KR102627697B1 - Skin diagnosis device using smart mirror and its driving method - Google Patents

Abstract

The present invention is a skin diagnosis device, which includes a device frame formed at the rear end of a mirror with a hollow portion of a certain size in the middle and an edge of a certain thickness, an LED module mounted on the four corners of the device frame, and the device frame. It consists of a camera module provided on the top, bottom, left, and right sides, a display module mounted in the hollow part of the device frame, and a lower cradle that supports the device frame and functions as a stand, and 2D and 3D images obtained by the camera module. It relates to a skin diagnosis device using a smart mirror, which can provide skin analysis and diagnosis results for the user's skin condition while displaying an image on the display module.

Description

Skin diagnosis device using a smart mirror and its operating method {SKIN DIAGNOSIS DEVICE USING SMART MIRROR AND ITS DRIVING METHOD}

The present invention relates to a skin diagnosis device, which provides skin diagnosis and skin care services while allowing the user to check 2D and 3D images of the skin condition captured using a 3D camera, an RGB camera, and an LED light source through a display device on the back of a smart mirror. It relates to a skin diagnosis device using a smart mirror that can be provided and a method of operating the same.

Recently, as the trend of focusing on appearance and cost-effectiveness has spread, the craze for ‘home care’, which involves taking care of one’s body and health at home, is growing stronger, and skin care is one of the main interests of modern people regardless of gender. However, it is true that receiving skin care or diagnosis every day is burdensome in terms of cost and time.

Accordingly, technologies are emerging that analyze the condition of your skin in the mirror you look at every day at home and provide solutions. A smart mirror is a mirror that not only performs its own function of reflecting the shape of an object using the reflection of light, but is also connected to the Internet and can simply output various information such as weather, calendar, and major news. It can be said to be a mirror, but it cannot reflect one's skin condition and provide customized information appropriate for it.

In addition, most 2D skin analysis products are capable of analyzing skin condition on the surface, such as analyzing spots, freckles, blemishes, and pigmentation using polarization images and UV images, but they also analyze wrinkles and scars based on 3D depth information. Accurate analysis is impossible.

In order to solve the above problem, in Korean Patent Publication No. 10-2020-00365656 (2020.04.07.), a two-dimensional image of the skin is obtained using a camera, and the two-dimensional image is obtained using a predetermined algorithm. Modeling, wherein the modeling step generates a phase shift map, removes the background, and performs height transformation based on data resolving the ambiguity of the phase shift map and phase map data of the reference plane to generate a three-dimensional image. As a 3D skin analysis method, a 3D skin scanning method is known, which allows analysis and diagnosis of skin by performing 3D modeling on the face, etc.

However, like the present invention, the conventional technology controls the user interface (UI) through user touch commands, voice commands, or gesture commands, and provides skin care through diagnosis through artificial intelligence-based complex data analysis based on the user's skin image. There is a difference between generating information and providing skin diagnosis service information.

The present invention was derived to solve the problems of the prior art described above. The purpose of the present invention is to function as a general mirror in normal times, scan the face or skin condition with a camera during skin diagnosis, and display the LCD display device. The goal is to provide a skin diagnosis device using a smart mirror that can display the user's skin and provide skin care services.

In addition, the present invention allows skin care service information to be provided according to commands through a user interface (UI) such as voice recognition, motion recognition, and touch recognition.

In addition, another purpose of the present invention is to provide skin condition information such as skin age and risk level based on artificial intelligence-based deep learning algorithm.

A skin diagnosis device using a smart mirror according to an aspect of the present invention to solve the above technical problem includes a device frame formed with a hollow portion of a certain size in the center at the rear end of the mirror and an edge of a certain thickness, and four sides of the device frame. An LED module mounted on an angle corner, a camera module provided on the top, bottom, left, and right sides of the device frame, a display module mounted on a hollow portion of the device frame, and a lower cradle that functions as a holder while supporting the device frame ( Cradle), and can display 2D and 3D images acquired by the camera module on the display module and provide skin analysis and diagnosis results for the user's skin condition.

In addition, the camera module of the present invention may be comprised of a 3D camera mounted at the top center of the device frame and an RGB camera mounted at the left, right, and bottom centers to capture the user's appearance from different directions.

In addition, the 3D camera of the present invention is a stereo camera method using two cameras, a structured light method using pattern lighting, or the emission time of a large amount of infrared dot light and the reflected return. This can be done using any one of TOF (Time Of Flight) 3D scan cameras that measure the blinking interval of light.

In addition, in the present invention, the center of the top, bottom, left, and right of the device frame is formed with a wing piece extending a certain length, and a camera is mounted on the wing piece so that the camera's angle of view can be arranged around the person's face. It can be.

In addition, the main control board including a control module, a communication module, a power module, and an audio input/output module is provided inside the cradle of the present invention, and the control module controls the shooting timing and timing of the 3D camera 141 and the RGB camera. Camera control unit that controls the shooting angle, etc., lighting control unit that controls the brightness, wavelength and duration of the LED module, screen control unit that controls the brightness and resolution of the display module, communication control unit for network connection and server linkage, and user through camera It may be characterized by comprising a motion recognition unit for detecting and processing gestures, etc., a voice recognition unit for detecting and processing the user's voice through a microphone device, and a touch recognition unit for controlling touch recognition of the touch film attached to the mirror. there is.

A method of driving a skin diagnosis device using a smart mirror according to another aspect of the present invention to solve the above technical problem is to control the brightness by operating lighting of a certain wavelength in an LED module provided in the device frame in response to a user's input command. , Entering the skin diagnosis mode to prepare for camera shooting for skin diagnosis by aligning the user's face with the virtual guide line provided by the display module and positioning it toward the camera, operating the 3D camera and RGB camera to detect the user's face or skin diagnosis area Camera shooting step to acquire the image, video confirmation step to confirm that the captured 2D and 3D video images are displayed through the display module on the back of the mirror after shooting, and viewing the captured image displayed through the LCD panel, which is the display module on the back of the mirror. It may include a skin diagnosis service step in which skin care services are provided from a skin care management server.

In addition, the present invention has the feature of classifying in detail the entire face area obtained from the image captured in the image confirmation step, giving weight to a specific area, or distinguishing and displaying the area of the area more clearly.

In addition, the present invention has the feature of providing a real-time chat window through a user interface in the skin diagnosis service step, and allowing feedback on advice and diagnosis results through real-time chat with experts other than designated hospitals and attending physicians.

In addition, in the present invention, in the skin diagnosis service step, an artificial intelligence (AI)-based diet control service, recommended exercise amount service, and makeup method suggestion service according to the skin condition diagnosis results can be provided through the diagnosis information service window of the display.

In addition, the skin diagnosis mode entry step, image confirmation step, and skin diagnosis service step of the present invention are characterized in that the user interface (UI) is controlled according to the recognition of any one of the user's touch command, voice command, or gesture command.

The present invention, based on the skin diagnosis device using the above-described smart mirror and its driving method, positions each camera to the outside so that the distance is wider, so that the camera's angle of view rotates around the person's face. This has the effect of securing high quality image data for the production of 3D skin modeling.

In addition, in the skin diagnosis service using the diagnostic device of the present invention, skin condition diagnosis data can be shared with a designated hospital, attending physician, or external expert, and when executing the real-time chat icon, advice and feedback can be received from skin experts. It has the effect of providing effective skin care.

1 is an exemplary diagram of a skin diagnosis device using a smart mirror according to an embodiment of the present invention.
Figure 2 is a front view showing the skin diagnosis device of Figure 1 in use.
Figure 3 is an example diagram showing the coupling relationship of the 3D scan skin diagnosis device according to Figure 1.
Figure 4 is a block diagram schematically showing the configuration of the main PCB and the modules included in the diagnostic device according to the present invention.
Figure 5a is a front view of a skin diagnosis device according to another embodiment of the present invention;
Figure 5b is an example diagram showing the use state of Figure 5a.
Figure 6 is a flowchart showing a method of operating a skin diagnosis device using a smart mirror according to the present invention.
FIG. 7 is an example diagram showing user usage status according to the driving order of the diagnostic device of FIG. 6.
Figure 8 is an exemplary diagram showing a skin diagnosis service screen in a display module according to the present invention.
Figure 9 is an example diagram showing a real-time chat screen in a skin diagnosis service.
Figure 10 is an example of receiving information on the recommended amount of exercise according to the results of body and skin condition check using AI according to the present invention.

Terms or words used in this specification and claims should not be construed as limited to their common or dictionary meanings, and the inventor may appropriately define the concept of terms in order to explain his or her invention in the best way. It must be interpreted with meaning and concept consistent with the technical idea of the present invention based on the principle that it is.

Therefore, the embodiments described in this specification and the configuration shown in the drawings are only one of the most preferred embodiments of the present invention and do not represent the entire technical idea of the present invention, and therefore, various equivalents that can replace them at the time of filing the present application It should be understood that variations and variations may exist.

Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the attached drawings.

Figure 1 is an exemplary diagram of a skin diagnosis device using a smart mirror according to an embodiment of the present invention, Figure 2 is a front view showing the use state of the device in Figure 1, and Figure 3 is a 3D scan skin diagnosis device according to Figure 1. This is an example diagram showing the combination relationship.

In one embodiment, the skin diagnosis device (hereinafter referred to as 'diagnostic device') using the smart mirror of the present invention according to FIGS. 1 to 3 functions as a regular mirror on the front and allows light sources on the back to be transmitted when the diagnostic device is operating. A device frame consisting of a working mirror 110, attached to and covered with the mirror 110, a hollow portion 121 of a certain size in the center, and an edge of a certain thickness for mounting a camera and an LED module. 120 and a lower cradle 170 that supports the device frame 120 and functions as a stand, and a power connection cable 180.

The mirror 110 can be said to be a half mirror in which a translucent mirror film is attached to the glass surface by attaching a very thin layer of metal such as nickel, aluminum, titanium, etc. in the form of a thin film. Because of this, the surface has a similar or identical color and reflectivity to a regular mirror, but if there is a bright light source on the back of the half mirror, the light is transmitted to the outside and can be seen.

Additionally, a touch recognition film is attached to the mirror 110 to enable recognition according to a user's touch operation, and a touch sensor may be provided accordingly.

At this time, LED modules 131 using light-emitting diodes (LEDs) are inserted and mounted at the four corners of the device frame 120 in the shape of an L or L at a certain length, and the light emitted from the LED module 131 is In order to diffuse the light, a light diffusion plate 132 is provided as a protective cover that covers the LED module 131 at the front end.

In addition, one or more camera modules 140 may be mounted at the top, left, right, and bottom center of the device frame 120 to scan the user's skin. In this case, the camera module 140 is installed at the top of the device frame 120 for 3D scanning. An RGB camera 142 may be installed at one or more of the left, right, and bottom sides of the camera 141.

Here, the 3D camera 141 may be in the form of a stereo camera that uses two cameras to recognize a three-dimensional sense of distance by imitating the human visual system, and can capture left and right images through a stereo vision system. By calculating disparity, 3D distance information and depth map can be obtained.

In addition, the 3D camera 141 measures the distance by analyzing the position of the pattern on the surface of the object by shining light with a pattern as a structured light method, or uses structured light using pattern lighting that can image the shape of the object. light) type 3D camera can be applied.

At this time, when applying a structured light type 3D camera, one of the two cameras in the stereo camera form is replaced with a projector that emits infrared structured light, and the light emitted by the infrared projector is replaced with a projector that emits infrared structured light. By analyzing the position of the pattern formed on the surface of this object using an algorithm and calculating a depth map in real time, it is possible to obtain 3D information.

In addition, the 3D camera 141 irradiates a large amount of infrared dot light to the facial skin or the recognition target skin to determine the distance or depth ( This is a method of acquiring a 3D image by measuring depth information, and can be applied with a 3D camera equipped with a TOF (Time Of Flight) receiving sensor. The TOF receiving sensor is an infrared image used to measure the phase difference according to time timing by ensuring that each cell consists of two pairs of detection sensors (receptors) and accurately synchronizes the flashing interval of the reflected light and the emitted light pulse. It can be called a sensor.

The present invention operates one of the 3D cameras described above on one side of the diagnostic device to obtain a skin scan image with a 3D structure, so that the 3D spatial structure and detailed skin condition of the face and skin area can be precisely measured. It will exist.

And the RGB camera 142 is a camera module consisting of either a CCD or CMOS sensor, and acquires 2D color images obtained by photographing the user's appearance from different directions. Accordingly, images taken with the RGB camera 142 from various sides of the face are mapped to images taken with the 3D camera 141, or the colors of the front and sides of the face are mapped based on 2D images taken from three locations. It can be used for precise analysis and diagnosis of the user's skin condition through analysis of texture components.

And the hollow part 121 of the device frame 120 is equipped with a display module 150 of a size corresponding to or smaller than the size of the hollow part 121 to display images captured or captured by the camera module 140. By displaying it, the user can check the condition of his or her face, skin, etc., and also receive the user's skin analysis results and diagnosis results.

For example, the display module 150 may be a 12.5-inch liquid crystal display (LCD) panel, or an organic light-emitting OLED (Organic Light Emitting Diode) display device.

The cradle 170 supports the lower part of the device frame 120 and serves as a stand. It serves as a cable connection terminal and an external wired network so that the power connection cable 180 of the diagnostic device can be connected to the LAN (Local Area Network). A terminal for connection may be provided, and the lower portion may be covered with a cradle cover 171.

In addition, inside the cradle 170, there is a main control board (Main PCB) 200 that controls the camera, display device, and lighting applied to the device frame 120, and interfaces with a skin care management server, etc. through an external network. It is provided.

Figure 4 is a block diagram schematically showing the configuration of the main PCB and the modules included in the diagnostic device according to the present invention. The main PCB 200 includes a control module 210, a communication module 220, a power module 230, and a voice It may include an input/output module 240.

The voice input/output module 240 can be said to include a microphone device that can receive a user's voice and a speaker device that outputs sound.

The communication module 220 may provide at least one access port (Ex, LAN port) to enable wired communication with other computing devices, or wireless communication such as WiFi, 3G, LTE, Bluetooth, etc. The method can be used to enable communication with a user's smartphone or external computing device.

Accordingly, the control module 210 can link with an external skin care management server through the communication module 220.

The skin care management server can be said to be a service platform that receives a user's skin image, creates skin care information through diagnosis based on artificial intelligence-based complex data analysis, and provides it to a diagnostic device.

Therefore, the control module 210 includes a camera control unit that controls the shooting timing and shooting angle of the 3D camera 141 and the RGB camera 142, and a lighting control unit that controls the brightness, wavelength, and duration of the LED module 131. , a screen control unit that controls the brightness and resolution of the display module 150, a motion recognition unit that detects and processes user gestures through a camera, and a user's voice is detected through a microphone device provided at a certain position of the cradle 170. It may include a voice recognition unit that processes and a touch recognition unit that controls touch recognition input from the mirror 110, and in addition, a communication control unit, lighting and display for network connection and server interworking through the communication module 220. It may further include a power control unit for controlling the on/off of the module and a function for controlling the internal storage device.

The control module 210 receives skin care information generated from the skin care management server, and when the user checks the condition of the face, skin, etc. through the display module 150, the control module 210 reports the user's skin analysis results and diagnosis results as a voice. It may further include a voice output unit that synthesizes and outputs the sound.

The control module 210 consists of a combination of hardware and software components, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, and a field programmable gate array (FPGA). ), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The control module 210 may execute an operating system (OS) and one or more software applications executed on the operating system.

FIG. 5A is a front view of a skin diagnosis device according to another embodiment of the present invention, and FIG. 5B is an exemplary diagram showing a state of use of FIG. 5A.

As shown in FIGS. 5A and 5B, the skin diagnosis device of the present invention has wing pieces 122a and 122b extending a certain length from the top, bottom, left and right sides of the device frame 120. At this time, the extending wing pieces 122a and 122b are formed to protrude further forward from the side of the mirror 110.

Accordingly, when the 3D camera 141 is mounted on the wing piece 122a extending upward, and the RGB camera 142 is mounted on the wing piece 122b extending left, right, and lower, the angle of view of the camera is centered on the person's face. It can be placed around the .

This is different from the embodiment shown in Figure 1, where each camera is positioned to the outside with a wider distance, and the camera's angle of view can be placed in a structure that rotates around the person's face, so it can create a three-dimensional image. This has the effect of securing high-quality image data for the production of skin modeling.

At this time, except for the position of the camera mounted on the wing pieces 122a and 122b, the functions and configurations of other devices and modules of the present invention are the same.

FIG. 6 is a flowchart showing a method of driving a skin diagnosis device using a smart mirror according to the present invention, and FIG. 7 is an example diagram showing a user usage state according to the driving order of the diagnosis device of FIG. 6.

Figure 8 is an exemplary diagram showing a skin diagnosis service screen in a display module according to the present invention, and Figure 9 is an exemplary diagram showing a real-time chat screen in the skin diagnosis service.

As shown in the figure, the operating method of the skin diagnosis device of the present invention includes a skin diagnosis mode entry step (S100), a camera shooting step (S200), an image confirmation step (S300), and a skin diagnosis service step (S400).

First, in normal mode, the mirror of the skin diagnosis device operates only as a regular mirror and is waiting for the user's input command.

That is, the skin diagnosis mode entry step (S100) can be entered by recognizing a touch command, voice command, or gesture command indicating the user's entry into the skin diagnosis mode by the control module in the normal mode.

The gesture command recognizes the hand trace information through a 3D camera using a stereo vision system, or applies a hand tracking model into three categories of static, dynamic, and trigger gestures according to the hand movement through a color (RGB) camera to provide an action command. It can be said to recognize .

Additionally, the cradle of the skin diagnosis device may be equipped with a user-operated input switch including an On/Off switch. Accordingly, it is possible to enter the skin diagnosis mode and perform other user commands through the input switch.

Here, the skin diagnosis mode is to control the brightness by operating lights of a certain wavelength from the LED modules provided at the four corners of the device frame in order to take camera photos for the user's skin diagnosis, and the user controls the brightness of the skin diagnosis target such as the face. It can be said to be a mode in which the camera is aligned according to the virtual guide line provided by the display module and positioned toward the camera to prepare for shooting.

In the camera shooting step (S200), when the user's position is fixed in the skin diagnosis mode entry step (S100) and the shooting angle is set according to the distance between the camera and the user, the 3D camera and the RGB camera are operated to capture the user's face or skin diagnosis unit. This can be said to be the stage of acquiring an image.

In the video confirmation step (S300), the user's face is photographed using 3D and RGB cameras on the top, bottom, left, and right sides of the mirror and an LED light source, and then the captured video image is displayed on the 2D and 3D screen through the LCD panel on the back of the mirror, and the user This can be said to be a confirmation step.

At this time, the diagnostic device of the present invention can divide the entire face area secured by the captured image into the forehead area, left and right eye area, left and right cheek area, left and right nasolabial area around the nose and mouth, and chin and neck area.

In other words, facial skin areas obtained from images captured with 3D cameras and RGB cameras can be divided by area and weighted to specific areas for more detailed diagnosis, or areas of specific areas can be distinguished and expressed more clearly. .

Additionally, the diagnostic device of the present invention transmits the captured image to the skin care management server through a communication module. In addition, the skin care management server can receive images of the user's skin, generate skin care information through diagnosis based on artificial intelligence-based complex data analysis, and provide skin diagnosis service information to a diagnostic device.

delete

The skin diagnosis service step (S400) is a step of receiving skin care services from the skin care management server while viewing the captured user image displayed through the LCD panel, which is a display module on the back of the mirror, as shown in FIGS. 7 to 9. You can.

At this time, the user interface (UI) displayed on the screen to receive skin care services may be divided into a menu icon and a diagnostic information service window.

delete

Figure 8 shows that data analyzed for the current skin condition is displayed in the form of a graph in the diagnosis information service window by touching the skin condition icon on the skin diagnosis service screen or following a command through voice or gesture.

In other words, through diagnosis of skin type, wrinkles, sebum, sensitivity, etc. analyzed with an artificial intelligence-based deep learning algorithm, it can be confirmed that information such as skin age and risk is displayed in a graph.

Additionally, a data sharing icon can be provided that allows users to share skin condition diagnosis data with a designated hospital, attending physician, or external expert and receive feedback. In addition, skin condition diagnosis data can be shared by linking to a smartphone application and providing a social networking service (SNS) sharing icon.

Figure 9 is an example diagram showing a real-time chat screen that receives advice and feedback from skin experts when running the real-time chat icon in the skin diagnosis service.

In addition, Figure 10 is an example of receiving information on the recommended amount of exercise according to the results of body and skin condition check using AI according to the present invention.

At this time, even in the skin diagnosis service step (S400), the user interface (UI) can be controlled through a user touch command, voice command, or gesture command.

As described above, preferred embodiments of the present invention have been described in the detailed description of the present invention, but those skilled in the art will understand without departing from the spirit and scope of the present invention as set forth in the following claims. It will be understood that various modifications and changes may be made to the present invention.

110: Mirror 120: Device frame
121: hollow section 122a, 122b: wing section
131: LED module 132: Light diffusion plate
140: Camera module 141: 3D camera
142: RGB camera 150: Display module
160: Rear cover 170: Cradle
171: Cradle cover 180: Power connection cable
200: Main PCB 210: Control module
220: communication module 230: power module
240: Voice input/output module

Claims (10)

A device frame formed by a hollow portion of a certain size in the center at the rear end of the mirror and a border of a certain thickness,
LED modules mounted on four corners of the device frame,
Camera modules provided on the top, bottom, left, and right sides of the device frame,
A display module mounted in the hollow part of the device frame and
It consists of a lower cradle that supports the device frame and functions as a stand,
2D and 3D images acquired by the camera module are displayed on the display module, and skin analysis and diagnosis results regarding the user's skin condition are provided,
The camera module consists of a 3D camera mounted at the top center of the device frame and an RGB camera mounted at the left, right, and bottom centers to capture the user's appearance from different directions,
The 3D camera is a 3D scan using a structured light method using pattern lighting or a TOF (Time Of Flight) method that measures the emission time of a large amount of infrared dot light and the blinking interval of the reflected light. Made with a camera,
A wing piece extending a certain length is formed at the center of the top, bottom, left, and right sides of the device frame, and the camera module is mounted on the wing piece so that the angle of view of the camera module can be arranged around the person's face. A skin diagnosis device using a smart mirror.
delete delete delete In claim 1,
Inside the cradle, a main control board including a control module, a communication module, a power module, and an audio input/output module is provided,
The control module is
A camera control unit that controls the shooting timing and shooting angle of the 3D camera 141 and the RGB camera,
A lighting control unit that controls the brightness, wavelength, and duration of the LED module,
A screen control unit that controls the brightness and resolution of the display module,
Communication control unit for network connection and server interworking,
A motion recognition unit to detect and process user gestures through a camera,
A voice recognition unit that detects and processes the user's voice through a microphone device, and
A skin diagnosis device using a smart mirror, comprising a touch recognition unit that controls touch recognition of a touch film attached to the mirror.
delete delete delete delete delete