KR102543172B1 - Method and system for collecting data for skin diagnosis based on artificail intellience through user terminal - Google Patents

Abstract

As a skin diagnosis system according to an embodiment for realizing the above object of the present invention, a terminal including a lighting unit capable of irradiating light to a user and a camera unit configured to photograph a user's face image, and a predetermined An artificial neural network model is trained using a 3D modeled learning 3D face image corresponding to a plurality of unspecified learning 2D face images including quality information and the learning 2D face image, and based on the learned artificial neural network model, the and a central server configured to convert a user's face image received from a terminal into a three-dimensional 3D face image, wherein the terminal includes a user's image data input in real time to the camera unit based on the artificial neural network model. and detecting quality information of the face image, and photographing the user's face image at a specific viewpoint if the quality information meets a predetermined criterion.

Description

Method and system for collecting data for skin diagnosis based on artificial intelligence through user terminals

The present invention relates to a method and system for collecting data for skin diagnosis based on artificial intelligence through a user terminal, and more particularly, to a user for obtaining a high-quality facial image suitable for skin diagnosis by utilizing a deep learning-based object recognition technology. The invention relates to a method and system for collecting data for skin diagnosis based on artificial intelligence through a terminal.

Recently, electronic communication technology has developed in relation to the Internet of Things (IoT) and smart home, and skin care is possible at a relatively low cost. After diagnosing the skin through a skin diagnosis device directly at home, people want to receive customized cosmetics recommendations. More and more people are doing it.

Reflecting this trend, various skin diagnosis devices including smart mirrors have recently been released. The smart mirror is a device that utilizes a mirror capable of recognizing a user's motion or voice as a display, and is used to diagnose the user's skin through analysis of a facial image taken by a camera.

Recently, smart mirrors with lighting functions have been released to reinforce these diagnosis functions, but unless they are professional skin diagnosis devices, there are bound to be limitations in accurately diagnosing skin conditions due to light spread or light reflection caused by external light. does not exist. In order to solve this problem, various smart mirrors capable of controlling lighting have been released, but lighting control is inevitably difficult according to changes in external conditions and environment, and due to different lighting specifications, depending on the manufacturer or specification of the smart mirror, skin diagnosis There are limitations that inevitably result in different results.

In addition, in terms of accuracy, overall skin conditions such as rashes can be grasped, but there is inevitably difficulty in precisely diagnosing lesions such as swelling in a local area and water warts. Therefore, in order to accurately diagnose skin based on facial images measured at home, it is necessary to collect high-quality facial images at the central server level, and based on this, a system that can perform unified skin diagnosis regardless of the type of smart mirror this is required

Republic of Korea Patent Publication No. 10-2022-0088219, June 27, 2022

One object of the present invention is based on a face image acquisition model managed by a central server regardless of the type of user terminal including a smart mirror, the location, size, brightness, and A high-quality face image with focus applied can be obtained.

In addition, one object of the present invention is to collect high-quality face images including a predetermined face direction from a smart mirror based on a face image acquisition model that has learned a face image including a predetermined face direction, and to collect different faces from each other. A three-dimensional face image is obtained based on a plurality of high-quality face images including directions.

As a skin diagnosis system according to an embodiment for realizing the above object of the present invention, a terminal including a lighting unit capable of irradiating light to a user and a camera unit configured to photograph a user's face image, and a predetermined An artificial neural network model is trained using a 3D modeled learning 3D face image corresponding to a plurality of unspecified learning 2D face images including quality information and the learning 2D face image, and based on the learned artificial neural network model, the and a central server configured to convert a user's face image received from a terminal into a three-dimensional 3D face image, wherein the terminal includes a user's image data input in real time to the camera unit based on the artificial neural network model. and detecting quality information of the face image, and photographing the user's face image at a specific viewpoint if the quality information meets a predetermined criterion.

In one embodiment of the present invention, the predetermined quality information may include at least one of position, size, brightness, and focus information of a face in the face image.

In one embodiment of the present invention, the lighting unit is configured to control at least one of a light amount and an irradiation angle, and when the quality information does not satisfy the predetermined criterion, the lighting unit controls the light amount or the irradiation angle can control.

In one embodiment of the present invention, the training 3D face image may be generated from the training 2D face images for different viewpoints.

In one embodiment of the present invention, the learning 3D face image is a 3D image obtained from a three-dimensional skin precision diagnostic device that obtains continuous 3D face images at various viewpoints by rotating the camera while the face to be studied is fixed, wherein the A training 2D face image may be extracted for each viewpoint from the training 3D face image.

In one embodiment of the present invention, the artificial neural network model additionally learns lesion information of a face included in a training 2D face image, and the terminal detects a lesion included in the user's face image, and the lesion Based on the location information of , it is possible to determine the specific viewpoint.

In one embodiment of the present invention, the specific viewpoint may include a plurality of different viewpoints.

In one embodiment of the present invention, the specific viewpoint may be the front or both sides of the user's face.

In one embodiment of the present invention, the artificial neural network model may be learned based on the learning 2D face images of different viewpoints including the specific viewpoint.

In one embodiment of the present invention, the artificial neural network model additionally learns viewpoint information of a learning 2D face image, and the terminal detects viewpoint information of a user's face image in the video data, and from the specific viewpoint When it is within a predetermined error range, video recording is started by controlling the camera unit, and when the face image at the specific time point is included in a video frame while the user moves the angle of the face, video recording is stopped, and the video recording is stopped. An image frame including a face image of a viewpoint may be extracted.

According to an embodiment of the present invention, regardless of the type of user terminal, based on a face image acquisition model managed by a central server, the position, size, brightness, and focus of a face corresponding to the quality of an image learned from a smart mirror A high-quality face image to which the back is applied may be obtained.

In addition, according to an embodiment of the present invention, a high-quality face image including a predetermined face direction can be collected from a smart mirror based on a face image acquisition model that has learned a face image including a predetermined face direction. , It is possible to obtain a three-dimensional face image based on a plurality of high-quality face images including different face directions. Accordingly, accurate skin diagnosis can be performed by precisely analyzing the condition of the lesion in the local area using a 3D image.

1 is a block diagram for explaining a skin diagnosis system according to an embodiment of the present invention.
FIG. 2 is block diagrams for explaining the skin diagnosis terminal of FIG. 1 .
3 is an exemplary diagram for explaining acquiring a user's face image based on a face image acquisition model learned using deep learning.
4 is another exemplary diagram for explaining acquiring a user's face image based on a face image acquisition model learned using deep learning.
5 is a block diagram illustrating a central server that converts a received 2D face image into a 3D face image using deep learning technology.
6 is a block diagram illustrating generating a face image acquisition model using deep learning technology.
7 is an exemplary view of converting a 2D face image into a 3D face image.
8 is a flowchart illustrating a method of obtaining a 3D face image for skin diagnosis according to an embodiment of the present invention.
9 is a flowchart illustrating a method of photographing a face image for skin diagnosis according to an embodiment of the present invention.

Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the accompanying drawings, but the same or similar components are given the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. The suffixes "module" and "unit" for components used in the following description are given or used together in consideration of ease of writing the specification, and do not have meanings or roles that are distinct from each other by themselves. In addition, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of a related known technology may obscure the gist of the embodiment disclosed in this specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in this specification, the technical idea disclosed in this specification is not limited by the accompanying drawings, and all changes included in the spirit and technical scope of the present invention , it should be understood to include equivalents or substitutes.

Terms including ordinal numbers, such as first and second, may be used to describe various components, but the components are not limited by the terms. These terms are only used for the purpose of distinguishing one component from another.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

In this application, terms such as "comprise" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

In this specification, a "unit" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Further, one unit may be realized using two or more hardware, and two or more units may be realized by one hardware.

In this specification, some of the operations or functions described as being performed by a terminal, device, or device may be performed instead by a server connected to the terminal, device, or device. Likewise, some of the operations or functions described as being performed by the server may also be performed by a terminal, apparatus, or device connected to the server.

The present invention can be implemented as computer readable code on a computer readable recording medium, and the computer readable recording medium includes all types of recording devices storing data that can be read by a computer system. . Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. In addition, the computer-readable recording medium may be distributed to computer systems connected through a network, so that computer-readable codes may be stored and executed in a distributed manner.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram for explaining a skin diagnosis system according to an embodiment of the present invention.

In the skin diagnosis system 1000 according to an embodiment, in capturing a user's face image using the skin diagnosis terminal 150, a face image of a desired quality is captured based on an image acquisition model in which a face image of a desired quality has been learned in advance. It may be a system that That is, the skin diagnosis terminal 150 is configured to detect the quality and view point information of a face image in image data input through a camera by utilizing object recognition technology using a deep learning algorithm. For example, as shown in FIG. 1 , the skin diagnosis terminal 150 detects the user's face image from different viewpoints when the quality information of the face image previously learned through the camera is detected. It is configured to photograph and transmit the obtained user's face image to the central server (300). The skin diagnosis system 1000 is configured to convert a 2D face image obtained through the skin diagnosis terminal 150 into a 3D face image for precise diagnosis of the user's skin.

The skin diagnosis system 1000 includes a user terminal 100 and a central server 300 . However, since the skin diagnosis system 1000 of FIG. 1 is only an embodiment of the present invention, the present invention is not limitedly interpreted through FIG. 1 .

For example, each component in FIG. 1 is generally connected through a network 10 . That is, as shown in FIG. 1 , at least one user terminal 100 may be connected to the central server 300 through the network 10 . In addition, the skin diagnosis system 1000 according to the present embodiment may further include a skin diagnosis terminal 150 for capturing a user's face image. For example, the skin diagnosis terminal 150 may be a smart mirror. The smart mirror is a device that utilizes a mirror capable of recognizing a user's motion or voice as a display, and may be configured to capture a user's face image through a camera. Although the user terminal 100 and the skin diagnosis terminal 150 have been described as independent and separate components, they are not limited thereto. For example, the user terminal 100 may be the skin diagnosis terminal 150 or may be included in the skin diagnosis terminal 150 .

Here, the network means a connection structure capable of exchanging information between nodes such as a plurality of terminals and servers, and examples of such networks include RF, 3rd Generation Partnership Project (3GPP) network, and Long Term Evolution (Term Evolution) network, 5GPP (5th Generation Partnership Project) network, WIMAX (World Interoperability for Microwave Access) network, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network) ), PAN (Personal Area Network), Bluetooth (Bluetooth) network, NFC network, satellite broadcasting network, analog broadcasting network, DMB (Digital Multimedia Broadcasting) network, etc. are included, but are not limited thereto.

In the following, the term at least one is defined as a term including singular and plural, and even if at least one term does not exist, each component may exist in singular or plural, and may mean singular or plural. It will be self-evident. In addition, the singular or plural number of each component may be changed according to embodiments.

The user terminal 100 is a computer or portable terminal owned by a person (hereinafter referred to as 'user') who intends to perform skin diagnosis on his or her face through the skin diagnosis system 1000, and the user can access the web ( It may be a terminal provided in the form of a web, application, or web app, which communicates with the central server 300 through the network 10 to sign up for membership, and then inputs user information. The user terminal 100 is connected to the skin diagnosis terminal 150 by wire or wirelessly, manages facial images taken through the skin diagnosis terminal 150, and communicates with the central server 300 to provide user information. It is configured to transmit the face image of the central server (300).

Here, the user terminal 100 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop, a desktop, a laptop, and the like equipped with a navigation system and a web browser. At this time, the user terminal 100 is, for example, a wireless communication device that ensures portability and mobility, navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) It may include all types of handheld-based wireless communication devices such as terminals, smartphones, smart pads, tablet PCs, and the like.

The skin diagnosis terminal 150 may be connected to the user terminal 100 through wired or wireless communication, and detect the quality of the user's face image based on the face image acquisition model generated by the central server 300. It consists of In addition, the skin diagnosis terminal 150 is configured to detect a user's face image at different viewpoints learned based on the face image acquisition model and capture a plurality of box face images taken at different viewpoints. Since the skin diagnosis terminal 150 captures a user's face image based on the face image acquisition model generated by the central server 300, it does not matter what type of product it is. For example, the skin diagnosis system 1000 can be applied to different skin diagnosis terminals 150a and 150b.

In addition, the plurality of facial images captured by the skin diagnosis terminal 150 are transmitted to the central server 300 via the user terminal 100, and the central server 300 precisely analyzes the user's skin. For this, it is configured to be converted into a three-dimensional 3D face image. Hereinafter, the skin diagnosis terminal 150 will be described in detail with reference to FIGS. 2 and 3 .

FIG. 2 is block diagrams for explaining the skin diagnosis terminal of FIG. 1 . 3 is an exemplary diagram for explaining acquiring a user's face image based on a face image acquisition model learned using deep learning.

1 to 3 , the skin diagnosis terminal 150 may include a camera unit 151, a lighting unit 152, a detection unit 153, an output unit 154, and a control unit 155.

The camera unit 151 is configured to capture at least one face image of the user. Specifically, the camera unit 151 is configured to process a plurality of frames (images) in one second, and is configured to acquire image data including at least one frame, and convert the obtained image data to the camera unit ( 151) and is configured to output to the connected detection unit 153.

On the other hand, in defining terms used in this specification, based on the time point of image capture, an image projected on a camera before the start of capturing is referred to as a "preview image" and a captured image is referred to as a "captured image". do. After shooting is finished, the "captured image" continues with the "preview image". In addition, "camera image" is used as a general term for "preview image" and "photographed image".

The camera unit 151 is a camera module configured to capture a user's face image, and includes one or more sensors (eg, a front sensor or a rear sensor), a lens, an image signal processor (ISP), or a flash (eg, a front sensor). LED or xenon lamp, etc.) may be included. For example, the camera unit 151 may include a plurality of cameras, which may have different focal points or viewing angles. That is, the plurality of cameras of the camera unit 151 may be disposed on the left and right sides and/or above and below the skin diagnosis terminal 150 in order to correct lens aberrations of each other.

Also, the camera unit 151 may include a camera having a zoom function (eg, a zoom-in and/or zoom-out function). In addition, the camera unit 151 may be configured as a general camera, a depth camera capable of capturing a depth image, or a high frame rate camera capable of high-speed capturing.

The camera unit 151 is configured to convert a preview image projected through the camera into image data and transmit the image data to the detection unit 153 in real time. At this time, the detection unit 153 recognizes the quality of the learned face image based on a pre-learned face image acquisition model in capturing the user's face image from the video data input in real time, and recognizes the pre-learned specific quality of the face image. When a face image of a viewpoint is detected, it is recognized as a trigger, and the controller 155 controls the camera unit 151 to take a picture. That is, the camera unit 151 is configured to photograph a subject, that is, a user's face, under the control of the controller 155 .

In addition, the camera unit 151 may be configured to include an auto-focus function and an auto-flash function in order to obtain clear image data. Alternatively, the camera unit 151 may be configured to focus or emit a flash based on the reliability of an object detected on the image data transmitted to the detection unit 153 . Accordingly, the camera unit 151, together with the lighting unit 152 described later, can minimize the influence from the environment in obtaining clear image data, detect objects in the image data with high reliability, and It is possible to obtain a high-quality face image.

The lighting unit 152 is configured to emit light so that the user's face image can be captured in a desired quality state. That is, the lighting unit 152 may include a plurality of LEDs capable of irradiating light. For example, a plurality of LEDs provided to the lighting unit 152 may be disposed along the circumference of the skin diagnosis terminal 150 .

The quality of an image required for skin diagnosis is optimized through the lighting unit 152 and lighting under constant conditions for external changes can be provided, so that accurate skin diagnosis can be made. In addition, the lighting unit 152 may be provided to adjust the light amount and angle so that the light of the LED can be irradiated to a desired position on the user's face. Accordingly, in consideration of the shadow cast on the user's skin, the light emitted from the lighting unit 152 may be irradiated to remove the shadow.

The detection unit 153 is configured to detect an object using a deep learning algorithm. For example, the detection unit 153 automatically detects an object at regular time intervals for image data input from the camera unit 151, and applies a tracking technology using a deep network to the detected object for real-time processing. , and design and train a deep network by applying a method of minimizing an area in which an object is searched. The deep network may be a neural network based on a deep neural network (DNN)-based deep learning algorithm, and the neural network is composed of an input layer, one or more hidden layers, and an output layer. . Here, the deep network may be applied to a neural network other than DNN, and for example, a neural network such as a Convolution Neural Network (CNN) or a Recurrent Neural Network (RNN) may be applied.

The detection unit 153 is configured to detect an object from at least one frame of input image data. For example, the detector 153 may detect an object by applying a video-based deep network to a plurality of consecutive frames of image data in the preview image or a predetermined number or more frames randomly extracted from them. In this case, motion of an object detected from adjacent frames may be tracked. Alternatively, the detection unit 153 may select a representative frame from among the input image data and detect the object by applying an image-based deep network based on the representative frame. For example, the representative frame may be a frame obtained when a face is detected in the preview image of the camera unit 151 through a separate sensor or the like.

Also, although not shown, the image data obtained from the camera unit 151 may be configured to be gray converted and equalized before being input to the detection unit 153 . Here, gray conversion is to convert an image with 3-channel color into a gray scale image, which is for effective smoothing processing. This is to give, that is, to widen the brightness distribution. Accordingly, it is possible to detect a face image with high reliability within the image data. Meanwhile, various well-known image processing techniques may also be applied to the image data, and a detailed description thereof will be omitted. In addition, it has been described as an example that the image data is processed before being input to the detection unit 153, but is not limited thereto.

For example, when the quality information of the face image pre-learned in the generation of the face image acquisition model is detected in the image data, the detection unit 153, when the quality information meets a predetermined criterion, the user at different time points. It is configured to detect a face image of. Specifically, the detection unit 153 may include a quality detection unit 153a that detects the quality of a face image and a face detection unit 153b that detects pre-learned face images of different viewpoints.

The quality detection unit 153a detects the quality of image data including a face image for accurate skin diagnosis. The quality detection unit 153a automatically obtains quality information such as the location, size, brightness, and focus of the face of the image data input from the camera unit 151 based on the face image acquisition model using a deep network. configured to extract. That is, the quality detector 153a may detect the quality of a face image in received video data based on the quality of a face image, which is the learning data of the face image acquisition model.

For example, the quality detection unit 153a is configured to detect the position of the face based on the detected face image, whether the face is placed in the center of the image data or whether a part of the face shape is cut off. In addition, based on the size of the face image as learning data, it is configured to detect whether the face size in the image data is a desired face size. For example, the quality detector 153a may extract size information of a face, which is an object to be detected, through a distance away from a camera. In addition, the quality detection unit 153a determines the location of light quantity, shade (shadow), light smearing, and light reflection of the face image in the image data according to the external environment, based on the illuminance and sharpness of each part of the face image, which is learning data. configured to detect.

Here, the light spread is due to the oil of the user's face, and the light reflection means that light is reflected by an external light source regardless of the illumination of the lighting unit 152 . That is, when light smearing occurs in the face image, the sharpness of the generated area is lowered, and when light reflection occurs, the generated area appears in white. That is, the quality detection unit 153a is configured to measure the amount of light according to the external environment or condition, and to detect whether light spread or light reflection occurs in the face image. In addition, based on the training data, it is possible to detect whether or not a face image in the image data is in focus. However, it is not limited thereto. For example, whether or not focus is correct is not detected by the quality detection unit 153a, but can be automatically focused according to an auto focus function of the camera unit 151.

In addition, the quality detection unit 153a may detect whether conditions required for precise skin diagnosis are satisfied, for example, whether the forehead is exposed by raising the bangs or whether a foreign substance is present in the face image. That is, the quality detector 153a may detect the quality of the face image in the video data based on the face image as learning data.

The face detection unit 153b is configured to detect a face in image data. The face detection unit 153b automatically determines whether or not a face is included in the video data input from the camera unit 151 and viewpoint information of the face image in the video data based on the face image acquisition model using a deep network. configured to extract. That is, the quality detector 153b may detect viewpoint information of a face image in received image data based on a face image at a specific viewpoint, which is learning data of the face image acquisition model.

For example, the face detection unit 153b determines viewpoint information of a face image on image data based on a face image detected according to specific viewpoint information, which is learning data, for example, whether it is a front face, a left face, a right face, or a 45-degree tilt. It is configured to detect whether or not the

The output unit 154 is configured to provide feedback to the user under the control of a control unit 155 to be described later. For example, the output unit 154 may include a display unit or a speaker unit. That is, in providing feedback, the skin diagnosis terminal 150, for example, a smart mirror, may output feedback on a mirror or output feedback through a speaker. For example, in order to capture a user's face image at a specific viewpoint, the output unit 154 provides feedback about the rotation of the face angle to the user based on the viewpoint information of the face image recognized by the detection unit 153b. can be printed out. Since it is impossible for the user to recognize the angle of the face and pose, the user can capture a face image at a desired viewpoint by finely adjusting the angle of the face based on the feedback from the output unit. Meanwhile, the face angle includes both up and down and left and right angles. Meanwhile, the face angle here is a numerical value according to a specific criterion through learning face image learning, and does not mean a face angle consciously manipulated by an actual user. For example, an up and down angle at which the forehead and the jaw line are perpendicular to the side may be set as the reference angle, and the reference angle may be reached by relatively manipulating the head according to the user. That is, analysis data of a common shape for precise skin diagnosis can be obtained by detecting and capturing a face image at a point in time to be photographed based on artificial intelligence, regardless of physical characteristics such as a user's turtle neck.

The control unit 155 is configured to output a feedback signal to the output unit 154 based on information detected by the detection unit 153 .

For example, if the quality information detected by the quality detector 153a does not meet a predetermined criterion, for example, the controller 155 determines that the face is not located in the center of the image data based on the face image. In this case, the output unit 154 may output feedback about the request for moving the position of the face. In addition, based on the size of the face image in the image data or the distance of the face as an object, feedback to move closer or farther may be output.

In addition, the control unit 155 controls the lighting unit 152 when a low amount of light or shade (shadow) of the face image in the video data is detected by the quality detection unit 153a, so that the LED is positioned at a desired position on the user's face. It is configured to control the amount of light or the irradiation angle so that the light of the light can be irradiated. For example, when a plurality of LEDs are installed at different positions, the amount of light of the LEDs may be individually controlled according to the position where the shade is generated. Individual control targets may be programmed in advance according to the shadow occurrence position. Accordingly, it is possible to remove shadows by controlling light for an insufficient amount of light or shadows on the user's face image. However, it is not limited thereto. The control unit 155 may output feedback for a user's manipulation request for adjusting the amount of light.

In addition, the control unit 155 may output feedback suggesting face washing and diagnosis through the output unit 154 when light smearing is detected in the face image in the video data by the quality detection unit 153a. there is. In addition, when the quality detection unit 153a detects light reflection in the face image in the video data, the output unit 154 may output feedback suggesting avoiding a position where an external light source is reflected by moving the position. there is.

In addition, when the face detection unit 153b detects viewpoint information of the user's face image in the video data, the control unit 155 outputs feedback instructing the user to rotate the user's face to a viewpoint corresponding to a specific viewpoint of the learned data. can do. The specific viewpoint is assumed to be two or more, and may include, for example, a total of three viewpoints of the user's front and left and right sides. That is, the controller 150 is configured to capture an image by controlling the camera unit 151 when the viewpoint of the user's face image corresponds to a predetermined specific viewpoint. That is, when there are a plurality of specific viewpoints, the controller 155 may output feedback for guiding rotation of the user's face to the next viewpoint while sequentially capturing images at each specific viewpoint. Accordingly, as shown in FIG. 3 , three face images of the front and both sides of the user may be acquired.

Also, the controller 155 may control overall operations of the skin diagnosis terminal 150 . For example, the control unit 155 controls the detection unit 153 to obtain a face image based on an artificial neural network model for a desired face within image data of a preview image received in real time from the camera unit 151. When the quality of the image is detected, the camera unit 151 is configured to control the camera unit 151 by determining whether it is a specific face angle, which is a preset trigger in capturing a user's face image in the camera image.

Specifically, the controller 155 extracts features from a plurality of frames of the image data to generate feature maps, calculates them, and calculates the position of the user's face and the face image in the camera image in real time. The quality is tracked, and if the quality is satisfied, it is configured to control the camera unit 151 by determining whether or not a specific angle, which is a preset trigger, is applied based on viewpoint information of the user's face image. Accordingly, when the user moves the face according to the feedback from the output unit without the need to fix the face at a specific face angle, the specific face angle triggers the photographing. However, it is not limited thereto. The viewpoint information of the user's face image in the video data is detected, and when it is within a predetermined error range from the specific viewpoint, the camera unit is controlled to start video recording, and within the image frame while the user moves the angle of the face. If the face image of the specific point in time is included, video recording may be stopped, and an image frame including the face image of the specific point in time may be extracted and obtained.

The control unit 155 may extract a region of interest (ROI) from a characteristic map of a frame of image data, and also determines whether there is a preset trigger in capturing a facial image of a region of interest, which is at least a part of a frame of image data. It can be set as the main area for judgment. For example, the controller 155 may set the main area to include the area where the user's face is recognized in the frame of the image data.

On the other hand, the generation and operation of the feature map is a layer implemented based on a deep neural network (DNN) consisting of an input layer, one or more hidden layers, and an output layer, for example, convolution Layers implemented based on convolution neural networks (CNNs) and/or recurrent neural networks (RNNs), rectified linear unit (RELU) layers, pooling layers, bias add ) layer, softmax layer, etc., may be implemented by a computation model including various computation layers.

Here, an artificial neural network (ANN) represents a computation model implemented in software or hardware that imitates the computational capability of a biological system by using a large number of artificial neurons connected by connection lines. In the artificial neural network, artificial neurons that simplify the functions of biological neurons are used. In addition, human cognitive function or learning process is performed by interconnecting them through connection lines having connection strength. That is, the calculation model used in the detection unit 153 may be an artificial neural network model in which quality and viewpoint information of a face image are learned.

That is, when the quality information of the face image previously learned through the camera is detected, the controller 155 captures the user's face image at different points in time when the quality information meets a predetermined criterion, and acquires the user's face image. It is configured to transmit face images at different viewpoints to the central server 300. On the other hand, in the present embodiment, based on the face image acquisition model, the quality of the face image and viewpoint information are detected in the image data as an example, but it is not limited thereto. For example, a quality learning model that learns the quality of a face image and a viewpoint learning model that learns viewpoint information of a face image can be distinguished. That is, after detecting the quality of the face image based on the quality learning model by the ensemble technique, camera shooting can be controlled at a specific viewpoint learned based on the viewpoint learning model. In addition, a face image may be captured based on a separate learning model for each viewpoint.

In addition, the control unit 155 according to the present embodiment has been described as capturing an image by controlling the camera unit 151 when the viewpoint of the user's face image corresponds to a predetermined specific viewpoint, which is learning data, but is limited to this. It doesn't work. For example, when a lesion is detected on the user's face image, the control unit 151 may control photographing by determining viewpoint information to be photographed according to the detected location. For example, referring to FIG. 4, when a lesion A is detected on the user's left cheek, the user's left cheek is rotated θ degrees to the right, for example, 45 degrees, so that the user's left cheek faces the front of the camera. A photographing time point may be determined as a time point rotated by 135 degrees further rotated by 90 degrees to the right. This is to determine the size (width) and protrusion (height) of the lesion (A) for precise skin diagnosis. to make this easier. For example, when there are multiple locations of lesions, the imaging time point may be determined according to the number of lesions, and in the case of lesions clustered in a certain area, the imaging time point may be determined based on the center of the grouped area by grouping the lesions. there is.

Meanwhile, the location of the lesion A may be detected using the face image acquisition model by additionally learning information about the lesion in the learning step of the face image acquisition model. Alternatively, for precise diagnosis of the forehead and left and right cheeks where lesions commonly occur, the time to be photographed is the front of the face image, both sides, 45 degrees to the right, the side rotated 135 degrees, and 45 degrees to the left. , can be determined as a total of 7 viewpoints on the side rotated by 135 degrees.

Meanwhile, in the case of FIG. 4 , it is assumed that the skin examination terminal 150 has a front camera facing the user's face on the premise that lens aberration does not occur, but the present invention is not limited thereto. For example, in the case of the skin diagnostic terminal 150 operated by a single camera at the center, a rotation angle of the face required for measuring the height of the lesion may be differently determined in consideration of the distance from the user. Alternatively, a picture may be taken after moving the position of the face left and right by providing feedback to the user so that the position of the lesion is located at the shortest distance from the position of the camera at a predetermined distance.

Meanwhile, the skin diagnosis terminal 150 according to the present embodiment has been described as including a camera unit 151, a lighting unit 152, a detection unit 153, an output unit 154, and a control unit 155 as an example, but is limited thereto. It doesn't work. Since the skin diagnosis system 1000 according to the present embodiment captures a user's face image based on the face image acquisition model generated by the central server 300, it does not matter what type of skin diagnosis terminal is used. For example, the first skin diagnosis terminal 150a may include only the camera unit 151 and the lighting unit 152, and the detection unit 153, the output unit 154, and the control unit 155 may include the user terminal 100 ) may perform the function. Unlike this, the second skin diagnosis terminal 150b may include only the camera unit 151, the lighting unit 152, and the output unit 154, and the detection unit 153 and the control unit 155 may include the user terminal 100 function can also be performed. Alternatively, the function of each component may be performed by the user terminal 100 without a separate skin diagnosis terminal 150 . At this time, it is assumed that the user terminal 100 is a smart phone.

Hereinafter, the conversion from the 2D face image (VIMG) of the user to the 3D face image (3DVIMG) in the central server 300 will be described with reference to FIGS. 5 to 7 .

5 is a block diagram illustrating a central server that converts a received 2D face image into a 3D face image using deep learning technology. 6 is a block diagram illustrating generating a face image acquisition model using deep learning technology. 7 is an exemplary view of converting a 2D face image into a 3D face image.

Referring to FIG. 5 , the central server 300 includes an input buffer 310 , at least one processing element 320 and an output buffer 330 . The central server 300 may further include a parameter buffer 340 and a memory 350 .

The input buffer 310 may receive a plurality of user's face images VIMG at different viewpoints.

The processing element 320 converts a plurality of 2D face images VIMG from different viewpoints into a 3D face image 3DVIMG according to embodiments of the present invention.

Specifically, the 2D face image (VIMG) is an image of a user's face, which is a photographing target, photographed using a general camera, and the x-coordinate, y-coordinate and gray scale value of the photographing target (or in the case of a color image, R, G and B values) may be included ([f(x, y)]). This is because the photographed face image is expressed on a two-dimensional plane. That is, the 2D face image VIMG includes information on coordinate values of an object to be captured on a two-dimensional plane, but does not include the depth of the object to be captured (ie, 3D position information).

The processing element 320 may include an image conversion unit 321 . The image converter 321 converts the 2D face image VIMG received from the input buffer 310 into information that is not included in each image of the 2D face image VIMG by using a pre-learned artificial neural network model. A coordinate value in a 3D space, that is, a depth value may be inferred.

Accordingly, a 3D image image (including information on the 3-dimensional position (f(x,y,z)) of the object to be captured is obtained from an image image including only the 2-dimensional position (f(x,y)) of the object to be captured. 3DVIMG) can be created.

That is, the processing element 320 converts the 3D face image 3DVIMG including the 3D coordinates of the object from the 2D face image VIMG in which each frame image includes only the 2D coordinates of the object to be captured. can create

For example, the processing element 320 generates feature maps by extracting features of each of the 2D face images (VIMG), calculates them, and determines the depth of each of the frame images. can be inferred. At this time, the generation and operation of the feature map is a layer implemented based on a convolution neural network (CNN) and / or a recurrent neural network (RNN), a rectified linear unit (RELU) layer, It may be implemented by a computation model including various computation layers such as a pooling layer, a bias add layer, a softmax layer, and the like.

The 3D face image 3DVIMG is a 3D image obtained by 3D modeling using 2D face images VIMG from different viewpoints. Accordingly, the 3D face image 3DVIMG may include information on 3D coordinates (x coordinate, y coordinate, and z coordinate) at each viewpoint ([f(x, y, z)]).

That is, since the 3D face image 3DVIMG includes 3D coordinate information of the photographed object, as shown in FIG. 7 , the user's face image can be rotated and viewed from various angles desired by the user. That is, the viewpoint of the 3D face image 3DVIMG can be freely changed to a position and direction desired by the user.

In the conventional case, a method of generating a 3D image using a technique such as stitching a plurality of images of the same object to be photographed at different viewpoints was used, but according to the present embodiment, A 3D face image may be generated from a 2D face image using the processor 320 that implements a pre-learned calculation model.

As described above, this is similar to how a person infers the overall invisible shape of an object only with the image of a specific viewpoint, through a pre-learned artificial neural network operation model, the entire facial contour, etc. This is because it can infer (infer depth). On the other hand, inputting a plurality of 2D face images of different viewpoints is because the more face images for each viewpoint are input, the more precise conversion into 3D face images is possible. That is, the present invention receives a 2D face image that can be simply requested from a user, and based on this, it is possible to obtain a 3D face image capable of precise diagnosis at all viewpoints. Accordingly, precise skin diagnosis can be obtained even when a relatively inexpensive skin diagnosis terminal is used, and the influence of shaking or external light due to a change in face angle can be minimized. In addition, it is possible to secure data in a unified way in precise skin diagnosis and data utilization.

In one embodiment, the processing element 320 may include a central processing unit (CPU), a graphic processing unit (GPU), and a neural processing unit to perform the plurality of operations described above. NPU), a digital signal processor (DSP), an image signal processor (ISP), and the like. According to embodiments, the processing element 320 may include a plurality of homogeneous processing devices among the above-described processing devices or may be implemented by including a plurality of heterogeneous processing devices.

In one embodiment, the processing element 320 may be implemented by including a plurality of processor cores to parallelly process the plurality of operations described above.

The output buffer 330 may store and output the 3D face image 3DVIMG as an output result as an operation result of the processing element 320 . For example, the output buffer 330 may include at least one register.

The parameter buffer 340 may store a plurality of parameters and/or a plurality of hyper parameters used for the processing element 320 to perform the plurality of operations described above. For example, the parameter buffer 340 may store parameters of an artificial neural network model learned through a learning process.

The memory 350 may temporarily or continuously store data processed or to be processed by the processing element 120 . For example, the memory 150 may include volatile memory such as dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, phase change random access memory (PRAM), and resistance random access memory), nano floating gate memory (NFGM), polymer random access memory (PoRAM), magnetic random access memory (MRAM), and ferroelectric random access memory (FRAM). According to embodiments, the memory 350 may be implemented in the form of a mass storage device such as a solid state drive (SSD), an embedded multimedia card (eMMC), or a universal flash storage (UFS).

Although not shown, it may further include a control unit for controlling overall operations of components to convert the 2D face image VIMG into a 3D face image 3DVIMG, a task manager for managing assignment of specific tasks, and the like.

6 is a block diagram illustrating generating a face image acquisition model using deep learning technology.

Referring to FIGS. 5 and 5 , the processing element 320 may further include a learning model generator 322 . The learning model generation unit 322 is configured to learn the learning 3D face image 3DTVIMG corresponding to the learning 2D face image TVIMG based on artificial intelligence.

Here, the learning 2D face image (TVIMG) and the learning 3D face image (3DTVIMG) may be configured as learning images of a desired quality to enable precise skin diagnosis. That is, the learning image may include a high-quality face image to which the location, size, brightness, focus, etc. of the face suitable for precise skin diagnosis are applied. That is, in pre-processing before learning, it is possible to compare with the quality of the user's face image by selecting high-quality learning data capable of accurate skin diagnosis. In addition, learning data serving as a control group for comparison of each quality item can be trained together. For example, by supervised learning, data that can be fed back in terms of the quality of a face image, such as when the amount of light is suitable for the entire face image or when shadows are partially generated, can be learned. In addition, information on each lesion may be supervised on the face image, which is learning data. Accordingly, imaging can be performed by detecting a lesion in the detection step and calculating information on a point in time at which a precise examination of the lesion can be performed.

Meanwhile, the corresponding learning 3D face image 3DTVIMG can be obtained from a learning 2D face image TVIMG taken at various viewpoints of a face image to be learned. For example, the learning 2D face image (TVIMG) can be acquired using a plurality of cameras photographed at various viewpoints. The learning 2D face image (TVIMG) may include two or more image sets obtained by capturing the same learning subject at different viewpoints during the same period of time using at least two or more cameras. For example, when the training 2D face image (TVIMG) is acquired using four cameras, the training 2D face image (TVIMG) is obtained from first to second learning face images taken at different viewpoints. It includes the fourth learning image images (TVIMG1, TVIMG2, TVIMG3, and TVIMG4). For example, if the capturing point of time of the user's face image is predetermined, the capturing point of time may be included among the points of view of the image images. Meanwhile, a photographing viewpoint corresponds to a face angle, and the face angle includes left and right angles and up and down angles. For example, an upper and lower angle of the face of the learning 2D face image TVIMG may be fixed as a reference angle.

That is, the learning 2D face image TVIMG can be acquired using at least two or more cameras. For example, the 2D face image TVIMG includes a first learning 2D face image TVIMG1 captured using a first camera, a second learning 2D face image TVIMG2 captured using a second camera, and a third learning 2D face image TVIMG2 captured using a second camera. A third training 2D face image (TVIMG3) captured using a camera and a fourth training 2D face image (TVIMG1) captured using a fourth camera may be included.

It is possible to acquire learning image images obtained by taking pictures of the subject for learning at the same time using the first to fourth cameras, and taking pictures of the subject for learning at different angles. If the axis of the plane of the image captured by the first camera is defined as the x-axis and the y-axis, when viewed on the xz plane, the angle between the first to fourth cameras adjacent to the learning shooting object as the center is 45 degrees , 90 degrees or 135 degrees, the first camera and the second camera may face each other, and the third camera and the fourth camera may face each other.

According to the present embodiment, the image conversion unit for converting image images into 3D image images repeatedly learns the same actions of a plurality of people, so that one frame image is obtained from an image image including only 2D coordinate information in one 2D face image. A 3D face image including 3D coordinate information may be inferred. That is, it is possible to infer a 3D face image only from a 2D face image in front of the face.

However, the skin diagnosis system 1000 of this embodiment is configured to acquire 2D face images of a plurality of users at different viewpoints for precise skin diagnosis. At this time, in the 3D conversion, a 3D face image is inferred by inputting 2D face images of different specific viewpoints, and at various viewpoints input, more precise face image information is weighted summed by post-processing. A 3D face image is obtained. That is, in the 3D face image, precise skin diagnosis at various viewpoints input is possible.

For example, in the 3D face image transformation, in order to more easily grasp the state at the location of the lesion, when the location of the lesion is detected on the user's face image in the step of detecting the face image of the skin diagnosis terminal, the detected Photographing may be controlled by determining viewpoint information to be photographed according to a location. That is, time point information at which input is required for user-customized precise skin diagnosis may be determined.

The learning 3D face image 3DTVIMG uses the pre-learned artificial neural network model of the image conversion unit 321 to infer the coordinates of the 3D space, that is, the depth value, which is information not included in the 2D face image. can be obtained

Here, view point information captured at different view points of the face image for learning of the same subject may be managed as a specific view point. That is, since the 2D face image at the specific point in time is learned as learning data, the user's 2D face image from the specific point in time can be requested for conversion of the user's 3D face image. That is, as described above, the skin diagnosis terminal 150 may take a picture by controlling a camera with the detection of a face at a corresponding specific time point as a trigger. However, it is not limited thereto. When the learning of the learning 2D face image and the learning 3D face image are performed at various times, capturing of the face image by the skin diagnosis terminal 150 is not limited to a specific time point.

The learning 3D face image (3DTVIMG) is a 3D image modeled in three dimensions using the learning 2D face image (TVIMG), and for the three-dimensional coordinates (x coordinate, y coordinate and z coordinate) of the face image at various viewpoints information may be included.

The learning 3D face image 3DTVIMG can be generated using a conventional technique of converting two or more images taken at different angles into 3D images. For example, images taken by multiple cameras can be generated. It is possible to use stitching technology that connects and attaches so that they can be viewed from 360 degrees.

Alternatively, the learning 3D face image 3DTVIMG can be acquired through a 3D skin precision diagnostic device, which is a professional mechanical equipment that acquires continuous 3D face images at various viewpoints by rotating the camera while the face to be studied is fixed. For example, the specialized mechanical equipment may be expensive equipment used in dermatology hospitals for precise skin diagnosis, and a lighting device maintains uniform light intensity throughout the skin during imaging. That is, 3D face images, which are professional data, can be provided as training data from an external server connected to a dermatology hospital, etc., and when the 3D face images trained by specialists are acquired, it is possible to extract learning 2D face images for each viewpoint among them, so that all viewpoints Data learning in can be made.

That is, the learning model generating unit 322 may train the artificial neural network model using the learning 2D face image (TVIMG) and the 3D learning face image (3DTVIMG). That is, supervised learning may be executed by using the learning 2D face image TVIMG and the 3D learning face image 3DTVIMG as a learning data set. It is possible to repeatedly obtain a learning data set for a plurality of subjects for learning, and to perform learning by repeating it. Accordingly, the face image acquisition model may be generated.

In addition, according to an embodiment of the present invention, a high-quality face image including a predetermined face direction can be collected from a smart mirror based on a face image acquisition model that has learned a face image including a predetermined face direction. , It is possible to obtain a three-dimensional face image based on a plurality of high-quality face images including different face directions. Accordingly, accurate skin diagnosis can be performed by precisely analyzing the condition of the lesion in the local area using a 3D image.

At least one of the components of the central server 300 according to this embodiment and a component for controlling them may be a hardware component, a software component executed by a hardware component such as a processor, or a combination thereof. It can be implemented in a recording medium readable by a computer or a device similar thereto.

According to the hardware implementation, ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), FLDs (programmable logic devices), FPGAs (field programmable gate arrays), processors, controllers It may be implemented using at least one of controllers, micro-controllers, microprocessors, and electrical units for performing other functions. In particular, the detection unit 153 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or an existing general-purpose processor (eg, CPU or application processor) or graphics-only processor (eg, GPU). It may be manufactured as a part of and mounted on the vision server 15.

According to the software implementation, it may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein, and the software code may be implemented as a software application written in an appropriate programming language. The software code may be stored in the memory of the vision server 15 and executed by a separate control unit. In particular, when the detection unit 153 is implemented as a software module (or a program module including instructions), the software module is a computer-readable non-transitory computer readable recording medium (non-transitory computer readable media). ) can be stored in Also, in this case, the software module may be provided by an Operating System (OS) or a predetermined application. Alternatively, some of the software modules may be provided by an Operating System (OS), and the other part may be provided by a predetermined application.

8 is a flowchart illustrating a method of converting a 2D face image (VIMG) into a 3D face image (3DVIMG) according to embodiments of the present invention.

Referring to FIG. 8 , the method of converting a 2D face image (VIMG) into a 3D face image (3DVIMG) includes a training face image (TVIMG) collection step (S100), a model generation step (S200), a user 2D face image (VIMG) It may include an input step (S300), an image conversion step (S400), and a 3D face image (3DVIMG) output step (S500).

In the learning face image TVIMG collecting step (S100), at least two or more cameras may be used to collect the learning face image TVIMG. The learning face image TVIMG obtained by taking pictures of the learning subject at different angles may be obtained by simultaneously photographing the learning subject using the first to fourth cameras. In this case, an angle between the first to fourth cameras adjacent to the learning recording target may be 45 degrees, 90 degrees, or 135 degrees.

In addition, a learning 3D face image (3DTVIMG) can be collected in this step.

The learning 3D face image (3DTVIMG) is a 3D image modeled in three dimensions using the learning 2D face image (TVIMG), and for the three-dimensional coordinates (x coordinate, y coordinate and z coordinate) of the face image at various viewpoints information may be included.

The learning 3D face image 3DTVIMG can be generated using a conventional technique of converting two or more images taken at different angles into 3D images. For example, images taken by multiple cameras can be generated. It is possible to use stitching technology that connects and attaches so that they can be viewed from 360 degrees.

Alternatively, the learning 3D face image 3DTVIMG can be acquired through a 3D skin precision diagnostic device, which is a professional mechanical equipment that acquires continuous 3D face images at various viewpoints by rotating the camera while the face to be studied is fixed. For example, the specialized mechanical equipment may be expensive equipment used in dermatology hospitals for precise skin diagnosis, and a lighting device maintains uniform light intensity throughout the skin during imaging. That is, 3D face images, which are expert data, can be provided as training data from an external server connected to a dermatology hospital, etc., and when a specialist's learning 3D face image is acquired, it is possible to acquire a learning 2D face image at a specific point in time. Data learning in can be made.

In the model generation step (S200), a 3D modeled learning 3D face image is generated using the learning face image (TVIMG), and an artificial face image is generated using the learning face image (TVIMG) and the learning 3D face image (3DTVIMG). A neural network model can be trained. For example, it is possible to repeatedly learn the same motions of a plurality of people, and detailed descriptions thereof are as described in FIGS. 1 to 7 . That is, a face image acquisition model for precise skin diagnosis is created.

In the step of inputting the user's 2D face image (VIMG) (S300), the face image (VIMG) to be 3D converted may be input to the artificial neural network model. The face image VIMG may be a plurality of face images captured at different viewpoints, and each may include 2D coordinate information of an object of the face image VIMG.

In the image conversion step (S400), a 3D face image 3DVIMG may be generated from the 2D face image VIMG using the learned artificial neural network model. The 3D face image 3DVIMG may include 3D location information of an object to be photographed in the face image VIMG. That is, the 3D face image 3DVIMG may include 3D coordinate information of the object to be captured. Accordingly, the view point of the 3D face image 3DVIMG can be variously changed according to the user's request.

In the outputting step (S500), the 3D face image 3DVIMG may be output.

9 is a flowchart illustrating a method of photographing a face image for skin diagnosis according to an embodiment of the present invention.

1 to 9, the method of capturing a face image for skin diagnosis includes a face image quality detection step (S210), an image quality control step (S220), a face image acquisition step (S230), and a face image transmission step. (S240) may be included.

The method for taking a face image for skin diagnosis is configured to capture a user's face image based on the face image acquisition model described in FIG. 8 .

In the face image quality detection step (S210), the quality of image data including the face image is detected for accurate skin diagnosis. Based on the face image acquisition model using a deep network, quality information such as the position, size, brightness, and focus of the face is automatically extracted for the image data input from the camera unit 151. That is, the quality of the face image in the received image data may be detected based on the face image quality information, which is the learning data of the face image acquisition model.

For example, based on the detected face image, the face position is detected based on whether the face is placed in the center of the image data or whether a part of the face shape is cut off. In addition, based on the size of the face image as learning data, it is configured to detect whether the face size in the image data is a desired face size. For example, size information may be extracted through a distance away from a camera with respect to a face, which is an object to be detected. In addition, based on the illuminance and sharpness of each part of the face image, which is learning data, the amount of light, shade (shadow), light smearing, and light reflection of the face image in the image data according to the external environment are configured to detect the location.

In addition, in this step, it is possible to detect whether conditions required for precise skin diagnosis are satisfied, for example, whether the forehead is exposed by raising the bangs or whether a foreign substance is present in the face image. That is, based on the face image as learning data, the quality of the face image in the video data can be detected.

In the image quality control step (S220), based on the information detected in the detection step, it is configured to output a feedback signal to the output unit 154.

For example, in this step, if the face is not located at the center of the image data based on the detected face image, feedback regarding the request for moving the face position may be output through the output unit 154 . In addition, based on the size of the face image in the image data or the distance of the face as an object, feedback to move closer or farther may be output.

In addition, when a low amount of light or shade (shadow) of the face image in the video data is detected, the lighting unit 152 is controlled to set the amount of light or the irradiation angle so that the light of the LED can be irradiated to a desired location on the user's face. configured to control. For example, when a plurality of LEDs are installed at different positions, the amount of light of the LEDs may be individually controlled according to the position where the shade is generated. Individual control targets may be programmed in advance according to the shadow occurrence position. Accordingly, it is possible to remove shadows by controlling light for an insufficient amount of light or shadows on the user's face image. However, it is not limited thereto. Feedback on the user's manipulation request for adjusting the amount of light may be output.

In addition, when light smearing is detected in the face image in the video data, feedback suggesting face washing and diagnosis may be output through the output unit 154, and when light reflection is detected in the face image, the Feedback suggesting to avoid a location where an external light source is reflected by moving the location may be output through the output unit 154 .

In the face image acquisition step (S230), when the quality information from the user's face image meets a predetermined criterion, the user's face image is captured at different viewpoints based on the face image acquisition model.

That is, when viewpoint information of the user's face image in the video data is detected, a feedback requesting rotation of the user's face to a viewpoint corresponding to a specific viewpoint of the learned data may be output. The specific viewpoint is assumed to be two or more, and may include, for example, a total of three viewpoints of the user's front and left and right sides. That is, when the viewpoint of the user's face image corresponds to a predetermined specific viewpoint, the camera unit 151 is controlled to capture an image. That is, when there are a plurality of specific viewpoints, the controller 155 may output feedback for guiding rotation of the user's face to the next viewpoint while sequentially capturing images at each specific viewpoint. Accordingly, as shown in FIG. 3 , three face images of the front and both sides of the user may be obtained.

In addition, in the face image acquisition, when a desired quality of the face image is detected in the image data of the preview image received in real time from the camera unit 151 based on the artificial neural network model, the user's face image is captured in the camera image. It is configured to control the camera unit 151 by determining whether it is a specific angle, which is a preset trigger.

Specifically, by extracting features from a plurality of frames of the image data to create feature maps, calculating them, tracking the position of the user's face and the quality of the face image in the camera image in real time, and When is satisfied, it is configured to control the camera unit 151 by determining whether it corresponds to a specific angle, which is a preset trigger, based on viewpoint information of a user's face image.

For example, a region of interest (ROI) may be extracted from a feature map of a frame of image data, and a region of interest (ROI) that is at least a part of a frame of image data may be used to determine whether there is a preset trigger in capturing a face image. It can be set as the main area for For example, the controller 155 may set the main area to include the area where the user's face is recognized in the frame of the image data. A detailed description of this is as described in FIGS. 1 to 7 .

In addition, in the corresponding step, it has been described that when the viewpoint of the user's face image corresponds to a predetermined specific viewpoint, which is learning data, an image is captured by controlling the camera unit 151, but the present invention is not limited thereto. For example, when a position of a lesion is detected on a user's face image, photographing may be controlled by determining viewpoint information to be photographed according to the detected position. For example, when a lesion is detected on the user's left cheek, the time point at which the user's left cheek is rotated 45 degrees to the right and the time point when the left cheek is rotated 135 degrees to the right to face the front of the camera may be determined. This is to determine the size (width) and degree of protrusion of the lesion for precise skin diagnosis. In the 3D face image transformation in the central server 300, which will be described later, it is easier to grasp the state at the location of the lesion. It is for Meanwhile, the location of the lesion may be detected using the face image acquisition model by additionally learning information about the lesion in the learning step of the face image acquisition model. Alternatively, for precise diagnosis of the forehead and left and right cheeks where lesions commonly occur, the time to be photographed is the front of the face image, both sides, 45 degrees to the right, the side rotated 135 degrees, and 45 degrees to the left. , can be determined as a total of 7 viewpoints on the side rotated by 135 degrees.

In the face image transmission step (S240), when the quality information of the face image previously learned through the camera is detected, the user's face image at different viewpoints is photographed, and the obtained user's face image at different viewpoints is recalled. It is configured to transmit to the central server (300).

The above-described method for capturing a face image according to an embodiment of the present invention may be executed by an application basically installed in a terminal (this may include a program included in a platform or an operating system basically installed in the terminal), and , It may be executed by an application (that is, a program) directly installed in the master terminal by a participant through an application providing server such as an application store server, an application or a web server related to the corresponding service. In this sense, the usability test matching method according to an embodiment of the present invention described above is implemented as an application (i.e., a program) that is basically installed in a terminal or directly installed by a participant, and is a computer-readable recording medium such as a terminal. can be recorded in

The above description of the present invention is for illustrative purposes, and those skilled in the art can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be construed as being included in the scope of the present invention. do.

The above-described present invention can be implemented as computer readable code (or application or software) on a medium on which a program is recorded. The above-described method of photographing a face image may be realized by a code stored in a memory or the like.

The computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. , and also includes those implemented in the form of a carrier wave (eg, transmission over the Internet). Also, the computer may include a processor. Accordingly, the above detailed description should not be construed as limiting in all respects and should be considered illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

100: user terminal
150: skin diagnosis terminal
151: camera unit
152: lighting unit
153: detection unit
154: output unit
155: control unit
300: central server

Claims (10)

As a skin diagnosis system,
A terminal including a lighting unit capable of irradiating light to a user and a camera unit configured to capture a user's face image; and
An artificial neural network model is trained using a 3D modeled learning 3D face image corresponding to a plurality of unspecified learning 2D face images including predetermined quality information and the learning 2D face image, and based on the learned artificial neural network model , A central server configured to convert the user's face image received from the terminal into a three-dimensional 3D face image,
The terminal, based on the artificial neural network model, detects the quality information of the user's face image in the video data input in real time to the camera unit, and if the quality information meets a predetermined criterion, the user is informed of the face angle. By outputting feedback on the rotation, it is configured to capture an image of the user's face at a specific viewpoint
The artificial neural network model additionally learns lesion information of the face included in the learning 2D face image, and the terminal detects the lesion included in the user's face image, and based on the location information of the lesion, the specific Skin diagnosis system, characterized in that for determining the point of view. According to claim 1,
The skin diagnosis system, characterized in that the predetermined quality information includes at least one of position, size, brightness and focus information of the face in the face image. According to claim 1,
Skin diagnosis characterized in that the lighting unit is configured to control at least one of a light amount and an irradiation angle, and controls the light amount or the irradiation angle by controlling the lighting unit when the quality information does not satisfy the predetermined criterion. system. According to claim 1,
The skin diagnosis system, characterized in that the learning 3D face image is generated from the learning 2D face images for different viewpoints. According to claim 1,
The learning 3D face image is a 3D image obtained from a 3D skin precision diagnostic device that acquires continuous 3D face images at various viewpoints by rotating the camera while the face to be studied is fixed, and the learning 2D face image is the learning 3D face image. Skin diagnosis system, characterized in that extracted from the image for each viewpoint. delete According to claim 1,
The skin diagnosis system, characterized in that the specific point of time includes a plurality of different points of view. According to claim 7,
The skin diagnosis system, characterized in that the specific time point is determined based on the center of the area in which the lesions are grouped when the lesions are clustered in a certain area. According to claim 1,
The artificial neural network model is learned based on learning 2D face images of different viewpoints including the specific viewpoint. According to claim 1,
In the artificial neural network model, viewpoint information of the learning 2D face image is additionally learned,
The terminal detects viewpoint information of the user's face image in the video data and, when the specific viewpoint is within a predetermined error range, controls the camera unit to start capturing a video, while the user moves the angle of the face. When a face image of the specific point in time is included in the image frame of the skin diagnosis system, characterized in that for stopping video recording and extracting an image frame including the face image of the specific point in time.

Images