TWM586599U - System for analyzing skin texture and skin lesion using artificial intelligence cloud based platform - Google Patents

Abstract

A system for analyzing skin texture and skin lesion using artificial intelligence cloud based platform is provided. The system includes an electronic device and a server. The server includes a storage device and a processor. The processor is coupled to the storage device, accesses and executes a plurality of modules stored in the storage device, the plurality of modules comprising: an information receiving module, receiving a captured image and a plurality of user parameters; a feature vector obtaining module, obtaining the first feature vector from the captured image, and the second feature vector from the plurality of user parameters; a skin parameter acquisition module, obtaining the output result of skin parameters from the first feature vector and the second feature vector; and a skin identification module, determining a skin identification result based on the output result of skin parameters.

Description

Artificial intelligence cloud skin texture and skin lesion identification system

The new creation relates to a skin texture and skin lesion detection technology, and more particularly to an artificial intelligence cloud skin and skin lesion identification system.

Generally speaking, in addition to judging the skin condition from the appearance, a dermatologist also comprehensively judges whether the skin has an abnormal condition through consultation. Based on the appearance and the results of the consultation, the doctor can make a preliminary judgment on the condition of the skin. For example, if the mole on the skin becomes significantly larger or has abnormal protrusions over time, it may be a precursor to a lesion. Once a lesion occurs, it takes time to treat and causes a burden on the body, so early detection of the disease and timely treatment is the best way to avoid suffering.

However, the current skin changes require professional judgment of a doctor. It is easy for general users to ignore skin changes, and it is difficult to determine whether the skin is abnormal. Therefore, how to know the skin condition effectively and clearly is one of the problems that those skilled in the art want to solve.

In view of this, the new creation provides an artificial intelligence cloud skin texture and skin lesion recognition system, which can simultaneously consider the skin image and the content of the user's answer to the question, and determine the skin recognition result based on the skin image and user parameters.

The novel creation provides an artificial intelligence cloud skin texture and skin lesion identification system, including an electronic device and a server. The electronic device obtains the captured image and a plurality of user parameters. A server is connected to the electronic device, and the server includes a storage device and a processor. The storage device stores a plurality of modules. The processor is coupled to the storage device, and accesses and executes the multiple modules stored in the storage device. The multiple modules include an information receiving module, a feature vector obtaining module, and a skin parameter obtaining module. Group and skin identification module. An information receiving module receives the captured image and the plurality of user parameters; a feature vector acquisition module obtains a first feature vector of the captured image and calculates a second feature vector of the plurality of user parameters The skin parameter acquisition module obtains an output result related to the skin parameter according to the first feature vector and the second feature vector; and the skin identification module determines a corresponding image to the captured image according to the output result Skin identification results.

In an embodiment of the novel creation, the operation of the feature vector obtaining module to obtain the first feature vector of the captured image includes: using a machine learning model to obtain the first feature vector of the captured image. .

In an embodiment of the novel creation, the operation of the feature vector obtaining module for calculating the second feature vector of the plurality of user parameters includes: using a vector The quantity represents each of the plurality of user parameters; the vectorized each of the plurality of user parameters is combined and input to a fully connected layer of a machine learning model to obtain the second feature vector.

In an embodiment of the novel creation, the plurality of user parameters include a combination of a gender parameter, an age parameter, an affected area size, a time parameter, or an affected area change parameter.

In an embodiment of the novel creation, the operation of the skin parameter obtaining module to obtain the output result related to the skin parameter according to the first feature vector and the second feature vector includes: The first feature vector and the second feature vector to obtain a merge vector; and inputting the merge vector to a fully connected layer of a machine learning model to obtain the output result, wherein the output result is associated with the skin quality parameter Target probability.

In an embodiment of the novel creation, the operation of the skin identification module to determine the skin identification result corresponding to the captured image according to the skin parameter includes: determining the corresponding skin identification result according to the output result. The skin quality recognition result of the captured image is described.

In an embodiment of the novel creation, the machine learning model includes a convolutional neural network or a deep neural network.

In order to make the above-mentioned features and advantages of the novel creation more comprehensible, embodiments are described below in detail with the accompanying drawings as follows.

1‧‧‧ Artificial Intelligence Cloud Skin and Skin Lesion Identification System

10‧‧‧ electronic device

11, 21‧‧‧ communication device

12, 22‧‧‧ processor

13, 23‧‧‧ storage devices

20‧‧‧Server

231‧‧‧Information receiving module

232‧‧‧Feature Vector Acquisition Module

233‧‧‧Skin parameter acquisition module

234‧‧‧Skin Identification Module

S301 ~ S304, S401 ~ S405‧‧‧step

FIG. 1 is a schematic diagram of an artificial intelligence cloud skin texture and skin lesion identification system according to an embodiment of the novel creation.

FIG. 2 is a block diagram of components of an electronic device and a server according to an embodiment of the present invention.

FIG. 3 is a flowchart of an artificial intelligence cloud skin and skin lesion identification method according to an embodiment of the novel creation.

FIG. 4 is a flowchart of a method for identifying skin texture and skin lesions in an artificial intelligence cloud according to an embodiment of the novel creation.

The novel creation considers the skin image and the content of the user's answer at the same time, uses the machine learning model to obtain the feature vector of the skin image, and calculates the feature vector of the user parameter. Then, according to the feature vector of the skin image and the feature vector of the user parameter, an output result related to the skin texture parameter is obtained to determine the skin recognition result. In this way, the skin image and the content of the user's answer to the question can be considered at the same time to determine the skin lesion or skin quality recognition result.

Some embodiments of this novel creation will be described in detail with reference to the accompanying drawings. The component symbols cited in the following description will be regarded as the same or similar components when the same component symbols appear in different drawings. These embodiments are only a part of the novel creation, and do not disclose all the possible implementations of the novel creation. More precisely, this These embodiments are just examples of the artificial intelligence cloud skin texture and skin lesion identification system in the scope of the patent application of this novel creation.

FIG. 1 is a schematic diagram of an artificial intelligence cloud skin texture and skin lesion identification system according to an embodiment of the novel creation. Referring to FIG. 1, the artificial intelligence cloud skin texture and skin lesion identification system 1 includes at least, but not limited to, an electronic device 10 and a server 20. The server 20 can be connected to multiple electronic devices 10 respectively.

FIG. 2 is a block diagram of components of an electronic device and a server according to an embodiment of the present invention. Referring to FIG. 2, the electronic device 10 may include, but is not limited to, a communication device 11, a processor 12, and a storage device 13. The electronic device 10 is, for example, a smart phone, a tablet computer, a notebook computer, a personal computer, or other devices having a computing function, and the present invention is not limited thereto. The server 20 may include, but is not limited to, a communication device 21, a processor 22, and a storage device 23. The server 20 is, for example, a computer host, a remote server, a background host, or other devices, and the present invention is not limited thereto.

The communication device 11 and the communication device 21 may support mobile communication such as third generation (3G), fourth generation (4G), fifth generation (5G), or later generations, Wi-Fi, Ethernet, and optical fiber networks. Wait for the communication transceiver to connect to the Internet. The server 20 is communicatively connected to the communication device 11 of the electronic device 10 through the communication device 21 to transmit data to and from the electronic device 10.

The processor 12 is coupled to the communication device 11 and the storage device 13, the processor 22 is coupled to the communication device 21 and the storage device 23, and the processor 12 and the processor 22 can access and execute the storage in the storage device 13 and the storage device 23, respectively. Multiple modules. In different embodiments, the processor 12 and the processor 22 may be, for example, central processing units, respectively. Unit (Central Processing Unit, CPU), or other programmable general purpose or special purpose microprocessor (Microprocessor), digital signal processor (DSP), programmable controller, special application Application Specific Integrated Circuits (ASIC), Programmable Logic Device (PLD) or other similar devices or a combination of these devices, the creation of this new type is not limited here.

The storage device 13 and the storage device 23 are, for example, any type of fixed or removable random access memory (RAM), read-only memory (ROM), and flash memory ( flash memory), a hard disk, or a similar component or a combination of the foregoing components to store programs executable by the processor 12 and the processor 22, respectively. In this embodiment, the storage device 23 is used to store buffered or permanent data, software modules (for example, information receiving module 231, feature vector acquisition module 232, skin parameter acquisition module 233, and skin identification module Group 234, etc.) and other materials or files, and the details thereof will be detailed in the subsequent embodiments.

FIG. 3 is a flowchart of an artificial intelligence cloud skin and skin lesion identification method according to an embodiment of the novel creation. Please refer to FIG. 2 and FIG. 3 at the same time. The method of this embodiment is applicable to the above-mentioned artificial intelligence cloud skin and skin lesion identification system 1. The following describes the devices and components of the electronic device 10 and the server 20 in this embodiment. Detailed steps of artificial intelligence cloud skin texture and skin lesion identification method. Those skilled in the art should understand that the software modules stored in the server 20 do not have to be executed on the server 20, but may be downloaded and stored in the storage device 13 of the electronic device 10, and executed by the electronic device 10. The software module performs artificial intelligence cloud skin and skin Lesion identification method.

First, the processor 22 accesses and executes the information receiving module 231 to receive the captured image and a plurality of user parameters (step S301). The captured image and user parameters can be received from the electronic device 10 by the communication device 21 in the server 20. In an embodiment, the captured image and a plurality of user parameters are obtained by the electronic device 10 first. In detail, the electronic device 10 is coupled to an image source device (not shown) and obtains a captured image from the image source device. The image source device may be a camera configured in the electronic device 10, or may be a device for storing images such as the storage device 13, an external memory card, or a remote server, and the new creation is not limited thereto. In other words, for example, the user operates the electronic device 10 to capture an image by using a camera, or operates to obtain a previously captured image from the device, and transmits the selected image to the server 20 as a captured image for subsequent operations. .

In addition, the server 20 provides a number of questions for the user to answer. After the user answers these questions through the electronic device 10, the answer results will be transmitted to the server 20 as user parameters for subsequent operations. Among them, the user answers the question through a user interface displayed on the electronic device 10, for example. The user interface may be a chat room of a communication software, a webpage, a voice assistant, or other software interfaces for interactive functions. limit.

Then, the processor 22 accesses and executes the feature vector obtaining module 232 to obtain a first feature vector of the captured image, and calculates a second feature vector of a plurality of user parameters (step S302).

In detail, in order to obtain the first feature vector of the captured image, the processor 22 First train the parameter values of each layer in the machine learning model through skin lesion image samples and user parameter samples. In an embodiment, the above machine learning model is, for example, a machine learning model constructed by using a technology such as a neural network. Taking a neural network as an example, there are many between the input layer and the output layer. Neurons and links consist of multiple hidden layers, and the number of nodes (neurons) in each layer is variable. A larger number of nodes can be used to enhance the robustness of this type of neural network. In this embodiment, the machine learning model is, for example, a Convolutional Neural Network (CNN) or a Deep Neural Network (DNN), and the novel creation is not limited thereto. Taking the convolutional neural network as an example, the parameter values corresponding to the skin lesion image can be input to the convolutional neural network as a machine learning model, and trained using backward propagation to use the final objective function. (loss / cost function) to update the parameters of each layer, and the parameter values of each layer in the learning model can be trained, where, for example, the mean square error is used as the objective function. Among them, each skin lesion image sample can be trained using a conventional convolutional neural network model architecture such as ResNet50, InceptionV3, and the like.

The image can then be input to a trained machine learning model to obtain image features. In one embodiment, the feature vector obtaining module 232 obtains a first feature vector of the captured image by using a machine learning model. That is, after training the machine learning model, the processor 22 inputs the captured image to the trained machine learning model, and extracts a first feature vector of the captured image.

On the other hand, the feature vector acquisition module 232 will also calculate multiple users The second eigenvector of the parameter. The feature vector acquisition module 232 uses vectors to represent user parameters, for example, and combines the vectorized user parameters and inputs them to the Fully Connected Layer of the machine learning model to obtain a second feature vector. Among them, the dimensions of the combined vectorized user parameters are related to the number of questions and options within the question.

In detail, the feature vector obtaining module 232 encodes user parameters received by the server 20 from the electronic device 10 using an indicator function. For example, if the question is the gender of the user, a vector (1,0,0) is generated when the user answers the gender is male; a vector (0,1,0) is generated when the user answers the gender is female; When the user does not want to answer the gender, a vector (0,0,1) is generated. After all the user parameters are encoded, the feature vector acquisition module 232 combines the encoded user parameters to obtain a merge vector, and inputs the merged merge vector to the fully connected layer to perform hybridization and output N-dimensional vector. The fully connected layer will consider the interaction between user parameters to generate a second feature vector with more vector dimensions than the original vector dimension of each user parameter. For example, a 16-dimensional vector is input to the fully connected layer. Can produce 256-dimensional vectors. In an embodiment, the plurality of user parameters include one or a combination of a gender parameter, an age parameter, an affected area size, a time parameter, or an affected area change parameter.

Then, the processor 22 accesses and executes the skin parameter acquisition module 233 to obtain an output result related to the skin parameter according to the first feature vector and the second feature vector (step S303). The skin parameter acquisition module 233 merges the first feature vector and the second feature vector to obtain a merge vector, and inputs the merge vector to a machine learning model. Full-connected layer of the model to obtain the output result, where the output result is related to the target probability of the skin parameter. In an embodiment, since the first feature vector obtained through the machine learning model is used to obtain a picture that may be a two-dimensional structure, the first feature vector may be converted into a one-dimensional space vector before being combined with the second feature vector to generate a merge. vector.

In detail, the skin parameter obtaining module 233 combines the first feature vector of the captured image obtained by the feature vector obtaining module 232 and the second feature vector calculated from a plurality of user parameters, and combines the first feature vector And the second feature vector is merged into a merge vector. Then, the skin parameter acquisition module 233 inputs the merged vector to the fully connected layer, and generates an output result at the output layer. The number of output results is related to the number of output results that you want to classify. Suppose that you want the output results to be divided into two categories (for example, no skin condition and skin condition), then there are two output categories in the output layer. Quality parameters, this new creation does not limit the number of output categories here. The probability that the final merge vector input to the fully connected layer will be transformed into each output category (between 0 and 1). In this embodiment, the skin quality parameters are, for example, "mole", "acne" or "skin condition", and are divided into "moles with lower malignant risk / moles with higher malignant risk", " Acne / non-acne "or" good skin condition / bad skin condition "and other output categories are correlated with the target probability of each skin quality parameter in each output category.

Finally, the processor 22 accesses and executes the skin recognition module 234 to determine a skin recognition result corresponding to the captured image according to the output result (step S304). The skin texture recognition module 234 determines a skin texture recognition result corresponding to the captured image according to the output result. In detail, the most probable category in the output is the most likely category.

Based on the above, in the embodiment of the novel creation, after inputting the image to the machine learning model to obtain the feature vector of the image, and using the fully connected layer to calculate the vector of the user parameters, the two vectors are combined as data to enter the full connection of the machine learning model. Layer and produce output results through the fully connected layer. In other words, in addition to considering picture information, this new type of creation also considers non-picture information. By building a machine learning model that can consider both picture and non-picture information, it can more realistically simulate the situation of clinical judgment of skin texture and Improved model accuracy.

The following embodiment takes “mole” as an example, in which the output category “mole” is divided into two skin parameters: “mole with lower risk of malignant change” and “mole with higher risk of malignant change”. In this embodiment, the volume is used. Product neural networks are examples of machine learning models. FIG. 4 is a flowchart of a method for identifying skin texture and skin lesions in an artificial intelligence cloud according to an embodiment of the novel creation. Referring to FIG. 4, first, the processor 22 receives the captured image and a plurality of user parameters (step S401). In this embodiment, the user uses the electronic device 10 to capture or select a captured image. The size of the captured image is, for example, set to 224x224 according to the input format and size of the conventional convolutional neural network. The captured image can be represented as a matrix of (224,224,3), where 3 represents the rank of the RGB color. And the user answers a number of questions provided by the server 20, among which the questions include, for example, "gender (male, female, do not want to answer)", "age (under 20 years old, 21-40 years old, 41-65 years old, 66 years old or older) ) "," Area of affected area (0.6 cm or less, greater than 0.6 cm) "," Longevity (less than or equal to 1 year, greater than 1 year and less than 2 years, greater than 2 years, no attention) "or" Change of affected area (recent There have been changes from month to month, and there has been no change in the last month. The processor 22 receives the information transmitted by the electronic device 10 Input captured image and multiple user parameters.

Next, the processor 22 uses the convolutional neural network to obtain a first feature vector of the captured image (step S4021). And the processor 22 calculates the second feature vectors of the plurality of user parameters (step S4022). The processor 22 inputs the captured image to a trained convolutional neural network to obtain a first feature vector of the captured image. The convolutional neural network is trained using the image of the "mole". And after the server 20 receives the user's answer, the processor 22 encodes the answer as a vector. For example, in this embodiment, if the user's answer is male, under 20, 0.6 cm or less, 1 year or less, the most recent The month changes, the vectorized answers are gender (1,0,0), age (1,0,0,0), affected area (1,0), age (1,0,0,0), and Affected area changes (1,0,0). Next, the processor 22 merges each of the vectorized user parameters in a dimension to obtain a merge vector, and the processor 22 inputs the merge vector to a fully connected layer of the machine learning model to obtain a second feature vector.

Next, the processor 22 combines the first feature vector and the second feature vector to obtain a merged vector (step S403). Next, the processor 22 inputs the merge vector to the fully connected layer of the convolutional neural network to obtain an output result (step S404). In this embodiment, the processor 22 merges the first feature vector and the second feature vector in dimensions to obtain a merge vector, and inputs the merge vector to a fully connected layer of the convolutional neural network to obtain an output result, where the output The results are related to the respective target probabilities of the two skin parameter parameters "moles with a lower risk of malignancy / moles with a higher risk of malignancy" in the output category "mole".

Finally, the processor 22 determines a skin quality recognition result corresponding to the captured image according to the output result (step S405). In this embodiment, if the skin quality parameter is in the output result, Mole with a lower risk of malignant change determines that the moles with a lower risk of malignancy are included in the captured image. If the probability of a skin parameter "mole with a higher risk of malignant change" is higher, it is determined that the moles with a higher risk of malignant change are included in the captured image Tall mole.

In another embodiment, if the convolutional neural network uses other images about lesions such as "acne" or images about skin texture such as "skin condition" to train, and targets "acne" or "skin condition" ”And other lesions or skin types propose different questions for judging the lesions or skin texture as user parameters, the model created by the new creative system can be used to help determine“ acne ”,“ skin condition ”or other lesions or Whether the skin image matches the specific lesion or skin condition.

In another embodiment, the artificial intelligence cloud skin and skin lesion identification model provided by the artificial intelligence cloud skin and skin lesion identification method provided by the novel creative embodiment may be trained using reverse transfer to utilize the final target Function to update the parameters of each layer to improve the recognition accuracy of the model.

In summary, the artificial intelligence cloud skin texture and skin lesion recognition system provided by this new creation can simultaneously consider the skin image and the content of the user's answer to the question, and obtain the feature vector of the image from the input image to the machine learning model, and use the full connection After computing the vector of user parameters in the layer, the feature vector of the image and the vector of user parameters are combined as a data input fully-connected layer of the machine learning model, and an output result is generated through the fully-connected layer. Thereby, the probability of each skin quality parameter can be obtained according to the feature vector of the skin image and the feature vector of the user parameter to determine the recognition result of the lesion or skin quality. In other words, in addition to considering picture information, this new type of creation also considers non-picture information. By creating a machine that can consider both picture and non-picture information Learning the model to more realistically simulate the situation in which the lesion or skin is judged clinically based on the condition of the affected area and the result of the question and answer to improve the accuracy of the model.

Although this new type of creation has been disclosed in the above examples, it is not intended to limit the new type of creation. Any person with ordinary knowledge in the technical field can make some changes and modifications without departing from the spirit and scope of this new type of creation. Retouching, so the protection scope of this new type of creation shall be determined by the scope of the attached patent application.

Claims (7)

An artificial intelligence cloud skin texture and skin lesion identification system includes:
An electronic device to obtain the captured image and a plurality of user parameters; and a server connected to the electronic device, the server including:
A storage device storing a plurality of modules; and a processor coupled to the storage device to access and execute the plurality of modules stored in the storage device, the plurality of modules including:
An information receiving module for receiving the captured image and the plurality of user parameters;
A feature vector obtaining module, obtaining a first feature vector of the captured image, and calculating a second feature vector of the plurality of user parameters;
A skin parameter acquisition module obtains output results related to skin parameters according to the first feature vector and the second feature vector; and a skin identification module determines corresponding to the extraction according to the output results Skin image recognition results. According to the artificial intelligence cloud skin and skin lesion identification system according to item 1 of the scope of patent application, the operation of the feature vector acquisition module to obtain the first feature vector of the captured image includes:
A machine learning model is used to obtain the first feature vector of the captured image. The artificial intelligence cloud skin and skin lesion identification system according to item 1 of the scope of the patent application, wherein the operation of the feature vector acquisition module calculating the second feature vector of the plurality of user parameters includes:
Using vectors to represent each of the plurality of user parameters; and merging and vectorizing each of the plurality of user parameters into a fully connected layer of a machine learning model to obtain the second feature vector. According to the artificial intelligence cloud skin texture and skin lesion identification system according to item 3 of the scope of the patent application, the plurality of user parameters include a combination of a gender parameter, an age parameter, an affected area size, a time parameter, or an affected area change parameter. The artificial intelligence cloud skin and skin lesion identification system according to item 1 of the scope of the patent application, wherein the skin parameter acquisition module obtains a correlation with the skin according to the first feature vector and the second feature vector. The operation of the output results of the quality parameters includes:
Merging the first feature vector and the second feature vector to obtain a merge vector; and inputting the merge vector to a fully connected layer of a machine learning model to obtain the output result, wherein the output result is associated with the skin Target probability of quality parameters. The artificial intelligence cloud skin and skin lesion identification system according to item 5 of the scope of patent application, wherein the skin identification module determines the skin identification result corresponding to the captured image according to the skin parameter The operations include:
Determine the skin quality recognition result corresponding to the captured image according to the output result. The artificial intelligence cloud skin and skin lesion identification system according to item 2 of the scope of patent application, wherein the machine learning model includes a convolutional neural network or a deep neural network.