KR20220097032A - Method for providing image processing solution for analysis of diaper stool image - Google Patents

Abstract

According to one embodiment, provided is a method for providing an image processing solution for analyzing a diaper feces image, performed by a device. The method for providing an image processing solution for analyzing a diaper feces image includes the steps of: obtaining an image of the diaper; identifying fecal objects and color codes from the image; pre-processing the feces object based on the color code; and, based on an image processing solution through deep learning, generating a stool condition determination result by processing the pre-processed stool object.

Description

How to provide an image processing solution for analysis of diaper stool images {METHOD FOR PROVIDING IMAGE PROCESSING SOLUTION FOR ANALYSIS OF DIAPER STOOL IMAGE}

The embodiments below relate to a technology for providing an image processing solution for analysis of diaper stool images.

The baby health screening market is growing rapidly. While the fertility rate is falling, the markets for baby products, baby food, and baby health are increasing rapidly as the interest in childrearing of the wealthy increases.

Until now, most of the diagnostic services using sensors are. The smart diaper alarm service, launched by Samsung's in-house startup Monit, is a method that notifies whether the urine is discharged through a smartphone application with a sensor of the diaper. The patent for “A diaper sensor for detecting and differentiating feces and urine.” applied by Google in 2016 is a method of attaching a physical sensor directly to a diaper, similar to Monit’s service established by Samsung’s in-house venture investment.

So far, there are sensing-based services for baby health diagnosis, such as ECG, oxygen level, nearby fine dust, and body temperature.

According to an embodiment, the purpose is to generate a stool state determination result by processing a pre-processed stool object based on an image processing solution through deep learning.

The object of the present invention is not limited to the object mentioned above, and other objects not mentioned will be clearly understood from the description below.

According to one embodiment, there is provided a method of providing an image processing solution for analysis of a diaper stool image, performed by a device, the method comprising: acquiring an image of a diaper; identifying a stool object and a color code from the image; preprocessing the stool object based on the color code; And, based on the image processing solution through deep learning, processing the pre-processed stool object to generate a stool state determination result, an image processing solution providing method for analysis of diaper stool images is provided.

According to an embodiment, based on an image processing solution through deep learning, a pre-processed stool object may be processed to generate a stool state determination result.

On the other hand, effects according to the embodiments are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those of ordinary skill in the art from the description below.

1 is a view for explaining a method of acquiring an image of a diaper according to an embodiment.
2 is a view for explaining a method of determining a diaper stool image according to an embodiment.
3 is a view for explaining a method of providing an image processing solution for analyzing a diaper stool image according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, since various changes may be made to the embodiments, the scope of the patent application is not limited or limited by these embodiments. It should be understood that all modifications, equivalents and substitutes for the embodiments are included in the scope of the rights.

Specific structural or functional descriptions of the embodiments are disclosed for purposes of illustration only, and may be changed and implemented in various forms. Accordingly, the embodiments are not limited to the specific disclosure form, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical spirit.

Although terms such as first or second may be used to describe various elements, these terms should be interpreted only for the purpose of distinguishing one element from another. For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component.

When a component is referred to as being “connected to” another component, it may be directly connected or connected to the other component, but it should be understood that another component may exist in between.

The terms used in the examples are used for the purpose of description only, and should not be construed as limiting. The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that a feature, number, step, operation, component, part, or a combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

In addition, in the description with reference to the accompanying drawings, the same components are given the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted. In describing the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

The embodiments may be implemented in various types of products, such as personal computers, laptop computers, tablet computers, smart phones, televisions, smart home appliances, intelligent cars, kiosks, wearable devices, and the like.

1 is a view for explaining a method of acquiring an image of a diaper according to an embodiment.

Referring to FIG. 1 , the device may acquire an image of a diaper. The device is a device for determining a diaper stool image, and may be implemented as, for example, a hardware module, a software module, or a combination thereof. The apparatus may identify a stool object and a color code from the image, preprocess the stool object based on the color code, and process the preprocessed stool object to generate a stool state determination result.

According to an embodiment, the device may be an electronic device. The electronic device may be a smart home appliance having a communication function. Smart home appliances include, for example, a television, a digital video disk (DVD) player, an audio device, a refrigerator, an air conditioner, a vacuum cleaner, an oven, a microwave oven, a washing machine, an air purifier, a set-top box, and a TV. It may include at least one of a box (eg, Samsung HomeSync™, Apple TV™, or Google TV™), game consoles, an electronic dictionary, an electronic key, a camcorder, or an electronic picture frame.

According to an embodiment, the electronic device includes various medical devices (eg, magnetic resonance angiography (MRA), magnetic resonance imaging (MRI), computed tomography (CT), imagers, ultrasound machines, etc.), navigation devices, and GPS receivers ( global positioning system receiver), EDR (event data recorder), FDR (flight data recorder), automotive infotainment device, marine electronic equipment (e.g. marine navigation system and gyro compass, etc.), avionics, security It may include at least one of a device, a head unit for a vehicle, an industrial or household robot, an automatic teller's machine (ATM) of a financial institution, or a point of sales (POS) of a store.

According to an embodiment, the electronic device is a piece of furniture or a building/structure including a communication function, an electronic board, an electronic signature receiving device, a projector, or various measurement devices. It may include at least one of devices (eg, water, electricity, gas, or radio wave measuring devices, etc.). The electronic device according to an embodiment may be a combination of one or more of the various devices described above. Also, the electronic device according to an embodiment may be a flexible device. Also, it is apparent to those skilled in the art that the electronic device according to an embodiment is not limited to the above-described devices. The term user used in various embodiments may refer to a person who uses an electronic device or a device (eg, an artificial intelligence electronic device) using the electronic device.

An electronic device according to an embodiment includes a processor, a memory, a user interface, and a communication interface, and may be connected to another electronic device through a network. The communication interface may transmit/receive data to and from another electronic device within a predetermined distance through a wired or wireless network or wired serial communication. The network enables wired and wireless communication between the electronic device and various entities according to an embodiment. The electronic device may communicate with various entities through a network, and the network may use standard communication technologies and/or protocols. At this time, the network includes, but is not limited to, the Internet, a local area network (LAN), a wireless local area network (Wireless LAN), a wide area network (WAN), a personal area network (PAN), and the like, A person skilled in the art of communication technology can recognize that it may be another type of network capable of transmitting and receiving information.

The portable terminal according to an embodiment may be an electronic device including a communication function. For example, the electronic device includes a smart phone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop personal computer (PC), and a laptop computer. PC (laptop personal computer), netbook computer (netbook computer), PDA (personal digital assistant), PMP (portable multimedia player), MP3 player, mobile medical device, camera, or wearable device (eg: at least one of a head-mounted-device (HMD) such as electronic glasses, electronic apparel, electronic bracelets, electronic necklaces, electronic accessories, electronic tattoos, smart cars, or smartwatches. can

2 is a view for explaining a method of determining a diaper stool image according to an embodiment.

Referring to FIG. 2 , when the device captures a diaper using a smartphone application, it may include a solution that analyzes the diaper image and provides health status information according to color and shape from the baby's feces.

For example, when a parent inputs basic information such as the baby's birth date, gender, and whether breastfeeding or not through the application, the condition of the stool analyzed based on the image of the diaper, the health diagnosis result and cause according to the condition of the stool, pediatrics You can receive customized stool status guide information according to the need for a visit and the number of days after birth.

It is possible to standardize images that change according to the shooting environment and camera specifications, attach a “color code” to one side of the diaper to prevent analysis distortion, and analyze images with “color codes” based on the learned model. .

From the point of view of the diaper consumer, the consumer group, baby stool is a measure of health, but parents who check their baby's stool every day while changing diapers can take pictures of the baby's stool and provide health-related information about the baby who has difficulty communicating. have.

From the point of view of a parent who is raising a child, which is the place of use, the baby's health status, child care information, and information about nearby hospitals can be provided from the diaper stool image.

From the diaper company's point of view, if a business alliance with a diaper company is promoted based on a diaper stool image analysis solution and "color code", the demand for a "smart diaper solution" may increase.

From the point of view of the pediatric hospital, since the diaper stool image can be used as an auxiliary for diagnosis, there may be inquiries for partnership with the baby stool analysis platform. can be connected

The device may analyze the diaper image and provide health status information according to color and shape from the baby's stool.

When basic information such as date of birth, gender, and whether breastfeeding is entered through the application, the condition of the stool analyzed based on the image of the diaper, the medical examination result and cause according to the condition of the stool, the need to visit the pediatrician, and the number of days after birth You can receive customized stool condition guide information.

A “color code” is attached to one side of the diaper, and an image containing the “color code” can be analyzed based on the trained model.

Pediatricians are already diagnosing health conditions from baby feces, and taking diapers to a specialist is recommended in pediatric diagnosis. If image processing technology is used, it is possible to train specialists on the diagnosis of diaper stool photos.

While the fertility rate is falling, the market for baby products, infant food, and baby health is increasing rapidly as the interest in childcare from the wealthy increases. There is a service based on sensing of information.

Just as Samsung’s in-house startup Monit is collaborating with Yuhan-Kimberly for a service that notifies whether urine is discharged from a sensor in the diaper, it attracts investment from diaper companies with a “color code” based diaper stool diagnosis solution, and through collaboration with pediatrics You can create a business model.

In order to correct image distortion, “color code” is attached to one side of the diaper, and the image with “color code” is applied to pre-trained SegNet and ResNet models to classify stool status, and for rapid pediatric diagnosis It features a technical arrangement for processing diaper stool images using “color codes” in a way that provides preliminary information.

Auggi, a startup founded by MIT/Harvard researchers in 2019, is writing a thesis that classifies stool images as constipation/normal/diarrhea, and is preparing a service based on this.

auggi simply provides results of stool image analysis. In contrast, according to an embodiment of the present invention, since labeling and training are performed according to the baby's birth date in order to analyze the baby's diaper stool image, which is difficult to express his/her condition in language, a standardized background image It is possible to determine a more accurate diagnosis result from In addition, compared to auggi, this founding item is technically differentiated in that it performs image pre-processing using "color code".

The patent “A diaper sensor for detecting and differentiating feces and urine.”, which was applied by Google in 2016, is a method of directly attaching a physical sensor to a diaper, similar to Monit’s service established by Samsung’s in-house venture investment. Although only collecting information by mounting a sensor, according to an embodiment of the present invention, there is a distinctive feature in that the color and shape of feces containing important health status information are analyzed.

After developing a diaper stool image diagnosis solution, diaper purchase coupons are presented as an incentive for shooting to secure users, and from the diaper company's point of view, it is possible to increase customer loyalty to diapers by providing baby health diagnosis information, even if it is costly. The obtained stool image can be additionally used for training to increase the solution accuracy.

Currently, we use a diaper sensor to provide a toilet and urine alarm, and as an additional service, we can secure users and increase sales by presenting a baby health diagnosis solution. ) + It is possible to extend the subscription service that includes diapers as an item by presenting it as a baby's health condition diagnosis.

Based on the preliminary results of the baby's health condition diagnosis, a nearby pediatrician can be recommended, and a profit model can be created by using the stool image for pediatric diagnosis. Business settlement is possible.

It is the baby's "feces condition", which is a measure of health abnormalities, and the health condition of a baby with limited communication can be diagnosed from the color and shape of the stool. required for an accurate diagnosis. For an accurate diagnosis, parents show a picture of their baby's stool to a pediatrician or bring a diaper to the doctor's office because the baby's stool, which changes according to the number of days after birth, is a measure of health. For example, during the first week of life, the condition of the stool changes every day, and after 5 days after birth, golden stool is excreted from breast milk, and after a week, the condition of the stool changes to yellow or brown depending on whether the child is breastfed, black stool or black stool. If you have gray stools, you should take a diaper with the stool on it and see a doctor. However, it is difficult to determine whether parents should visit the hospital by looking at the state of the feces containing a lot of health information, and it is also inconvenient to bring diapers to the hospital.

Diagnosis technology using machine learning technology is developing, and the baby health diagnosis market is rapidly growing due to the development of artificial intelligence technology. Until now, most services that diagnose using the information collected using sensors are (ECG, oxygen level, nearby microscopic dust, body temperature, etc.). As the use of artificial neural networks such as CNNs increases, the technology for analyzing medical information through images is also advancing. The number of use cases for radiologists to diagnose through machine learning-based MRI/CT analysis technology is increasing, and the popularization of medical diagnostic services is accelerating.

It is possible to diagnose a baby's health condition by "analyzing a stool image", and according to an embodiment of the present invention, a stool containing baby health diagnosis information is photographed, and the photographed image is analyzed using deep learning technology to determine a specialist Can provide preliminary and supportive baby health-related information. In a situation where the smart diaper market is also expanding and services that combine creative IT technology with traditional diapers are being launched one after another, if a diagnostic service is launched without a separate sensor from feces directly related to baby health, it You can connect pediatrics.

A classification system for diaper stool diagnosis can be established. There is a medical classification system called the Bristol stool scale that diagnoses health conditions such as chronic gastrointestinal disease through the shape and color of adult stool. A baby's stool changes shape and color depending on the number of days of birth and whether or not she is breastfed, and the digestive ability is incomplete in the process of forming teeth and digestive organs. It is important to provide to parents. In the initial implementation stage, it is necessary to start by providing simple information such as whether there is an abnormality, but as the solution perfection increases, it is possible to build a classification system suitable for deep learning-based image processing technology by securing a large amount of images.

It is possible to provide an advanced solution from 'Monit''s smart diaper alarm. In 2016, Monit, a startup affiliated with Samsung Electronics' in-house venture program 'C-LAB', launched a service that alerts you to urination using a 'sensor that observes the inside of a diaper in real time' as a result of about a year of research and development, and Yuhan-Kimberly Sales of 1.2 billion won were achieved by launching a joint product with A service such as monit is currently a method of attaching a sensor to a diaper, but there is no service that utilizes health information obtained from a stool condition so far, and according to an embodiment of the present invention, preliminary diagnosis of a health condition from baby stool information It is judged to be able to secure a new market in the baby health market by implementing a solution that

A health information guide that can be checked through baby stool can be provided, and information necessary for parenting can be provided in a form suitable for the baby based on information about the baby, such as the number of months after birth, as well as analyzing the stool image of the diaper. . For example, it can provide information such as food suitable for the number of months, how to train a toilet habit, and how to learn the senses. By acquiring an image of baby stool and tracking its history, you can continuously monitor the baby's health.

The pediatric care process can be streamlined according to the circumstances and needs. Through analysis of baby stool image, parents can identify whether pediatric care is required, and if necessary, it can be connected to a nearby hospital reservation service to promote prompt treatment. During hospital diagnosis, diaper stool imaging results can be provided to provide information necessary for an accurate diagnosis.

It is possible to implement an application for securing a diaper stool image. A diaper stool imaging application can be provided to secure a sample image for training, and as a step before the diagnostic result solution is advanced, an application for pre-processing to a trainingable standard can be implemented by photographing the diaper stool. According to a 2019 paper on classification of stool images (Augmenting Gastrointestinal Health: A Deep Learning Approach to Human Stool Recognition and Characterization in Macroscopic Images, https://arxiv.org/ftp/arxiv/papers/1903/1903.10578.pdf), 650 When training several images, the accuracy of the identification and classification model is about 70% or more, and when training 25200 images, an accuracy of 99.4% can be secured. According to an embodiment of the present invention, it is possible to implement an application for securing a training sample.

The obtained diaper stool image can be labeled. Labeling of images obtained by recruiting two pediatricians can be performed, and three classification models of constipation/normal/diarrhea can be constructed first.

SegNet and ResNet can be trained based on labeled images. First, we can label 500 training images, and based on the labeled training images, we can train a model to identify stool shapes with SegNet. <SegNet architecture, SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation> It is possible to train a model that performs color-based classification using ResNet on stools identified based on SegNet. <Network architectures for ImageNet, Deep Residual Learning for Image Recognition>

Diaper stool image-based diagnostic solutions can be implemented. By utilizing SegNet and ResNet that have learned labeled images, a solution for analyzing diaper stool images can be implemented.

An application equipped with an image distortion correction function can be implemented by utilizing a diaper with "color code" attached. Distortion problems may occur due to image conditions that change depending on the shooting environment or the specifications of the image sensor. It can improve the accuracy of diagnosis. Once the learning of SegNet and ResNet is completed, they can collaborate with diaper companies to attract investment while developing a business model that utilizes “color codes”. In this process, VC investment can also be attracted.

Business models can be upgraded through VC investment attraction and cloud funding. By attracting seed investment, you can increase capital, perform team building, and link accelerating. Cloud funding can promote service and generate revenue.

3 is a diagram for explaining a method of providing an image processing solution for analyzing a diaper stool image according to an embodiment.

First, the Change Deep Neural Network. Regarding Reason for change, in Reason for change, the deep learning model used previously is Segnet, and the technology is a thesis presented in 2015. occurred. Of course, there are model problems, but since the amount of data used for training was small, good results could be obtained by increasing the data, but data could not be obtained under the current situation.

Referring to (a) of FIG. 3 , the gray part is the part recognized as a patch in deep learning, and the white part is the part recognized as poop. Therefore, in order to compensate for the lack of data as much as possible, we proceeded in the direction of trying to use a better model.

Referring to (b) of FIG. 3 , in the Change Deep Neural Network, it was attempted to utilize the deeplab v3 model released in 2017 rather than Segnet if it was used previously. In this model, Resnet-50 and Resnet-101 can be used as backbones, and Resnet-50 was used in this part. This model was selected because it has the lightness, speed, and accuracy performance of the model, and it is also worth considering using the backbone as Resnet-101 to further increase the accuracy or the situation in which the recognition is slow. Therefore, the model was changed by running training including the newly acquired data in the above model.

Regarding Color Detection in Patch, in the previously used Patch Color Detection part, after recognizing the patch through deep learning, calculations were performed using cv2.findcontours and cv2.minarearect of opencv.

Referring to (c) of Figure 3, the red dot is a predicted rectangle. The blue dots are the patch color locations. I was able to confirm that it was recognized as the image. The algorithm had a problem in that it could not predict a correct rectangle if the patch was lying or curved.

Therefore, in order to solve the problem, cv2.getPerspectiveTransform and cv2.warpPerspective are used to detect only the patch part, and then crop only the relevant area for analysis.

Referring to (d) of FIG. 3 , a Color value is obtained based on the point on the right in this way.

In relation to Visualization for Patch Colors, the data distribution of the corresponding color is as shown in (e) of FIG. 3 .

Referring to (e) of FIG. 3 , the distribution shows the distribution of data in a blue circle for each patch color.

Regarding Using Mean, Standard derivative for delete outlier data in patch, the graph shown above is a request to remove noise on the image after AWS. The Gaussian Distribution used is shown here. In order to remove outliers, only the data included in “-1 * std < value < 1 * std” corresponding to 68.26% of the reliability of the distribution is used, and the rest of the data is deleted. . Therefore, the influence on the transformation model by the outlier data was removed.

Regarding evaluation, MSE (Mean Squared Error) was used as an index indicating how well the model was transformed, and the total experimental cases are as follows.

1. In transformation via Linear Regression, a) remove bouncing values from patch color x, b) remove bouncing values from patch color o

2. In transform via BeyesianRidge a) remove bouncing values from patch color x, b) remove bouncing values from patch color o

3. In transform via AdaBoostRegressor a) remove bouncing values from patch color x, b) remove bouncing values from patch color o

1. With respect to transformation through linear regression, referring to FIG. 3(f), when looking at the output result, it was confirmed that removing the outlier data has better accuracy than not removing it.

2. In relation to conversion through the BeyesianRidge, referring to FIG. 3 (g), it was possible to confirm the output result.

3. Regarding the conversion through AdaBoostRegressor, referring to Fig. 3 (h), if you look at the model, the mse value is very low, so the patch color itself is very well fitting, but on the contrary, it fits only the patch too much, so it is an image of another part showed a tendency to destroy Also, since the model is too heavy, it takes a lot of computation time.

The embodiments described above may be implemented by a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the apparatus, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA) array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (1)

A method of providing an image processing solution for analysis of diaper stool images performed by a device, the method comprising:
acquiring an image of the diaper;
identifying a stool object and a color code from the image;
preprocessing the stool object based on the color code; and
Based on an image processing solution through deep learning, comprising the step of processing the pre-processed stool object to generate a stool state determination result,
A method of providing an image processing solution for analysis of diaper stool images.

Images