KR102237583B1 - Estimating bilirubin levels - Google Patents

Abstract

Systems, methods and devices are provided for estimating bilirubin levels. In one aspect, a method for estimating the level of bilirubin in a patient includes receiving image data for at least one image comprising a color correction target and an area of the patient's skin. Color-balanced image data for the skin area is generated based on a subset of image data corresponding to the skin area and color correction target. The patient's bilirubin level is estimated based on color-balanced image data for the skin area.

Description

Estimation of bilirubin level {ESTIMATING BILIRUBIN LEVELS}

This application claims priority to U.S. Provisional Application No. 61/777,097, filed March 12, 2013, which application is incorporated herein by reference.

An estimated 60-84% of neonates have neonatal jaundice, which becomes yellowing of the skin caused by the accumulation of natural compounds produced by the destruction of red blood cells, i.e., excessive bilirubin (hyperbilirubinemia). Although these conditions are typically harmless and heal spontaneously within a few days, very elevated bilirubin levels can cause irreversible damage to nuclear jaundice, i.e. deafness, cerebral palsy, deep developmental delay, or even death. It can lead to neurological lesions.

Current approaches to monitoring bilirubin levels in infants typically require repeated testing in a hospital setting. The blood level of bilirubin can be determined either through a transcutaneous bilirubinometer (TcB) measurement achieved using a non-invasive or expensive tool, or by total serum bilirubin (TSB) measured from a blood sample. These tests are often not available in resource-poor settings, thus delaying early detection and treatment of nuclear jaundice. Visual assessment, often used as an alternative to these tests, is often inaccurate and can be confused by factors such as lighting or skin tone. Therefore, there is a need for an improved approach to provide non-invasive and cost-effective screening for excessive bilirubin levels.

Systems, methods and devices are provided for estimating bilirubin levels. In many embodiments, a mobile device is used to capture image data of a color correction target and a patient's skin. Image data is used to generate an estimate of the bilirubin level. Image processing may include converting image data into a plurality of different color spaces to facilitate evaluation of the overall yellowness of the skin while compensating for color differences caused by lighting, skin color, and other potential confounding factors. . The screening techniques described herein do not require specialized medical equipment and can be implemented by users (e.g., patients, medical professionals, community health personnel) in an outpatient clinic (e.g., a patient's home). , Improve the convenience, accessibility and cost-effectiveness of bilirubin monitoring.

Thus, in a first aspect, a method for estimating the level of bilirubin in a patient is provided. The method includes receiving image data for at least one image comprising a color correction target and an area of the patient's skin. Color-balanced image data for the skin region is generated based on a subset of image data corresponding to the skin region and a color correction target. The patient's bilirubin level is estimated based on color-balanced image data for the skin area. In many embodiments, image data may be acquired with any suitable imaging device. The imaging device may collect image data independent of additional attachments or equipment, such as external lenses, filters or other specialized hardware.

The bilirubin level may be estimated by using only image data of a skin color of a patient at a specific time point or by comparing basic skin color image data and skin color image data. For example, the method may further comprise receiving baseline skin color data for a corresponding patient (e.g., approximately 0 when the baseline data is obtained within 24 hours after birth) when the patient has a baseline bilirubin level. . The bilirubin level may be estimated based on one or more differences between the color-balanced image data for the skin area and the basic skin color data. Base skin color data for a patient can be generated by capturing base image data for a patient when the patient has a baseline bilirubin level. The basic image data may correspond to at least one image including a basic color correction target and a skin area. Based on a subset of the basic image data corresponding to the skin area and the basic color correction target, color-balanced basic image data for the skin area may be generated. Basic skin color data may be generated based on the color-balanced basic image data for the skin area.

Standardized color correction targets can be used to facilitate the color balancing process. The color correction target may include a plurality of standardized color areas including a white area. The standardized color area may include a black area, a gray area, a light maroon area, a cyan area, a magenta area, a yellow area, and a dark brown area. The color correction target may at least partially define an opening configured to expose the skin region to enable capturing of image data for the skin region. The standardized color gamut can be arranged in a known arrangement surrounding the opening. Accordingly, the process for generating color-balanced image data for a skin region is to identify a subset of image data corresponding to the exposed skin region and a subset of image data corresponding to the white region, the received image data May include processing. The white area data can be processed to determine the observed color values for the white area. Color-balanced image data for the exposed skin area may be generated based on the observed color values for the white area. The observed color values for the white region may include any suitable color space values such as red, green and blue (RGB) color space values.

Image data may be converted into a plurality of different color spaces to detect yellow discoloration of the skin. For example, the color-balanced image data for the skin area may include RGB color space data, and a method of estimating the level of bilirubin of a patient is the color of the exposed skin area for at least one other color space. In order to generate the balanced image data, the step of converting the RGB color space data into at least one other color space may be further included. The at least one color space includes: (a) a cyan, magenta, yellow, and black (CMYK) color space; (b) YCbCr color space; And/or (c) Lab color space.

A plurality of achromatic and chromatic features may be generated based on image data. In some instances, the received image data may include images acquired using flash lighting and images acquired without using flash lighting. Estimating the bilirubin level may include processing a plurality of normalized achromatic and chromatic features to select a first estimated range of bilirubin levels from one of the plurality of different bilirubin ranges. The feature can be processed using an approach based on a selected first estimated range of bilirubin levels to produce a final estimate of the bilirubin level. The plurality of different bilirubin ranges may include the low range, the middle range and the high range. The plurality of features may include selected color values of the skin region for a plurality of different color spaces. In some instances, the plurality of features may include calculation of a color gradient across the skin area.

Estimating the bilirubin level may involve performing one or more regressions. For example, processing features to select a first estimated range of bilirubin levels may include (a) linear regression, (b) encapsulated k-Nearest Neighbor regression. ), (c) lasso regression, (d) LARS regression, (e) elastic net regression, (f) support vector regression using a linear kernel vector regression), (g) support vector regression that assigns high weights to higher-rated bilirubin values, and (h) random forest regression. It may include the step of performing. Processing the features to select the first estimated range of bilirubin levels includes (a) linear regression analysis, (b) encapsulated k-nearest neighbor regression analysis, (c) Lasso regression analysis, and (d) LARS regression analysis. , (e) elastic net regression, (f) support vector regression using a linear kernel, (g) support vector regression that assigns high weights to high-grade bilirubin values, and (h) random forest regression. It may include performing a series of regression analysis. The step of using the processing approach based on the selected first estimated range of bilirubin levels is the final use of the selected first estimated range of bilirubin levels and a plurality of normalized achromatic and chromatic features as features for the final random forest regression. It may include performing a random forest regression analysis. In some instances, estimating the bilirubin level includes determining a color space value for the patient's skin and using a processing approach based on the determined color space value to estimate the bilirubin level. The regression equation may also include features from the base image (eg, color space values from one or more color spaces).

In another aspect, a mobile device configured to estimate the level of bilirubin in a patient is provided. The device includes a camera operable to capture image data for a field of view, a processor operatively coupled with the camera, and a data storage device operably coupled with the processor. The data storage device may store instructions, which, when executed by the processor, cause the processor to receive image data for an image captured by the camera, the image including a color correction target and an area of the patient's skin. . The instructions cause the processor to generate color-balanced image data for the skin region, based on the subset of image data corresponding to the skin region and color correction target, and based on the color-balanced image data for the skin region. Thus, the patient's bilirubin level can be estimated. In many embodiments, the mobile device may be used to estimate the bilirubin level regardless of any external attachments (eg, lenses, filters) to the mobile device or any other specialized mobile device equipment.

The color correction target may at least partially define an opening configured to expose the skin region, and may include a plurality of standardized color regions including a white region in order to enable capturing of image data for the skin region. . The instructions may cause the processor to process the received image data to identify the subset of image data corresponding to the exposed skin area and the subset of image data corresponding to the white area. The white area data can be processed to determine the observed color values for the white area. Based on the observed color value for the white area, color-balanced RGB image data for the exposed skin area may be generated. By converting the color-balanced RGB image data into at least one other color space, color-balanced image data of an exposed skin area for at least one other color space may be generated. A plurality of normalized achromatic and chromatic features may be processed to select a first estimated range of bilirubin levels from one of a plurality of different bilirubin ranges. The feature can be processed using an approach based on a selected first estimated range of bilirubin levels to produce a final estimate of the bilirubin level.

Alternatively or additionally, color-balanced image data for the exposed skin area may be processed to determine a color space value for the patient's skin. A plurality of normalized achromatic and chromatic features can be processed using an approach based on the determined patient's skin color space value to estimate the bilirubin level.

The mobile devices described herein may further include a flash unit operable to selectively illuminate a field of view. Received image data processed to estimate the bilirubin level may include an image captured with a field of view illuminated by the flash unit and an image captured with a field of view not illuminated by the flash unit.

In another aspect, a method for estimating the level of bilirubin in a patient is provided. The method includes receiving image data from a mobile device for an image comprising a color correction target and a skin area of a patient. Color-balanced image data for the skin region may be generated based on a subset of image data corresponding to the skin region and a color correction target through one or more processors. The patient's bilirubin level may be estimated based on color-balanced image data for the skin area through one or more processors. The estimated bilirubin level may be transmitted to the mobile device. In many embodiments, at least one of receiving the image data and transmitting the estimated bilirubin level is performed using text messaging service (SMS) text messaging. Image data can be acquired by the mobile device without the use of external attachments, hardware add-ons, or any other specialized equipment.

Other objects and features of the present invention will become apparent upon examination of the specification, claims, and accompanying drawings.

The novel features of the invention are set out in particular in the appended claims. Features and advantages of the present invention will be more readily understood upon reference to the following detailed description and accompanying drawings, which present exemplary embodiments in which the principles of the present invention are utilized.

1A shows guidelines for using phototherapy to treat neonatal jaundice, according to an embodiment;

1B shows guidelines for using exchange transfusions to treat neonatal jaundice, according to an embodiment;

1C depicts a Bhutani nomogram for assessing the risk associated with neonatal jaundice;

2 depicts using a mobile device to capture image data used to estimate a patient's bilirubin level, in accordance with a number of embodiments;

3A-3D illustrate a color correction target used with a mobile device to capture image data used to estimate bilirubin levels, in accordance with a number of embodiments;

4A-4C illustrate an exemplary user interface of a mobile application for estimating a bilirubin level, in accordance with a number of embodiments;

5A-5F illustrate exemplary image data collected for use in estimating bilirubin levels, in accordance with a number of embodiments;

6 depicts a method for estimating a patient's bilirubin level, in accordance with a number of embodiments;

7 illustrates a method for generating color-balanced image data for a patient's skin, in accordance with a number of embodiments;

8 shows the identification of a standardized color gamut in an image, according to a number of embodiments;

9A illustrates a method for estimating a patient's bilirubin level, in accordance with a number of embodiments;

9B shows another example of a method for estimating a patient's bilirubin level, according to a number of embodiments;

10 depicts a method for estimating a patient's bilirubin level, in accordance with a number of embodiments;

11A shows a method for estimating a bilirubin level using basic skin color data, according to a number of embodiments;

11B shows a method for generating basic skin color data, according to a number of embodiments;

12 illustrates a mobile device for estimating a bilirubin level, in accordance with a number of embodiments;

13 illustrates a mobile device in communication with a data processing system for estimating a bilirubin level, in accordance with a number of embodiments.

The systems, devices, and methods described herein provide an improved approach for estimating bilirubin levels in a patient (eg, infant or adult). A mobile device configured with suitable software can be used to capture an image of the patient's skin and a standardized color correction target. The image data of the skin and the target may be used to generate color-balanced image data that is analyzed to estimate a patient's bilirubin level. For example, color-balanced image data can be transformed into multiple different color spaces to extract features representing yellowing of the skin, and these features can be used in multiple regressions to generate bilirubin estimates. . In contrast to conventional bilirubin level measurement methods that rely on invasive blood testing or utilize expensive instrumentation, the systems, devices and methods described herein are easily performed by non-medical personnel using personal mobile devices. It enables convenient, portable, and inexpensive bilirubin level estimation that can be made, thus improving the accessibility and cost-effectiveness of bilirubin monitoring. Advantageously, the methods described herein can be performed on a mobile device without requiring the use of external attachments, hardware add-ons, or any other specialized mobile device equipment. In particular, the disclosed technique provides accurate bilirubin level estimation over a wide range of bilirubin concentrations, as opposed to TcB measurements that are less accurate at high bilirubin levels. Additionally, the methods described herein improve the accuracy and flexibility of non-invasive bilirubin level estimation, taking into account the diversity of different lighting states as well as skin color. In addition, the use of a mobile device software platform enables easy and fast updating of the estimation methods and algorithms described herein, thus allowing improvements and upgrades to be made as needed.

Referring now to the drawings, FIG. 1A illustrates guidelines for using phototherapy to treat neonatal jaundice. Similarly, FIG. 1B illustrates guidelines for using exchange transfusions to treat neonatal jaundice. These guidelines can be used by healthcare professionals to determine the appropriate course of treatment based on the infant's age, total serum bilirubin (TSB), number of weeks of gestation (eg, over 35 weeks), and other risk factors. Risk factors may include alloimmune hemolytic disease, G6PD deficiency, asphyxiation, marked lethargy, temperature instability, sepsis, acidosis, or albumin levels lower than 3.0 grams per deciliter. The curves shown in FIGS. 1A and 1B represent exemplary treatment thresholds for low risk, medium risk and high island infants.

FIG. 1C illustrates a Bhutani nomogram 110 for evaluating the risk associated with jaundice in a newborn child. The butani nomogram 110 may be used by a medical professional to evaluate an infant's risk for developing hyperbilirubinemia based on the infant's postnatal age and bilirubin level. For example, the butani nomogram 110 may be used to define a low-risk zone 114, a low-medium-risk zone 116, a high-medium-risk zone 118, and a high-risk zone 120. May include a percentile curve 112 of.

2 illustrates a mobile device-based estimation of a bilirubin level, in accordance with multiple embodiments. An imaging device such as a mobile device 202 (eg, a smartphone, tablet) is used to capture images of the color correction target 208 and the skin area 204 of the patient 206. Areas of skin suitable for the approaches described herein include the forehead and sternum, as well as any other significantly flat areas of the skin that are likely to be uniformly illuminated. Additionally, because jaundice typically first appears on the forehead and progresses slowly downwards in the body, areas closer to the forehead can potentially be more beneficial for diagnosis. Thus, in many embodiments, the color correction target 208 is on the abdomen of the patient 206 near the sternum. The mobile device 202 includes a camera for recording image data (not shown) as well as a mobile software application for estimating the bilirubin level based on the image data (“mobile app” or “app”). It includes a display 210 used to show a user interface (UI) of. The mobile device 202 may be a personal device of a user (e.g., a parent, a medical professional, a community health worker, etc.), so that the user may, in order to perform the methods described herein, the color correction target 208 No other instrument equipment or hardware is required, and only the mobile app needs to be installed on one's own personal device. In particular, mobile device 202 may be used to practice the methods described herein independent of any additional attachments or accessories of the mobile device, such as external lenses, filters or other specialized mobile device equipment. Additional details regarding software and hardware components suitable for mobile device 202 are provided in more detail below.

3A-3D illustrate a color correction target that may be used with a mobile device to estimate bilirubin levels, in accordance with a number of embodiments. The color correction targets described herein (also referred to as “color calibration cards” or “color cards”) are lighting that affects the resulting color balance of skin image data. Or it may be used to account for differences in other environmental conditions. Referring to Fig. 3A, a color correction target 300 may be provided on a rectangular card 302. The card 302 is a patient (e.g., It may be of any size suitable for placement on the skin of an infant), for example a cardstock of approximately the size of a light to dark. In some cases, the card 302 is designed to prevent the spread of pathogens among patients. The color correction target 300 may include a plurality of standardized colored regions 304 that may be printed on the card 302. 304) may be any suitable size, number, or shape (eg, square, rectangle, polygon, circular ellipse, etc.) For example, the color correction target 300 may have eight equal sized square colors. Each of the standardized color regions 304 is shown to have a different color (eg, black, gray, white, cyan, magenta, yellow, pale maroon, dark brown). And positioned in a known arrangement on the card 302. For example, in the embodiment of Figure 3A, 304a is a black area, and 304b is a gray area (eg, 50% gray). ), 304c is a white area, 304d is a pale maroon color (e.g., first skin color), 304e is a cyan area, 304f is a magenta area, 304g is a yellow area, and 304h is a dark brown area (e.g. For example, the second skin color). Other arrangements and combinations of colors may also be used. In addition, the back side of the card 302 (not shown) shows that the card 302 is It may include one or more areas of adhesion to allow removably adherence to the skin. The back side may also include related instructions, such as instructions for the user on how to download the attached mobile app on the user's personal mobile device.

3B illustrates an alternative embodiment of a color correction target 320. The color correction target 320 is substantially similar to the color correction target 300 except that the color correction target 320 also defines an opening 322. The opening 322 may be positioned so that the skin region of interest is exposed through the opening 322 when the color correction target 300 is placed on the patient. The opening 322 is defined entirely (eg, completely enclosed) or partially defined (eg, partially enclosed) by the target 320. A plurality of standardized color regions 324 including white regions may be positioned around openings 322 in a known arrangement. The embodiment of FIG. 3B further includes light gray, medium light gray, and dark gray areas for a total of 10 different normalized color areas. 3C illustrates a color correction target 340 provided on a square card 342, with a plurality of standardized color regions 344 surrounding the square opening 346. Some of the color areas 344 may include two or more colors, such as the color area 348 including a central black square bordered by white. 3D illustrates a color correction target 360 having a plurality of standardized color gamuts 362 surrounding an opening 364. The opening 364 may be adjacent to some or all of the color area 362. Color gamut 362 is shown in FIG. 3D as a series of rectangular areas with different aspect ratios. The color, geometry, and arrangement of the color gamut of the color correction target described herein may be selected based on the image processing and analysis method to be used, such as the embodiments discussed below.

4A-4C illustrate an exemplary user interface (UI) of a mobile app for estimating a bilirubin level, in accordance with a number of embodiments. 4A illustrates a summary UI 400 for displaying various statistics and metrics pertaining to a patient. UI 400 includes patient identification information 402 (e.g., patient name, study ID number), time of birth 404, and any available bilirubin level results 406 (e.g., TSB and/or TcB). Result). The UI 400 may also include a button 408 or other interactive element that allows a user to collect image data (eg, a video sample or a photo sample) of a patient. In some cases, video samples can be beneficial in eliminating motion blur problems. 4B illustrates a command UI 420 to help the user capture the patient's already data. The UI 420 may include a graphic and/or text command 422 instructing the user to perform an appropriate step. For example, command 422 may instruct the user to place a color correction target on the patient's abdomen below the sternum. In many embodiments, instructions 422 are adapted to be easily understood by individuals without medical education, allowing non-medical professionals to operate the mobile app. 4C illustrates a live preview 440 displayed to the user during the image capture process. The UI 440 may include a video preview 442 of the current field of view of the camera of the mobile device. The UI 440 may include one or more positioning targets 444 to assist the user in positioning the mobile device. For example, the positioning target 444 may be a box, frame, or gamut overlaid on the video preview 422 to indicate placing the color correction target in an appropriate position. The size and position of the positioning target 444 may be selected to limit the distance of the mobile device from the patient to within an optimal range and also to ensure that the field of view sufficiently captures the appropriate skin area and color correction target. Once the fields of the view are correctly aligned, the user can record image data by tapping the record button 446. Additional details regarding the image collection and analysis process are provided below.

5A-5F illustrate exemplary image data collected by a mobile app according to a number of embodiments. After the recording process, the mobile app can display image data to the user for review and approval. In addition, in order to ensure that the captured image is of sufficient quality for processing and analysis, the mobile app may include a function to detect image quality, and images that do not meet the quality threshold and need to be retaken. You can notify the user. 5A illustrates an image 500 of sufficient quality that the lighting is satisfactory and can clearly see both the patient and the color correction target. 5B illustrates a defective image 502 in which portions of the image are obstructed by flashes. 5C illustrates a defective image 504 where the image is too bright. 5D illustrates a defective image 506 in which the color correction target is partially obstructed. 5E illustrates a defective image 508 in which the image has been partially blurred by a shadow. 5F illustrates a defective image 510 where the image is too dark.

6 illustrates a method 600 for estimating a patient's bilirubin level in accordance with a number of embodiments. Method 600, as with all other methods described herein, includes any of the devices and systems disclosed herein, such as a mobile device or a separate computing system (e.g., a remote server) in communication with a mobile device. Can be implemented using. Certain steps of method 600, like all other methods described herein, may be optional or may be combined with or used in place of the appropriate steps of other methods disclosed herein.

In operation 610, image data for at least one image including a color correction target and a region of the patient's skin is received. Image data may be collected using the camera of the mobile device as previously described herein. Image data may include photographic data, video data, or any suitable combination thereof. Photo data and video data can be captured sequentially or simultaneously (eg, images are taken during video recording). Image data may be obtained with and/or without flash lighting. In some cases, flash lighting can be used to offset environmental lighting, so the lighting of the resulting image is determined solely by the contribution of the flash lighting. This may be beneficial for producing more consistent lighting, or in situations where environmental lighting is suboptimal (eg, too dark, strong colors, etc.).

In many embodiments, image data is collected using a flash unit of the mobile device as well as a mobile app that controls the number and sequence of images taken. For example, the app can turn on the flash unit during the initial positioning of the mobile device, so the user can evaluate the amount of flash in the image (e.g. via video preview) to reduce the flash or The device can be repositioned as needed to remove it. During the recording process, the app first acquires image data by turning the flash unit on and then off the flash unit to acquire the image data, creating two sets of images, each with a flash light and no flash light. You can control your mobile device to do so. For example, when the user initiates the image capture processor (by pressing the record button), the app is configured to record video of the patient and calibration target by turning on the flash unit during the first half and turning off the flash unit during the second half. Can be. The app can also capture two photos taken during the first half and the second half of the video recording, respectively. The total length of the video can be any suitable time, for example approximately 10 seconds. The timing of the image capture process can be further configured to ensure that the mobile device's image sensor (eg, a charge-coupled device (CCD) array) stabilizes before the next image is taken. For example, the app may include a specific amount of delay (eg, 3 to 4 seconds) before recording each set of images.

As previously discussed, once the image capture sequence is complete, the mobile app can already analyze the data to determine whether the image data is of sufficient quality for subsequent use. For example, an app may implement an image analysis technique (eg, computer vision) suitable for evaluating image quality. An exemplary procedure is to extract a color correction target from the captured image data, (e.g., to determine whether the standard deviation of pixel values for each color gamut falls below a predetermined threshold) of the correction target. It entails checking color consistency across each gamut, and recommending that the user retake any images that fail this test. In some instances, the app may be configured to capture multiple sets of image data per session to maximize the likelihood that at least a portion of the image data will meet quality standards. Thereafter, as described in more detail below, the approved image may be processed on the mobile device or transmitted to a separate computing system for processing.

In operation 620, color-balanced image data for the skin region is generated based on the subset of image data corresponding to the skin region and the color correction target. Color balancing is used to compensate for different lighting conditions because the color content of image data can vary considerably with environmental lighting (e.g., intensity, color, type (halogen fluorescent lighting, natural lighting, etc.)). Can be done. In many embodiments, the image data is initially normalized by dividing the three red, green, and blue (RGB) color channels by the total luminescence of the image. In addition, as discussed below, the observed pixel color values for one or more of the standardized color gamuts of the color correction target can be used to determine the color balancing adjustment to be applied to the skin area in the image data.

In operation 630, the patient's bilirubin level is estimated based on color-balanced image data for the skin area. Since hyperbilirubinemia causes yellowish discoloration of the skin, the bilirubin level can be determined based on the amount of yellow in the color-balanced skin area image data. This determination can be made using any suitable technique, for example by transforming the color-balanced image data into a plurality of different color spaces selected to facilitate quantifying the overall yellowing of the skin. Image features generated from the transformed image data can be input into a series of machine learning regressions designed to estimate the concentration of bilirubin in the patient's body. This approach is discussed in more detail below.

7 illustrates a method 700 for generating color-balanced image data in accordance with a number of embodiments. The method 700 may be applied to photographic and/or video data obtained from a skin region and color correction target of a patient, as previously discussed. In operation 710, the received image data is processed to identify image data corresponding to the exposed skin area and image data corresponding to the white area. For example, the image data may be segmented to extract pixel values of a skin region of interest (sternum, forehead) and a color region of a color correction target. For the color correction target, if the mobile app UI includes a positioning target for aligning the correction target (for example, the positioning target 444 in FIG. 4C), the approximate pixel coordinates of the color correction target in the image data are previously Is known, and thus the search space for the color gamut may be limited. The positioning of the gamut can be determined by identifying at least two gamuts on the correction target.

8 illustrates the identification of a standardized color gamut in an image, according to a number of embodiments. The gamut segmentation process can be implemented through a suitable algorithm configured to apply a predetermined threshold to the image. In many embodiments, since the cyan, magenta, and yellow regions have individual colors and relatively high saturation, the algorithm initially attempts to identify at least two of these regions. The algorithm converts image data into a hue, saturation, and value (HSV) color space, and applies an empirically determined threshold to the hue and saturation channels to obtain a threshold color image 800 and a threshold saturation image 802. . The "AND" operation can be performed on the two threshold images to separate a specific color gamut from the rest of the image, creating a segmented image 804. In addition, since the approximate size of each gamut is predetermined, this information can be used to distinguish gamut from noise in the image data (e.g., using edge detection, morphological operations, etc.). Can be used to Thereafter, the algorithm may use contour detection to identify the boundary of a color gamut when contour smoothing is performed using an appropriate technique such as a Douglas-Peuker algorithm. Since the overall arrangement of the color regions is known, once the positions of the two color regions are determined, the overall orientation of the correction target can be calculated and used to extrapolate the positions of the other regions including the white region.

에 대하여, 피부 영역에 대한 조절된 색 값(R, G, B)는 하기의 식에 의해 획득될 수 있으며, In operation 720, the white area data is processed to determine an observed color value for the white area. In operation 730, color-balanced image data for the exposed skin area is generated based on the observed color values for the white area. The observed color value for the white area may include an RGB color value that can be used to adjust the RGB value of the image data for the skin area. For example, the observed color value for the white area

For, the adjusted color values (R, G, B) for the skin area can be obtained by the following equation,

는 색 보정 타겟상의 백색 영역의 미가공의 관찰된 색 값이다. here,

Is the raw observed color value of the white area on the color correction target.

9A illustrates a method 900 for estimating a patient's bilirubin level in accordance with a number of embodiments. Method 900 may be practiced using color-balanced RGB image data obtained as previously discussed herein. In operation 910, the color-balanced RGB image data is converted to at least one other color space to generate color-balanced image data of an exposed skin area of at least one other color space. As previously discussed, yellowing of color-balanced skin area image data correlates with the bilirubin level in the patient's body. In many embodiments, the mobile device's image sensor (e.g., a CCD or CMOS sensor) interpolates the reflected light into red, green, and blue wavelengths, which makes it clear that the image sensor captures reflections in the yellow wavelength band. Can be prevented. Thus, the color-balanced RGB data may include a plurality of different color spaces, for example cyan, magenta and yellow (CMY); Cyan, magenta, yellow and black (CMYK); YCbCr; Or it can be converted to Lab color space. Any suitable number and combination of color spaces may be used. For example, in many embodiments, RGB image data may be converted to CMYK, YCbCr, and Lab color spaces. Alternatively, RGB image data can be converted into CMY, YCbCr and Lab color spaces.

In operation 920, a plurality of normalized achromatic and chromatic features are processed to select a first estimated range of bilirubin levels from a plurality of different bilirubin ranges. The features may be achromatic and/or chromatic (eg, luminescence) values for the skin area, each feature corresponding to a color space value of the used color space. In some instances, a feature may include calculation of one or more color gradients across a skin area. Any suitable number and combination of features may be used. For example, three features can be extracted from each of four color spaces (e.g., RGB, CMY, YCbCr, Lab; or RGB, CMYK, YCbCr, Lab) to obtain 12 achromatic and chromatic features. . In addition, features are individually extracted from image data acquired with and without flash illumination, respectively, resulting in a total of 24 chromatic and achromatic features. Features may be normalized based on the overall luminance of the image as previously described herein with respect to operation 620 of method 600. The extracted features can be used as input to machine learning regression analysis used to estimate the bilirubin level. Regression analysis may utilize some or all of the extracted features, and an optimal subset of features to be used is selected based on machine learning regression analysis. The machine learning regression described herein can be trained on an appropriate data set, such as clinical patient data, and incorporate any suitable number and combination of parametric and non-parametric regression models. I can. An exemplary regression analysis suitable for use with the methods described herein is provided below. In many embodiments, the initially computed feature and the output of the selected regression are used to classify the estimated bilirubin level into one of a plurality of different bilirubin ranges, e.g., low, medium and high ranges.

In operation 930, the feature is processed using a processing approach based on a selected first estimated range of bilirubin levels to produce a final estimate of the bilirubin level. Similar to operation 920, the normalized achromatic and chromatic features can be used to inform one or more machine learning regressions. The input to the regression analysis may differ based on whether the first estimated range of bilirubin levels is low, medium, or high, as provided in more detail below. For example, the classification of the first estimated range can be used as an input to machine learning regression analysis. The results of the initial regression analysis performed in operation 920 can also be used as input. This "two-tiered" approach to bilirubin estimation can be used to produce more accurate estimation results compared to direct estimation.

9B illustrates a method 950 for estimating a patient's bilirubin level in accordance with a number of embodiments. Method 950 may be implemented in combination with method 900 to obtain a more accurate estimate of the bilirubin level. Similar to method 900, method 950 may be practiced using color-balanced RGB image data obtained as previously described herein. In operation 960, the color-balanced RGB image data is color-balanced image data of the exposed skin region for at least one other color space as discussed above with respect to operation 910 of method 900. Is transformed into at least one other color space to generate.

In operation 970, the color-balanced image data for the exposed skin area is processed to determine a color space value for the patient's skin. The color space value for the patient's skin can be used to classify the patient's skin color into one of a plurality of different skin color types, for example light skin, medium skin and dark-skin. The skin color type may be related to the race and/or ethnic color of the patient. The skin color type may be determined based on a color correction target obtained from color-balanced RGB image data and/or a color value of a skin area. For example, the skin region may be compared with one or more standardized color regions (eg, first and second skin color regions) of a color correction target to determine a skin color type.

In operation 980, a plurality of normalized achromatic and chromatic features are processed using an approach based on the determined skin color to estimate the bilirubin level. Normalized achromatic and chromatic features may be extracted from color-balanced image data for one or more different color spaces as described above with respect to method 900. The feature can be used as input to an appropriate machine learning regression analysis, along with the determined skin color, similar to operation 930 of method 900.

10 illustrates a method 1000 for estimating a patient's bilirubin level in accordance with a number of embodiments. The method 1000 includes acquiring an image from a camera (1002), color balancing image data (1004), performing feature extraction from color-balanced image data (1006), machine learning based on features. Performing a regression analysis (1008), and generating a bilirubin estimate (1010).

Image data may be obtained from the camera of the mobile device under the control of an appropriate mobile app (act 1002). A part of the image data may be obtained with flash lighting, and a part of the image data may be obtained without flash lighting. Once the app has verified that the image is of sufficient quality for processing and analysis, the image data can be color balanced (act 1004). Color balancing involves identifying a subset of image data corresponding to a color correction target and automatically segmenting the standardized gamut of the target using the thresholding method previously described herein (operation 1012). ). The segmented white region may be used to white balance the image data (operation 1014) to generate color-balanced image data. The color-balanced image data may be RGB image data. One or more features may be color. -May be extracted from the balanced image data (operation 1006) The feature extraction process is to convert the image data from an RGB color space to a plurality of different color spaces as previously described herein (operation 1016). The transformed image data can then be used to calculate a plurality of normalized achromatic and chromatic features (operation 1018).

Some or all of the extracted features may be used as input to machine learning regression (act 1008). Any suitable number and/or combination of regressions can be used. For example, the initial set of machine learning regressions may include five different regressions (actions 1020, 1022, 1024, 1026 and 1028). For example, the first regression analysis may include one or more encapsulated k-Nearest Neighbor (kNN) regression (operation 1020). Such regression analysis can utilize a database of known features and bilirubin values. When an unknown test vector is analyzed, a convex hull can be found around the test vector in the database of features. Then, the feature points from the convex hull can be used in linear regression analysis. A new regression analysis can be built every time a new test point is analyzed. The parameters for generating the convex hull may be a normalized value of luminance (eg, from a YCbCr color space transform) and a “green” channel (eg, from an RGB color space). This can be used to ensure that the convex hull calculation occurs only in two dimensions, thus ensuring that the number of points in the convex hull is manageable (eg, approximately 4-6 points).

The second regression analysis may include one or more lasso regressions and/or one or more least angle regressions (LARS) (act 1022). LARS can help to determine which of the total set of extracted features is most useful using a variant of forward feature selection. For example, the best predictor(s) from a set of features can be selected by developing a single-feature linear regression from each feature. The most correlated output can be selected as the "first" feature. These predictions can be subtracted from the output to obtain the residuals. The algorithm may differ from other forward feature selection algorithms in that it attempts to find another feature that has a correlation to the residual approximately equal to the first feature on the output. Then, the algorithm can find the "equiangular" direction between the two estimates, and along the equiangular direction it can find a third feature that maximizes the correlation for the new residual. Thereafter, new angles can be discovered from the previous features and new features added to the set. Features can be added in this way until the desired accuracy is satisfied.

The third regression analysis may include one or more elastic net regressions, also known as elastic net algorithms (act 1024). Elastic net regression is a combination of Lasso regression and ridge regression. However, instead of using only forward feature selection, the algorithm can also use the L1 and L2 norms in its objective function. This allows the algorithm to relate to LARS and Lasso, but to a specific "backoff" adjustment so that the algorithm can be more stable. The parameters can be cross-validated using a stepwise exhaustive process.

The fourth regression analysis may include one or more support vector (SV) regression analysis (act 1026). Two SV regressions can be used to capture possible non-linear relationships between image data and bilirubin levels. SV regression analysis can be used to find a linear regression function in a high-dimensional feature space. The input data can then be mapped into space using a potential nonlinear function. The first SV regression analysis may use a linear kernel, and the second SV regression analysis may use a nonlinear radial basis function to assign high weights to high-grade bilirubin values.

The fifth regression analysis may include one or more random forest regressions (act 1028). For example, the fifth regression analysis can use a random forest regression analysis with 500 trees. Random forest is a collection of estimates. It can use many "classifying" decision trees for various sub-samples of the dataset. The outputs of these trees can be averaged to improve prediction accuracy and control over-fitting. Each tree can be generated (replaced) using random sub-samples.

In many embodiments, other types of regressions may also be used in addition to or instead of one or more of the regressions described herein. For example, more than one linear regression analysis may also be used. Method 1000 may be practiced using any suitable type of regression analysis, including linear and non-linear regression analysis.

After the initial regression analysis, one or more multi-layer classifiers may be used (act 1030). For example, as previously described herein, the initially computed feature, along with the output of the initial set of regressors, classifies the first estimated range of bilirubin levels into one of a plurality of different bilirubin ranges. Can be used to The classifier may include a random forest classifier, a support vector machine, and a k-Nearest Neighbor (k=3). The results of all classifiers, as well as the log-likelihood for each class, can be used as a feature for the final stacked regression, which can be the final random forest regression (behavior( 1032)). The results of the initial regression analysis and the original extracted features can also be used as features for this final stage regression analysis. The final regressor can be trained using any suitable machine learning algorithm such as AdaBoost. The final regressor can be used as the final estimate 1010 of the bilirubin level (eg, measured in milligrams per deciliter). To prevent overfitting, LOOCV (Leave-One-Out Cross-Validation) can be used at all levels of learning.

11A illustrates a method 1100 for estimating a patient's bilirubin level using basic skin color data, in accordance with multiple embodiments. Method 1100 is similar to method 600, except that the patient's current skin color data is compared to baseline skin color data to generate a bilirubin estimate. This approach can be used to compensate for different skin colors due to factors that can make the image analysis wrong, for example the patient's race or ethnicity.

In operation 1110, basic skin color data for the patient is received, and the basic skin color data corresponds when the patient has a reference bilirubin level. The reference bilirubin level may be a known bilirubin level (eg, based on TSB or TcB testing). If the patient is an infant, basic skin color data can be collected within 24 hours of the infant's birth when the bilirubin level is typically very low, for example when the bilirubin level is approximately zero. A suitable method for collecting and generating basic skin color data is described below.

In operation 1120, image data for at least one image including a color correction target and a region of the patient's skin is received. In operation 1130, color-balanced image data for the skin region is generated based on a subset of image data corresponding to the skin region and a color correction target. The image data collection and color-balancing process may be similar to the process previously described herein, respectively, with respect to operations 610 and 620 of method 600. Operations 1120 and 1130 may be performed at any time after the base skin color data is received, and any suitable time period ( For example, it may be repeated during the first 4 to 5 days of the infant period).

11B illustrates a method 1150 for generating basic skin color data in accordance with a number of embodiments. In operation 1160, basic image data for the patient is captured when the patient has a reference bilirubin level, and the basic image data corresponds to at least one image including a basic color correction target and a skin area. The collection of basic image data may be similar to the collection procedure previously described herein with respect to operation 610 of method 600. The basic color correction target may be the same as the color correction target used to collect normal image data or may be a different color correction target. Similarly, the basic image data may include the same skin region or different skin regions as the normal image data. As explained above, the reference bilirubin level may be any known bilirubin level, for example a bilirubin level determined by testing or taken within 24 hours postnatal.

In operation 1170, color-balanced basic image data for the skin area is generated based on the basic image data. Generation of color-balanced basic image data can be performed using any of the techniques previously described herein with respect to normal image data. In operation 1180, the basic skin color data is generated based on the color-balanced basic image data for the skin area. As discussed above, this process involves extracting features from the color-balanced base image data and using the features for machine learning regression analysis.

12 illustrates a mobile device 1200 for estimating a patient's bilirubin level in accordance with a number of embodiments. Mobile device 1200 includes a camera 1202 suitable for capturing image data and a flash unit 1204 suitable for providing flash illumination. The camera 1202 and the flash unit 1204 may be built-in hardware of the mobile device 1200. Alternatively, the camera 1202 and the flash unit 1204 may be provided separately from the mobile device 1200 and coupled to the mobile device 1200 (eg, via wired or wireless communication). have. In some instances, the mobile app described herein can detect whether the camera 1202 is capable of capturing an image with a sufficiently high resolution for subsequent image analysis, and notify the user if these criteria are not met. have.

Mobile device 1200 also includes an input unit 1206 for receiving input from a user and a display 1208 for displaying content to the user. The input unit 1206 may include a keyboard, a mouse, a touch screen, a joystick, and the like. The input unit 1206 may also be configured to accept a voice command or a gesture command. The display 1208 may include a monitor, a screen, a touch screen, and the like. In some instances, input unit 1206 and display 1208 may be implemented across shared hardware (eg, the same touchscreen is used to accept input and display output). As previously described, the display 1208 may display to the user one or more suitable UIs, such as the UI of the mobile app for estimating the bilirubin level.

The mobile device 1200 includes one or more processors 1210, a memory or other data storage device 1212 or communication unit 1216 that stores image data as well as one or more software modules 1214. The processor 1210 may be operatively coupled to the camera 1202 and/or the flash unit 1204 to control one or more functions (eg, a recording function, a zoom function, a flash lighting function). Processor 1210 may also be operatively coupled to memory 1212 such that processor 1210 can receive and execute instructions provided by software module 1214. The software module 1214 may be implemented as part of the mobile app described herein and may provide instructions for performing one or more operations of the previously discussed methods. For example, the software module 1214 may cause the mobile device 1200 to capture image data of the patient and correction target. In some instances, software module 1214 may perform some or all of the image processing and analysis tasks disclosed above (eg, color balancing, feature extraction, machine learning regression). The software module 1214 may be adapted to a number of different types of mobile devices. In addition, the mobile device 1200 receives and installs software updates, such as updates that improve one or more image capture, processing, and analysis algorithms, so that the mobile app can be configured to be easily and quickly upgraded as needed.

The communication unit 1216 of the mobile device 1200 provides data (e.g., image data, bilirubin estimates, software updates, etc.) between the mobile device 1200 and an individual device or system, e.g., a remote server or other computing system. May be configured to receive and/or transmit. The communication unit may use any suitable combination of wired or wireless communication methods including Wi-Fi communication. In some instances, communication between mobile device 1200 and individual devices may be performed using text messaging service (SMS) text messaging. The communication unit 1216 is also operatively coupled to the processor 1210 such that data communication to and from the mobile device 1200 can be controlled based on instructions provided by the software module 1214. Can be.

13 illustrates a mobile device 1300 in communication with a data processing system 1302 for estimating a bilirubin level in accordance with a number of embodiments. Mobile device 1300 may include any of the components previously described herein with respect to mobile device 1200 of FIG. 12. Components of data processing system 1302 may be implemented across any suitable combination of physical and/or virtual computing resources (eg, virtual machines) including distributed computing resources (in the “cloud”). have. In many embodiments, data processing system 1302 is a remote server configured to communicate with a plurality of user mobile devices including mobile device 1300. Communication may utilize any suitable wired or wireless communication method as described above.

The data processing system 1302 includes one or more processors 1304, a memory or other data storage device 1306 that stores one or more software modules 1308, and a communication unit 1310. Communication unit 1310 is used to communicate data (e.g., image data, bilirubin estimate, software update, etc.) between system 1302 and mobile device 1300 (e.g., via SMS text messaging). I can. For example, the communication unit 1310 may receive image data provided by the mobile device 1300, for example, image data that has not yet been color-balanced. Data obtained from the mobile device 1300 may be stored in the memory 1306. The software module 1308 includes instructions executable by the processor 1304 to process and analyze image data (e.g., color balancing, feature extraction, etc.), such as by performing one or more operations of the methods described herein. Machine learning regression analysis). Processor 1304 may output an estimate of the patient's bilirubin level, which may be stored in memory 1306 and/or transmitted to mobile device 1300. In some instances, depending on user preferences, results may also be transmitted to a third party, for example a medical professional who may review the results and provide additional instructions to the user as needed.

In an alternative embodiment, the mobile device described herein may be configured to illuminate the skin of a patient with light of different wavelengths (eg, 460 nm, 540 nm) and use a camera to capture an image of the illuminated skin. The timing and sequence of the lighting can be controlled by the mobile app. The mobile app can analyze the collected image data to measure the intensity of light of different wavelengths reflected from the skin area. In many embodiments, the absorption of different wavelengths is different depending on the color of the skin, including yellow discoloration. Some wavelengths can be influenced by the level of bilirubin, so the intensity of the wavelength indicates the amount of bilirubin in the patient's body. Thus, in order to allow wavelength absorption to be used as an input for estimating the patient's bilirubin level, an appropriate machine learning regression analysis and/or model can be developed. Advantageously, this approach can be more robust for different environmental and situational conditions.

In many embodiments, the mobile device is a front-facing camera (e.g., on the same side of the mobile device as the screen) that can be used to capture image data (e.g., photos, videos) used to evaluate ambient lighting conditions. Placed camera). These data provide alternative and/or additional ambient lighting information that can be used to normalize the color data of the patient's skin during estimation of the patient's bilirubin level.

The various techniques described herein are storable on storage media and computer-readable media, and may be partially or fully implemented using code executable by one or more processors of a computer system. Storage media and computer-readable media for containing code or portions of code include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD), or other optical storage, magnetic cassette, magnetic Including tape, magnetic disk storage or other magnetic storage device, solid state drive (SSD), or other solid state storage device, or any other medium that can be used to store desired information and that can be accessed by a system device, Volatile and non-volatile, removable and non-removable media implemented in any method or technology to store and/or transmit information such as computer readable instructions, data instructions, program modules, or other data ( However, it may include any suitable media known or used in the art, including, but not limited to) storage media and communication media. Based on the disclosure and teachings provided herein, those skilled in the art will recognize other ways and/or methods for implementing various embodiments.

While preferred embodiments of the present invention have been presented and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, modifications and substitutions will now be made by those skilled in the art without departing from the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be used in practicing the invention. It is intended that the following claims define the scope of the invention, whereby methods and structures within the scope of these claims and their equivalents should be covered by the claims.

Claims (27)

As a method of estimating the level of bilirubin in a patient,
Receiving image data for at least one image including a color correction target and a region of the patient's skin;
Generating color-balanced image data for the skin area based on the subset of the image data corresponding to the skin area and the color correction target-a subset of the image data Contains at least one area of the image; And
Estimating the bilirubin level of the patient based on the color-balanced image data for the skin area,
Estimating the bilirubin level,
Processing a plurality of normalized achromatic and chromatic features to select a first estimated range of the bilirubin level from one of a plurality of different bilirubin ranges; And
Processing the plurality of normalized achromatic and chromatic features using an approach based on a selected first estimated range of bilirubin levels to generate a final estimate of the bilirubin level. How to estimate. The method according to claim 1,
Receiving base skin color data corresponding to the patient when the patient has a reference bilirubin level;
The estimating the bilirubin level is based on a comparison between the color-balanced image data for the skin area and the basic skin color data. The method according to claim 2,
The basic skin color data for the patient is,
Capturing basic image data for the patient when the patient has the reference bilirubin level, the basic image data corresponding to at least one image comprising a basic color correction target and the skin area;
Based on a subset of the basic image data corresponding to the skin area and the basic color correction target, color-balanced basic image data for the skin area is generated, and the subset of the basic image data is at least one Includes an area of the image -;
By generating the basic skin color data based on the color-balanced basic image data for the skin area,
Generated, a method of estimating the level of bilirubin in a patient. The method according to claim 1,
The color correction target includes a plurality of standardized color areas including white areas. The method of claim 4,
The standardized color gamut includes a black area, a gray area, a light brown area, a cyan area, a magenta area, a yellow area, and a dark brown area. How to. The method of claim 4,
The color correction target at least partially defines an opening configured to expose the skin region to enable capturing of image data for the skin region. The method of claim 6,
The normalized color gamut is arranged in a known arrangement surrounding the opening. The method of claim 6,
The step of generating color-balanced image data for the skin area,
Processing the received image data to identify a subset of image data corresponding to the exposed skin area and a subset of the image data corresponding to the white area-the subset of image data comprises the at least one Contains an area of the image of -;
Processing the image data corresponding to the white area to determine an observed color value for the white area; And
And generating color-balanced image data for the exposed skin area based on the observed color value for the white area. The method of claim 8,
The observed color values for the white region include red, green and blue (RGB) color space values. The method according to claim 1,
The color-balanced image data for the skin area includes RGB color space data;
The method further comprises converting the RGB color space data to the at least one other color space to generate color-balanced image data of the exposed skin area for at least one other color space. How to estimate the level of bilirubin in. The method of claim 10,
The at least one other color space may include: (a) a cyan, magenta, yellow, and black (CMYK) color space; (b) YCbCr color space; Or (c) a method for estimating the level of bilirubin in a patient, including the Lab color space. The method of claim 10,
The at least one other color space may include: (a) a cyan, magenta, yellow (CMY) color space; (b) YCbCr color space; Or (c) a method for estimating the level of bilirubin in a patient, including the Lab color space. The method of claim 11,
A method of estimating the level of bilirubin in a patient, wherein the received image data includes an image obtained using flash illumination and an image obtained without using flash illumination. delete The method according to claim 1,
The method of estimating the level of bilirubin in a patient, wherein the plurality of different bilirubin ranges include a low range, a mid range and a high range. The method according to claim 1,
The method of estimating the level of bilirubin in a patient, wherein the plurality of normalized achromatic and chromatic features include selected color values of skin regions for a plurality of different color spaces. The method of claim 1, wherein the plurality of normalized achromatic and chromatic features comprises calculating a color gradient across the skin area. The method according to claim 1,
The step of processing the plurality of normalized achromatic and chromatic features to select the first estimated range of the bilirubin level includes (a) linear regression, (b) encapsulated k-proximity neighbor regression analysis. (k-Nearest Neighbor regression), (c) lasso regression, (d) LARS regression, (e) elastic net regression, (f) using a linear kernel Support vector regression, (g) support vector regression that assigns high weights to higher-rated bilirubin values, and (h) random forest regression. A method of estimating the level of bilirubin in a patient, comprising the step of performing a series of regression analyzes. The method according to claim 1,
The step of processing the plurality of normalized achromatic and chromatic features to select the first estimated range of the bilirubin level includes (a) linear regression analysis, (b) encapsulated k-nearest neighbor regression analysis, and (c) Lasso regression analysis, (d) LARS regression analysis, (e) elastic net regression analysis, (f) support vector regression analysis using a linear kernel, (g) support vector regression analysis that assigns high weights to high-grade bilirubin values. And (h) performing a series of regression analyzes including random forest regression analysis. The method according to claim 1,
The step of using a processing approach based on the selected first estimated range of the bilirubin level comprises a selected first estimated range of the bilirubin level as a feature for a final random forest regression and the plurality of normalized achromatic and chromatic features. A method for estimating the level of bilirubin in a patient comprising the step of performing the final random forest regression analysis using. The method according to claim 1,
The step of estimating the bilirubin level comprises determining a color space value for the patient's skin and using a processing approach based on the determined patient's skin color space value to estimate the bilirubin level. How to estimate the level of bilirubin in. A mobile device configured to estimate the level of bilirubin in a patient, comprising:
A camera operable to capture image data for a field of view;
A processor operatively coupled to the camera; And
A data storage device operatively coupled with the processor and storing instructions;
The instruction, when executed by the processor, causes the processor to:
Receive image data for an image including a color correction target captured by the camera and an area of the patient's skin;
Based on the skin region and a subset of image data corresponding to the color correction target, to generate color-balanced image data for the skin region, wherein the subset of the image data includes at least one region of the image. Contains-; And,
Processing a plurality of normalized achromatic and chromatic features to select a first estimated range of the bilirubin level from one of a plurality of different bilirubin ranges;
By processing the plurality of normalized achromatic and chromatic features using an approach based on a selected first estimated range of the bilirubin level to produce a final estimate of the bilirubin level.
And estimating the patient's bilirubin level based on the color-balanced image data for the skin area. The method of claim 22,
The color correction target at least partially defines an opening configured to expose the skin area to enable capturing of image data for the skin area, and includes a plurality of standardized color areas including a white area And;
The instruction causes the processor to:
Process the received image data to identify a subset of image data corresponding to the exposed skin area and a subset of the image data corresponding to the white area, wherein the subset of image data comprises the at least one Contains an area of the image of -;
Process the image data corresponding to the white area to determine an observed color value for the white area;
Based on the observed color value for the white area, generate color-balanced RGB image data including RGB color space data for the exposed skin area; And
Converting the color-balanced RGB image data into at least one other color space to generate color-balanced image data of the exposed skin area for the at least one other color space. The method of claim 22,
The color correction target at least partially defines an opening configured to expose the skin area to enable capturing of image data for the skin area, and includes a plurality of standardized color areas including a white area And;
The instruction causes the processor to:
Process the received image data to identify a subset of image data corresponding to the exposed skin area and a subset of the image data corresponding to the white area, wherein the subset of image data comprises the at least one Contains an area of the image of -;
Process the image data corresponding to the white area to determine an observed color value for the white area;
Based on the observed color value for the white area, generate color-balanced RGB image data including RGB color space data for the exposed skin area;
Converting the color-balanced RGB image data into at least one other color space, thereby generating color-balanced image data of the exposed skin area for the at least one other color space;
Process color-balanced image data for the exposed skin area to determine a skin color for the patient;
Processing the plurality of normalized achromatic and chromatic features using an approach based on the determined skin color to estimate the bilirubin level. The method of claim 22 or 24,
Further comprising a flash unit operable to selectively illuminate the field of view;
The mobile device, wherein the received image data processed to estimate the bilirubin level comprises an image captured with a field of view illuminated by the flash unit and an image captured with a field of view not illuminated by the flash unit. As a method for estimating the level of bilirubin in a patient,
Receiving image data for an image including a color correction target and a skin area of the patient from the mobile device;
Generating, through one or more processors, color-balanced image data for the skin region based on a subset of image data corresponding to the skin region and the color correction target-the subset of image data is at least Contains an area of one image;
Estimating a bilirubin level of the patient based on color-balanced image data for the skin region, through the one or more processors; And
Transmitting the estimated bilirubin level to the mobile device,
Estimating the bilirubin level,
Processing a plurality of normalized achromatic and chromatic features to select a first estimated range of the bilirubin level from one of a plurality of different bilirubin ranges; And
Processing the plurality of normalized achromatic and chromatic features using an approach based on a selected first estimated range of bilirubin levels to generate a final estimate of the bilirubin level. Method to estimate. The method of claim 26,
At least one of receiving the image data and transmitting the estimated bilirubin level is performed using text messaging service (SMS) text messaging.

Images