KR102242479B1 - Digital Breathing Stethoscope Method Using Skin Image - Google Patents

Abstract

Digital respiration auscultation method using a skin image includes: detecting a skin area from a photographed image, extracting a bio-signal from the detected skin area, extracting a respiration signal from the extracted bio-signal, and extracting the breathing Amplifying a signal, interpolating the amplified breathing signal by applying an audible sampling frequency, detecting the upper and lower peaks of the interpolated breathing signal, and calculating the upper and lower peaks of the breathing signal. Distinguishing the inhalation and exhalation of the breathing signal by using, outputting the breathing signal as a vibration by separating the inhalation and exhalation, and interpolating a preset sound source signal corresponding to the inhalation length and exhalation length of the breathing signal. , Adjusting the amplitude of the interpolated sound source signal according to the breathing intensity, outputting the sound by combining the adjusted sound source signal and the breathing signal, providing the breathing state information using the breathing signal, and It includes the step of measuring the breathing intensity through the breathing signal.

Description

Digital Breathing Stethoscope Method Using Skin Image}

The technology to be described below relates to a digital breathing auscultation and diagnosis method based on the five senses using a skin image.

The conventional respiration meter used a separate contact type meter or sensor such as a temperature and humidity sensor or a carbon dioxide sensor. When a separate measuring device was not used, respiration and lung capacity were measured using a microphone or camera of a smart device. When using a microphone, the user measured the maximum inhaled and exhaled sound through the microphone of the smart device. When using a camera, the movement of the chest was measured by removing the top, or when using a face image, only the respiratory rate was presented.

In the conventional contact-type respiration meter, a user individually purchased a separate meter. The breathing meter must be purchased directly, and even if the measurement is performed while the user is wearing clothes, there is a limitation that the contact type measurement method is used.

In the case of a non-contact breathing meter, use the smart device's microphone or camera. In the case of measuring lung function through a microphone of a smart device, only forced lung capacity can be measured among the items required for lung function test, and there is a difference from contact type equipment because there is no classification of lung capacity measurement values according to sex. In the case of using the camera function of a smart device, the user had to take off the top for chest recognition. When the top is not removed, the application of the smart device measures through the movement of the user's chest and abdomen. There was an error due to. In addition, when using a skin image, only the respiratory rate is presented as a numerical value, and there is a problem that there are some errors.

Korean Patent Publication No. 10-2014-0059404

In order to overcome the limitations of existing measuring devices, the technology described below does not use additional measuring devices, but simply breathes by performing a contact method using the acquired skin image without the inconvenience of undressing the top with a camera of a smart device owned by an individual. It not only provides the number and lung capacity, but also provides diagnostic information such as respiratory status, respiratory disease, lung function, and lung disease, including respiratory rate, and through the five senses such as sight, hearing, and touch using hearing sound, vibration, and graphs. It is intended to provide a digital breathing auscultation method using skin images capable of intuitive condition diagnosis.

According to an aspect of the present invention, detecting a skin area from a photographed image, extracting a biosignal from the detected skin area, extracting a breathing signal from the extracted biosignal, amplifying the extracted breathing signal Step, Interpolating the amplified breathing signal by applying an audible sampling frequency, outputting the breathing signal as a vibration by separating the inhalation and exhalation, and outputting a preset sound source signal to the inhalation length and the exhalation length of the breathing signal. Interpolating, adjusting the amplitude of the interpolated sound source signal according to the breathing intensity, combining the adjusted sound source signal and the breathing signal and outputting it as a sound, providing breathing status information using the breathing signal, and the breathing signal It provides a digital breathing auscultation method using a skin image including the step of measuring the breathing intensity through.

In addition, according to another aspect of the present invention, the amplified breathing signal may be expanded and interpolated using an audible sampling frequency.

In addition, according to another aspect of the present invention, after the step of interpolating the amplified respiration signal by applying an audible sampling frequency, the upper peak and the lower peak of the interpolated respiration signal are detected, and the upper peak of the respiration signal It may include the step of determining the inhalation and exhalation length by separating the inhalation and exhalation of the breathing signal by using and the lower peak.

In addition, according to another aspect of the present invention, it may include the step of interpolating or reducing a preset sound source signal corresponding to the inhalation length and the exhalation length of the extracted respiration signal.

In addition, according to another aspect of the present invention, the step of adjusting the amplitude of the interpolated and reduced sound source signal according to the breathing intensity is adjusted to increase the amplitude as the intensity of the breathing increases, and the amplitude is increased as the breathing intensity is weakened. Can be adjusted to be small.

In addition, according to another aspect of the present invention, it is possible to sense or measure the breathing state with sound by combining the adjusted sound source signal with the breathing signal.

In addition, according to another aspect of the present invention, it is possible to output the breathing signal as a graph.

In addition, according to another aspect of the present invention, the step of outputting the breathing signal as vibration is adjusted so that the interval of the vibration increases as the intensity increases in inhalation based on the intensity of breathing, and the vibration increases as the intensity decreases in exhalation. It can be output by adjusting so that the interval of is shortened.

In addition, according to another aspect of the present invention, the respiration intensity or respiration rate may be extracted or calculated using the upper peak and the lower peak of the respiration signal.

In addition, according to another aspect of the present invention, the step of diagnosing the respiration intensity may diagnose or measure the respiration intensity by using the amplitude of the respiration signal.

In addition, according to another aspect of the present invention, further comprising the step of diagnosing a respiratory disease, and the step of diagnosing a respiratory disease uses the intensity of breathing, the regularity of breathing, and the number of breaths measured through the breathing signal. It can be diagnosed.

According to another aspect of the present invention, the steps of detecting a skin area from an image that consecutively exhales maximum after the maximum inhalation taken, extracting a biosignal from the detected skin area, extracting a breathing signal from the extracted biosignal, Detecting the upper and lower peaks of the extracted respiration signal, measuring the peak point of the respiration curve using the detected upper and lower peaks, the first forced expiratory volume (FEV1) using the detected respiration curve, Measuring the second forced expiratory volume (FEV2), the third forced expiratory volume (FEV3), the measured first forced expiratory volume (FEV1), the measured second forced expiratory volume (FEV2), and the measured third forced expiratory volume (FEV3). It provides a digital breathing auscultation method using a skin image including the step of measuring the forced pulmonary capacity (FVC) by utilizing and providing lung function information using the measured forced pulmonary capacity (FVC).

In addition, according to another aspect of the present invention, the step of measuring the peak point of the respiration curve using the detected upper peak and the lower peak can measure the respiratory rate and the respiration intensity through the measured peak point of the curve. have.

In addition, according to another aspect of the present invention, the respiration rate and respiration intensity are measured by measuring the upper and lower peaks of the detected respiration signal to detect the inhalation and exhalation portions, and the height of the upper peak and the lower peak. You can measure the number of breaths and the intensity of breathing through the height and number of peaks.

Further, according to another aspect of the present invention, the first forced expiratory volume (FEV1), the second expiratory volume (FEV2), and the third forced expiratory volume (FEV3) are t1 after 1 second from the lower peak value of the detected respiration curve. H1 at the time point, h2 at the time point t2 after 2 seconds, and h3 at the time point t3 after 3 seconds can be measured by applying to the forced expiratory volume regression analysis equation.

In addition, according to another aspect of the present invention, the forced lung capacity (FVC) may use Equation 1.

In addition, according to another aspect of the present invention, the step of providing pulmonary function information using the measured forced lung capacity (FVC) is a percent lung capacity ( %FVC) and 1 second rate (FEV1/FVC) can be calculated to provide information on lung function.

In addition, according to another aspect of the present invention, the step of providing lung function information using the measured forced lung capacity (FVC) is normal ventilation if the lung capacity (%FVC) is 80% or more and 1 second rate (FEV1) is 70% or more. If the lung capacity (%FVC) 80% or more and 1 second rate (FEV1) is 70% or less, it is diagnosed as obstructive ventilation disorder, and if the lung capacity (%FVC) 80% or less and 1 second rate (FEV1) is 70% or more, it is diagnosed as restrictive ventilation disorder. If the lung capacity (%FVC) is 80% or less and 1 sec rate (FEV1) is 70% or less, it can be diagnosed as mixed ventilation disorder.

According to another aspect of the present invention, detecting a skin area from a photographed image, extracting a biosignal from the detected skin area, extracting a breathing signal from the extracted biosignal, and converting the extracted breathing signal to a sound source. Digital respiration auscultation method using skin images comprising interpolating at the same sample rate as and calculating an emphasis filter coefficient on the interpolated respiration signal, and filtering a sound source signal using the calculated emphasis filter coefficient Provides.

In addition, according to another aspect of the present invention, the step of interpolating the extracted respiration signal to the same sample rate as the sound source may expand the number of samples in response to the detected respiration signal to a preset sound source. .

In addition, according to another aspect of the present invention, the step of calculating the coefficient of the enhancement filter may be calculated using Equation 4.

In addition, according to another aspect of the present invention, in the step of filtering the sound source signal using the calculated filter coefficient, the calculated emphasis filter coefficient according to the time may be calculated using Equation (5).

The technology described below is a non-contact method, without purchasing separate measurement equipment, and conveniently and accurately measuring breathing and lung function using the image of a smart device. You can intuitively auscultate through the five senses such as touch.

1 is an example showing the configuration of various electronic devices capable of digital breathing auscultation using a skin image.
2 to 6 are flow charts showing an embodiment of a method for digital breathing auscultation using a skin image.
7 and 8 are flow charts showing another embodiment of a method for performing digital breathing auscultation using skin images.
9 to 10 are flow charts showing another embodiment of a method for digital respiration auscultation using a skin image.
Referring to FIG. 11, it is an exemplary diagram for generating a sound source reflecting breathing using frequency enhancement according to breathing using a skin image according to an embodiment of the present invention.

The technology described below may be changed in various ways and may have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the technology to be described below with respect to a specific embodiment, and it should be understood to include all changes, equivalents, or substitutes included in the spirit and scope of the technology to be described below.

Terms such as 1st, 2nd, A, B, etc. may be used to describe various components, but the components are not limited by the above terms, and only for the purpose of distinguishing one component from other components. Is only used. For example, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component without departing from the scope of the rights of the technology described below. The term and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.

In terms of the terms used in the present specification, expressions in the singular should be understood as including plural expressions unless clearly interpreted differently in context, and terms such as "includes" are specified features, numbers, steps, actions, and components. It is to be understood that the presence or addition of one or more other features or numbers, step-acting components, parts or combinations thereof is not meant to imply the presence of, parts, or combinations thereof.

Prior to the detailed description of the drawings, it is intended to clarify that the division of the constituent parts in the present specification is merely divided by the main function that each constituent part is responsible for. That is, two or more constituent parts to be described below may be combined into one constituent part, or one constituent part may be divided into two or more for each more subdivided function. In addition, each of the constituent units to be described below may additionally perform some or all of the functions of other constituent units in addition to its own main function, and some of the main functions of each constituent unit are different. It goes without saying that it can also be performed exclusively by.

In addition, in performing the method or operation method, each of the processes constituting the method may occur differently from the specified order unless a specific order is clearly stated in the context. That is, each of the processes may occur in the same order as the specified order, may be performed substantially simultaneously, or may be performed in the reverse order.

Looking at (a) of FIG. 1, the present invention extracts, amplifies, and interpolates a breathing signal in a non-contact manner using a smart device 50, and based on this, the presence or absence of an abnormal condition is diagnosed by vibration, listening, and graphs. I can confirm. The user may photograph a face with a camera built into the smart device 50. For example, the smart device may include a mobile phone, a smart phone, a wearable device, and the like in which at least one camera is embedded. The smart device 50 extracts or calculates the breathing signal for the skin area included in the photographed source image, performs an appropriate amplification of the audible range, interpolation with a variety of sampling frequencies, and outputs as graphs, sounds, and vibrations. By checking additional information such as the calculated breathing intensity, respiratory rate, breathing status, and lung function, you can check the presence or absence of abnormal conditions along with diagnosis of the condition by vibration, hearing, and graphs. A detailed process for this will be described later.

Referring to FIG. 1A, the smart device 50 may include a camera 51, a storage device 52, a control device 53, and an output device 54.

The camera 51 may photograph a user and obtain a source image using the photographed image. The camera may include a camera built into a smart device, a general camera, or an infrared camera.

The control device 53 receives the source image from the camera 51 and extracts or calculates the breathing signal for the skin region included in the source image, and performs interpolation with appropriate amplification of a range that can be heard and a variety of sampling frequencies. Performing, auscultation in a way that outputs graphs, sounds, and vibrations, and by checking additional information such as calculated breathing intensity, respiratory rate, breathing status, and lung function, abnormal conditions along with diagnosis of conditions by vibration, hearing, and graphs You can check the presence or absence. The control device 53 can be referred to as an arithmetic device or a controller.

The storage device 52 may be electrically connected to the camera 51 or the control device 53. The storage device 52 may temporarily store a source image supplied from the camera 51.

The output device 54 may output a diagnosis of a user's condition along with measurement of pulse rate and respiration rate, respiration intensity and respiration condition, and lung capacity. The output device 54 may check additional information such as calculated breathing intensity, respiratory rate, breathing state, lung function, etc., and output it to check the presence or absence of abnormal conditions along with diagnosis of the condition by vibration, hearing, and graphs. . For example, the output device 54 may include a display unit. The display unit may display the presence or absence of an abnormal condition as well as a condition diagnosis by vibration, audible sound, and a graph.

Looking at (b) of FIG. 1, the present invention extracts, amplifies, and interpolates a breathing signal in a non-contact manner using a device such as a computer 85, and based on this, vibration, listening, and abnormal conditions along with a status check by a graph You can judge the presence or absence. The user can photograph a face with the camera 81 connected to the computer 85. The computer 85 receives the source image from the camera 81, extracts or calculates the breathing signal for the skin area included in the source image, and performs interpolation with an appropriate amplification of the audible range and a variety of sampling frequencies. And, auscultation is performed in a way that outputs graphs, sounds, and vibrations, and by checking additional information such as calculated breathing intensity, respiratory rate, breathing status, and lung function, it is possible to diagnose abnormalities with intuitive condition diagnosis by vibration, hearing and graph. You can determine the presence or absence of a state.

Referring to (c) of FIG. 1, according to the present invention, by providing an image acquired by the user terminal 91 to a server 95 in a remote location, health care can be provided remotely. The user can take a picture of the user's face with a camera built into the user terminal 91. The user terminal 91 may transmit the captured source image to the server 95 through a network. In this case, the user terminal 91 may include a communication module for data transmission. The server 95 receives the source image from the user terminal 91, extracts or calculates the breathing signal for the skin area included in the source image, and performs interpolation with an appropriate amplification of a range that can be heard and a variety of sampling frequencies. And, auscultation is performed in a way that outputs graphs, sounds, and vibrations, and by checking additional information such as calculated breathing intensity, respiratory rate, breathing status, and lung function, it is possible to diagnose abnormalities with intuitive condition diagnosis by vibration, hearing and graph. You can check the status. The server 95 may provide or transmit the presence or absence of an abnormal condition to the user terminal 91 along with a condition diagnosis by vibration, listening sound, and graph.

In some cases, the user terminal 91 receives the source image, extracts or calculates a breathing signal for the skin region included in the source image, and performs interpolation with appropriate amplification of the range that can be heard and sampling frequencies of various ranges. , Graph, sound, vibration, and auscultation, calculated breathing intensity, respiratory rate, breathing status, lung function By checking additional information such as vibration, hearing sound, and intuitive condition diagnosis by graph, it is possible to check the presence or absence of an abnormal condition, and transmit this to the server 95. In this case, the server 95 may store a condition diagnosis by vibration, listening sound, and a graph and the presence or absence of an abnormal condition.

As described above, various electronic devices may extract, amplify, and interpolate a breathing signal in a non-contact manner, and based on this, it is possible to check the presence or absence of an abnormal condition along with vibration, hearing sound, and condition diagnosis by a graph. For convenience of explanation, it will be described that the computer device extracts or calculates a breathing signal.

2 to 6, an example of a digital breathing auscultation method using a skin image according to an embodiment of the present invention may be shown.

First, the computer device may capture an image and detect skin (110). The computer device may take an image using a camera (process (a) of FIG. 3). The image may be referred to as a source image. The computer device may detect the skin area from the source image captured by the camera (process (b) of FIG. 3). For example, the skin area may be not only a face image but also various body parts. Here, the description will focus on the face image.

The computer device may detect a skin color from a face image, and may detect or set a skin region of interest from the detected skin image. At this time, the size of the skin region of interest can be flexibly adjusted. The computer device may detect or set the skin region of interest using various algorithms. At this time, the algorithm for detecting the skin region of interest may use various techniques known in the art.

The computer device may extract color average data for the region of interest in the skin (120). The color average data may be referred to as average color data or an average color data value. The computer device may extract or calculate color average data for the entire extracted skin area (process (c) of FIG. 3). Also, the computer device may extract or calculate color average data for a specific skin area from the acquired image. The computer device may continuously extract or calculate color data from consecutive images (continuous frames).

Various values may be used for color data (process (d) of FIG. 3). For example, (1) color data may use at least one of an R value, a G value, and a B value based on an RGB color system. Color data may use color average data for at least one of an R value, a G value, and a B value. (2) The computer device can convert the RGB color system to another color system. For example, the computer device may convert RGB color systems into various color systems such as YUV, HSV, YCbCr, YCgCo, and the like. In this case, the color data may use one of the color difference components that are less affected by the surrounding environment (illumination, etc.). For example, in the case of YCbCr, at least one of a Cb value or a Cr value may be used. In the case of YCgCo, at least one of a Cg value or a Co value may be used. Furthermore, one of the two color difference components, which is more robust to changes in illuminance, can be used. For example, in the case of YCgCo, only the Cg value can be used. In this case, the computer device may extract the average value of the Cg color data of the skin region as color data. (3) Furthermore, the color data may be a value obtained by applying a weight to at least one or more color components in various color systems such as RGB, YUV, HSV, YCbCr, YCgCo, and the like. When color components are combined, the color data may be a value obtained by adding different weights according to a color system and a type of color component. The computer device may change the source image having the RGB color system to the YCgCo color system, and it is assumed that the computer device obtains and uses a Cg value from YcgCo.

The computer device may apply Fast Fourier Transform (FFT) to the average value of color data. For example, the computer device may calculate the cutoff frequency through frequency analysis using a spectrogram and an FFT on the average of the Cg values in each frame (process (e) of FIG. 3). The computer device may calculate a respiration signal after applying a filtering, for example, BPF (Bnad Pass Filter) or MAF (Moving Average Filter) (process (f) of FIG. 3) to color data using the calculated cutoff frequency ( 130) (process (g) of FIG. 3).

The computer device may interpolate and amplify the calculated respiratory signal (140). The computer device can detect and adjust the upper and lower peaks of each peak of the calculated or acquired respiration signal (process (h) of Fig. 4) using a peak search algorithm (150) (process (i) of Fig. 4). . The computer device may detect an inhalation portion and an exhalation portion by distinguishing the detected upper peak and the lower peak (160).

It is a part that rises from the inhaled lower peak to the upper peak, and may be a part that goes down from the exhaled upper peak to the lower peak.

The computer device can measure or detect the breathing length by using the detected inhalation portion and the exhalation portion to distinguish between inhalation or exhalation (process (j) of FIG. 4) (process (k) of FIG. 4). The computer device may measure or calculate the number of breaths using the peak interval of the calculated breathing signal (process (l) of FIG. 4).

And computer devices can amplify breathing signals. Since the calculated amplitude of the breathing signal has a low amplitude, problems may occur during output. To solve this problem, a computer device can amplify the breathing signal to an appropriate size for sound output.

Thereafter, the computer device may obtain the length and time of inhalation and exhalation, and then subsample or interpolate the previously prepared breathing sound source signal according to the length of inhalation and exhalation (170). For example, a computer device may measure the length of inhalation and exhalation per breath through the peak interval (171). The computer device may interpolate or reduce the preset inhalation sound source signal to correspond to the measured inhalation length (172). Alternatively, the computer device may interpolate or reduce the preset exhalation sound source signal to correspond to the measured exhalation length (173).

In the case of the interpolation method, since the types and sampling frequencies are varied, the computer device can accurately analyze the calculated breathing signal, and select and use an appropriate interpolation method based on the analyzed breathing signal so that the sound output is smooth.

The computer device can adjust the intensity of the breathing sound according to the length of inhalation and exhalation resulting from the peak detection of the breathing signal from the interpolated sound source signal in order to adjust the intensity of the breathing sound output sound according to the intensity of breathing ( 180). That is, the computer device can adjust the breathing sound source signal prepared in advance in the inhalation and exhalation parts for the auscultation sound output according to the inhalation and exhalation length of the breathing signal, and then adjust the amplitude according to the breathing intensity. For example, the computer device may increase the amplitude of the sound source signal when the size of the breath is large, and decrease the amplitude of the sound source signal when the size of the breath is small.

Finally, the computer device may add the converted sound source signal to the breathing signal after adjusting the amplitude of the sound source signal according to the inhalation intensity (181). Alternatively, the computer device may add the converted sound source signal to the breathing signal after adjusting the amplitude of the sound source signal according to the exhalation intensity (182). The computer device may output the adjusted inhalation and exhalation sound source signals as sound together with a graph after adding the previously calculated breathing signal and the graph (190). Accordingly, the computer device is capable of diagnosing the user's condition through listening sound (220).

In addition, the computer device can classify the inhalation and exhalation parts based on the output breathing signal and the detected peak. The computer device distinguishes by adjusting the intensity or speed of vibration in each inhalation part and exhalation part based on the detection peak. That is, the computer device calculates the breathing signal through filtering the extracted color data, detects the upper peak and the lower peak, and then divides the inhalation and exhalation parts, and according to the intensity of each inhalation and exhalation part, the vibration device frequency and vibration The speed or vibration intensity can be adjusted (200).

For example, the computer device loosens the interval of vibration as the intensity of the inhalation portion increases based on the strength of the signal (for example, the inhalation portion may be set to vibration 70 ms and standby 30 ms based on 100 ms. (vibration). :Atmospheric = 70:30, based on 100ms)) (201), as the intensity decreases from the exhalation part, the interval of the vibration is densely (202), and the vibration is output (e.g., the exhalation part is set to vibration 95ms, standby 5ms. Can (vibration: atmospheric = 95:5, based on 100ms)) can (210).

The user intuitively feels breathing and enables the user's condition diagnosis (respiratory disease diagnosis or respiratory intensity diagnosis) through vibration (230, 240).

As described above, the computer device performs amplification and interpolation for smooth auscultation sound output, and in the case of amplification, it can be performed in various ranges. In addition, the computer device can diagnose the user's condition through audible sound and graph using the auscultation sound generated through the breathing signal, or diagnose the user's condition by using the vibration adjusted through the breathing signal. For example, the user's condition diagnosis may include a user condition self-diagnosis 220, a respiratory disease diagnosis 230, and a respiratory intensity diagnosis 240 through sound listening and vibration.

7 to 8, another example of a digital breathing auscultation method using a skin image according to an embodiment of the present invention may be shown.

The computer device may capture a face image of a user who continuously exhales after the maximum inhalation using the camera of the smart device (310), and detects the face area and the skin area (320).

In addition, the computer device may detect the upper peak and the lower peak by applying a moving average filter (MAF), a band pass filter (BPF), or the like to the Cg color data 330 in the designated skin region of interest (340). A detailed description of this has been described above, so it will be omitted.

The computer device may measure the respiration rate and respiration intensity through the peak point of the detected respiration signal (350). The computer device calculates the difference values (h1, h2, h3) between the first, second, and third color data corresponding to the average color data value at 1-second intervals (t1, t2, t3) from the detected negative peak. Can be detected (351).

Specifically, the computer device calculates h1 at the point t1 after 1 second from the smallest negative peak value, h2 at the point t2 after 2 seconds, and h3 at the point t3 after 3 seconds, and calculates the calculated h1, By applying h2 and h3 to the forced expiratory volume first regression analysis equation (355), the first, second, and third forced expiratory volumes (FEV ₁ , FEV ₂ , and FEV ₃₎ may be estimated (352).

In addition, the computer device applies the estimated first, second, and third forced expiratory volumes to the second regression analysis equation of the forced expiratory volume (356), thereby improving the first, second, and third forced expiratory volumes (FEV ₁ , FEV _{2 ).} ,FEV ₃₎ can be measured (353). And the computer device can finally measure the FVC (354). FVC means forced lung capacity or effort lung capacity, and can represent the amount of air that can be exhaled as much as possible while inhaling air to the maximum.

The forced lung capacity (FVC) _{can be calculated through FEV 1,} FEV _{2, and} FEV ₃ , and is shown in Equation 1 below.

As described above, the computer device can diagnose respiratory diseases, lung diseases, etc. through respiratory rate and respiratory intensity by using the respiratory signal (360,370).

The computer device measures the respiration rate through the calculated respiration signal and divides it into normal breathing, tachy breathing, slow breathing, and apnea according to the frequency of breathing.If the breathing rate per minute continues 12 to 20 times, the number of breaths per minute is 20 or more. If it persists, it can be diagnosed as tachypnea, slow breathing if the respiratory rate per minute continues to be less than 12, and apnea if breathing stops.

In addition, through this, the computer device can diagnose various respiratory diseases through Biot's breathing when the change from apnea to multiple breathing is repeated, and a respiratory rate of 25 or more breaths per minute can be a sign of pneumonia and pulmonary thrombosis.

The computer device estimates the first, second, and third forced expiratory volumes (FEV ₁ , FEV ₂ , FEV ₃ ) using the peak points of the breathing signal, and through this, the percent lung capacity (%FVC) and the first second rate (FEV ₁ /FVC) are estimated. ). Lung disease pathological signs can be diagnosed through %FVC and 1 second rate, and pulmonary capacity (%FVC) 80% or more 1 second rate (FEV ₁ ) 70% or more at a time, normal ventilation, lung capacity (%FVC) 80% or more 1 second rate (FEV) _{1) Obstructive ventilation disorder at one time} less than 70%, lung capacity (%FVC) 80% or less 1 second rate (FEV ₁ ) 70% or more at one time constrained ventilation disorder, lung capacity (%FVC) 80% or less 1 second rate (FEV ₁ ) 70% It can be diagnosed as mixed ventilation disorder at the following time.

9 and 10, another example of a digital breathing auscultation method using a skin image according to an embodiment of the present invention may be shown.

The computer device may use the skin image detected by the camera to calculate the respiratory rate (510, 520, processes (a1) and (b1) of FIG. 10). Here, a description of the detection of the skin image has been described in detail above, and thus will be omitted.

The computer device may generate or extract a pulse wave signal by filtering color data for the skin region (530) (process (c1) of FIG. 10). In addition, the computer device may apply the FFT to the generated pulse wave signal 540 (process (d1) of FIG. 10).

The computer device may calculate the largest value observed in the frequency domain of the pulse wave signal extracted through the skin image or a frequency value having a large power (550) (process (e1) of FIG. 10). And the computer device may measure the respiration rate based on the calculated frequency value (560). To this end, the computer device may limit or set a frequency band for detecting a frequency component that is a reference of the respiratory rate (process (f1) of FIG. 10).

The computer device may determine a frequency range for calculating the respiration rate and measure the respiration rate using a frequency value having the greatest power within the determined frequency range. In addition, the computer device may extract a frequency value having the greatest power in a frequency range related to the pulse signal in order to determine a specific frequency range for calculating the respiration rate (process (g1) of FIG. 10).

For example, in a normal person, the pulse rate per minute can be measured from 40 to 200 depending on the state of stability or excitement. For example, you can basically limit the frequency range you want to observe from 0.7Hz to 3.2Hz. You can observe the frequency range within the set frequency range and determine two frequency ranges according to the frequency value that has the greatest power in the frequency range.

에서

범위를 호흡수 측정을 위한 제 1 주파수 범위라고 명명할 수 있다. For example, if the frequency value with the greatest power in the frequency range related to the pulse is greater than 1.4,

in

The range may be referred to as the first frequency range for respiration rate measurement.

Thereafter, when the frequency value having the greatest power in the frequency range related to the pulse is less than 1.4, the computer device may determine the second frequency range for measuring the respiration rate from 0.1 Hz to 0.35 Hz.

Finally, the respiration rate can be estimated using the frequency value having the greatest power in the frequency range set for respiration rate measurement, and the improved respiration rate can be calculated by applying the estimated respiration rate to the regression analysis function (Fig. 10). (h1) process).

The process of determining the frequency range for respiration rate measurement is as follows.

The computer device applies the FFT to the average value of color data calculated from the face image, and if the frequency value having the highest power within the frequency range (0.7Hz ~ 3.2Hz) related to the pulse is 1.4Hz or more, the computer device uses the following math: Equation 2 can be used to determine the first frequency range.

In the above equation, f _pulse is the frequency value that has the greatest power within the frequency range (0.7Hz ~ 3.2Hz) related to the pulse, and f _respiration is the frequency value that has the greatest power in the respiration rate measurement frequency range set using _{the f pulse.} Can mean

The computer device may determine a frequency value (frequency value for measuring respiration rate) having the greatest power in the frequency range determined by Equation 2.

From the result of applying the FFT to the average value of color data calculated from the face image, if the frequency value having the greatest power within the frequency range (0.7Hz ~ 3.2Hz) related to the pulse is less than 1.4, the computer device uses Equation 3 below. Can be used to determine the second frequency range.

In this case, the computer device may determine a frequency value (frequency value for measuring respiration rate) having the greatest power in the frequency range determined by Equation 3.

Regression analysis is an analysis method that measures the fit after obtaining a model between two variables for continuous variables. In the regression analysis, a function (formula) for calculating a specific value may be determined by using certain sample data in advance (process (i1) of FIG. 10). The regression analysis function may be expressed as an equation representing a regression line or a regression curve. In this case, as the regression analysis function, various conventionally known equations may be used.

Referring to FIG. 11, an example of generating a sound source reflecting breath using frequency enhancement according to breathing using a skin image according to an embodiment of the present invention may be shown.

First, the computer device can capture an image and detect skin (a2). The computer device may extract color average data for the skin region of interest (b2). A detailed description of this has already been described with reference to FIGS. 2 to 6, and thus will be omitted here.

The computer device can calculate the respiration signal after applying filtering (Bnad Pass Filter, etc.) to the average value of color data (c2). And the computer device can interpolate the calculated breathing signal (d2). For example, detection of a respiration signal using an image may have a sampling rate of 30 Hz because it is calculated through color change from an image input at 30 frames per second.

Since the computer device needs to expand the number of samples for smooth processing in consideration of the time with a general sound source, it can be expanded by interpolating the number of samples of the breathing signal to maintain the same sampling rate as the user's preferred sound source. I can.

The computer device may use the sound source stored in the user's sound source file storage unit or the sound source file storage f2. The sound source stored in the sound source file storage may be a user's preferred sound source. The computer device may select a sound source file stored in the sound source file storage f2 or calculate a sample rate for the selected sound source file (g2).

The computer device inputs a signal of the preferred sound source file to perform high/low frequency enhancement of the selected sound source signal (h2). Alternatively, the computer device may interpolate the selected sound source file (d2).

In addition, the computer device can use a frequency enhancement technique using high-pass emphasis and low-pass emphasis filtering to tune the sound source reflecting the breath.

는 아래 수학식 4를 활용하여 계산할 수 있다(e2).Frequency-enhanced filter coefficient in interpolated breathing signal

Can be calculated using Equation 4 below (e2).

In Equation 4 above, r[n] may represent the interpolated respiratory signal, and r _p and r _n may represent the maximum and minimum values of the respiratory signal, respectively. Since the general emphasis filter coefficient ranges from -0.95 to 0.95, r _p and r _n add 5% to the maximum and minimum values of the respiratory signal, respectively, so that an appropriate range of emphasis filter coefficients can be generated.

It is possible to perform frequency enhancement of the preferred sound source according to the user's breathing state by matching the emphasis filter coefficient according to the time calculated from the input breathing signal to Equation 5 below (i2).

는 강조필터 계수, x는 음원신호이며 y는 최종 산출된 변경된 음원신호를 나타낼 수 있다.In Equation 5 above

Denotes an enhancement filter coefficient, x denotes a sound source signal, and y denotes a finally calculated changed sound source signal.

When the breath input by applying Equation 5 above has a positive (+) value, high-pass emphasis can be performed, and when it has a negative (-) value, low-pass emphasis can be performed. The intensity of the emphasis can be applied differently depending on the amplitude range of the current breathing signal.

As a result, by generating and outputting a sound source reflecting the user's breathing, it is possible to self-auscultate and diagnose the user's current breathing state.

As described above, the digital breathing auscultation method using a skin image according to an embodiment of the present invention deviates from the conventional contact type auscultation method and transmits a breathing signal from a skin image input through a general camera such as a camera of a smart device It can calculate and process appropriate amplification and interpolation to produce a signal that can be auscultated.

In addition, the digital breathing auscultation method using a skin image according to an embodiment of the present invention does not require a separate device when developed in the form of an application of a smart device, and thus can be provided to general users through fast and convenient auscultation sound and vibration. In this way, a health condition can be intuitively diagnosed.

As a result, it can effectively replace the method of using a separate device required for the existing digital auscultation. In addition, by providing diagnosis information such as respiratory rate, respiratory intensity, respiratory disease, and lung disease in addition, it is possible to diagnose the condition using additional diagnostic information in addition to the condition diagnosis through listening and vibration.

Embodiments of the present invention and the accompanying drawings are merely illustrative of some of the technical ideas included in the above-described technology, and those skilled in the art within the scope of the technical ideas included in the above-described specification and drawings It will be apparent that both modified examples and specific embodiments that can be easily inferred are included in the scope of the rights of the above-described technology.

50: smart device
51: camera
52: storage device
53: control device
54: output device
81: camera
85: computer
91: user terminal
95: server

Claims (22)

Detecting, by a computing device, a skin area from an image captured using a camera;
Extracting a bio-signal from the skin area in which the computing device is detected;
Extracting, by the computing device, a respiration signal from the extracted biological signal;
Amplifying, by the computing device, the extracted breathing signal;
Interpolating, by the computing device, the amplified breathing signal by applying an audible sampling frequency;
Outputting, by the computing device, the breathing signal as vibration by separating the breathing signal from inhalation and exhalation;
Interpolating, by the computing device, a preset sound source signal corresponding to an inhalation length and an exhalation length of the breathing signal;
Adjusting, by the computing device, an amplitude of the interpolated sound source signal according to a breathing intensity;
Combining, by the computing device, the adjusted sound source signal and the breathing signal and outputting a sound;
Providing, by the computing device, breathing state information using the breathing signal; And
Digital breathing auscultation method using a skin image comprising a; step of the computing device measuring the breathing intensity through the breathing signal. The method of claim 1,
The amplified breathing signal is a digital breathing auscultation method using a skin image that is expanded and interpolated using an audible sampling frequency. The method of claim 1,
After the step of interpolating the amplified breathing signal by applying an audible sampling frequency,
And determining, by the computing device, an upper peak and a lower peak of the interpolated respiration signal, distinguishing between inhalation and exhalation of the respiration signal by using the upper peak and the lower peak of the respiration signal, and obtaining an inhalation and exhalation length. Digital breathing auscultation method using skin image. The method of claim 1,
Digital breathing auscultation method using a skin image comprising the step of interpolating or reducing, by the computing device, a preset sound source signal according to the inhalation length and the expiration length of the extracted breathing signal. The method of claim 1,
Adjusting the amplitude of the interpolated and reduced sound source signal according to the breathing intensity
A digital breathing auscultation method using a skin image in which the computing device adjusts the amplitude to increase as the intensity of breathing increases, and adjusts the amplitude to decrease as the intensity of breathing increases. The method of claim 1,
Digital breathing auscultation method using a skin image for sensing or measuring a breathing state by sound by combining the sound source signal adjusted by the computing device with the breathing signal. The method of claim 1,
Digital breathing auscultation method using a skin image in which the computing device outputs the breathing signal as a graph. The method of claim 1,
The step of outputting the breathing signal as vibration
Digital breathing using a skin image that the computing device adjusts so that the interval of vibration increases as the intensity increases in the inhalation based on the breathing intensity, and the interval of the vibration decreases as the intensity decreases in the exhalation. Auscultation method. The method of claim 1,
The respiration intensity or respiration rate is a digital respiration auscultation method using a skin image using an image extracted or calculated using an upper peak and a lower peak of the respiration signal. The method of claim 1,
The step of measuring the breathing intensity
Digital breathing auscultation method using a skin image, characterized in that the computing device uses the amplitude of the breathing signal. The method of claim 1,
The computing device further comprises providing information for predicting a respiratory disease,
Providing information for predicting the respiratory disease
Digital breathing auscultation method using a skin image, characterized in that using the intensity of the breathing, the regularity of breathing, and the number of breaths measured through the breathing signal. Detecting, by a computing device, a skin area in an image that is continuously exhaled after maximum inhalation taken using a camera;
Extracting a bio-signal from the skin area in which the computing device is detected;
Extracting, by the computing device, a respiration signal from the extracted biological signal;
Detecting, by the computing device, an upper peak and a lower peak of the extracted respiration signal;
Measuring, by the computing device, a peak point of a breathing curve using the detected upper peak and the lower peak;
Measuring, by the computing device, a first forced expiratory volume (FEV1), a second forced expiratory volume (FEV2), and a third forced expiratory volume (FEV3) using the detected breathing curve;
Measuring, by the computing device, a forced lung capacity (FVC) using the measured first forced expiratory volume (FEV1), the measured second forced expiratory volume (FEV2), and the measured third forced expiratory volume (FEV3); And
Providing, by the computing device, lung function information using the measured forced lung capacity (FVC);
Digital breathing auscultation method using a skin image comprising a. The method of claim 12,
Measuring the peak point of the breathing curve using the detected upper peak and the lower peak
Digital breath auscultation method using a skin image in which the computing device measures the respiration rate and respiration intensity through the measured peak point of the curve. The method of claim 13,
The respiration rate and the respiration intensity measurement
The computing device detects an inhalation portion and an exhalation portion by measuring the upper peak and the lower peak of the detected respiration signal,
Digital breathing auscultation method using a skin image in which the computing device measures the number of breaths and the intensity of breaths through the height of the upper peak, the height of the lower peak, and the number of peaks. The method of claim 12,
The first forced expiration amount (FEV1), the second forced expiration amount (FEV2), and the third forced expiration amount (FEV3) are
Measured by applying h1 at time t1 after 1 second, h2 at time t2 after 2 seconds, and h3 at time t3 after 3 seconds from the lower peak value of the respiration curve detected by the computing device to a forced expiratory volume regression analysis equation Digital breathing auscultation method using skin image. The method of claim 12,
The forced lung capacity (FVC) is a digital breathing auscultation method using a skin image calculated using Equation 1.
[Equation 1]

The method of claim 12,
Providing lung function information using the measured forced lung capacity (FVC)
The computing device calculates percent lung capacity (%FVC) and 1 second rate (FEV1/FVC) using the forced vital capacity (FVC) and the first forced expiratory capacity (FEV1), and uses a skin image that provides information on the lung function. Digital breathing auscultation method. The method of claim 17,
Providing lung function information using the measured forced lung capacity (FVC)
If the computing device is more than 80% of the lung capacity (%FVC) and more than 70% of the 1 second rate (FEV1), it provides information that predicts normal ventilation,
If the computing device is more than 80% of the lung capacity (%FVC) and less than 70% of the 1 second rate (FEV1), it provides information that predicts obstructive ventilation disorder,
If the computing device is less than 80% of the lung capacity (%FVC) or more than 70% of the 1 second rate (FEV1), it provides information that predicts a constrained ventilation disorder,
When the computing device is less than 80% of the lung capacity (%FVC) or less than 70% of the 1 second rate (FEV1), a digital breathing auscultation method using a skin image provides information predicting a mixed ventilation disorder. Detecting, by a computing device, a skin area from an image captured using a camera;
Extracting a bio-signal from the skin area in which the computing device is detected;
Extracting, by the computing device, a respiration signal from the extracted biological signal;
Interpolating the breathing signal extracted by the computing device at the same sample rate as the sound source;
Calculating, by the computing device, an emphasis filter coefficient on the interpolated breathing signal; And
Filtering the sound source signal by the computing device using the calculated enhancement filter coefficient; Including,
The step of calculating the emphasis filter coefficient includes a digital breathing auscultation method using a skin image calculated using Equation 4 below,
[Equation 4]

In Equation 4,

Is the emphasis filter coefficient, r[n] is the interpolated respiratory signal, and r _p and r _n are the maximum and minimum values of the respiratory signal, respectively. The method of claim 19,
Interpolating the extracted breathing signal to the same number of samples as the sound source (sample rate)
Digital breathing auscultation method using a skin image in which the computing device expands the number of samples in response to a preset sound source with the detected breathing signal. delete The method of claim 19,
Filtering the sound source signal using the enhancement filter coefficient
Digital breathing auscultation method using a skin image calculated using Equation 5 below.
[Equation 5]

In Equation 5, x is a sound source signal and y is a finally calculated modified sound source signal.

Images