KR102054213B1 - Respiratory measurement system using thermovision camera - Google Patents

Abstract

The present invention captures the entire face using a thermal imaging camera (infrared camera), and uses a KLT point tracker (KLT point tracker, Kanade-Lucas-Tomasi Feature Tracker) to capture temperature changes in the region of interest. Detects the features of each frame, which are pixels, obtains the temperature variation of the detected features of each frame, obtains the sum of the temperature variation of each feature in the frame as the sum of the temperature variation of each frame, and adds the sum of the temperature variation of all the frames. To calculate respiratory volume and respiratory rate by detecting peaks and valleys from respiratory signals by applying continuous wavelet transformation to respiratory related data. Respiratory measurement system using an image camera.
The present invention can be used as a test that can replace the pulmonary function test in patients who can not perform the pulmonary function test. In addition, it can be used for the purpose of monitoring the breathing in patients undergoing sedation and can also measure the breathing quantitatively if the respiratory muscle capacity is decreased and the respiratory therapy is needed. It can be used as a tool and also as a tool for analyzing and monitoring driver breathing patterns.

Description

Respiratory measurement system using thermovision camera}

The present invention captures the entire face using a thermal imaging camera (infrared camera), and uses a KLT point tracker (KLT point tracker, Kanade-Lucas-Tomasi Feature Tracker) to capture temperature changes in the region of interest. Detects the features of each frame, which are pixels, obtains the temperature variation of the detected features of each frame, obtains the sum of the temperature variation of each feature in the frame as the sum of the temperature variation of each frame, and adds the sum of the temperature variation of all the frames. To calculate respiratory volume and respiratory rate by detecting peaks and valleys from respiratory signals by applying continuous wavelet transformation to respiratory related data. Respiratory measurement system using an image camera.

The lungs take in oxygen and release carbon dioxide from the body. To diagnose respiratory diseases, various imaging tests (CT, MRI, PET, perfusion scan) and blood tests (arterial blood gas test, tumor marker test, general blood test) can be performed. However, for the functional evaluation of the lungs, only the pulmonary function test is currently the only tool that evaluates the functional aspects of the lungs as an objective indicator and the reproducibility has been verified.

Among them, spirometry is a test that measures the amount of air that a patient can exhale after inhaling as much as possible. Pulmonary function test does not identify a specific disease but diagnoses the disease because it shows unique characteristics according to the disease. It helps to make an early diagnosis or to assess the clinical progress step by step. In addition, it is possible to suggest a solution to the clinical situation in various lung diseases by reflecting the change in the severity of the disease with the change of measurement value, and to help determine the effect before and after the treatment or the reversibility of the treatment. As a representative example, the pulmonary function test helps to diagnose respiratory diseases such as asthma and chronic obstructive pulmonary disease (COPD), and it is also used as a method of checking the respiratory health for abnormalities. . The main indicators to be measured are the forced vital capacity (FVC), which measures the flow rate during the exhalation, and the forced expiratory volume in one, the flow rate measured during the first 1 second of the FVC measurement process. second, FEV1). There is also a bronchodilator test to observe changes in measurements after bronchodilator inhalation.

In general, the spirometry is configured to inhale or exhale with the mouthpiece at the end of the tube connected to the machine.

In recent years, health workers are also rapidly increasing interest in spirometry, and the products already on the market make existing lung function testers portable, and the basic principle of operation is the same. It is used for the purpose of improving the respiratory ability through the repeated training by biting the mouthpiece.

However, such spirometry has limitations in use. For example, if the patient is diagnosed with a respiratory-borne infectious disease, the mouthpiece cannot be bitten, the coordination or impairment of consciousness cannot be performed. In addition, since the test is generally performed in a sitting state, the test may be difficult for a patient who cannot sit.

 Therefore, there is a need for a respiratory volume measurement system that can measure respiratory volume non-invasively and non-involuntarily.

In general, during inspiration, the external cold air receives the temperature of the nasal cavity (nostril, nostril, nose) in the form of convection, conduction, and radiation, thereby lowering the temperature of the nasal cavity.

In the prior art, Korean Patent No. 10-1242755 relates to a system for measuring and monitoring a biological signal of a newborn baby in an incubator in a non-contact manner, and in particular, sets a ROI (Region Of Interest) in a newborn baby's face by a thermal imaging camera. By measuring the respiration of the newborn, and by measuring the temperature of the newborn baby through the thermal body extraction, and relates to a newborn monitoring system using a thermal imaging camera implemented to enable remote monitoring of the health status of the newborn. In Korean Patent No. 10-1242755, in a thermal image, a nasal cavity is extracted by using a face region as a region of interest, and a respiratory rate of a newborn is detected through a change in the temperature of the nose in a thermal image of the nose. However, simply estimating the respiratory rate based on changes in the temperature of the nose can not provide accurate respiratory rate. Even if it is under the nose, the temperature of the thermal image will be different according to the position.If the subject moves, the face may be off the screen or obstructed, so it is not easy to find the exact position under the nose, and it is not an incubator. In Esau, there may be noise caused by other factors, and simply using the body temperature as the respiration signal does not provide an accurate respiration value.

As another prior art, there is a stress index measuring system using a thermal image of the Korean Patent Publication No. 10-2014-0057867, the nose is set as the region of interest in the face region of the thermal image, the respiratory volume is detected by changing the temperature of the nose, Count the number of times the temperature change is zero for a certain period of time (for example, 5 seconds or 10 seconds), divide this value by time, calculate the respiratory rate per second, and multiply the calculated value by 60 to convert to respiratory rate per minute do. In the case of Korea Patent Publication No. 10-2014-0057867, it is not possible to obtain an accurate respiration rate by detecting a change in the nasal temperature as a change in respiration volume and simply estimating the respiration rate. Even if it is under the nose, the temperature according to the thermal image will be different according to the position, and if the subject moves, the face may be off the screen or obscured if the subject moves. There may be noise caused by other factors, so the exact breathing value cannot be obtained simply by using the body temperature as the breathing signal.

In general, when a thermal imaging camera is used, it is important to accurately measure a thermal image under the nose of a subject, but because the subject is moving, the face may move away from the screen or may be covered, and thus the thermal image cannot be measured. In addition, there is a possibility that noise caused by the external environment may enter.

Therefore, the present invention is non-invasive, unrestrained, and can measure the respiratory volume and respiratory rate even if the face is off the screen or obscured, remove the noise, can measure the respiratory volume with higher accuracy Suggests a respiratory volume measurement system.

The technical problem to be solved by the present invention is to shoot the front face using a thermal imaging camera (infrared camera), and by using a KLT feature tracker in the captured image, the feature of each frame that is a pixel having a temperature change in the region of interest The temperature variation of each feature detected in the frame, the sum of the temperature variation of each feature in the frame as the sum of the temperature change of each frame, and integrate the sum of the temperature change of all the frames over time Respiratory measurement using a thermal imaging camera, which calculates respiratory volume and respiratory rate by detecting peaks and valleys from respiratory signals by applying continuous wavelet transform to respiratory data. To provide a system.

Another technical problem to be solved by the present invention, the front of the face around the nose hole using a thermal imaging camera, using the KLT feature tracker in the captured image, while tracking the breath-related areas, both eyes and By using the three tracking points of the nose to infer each other's position, according to the subject's movement, the face can be continuously tracked even if it is off the screen or obscured, thereby detecting the respiratory volume and respiratory rate To provide a breathing measurement system using a thermal imaging camera.

Another technical problem to be solved by the present invention, after detecting the features of each frame detected using the KLT feature tracker, each frame, using 3D-DBSCAN to remove noise, the noise of each frame is removed In order to detect the temperature variation of the features of the region, from the feature of the region of interest of the image of each frame, the corresponding feature of the region of interest of the image of the previous frame is subtracted and detected as the temperature variation of the feature of the region of interest of each frame. The present invention provides a respiration measurement system using a thermal imaging camera that calculates the temperature variation for each frame by summing the temperature variation for each feature when the absolute value of the detected temperature variation for each feature is greater than a preset temperature variation reference value.

In order to solve the above problems, in the breath measurement method using the thermal imaging camera of the present invention, in the frame of the thermal image of the face, which is received from the thermal imaging camera and each pixel represents a temperature, the calculation processing unit includes a nose and two eyes Using the KLT feature tracker to detect a region of interest in the nose portion, using the tracking point as a tracking point; After the breathing-related area tracking step, the operation processor obtains a difference image obtained by subtracting the image of the ROI of each frame from the image of the ROI of the next frame in the thermal image of the face received from the thermal imaging camera. The value of each pixel of the difference image is the value of the temperature variation, and the temperature variation value is added to each frame to be sequentially stored in the memory unit as the sum of the temperature change amount per frame, and the temperature for each frame of all the frames stored for a predetermined time. And a respiratory signal extraction step of detecting the respiratory signal by applying continuous wavelet transform to the respiratory related data by reading the sum of the change amounts from the memory unit and integrating according to time.

After the breathing signal extraction step, the calculation unit, detecting the peak and valley in the breathing signal and detecting the respiratory rate and respiratory volume using the detected peak and valley; further comprises a;

In the respiratory-related area tracking step, the processing unit obtains a minimum temperature value and a maximum temperature value in each frame of the subject's face thermal image received from the thermal imaging camera, and the temperature of each pixel of each frame is determined by the minimum temperature value. A normalization step of performing normalization to have a value equal to 256 between the maximum temperature values; An angle correction step of correcting, by the calculation processing unit, a nose of each frame normalized in the normalization step to be equal to a preset initial nose angle; The arithmetic unit sets candidates of the region of interest using the tracking points of the nose and two eyes in each frame whose angles are corrected in the angle correction step, extracts feature points from the candidates of the region of interest, and extracts the extracted feature points. 1, which represents a moving distance between the feature point and the corresponding feature point in a subsequent previous frame in a matrix, corrects an image of the current frame according to the extracted matrix, and obtains a region of interest in the corrected current frame. Tea feature extraction step; includes.

In addition, in the respiration-related region tracking step, the calculation processing unit obtains an average temperature in the region of interest of each frame obtained by the primary feature extraction step, and obtains the calculation region as the average region of interest for each frame. A first average temperature and deviation calculation step of obtaining a standard deviation of temperatures within the frame as a region of interest temperature deviation for each frame; The calculation processor compares the ROI deviation obtained in the average temperature and deviation calculation step with a preset deviation reference value, and determines whether the ROI deviation is greater than the deviation reference value, and if the ROI deviation is greater than the deviation reference value, Determining whether a primary deviation criterion is exceeded by removing features in the region of interest and going to a re-detection of a tracking point; After the step of determining whether the primary deviation reference value is exceeded, the operation processor compares the number of feature points obtained in the ROI with the preset feature number reference value, and if the number of feature points is smaller than the feature number reference value, removes the features in the ROI. Determining whether the feature number is less than the threshold value, and going to the tracking point redetection step; If the deviation of the region of interest is greater than the deviation criterion in the step of determining whether the primary deviation criterion is exceeded, or the number of feature points is less than the feature number criterion in the determination of whether the feature deviation is less than the criterion, the calculation point of the nose is incorrectly obtained. Determine the tracking point movement distance, which is the distance traveled from each position of the two eye tracking points of the previous frame following the current frame to each position of the two eye tracking points of the current frame, and the two tracking points A tracking point redetection step of obtaining a tracking point moving distance average, which is an average of moving distances, and finding a tracking point of the nose of the previous frame and a modified tracking point of the nose using the tracking point moving distance average. .

In addition, the respiratory-related area tracking step, the operation processing unit, using the modified tracking point of the nose obtained in the tracking point re-detection step and the tracking point of the two eyes, to obtain the region of interest and extract the feature, 2 Tea feature extraction step; The arithmetic processing unit calculates the standard deviation of temperatures in the ROI of each frame by averaging the temperatures in the ROIs of each frame obtained in the second feature extraction step, and calculates the standard deviation of the temperatures in the ROIs of each frame. Calculating a second average temperature and a deviation as a region of interest temperature deviation; The calculation processing unit compares the ROI temperature deviation obtained in the second average temperature and the deviation calculation step with the deviation reference value to determine whether the ROI temperature deviation is greater than the deviation reference value, and if the ROI temperature deviation is greater than the deviation reference value Determining whether or not the second deviation threshold value is exceeded, by ending tracking (tracking); If the end flag is set as the end switch is pressed, or if the current frame corresponds to the last frame of the predetermined predetermined time interval, the processing unit ends tracking, otherwise, the face thermal image of the next frame. Receiving a frame of, and goes to the normalization step, the tracking end determination step; further comprises.

In the respiratory signal extraction step, the calculation processor obtains a difference image obtained by subtracting an image of the ROI of each frame from an image of the ROI of each frame in succession from the thermal image of the face received from the thermal imaging camera. A temperature variation detection step by frame, wherein the value of each pixel of the image is a temperature variation value; A noise removing step of the arithmetic processing unit to remove a noise point by using the 3D-DBSCAN in a region of interest of each frame of the difference image which has undergone the step of detecting the temperature variation per frame; The step of comparing the temperature variation reference value, wherein the processing unit removes the value of the temperature variation when the absolute value of the temperature variation value in each frame of the difference image which has undergone the noise removing step is smaller than or equal to the preset temperature variation reference value. ; And a temperature change amount calculating step of calculating, by the calculation processing unit, summing each temperature change in each frame of the difference image, calculating the amount of temperature change for each frame, and storing the frame according to the frame order.

In addition, in the breath signal extraction step, the calculation processing unit sequentially reads the sums of the temperature changes for each frame obtained in the step of calculating the temperature change from the memory unit, integrates the temperature changes for each frame according to time (in the t-axis), and calculates the integration result. Respiratory data detection, using respiratory data; A smoothing and detrending step, wherein the computing unit performs smoothing and detrending of the respiratory data detected in the respiratory data detection step; A continuous wavelet transform step of performing, by the processing unit, continuous wavelet transform (CWT) on the respiratory data on which smoothing and detrending is performed, and displaying frequency scales for each frequency; The calculation processing unit obtains the sum of the frequency scales for each frequency based on the result of the continuous wavelet transformation, and among the obtained sums of frequency scales for the frequencies, the respiratory frequency is the frequency at which the sum of the frequency scales for each frequency is the largest. Respiratory frequency detection step of storing each frequency scale of the memory unit as a breathing signal; further includes.

In the respiratory rate and respiratory rate detection step, the calculation processing unit obtains an inflection point by first differentiating the respiration signal obtained in the respiration signal extraction step, and uses the inflection point to detect peaks and valleys. In the peaks and valleys detected in the peak and valley detection step, the calculation processing unit calculates the number of pairs of peaks and valleys that appear sequentially after the peaks and valleys during a predetermined time interval. It includes; detecting the respiratory rate, respiratory rate;

In the respiratory rate and respiratory rate detection step, the computing unit detects the amplitude of the peak and valley pairs that appear sequentially after the peaks and valleys as the respiratory volume of the respiration cycle including the peak and valley pairs, or the peak and valleys. Detects the respiratory cycle with a pair of peaks and valleys that are sequentially followed, subtracts the respiratory signal of each respiratory cycle, and subtracts from the respiratory signals of all subsequent respiratory cycles to determine the change in each respiratory cycle, It may further include a; respiratory volume detection step, summed over time to obtain a respiratory volume.

When the change amount of each breathing cycle is summed over time, it is added based on the starting point of each breathing cycle.

The invention also features a recording medium storing a computer program source for a respiration measurement method using a thermal imaging camera of the invention.

In another aspect, the present invention, in the respiration measurement system using a thermal imaging camera, a thermal imaging camera including an image analysis unit including a processing unit for detecting a breathing signal by analyzing the thermal image of the front face received from the thermal imaging camera In the frame of the thermal image of the face, which is received from the thermal imaging camera and each pixel represents a temperature, the arithmetic processing unit uses the KLT feature tracker with the nose and two eyes as tracking points, and the nose portion. Detects a region of interest of the camera and obtains a difference image obtained by subtracting an image of the region of interest of each frame from an image of the region of interest of a subsequent frame from a thermal image of the face received from the thermal imaging camera. The pixel value is used as the temperature variation value, and the temperature variation value is summed for each frame, and is sequentially stored in the memory unit as the sum of the temperature variation amounts for each frame. The respiratory data is obtained by reading the sum of the temperature change for each frame of every frame stored for a predetermined time from the memory unit and integrating according to the time, and detecting the respiratory signal by applying continuous wavelet transform to the respiratory data. .

 The computing unit detects peaks and valleys from the detected breathing signals, and detects respiratory rate and respiratory volume using the detected peaks and valleys.

In order to obtain a region of interest using the KLT feature tracker, the calculation unit normalizes the temperature of each pixel of each frame of the subject's face thermal image received from the thermal imager to have an 8-bit value, and normalizes the value of the temperature. Correct the image of each frame such that the nose angle of the image of each frame having the same as the preset nose angle, obtain a region of interest by applying a KLT feature tracking algorithm to the image of the corrected frame, The region of interest is also obtained from the frame of the facial thermal image received from the thermal image camera, which is a raw image corresponding to the region of interest of the corrected frame.

The computing unit performs smoothing and detrending of breathing data, performs continuous wavelet transform (CWT) on breathing data performed by smoothing and detrending, and performs continuous wavelet transform on a frequency-by-frequency basis. The sum of the frequency scales is obtained, and among the obtained sums of frequency scales for each frequency, the frequency having the largest sum of frequency scales for each frequency is the respiratory frequency, and each frequency scale of the respiration frequency is stored as a respiration signal in the memory unit.

The calculation processing unit obtains an inflection point by first-ordering the respiration signal, detects peaks and valleys using the inflection points, and at the detected peaks and valleys during a predetermined time interval. The number of pairs of peaks and valleys appearing successively after the peaks and valleys is detected as the respiratory rate.

The computational processing unit detects the respiratory cycle with a pair of peaks and valleys that appear in succession of peaks and valleys, subtracts the respiratory signals of each respiratory cycle from the respiratory signals of all subsequent respiratory cycles, and calculates the amount of change for each respiratory cycle. Alternatively, the amplitudes of the peak and valley pairs appearing sequentially after the peaks and valleys are detected as the respiratory volume of the respiratory cycle including the peak and valley pairs.

In the breath measurement system using the thermal imaging camera of the present invention, the front face is photographed using a thermal imaging camera (infrared camera), and a frame having a temperature change in a region of interest using a KLT feature tracker in the captured image. Detects the features of each feature, obtains the temperature variation of the detected features of each frame, calculates the sum of the temperature variation of each feature in the frame as the temperature change of each frame, and integrates the temperature change of all the frames over time as breath-related data. Respiratory signal is detected by applying continuous wavelet transform to respiratory data, and peak and valley are detected from respiratory signal to calculate respiratory volume and respiratory rate. Can be detected.

In addition, the present invention, the front of the face around the nose portion (nostril) using a thermal imaging camera, using the KLT feature tracker in the captured image, while tracking the breath-related area, the eyes of both eyes and nose 3 tracking points can be used to infer each other's position, so that even if the face is off the screen or hidden, it can continuously track the volume and the number of breaths.

In addition, the present invention, using the 3D-DBSCAN in the portion extracted through tracking, to remove the noise, to extract the data (raw data) in the portion from which the noise is removed, from the extracted data, frequency using CWT Respiratory volume and respiratory rate can be detected more accurately by extracting only the part related to respiration in the domain to the respiratory area.

Respiration measurement system using a thermal imaging camera of the present invention can enable a non-invasive method of respiratory pattern, one-time breathing volume, effort exertion, comparison of the effects before and after the drug, respiratory rehabilitation treatment.

First, it can be used as an alternative to the pulmonary function test in patients who cannot perform the pulmonary function test. Non-invasive, relatively inexpensive, non-invasive, relatively inexpensive methods for patients who were unable to test for pulmonary function, such as patients with bronchial incisions that could not bite the mouthpiece, patients who could not sit, and patients with respiratory-borne infectious diseases such as MERS. As an alternative to pulmonary function tests.

Secondly, it can be used for the purpose of monitoring the breathing in patients who are sedated. Recently, in the medical field, such as gastrointestinal endoscopy and MRI, which requires imaging for more than 30 minutes, frequency of administering sedatives for monitoring is increasing for the purpose of patient comfort and ease of treatment. Peripheral arterial oxygen saturation is measured to monitor respiratory depression due to oversettling, but there is a limit of monitoring due to lack of accuracy and rapidity. The present invention can be monitored non-invasive real-time breathing can be helpful in terms of patient safety.

Third, in the medical field, if the capacity of the respiratory muscles is reduced and respiratory rehabilitation is needed, quantitatively measuring respiration may be helpful. In particular, by incorporating the present invention into rehabilitation exercise equipment, mobile devices, can be used to enhance the effect of exercise through biofeedback during respiratory rehabilitation treatment, there is no restriction in the place, there is a possibility to use in home care monitoring system.

Fourth, there is potential for use as a screening tool for diagnosis of respiratory diseases. Currently, there are no tests for screening for respiratory diseases except medical examination, consultation, and chest radiography. The algorithms of the present technology can be analyzed and used to screen patients at the stage of disease diagnosis.

In addition, the respiration measurement system using the thermal imaging camera of the present invention can be utilized in the development of a health device for the general public in addition to the medical environment. After the wide-area drowsiness driving accident that led to the recent fatal accident, the drowsiness driving prevention device has attracted attention again. Using the present invention, it is also possible to analyze the breathing pattern of the driver and to use it for monitoring purposes.

In addition, living in an environment where microscopic dust indexes are released in daily weather forecasts, modern people are not only concerned about respiratory disease and worsening, but also pay attention to improving respiratory ability. Fitness bands such as Mi-Band is expanding the base in the health market and can be used to measure the lung capacity before and after exercise by interlocking the breath measurement system using the thermal imaging camera of the present invention.

1 is a conceptual diagram of a respiration measurement system using a thermal imaging camera of the present invention.
Figure 2 is a block diagram schematically showing the configuration of a respiration measurement system using a thermal imaging camera of the invention.
FIG. 3 is a flowchart schematically illustrating a method of detecting a respiratory rate using thermal image data received from a thermal imaging camera in the image analyzer 200 of FIG. 2.
4A is a flowchart schematically illustrating a KLT feature tracking algorithm according to an embodiment of the present invention.
FIG. 4B is a flowchart for describing the primary feature extraction step S150 of FIG. 4A.
4C is an explanatory diagram illustrating a process of extracting a region of interest from feature extraction in FIG. 4B.
FIG. 5 is a flowchart for schematically describing a respiratory signal extraction step of FIG. 3.
FIG. 6 is a flowchart for schematically describing a respiratory rate calculation step of FIG. 3.
7 is an example of a result of performing a temperature variation detection step and a temperature variation reference value comparison step for each feature in the breath signal extraction step in FIG.
8 is an example of a result of removing noise by using the 3D-DBSCAN in the noise removing step of the breath signal extraction step (S200).
9 is an example of the result of the temperature change amount calculation step.
10 is an example of the results of the respiratory data detection step.
11 is an example of the results of the continuous wavelet transform step.
12 is a view for explaining the respiratory frequency detection step.
13 is an example of the results of the peak and valley detection steps.
14 illustrates correcting the angle in the thermal image rod and angle correcting step.
15 illustrates an example of extracting a feature in the first feature extraction step.
FIG. 16 is an explanatory diagram for explaining an average temperature and a deviation of a region of interest in each frame in the present invention.

Hereinafter, a respiration measurement system using a thermal imaging camera of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a respiration measurement system using a thermal imaging camera of the present invention, Figure 2 is a block diagram schematically showing the configuration of a respiration measurement system using a thermal imaging camera of the present invention.

The face of the subject 50 is photographed by the thermal imaging camera (infrared camera) 100, converted into a digital signal, and transmitted to the image analyzer 200. In this case, the image captured by the thermal camera 100 is converted into a digital signal and transmitted to the image analyzer 200, or the image captured by the thermal camera 100 is transferred through an A / D converter (not shown). The digital signal may be converted into a digital signal and transmitted to the image analyzer 200.

The thermal imaging camera 100 may use an infrared camera. The thermal imaging camera 100 photographs the front of the subject's face, but inevitably photographs both eyes and nose of the subject 50. Shooting can acquire data at 10 times per second (10 Hz), and you get thermal imaging data in absolute temperature.

In general, body temperature changes according to the body's breathing, which causes cold air to move around the body during inspiration, lowering the temperature (ie, the temperature of the part of the body related to breathing), and warming the air inside the body during exhalation. Let's do it. Therefore, the present invention, while photographing the front of the subject's face, the image is taken mainly around the nastril portion through which air passes.

Also, in general, in the case of an object having a temperature, infrared radiation is always emitted, and the emitted infrared rays have a shorter wavelength as the temperature increases, and the frequency increases to have a higher energy, which is an infrared camera (IR camera). Measure and convert to temperature.

FLIR's T420 can be used as an infrared camera and extracts only parts with a temperature change of more than NETD (noise equivalent Temperature difference), which has a resolution of 320x240, a NETD of 0.05, and a minimum focal length of 50 cm. The subject can be measured at a distance of approximately 50-60 cm. However, the present invention is not intended to limit the present invention, and any thermal imaging camera may be used as long as the thermal image is detected from the face of the subject 50. In general, NETD is different for each camera specification, and is set to 0.05 here, but is not intended to limit the present invention.

The image analyzer 200 is a means for detecting a breathing signal by analyzing an image received from the thermal imaging camera 100, and may be made of a general computer, a laptop, a microprocessor, or the like. The image analyzer 200 may include an operation processor 210, a memory 220, an output 230, and a key input 250.

The calculation processor 210 tracks a breathing-related area by using a KLT point tracker (KLT point tracker, Kanade-Lucas-Tomasi Feature Tracker) in the image received from the thermal imaging camera 100, thereby affecting breathing. Extracts the region of interest, which is only a part of the region, extracts only the portion of the region of interest that is related to breathing in the frequency domain from the region of interest, and applies the extracted breathing region to the volumetric expression to calculate the volume waveform. Calculate, find peaks and valleys in the waveform, and calculate respiratory volume and respiratory rate.

In other words, the calculation processing unit 210 is a KLT feature tracking algorithm for each pixel of each frame of the subject's face thermal image received from the thermal imaging camera 100, that is, absolute temperature values (hereinafter referred to as temperature values for convenience of description). Normalize to have an 8-bit value that can be applied, and correct the image of each frame so that the nose angle of the image of each frame having the normalized temperature value is equal to the preset nose angle, Apply the KLT feature tracking algorithm to find the region of interest. Accordingly, when the ROI is obtained from the corrected image, the ROI is obtained from a frame of the original image corresponding to the original image, that is, the subject's face thermal image received from the thermal camera 100. That is, one region of interest is obtained for each frame of the normalized and corrected image through the KLT feature tracking algorithm, and correspondingly, each of the original subject's face thermal images received from the non-normalized thermal imager 100 is correspondingly obtained. The region of interest is also obtained in the frame.

The operation processor 210 stores the ROIs of each frame of the subject's face thermal image received from the thermal imaging camera 100 in the memory unit 220, and stores each region of the frame of the subject's face thermal image stored in the memory unit 220. The region of interest is subtracted from the region of interest of each subsequent frame to obtain a difference image of the region of interest of each frame, and noise is removed by applying 3D-DBSCAN from the difference image of the region of interest of each frame obtained. If the pixel value of the difference image of the ROI of each frame from which noise is removed, that is, the temperature variation value is smaller than or equal to the preset temperature variation reference value, is removed, and only if the temperature variation value is larger than the reference region Is stored as a temperature variation value (i.e., temperature variation data) of each frame, and the temperature variation value (temperature variation data) is summed for each region of interest of each frame, and is obtained as the temperature variation amount for each frame. 220) sequentially.

The calculation processor 210 reads the stored frame temperature changes from the memory unit 220 in time order and integrates the frame temperature changes according to time (that is, on the t-axis) to obtain raw data. .

The operation processor 210 performs smoothing of the detected respiratory data, performs detrending, and performs continuous wavelet transform of the respiratory data performed by smoothing and detrending. ) Is expressed by the frequency-specific scales (intensity, magnitude), and the sum of the frequency-specific scales (sum-by-frequency intensity) is obtained from the results of continuous wavelet transform. The frequency (ie, the frequency at which the sum of the scales for each frequency is the largest) is the respiratory frequency (more specifically, the frequency most relevant to respiration), and each scale of the respiratory frequency is used as the respiration signal. Store in Here, when the scales of the breathing signals are plotted over time, the scales become breathing waveforms. Therefore, this breathing waveform is a waveform (data) of the time domain.

The calculation processing unit 210 obtains an inflection point by first-order differentiating a respiration signal, detects peaks and valleys therefrom, and at the detected peaks and valleys for a predetermined time interval. (Per second, or per minute or 30 seconds), the number of consecutive peak and valley pairs, i.e., the number of peaks to valleys, is detected as the respiration rate (RR).

Respiratory volume can be obtained in two ways.

The first method detects the amplitude of a pair of consecutive peaks and valleys, i.e., the amplitude of the signal that changes from peak to valley, in the breathing volume of the cycle.

In the second method, the breathing waveform is one breathing cycle from one peak to the next, and the breathing signal of each breathing cycle is subtracted from the breathing signals of the previous breathing cycles, and the change amount for each breathing cycle is obtained. Sum the changes and find the breath volume. In some cases, the respiratory cycle may be a time interval from one peak and valley pair to the next previous peak and valley pair.

That is, the amount of change in each breathing cycle is obtained by subtracting the breathing signal (each sample) of the current breathing cycle from each corresponding breathing signal (each sample) in the subsequent entire breathing cycle. For example, from the first breathing signal (sample) of the current breathing cycle, subtract the first breathing signal (sample) of the previous breathing cycle to find the first breathing change of the current breathing cycle. Obtain the change in respiration for each cycle. The respiratory volume is calculated by summing the respiratory changes in each cycle. Here, summing the respiratory changes in each cycle by time zone is the first respiratory variance signal in each cycle, adding up as the first respiratory rate signal, and adding the second respiratory variance signal in each cycle, in order to make the second respiratory rate signal. The respiratory volume is calculated by summing the respiratory volume change signals.

The memory unit 220 stores the amount of breath and the number of breaths received from the calculation processing unit 210.

The output unit 230 outputs the amount of breath and the number of breaths received from the calculation processing unit 210.

The key input unit 250 may include a graphic user interface, through which an initial tracking point may be set, and a start / stop switch may be included.

3 is a flowchart schematically illustrating a method of detecting a respiratory volume using thermal image data received from a thermal camera in the image analyzer 200 of FIG. 2.

In the thermal image receiving step (S10), the calculation processing unit 210 of the image analyzer 200 receives a thermal face image (thermal image) of the subject from the thermal imaging camera 100 and receives a memory unit ( Temporarily store in 220). In the thermal image receiving step S10, a thermal image may be received for each frame, but a thermal image of one frame at a time may be received, or a thermal image of frames for a predetermined time may be received. For example, you can get a 320x240 Absolute Temperature matrix in front of your face, 10 frames per second. The data obtained from the thermal imaging camera 100 may be referred to as temperature data or temperature domain data. In this way, it can be distinguished from normalized data to apply the KLT feature tracker algorithm.

Respiratory-related area tracking step (S100), each pixel of each frame of the subject's face thermal image (ie, the subject's face thermal image received from the thermal imaging camera 100) received in the thermal image receiving step (S10), that is, Normalize the temperature values (absolute temperature values) to have 8-bit values, correct the image of each frame so that the nose angle of the image of each frame having the normalized temperature value is equal to the preset nose angle, and correct A KLT feature tracker, that is, a KLT point tracking algorithm, is used in each frame of the image, and two tracking points are used for each eye and nose in each frame of the face thermal image. The breath-related area, that is, the nose or only the nose part, is extracted as a region of interest (ROI), and at this time, a feature is extracted for tracking. Here, the feature is a feature detected by a specific calculation method (for example, a minimum eigenvalue method, etc.), which is also called a KLT feature in the KLT feature tracking algorithm, and since it is well known, a detailed description thereof will be omitted. The detected KLT feature mainly detects a border line part, a corner (corner) part, etc. (for example, a border line or a corner around the nose in a face image). Tracking and tracking the positions of two eyes and noses of the subject 50, which are three tracking points, using the three tracking points to estimate each other's position, and a predetermined position from three tracking points or from the nose A region of interest (ROI) located at is detected, and features of the region of interest are extracted and tracked.

Prior to applying the KLT feature tracking algorithm, at the beginning of the breath-related area tracking step S100, the position of the nose is positioned in the first face thermal image frame of the continuous facial thermal image frame coming from the thermal image receiving step S10. The thermal image is positioned to be aligned with a predetermined predetermined position. In addition, the position of the nose and the two eyes in the first face thermal image frame, the size to be used as a region of interest (ROI), and the angle of the inclined face (or nose angle) of the face thermal image ) And other KLT feature tracking algorithm parameters for facial thermal imaging. Accordingly, the memory unit 220 stores the coordinates of the nose and eyes, the angle of the nose (angle of the face), the size of the ROI, and the like. The parameters can be changed according to the required precision and the degree of movement of the patient.

Thereafter, the image of each frame received in the thermal image receiving step S10 is normalized to have an 8 bit value, and the nose angle of the image of each frame having the normalized temperature value is equal to the preset nose angle. The ROI is obtained by correcting the image of each frame and applying the KLT feature tracking algorithm to the corrected image of each frame. In other words, the portion of the nose that is substantially affected by respiration is extracted as the region of interest.

The subject 50 may have a movement due to breathing and other reasons, and if the face moves away from the screen of the camera or is covered by the movement of the subject 50, among the three tracking points in the face thermal image. If only a portion remains, the tracking point remaining in the face thermal image is used to estimate a location of the tracking point not in the face thermal image. That is, it uses a KLT feature tracking algorithm to track each point as the center of each eye as a point, the center of the tip of the nose as a point, and uses the characteristics of the three position coordinates. When is not traceable, the position of the untraceable point, i.e., the coordinate, can be inferred to keep track.

Respiratory-related area tracking step (S100) is a step for detecting a region of interest, in this way, the region of interest is obtained from the image of the frame normalized and corrected by the KLT feature tracking algorithm, and thus, the original image, the heat The region of interest is also obtained for the frame of the subject's face thermal image received in the image receiving step (S10).

In the present invention, for convenience of explanation, the normalized image (image) in the breath-related area tracking step (S100) is called an image (data) of the image domain and is received in the thermal image receiving step (S10), that is, the thermal image. An image (image) received from the camera 100 is called an image (data) of the temperature domain.

In the respiratory signal extraction step (S200), the temperature change amount of each frame is obtained from the ROI of each frame of the face thermal image detected in the respiration-related area tracking step (S100), and the respiration signal is calculated using the temperature change amount of each frame. Extracting the breathing signal. This is obtained by subtracting the region of interest of each frame of the subject's face thermal image received from the thermal imaging camera 100 to the region of interest of successive previous frames, and obtaining the difference image of the region of interest of each frame. 3D-DBSCAN is applied to the difference image to remove noise, and when the pixel value of the difference image of the region of interest of each frame from which the noise is removed, that is, the temperature variation value is smaller than or equal to the preset temperature variation reference value, is removed. Only when the temperature variation is greater than the reference value, the temperature variation value (ie, temperature variation data) of the ROI of each frame is stored, and the temperature variation value (temperature variation data) is added to each region of interest of each frame, and the frame The respiration data (raw data) are obtained by integrating the obtained temperature changes per frame with time (that is, on the t-axis). After smoothing the detected respiratory data, detrend is performed, and continuous wavelet transform (CWT) of the respiratory data smoothed and detrended is performed for each frequency. Expressed in scales (intensity, magnitude), the sum of the frequency-specific scales (the sum of the frequency-specific strengths) is obtained from the result of the continuous wavelet transformation. The frequency (ie, the frequency at which the sum of the scales for each frequency is the largest) is the respiratory frequency (more specifically, the frequency most relevant to respiration), and each scale of the respiratory frequency is used as the respiration signal. Store in

In the respiratory rate calculation step (S300), a peak and a valley are detected in the volume waveform, and at the detected peaks and valleys, for a predetermined time interval (per second, Or 1 minute or 30 seconds), the number of consecutive peaks and valley pairs, i.e., the number of peak to valley changes, is detected as the respiration rate (RR) and the respiratory volume is also determined. As described above, the respiratory volume can be obtained by two methods, but it is assumed here by the second method. In other words, one breathing cycle from one peak to the next is one breathing cycle.The breathing signal of each breathing cycle is subtracted from the breathing signals of all previous breathing cycles, and the change of each breathing cycle is calculated. Add up to get your breathing volume.

4A is a flowchart for schematically describing a KLT feature tracking algorithm according to an embodiment of the present invention, FIG. 4B is a flowchart for explaining a primary feature extraction step S150 of FIG. 4A, and FIG. 4C is a feature extraction for FIG. 4B. An explanatory diagram for explaining a process of extracting a region of interest from an image.

The KLT feature tracking algorithm is executed in the calculation processing unit 210 and is included in the respiratory related area tracking step (S100).

The initialization step S105 and the operation processor 210 read the deviation reference value, the feature number reference value, and the like from the memory unit 220.

In the initial tracking point setting step (S110), the user specifies the position of the nose and the position of the two eyes in the first frame of the subject's face thermal image received in the thermal image receiving step (S10) through the key input unit 250, The operation processor 210 receives a nose position and two eye positions from the key input unit 250, that is, a nose coordinate and two eye coordinates, and stores the nose position and two eye coordinates in the memory unit 220 as an initial tracking point. Here, two eyes of the examinee may be referred to as a first eye and a second eye. At this time, the ROI size and location may also be set.

In the initial nose angle setting step (S115), the user specifies the angle of the nose in the first frame of the subject's face thermal image received in the thermal image receiving step (S10), and the operation processor 210 is a nose from the key input unit 250 Receives the angle of, and stores in the memory unit 220 as the initial angle of the nose.

In the thermal image loading step S120, a thermal image of a subject face received in the thermal image receiving step S10 and temporarily stored in the memory unit 220 is read in units of frames.

In the normalization step S130, each pixel of each frame of the subject's face thermal image received in the thermal image loading step S120, that is, between absolute temperature data and a minimum temperature max value, is determined. Normalization is performed by imaging to 256 equal values (8 bits).

Here, the normalization method calculates the median and maximum temperatures for each frame of the face thermal image. The difference between the maximum temperature and the intermediate temperature of each frame of the face thermal image, that is, the value obtained by subtracting the median from the maximum temperature max is obtained. In other words, find difference = max-median. Next, the minimum temperature min of each frame of the face thermal image is obtained, and the minimum temperature min is obtained by subtracting the difference between the maximum temperature and the median temperature from the median temperature. . That is, min = median-difference is found. In this case, a temperature range is referred to between a minimum temperature value and a maximum temperature value. Each pixel of each frame of the face thermal image is converted into and stored at a temperature range, that is, a value obtained by dividing the minimum temperature (min) value and the maximum temperature (max) value by 256 equally. For example, when the median value is 36.1 degrees and the maximum temperature value is 38.1 degrees, the difference value is 2 degrees and the minimum temperature min value is 34.1 degrees. 8bit image is created by converting 34.1 ~ 38.1 degrees into 256 equals.

In the angle correction step S135, the nose of each frame of the subject's face thermal image normalized in the normalization step S130 is corrected to be equal to the initial nose angle set in the initial nose angle setting step S115, that is, the initial By comparing the nose angle and the nose of each frame of the subject's facial thermal image received after the first frame, it is determined whether the nose is skewed and if it is skewed, the nose of the facial thermal image frame has the same angle as the initial nose angle. To correct the face thermal image frame.

That is, the angle correction step S135 includes the frames of the continuous facial thermal images read from the thermal image reception step S10 and normalized in the normalization step S130, by the initial nose angle, the first facial column. The thermal image is positioned so that the position of the nose in the image frame is aligned with a predetermined predetermined position.

In the first feature extraction step (S150), the first eye and the second eye to be used for tracking in each frame of the subject's face thermal image, normalized in the normalization step (S130) and angle corrected in the angle correction step (S135). Extract the eye and nose tracking points and, using these three tracking points (or using the nose tracking points), are candidates for the region of interest (nose or nasal or nasal) (FIG. 4C). Set the blue squares of the < RTI ID = 0.0 > and < / RTI > extract a feature point (green " + " portion of FIG. 4C) from the candidate of the region of interest (S153), and feature points corresponding to the feature points in subsequent frames ( The coordinates of (S154), the extracted feature points (coordinates of), and the distance (movement degree) between the feature points corresponding to the feature points (coordinates of the corresponding points) in the previous frame are represented by a matrix (S155). ), According to the extracted matrix, the image of the current frame ( That is, correction is performed to warp the subject's face thermal image output in the angle correction step S135 (S156), and the ROI (red square part of FIG. 4C) is obtained.

In the present invention, the tracking basically tracks the nose area in the face thermal image, and in some cases it is difficult to track the nose by the movement of the patient, in which case to estimate the position of the nose Also track the position of both eyes. That is, when the location of the nose cannot be determined, the location of the nose is estimated using information of two eyes.

A situation where it is difficult to track the nose is when the KLT fails to specify the region of interest or the desired portion of the nose. In this case, first, when the nose leaves the screen (out of the range of the tracking point) , Second, when hands and other objects pass over the face (no nose features found, and changes in temperature bands, such as the average temperature and deviation of the nose), third, when the angle of the front face changes and the position cannot be determined. (No feature found in the nose). In this case, as the prediction method, since the positions of the two eyes do not change, the position of the nose is estimated by the average of the movement trajectories of the two eyes.

In the present invention, individual tracking is performed for each of the two eyes (ie, the first eye and the second eye) and the nose of the subject, which are the tracking points, and thus the region of interest is set.

The KLT sets a region of interest in the image domain and extracts feature points of the image within the region of interest. The transform matrix is obtained by calculating how the feature points change in the next frame with the information of the feature points. Using this, several slices containing the same region of interest can be obtained. Here, the feature extraction may basically use a mininum eigen feature (minimum eigenvalue) algorithm (method), or may use various feature extraction methods.

In the first average temperature and deviation calculation step (S155), the temperature in the region of interest of each frame obtained in the first feature extraction step (S150) is averaged to obtain the average region of interest for each frame, and the interest to the current frame. Obtain the average region of interest of the total frame by averaging the region average temperature, and determine the standard deviation of the temperatures in the region of interest of each frame (i.e. how far from the region of interest the average temperature of each frame is ) Is obtained as 'region of interest temperature deviation of each frame', and the temperature of each feature in the region of interest of the current frame is averaged with the temperatures of the corresponding features of all frames, and the average temperature of each feature is obtained. The temperature deviation of each feature of each frame, which is the absolute value of the difference between the current temperature of each feature of each frame and the average temperature of each feature, is obtained.

That is, the average temperature and deviation calculation step (S155) calculates the ROI average temperature of each frame, the ROI temperature deviation of each frame, the ROI average temperature of the total frames, the average temperature of each feature, and the deviation of each feature. Therefore, some of them (eg, average temperature for each feature, deviation for each feature, and average region of interest of the total frame) may be omitted.

In the determination of whether the primary deviation reference value is exceeded (S160), the deviation of the ROI obtained from the average temperature and the deviation calculation step (S130) is compared with the deviation reference value to determine whether the ROI deviation is greater than the deviation reference value. If the deviation is larger than the deviation reference value, the features are incorrectly detected, and the features in the region of interest are removed (S166), and the tracking point redetection step (S167) is performed.

Here, the deviation reference value may be a value set and stored by the user in the initial setting step or a factory stored value, or may be a human body temperature change limit value or larger value.

In FIG. 16, the horizontal axis (x-axis) represents each frame according to time, and the red line represents the ROI average temperature for each frame, and the blue line shows the ROI average temperature for each frame and the ROI temperature for each frame. The green line is a graph showing the sum of the deviations, and the green line is a graph indicating that the region of interest temperature by frame is subtracted from the region of interest temperature by frame. In other words, if the same face area is tracked (tracking) normally (because the change in temperature due to breathing is very small and the effect is small when the average is taken), the deviation from the mean is maintained or does not change rapidly. However, when the contaminated part comes in, the average and deviation of the temperature change drastically. Therefore, in the case of a change in the frames 6-7 of FIG. 16, it is determined that the ROI is wrong. That is, it is determined that the ROI is not a nose.

In the determining of whether the feature number is less than the reference value (S165), after determining whether the feature deviation is exceeded, the number of feature points is determined by comparing the number of feature points obtained in the ROI with a predetermined feature number reference value. If smaller, the feature points are erroneously detected, and features in the region of interest are removed (S166), and the tracking point redetection step (S167) is performed.

When the features in the region of interest are removed (S166), the region of interest temperature, the region of interest average temperature, the average temperature of each feature, and the deviations of the features obtained for the current frame are also removed.

If the region of interest deviation is greater than the deviation reference value in the first deviation reference value determination step (S160), or if the number of features is less than the feature number reference value in the determination whether the feature number reference value is less than the reference value (S165), this indicates nose-related tracking points. It is incorrectly determined that the area of interest is incorrectly obtained.

In the tracking point redetection step (S167), if the region of interest deviation is greater than the deviation reference value in the first deviation reference value determination step (S160), or the feature number reference value is determined in the determination step (S165) feature number reference value If smaller, the wrong tracking points of the nose are obtained, and for the remaining two tracking points (ie two eye tracking points), the remaining two tracking points of the previous frame following the current frame (ie two eye tracking points) Points) From each position, the tracking point movement distance, which is the distance traveled to each position of the tracking point (i.e. two eye tracking points) of the current frame, is obtained, and the tracking point is an average of the two tracking point movement distances. The distance of the nose corrected using the moving distance average of the previous frame's nose and the tracking point moving distance average Calculate the points again. That is, the corrected nose tracking point calculates the nose tracking point as the tracking point of the nose of the previous frame is moved by the tracking point moving distance average.

Here, the tracking point moving distance is from the position of the tracking point of the previous frame subsequent to the current frame. The distance to the position of the corresponding tracking point of the current frame.

In the second feature extraction step S170, using the modified nose tracking point obtained in the tracking point redetection step S167 and two eye tracking points (or using the modified nose tracking point). The region of interest is obtained, and features of the region of interest are extracted.

In the second average temperature and deviation calculation step (S175), the temperature in the region of interest of the current frame obtained in the second feature extraction step (S170) is averaged to obtain the 'interest region average temperature for each frame', and the region of interest to the current frame. The average temperature is used to obtain the average region of interest of the total frame, and the standard deviation of the temperatures in the region of interest of the current frame (ie, how far the temperature in the region of interest of each frame is from the average region of interest of each frame). ) Is calculated as a 'region of interest temperature deviation for each frame', and the temperature of each feature in the region of interest of the current frame is averaged with the temperatures of the corresponding features of all frames, and is calculated as the average temperature for each feature. The temperature deviation of each feature of each frame, which is the absolute difference between the current temperature of each feature of the star and the average temperature of each feature, is obtained.

In step S180 of determining whether the second deviation reference value is exceeded, it is determined whether the ROI deviation is greater than the deviation reference value by comparing the deviation of the ROI temperature obtained in the second average temperature and the deviation calculation step S175 with the deviation reference value. If the region of interest temperature deviation is greater than the deviation reference value, the features are erroneously detected, and further tracking (tracking) is impossible, thereby terminating the tracking.

Here, if the ROI temperature deviation is less than or equal to the deviation reference value, if the ROI temperature detected in the current frame exists between the ROI mean temperature and the deviation reference value, the features are properly detected, and tracking (tracking) is performed. Impossible elements are gone.

In the tracking end determination step (S185), if the end flag is set as the end switch is pressed, or if the current facial thermography frame corresponds to the last facial thermography frame at a predetermined time interval, the tracking is performed. If not, go to the thermal image loading step (S120) to receive a face thermal image of the next frame.

In the primary feature extraction step (S150), the tracking point redetection step (S167), and the secondary feature extraction step (S170), when selecting (extraction) tracking points and features required for tracking, It is desirable to select (extract) tracking points and features having high reproducibility, number, position, and scale invariance. It can select (extract) tracking points and features in various ways such as mineigen, Harris, SURF, BRISK, MSER, FAST, etc. from the location to track KLT. The extracted tracking points and features are compared to the tracking points and features extracted in the next frame, and the portion changed between the frames and the degree thereof are calculated. At this time, according to the state of the subject, it is possible to specify the maximum value that can be changed, and to set the tolerance. Image Pyramid is used for larger change detection, and Bidirectional Error is verified to lower the error rate that appears when the feature disappears. Using the calculated result, the transform matrix is extracted, the face thermal image is warp using the extracted matrix, and the ROI is extracted.

In the present invention, the subject's face thermal image received from the thermal imaging camera 100 is normalized and corrected through the KLT feature tracking algorithm, and one ROI is obtained for each frame of the normalized and corrected image. The region of interest is also obtained from each frame of the original subject's face thermal image received from the thermal imaging camera 100, which is not normalized, and is stored in the memory unit 220.

Next, the breath signal extraction step S200 performed by the calculation processing unit 210 will be described.

Respiratory signal extraction step (S200) is performed using the region of interest obtained for each frame of the original subject's face thermal image received from the thermal imaging camera (100).

FIG. 5 is a flowchart for schematically describing a respiratory signal extraction step of FIG. 3.

In the frame-specific temperature shift detection step (S225), the frame of the original subject's face thermal image received in the thermal image receiving step (S10), which corresponds to the image of the region of interest for each frame obtained in the breath-related area tracking step (S100). The region of interest of each frame is read from the memory unit 220 and the region of interest of each frame of the subject's face thermal image read from the memory unit 220 is subtracted. Obtain the video. That is, from the temperatures (pixels) of the region of interest of the image of the current frame, the corresponding temperature (pixel) of the region of interest of the image of the previous frame is subtracted and detected as the temperature variation of the region of interest of the current frame.

In the noise removing step (S220), the noise point is removed using the 3D-DBSCAN in the region of interest of each frame that has undergone the frame temperature detection step (S225). In other words, the noise removing step S220 considers the temperature variation of the region of interest of the current frame output in the temperature variation detecting step S225, that is, the portion of the density of the temperature variation portion having a low density in the difference image as 3D. Remove using DBSCAN. DBSCAN is well known in Wikipedia and the like, and a detailed description thereof is omitted.

In particular, the existing DBSCAN calculates the density on the 2d Image screen, clustering, but in the present invention improved to 3D-DBSCAN, when calculating the density to obtain the information from the 2D Image and the time axis to calculate the density. DBSCAN algorithm is used to remove the noise. However, the present invention is not intended to limit the present invention, and any method may be used as long as it removes noise in time and space.

In the temperature variation reference value comparing step S230, after the noise removing step S220, when the absolute value of the temperature variation of each frame is smaller than or equal to the preset temperature variation reference value, the temperature variation is removed, and only when the temperature variation is larger than the reference value. Is stored as the temperature variation of each frame. In this case, the temperature variation reference value may be a minimum temperature change amount (temperature sensitivity, NETD) given according to the camera. That is, the minimum temperature change amount is a value set according to the camera and may be set at the factory or set by the user at the beginning of use, which is stored in the memory unit 220.

에 의해 구하여진다. 이렇게 구하여진 온도변화량의 합들을 프레임에 따라, 다시말해 시간에 따라 순차적으로 그래프상에 나타낼 수 있으며, 본 발명에서는 이를 온도 변화량 그래프라 설명의 편의상 정의한다. In the temperature change amount calculating step (S235), after the temperature change reference value comparing step (S230), the temperature change is added for each stored frame, and this is calculated as the sum of the temperature change amounts of each frame, and the sum of the temperature change amounts of each frame is calculated by the memory unit 220. In order). If a frame has a resolution of 320x240, the sum of the changes in temperature

Obtained by The sums of the temperature change amounts thus obtained may be sequentially displayed on the graph according to the frame, that is, over time. In the present invention, this is defined for convenience of description as a temperature change graph.

In the respiratory data detection step (S250), the calculation processing unit 210 reads the sum of the temperature change amounts for each frame obtained in the temperature change calculation step (S235) from the memory unit 220 in a frame order, that is, in a time order, and the temperature for each frame. The changes are integrated over time (ie, on the t axis) to obtain the raw data.

Here, the number of data to be read is determined by the number of preset data or a predetermined time interval, which may be determined at the time of failure or may be determined by a user at the beginning of use of the system.

That is, since the temperature change amount (more precisely, the sum of the temperature change amount per frame) is proportional to the breathing, when the temperature change amount is integrated over time (that is, in the t-axis), the respiration waveform can be obtained. Respiratory data (raw data).

In the smoothing and detrending step (S255), after smoothing the breathing data (raw data) detected in the breathing data detecting step (S260), detrending is performed.

Smoothing can be performed using a moving average filter or a Gaussian filter. This redistributes the distribution of the light and shade, so that the light and dark become constant, and the black and white part becomes clear.

In general, detrend is a method that removes the average or linear trend from the data to clearly reveal the fluctuation. There is a method such as DFA (Detrended Fluctuation Analysis), and a computer language (for example, c language) is "detrend." Command for ".

In the smoothing and detrending step S255, continuous wavelet transform (CWT) is performed on the respiratory data that have been smoothed and detrended to represent frequency-specific intensities, that is, converted into values on the frequency domain.

In the continuous wavelet transform step (S260), the continuous wavelet transform (CWT) is performed on the respiration data obtained by smoothing and detrending in the smoothing and detrending step (S270) to represent scales (strength and magnitude) for each frequency, that is, frequency Convert to values on the domain.

In the respiratory frequency detection step (S265), the result of performing the continuous wavelet transform in the continuous wavelet transform step (S260), obtains the sum of the scales (intensity, magnitude) for each frequency, and the sum of the scales (intensity, magnitude) for each frequency is the most. The large frequency (ie, the maximum frequency) is referred to as the breathing frequency (more specifically, the frequency most relevant to breathing), and each scale of the breathing frequency is stored in the memory unit 220 as a breathing signal.

In the respiratory waveform output step (S270), the respiratory signal obtained in the respiratory frequency detection step (S265) is output as a graph.

Next, a description will be given of the respiratory rate calculation step (S300) performed by the calculation processing unit 210. That is, the process of calculating the respiratory rate using the respiratory signal obtained in the respiratory frequency detection step S265 of the respiratory signal extraction step S200 will be described.

FIG. 6 is a flowchart for schematically describing a respiratory rate calculation step of FIG. 3.

In the peak and valley detection step (S310), the respiration signal obtained in the respiratory frequency detection step (S265) is first differentiated to obtain an inflection point, and the peak and the valley are detected therefrom.

In the respiratory rate detection step S320, in the peaks and valleys detected in the peak and valley detection step S310, for a predetermined time interval (per second, or per minute or 30 seconds), the peak The number of pairs of sequential valleys successively, ie, the number of times the peak changes from valley to valley, is detected as the respiratory rate.

In the respiratory rate detection step (S330), the respiratory signal obtained in the respiratory frequency detection step (S265), the respiratory cycle is detected, and the respiratory signal of each respiratory cycle is subtracted from the respiratory signal of each subsequent respiratory cycle for each respiratory cycle. Find the amount of change and add the amount of change for each breathing cycle over time to get the amount of breath.

Here, the respiratory volume detection step (S330) is the second method of the respiratory volume detection method described above, which may also be performed by the first method of the respiratory volume detection method described above.

FIG. 7 is an example of a result of performing the temperature variation detection step S225 and the temperature variation reference value comparison step S230 for each feature in the respiratory signal extraction step S200 in FIG. 5. That is, FIG. 7 is an example of the output of the previous step of the temperature change amount calculation step (S235) of the breath signal extraction step (S200) of FIG.

FIG. 7A is a thermal image of the previous frame t0 subsequent to the current frame t1, FIG. 7B is a thermal image of the current frame t1, and FIG. 7C is a current frame (t1) is a difference image of the thermal image of the previous frame (t0).

FIG. 7D illustrates a case where the absolute value of the difference image of the thermal image of the previous frame t0 subsequent to the current frame t1 is removed when it is smaller than or equal to the temperature variation reference value. At this time, the temperature variation reference value may be 0,05.

FIG. 7E illustrates position information in the thermal image of FIG. 7D.

8 is an example of a result of removing noise by using the 3D-DBSCAN in the noise removing step (S220) of the breath signal extraction step (S200).

In the present invention, the DBSCAN algorithm is used to remove the noise. However, the present invention is not intended to limit the present invention, and any algorithm that removes noise in time and space may be used. Also, among various clustering methods, DBSCAN can express arbitrary shape, so it is better to cluster various forms than K-means and K-medoids clustering. In FIG. 8, black is a point determined as noise, and red is a clustered point.

In general, DBSCAN clusters by the density of dots. That is, clustering is less where there is no change in temperature, and clustering is good where there is change in temperature. In this clustering process, the clustering condition removes noise by considering the density within the same time and the density along the time axis in three dimensions.

9 is an example of the result of the temperature change amount calculating step (S235).

9A illustrates a temperature variation of one frame as a result of the temperature variation detection step S225 for each feature.

FIG. 9B is a graph showing the temperature change amount for each frame as a result of the temperature change amount calculation step S235. In FIG. 9B, the x-axis represents frames over time, and as a result, the x-axis represents time, and the y-axis represents the magnitude of the temperature change amount.

10 is an example of the results of the respiratory data detection step (S250).

10 (a) is an example of the result of the temperature change amount calculation step (S235), and shows a graph of the temperature change amount by frame, and FIG. 10 (b) is an example of the result of the respiration data detection step (S250). Respiration data (raw data) is obtained by integrating the temperature change amounts of each frame in FIG. 10 (a) with time (that is, in the t-axis).

In FIG. 10A, the x-axis represents frames over time, and as a result, the x-axis represents time, and the y-axis represents the magnitude of the temperature change amount. In FIG. 10B, the x axis represents a frame according to time, that is, time, and the y axis represents an integrated value of the temperature change amount.

11 is an example of the result of the continuous wavelet transform step (S260).

FIG. 11B is an example of the results of the respiratory data detection step S250 and is an example of the respiratory data obtained by integrating the temperature change amounts for each frame with time (that is, in the t-axis).

FIG. 11A illustrates continuous wavelet transform (CWT) of the respiration data of FIG. 11B to show intensity for each frequency.

12 is a view for explaining the respiratory frequency detection step (S265).

FIG. 12 shows the results of continuous wavelet transform in the continuous wavelet transform step (S260), and among them, the frequency band having the largest sum of intensities for each frequency (indicated by the solid line with arrows in FIG. 12) is detected as the respiratory frequency band. . That is, in the respiratory frequency detection step (S265), all the frames are used to extract the scale band in which the sum of the intensity is maximum, which is the frequency extraction that is dominant.

13 is an example of the results of the peak and valley detection step (S310).

Figure 13 (a) is an example of the results of the respiratory frequency detection step (S265).

FIG. 13B is an example of a result of detecting a peak and a valley from the respiration signal of FIG. 13A. In FIG. 13B, a part with a peak is displayed in red, and a part with a valley is displayed in blue.

14 illustrates correcting the angle in the thermal image rod and angle correcting step (S120).

14 (a) shows the angle of the nose designated by the user in the initial nose angle setting step (S115),

FIG. 14B shows that the output of the normalization step S130 is corrected to be equal to the initial nose angle of FIG. 14A in the angle correction step S135.

FIG. 14C illustrates a region of interest by correcting a nose angle as illustrated in FIG. 14B, and an interest in a frame of an original thermal image input in a thermal image receiving step corresponding to the detected region of interest. Indicates that the area is output.

15 illustrates an example of extracting a feature in the first feature extraction step (S150). That is, it extracts a number of features to be used for tracking in the region of interest, i.

The present invention as described above, although been described and specific examples, the invention is not limited to the embodiments described above, which those skilled in the art to which the invention pertains many modifications to the described and Modifications are possible. Therefore, the spirit of the present invention should be grasped only by the claims set out below, and all equivalent or equivalent modifications thereof will belong to the scope of the present invention.

50: test subject 100: thermal imaging camera
200: image analysis unit 210: arithmetic processing unit
220: memory unit 230: output unit
250: key input unit

Claims (19)

In a frame of a thermal image of a face, which is received from a thermal camera and each pixel represents a temperature, a calculation processing unit detects a region of interest in the nose portion using a KLT feature tracker with the nose and two eyes as tracking points. Respiratory related area tracking step;
After the breathing-related area tracking step, the operation processor obtains a difference image obtained by subtracting the image of the ROI of each frame from the image of the ROI of the next frame in the thermal image of the face received from the thermal imaging camera. The value of each pixel of the difference image is the value of the temperature variation, and the temperature variation value is added to each frame to be sequentially stored in the memory unit as the sum of the temperature change amount per frame, and the temperature for each frame of all the frames stored for a predetermined time. A respiratory signal extraction step of reading the sum of the change amounts from the memory unit and integrating according to time to respiratory related data, and detecting respiratory signals by applying continuous wavelet transform to respiratory related data;
Includes, the breathing related area tracking step,
The calculation processing unit obtains a minimum temperature value and a maximum temperature value in each frame of the subject's face thermal image received from the thermal imaging camera, and the temperature of each pixel of each frame is divided into 256 equal parts between the minimum temperature value and the maximum temperature value. A normalization step of performing normalization to have a value;
An angle correction step of correcting, by the calculation processing unit, a nose of each frame normalized in the normalization step to be equal to a preset initial nose angle;
The arithmetic unit sets candidates of the region of interest using the tracking points of the nose and two eyes in each frame whose angles are corrected in the angle correction step, extracts feature points from the candidates of the region of interest, and extracts the extracted feature points. 1, which represents a moving distance between the feature point and the corresponding feature point in a subsequent previous frame in a matrix, corrects an image of the current frame according to the extracted matrix, and obtains a region of interest in the corrected current frame. Tea feature extraction step;
Respiratory measurement method using a thermal imaging camera, comprising a. The method of claim 1,
After the breathing signal extraction step, the calculation unit, detecting the peaks and valleys in the breathing signal and using the detected peaks and valleys to detect the respiratory rate and respiratory volume, respiratory rate and respiratory rate detection step;
Respiratory measurement method using a thermal imaging camera, characterized in that it further comprises. delete According to claim 2, Respiratory-related area tracking step,
The arithmetic processing unit calculates the standard deviation of temperatures in the region of interest of each frame by averaging the temperatures in the region of interest of each frame obtained in the primary feature extraction step and calculating the standard deviation of the temperatures in the region of interest of each frame Calculating a mean temperature and deviation of the region of interest;
The calculation processing unit compares the ROI deviation obtained in the average temperature and deviation calculation step with a preset deviation reference value, and determines whether the ROI deviation is greater than the deviation reference value, and if the ROI deviation is greater than the deviation reference value, Determining whether a primary deviation criterion is exceeded by removing features in the region of interest and going to a re-detection of a tracking point;
After the step of determining whether the primary deviation reference value is exceeded, the operation processor compares the number of feature points obtained in the ROI with the preset feature number reference value, and if the number of feature points is smaller than the feature number reference value, removes the features in the ROI. Determining whether the feature number is less than the threshold value, and going to the tracking point redetection step;
If the deviation of the region of interest is greater than the deviation criterion in the step of determining whether the primary deviation criterion is exceeded, or the number of feature points is less than the feature number criterion in the determination of whether the feature deviation is less than the criterion, the wrong tracking points are determined. Determine the tracking point movement distance, which is the distance traveled from each position of the two eye tracking points of the previous frame following the current frame to each position of the two eye tracking points of the current frame, and the two tracking points A tracking point redetection step of obtaining a tracking point moving distance average, which is an average of moving distances, and a tracking point of a nose of a previous frame and a modified tracking point of a nose using a tracking point moving distance average;
Respiratory measurement method using a thermal imaging camera, comprising a. According to claim 4, Respiratory-related area tracking step,
A second feature extracting step of calculating a region of interest using the modified nose tracking point obtained in the tracking point redetection step and two eye tracking points, and extracting a feature;
The arithmetic processing unit calculates the standard deviation of temperatures in the ROI of each frame by averaging the temperatures in the ROIs of each frame obtained in the second feature extraction step and calculating the standard deviation of the temperatures in the ROIs of each frame. Calculating a second average temperature and a deviation as a region of interest temperature deviation;
The calculation processing unit compares the ROI temperature deviation obtained in the second average temperature and the deviation calculation step with the deviation reference value to determine whether the ROI temperature deviation is greater than the deviation reference value, and if the ROI temperature deviation is greater than the deviation reference value Determining whether or not the second deviation threshold value is exceeded, by ending tracking (tracking);
If the end flag is set as the end switch is pressed, or if the current frame corresponds to the last frame of the predetermined predetermined time interval, the processing unit ends tracking, otherwise, the face thermal image of the next frame. Receiving a frame and going to a normalization step, determining whether to finish tracking;
Respiratory measurement method using a thermal imaging camera, comprising a. The method of claim 2, wherein the respiratory signal extraction step,
The calculation processing unit obtains a difference image obtained by subtracting an image of the ROI of each frame from the image of the ROI of each frame in the thermal image of the face received from the IR camera, and calculating a value of each pixel of the difference image. A temperature variation detection step by frame, wherein the temperature variation is a value;
A noise removing step of the arithmetic processing unit to remove a noise point by using the 3D-DBSCAN in a region of interest of each frame of the difference image which has undergone the step of detecting the temperature variation per frame;
The step of comparing the temperature variation reference value, wherein the processing unit removes the value of the temperature variation when the absolute value of the temperature variation value in each frame of the difference image which has undergone the noise removing step is smaller than or equal to the preset temperature variation reference value. ;
A temperature change calculation step of the calculation processing unit summing each temperature change in each frame of the difference image after the temperature change reference value comparison step, calculating the amount of temperature change for each frame, and storing the result in the frame order;
Respiratory measurement method using a thermal imaging camera, comprising a. The method of claim 6, wherein the respiratory signal extraction step,
The calculation processing unit sequentially reads the sums of temperature changes per frame obtained in the temperature variation calculation step from the memory unit, integrates the temperature changes per frame over time (in the t-axis), and converts the integration result into raw data. Respiratory data detection;
A smoothing and detrending step, wherein the computing unit performs smoothing and detrending of the respiratory data detected in the respiratory data detection step;
A continuous wavelet transform step of performing, by the processing unit, continuous wavelet transform (CWT) on the respiratory data on which smoothing and detrending is performed, and displaying frequency scales for each frequency;
The calculation processing unit obtains the sum of the frequency scales for each frequency based on the result of the continuous wavelet transformation, and among the obtained sums of frequency scales for the frequencies, the respiratory frequency is the frequency at which the sum of the frequency scales for each frequency is the largest. Respiratory frequency detection step of storing each frequency scale of the memory unit as a respiration signal;
Respiratory measurement method using a thermal imaging camera, comprising a. According to claim 2, Respiratory rate and respiratory rate detection step
A peak and valley detection step of calculating an inflection point by first-differentiating the respiration signal obtained in the respiration signal extraction step, and detecting a peak and a valley using the inflection point;
In the peaks and valleys detected in the peak and valley detection step, the calculation processing unit calculates the number of pairs of peaks and valleys that appear sequentially after the peaks and valleys during a predetermined time interval as the respiratory rate. Detecting, respiratory rate detection step;
Respiratory measurement method using a thermal imaging camera, comprising a. According to claim 8, Respiratory rate and respiratory rate detection step,
A respiratory rate detection step of detecting, by the calculation processing unit, the amplitude of the pair of peaks and valleys that appear sequentially after the peaks and valleys as the respiratory volume of the respiratory cycle including the pair of peaks and valleys;
Respiratory measurement method using a thermal imaging camera, comprising a. According to claim 8, Respiratory rate and respiratory rate detection step,
Detecting the respiratory cycle with pairs of peaks and valleys that appear in succession of peaks and valleys, and subtracting the respiratory signals of each respiratory cycle from the respiratory signals of all subsequent respiratory cycles to determine the change in each respiratory cycle. Respiratory volume detection step of calculating the respiratory volume by summing the amount of change over time;
Respiratory measurement method using a thermal imaging camera, comprising a. The method of claim 10,
When summing the change amount of each breathing cycle over time, the calculation processing unit is added to the basis of the start point of each breathing cycle, characterized in that the respiration measurement method using a thermal imaging camera. The recording medium according to any one of claims 1, 2 and 4 to 11, storing a computer program source for a respiration measurement method using a thermal imaging camera of the present invention. A thermal image camera, and an image analyzer including a calculation processor configured to detect a breathing signal by analyzing a thermal image of the front face received from the thermal camera, wherein the calculation processor comprises:
In a frame of a thermal image of a face, which is received from a thermal camera and each pixel represents a temperature, a calculation processing unit detects a region of interest in the nose portion using a KLT feature tracker with the nose and two eyes as tracking points. and,
In a thermal image of a face received from a thermal imager, a difference image obtained by subtracting an image of a region of interest of each frame and an image of a region of interest of a previous frame is obtained, and the value of each pixel of the difference image is obtained by a temperature variation. It adds the temperature variation value for each frame and saves it sequentially in the memory unit as the sum of the temperature change for each frame, and reads the sum of the temperature change for each frame of all the frames stored during the preset time from the memory unit. In the respiration measurement system using a thermal imaging camera, integrating according to the respiration data and detecting the respiration signal by applying a continuous wavelet transform to the respiration data,
The operation processing unit,
In order to obtain a region of interest using a KLT feature tracker, the temperature of each pixel of each frame of the subject's face thermal image received from the thermal imaging camera is normalized to have an 8-bit value, and each frame having a normalized temperature value is obtained. The image of each frame is corrected such that the nose angle of the image is equal to the preset nose angle, the region of interest is obtained by applying a KLT feature tracking algorithm to the image of the corrected frame, and the feature points within the region of interest are detected. And a region of interest is obtained from a frame of a face thermal image received from a thermal image camera, which is an original image corresponding to the region of interest of a corrected frame. The method of claim 13,
The calculation processor detects peaks and valleys from the detected breathing signals, and detects respiratory rate and respiratory volume using the detected peaks and valleys. delete The method of claim 14, wherein the operation processing unit,
In the results of smoothing and detrending breathing data, performing continuous wavelet transform (CWT) on the smoothing and detrending breathing data, and performing continuous wavelet transform, the sum of frequency scales for each frequency It is characterized in that, the frequency of the frequency scale for each frequency of the sum of the obtained frequency scales, the maximum frequency is the respiratory frequency, characterized in that each frequency scale of the respiratory frequency is stored in the memory unit as a breathing signal, Respiration measurement system using a thermal imaging camera. The method of claim 16, wherein the operation processing unit,
The indifference point is obtained by first-order differentiation of the respiration signal, the peak and the valley are detected using the inflection point, and at the detected peak and the valley, the peak and the valley for a predetermined time interval. The number of pairs of peaks and valleys which appear sequentially afterwards is detected as the number of breaths, Respiration measurement system using a thermal imaging camera. The method of claim 17, wherein the processing unit,
Detecting the respiratory cycle with pairs of peaks and valleys that appear in succession of peaks and valleys, and subtracting the respiratory signals of each respiratory cycle from the respiratory signals of all subsequent respiratory cycles to determine the change in each respiratory cycle. Respiration measurement system using a thermal imaging camera, characterized in that the sum of the amount of change over time to obtain a respiratory volume. The method of claim 17, wherein the processing unit,
A respiration measurement system using a thermal imaging camera, characterized by detecting the amplitude of a pair of peaks and valleys that appear sequentially after the peaks and valleys as the respiratory volume of the respiratory cycle including the pairs of peaks and valleys.

Images