KR20190060243A - Respiratory measurement system using thermovision camera - Google Patents

Abstract

The present invention relates to a respiration measurement system using a thermal imaging camera, which photographs the front of a face by using a thermal imaging camera (infrared camera), detects features for each frame which is a pixel having a temperature change within an area of interest by using a Kanade-Lucas-Tomasi feature tracker (KLT point tracker) in a photographed image, obtains a temperature variation for the detected features for each frame, obtains a sum of temperature variation of each feature in a frame as a sum of temperature change amounts for each frame, integrates a sum of temperature change amounts of all frames according to time to obtain the same as respiration-related data, detects respiration signals by applying continuous wavelet transformation to the respiration-related data, and detects peaks and valleys from the respiration signals to calculate respiration volume and respiration rate. The respiration measurement system of the present invention can be used for an inspection which can replace a pulmonary function test for patients who cannot perform the pulmonary function test, and also can be used for monitoring respiration of patients undergoing a treatment under sedation. In addition, if capacity of respiratory muscle is reduced and respiration rehabilitation is required, the respiration measurement system can quantitatively measure respiration, can be used as a screening tool for diagnosis of respiratory diseases, and can be used for analyzing and monitoring a respiration pattern of a driver.

Description

[0001] The present invention relates to a respiratory measurement system using a thermographic camera,

The present invention relates to a method for detecting a temperature change in a region of interest using a KLT point tracker (Kanade-Lucas-Tomasi Feature Tracker) in a photographed image by photographing an entire face of the face using a thermal imaging camera (infrared camera) The sum of the temperature variations of the respective features in the frame is obtained as the sum of the temperature variation amounts of the respective frames and the sum of the temperature variation amounts of all the frames is set as the time To calculate respiration-related data by integrating the respiration-related data, detecting a respiration signal by applying a continuous wavelet transform to the respiration-related data, and detecting peaks and valleys in the respiration signal, To a respiration measuring system using an image camera.

The lungs take in oxygen and release carbon dioxide from the body. For the diagnosis of respiratory diseases, several imaging tests (CT, MRI, PET, perfusion scan) and blood tests (arterial blood gas test, tumor marker, general blood test) can be performed. However, for the functional evaluation of the lungs, only pulmonary function testing is currently the only tool to evaluate the functional aspect of the lung as an objective indicator.

Among them, the spirometry test is a test to measure the amount of air that can be released after the patient breathes as much as possible. Although the pulmonary function test does not diagnose a specific disease, it shows a characteristic characteristic of the disease. It helps to diagnose early, or to evaluate the progress of the clinical stage. In addition, changes in the measured values reflect changes in the severity of the disease, suggesting solutions to clinical situations in various lung diseases, and can help determine the effectiveness of treatment before and after treatment, and the reversibility of treatment. For example, pulmonary function tests may aid in the diagnosis of respiratory diseases such as asthma or chronic obstructive pulmonary disease (COPD), and may also be used as a means of assessing the health of the respiratory system . The main parameters measured here are forced vital capacity (FVC), which measures the flow discharged during effort expiration, and forced expiratory volume in one, which measures the flow discharged during the first 1 second of the FVC measurement procedure. second, FEV1). There is also a bronchodilator test to observe changes in the measured value after inhalation of the bronchodilator.

Generally, a spirometer consists of breathing or exhaling a mouthpiece at the end of a tube connected to the machine.

Healthy people have recently been increasing their interest in spirometry, and products already on the market have made the existing lung function tests portable, and the basic working principle is the same. It is used for the purpose of measuring breathing capacity and improving respiration ability through repeated training by wearing mouthpiece.

However, such a spirometer has a limitation in use. For example, if a patient is diagnosed with a respiratory-tract infectious disease, a mouthpiece can not be eaten, and co-ordination is impaired or unconscious, lung function tests can not be conducted. In addition, since the examination is usually performed while sitting, it may be difficult to perform a test for a patient who can not sit.

 Thus, a non-invasive, non-limiting, tidal volume measurement system capable of measuring volume is desired.

In general, cold air outside the nasal cavity absorbs the temperature of the nasal cavities (nostril, nostril, and nasal cavity) in the form of convection, conduction, and radiation, thereby lowering the temperature of the nasal cavity.

Korean Patent No. 10-1242755 discloses a system for measuring and monitoring vital signs of a newborn baby in an incubator in a noncontact manner. In particular, ROI (Region Of Interest) is set in the face of a newborn baby by a thermal imaging camera To a neonatal monitoring system using a thermal imaging camera that is capable of remote monitoring of the health condition of a newborn by measuring the respiration of the neonate and measuring the temperature of the neonate through thermal body extraction. In Korean Patent No. 10-1242755, the nasal cavity is extracted with the face region as a region of interest in the deteriorated state, and the respiratory rate of the newborn baby is detected through a change in the subcutaneous temperature and in the deteriorated state of the nasal cavity. However, accurate respiratory rate can not be obtained by simply estimating the number of breaths due to changes in submergence temperature. Even if it is a subcontract, depending on the position, the temperature depending on the thermal image will be different. When the subject moves, the face may be out of the screen or may be blocked. It is not easy to find the position of the subconsumer correctly. , There may be noise due to other factors, and an accurate breathing value can not be obtained by simply using the body temperature as a breathing signal.

Another prior art is disclosed in Korean Patent No. 10-2014-0057867, which is a system for measuring stress index using a thermal image. In the face region of a thermal image, a submucosa is set as a region of interest, The number of times the temperature change amount is zero for a predetermined time (for example, 5 seconds or 10 seconds) is counted, and this value is divided by time to calculate the number of respirations per second. The calculated value is multiplied by 60, do. Also in the case of Korean Patent Laid-Open No. 10-2014-0057867, accurate breathing can not be obtained by simply detecting the change in the submucous temperature as a change in the breathing volume and simply estimating the breathing rate. Even if it is a nose, the temperature according to the thermal image will be different depending on the position. Also, if the subject moves, the face may be out of the screen or blocked when the subject moves, so that it is not easy to find the position of the nose correctly , Noise due to other factors, etc., and therefore, accurate respiration values can not be obtained by simply using the body temperature as a respiration signal.

In general, when using an infrared camera, it is important to accurately measure the thermal image of the subject's nose. Since the subject moves, the face may be out of the screen or may be occluded. If the thermal image can not be measured There is a possibility that noise due to the external environment may enter.

Accordingly, the present invention is a noninvasive, non-binding, non-invasive, non-limiting method capable of measuring respiratory volume and respiratory rate even when a face is out of the screen or occluded, and can remove noise, , A tidal volume measurement system.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and it is an object of the present invention to provide a method and apparatus for photographing an entire face of a face using a thermal imaging camera (infrared camera), using a KLT feature tracker in the captured image, The sum of the temperature variations of the respective features in the frame is obtained as the sum of the temperature variation amounts of the respective frames, and the sum of the temperature variation amounts of all the frames is integrated over time, And breathing signals are detected by applying continuous wavelet transform to the breathing data, and breathing and respiration are calculated by detecting peaks and valleys in the breathing signal. System.

Another object of the present invention is to provide a method of tracking a breathing related area using a KLT feature tracker in a captured image by photographing an entire face around a nose hole using a thermal camera, The tracking points of the nose of the nose are used to infer the positions of the nose and the nose of the nose so that the respiration rate and the respiration rate can be detected by tracking the face even if the face is out of the screen or blocked. , And a respiration measurement system using an infrared camera.

Another object of the present invention is to provide a method and apparatus for detecting a feature of each frame detected using a KLT feature tracker, removing noise using each frame using a 3D-DBSCAN, In order to detect the temperature variation of the characteristics of the region, the characteristic of the interest region of the image of each frame is subtracted from the characteristic of the region of interest of the image of the succeeding previous frame in the characteristic of the image of each frame and detected as the temperature variation of the characteristic of the region of interest And calculating a temperature change amount for each frame by summing the temperature variation of each feature when the absolute value of the detected temperature variation is larger than a predetermined temperature variation reference value to obtain a frame temperature change amount.

In order to solve the above problem, in a breathing measurement method using a thermal imaging camera of the present invention, in a frame of a thermal image of a face received from a thermal imaging camera and each pixel representing temperature, Using the KLT feature tracker as a tracking point to detect a region of interest of the nose portion; After the respiration-related area tracking step, the arithmetic processing unit obtains a difference image obtained by subtracting the image of the ROI of the previous frame from the image of the ROI of each frame in the thermal image of the face received from the thermal imaging camera, The value of each pixel of the difference image is set as the value of the temperature variation, the temperature variation values for each frame are added up, and are sequentially stored in the memory as the sum of the temperature variation for each frame. And a respiration signal extracting step of extracting a sum of the change amounts from the memory unit, integrating the sum with time to make breathing-related data, and applying a continuous wavelet transform to breathing-related data to detect a breathing signal.

After the breathing signal extraction step, the operation processing unit further includes a respiration rate and respiration rate detection step of detecting peaks and valleys in the respiration signal and detecting respiration rate and respiration rate using the detected peak and valley.

In the breathing-related area tracking step, the calculation processing unit calculates a minimum temperature value and a maximum temperature value in each frame of the subject's face thermal image received from the thermal imaging camera, and determines whether the temperature of each pixel of each frame is smaller than the minimum temperature value A normalization step of performing normalization so as to have a value obtained by dividing the maximum temperature value by 256; An arithmetic processing step of the arithmetic processing part performing correction so that the nose of each frame normalized in the normalizing step is equal to a predetermined initial nose angle; The arithmetic processing unit sets the candidate of the ROI using the nose and the two eye tracking points in each frame whose angles are corrected in the angle correction step, extracts the minutiae from the candidates of the ROI, A moving distance between the minutiae and the minutiae corresponding to the previous frame is represented by a matrix (Transform Matrix), the image of the current frame is corrected according to the extracted matrix, and the region of interest is corrected by 1 And a car feature extraction step.

In the respiration-related area tracking step, the arithmetic processing unit averages the temperatures in the ROIs of the respective frames obtained in the first-ROI extraction step to obtain an arithmetic processing unit as 'ROIs of interest per frame' A first average temperature and a deviation calculating step of obtaining a standard deviation of temperatures within a frame as a 'region-of-interest temperature difference of interest'; The calculation processing unit compares the deviation of the region of interest calculated in the average temperature and deviation calculation step with a predetermined deviation reference value to determine whether the difference region of interest is greater than the deviation reference value. If the difference region of interest is larger than the deviation reference value, Determining whether a first deviation reference value is exceeded, removing features in the region of interest and going to a tracking point re-detection step; The arithmetic processing unit compares the number of feature points found in the ROI with the predetermined number of feature reference values after the step of determining whether the first deviation reference value is exceeded and if the number of feature points is smaller than the feature number reference value, Determining whether a characteristic number is less than a reference value, to go to a tracking point re-detection step; If the number of feature points is smaller than the feature number reference value in the step of determining whether or not the region of interest difference is larger than the deviation reference value or whether the feature region is smaller than the reference value in the step of determining whether the first deviation reference value is exceeded, , A tracking point movement distance which is a distance moved from each position of the two eye tracking points of the previous frame following the current frame to each position of the two eye tracking points of the current frame is obtained, And a tracking point re-detecting step of obtaining a tracking point moving distance average, which is an average of the moving distance, and obtaining a tracking point of the corrected nose using the tracking point of the nose of the previous frame and the moving average of the tracking point .

The respiration-related area tracking step is a step of tracking the respiratory-related area. In the respiration-related area tracking step, the calculation processing part obtains the region of interest using the tracking point of the modified nose and the tracking points of the two eyes obtained in the tracking point- Car feature extraction step; The arithmetic processing unit averages the temperatures in the interest region of each frame obtained in the second feature extraction step to obtain the arithmetic processing unit as the 'average region of interest per frame', and calculates the standard deviation of the temperatures in the interest region of each frame as' A second average temperature and a deviation calculating step of calculating a second average temperature and a deviation of a region of interest; The arithmetic processing unit compares the temperature difference of the region of interest calculated in the secondary average temperature and deviation calculating step with the deviation reference value to determine whether the temperature of the region of interest is greater than the deviation reference value and if the temperature of the region of interest is greater than the deviation reference value Determining whether a secondary deviation reference value is exceeded to end tracking (tracking); The arithmetic processing unit ends the tracing if the end flag is set as the end switch is pressed or if the current frame corresponds to the last frame of the initially set predetermined time interval and if not, And a normalization step of determining whether or not to end the tracking.

The breathing signal extracting step includes a calculation processing step of obtaining a difference image obtained by subtracting the image of the ROI of the previous frame of the image of the ROI of each frame from the thermal image of the face received from the thermal camera, A frame-by-frame temperature variation detecting step of setting a value of each pixel of the image as a value of a temperature variation; A noise removing step of removing a noise point using a 3D-DBSCAN in a region of interest of each frame of the difference image through the frame-by-frame temperature shift detection step; And the arithmetic processing unit removes the value of the temperature variation when the absolute value of the temperature variation value in each frame of the difference image through the noise elimination step is smaller than or equal to a preset reference value of the temperature variation, ; And the arithmetic processing unit includes a temperature change amount calculating step of summing the respective temperature variations within each frame of the difference image after the temperature variation reference value comparing step to obtain the temperature variation amount per frame and storing the obtained temperature variation amount in accordance with the frame order.

In the breathing signal extracting step, the sum of the temperature change amounts for each frame obtained in the calculating step of the temperature change processing section is sequentially read from the memory section, the temperature change amounts for each frame are integrated over time (on the t axis) A step of detecting breathing data as raw data; A smoothing and dedisting step of smoothing and detrending the raw data detected in the breathing data detection step; A continuous wavelet transform step of performing a continuous wavelet transform (CWT) on the breathing data subjected to the smoothing and detouring, and by frequency scales according to frequency; The arithmetic processing unit obtains the sum of the frequency scales for each frequency on the basis of the result of performing the continuous wavelet transform and sets the frequency having the largest sum of the frequency scales on the frequency among the sum of the obtained frequency scales as the respiration frequency, And a respiratory frequency detecting step of storing each frequency scale of the respiration signal in the memory unit as a respiration signal.

The breathing and breathing amount detecting step is a step of detecting a peak and a valley by using an inflection point to obtain an inflection point by first differentiating a breathing signal obtained in the breathing signal extracting step, Detecting a number of pairs of peaks and valleys appearing sequentially in succession in peaks and valleys in a predetermined time interval in peaks and valleys detected in the peak and valley detection step; And a breathing detection step of detecting as breathing water,

In the breathing water and respiratory volume detecting step, the arithmetic processing unit detects the amplitude of a pair of peaks and valleys in which peaks and valleys are successively consecutively consecutively detected as the respiration amount of the breathing cycle including the pair of peaks and valleys, The respiratory cycle of each respiratory cycle is subtracted from the respiratory signal of the previous respiratory cycle to obtain the amount of change per respiratory cycle and the amount of change per respiratory cycle And a tidal volume detecting step of summing the tidal volume with respect to time to obtain the tidal volume as a breath volume.

When the changes in the respiratory cycles are summed over time, they are added based on the starting point of each respiratory cycle.

The present invention further provides a recording medium storing a computer program source for a breath measurement method using the thermal imaging camera of the present invention.

The present invention also provides a respiration measurement system using a thermal imaging camera, including a thermal imaging camera, and an image analysis unit including an arithmetic processing unit for analyzing a thermal image of the entire face received from the thermal imaging camera and detecting a breathing signal , The arithmetic processing unit uses the nose and the two eyes as the tracking points in the frame of the thermal image of the face received from the thermal imaging camera and each pixel indicates the temperature, and uses the KLT feature tracker, A difference image obtained by subtracting an image of a region of interest of a previous frame and an image of a region of interest of each frame in a thermal image of a face received from a thermal imaging camera; The value of the pixel is taken as the value of the temperature variation, and the temperature variation values for each frame are added up and stored sequentially in the memory as the sum of the temperature variation for each frame , The sum of the temperature variation of each frame of every frame stored for a predetermined time is read from the memory unit and is integrated according to time as breathing data and a breathing signal is detected by applying a continuous wavelet transform to the breathing data .

 The arithmetic processing unit detects peaks and valleys in the detected respiration signal, and detects respiration and respiration using the detected peak and valley.

In order to obtain the region of interest using the KLT feature tracker, the operation processing unit performs normalization so that the temperatures of the respective pixels of each frame of the subject face thermal image received from the thermal imaging camera have 8 bit values, and the normalized temperature value The image of each frame is corrected so that the angle of the nose of the image of each frame having the predetermined angle is equal to the predetermined angle of the nose and the KLT feature tracking algorithm is applied to the image of the corrected frame to obtain the region of interest, The feature point is detected and the region of interest is also found in the frame of the face image received from the thermal camera, which is the original image corresponding to the region of interest of the corrected frame.

The arithmetic processing unit performs smoothing and detrending of respiration data, performs continuous wavelet transform (CWT) on breathing data subjected to smoothing and detouring, and performs continuous wavelet transform on the basis of frequency The sum of the frequency scales is obtained. The frequency having the largest sum of the frequency scales according to the frequency among the sum of the obtained frequency scales is stored as a breathing frequency, and each frequency scale of the breathing frequency is stored in the memory as a breathing signal.

The arithmetic processing unit performs a first differentiation of the respiratory signal to obtain an inflection point, uses the inflection point to detect peaks and valleys, and detects, in the detected peaks and valleys, , The number of pairs of peaks and valleys in which the peaks and the valleys sequentially appear successively are detected as respiration numbers.

The arithmetic processing unit detects a respiratory cycle in pairs of peaks and valleys in which peaks and valleys appear in succession, and subtracts a respiration signal of each respiratory cycle from a respiration signal of successive respiratory cycles to calculate a change amount per respiratory cycle Or the amplitude of a pair of peaks and valleys in which peaks and valleys appear in series are detected as the respiratory volume of the respiratory cycle including the pair of peaks and valleys.

The breathing measurement system using the thermal imaging camera of the present invention photographs the entire face of the face using a thermal imaging camera (infrared camera), and uses the KLT feature tracker in the captured image to generate a breathing image The temperature variations of the detected features are obtained, the sum of the temperature variations of the respective features in the frame is obtained as the temperature variation per frame, the temperature variation of all the frames is integrated over time, And the respiration-related data is applied to the continuous wavelet transform to detect the respiration signal, and the peak and the valley in the respiration signal are detected to calculate the respiratory volume and respiratory rate, Can be detected.

The present invention also provides a method of tracking a respiratory-related area using a KLT feature tracker in a captured image, photographing an entire face around a nostril using a thermal camera, 3 tracking points are used to infer the positions of each other, and even when the face is out of the screen or covered, tracking can be continuously performed to detect the amount of respiration and the number of breaths.

In the present invention, noise is removed by using 3D-DBSCAN in the portion extracted through tracking, raw data is extracted from the noise-removed portion, and frequency data is extracted from the extracted data using a CWT The respiration region can be extracted more accurately by extracting only the portion related to respiration in the domain.

The respiration measurement system using the thermal imaging camera of the present invention can enable the patient's respiratory pattern, one-time breathing volume, effort spirometry, comparison of effect of post-drug treatment and respiratory rehabilitation treatment by non-invasive method.

First, it can be used as a surrogate for pulmonary function tests in patients who can not perform pulmonary function tests. Non-contact, non-invasive, relatively inexpensive, non-invasive, non-invasive, non-invasive treatment for patients who were unable to undergo pulmonary function testing, such as patients with bronchial incisions that can not tolerate mouthpieces, May replace pulmonary function tests.

Second, it can be used for respiratory monitoring for patients under sedation. In gastrointestinal endoscopy, image examinations such as MRI, which requires imaging for more than 30 minutes, the frequency of administering sedation is increasing in the medical field in recent years, for the purpose of patient comfort and ease of procedure. And arterial oxygen saturation in order to monitor respiratory depression due to sedation, but there are limitations in monitoring because of its accuracy and rapidity. The present invention can non-invasively monitor real-time breathing, which can be helpful in terms of patient safety.

Third, when respiratory rehabilitation is needed in the medical field due to decreased ability of the respiratory muscles, quantitative measurement of respiration may be helpful. In particular, the present invention can be applied to a rehabilitation exercise device and a mobile device to enhance the effect of exercise through biofeedback during respiratory rehabilitation therapy.

Fourth, there is a possibility to use it as a screening tool for diagnosis of respiratory diseases. Currently, there are no tests that can screen for respiratory diseases except for consultation, counseling, and chest X-ray. The algorithm of this technique can be analyzed and used to screen patients at the pre-diagnosis stage.

In addition, the respiration measurement system using the thermal imaging camera of the present invention can be utilized for the development of a health device for a general public in addition to a medical environment. After a recent bus accident that caused a large-scale casualty, the drowsiness driving device is receiving attention again. With the present invention, it is also possible to analyze the respiratory pattern of the driver and utilize it as a monitoring purpose.

In daily life weather forecasting, living in an environment that announces the fine dust index, modern people are concerned not only for respiratory diseases but also for improving breathing ability. And a band such as an American band is expanding the bottom in the health market. Therefore, when the respiration measurement system using the thermal imaging camera of the present invention is interlocked, it can be used to measure the post-exercise spirometric capacity.

1 is a conceptual diagram of a respiration measurement system using a thermal imaging camera of the present invention.
2 is a block diagram schematically showing a configuration of a respiration measurement system using a thermal imaging camera of the present invention.
3 is a flowchart schematically illustrating a method of detecting the amount of breath using thermal image data received from a thermal imaging camera in the image analysis unit 200 of FIG.
4A is a flow chart for schematically explaining a KLT feature tracking algorithm of an embodiment of the present invention.
FIG. 4B is a flowchart illustrating the primary feature extraction step (S150) of FIG. 4A.
FIG. 4C is an explanatory diagram illustrating a process of extracting a region of interest from feature extraction in FIG. 4B.
5 is a flowchart schematically illustrating the respiration signal extracting step of FIG.
FIG. 6 is a flowchart for schematically explaining the breathing amount calculation step of FIG. 3;
FIG. 7 is an example of the result of performing the step of detecting the characteristic temperature difference and the step of comparing the temperature variation reference value in the breathing signal extracting step in FIG.
FIG. 8 is an example of a result of removing noise using the 3D-DBSCAN in the noise removing step of the breathing signal extracting step (S200).
9 is an example of a result of the temperature change amount calculating step.
10 is an example of the result of the breathing data detection step.
11 is an example of the result of the continuous wavelet transform step.
12 is a view for explaining the breathing frequency detecting step.
13 is an example of the result of the peak and valley detection step.
14 shows correction of the angle in the thermal image load and angle correction step.
FIG. 15 shows an example in which features are extracted in the primary feature extraction step.
16 is an explanatory diagram for explaining an average temperature and a deviation of a region of interest for each frame in the present invention.

Hereinafter, a respiration measurement system using a thermal imaging camera according to the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a conceptual diagram of a breathing measurement system using a thermal imaging camera of the present invention, and FIG. 2 is a block diagram schematically showing a configuration of a breathing measurement system using a thermal imaging camera of the present invention.

(Infrared camera) 100, converts the face of the subject 50 into a digital signal, and transmits the digital signal to the image analysis unit 200. [ At this time, the radiographic camera 100 converts the photographed image into a digital signal and transmits the digital signal to the image analysis unit 200. Alternatively, the photographed image of the thermal imaging camera 100 is transmitted through an A / D converter (not shown) Converted into a digital signal, and transmitted to the image analysis unit 200.

The thermal imaging camera 100 may use an infrared camera. The thermal imaging camera 100 photographs the face of the examinee so that the entire face of the examinee appears. The photographing can acquire data at 10 times per second (10 Hz), and thermal image data represented by absolute temperature is obtained.

Generally, the body temperature changes according to the breathing of the human body. This is because the cold air moves on the basis of the human body during intake, lowering the temperature (that is, the temperature of the body part related to breathing) . Accordingly, the present invention takes a photograph of the entire face of a subject, and photographs the nastril portion through which air passes substantially.

Generally, in the case of an object having a temperature, an infrared ray always emits. As the temperature of the emitted infrared ray increases, the wavelength becomes shorter, the frequency increases, and the energy becomes higher. And the temperature is converted.

FLIR's T420 can be used as an infrared camera, and only the part with a temperature change over NETD (noise equivalent temperature difference) is extracted. It has a resolution of 320x240, a NETD of 0.05 and a minimum focal length of 50 cm , The subject can be measured at a distance of approximately 50 to 60 cm. However, the present invention is not limited to this, and any thermal imaging camera may be used as long as it detects a thermal image on the face of the subject 50. Generally, NETD differs from camera specification to 0.05, which is set at 0.05, but this is not intended to limit the present invention.

The image analysis unit 200 may be a general computer, a notebook computer, a microprocessor, or the like as a means for analyzing an image received from the thermal imaging camera 100 and detecting a respiration signal. The image analysis unit 200 includes an operation processing unit 210, a memory unit 220, an output unit 230, and a key input unit 250.

The operation processing unit 210 tracks a respiration-related area using a KLT point tracker (Kanter-Lucas-Tomasi Feature Tracker) on the image received from the thermal imaging camera 100, And extracts only the part related to the respiration in the frequency domain in the extracted part, that is, the interest area, and extracts the volume waveform by applying the extracted respiration area to the calculation of the volume of breath, Calculate the volume and the number of breaths by looking for the peak and valley in the waveform.

In other words, the arithmetic processing unit 210 determines whether or not each pixel of each frame of the subject face image received from the thermal imaging camera 100, that is, an absolute temperature value (hereinafter referred to as a temperature value for convenience of explanation) , And corrects the image of each frame so that the angle of the nose of the image of each frame having the normalized temperature value is equal to the preset nose angle, KLT feature tracking algorithm is applied to obtain the region of interest. Accordingly, if the region of interest is found in the corrected image, the region of interest is found in the frame of the original image corresponding thereto, that is, the subject's face image received from the thermal imaging camera 100. That is, one region of interest is obtained for each frame of the normalized and corrected image through the KLT feature tracking algorithm. Corresponding to this, an image of the original subject face thermal image received from the thermal imaging camera 100, which is not normalized, The area of interest is also found in the frame.

The arithmetic processing unit 210 stores the regions of interest of each frame of the subject face image received from the thermal imaging camera 100 in the memory unit 220 and stores the area of interest of each frame of the subject face image stored in the memory unit 220 By subtracting the region of interest from the region of interest in successive consecutive frames, the difference image of the region of interest of each frame is obtained, and noise is removed by applying 3D-DBSCAN to the difference image of the region of interest of each frame. When the pixel value of the difference image of the interest region of each frame from which the noise is removed, that is, the temperature variation value is smaller than or equal to the predetermined temperature variation reference value, only the case where the difference value is larger than the temperature variation reference value, The temperature variation value (temperature variation data) is obtained for each frame of interest in each frame and is obtained as a temperature variation amount for each frame, and the frame-by-frame temperature variation amount is stored in the memory section 220).

The arithmetic processing unit 210 reads the stored temperature-dependent amounts of each frame from the memory unit 220 in time order, integrates the temperature-dependent amounts of the respective frames with time (i.e., t-axis), and obtains raw data .

The arithmetic processing unit 210 performs smoothing on the detected raw data and then performs detrend and applies the smoothed and detented breath data to a Continuous Wavelet Transform (Intensity and size), and the sum of the scales according to frequency (intensity sum of frequencies) is obtained from the result of continuous wavelet transform. (More specifically, the frequency most related to breathing), and the scales of the breathing frequency are stored in the memory unit 220 as respiration signals, . Here, when the respiration signal of each scale of the respiratory frequency is plotted according to time, it becomes a respiratory waveform. Therefore, this breathing waveform becomes a waveform (data) in the time domain.

The arithmetic processing unit 210 firstly differentiates the respiration signal to obtain an inflection point, detects peaks and valleys from the inflection points, and detects the peak and valley in the detected peaks and valleys for a predetermined time interval The number of successive peaks and valley pairs, that is, the number of times of changing from peak to valley, is detected as a respiration rate (RR) in a predetermined time (per second, per minute or every 30 seconds)

Tidal volume can be obtained in two ways.

The first method detects the amplitude of a pair of successive peaks and valleys in the respiration signal, that is, the amplitude magnitude of the signal that changes from peak to valley, by the amount of respiration of that period.

In the second method, the respiratory signal of each respiratory cycle is subtracted from the respiratory signal of the succeeding respiratory cycle, and the amount of change of each respiratory cycle is obtained by taking each respiratory cycle from one peak to the next peak in the respiratory waveform, The amount of change is summed to obtain the volume. In some cases, the respiratory cycle can be a time interval from one peak and valley pair to the previous peak and valley pair.

That is, the amount of change per breathing cycle is obtained by subtracting the respiration signal (each sample) of the current breathing cycle from each respiration signal (each sample) corresponding to the previous breathing cycle in succession to obtain the breathing change amount. For example, in the first breathing signal (sample) of the current breathing cycle, the first breathing signal (sample) of the previous breathing cycle is subtracted to obtain the first breathing change of the breathing cycle, To obtain the respiratory variation of each cycle. The respiration rate of each cycle is calculated by summing up the respiratory change amount in each time period. Here, the sum of the respiration change amounts of the respective periods is calculated by adding the first respiration change amount signals of the respective cycles to the first respiration amount signal, and adding the second respiration change amount signals of each cycle to the second respiration amount signal, And the respiration amount is calculated by summing the respiration change amount signals.

The memory unit 220 stores the respiration amount and the respiration amount received from the arithmetic processing unit 210.

The output unit 230 outputs the respiration amount and the respiration amount received from the operation processing unit 210.

The key input unit 250 may include a graphical user interface to set an initial tracking point, and includes a start / stop switch and the like.

3 is a flowchart schematically illustrating a method of detecting the amount of breath using thermal image data received from a thermal imaging camera in the image analysis unit 200 of FIG.

In the thermal image reception step S10, the operation processing unit 210 of the image analysis unit 200 receives the thermal face image (thermal image) of the subject from the thermal imaging camera 100, 220). In the thermal image receiving step (S10), a column image may be received for each frame, a column image may be received one frame at a time, or a column image of frames may be received for a predetermined time. For example, you can get a 320x240 Absolute Temperature matrix across the face at 10 frames per second. The data obtained from the thermal imaging camera 100 in this way can be referred to as temperature data or temperature domain data. By marking this way, we can distinguish it from the normalized data to apply the KLT feature tracker algorithm.

In the respiration-related area tracking step S100, each pixel of each frame of the subject's face thermal image (i.e., the subject's face thermal image received from the thermal camera 100) received in the thermal image receiving step S10, The image of each frame is corrected so that the temperature value (absolute temperature value) has an 8-bit value and the angle of the nose of the image of each frame having the normalized temperature value is equal to the preset nose angle, The KLT feature tracking algorithm (KLT point tracking algorithm) is used for each frame image, and two tracking points are used for two eyes and noses in each frame of the face image. The respiration-related region, that is, the nose portion or the nose portion, is extracted as a region of interest (ROI), and features are extracted for tracking. Here, the feature is a feature detected by a specific calculation method (e. G., A minimum eigenvalue method and the like), also called a KLT feature in the KLT feature tracking algorithm, which is well known and therefore will not be described in detail. The detected KLT feature mainly detects a boundary part, a corner part (e.g., a border around the nose or a corner in the face image) of the face, and the like. (Tracking) the positions of the two eyes and nose of the subject 50 as three tracking points, estimates the position of each of the three eyes using the three tracking points, (ROI) located in the region of interest, and extracts and tracks features within the region of interest.

Prior to applying the KLT feature tracking algorithm, at the beginning of the respiration-related area tracking step (S100), the position of the nose in the first face image frame of the consecutive face image sequence incoming from the thermal image receiving step (S10) The thermal image is positioned so as to be aligned with a predetermined position. In addition, the position of the nose and the eyes in the first face image frame is designated, and the size to be used as a region of interest (ROI) and the angle of the oblique face of the face image ), And sets the parameters of the other KLT feature tracking algorithm to the face image. Accordingly, the coordinates of the nose and two eyes, the angle of the nose (angle of the face), the size of the region of interest, and the like are stored in the memory unit 220. The parameters can be changed according to the accuracy required and the degree of patient motion.

Then, the image of each frame received in the thermal image receiving step (S10) is normalized so as to have an 8-bit value, and the angle of the nose of the image of each frame having the normalized temperature value is made equal to the predetermined nose angle , The image of each frame is corrected, and the ROI is obtained by applying a KLT feature tracking algorithm to each image of the corrected frame. That is, a part of the temperature change of the nose portion which substantially affects the respiration is extracted as the region of interest.

The subject 50 may have motion for breathing and other reasons and if the face of the examinee 50 moves out of the camera screen or is obscured by the movement of the subject 50, If only a part remains, the position of the tracking point which is not in the face image is estimated by using the tracking point remaining in the face image. That is, the KLT feature tracking algorithm is used to track each of the three points with the center of each eye as a point, the center of the nose as a point, and the characteristics of three position coordinates, Can not be traced, the position of the non-traceable point, that is, the coordinates, can be inferred and continuously tracked.

The breathing-related region tracking step S100 is a step for detecting a region of interest, thereby obtaining a region of interest in the image of the normalized and corrected frame through the KLT feature tracking algorithm, A region of interest is also obtained for the frame of the subject's face thermal image received in the image receiving step (S10).

In the present invention, for convenience of explanation, the image (image) normalized in the breathing-related area tracking step S100 is referred to as an image domain of the image domain, and the image received in the thermal image receiving step S10, The image (image) received from the camera 100 is referred to as an image (data) in the temperature domain.

In the breathing signal extraction step (S200), the temperature change amount per frame is obtained from the ROI of each frame of the face image detected in the respiration-related area tracking step (S100), and the respiration signal is calculated using the frame- And extracting the respiration signal. This subtracts the region of interest of each frame of the subject's face thermal image received from the thermal imaging camera 100 with the region of interest of the previous frame successively to obtain a difference image of the region of interest of each frame, The 3D-DBSCAN is applied to the difference image to remove the noise. When the pixel value of the difference image of the interest region of each frame from which the noise is removed, that is, the temperature variation value is smaller than or equal to the preset reference value, (Temperature shift data) of each frame, stores the temperature shift value (temperature shift data) of interest in each frame, and adds the temperature shift value (temperature shift data) for each frame of interest in each frame, And the temperature changes per frame obtained are integrated over time (that is, on the t axis) to obtain the raw data. After the detected raw data is smoothed, detrend is performed, and breathing data subjected to smoothing and detouring are subjected to continuous wavelet transform (CWT) (Intensity, size), and the sum of the scales on a frequency basis (intensity sum by frequency) is obtained from the result of continuous wavelet transformation. (More specifically, the frequency most related to breathing), and the scales of the breathing frequency are stored in the memory unit 220 as respiration signals, .

In the volume calculation step S300, a peak and a valley are detected in a volume waveform, and in the detected peaks and valleys, for a predetermined time interval (per second, Or per minute or 30 seconds), the number of successive peaks and valley pairs, i.e., the number of times the peak to valley is changed, is detected as the respiration rate (RR) and the volume is also determined. The amount of respiration can be obtained by two methods as described above, but here, it is determined by the second method. In other words, the respiratory signal of one respiratory cycle is subtracted from the respiratory signal of the next respiratory cycle, and the amount of change of each respiratory cycle is obtained. The sum of the breaths is obtained.

FIG. 4A is a flowchart for schematically explaining a KLT feature tracking algorithm of an embodiment of the present invention, FIG. 4B is a flowchart for explaining a first feature extraction step (S150) of FIG. 4A, FIG. And extracting the region of interest from the extracted region.

The KLT feature tracking algorithm is executed in the arithmetic processing unit 210 and included in the respiratory-related area tracking step S100.

In the initializing step (S105), the arithmetic processing unit 210 reads the deviation reference value, feature number reference value, and the like from the memory unit 220.

In the initial tracking point setting step S110, the user designates the position of the nose and the positions of the two eyes in the first frame of the subject's face image received in the thermal image receiving step (S10) through the key input unit 250, The operation processing unit 210 receives the position of the nose and the positions of the two eyes, that is, the coordinates of the nose and the coordinates of the two eyes, from the key input unit 250, and stores them in the memory unit 220 as an initial tracking point. Here, the two eyes of the examinee can be called the first eye and the second eye. At this time, the area of interest, the position, and the like can also be set.

In the initial co angle setting step S115, the user specifies the angle of the nose in the first frame of the subject face image received in the thermal image receiving step S10, and the arithmetic processing unit 210 receives the nose from the key input unit 250, And stores it in the memory unit 220 as the angle of the initial nose.

In the thermal image loading step (S120), the subject face image received in the thermal image receiving step (S10) and temporarily stored in the memory unit (220) is read frame by frame.

In the normalization step (S130), in each pixel of each frame of the subject face thermal image received in the thermal image load step (S120), that is, in the absolute temperature data, the difference between the minimum temperature (min) value and the maximum temperature Normalization is performed to image the image into 256 divided values (8 bits).

Here, the normalization method obtains the median temperature and the maximum temperature max for each frame of the face image. Then, a difference between the maximum temperature and the intermediate temperature of each frame of the face image is calculated, that is, the maximum temperature max minus the median. In other words, find difference = max - median. Next, the minimum temperature (min) of each frame of the face image is obtained, and the minimum temperature (min) is a value obtained by subtracting the difference between the maximum temperature and the intermediate temperature at the median . That is, min = median - difference is obtained. The range between the minimum temperature (min) and the maximum temperature (max) is called the range. Each pixel of each frame of the face thermal image is converted into a value divided into 256 ranges between the temperature range, i.e., the minimum temperature (min) value and the maximum temperature (max) value. For example, when the median value is 36.1 degrees and the maximum temperature value is 38.1 degrees, the difference value is 2 degrees and the minimum temperature value is 34.1 degrees. 34.1 to 38.1 degrees are divided into 256 equal parts to produce an 8-bit image.

In the angle correcting step S135, the nose of each frame of the subject's face image normalized in the normalizing step S130 is corrected to be equal to the initial nose angle set in the initial nose angle setting step S115, that is, If the nose of the face image frame is judged to be skewed, the angle of the nose of the face image frame is the same as the angle of the initial nose angle, And corrects the face image frame.

That is, the angle correcting step S135 is a step of correcting the frames of the continuous face image, which are read from the thermal image receiving step S10 and normalized in the normalizing step S130, The thermal image is positioned so that the position of the nose in the image frame is adjusted to a predetermined position.

In the first feature extraction step S150, the first eye and the second eye to be used for tracking in each frame of the subject's face thermal image, which is normalized in the normalization step S130 and the angle is corrected in the angle correction step S135, The eye and nose tracking points are extracted and candidates of the region of interest (the nose portion or the nasal cavity or the nose portion) using the three tracking points (or using the tracking points of the nose) 4C) (S153), and a feature point (a blue square portion of a blue color) of the region of interest The coordinates of the extracted feature points and the movement distance of the feature points corresponding to the feature points in the previous frame are represented by a matrix (Transform Matrix) (S155) ), The image of the current frame ( (The subject's face thermal image output from the angle correction step S135) (warp) (S156), and obtains the ROI (red square portion in FIG.

In the present invention, tracking is basically tracking the nose portion in the facial image, and in some cases, it is difficult to track the nose by the movement of the patient. In order to estimate the position of the nose, The position of both eyes is also tracked. That is, when the position of the nose can not be grasped, the position of the nose is estimated using the information of the two eyes.

A situation where it is difficult to track the nose is when the KLT can not designate the area of interest itself or specify a desired area. In this case, first, when the nose is out of the screen (out of the range of the tracking point) , Secondly, when hand and other objects pass over the face (no feature of the nose is found, and the temperature range of the nose changes, such as the average temperature and the deviation of the nose). Third, when the angle of the front face changes, (No feature found in the nose). In this case, as the prediction method, since the positions of the two eyes are not changed, the position of the nose is estimated as an average of the movement trajectories of the two eyes.

In the present invention, tracking tracking is performed for each of the two eyes (i.e., the first eye and the second eye) and the nose of the subject, which are tracking points, and the ROI is set accordingly.

KLT sets the region of interest in the image domain and extracts the feature points of the image within the region of interest. The Transform matrix is obtained by calculating how the feature points change in the next frame with the feature point information. Using this, we can obtain several slices containing the same region of interest. Here, feature extraction can be basically performed using a mininum eigen feature (minimum eigenvalue) algorithm (method), or various other feature extraction methods can be used.

In the first average temperature and deviation calculation step S155, the temperatures in the ROIs of the respective frames obtained in the first feature extraction step S150 are averaged to obtain 'ROIs of interest per frame' The average temperature of the region is averaged to obtain the 'average temperature of the region of interest of the total frame', and the standard deviation of the temperatures within the region of interest of each frame (i.e., how far the temperatures in the region of interest per frame are separated from the average region of interest per frame The temperature of each feature within the region of interest of the current frame is averaged with the temperatures of the corresponding feature of the previous frames to obtain an average temperature for each feature , The temperature deviation of each feature of each frame, which is the absolute value of the difference between the current temperature of each feature of each frame and the average temperature of each feature, is obtained.

That is, in the average temperature and deviation calculation step S155, the average temperature of the ROI for each frame, the ROI of the ROI for each frame, the ROI average temperature, the average temperature for each feature, Accordingly, some of them (for example, the average temperature of each feature, the deviation by feature, and the average temperature of the region of interest of the total frame) may be omitted.

In step S160 of determining whether or not the first deviation reference value is exceeded, it is determined whether the region of interest difference is larger than the deviation reference value by comparing the region of interest difference calculated in the average temperature and deviation calculation step S130 with a deviation reference value. If the deviation is larger than the deviation reference value, it is determined that the features are erroneously detected, the features in the RO are removed (S166), and the tracking point re-detection step S167 is performed.

Here, the deviation reference value may be a value set or stored by the user in the initial setting step, a value stored at the time of shipment from the factory, or a human body temperature change limit value or larger value.

In FIG. 16, the horizontal axis (x axis) represents each frame with time, the red line is the average temperature of the interest region per frame, the blue line indicates the average temperature of the region of interest per frame, And the green line is a graph showing that the average temperature of the region of interest for each frame minus the region-of-interest temperature deviation for each frame. In other words, if you are tracking (tracking) the same face part normally (the change in temperature due to respiration is very small and the effect is less if you take the average), the mean and deviation are maintained or do not change suddenly even if they change. However, when contaminated parts come in, the average and deviation of the temperature changes rapidly. Therefore, in the case of a change in frames 6-7 in Fig. 16, it is determined that the region of interest is erroneous. That is, it is determined that the region of interest is a non-nose portion.

In step S165, it is determined whether the number of feature points is less than a predetermined reference value. If the number of feature points is less than the reference value, If the feature point is smaller than the reference point, the feature points are erroneously detected, the features in the region of interest are removed (S166), and the tracking point re-detection step S167 is performed.

When the features in the RO are removed (S166), the ROI, the ROI, the average temperature, and the feature deviations for the current frame are also removed.

If the number of features is smaller than the feature number reference value in the step S165 of determining whether or not the primary deviation reference value is exceeded in the step S160 of determining whether the region of interest difference is larger than the deviation reference value or whether or not the feature number reference value is smaller, It is judged that the area of interest is wrongly obtained by this.

If it is determined in step S167 that the primary deviation reference value has exceeded the deviation reference value in step S160 or if the number of feature points in the determination step S165 is less than the feature number reference value (I.e., two eye tracking points), the remaining two tracking points of the previous frame subsequent to the current frame (i.e., the two eye tracking tracking points) (I.e., two eye tracking points) of the current frame from the respective positions of the two tracking points, and obtains the tracking point movement distance, which is the average of the two tracking point movement distances, The moving distance average is obtained, and the track of the nose of the previous frame and the track of the corrected nose Calculate the points again. That is, the tracking point of the modified nose is determined as the tracking point of the nose of the previous frame is shifted by the average of the movement distance of the tracking point, and the tracking point of the nose is obtained.

Here, the tracking point movement distance is from the position of the tracking point of the previous frame following the current frame. The distance to the position of the corresponding tracking point of the current frame.

In the second feature extraction step (S170), the corrected tracking point of the nose and the tracking point of the two eyes (or using the modified tracking point of the nose) obtained in the tracking point re-detection step (S167) , A region of interest is sought, and features within the region of interest are extracted.

The temperature in the ROI of the current frame obtained in the second feature extraction step S170 is calculated as a 'ROI of interest per frame' in the second average temperature and deviation calculation step S175, The temperature is averaged to obtain the 'average temperature of the region of interest of the total frame', and the standard deviation of the temperatures within the region of interest of the current frame (i.e., how far the temperatures in the region of interest per frame are separated from the average temperature of the regions of interest per frame Is obtained as an average temperature of each feature, and the temperature of each feature in the region of interest of the current frame is averaged with the temperatures of the corresponding feature of the previous frames, The temperature deviation of each feature of each frame, which is the absolute value of the difference between the present temperature of each feature of the star and the average temperature of each feature, is obtained.

In step S180, it is determined whether the region of interest difference is greater than the deviation reference value by comparing the region of interest temperature difference calculated in the secondary average temperature and deviation calculation step S175 with a deviation reference value. If the region of interest temperature difference is larger than the deviation reference value, the tracking is terminated because no further tracking (tracking) is possible because the features are detected incorrectly.

If the region of interest temperature difference is less than or equal to the deviation reference value, if the region of interest detected in the current frame exists between the average region of interest and the deviation reference value, The impossible element has disappeared.

If it is determined in step S185 that the end flag is set as the end switch is pressed or if the current face image frame corresponds to the last face image frame in the initially set predetermined time interval, Otherwise, the process goes to the step of loading the thermal image (S120) and receives the face image of the next frame.

When extracting (extracting) tracking points and features necessary for tracking in the primary feature extraction step S150, the tracking point re-detection step S167, and the secondary feature extraction step S170, It is desirable to select (extract) tracking points and features with high reproducibility, number, position, and scale invariance. It is possible to select (extract) tracking points and features by various methods such as mineigen, Harris, SURF, BRISK, MSER, FAST and the like at a position where KLT tracking is to be performed. The extracted tracking points, features are compared with tracking points, features extracted from the next frame, and the changed portions and the degree of the differences between the frames are calculated. At this time, the maximum value that can be changed can be specified according to the state of the subject, and the tolerance can be set. The image pyramid is used for larger change detection, and the bidirectional error is verified to lower the error rate that appears when the feature disappears. Using the calculated result, the transform matrix is extracted, and the image of the face is warped using the extracted matrix, and the ROI of interest is extracted.

In the present invention, the subject's face thermal image received from the thermal imaging camera 100 is normalized and corrected through the KLT feature tracking algorithm, and one region of interest is obtained for each frame of the normalized and corrected image, In the frame of the original subject face image received from the thermal imaging camera 100, which is not normalized, the region of interest is obtained and stored in the memory unit 220.

Next, the respiration signal extraction step (S200) performed in the arithmetic processing unit 210 will be described.

The breathing signal extracting step S200 is performed using the region of interest obtained for each frame of the original subject's face thermal image received from the thermal imaging camera 100. [

5 is a flowchart schematically illustrating the respiration signal extracting step of FIG.

In the frame-by-frame temperature variation detection step (S225), the frame of the original subject face image received in the thermal image reception step (S10) corresponding to the frame of interest per frame obtained in the breath tracking area tracking step (S100) The region of interest of interest is read from the memory unit 220 and the region of interest of the previous frame is subtracted from the region of interest of each frame of the subject's face image read from the memory unit 220, Obtain the image. That is, the temperature (pixels) of the ROI of the current frame is subtracted from the temperature (pixel) of the ROI of the image of the previous frame and detected as the temperature variation of the ROI in the current frame.

In the noise removal step S220, the noise region is removed using the 3D-DBSCAN for the region of interest of each frame that has been subjected to the frame-by-frame temperature variation detection step S225. In other words, the noise removing step S220 regards the portion of the temperature difference of the current frame, i.e., the difference image of the temperature variation portion of the difference image, output from the temperature variation detecting step S225 as noise, - Remove with DBSCAN. DBSCAN is widely known in Wikipedia and the like, and a detailed description thereof will be omitted.

In particular, the existing DBSCAN calculates the density on the 2d Image screen and performs clustering. In the present invention, it improves to 3D-DBSCAN, and when the density is calculated, the information is also obtained on the 2D image and the time axis to calculate the density. Although the DBSCAN algorithm is used to remove noise, the present invention is not limited to this, and any method can be used as long as it is a method of removing noise in time and space.

If the absolute value of the temperature variation of each frame after the noise removal step S220 is smaller than or equal to the predetermined temperature variation reference value is removed in step S230 of comparing the temperature variation reference value, As the temperature variation of each frame. At this time, the temperature variation reference value may be a minimum temperature change amount (temperature sensitivity, NETD) given according to the camera. That is, the minimum temperature change amount is a value set according to the camera, and may be set at the time of factory shipment, or may be set by the user at the beginning of use, which is stored in the memory unit 220.

에 의해 구하여진다. 이렇게 구하여진 온도변화량의 합들을 프레임에 따라, 다시말해 시간에 따라 순차적으로 그래프상에 나타낼 수 있으며, 본 발명에서는 이를 온도 변화량 그래프라 설명의 편의상 정의한다. After the temperature variation reference value comparison step S230, the temperature variation is calculated for each stored frame by the sum of the temperature variation amounts for each frame, and the sum of the temperature variation amounts for each frame is stored in the memory 220 ). If one frame has a resolution of 320x240, then the sum of the temperature variations is

. The sums of temperature variations obtained in this manner can be displayed on the graph sequentially in accordance with the frame, that is, according to time. In the present invention, this graph is referred to as a temperature variation graph for convenience of explanation.

In the respiration data detection step S250, the arithmetic processing unit 210 reads the frame-by-frame temperature change amount sums obtained in the temperature change amount calculation step S235 from the memory unit 220 according to the frame order, The changes are integrated over time (i.e., on the t-axis) to obtain the raw data.

Here, the number of data to be read is determined by the number of predetermined data or a predetermined time interval, which may be set at the time of a faulty shipment, or may be set by the user at the beginning of use of the system.

That is, since the temperature change amount (more precisely, the sum of the temperature change amounts per frame) is proportional to the respiration, the respiratory waveform can be obtained by integrating the temperature change amount with time (i.e., on the t axis) Let it be the raw data.

Smoothing of the raw data detected in the breathing data detection step S260 is performed in the smoothing and deduction step S255 and then detrend is performed.

The smoothing can be performed using a moving average filter or a Gaussian filter. This redistributes the distribution of light and darkness so that the light and dark areas become clearly visible.

In general, a trend is a method of removing an average value or a linear trend from the data to clearly hear the fluctuation, and there is a method such as a Detred Fluency Analysis (DFA), and a computer language (for example, c language) &Quot; for < / RTI >

In the smoothing and desarching step S255, continuous wavelet transform (CWT) is performed on the breathing data subjected to the smoothing and detouring, and the frequency-dependent intensities are represented, that is, converted into values in the frequency domain.

In the continuous wavelet transform step S260, continuous wavelet transform (CWT) is performed on the breathing data subjected to the smoothing and trending in the smoothing and deduction step S270 to display frequency-dependent scales (intensity and size) To values on the domain.

In the breathing frequency detection step S265, the sum of frequency-dependent scales (intensity, size) is obtained from the result of continuous wavelet transform in the continuous wavelet transform step S260, and the sum of frequency- (More specifically, the frequency most relevant to breathing), and stores each scale of the breathing frequency in the memory unit 220 as a breathing signal.

In the breathing waveform output step (S270), a breathing signal obtained in the breathing frequency detection step (S265) is outputted as a graph.

The following describes the breathing amount calculation step (S300) performed in the calculation processing unit 210. [ That is, a process of calculating the breathing volume using the breathing signal obtained in the breathing frequency detection step S265 of the breathing signal extraction step S200 will be described.

FIG. 6 is a flowchart for schematically explaining the breathing amount calculation step of FIG. 3;

In the peak and valley detection step (S310), the respiratory signal obtained in the respiratory frequency detection step (S265) is first differentiated to obtain the inflection point, and peaks and valleys are detected therefrom.

In the respiratory rate detection step S320, in the peaks and valleys detected in the peak and valley detection step S310, during a predetermined time interval (per second, or per minute or 30 seconds) And the number of pairs in which the valley sequentially changes, that is, the number of times the peak changes to the valley, is detected as the breathing number.

In the respiration detection step S330, the respiration cycle is detected in the respiration signal obtained in the respiration frequency detection step S265, and the respiration signal of each respiration cycle is subtracted from the respiration signal of the succeeding respiration cycle, The amount of change is calculated, and the amount of change by each respiratory cycle is added up with time to obtain the amount of respiration.

Here, the volume detection step S330 is the second method of the above-described volume detection method, which may be the first of the above-described volume detection methods.

FIG. 7 is an example of a result of performing the feature-based temperature variation detection step S225 and the temperature variation reference value comparison step S230 in the breathing signal extraction step S200 in FIG. That is, FIG. 7 shows an example of the output of the previous stage of the temperature change calculation step S235 of the breathing signal extraction step S200 of FIG.

7 (a) is a column image of the previous frame t0 subsequent to the current frame t1, FIG. 7 (b) is a column image of the current frame t1, (t1) and the difference between the thermal image of the previous frame (t0) and the thermal image of the previous frame (t0).

FIG. 7D shows a case where the absolute value of the difference image with respect to the thermal image of the previous frame t0 subsequent to the current frame t1 is compared with the temperature variation reference value and removed when the difference is smaller or equal. At this time, the temperature variation reference value can be set to 0, 05.

7 (e) shows positional information in the column image of (d) in Fig.

8 is an example of a result of removing noise using the 3D-DBSCAN in the noise removing step S220 of the breathing signal extracting step S200.

Although the DBSCAN algorithm is used to remove noise in the present invention, the present invention is not limited to this, and any algorithm that removes noise in a time-space manner can be used. In addition, DBSCAN can express arbitrary shape among various clustering methods, so clustering can be done more variously than K-means and K-medoid clustering. In FIG. 8, black is a point determined as noise, and red is a clustered point.

In general, DBSCAN is clustering based on the density of dots, that is, clustering is performed in a place where there is no change in temperature, and in a place where there is a change in temperature. In this clustering process, the clustering conditions are removed three-dimensionally by considering the density within the same time and the density along the time axis.

9 is an example of the result of the temperature change amount calculation step (S235).

FIG. 9A shows the temperature variation of one frame as a result of the characteristic temperature shift detection step (S225).

FIG. 9B is a graph showing the temperature change amount according to each frame as a result of the temperature change amount calculation step (S235). In FIG. 9 (b), the x-axis represents frames according to time, which can be said to represent time, and the y-axis represents the magnitude of the temperature variation.

10 is an example of a result of the respiration data detection step (S250).

10A is a graph showing the amount of temperature change per frame and FIG. 10B is an example of a result of the respiration data detection step S250, which is an example of the result of the temperature change amount calculation step S235. 10A is an example of obtaining the raw data by integrating the frame-by-frame temperature change amounts with time (i.e., along the t axis).

In FIG. 10 (a), the x-axis represents frames according to time, which can be regarded as a result, and the y-axis represents the magnitude of the temperature change amount. 10 (b), the x-axis represents a time-dependent frame, that is, time, and the y-axis represents an integrated value of the temperature change amount.

11 is an example of the result of the continuous wavelet transform step S260.

11B is an example of the result of the respiration data detection step S250, which is an example of raw data obtained by integrating the frame-by-frame temperature variation amounts over time (i.e., on the t axis).

11 (a) shows the frequency-dependent intensities by performing continuous wavelet transform (CWT) on the respiration data of Fig. 11 (b).

12 is a diagram for explaining the breathing frequency detection step (S265).

12 shows a result of performing continuous wavelet transform in the continuous wavelet transform step S260, and a frequency band (represented by a solid line with an arrow in Fig. 12) having the largest sum of intensities of frequencies is detected as a respiration frequency band . That is, in the breathing frequency detection step (S265), a scale band that maximizes the sum of the intensities is extracted using all the frames, which is a frequency-related (resp.

13 is an example of the result of the peak and valley detection step S310.

Fig. 13A is an example of the result of the breathing frequency detection step S265.

FIG. 13B is an example of a result of detecting a peak and a valley from the respiration signal of FIG. 13A. In FIG. 13 (b), the peak portion is indicated by red color, and the valley portion is indicated by blue color.

14 shows correction of the angle in the thermal image load and angle correction step S120.

14 (a) shows the angle of the nose designated by the user in the initial angle setting step S115,

Fig. 14B shows that the output of the normalization step S130 is corrected so as to be equal to the initial nose angle in Fig. 14A in the angle correction step S135.

FIG. 14C is a diagram showing a state in which the eye region of interest is corrected by correcting the angle of the nose as shown in FIG. 14B, and the interest in the frame of the original column image input in the column image receiving step corresponding to the detected region of interest Area is output.

FIG. 15 shows an example of extracting features in the primary feature extraction step (S150). That is, a plurality of features to be used for tracking in the region of interest, that is, the nose portion (or nasal portion, or nose portion) is extracted.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, Modification is possible. Accordingly, it is intended that the scope of the invention be defined by the claims appended hereto, and that all equivalent or equivalent variations thereof fall within the scope of the present invention.

50: Subject 100: Thermal imaging camera
200: Image analysis unit 210: Operation processing unit
220: memory unit 230: output unit
250: key input unit

Claims (19)

In the frame of the thermal image of the face received from the thermal imaging camera and representing the temperature of each pixel, the calculation processing unit detects the region of interest of the nose portion by using the nose and two eyes as tracking points and using the KLT feature tracker Respiratory-related area tracking step;
After the respiration-related area tracking step, the arithmetic processing unit obtains a difference image obtained by subtracting the image of the ROI of the previous frame from the image of the ROI of each frame in the thermal image of the face received from the thermal imaging camera, The value of each pixel of the difference image is set as the value of the temperature variation, the temperature variation values for each frame are added up, and are sequentially stored in the memory as the sum of the temperature variation for each frame. A respiration signal extraction step of reading the sum of the change amounts from the memory unit and integrating them according to time to obtain respiration-related data, and applying a continuous wavelet transform to the respiration-related data;
Wherein the breathing method comprises the steps of: The method according to claim 1,
A breathing and breathing volume detecting step of detecting a peak and a valley in the respiration signal and detecting a breathing volume and a breathing volume using the detected peak and valley;
Further comprising the steps of: measuring a breathing rate of the subject; 3. The method of claim 2, wherein breathing-
The arithmetic processing unit calculates a minimum temperature value and a maximum temperature value in each frame of the subject's face thermal image received from the thermal imaging camera and determines whether the temperature of each pixel of each frame is divided into 256 A normalization step of performing normalization so as to have a value;
An arithmetic processing step of the arithmetic processing part performing correction so that the nose of each frame normalized in the normalizing step is equal to a predetermined initial nose angle;
The arithmetic processing unit sets the candidate of the ROI using the nose and the two eye tracking points in each frame whose angles are corrected in the angle correction step, extracts the minutiae from the candidates of the ROI, A moving distance between the minutiae and the minutiae corresponding to the previous frame is represented by a matrix (Transform Matrix), the image of the current frame is corrected according to the extracted matrix, and the region of interest is corrected by 1 Car feature extraction step;
Wherein the breathing method comprises the steps of: 4. The method according to claim 3,
The arithmetic processing unit averages the temperatures in the interest region of each frame obtained in the first feature extraction step to obtain the arithmetic processing unit as the 'average region of interest per frame', and calculates the standard deviation of the temperatures in the interest region of each frame as' As a region of interest temperature difference of the target region;
The calculation processing unit compares the deviation of the region of interest calculated in the average temperature and deviation calculation step with a predetermined deviation reference value to determine whether the difference region of interest is greater than the deviation reference value. If the difference region of interest is larger than the deviation reference value, Determining whether a first deviation reference value is exceeded, removing features in the region of interest and going to a tracking point re-detection step;
The arithmetic processing unit compares the number of feature points found in the ROI with the predetermined number of feature reference values after the step of determining whether the first deviation reference value is exceeded and if the number of feature points is smaller than the feature number reference value, Determining whether a characteristic number is less than a reference value, to go to a tracking point re-detection step;
If the number of feature points is smaller than the feature number reference value in the step of determining whether or not the region of interest difference is larger than the deviation reference value or whether the feature region is smaller than the reference value in the step of determining whether the first deviation reference value is exceeded, , A tracking point movement distance which is a distance moved from each position of the two eye tracking points of the previous frame following the current frame to each position of the two eye tracking points of the current frame is obtained, A tracking point re-detection step of obtaining a tracking point moving distance average, which is an average of movement distances, and obtaining a tracking point of a corrected nose using a tracking point of a nose of a previous frame and an average tracking point moving distance;
Wherein the breathing method comprises the steps of: 5. The method of claim 4, wherein breathing-
A second feature extraction step of calculating a region of interest and extracting a feature using the tracking point of the modified nose and the tracking points of two eyes obtained in the tracking point re-detection step;
The arithmetic processing unit averages the temperatures in the interest region of each frame obtained in the second feature extraction step to obtain the arithmetic processing unit as the 'average region of interest per frame', and calculates the standard deviation of the temperatures in the interest region of each frame as' A second average temperature and a deviation calculating step of calculating a second average temperature and a deviation of a region of interest;
The arithmetic processing unit compares the temperature difference of the region of interest calculated in the secondary average temperature and deviation calculating step with the deviation reference value to determine whether the temperature of the region of interest is greater than the deviation reference value and if the temperature of the region of interest is greater than the deviation reference value Determining whether a secondary deviation reference value is exceeded to end tracking (tracking);
The arithmetic processing unit ends the tracing if the end flag is set as the end switch is pressed or if the current frame corresponds to the last frame of the initially set predetermined time interval and if not, A step of determining whether or not to end the tracing of the frame;
Wherein the breathing method comprises the steps of: 3. The method according to claim 2,
The arithmetic processing unit calculates a difference image obtained by subtracting an image of a region of interest of a previous frame of an image of a region of interest of each frame from a thermal image of a face received from a thermal imaging camera, As a value of the temperature variation;
A noise removing step of removing a noise point using a 3D-DBSCAN in a region of interest of each frame of the difference image through the frame-by-frame temperature shift detection step;
And the arithmetic processing unit removes the value of the temperature variation when the absolute value of the temperature variation value in each frame of the difference image through the noise elimination step is smaller than or equal to a preset reference value of the temperature variation, ;
A calculation step of calculating a temperature variation amount by summing each temperature variation in each frame of the difference image after the temperature variation reference value comparison step and obtaining the temperature variation amount for each frame and storing it in accordance with the frame order;
Wherein the breathing method comprises the steps of: 7. The method according to claim 6,
The arithmetic processing unit sequentially reads the temperature change amount sums for each frame obtained in the temperature change amount calculating step from the memory unit, integrates the temperature change amounts for each frame with time (on the t axis), and uses the integration result as raw data , Respiration data detection step;
A smoothing and dedisting step of smoothing and detrending the raw data detected in the breathing data detection step;
A continuous wavelet transform step of performing a continuous wavelet transform (CWT) on the breathing data subjected to the smoothing and detouring, and by frequency scales according to frequency;
The arithmetic processing unit obtains the sum of the frequency scales for each frequency on the basis of the result of performing the continuous wavelet transform and sets the frequency having the largest sum of the frequency scales on the frequency among the sum of the obtained frequency scales as the respiration frequency, A respiration frequency detecting step of storing each frequency scale of the breathing signal in a memory unit;
Wherein the breathing method comprises the steps of: The method according to claim 2, wherein the breathing and breathing volume detection step
A calculation and processing section for performing a first differentiation of the respiration signal obtained in the respiration signal extracting step to obtain an inflection point and detecting a peak and a valley using the inflection point;
The arithmetic processing unit calculates the number of pairs of peaks and valleys in which peaks and valleys appear in succession in a predetermined time interval in the peaks and valleys detected in the peak and valley detection step as the number of respiration Detecting a respiration rate;
Wherein the breathing method comprises the steps of: 9. The method according to claim 8, wherein the breathing and breathing amount detection step comprises:
Wherein the arithmetic processing unit detects an amplitude of a pair of peaks and valleys sequentially appearing in succession in a peak and a valley as a volume of respiration in a respiration cycle including a pair of peaks and valleys;
Wherein the breathing method comprises the steps of: 9. The method according to claim 8, wherein the breathing and breathing amount detection step comprises:
A respiratory cycle is detected by a pair of peaks and valleys in which peaks and valleys appear in succession, and the respiratory signal of each respiratory cycle is subtracted from the respiratory signal of the succeeding respiratory cycle to obtain a change amount for each respiratory cycle, A tidal volume detecting step of summing up the change amount per unit time and obtaining the tidal volume as a breath volume;
Wherein the breathing method comprises the steps of: 11. The method of claim 10,
Wherein the operation processing unit sums the start point of each breathing cycle based on the sum of the change amounts for each breathing cycle with time. The recording medium according to any one of claims 1 to 11, wherein the computer program source for the breath measurement method using the thermal imaging camera of the present invention is stored. And an image analyzing unit for analyzing a thermal image of the entire face received from the thermal imaging camera and detecting a breathing signal, the breathing measuring system comprising:
The arithmetic processing unit,
In the frame of the thermal image of the face received from the thermal imaging camera and representing the temperature of each pixel, the calculation processing unit detects the region of interest of the nose portion by using the nose and two eyes as tracking points and using the KLT feature tracker and,
A difference image obtained by subtracting the image of the ROI of the previous frame and the image of the ROI of the previous frame is obtained from the thermal image of the face received from the thermal imaging camera, The sum of the temperature variation values for each frame is sequentially stored in the memory unit as the sum of the temperature variation amounts for each frame and the sum of the temperature variation amounts of all the frames stored in the predetermined time period is read from the memory unit, Wherein the respiration data is integrated into the breathing data, and the breathing signal is detected by applying a continuous wavelet transform to the breathing data. 14. The method of claim 13,
Wherein the calculation processing unit detects peaks and valleys in the detected respiration signal and detects the respiration rate and respiration rate using the detected peak and valley. 15. The image processing apparatus according to claim 14,
In order to obtain the region of interest using the KLT feature tracker, normalization is performed so that the temperatures of the respective pixels of each frame of the subject face thermal image received from the thermal imaging camera have 8 bit values, and each frame having a normalized temperature value The image of each frame is corrected so that the angle of the nose of the image of the region of interest is equal to the preset nose angle and the KLT feature tracking algorithm is applied to the image of the corrected frame to find the region of interest, Wherein the region of interest is also found in the frame of the face image received from the thermal imaging camera, which is an original image corresponding to the region of interest of the corrected frame. 16. The image processing apparatus according to claim 15,
The breathing data is subjected to smoothing and detrend, continuous wavelet transform (CWT) is performed on the breathing data subjected to smoothing and detouring, and in the result of performing continuous wavelet transform, sum of frequency scales And the frequency scale of the respiratory frequency is stored in the memory as a respiration signal. The breathing apparatus according to claim 1, Breathing measurement system using infrared camera. 17. The image processing apparatus according to claim 16,
A peak and a valley are detected using an inflection point, and in the detected peaks and valleys, during a predetermined time interval, a peak and a valley are detected, Wherein the number of pairs of peaks and valleys sequentially appearing in series is detected as the number of breaths. 18. The image processing apparatus according to claim 17,
A respiratory cycle is detected by a pair of peaks and valleys in which peaks and valleys appear in succession, and the respiratory signal of each respiratory cycle is subtracted from the respiratory signal of the succeeding respiratory cycle to obtain a change amount for each respiratory cycle, Wherein the breath change amount is calculated as a breath volume by summing up a change amount per minute with respect to time. 18. The image processing apparatus according to claim 17,
Characterized in that the amplitude of the pair of peaks and valleys appearing sequentially in succession from the peak and the valley is detected as the respiratory volume of the respiratory cycle including the pair of peaks and valleys.

Images