KR102275263B1 - Respiratory measurement system using depth camera - Google Patents

Abstract

The present invention uses a depth camera to photograph the breathing of the subject, obtains a difference image arrangement in which the difference between each other is calculated from the obtained depth image arrangement, and to separate the breathing region of the difference image. In the difference image, by performing level set using the shape prior information and the region information function, the breathing region of the difference image is separated, or the difference image and the continuous depth image stored in the memory unit are recorded in the depth image arrangement step. By applying to the learned ResU-net (Residual U-net), it is possible to separate the respiration area from the difference image and calculate the respiration volume from the separated respiration area to measure the respiration volume non-invasively, non-constrained and with high accuracy. It relates to a respiratory volume measurement system and method using a depth camera.
Respiratory volume measurement method using a depth camera of the present invention, a pre-processing step of a calculation processing unit receiving a depth image array of the subject's upper body from the depth camera, and removing noise including impulse noise from the received depth image arrays; The arithmetic processing unit obtains a difference image by subtracting the difference between the depth values of the corresponding pixels from the temporally consecutive previous frames from the depth values of each pixel of each frame of the depth image array from which the noise has been removed in the preprocessing step, and the memory unit stored sequentially in . difference image arrangement step; In the difference image obtained in the difference image arrangement step, a breathing region separation step of separating the breathing region of the difference image by performing level set using the shape prior information and the region information function;
In addition, in the method for measuring respiration volume using a depth camera of the present invention, the calculation processing unit receives the depth image arrangement of the subject's upper body from the depth camera, and performs pre-processing of removing noise including impulse noise from the received depth image arrangement. , pretreatment step; The operation processing unit may include: a depth image arranging step of sequentially storing continuous depth images from which noise has been removed in the preprocessing step in the memory unit; The arithmetic processing unit obtains a difference image by subtracting the difference between the depth value of the pixel in the temporally consecutive previous frames from the depth value of each pixel of each frame of the continuous depth image array from which the noise has been removed in the preprocessing step, stored sequentially in the memory unit. difference image arrangement step; The difference image obtained in the difference image arrangement step and the continuous depth image stored in the memory unit in the depth image arrangement step are applied to the pre-learned ResU-net (Residual U-net) to separate the breathing area of the difference image. It is characterized in that it includes; a region separation step.

Description

Respiratory measurement system using depth camera}

The present invention uses a difference image array and a region information function that calculates the difference between each other in the acquired depth image array by photographing the subject's breathing using a depth camera, using region-based information By applying the level set method, only the part where there is a change due to respiration in the depth image is separated into the respiration area, and the respiration volume is calculated in the separated respiration area to measure the respiration volume with high accuracy without invasiveness. It relates to a respiration volume measurement system using a depth camera.

The lungs are responsible for taking in oxygen and expelling carbon dioxide from the body. For the diagnosis of respiratory diseases, various imaging tests (CT, MRI, PET, perfusion scan) and blood tests (arterial blood gas test, tumor indicator test, general blood test), etc. can be performed. However, for the functional evaluation of the lungs, the reproducibility of the lung function test has been verified as the only tool that evaluates the functional aspects of the lung as an objective index.

Among them, spirometry is a test method that measures the amount of air that a patient can exhale after inhaling as much as possible. Although the lung function test does not diagnose a specific disease, it shows unique characteristics according to the disease. It helps to diagnose early or evaluate clinical progress in stages. In addition, by reflecting the change in the severity of the disease by changing the measurement value, it can provide solutions to clinical situations in various lung diseases, and can help to determine the effect before and after treatment or whether the treatment is reversible. As a representative example, the lung function test helps in the diagnosis of respiratory diseases such as asthma and chronic obstructive pulmonary disease (COPD), and is also used as a method to determine the presence or absence of abnormalities by checking the health status of the respiratory system. . The main indicators to measure here are forced vital capacity (FVC), which measures the flow rate discharged during effort expiratory, and forced expiratory volume in one second, which measures the flow rate during the first second during the FVC measurement process. second, FEV1). There is also a bronchodilator test that looks for fluctuations in measurements after inhalation of the bronchodilator.

In general, spirometry consists of inhaling or exhaling through a mouthpiece at the end of a tube connected to the machine.

Even healthy people have recently increased interest in spirometry, and products that are already on the market make the existing lung function tester portable, and the basic operating principle is the same. It is used for the purpose of measuring the respiratory capacity by biting the mouthpiece and improving the breathing ability through repeated training.

However, these spirometry devices have limitations in use. For example, if a patient is diagnosed with a respiratory-borne infectious disease, cannot put the mouthpiece in the mouth, cannot cooperate, or is unconscious, the lung function test cannot be performed. In addition, since the test is generally performed in a sitting state, it may be difficult for a patient who cannot sit down.

Accordingly, there is a need for a respiratory volume measurement system capable of measuring respiration volume in a non-invasive, non-constrained manner.

As a prior art, Korean Patent Application Laid-Open No. 10-2016-0126238 relates to a device for monitoring breathing rate during sleep using a depth camera, and designates 150 * 75 pixels from the left/top of the chest of a sleeper as an ROI (region of interest), The respiration rate is estimated by averaging the depth values from the ROI. However, when a region of interest is designated in this way, it is not possible to obtain an accurate result because the part not related to respiration is reflected, and the respiration rate cannot be accurately known only by estimating the respiration rate by simply calculating the average of the depth values. .

Therefore, the present invention, non-invasive, non-constrained, with higher accuracy, which can measure the respiration volume, proposes a respiratory volume measurement system.

The problem to be solved by the present invention is to use a difference image arrangement and a region information function in which a difference between each other is calculated from the acquired depth image arrangement by photographing the subject's breathing using a depth (depth) camera. , by applying the level set method using area-based information, separating only the part where there is a change due to respiration in the depth image into the respiration area, and calculating the respiration volume in the separated respiration area, providing a respiration volume measurement system using a depth camera will do

Another problem to be solved by the present invention is, in the pre-processing of the acquired depth image, in the respiratory rate measurement system using a depth camera, using a median filter and a clipping technique in the temporal and spatial domains. It is to remove the error value from the image data.

Another problem to be solved by the present invention is to separate only the area affected by respiration in the depth image using the level set method, and the level set is to use a method combining the area information function and the dictionary information function.

Another problem to be solved by the present invention is to calculate the volume waveform by applying the separated respiration area to the respiration volume calculation formula, find the peak and valley in the respiration volume waveform, and calculate the respiration volume.

In order to solve the above problems, in a first embodiment of a method for measuring respiration using a depth camera of the present invention, the operation processing unit receives the depth image arrangement of the subject's upper body from the depth camera, and impulse noise from the received depth image arrangements A pre-processing step of removing noise including; The arithmetic processing unit obtains a difference image by subtracting the difference between the depth values of the corresponding pixels from the temporally consecutive previous frames from the depth values of each pixel of each frame of the depth image array from which the noise has been removed in the preprocessing step, and the memory unit stored sequentially in . difference image arrangement step; In the difference image obtained in the difference image arrangement step, a breathing region separation step of separating the breathing region of the difference image by performing level set using the shape prior information and the region information function;

In addition, in the second embodiment of the method of measuring the amount of respiration using the depth camera of the present invention, the operation processing unit receives the depth image arrangement of the subject's upper body from the depth camera, and noise including impulse noise from the received depth image arrangements performing a pretreatment to remove, a pretreatment step; The arithmetic processing unit may include: a depth image arranging step of sequentially storing continuous depth images from which noise has been removed in the preprocessing step in the memory unit; The arithmetic processing unit obtains a difference image by subtracting the difference between the depth value of the pixel in the temporally consecutive previous frames from the depth value of each pixel of each frame of the continuous depth image array from which the noise has been removed in the preprocessing step, stored sequentially in the memory unit. Difference image arrangement step; The difference image obtained in the difference image arrangement step and the continuous depth image stored in the memory unit in the depth image arrangement step are applied to the pre-learned ResU-net (Residual U-net), and the breathing region of the difference image It is characterized in that it comprises a; breathing zone separation step to separate the.

In the breathing region separation step of the first embodiment, the operation processing unit separates the rib cage and the abdomen region from the depth image by using the shape dictionary information for separating only the torso region, which is the rib cage and the abdomen region. Level set using the shape dictionary information step; The operation processing unit calculates a weight having a value between 0 and 1 by using the image of the respiration-related region of the previous frame in succession, and applies the weight to the shape dictionary information to which the region information function is applied, thereby adaptive region information an area information function obtaining step of detecting a function; A level set using a region information function, where the operation processing unit separates the breathing region of the difference image by using a region information function and a CVM (Chan-Vese model) technique on the difference image obtained in the difference image arrangement step step; may include.

In the first embodiment or the second embodiment, the preprocessing step is to perform median value filtering on the depth values of each pixel according to the temporal domain in successive depth images, and filter the median value from the depth values of each pixel according to the spatial domain. performing spatiotemporal median filtering; In successive depth images that have been median-filtered in the temporal and spatial domains in the spatiotemporal median filtering step, clipping is performed on the depth values of each pixel along the temporal domain based on the temporal domain clipping threshold, and a spatiotemporal clipping step of clipping the depth values based on a spatial domain clipping threshold; may include.

In the area information function acquisition step of the first embodiment, when the operation processing unit applies a signed distance function (SDF) to the area separated by using the shape dictionary information, the boundary line is set as 0, and , as the distance from the reference is increased, the inner region of the separated region has a positive value, the outer region of the separated region has a negative value, and in the region to which the SDF function is applied, a value exceeding a certain threshold or a predetermined value It includes treating values within the size unit as the same value, and applying a unit step function (Heaviside) function to the outer area.

In the breathing region separation step of the second embodiment, the operation processing unit receives the preprocessed depth image stored in the depth image arrangement step (S130) from the memory unit, and inputs it to the pre-learned first ResU-Net, and the first ResU - ResU-Net to receive a segmentation map that separates only the body region, which is the thoracic and abdominal region, output from the Net, respiration region segmentation step in the spatial region using ResU-Net; The arithmetic processing unit may include: a segmentation map arrangement step in which only the torso image is separated, in which the segmentation map obtained by separating only the body image of the person received from the first ResU-Net is sequentially temporarily stored in the memory unit; The arithmetic processing unit separates only the torso image by multiplying the segmentation map in which only the torso image is separated, output in the breathing region segmentation step in the spatial domain using ResU-Net, by the difference image obtained in the difference image arrangement step in units of pixels a multiplication step between a segmentation map and a difference image; The arithmetic processing unit inputs, as an input of the previously learned second ResU-Net, the product of the segmentation map in which only the body image is separated and the segmentation map in which only the torso image is separated, which is output in the multiplication step of the difference image, and the difference image, 2 As the output of ResU-Net, receiving an image in which only a respiration-related area is separated from the difference image, which is a segmentation map in which only a respiration-related area is separated from the second image, is received, a respiration area segmentation step in the time domain using ResU-Net; includes

The first ResU-Net is a machine-learning artificial neural network using, as labels, depth images from which noise has been removed in the pre-processing step and an image obtained by separating only the body region from the depth image from which noise has been removed. 2 ResU-Net is a segmentation map in which only the body image is separated and the difference image output in the multiplication step of the difference image and the image in which only the body region is separated in units of pixels are multiplied in units of pixels, and in the difference image It is a machine-learning artificial neural network using an image that isolates only the respiration-related area as a label.

For the label used in machine learning of the first ResU-Net, a label image can be created using a level set using shape prior information, and the label used in machine learning of the second ResU-Net is a level using the area information function. A label image may be generated using the set.

In the second embodiment, the arithmetic processing unit checks whether the value of the counter is equal to the preset number of difference image sequences, if not, increments the counter by 1, and returns to the breathing region division step in the spatial domain using ResU-Net It may further include; judging whether or not the breathing zone separation process is finished.

In the first embodiment, the arithmetic processing unit checks whether the counter value is equal to the preset number of difference image sequences, and if not, increments the counter by 1, and returns to the level set stage using shape prior information, breathing region separation It may further include; determining whether the process is complete or not.

In the first embodiment and the second embodiment, if the value of the counter is equal to the preset number of difference image arrays in the step of determining whether the breathing zone separation process is finished, the calculation processing unit, the difference obtained in the breathing zone separation step In frames of images in which only the region related to breathing is separated from the image, the depth value of the breathing region of each frame is summed to obtain a respiration-related signal for each frame, and the preceding pre-em respiration-related signal is subtracted from the respiration-related signal of the current frame to obtain the respiration change for each frame, multiply the respiration change for each frame by the unit volume value of the preset pixel to obtain the respiration change for each frame converted to the actual size, and graph the converted amount of respiration per frame according to the time sequence Respiratory waveform detection step, indicating the respiratory waveform by displaying; Detecting peaks and valleys in the respiration waveform, and calculating the respiration rate and respiration rate using the detected peaks and valleys, a respiration rate and respiration rate detection step; may further include.

In the step of detecting the respiration rate and respiration volume, the calculation processing unit first differentiates the respiration signal obtained in the respiration waveform detecting step to obtain an inflection point, and using the inflection point to detect a peak and a valley, peak and valley detection step; In the peak and valley detected in the peak and valley detection step, the number of pairs of peaks and valleys in which the peaks and valleys appear one after another during a predetermined time interval, as the respiration rate Detecting, respiration detection step; may include.

Respiratory rate and respiration volume detection step, the operation processing unit, the peak and valley pair of peaks and valleys appearing in succession in sequence, the respiration volume detection step of detecting the respiration volume of the respiration cycle including the pair of peaks and valleys; may include

In the step of detecting the respiration rate and volume, the respiration cycle is detected as a pair of peaks and valleys in which peaks and valleys appear sequentially in succession, and the respiration signal of each respiration cycle is subtracted from the respiration signal of the previous respiration cycle in succession to each respiration cycle. Respiratory volume detection step of obtaining a star change amount, summing the change amount for each respiratory cycle over time to obtain a respiration amount; may include.

When the amount of change for each respiration cycle is summed over time, the calculation processing unit may add up based on the starting point of each respiration cycle.

It features a recording medium storing a computer program source for the method of measuring respiration using a depth camera in the present invention.

In addition, the respiration measurement system using the depth camera of the first embodiment of the present invention, the depth camera, by analyzing the depth image of the subject's upper body received from the depth camera includes an image analysis unit comprising an operation processing unit for detecting a respiration signal In the respiration measurement system using a depth camera, the operation processing unit receives the depth image array of the subject's upper body from the depth camera, and performs preprocessing for removing noise including impulse noise from the received depth image arrays, and , of the preprocessed depth image array, from the depth value of each pixel of each frame, subtracting the difference of the depth value of the corresponding pixel from the previous frame in temporal succession to obtain a difference image array, sequentially storing it in the memory unit, and the depth It is characterized in that the breathing region is separated by performing level set using the shape prior information and the region information function in the difference image arrangement of the image.

In addition, the respiration measurement system using the depth camera of the second embodiment of the present invention, the depth camera, by analyzing the depth image of the subject's upper body received from the depth camera includes an image analysis unit comprising an operation processing unit for detecting a breathing signal In the respiration measurement system using a depth camera, the operation processing unit receives the depth image arrangement of the subject's upper body from the depth camera, and performs preprocessing for removing noise including impulse noise from the received depth image arrangement, The preprocessed continuous depth image array is sequentially stored in the memory unit, and the depth value of each pixel of each frame of the preprocessed depth image array is subtracted by subtracting the difference between the depth values of the corresponding pixels from the previous frames consecutively in time. The difference image arrangement is obtained and stored sequentially in the memory unit, and the difference image arrangement and the preprocessed continuous depth image received from the memory unit are applied to the pre-learned ResU-net (Residual U-net). It is characterized in that the breathing zone is separated.

In order to separate the breathing region by performing level set in the respiration measurement system using the depth camera of the first embodiment, the arithmetic processing unit uses the shape dictionary information for separating only the rib cage and the abdomen region in the difference image arrangement of the depth image, In the depth image, the torso region, which is the thoracic and abdominal region, is separated, and a weight having a value between 0 and 1 is calculated using the images of the breathing-related region of the previous frame in succession, and in the shape prior information to which the region information function is applied, the By applying a weight, the adaptive region information function is detected, and the breathing region of the difference image is separated by using the region information function and the CVM (Chan-Vese model) technique in the difference image arrangement.

In the respiration measurement system using the depth camera of the second embodiment, the difference image arrangement and the pre-processed continuous depth image are applied to the pre-learned ResU-net (Residual U-net) to separate the breathing area of the difference image, The arithmetic processing unit inputs the preprocessed depth image received from the memory unit into the pre-learned first ResU-Net, and outputs a segmentation map that separates only the torso image, which is the thorax and abdomen region, output from the first ResU-Net, The segmentation map obtained by receiving and separating only the torso image received from the first ResU-Net is sequentially temporarily stored in the memory unit, and the segmentation map obtained by separating only the torso image is multiplied by the difference image in units of pixels, 2 As an input of ResU-Net, the product of a segmentation map and a difference image that separates only the body image is input, and as an output of the second ResU-Net, a segmentation map that separates only a region related to respiration from the difference image, respiration and An image in which only the relevant area is separated is received.

Respiratory volume measurement system using a depth camera of the present invention is a difference image arrangement and area information function in which the difference between the two images is taken using a depth (depth) camera to capture the subject's breathing, and the difference is calculated from the obtained depth image arrangement. By applying the level set method using area-based information using , only the part where there is a change due to respiration in the depth image is separated into the respiration area, and the respiration volume is calculated in the separated respiration area, making it non-invasive and non-constrained. In this way, the respiration rate can be accurately obtained.

In addition, the present invention provides an error value in depth image data using a median filter and a clipping technique in the temporal and spatial domains in the pre-processing of the acquired depth image in the respiratory rate measurement system using a depth camera. to remove

In addition, the present invention separates only the region affected by respiration in the depth image using the level set method, and the level set uses a method combining the region information function and the prior information function.

In addition, the present invention calculates the volume waveform by applying the separated respiration region to the respiration volume calculation formula, and calculates the respiration volume by finding peaks and valleys in the respiration volume waveform.

Therefore, in the present invention, since the error value is removed and only the region affected by the respiration is extracted to calculate the respiration volume, it is possible to detect the respiration volume more accurately and non-invasively.

Respiratory volume measurement system using a depth camera of the present invention can enable a patient's breathing pattern, tidal volume, effort lung capacity, comparison of effects of treatment before and after drugs, and respiratory rehabilitation in a non-invasive way.

As an area where the respiratory volume measurement system using the depth camera of the present invention can be used, first, it can be used as a test that can replace the lung function test for patients who cannot perform the lung function test. Non-contact, non-invasive, relatively low cost for patients who could not perform lung function tests with conventional methods, such as patients with bronchectomy who cannot bite the mouthpiece, patients who cannot sit down, and patients who are being treated for respiratory-mediated infectious diseases such as MERS. can be substituted for lung function tests.

Second, it can be used for respiration monitoring in patients undergoing subsedation procedures. In recent medical fields, such as gastrointestinal endoscopy and MRI, which requires images to be obtained for more than 30 minutes, the frequency of administering sedatives under monitoring for the purpose of patient comfort and ease of operation is increasing. Although peripheral arterial oxygen saturation is being measured to monitor respiratory depression due to hypersedation, there are limitations in monitoring due to poor accuracy and speed. The present invention can non-invasively monitor real-time respiration, which can be helpful in terms of patient safety.

Third, in the medical field, when the ability of the respiratory muscles is reduced and respiratory rehabilitation treatment is required, it can be helpful to quantitatively measure respiration. In particular, by grafting the present invention to rehabilitation exercise equipment and mobile devices, it can be used to enhance the effect of exercise through biofeedback during respiratory rehabilitation treatment.

Fourth, it has the potential to be used as a screening tool for the diagnosis of respiratory diseases. Currently, there are no tests that can screen for respiratory diseases other than examination, counseling, and chest radiography in the health checkup items of the National Health Insurance. By analyzing the algorithm of the present technology, it can be used to screen patients in the pre-diagnosis stage.

In addition, the respiratory rate measurement system using the depth camera of the present invention can be utilized in the development of health devices for the general public in addition to the medical environment. Recently, after a wide-area bus drowsy driving accident that resulted in a large-scale fatal accident, the device for preventing drowsy driving is receiving attention again. By using the present invention, it is also possible to analyze the driver's breathing pattern and utilize it as a monitoring purpose.

In addition, living in an environment in which the fine dust index is announced in the daily weather forecast, modern people are not only concerned about the morbidity and exacerbation of respiratory diseases, but are also paying attention to improving their breathing ability. Since fitness bands such as Mi Band are expanding their base in the health market, if the respiratory volume measurement system using the depth camera of the present invention is linked, it can be used to measure the lung capacity before, during, and after exercise.

1 is a conceptual diagram for schematically explaining a respiratory volume measurement system using a depth camera of the present invention.
Figure 2 is a block diagram schematically showing the configuration of the respiration measurement system using the depth camera of the invention.
FIG. 3 is a diagram for explaining data pre-processing in the time domain in the operation processing unit of FIG. 2 .
FIG. 4 is a diagram for explaining data pre-processing in a spatial domain in the operation processing unit of FIG. 2 .
5 is a diagram for explaining shape dictionary information.
FIG. 6 is a view for explaining division of a breathing region by applying a level set method using a CVM and a region information function in the operation processing unit of FIG. 2 .
7 is an explanatory diagram for explaining a region information function using a breathing-related region in the operation processing unit of FIG. 2 .
FIG. 8 is an explanatory diagram for explaining a process of generating a region information function using a respiration-related region in the operation processing unit of FIG. 2 .
9 is a flowchart schematically illustrating a method of detecting a respiration volume by using a depth image received from a depth camera in the image analysis unit of FIG. 2 and applying a level set method.
10 is a flowchart illustrating the step of separating the breathing area of FIG. 9 .
Figure 11 shows an example of the breathing zone separation of the existing method, and an example of the breathing zone separation of the present invention.
12 is a comparative example of respiratory volume waveforms when the breathing zone separation method of the existing method, the breathing zone separation method of the present invention, and a ventilator are respectively applied in a normal environment.
13 is a comparative example of respiratory volume waveforms in the case of applying the breathing zone separation method of the existing method, the breathing zone separation method of the present invention, and a ventilator in an environment with arm movement.
14 is an explanatory diagram for explaining a respiration domain division process in a spatial domain using ResU-Net.
15 is an explanatory diagram for explaining the respiration domain division process in the time domain using ResU-Net.
16 is a flowchart schematically illustrating a method of detecting a respiration volume by using a depth image received from a depth camera in the image analysis unit of FIG. 2 and applying a ResU-net (Residual U-net) method.

Hereinafter, the respiratory volume measurement system using the depth camera of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram for schematically explaining a respiration volume measurement system using a depth camera of the present invention, Figure 2 is a block diagram schematically showing the configuration of a respiration measurement system using a depth camera of the present invention.

Figure 1 (a) shows that the subject 50 is lying down using the respiratory rate measuring system to measure the respiration, Figure 1 (b) is using the respiratory rate measuring system in a state where the subject 50 is sitting to measure respiration.

In FIG. 1, although breathing is measured when lying or sitting, the present invention is not limited thereto, and any posture may be used as long as the subject's upper body can be continuously photographed.

The subject 50 breathes in a sitting or lying position in front of the depth camera 100 , and transmits a depth image obtained by photographing breathing in the depth camera 100 to the image analysis unit 200 .

The depth image is measured by the depth camera 100 at a rate of 16 frames. That is, since 16 depth images are acquired per second, the number of images increases as the recording time increases.

The operation processing unit 210 of the image analysis unit 200 removes an error value from the depth image data using a median filter and a clipping technique in the temporal and spatial domains. go through a pre-processing process. That is, median value filtering and clipping are performed in the temporal domain, and median value filtering and clipping are performed in the spatial domain. In this case, median filtering and clipping may be performed in the spatial domain, and median filtering and clipping may be performed in the temporal domain.

Continuous depth images are acquired over time by photographing a person's breathing using a depth camera.

When depth values are sequentially displayed over time in a specific pixel A in a continuous depth image, data of depth values over time for a specific pixel (eg, A of FIG. 3 ) can be obtained. In other words, pixel values (ie, depth values) of the same area are sequentially stored over time. That is, data of depth values according to time for a specific pixel are arranged and stored, and in this stored data (that is, data of depth values according to time for a specific pixel), using a window of a predetermined size, Median value filtering, that is, median value filtering in the time domain, is performed through the median filter, and clipping in which impulses and the like are removed from the result of median filtering in the time domain, that is, clipping in the time domain is performed.

In addition, in each of the successive depth images clipped in the time domain, in the depth values for each pixel (ie, the values of each pixel), using a window of a predetermined size, filtering the median value through the median value filter; That is, median filtering is performed in the spatial domain, and clipping for removing impulses and the like from the result (ie, image) of the median filtering in the spatial domain, that is, clipping in the spatial domain is performed.

Here, the median filter has a window of a predetermined size among temporally or spatially arranged pixels (image elements), that is, a predetermined number of pixels according to the temporally or spatially arranged window size. In the field, the median value of the periphery of a given pixel (image element) is calculated, and the calculated median value pixel (image element) is used as a representative brightness value and applied to the point of each damaged or lost pixel (image element), which is As it is known in the knowledge encyclopedia, a more detailed description will be omitted.

Here, the clipping removes impulses having a predetermined size or more from depth values of pixels in the temporal or spatial domain, which are results of intermediate value filtering in the temporal or spatial domain.

3 is a diagram for explaining data preprocessing (ie, intermediate value filtering and clipping) in the time domain in the arithmetic processing unit of FIG. 2 , and FIG. 4 is a diagram for explaining data preprocessing in the spatial domain in the arithmetic processing unit of FIG. 2 It is a drawing.

Figure 3 (a) shows a depth image taken continuously over time while photographing a person breathing using a depth camera. FIG. 3B is a graph showing the depth value of a specific pixel A over time in the time-dependent depth images of FIG. 3A . FIG. 3C is a graph showing the depth value of a specific pixel A over time in FIG. 3B .

To elaborate on the median filtering in the temporal and spatial domains, continuous depth images according to time are acquired as shown in FIG. 3 ( a ) by photographing a person breathing using a depth camera. In this case, the acquired depth image includes error values. In order to remove this, the error value is removed by using a median filter and clipping technique in the temporal and spatial domains. For example, when a depth value in a specific pixel A is displayed over time in the continuous depth image of FIG. 3A , it is displayed as shown in FIG. 3B . Median value filtering in the time domain is by covering a window of a predetermined size on the values that display the depth value in a specific pixel A over time, sorting in the order of size within the window, and finding the median value among the sorted values, Let the values in the window be the intermediate values. In the case of data preprocessing in the time domain, a one-dimensional signal indicating a change in the depth value of the pixel position for all pixels using a median value filter, that is, a signal as shown in Fig. 3(b), is subjected to median value filtering. , during the filtering process, the impulse noise seen in the signal, that is, the signal as shown in Fig. 3 (b) is removed to obtain a signal from which the noise value is removed as shown in Fig. 3 (c), and this signal is applied to respiration Since it represents a change in the depth image according to the flow, this signal is referred to as a breathing signal. If the temporal domain represents a change in the depth value over time, in the spatial domain, a 2D median filter is applied to the depth image at a specific point in time to remove the spatial error value of the image.

Clipping refers to making an error value that is not removed from the median filtering process in temporal and spatial domain filtering to 0. The error value has a very large depth value (eg, 12000 mm, etc.) as shown in FIG. 3(b). After filtering the median values in the temporal and spatial domains for these error values, the error values that are not removed are detected and the values are filled with 0 instead.

The operation processing unit 210 of the image analysis unit 200 temporarily stores images preprocessed through the preprocessing process, that is, depth images in order (arranged) in the memory unit 220 .

Also, in the images preprocessed through the preprocessing process, the operation processing unit 210 of the image analysis unit 200 is configured to have a difference in depth values of predetermined pixels in temporally consecutive images, that is, a depth value of a pixel in the current image. , the difference images are obtained by subtracting the depth value of the corresponding pixel of the successive previous image and temporarily stored in the memory unit 220 .

And, the operation processing unit 210 of the image analysis unit 200 separates only the region affected by respiration in the depth image preprocessed by the preprocessing process. This can be called the respiratory zone separation process. Respiratory region separation process includes a method of separating using a level set method and a method of separating using ResU-net (Residual U-net, residual U-net).

First, the breathing area separation process using the level set method will be described.

That is, in the depth image preprocessed by the preprocessing process, only the region affected by respiration is separated using the level set method. In this breathing region separation process, a level set process using shape prior information, an adaptive region information function generation process, and a level set process using region-based information are sequentially performed.

When explaining the level set process using the shape prior information, after the pre-processing process, the level set for separating the rib cage and the abdomen region from the depth image is performed using the pre-stored shape prior information (that is, the shape of the body of a certain shape). . In this way, only the breathing-related region, that is, the rib cage and the abdomen region is separated and stored in the memory unit 220 . That is, in the level set method using shape dictionary information, a shape of a torso of a predetermined shape is given as an input in advance, and the chest and abdomen regions are separated from the depth image by using the input.

Here, the shape dictionary information indicates an area including the chest and abdomen where changes occur by respiration as shown in FIG. 5 , which may be information stored at the time of shipment from the factory or information stored by a user at the beginning of use. FIG. 5A is an example of shape dictionary information, and a portion except for shapes representing the rib cage and abdomen (ie, torso) is an image made of black. FIG. 5B is a diagram for explaining application of shape dictionary information.

Next, explaining the process of generating the adaptive region information function, the image of the entire respiration-related region is read in succession, the weight is calculated using this, and the weight is reflected in the shape prior information to which the region information function is applied, and the adaptive region Detect information function.

Here, the application of the area information function is to apply a signed distance function (SDF) to a predetermined image, with the boundary line being 0 as a reference, and as the distance from the reference is increased, the inner area is a positive number, the outer area is The region has a negative value, and in the image of the region to which the SDF function is applied, a value exceeding a certain threshold (or a value within a predetermined size unit) is treated as the same value, and then the external region is converted to a unit step function (Heaviside) A function is used to transform a constantly decreasing value to decrease with a steep slope.

Here, the weight is a value between 0 and 1, and is a value empirically determined by the user. For example, the image-related weight of the current respiration-related area (eg, a) and the image-related weight of the previous respiration-related area (eg, 1-a) may be summed to be 1.

Next, a level setting process using region-based information will be described. The level set method using area-based information is used to separate only the area where the depth value actually changes due to respiration from the images of the chest and abdomen areas separated through the level set process using the shape prior information. The level set using region-based information is not applied to the depth image, but is applied to the image in which the difference between the depth images is calculated. That is, by applying the level set method using the region-based information to the above-described difference image, only the breathing-related region (ie, the region in which a change in depth value occurs due to respiration) is extracted and stored in the memory 220 . Here, the level set uses a method in which the area information function and the dictionary information function are combined. The divided breathing region is used as prior information when dividing the breathing region in the next difference image, and the weight is reflected when generating the region information function to generate an adaptive region information function.

In the level set method using region-based information, the above-described region information function is added to the energy function by using a respiration-related region separated using a level set using shape dictionary information.

FIG. 6 is a diagram for explaining division of a breathing region by applying a region-based level set method according to a Chan & Vese model (CVM) and a region information function in the operation processing unit of FIG. 2 .

6A is an image obtained by obtaining the difference between the depth images of the rib cage and the abdomen, that is, the difference image. In the difference image of FIG. 6A , an area where a large change in depth value occurs is displayed in white, and an area in which the change in depth value is small is displayed in black. In the image of Fig. 6 (a), only the region changed by respiration inside the respiration-related region should be extracted, Fig. 6 (b) shows the CVM (Chan & Vese model, chan) in the image of Fig. 6 (a). This is an example of the result of separating the breathing area by applying the technique of Besse). 6 (c) and (d) are shown by applying the technique of CVM (Chan & Vese model, Chan Besse) to the image of FIG. 6 (a) to separate the breathing region, but by differentiating the internal function of the region-based information. These are the results of dividing the breathing area.

As shown in (c) to (d) of Figure 6, it can be seen that only the area in which the change occurs by respiration within the respiration-related area is separated. That is, a region information function is created using the breathing-related region separated by the level set method using the shape dictionary information, and the breathing region is separated by applying the level set method using the region-based information using this.

Segmentation of an image The implementation of the level set method (Chan-Vese) is to segment the image using the level set method, where segmentation is to cut out a region of interest in the image (image), level set (Level set) Law, developed by Osher and Sethean, is a framework for segmentation. In this framework, the inside and outside of the region are expressed as follows by the sign of the negative function φ called the level set function.

Here, r is the pixel position (coordinate distance), and t is the number of iterations of the CVM technique, that is, the Chan Besse algorithm, and has a temporal concept, so it can also be said to represent time. In addition, Ω is the boundary of the region to be divided, that is, the respiration-related region (ie, a parabola or a parabola of the Chan Besse algorithm). That is, at time t, the level set function φ(r, t) of the pixel located at the pixel position r has a value less than 0 (negative value) if the pixel position r belongs to the respiration-related region Ω, and the pixel position r is breathing. If it is the boundary of the relevant region Ω, it has a value of 0, and if the pixel position r does not belong to the respiration-related region Ω (ie, it belongs to the outside of Ω), it has a value greater than 0 (positive value).

The shape dictionary information in the image area is applied to the level set method by adding the shape dictionary information energy term to the energy term of the existing Chan-Vese level set method as shown in Equation (1). That is, when the shape dictionary information is applied to the image region, the energy (E _total ( ) ) of the entire image according to the level set function φ can be expressed as Equation 1 .

Here, φ is the segmentation curve at the current time in the chest and abdominal depth images, φ ₀ is the input shape prior information, and E _cv (φ) is the energy ( That is, the energy term of the image by the Chan-Vese level set method), E _shape (φ) is the shape dictionary information energy (ie, the shape dictionary information energy term), and a is the weight. d(φ, φ ₀ ) ² represents the distance function of the split curve at the current time point and the shape of the shape prior information in the depth image of the chest and abdomen, that is, d(φ, φ ₀ ) is the In the depth image, it represents the distance between the segmentation curve of the current view and the shape of the shape dictionary information. As in Equation 1, it is expressed as an expression in which the shape dictionary information term is multiplied by the weight a. Depending on whether a large value or a small value is given to the weight a, the energy given by the existing Chan-Vese method is either large or small.

d(φ, φ ₀ ) ² is represented by Equation (2).

는 분할곡선에서 해당 픽셀 위치(픽셀 좌표)이며,

는 현재시점의 분할곡선을 나타내며,

는 입력한 형상 사전정보의 분할곡선을 나타낸다. here,

is the pixel position (pixel coordinates) on the segmentation curve,

represents the split curve at the present time,

represents the segmentation curve of the input shape prior information.

In addition, the step function (ie, H(x)) can be defined as follows.

H(x) refers to extracting and subtracting only the inner region of each image.

As in Equation 2, φ ₀ is the input shape dictionary information and φ is the split curve at the current time. If the shape of the current segmentation curve and the shape dictionary information is the same, the energy of the shape dictionary information term is not generated. When the shape of the split curve and the shape dictionary information is different, energy is generated and the split curve is transformed into a form that generates energy, that is, in this case, the split curve is transformed into a form that generates energy so as to be the same as the shape dictionary information. . By repeatedly deforming the shape of the dividing curve, a shape similar to the shape of the input shape dictionary information is divided. That is, the breathing-related region is separated by using the shape dictionary information input in the acquired depth image by the level set method using the shape dictionary information.

In the present invention, in the depth image that has undergone the preprocessing process, only the region affected by respiration is separated using the level set method, but the level set method is a method of combining a region information function and a prior information function Using a breathing zone separation process.

In general, a level set refers to a set formed by a whole body belonging to a domain that allows a given image to obtain a predetermined value. For example, for a real-valued function f of n-variables, the equivalence set for real-valued c is given by Lc(f)= {(x1,…,Xn)┃f(x1,…,Xn) =c} . In the case of two variables, the isoset set is called a level curve, contour line, isoline, etc. by drawing a curve, and in the case of three variables, the isoset set is a level surface, isoline, etc. It is called an isosurface, and the higher-dimensional case is called a level hypersurface.

In addition, the division of a predetermined region by applying the region-based level set method according to the Chan & Vese model (CVM) and the region information function is a well-known technique, and a detailed description thereof will be omitted. For example, it is known in "Level set segmentation with both shape and intensity priors" published by IEEE in 2009 by Siqi Chen, Radke R.J.

The present invention is to separate only a region affected by respiration in a depth image using the level set method. In general, the major body parts involved in breathing in the body are the airways, lungs, and blood vessels connected to the lungs, and muscles involved in breathing. Among these organs, the muscles involved in breathing cause changes that are observable in the camera. The muscles involved in breathing include the diaphragm, intercostal muscles, and muscles that assist in breathing. During the breathing process, at the beginning of inspiration, the chest cavity is enlarged by contraction of the diaphragm and intercostal muscles. When the dome-shaped diaphragm contracts, its shape flattens and the intercostal muscles lift the ribs and expand the volume of the rib cage. At this time, as the diaphragm contracts and flattens, some of the organs in the abdominal cavity are pushed out. As the displaced organs are supported by the spine in the back and the pelvic bones below, they are pushed toward the sides and abdomen, causing the abdomen to swell. Conversely, during exhalation, the intercostal muscles relax and the ribs descend and the diaphragm relaxes and rises to its original position, reducing the volume of the chest cavity and returning the swollen abdomen due to contraction of the diaphragm. This change in volume during respiration shows a change that can be observed in the depth camera. Therefore, in the present invention, in the depth image, the ribcage and abdominal regions whose volume is changed by the muscles related to respiration are defined as respiration-related regions.

The adaptive region information function will be further described. FIG. 7 is an explanatory diagram for explaining the adaptive region information function using the breathing-related region in the operation processing unit of FIG. 2 , and FIG. 8 is the breathing-related region in the operation processing unit of FIG. It is an explanatory diagram for explaining the generation process of the area information function using

The left coordinate in (a) of FIG. 8 indicates the degree of equality, that is, as a result of applying the level set division method using shape dictionary information, the area indicated by 1 is a separated area. Here, when a signed distance function (SDF) is used, the boundary line of FIG. 8 (a) is set as 0 as a reference, and as the distance from the reference is increased, the inner area has a positive value and the outer area has a negative value. . This is expressed in Figure 8 (b). In (b) of FIG. 8 , a value exceeding a predetermined threshold value in the inner region is treated as the same value as shown in FIG. 8 ( c ). (d) of FIG. 8 is a result of transforming the external region to decrease with a steep slope using the Heaviside function, and FIG. 8 (d) is used as the final region information function. A positive value in (d) of FIG. 8 means that it is regarded as a respiration-related area and that when performing level set division, force is applied to separate the respiration area. Referring to (d) of FIG. 8 , the separated respiration-related area has a positive value, and the external value has a larger negative value as the distance from the respiration-related area increases.

If you look at the difference image, you can see that the difference in depth value occurs in areas other than the breathing area due to arm movement or noise and is displayed in white. When dividing by Chan&Vese's method, all these regions are divided, but when level set division using the region-based information function using Fig. 8(d), the internal region is divided except the external region and combined with Chan&Vese's energy function, Areas where there is no change in the area are excluded from the partitioned area.

The present invention applies a level set method using region-based information using a difference image array in which a difference is calculated from a depth image array and a region information function, and as a result, a level set method using region-based information Through this, the respiration area, which is an area affected by respiration, is separated from the depth image. That is, in the present invention, the level set method using shape dictionary information means that the chest and abdomen regions, which are the breathing-related regions defined above, are separated, and the breathing region is a region in which a change in volume occurs during actual breathing. The change appears as a change in the depth value. Therefore, when the difference image of the depth image is calculated according to the time series, a place with a change in volume is displayed in white, and a place where there is no volume change is displayed in black. However, as shown in (a) of FIG. 6 , it can be seen that other regions are also displayed in white due to noise and arm movement. Here, the region affected by respiration means a region in which a change in depth value appears in the chest and abdomen in the difference image of the depth image.

In the present invention, the divided respiration area is used as prior information when segmenting the respiration area in the next difference image, and the weight is reflected when generating the area information function to generate an adaptive area information function. That is, if the breathing region is divided in the difference image, the result is as shown in (a) of FIG. 7 . As shown in FIG. 7(b), it can be seen that a specific weight is generated in the divided region by calculating the weight when dividing the breathing region in the next difference image by using (a) of FIG. 7 . 7 (c) is a breathing-related area detected by the level set method using shape prior information, and when a signed distance function (SDF) and a unit step function (Heaviside) function are applied to this breathing-related area, FIG. (d), the adaptive area information function is detected by reflecting the weight and the result of applying the area information function to the shape prior information in FIG. 7(d).

In the depth image, the volume change due to respiration does not occur suddenly, but the change gradually appears and disappears in a specific area (chest, abdomen) and repeats. In order to reflect these characteristics, an adaptive region information function is generated using the result of previously divided into respiration regions as weights. If the weights are not reflected, (d) in FIG. 7(c) is calculated and a region information function is generated using it. This reflects most likely the breathing zone.

In the present invention, successive depth images are acquired, and a difference image of each depth image is calculated according to a time series. The difference image calculated for each successive depth image in this way generates a difference image array. For each difference image, a breathing region is divided using a level set method using region-based information. In other words, the respiration region is separated by repeating the series of steps for all secondary image sequences. That is, the breathing area for the difference image is separated.

In the depth image preprocessed by the preprocessing process, only the area affected by respiration is separated using the level set method. In this case, a level set process using shape prior information, an adaptive region information function generation process, and a level set process using region-based information are sequentially performed.

Next, the respiratory area separation process using ResU-net (Residual U-net, residual U-net) will be described.

This breathing area separation process uses a deep learning technique to segment only the respiration-related area in the depth image acquired from the depth camera. That is, this respiration region separation process uses the ResU-net technique, which is a mixture of U-net, which is often used in segmentation tasks, and Resnet (residual net), which is a method of deeply stacking convolution layers.

U-Net is one of deep learning technologies. It is a convolutional neural network developed for medical image segmentation, and has a U-shaped network structure. ResU-net (Residual U-net) is the addition of the Residual Network function to U-Net, which increases the precision and accuracy of medical image segmentation. ResU-net (Residual U-net) is a well-known technology and a detailed description thereof will be omitted. For example, about ResU-net (Residual U-net), it is known in "RGB-NIR Demosaicing Using Deep Residual U-Net" published to IEEE by Ivana Shopovska et al. in 2019, and also in January 2019 It is known in "Recurrent residual U-Net for medical image segmentation" published by Alom MZ et al. in J. of Medical Imaging, and also in the Journal of Medical Imaging in January 2019 (J. of Medical Imaging). ) in "Recurrent Residual Convolutional Neural Network based on U-Net (R2U-Net) for Medical Image Segmentation" published by Alom MZ et al.

The operation processing unit 210 of the image analysis unit 200 uses ResU-Net to segment a breathing region in a spatial domain, a multiplication process between a segmentation map and a difference image array, and a breathing region in a time domain using ResU-Net It includes the process of repeating the division process several times.

In the respiration domain segmentation process in the spatial domain using ResU-Net (Spatial Segmentation using ResU-Net), as an input of ResU-Net, the preprocessed depth image (array) received from the memory unit 220 is input, As the output of ResU-Net, a segmentation map of the respiration area (the boundary of the respiration area) in the spatial domain, that is, a segmentation map obtained by separating only the human body image is output. That is, by separating only the body of a person from the depth image, only the breathing area is extracted from the spatial domain.

The artificial neural network (first ResU-Net) applied to the respiration region segmentation process in the spatial domain using ResU-Net separates only the human body region from the preprocessed depth images (arrays) and the depth image. Using an image as a label, it may be a machine-learned artificial neural network, which may have been learned, and as shown in FIG. 14 , the contracting path, which is a part that gradually reduces the image, and the extension path, which is a part that grows the image ( expanding path).

The multiplication process between the segmentation map in which only the body image is separated and the difference image is the output of the breathing region segmentation process in the spatial domain using ResU-Net, that is, the segmentation map in which only the human body image is separated, and the memory unit 220 . Multiply the difference image received from the pixel unit.

In ResU-Net's time segmentation using ResU-Net, as an input of ResU-Net, the output of the multiplication process between the segmentation map separated only the body image and the difference image (that is, the Segmentation Map of the respiration area (the boundary of the respiration area) in the time domain for this input as an output of ResU-Net, that is, as an output of ResU-Net A segmentation map is output in which only a region related to respiration is separated from the difference image.

Here, the arithmetic processing unit 210 multiplies the difference image received from the memory unit 220 and the image obtained by separating only the body region in pixel units and uses it as an input of ResU-Net again, which is in the difference image. The area corresponding to the body area enters the input of ResU-Net. And the image output from ResU-Net is an image in which only the respiration-related area is separated from the secondary image.

ResU-Net (Second ResU-Net) of the respiration region segmentation process in the time domain using ResU-Net is, in advance, the image corresponding to the body region from the difference image (that is, the difference image and the body region only). It may be machine learning by using as a label an image obtained by multiplying an image by pixel) and an image in which only a region related to respiration is separated from the secondary image, and as shown in FIG. 15, the contraction path, which is a part that gradually reduces the image It has a contracting path and an expanding path, which is a part that grows an image.

The label used in machine learning generates a label image using the above-described level set method.

In this way, the process of repeating the respiratory region segmentation process in the spatial domain using ResU-Net, the multiplication process between the segmentation map that separated only the body image and the difference image, and the respiratory region segmentation process in the temporal domain using ResU-Net several times is repeated. Through this, a segmentation map is finally generated in which a region related to respiration is separated from the difference image, and the respiration amount is calculated using this.

As a result, in the respiratory region separation process using the Level Set method, or in the respiratory region separation process using ResU-net (Residual U-net, residual U-net), the breathing region for the difference image is separated. .

In the respiratory zone separation process using the level set method, or in the respiratory zone separation process using ResU-net (Residual U-net, residual U-net), apply the separated breathing zone to the respiratory volume calculation formula to generate a respiratory volume waveform (Volume Waveform) is detected, and the respiration rate and respiration rate are calculated by finding the peak and valley in the respiration rate waveform. For example, by summing the depth values of the breathing area in each image (each frame), the summed depth value for each frame is obtained as a respiration-related signal for each frame, and the preceding pre-M respiration-related signal is subtracted from the current frame respiration-related signal. to obtain as the amount of change in respiration per frame, multiply the obtained amount of change in respiration per frame by the unit volume value of a preset pixel to obtain the amount of change in respiration per frame converted to the actual size, and the converted amount of respiration per frame by the memory unit 220 ) can be sequentially stored, and the stored respiration conversion amount for each frame can be read in chronological order and displayed as a graph to indicate a respiration waveform.

Here, since the respiration variation for each frame is obtained using pixels, it is necessary to convert the respiration variation for each frame into an actual size. In order to convert the respiration variation for each frame into an actual size, the respiration variation for each frame is obtained by multiplying the unit volume of the pixel (ie, the actual volume of the pixel). The unit volume of a pixel (ie, the actual volume of the pixel) is a value obtained by multiplying the actual horizontal length and the actual vertical length of the pixel (ie, the actual horizontal length and the actual vertical length represented by the pixel).

Multiply the amount of change in respiration per frame to obtain the amount of change in respiration per frame converted to the actual size.

The operation processing unit 210 obtains an inflection point by first differentiating the respiratory signal, and detects a peak and a valley therefrom, and from the detected peaks and valleys, for a predetermined time interval. At (per second, or per minute, or per 30 seconds), the number of consecutive peak-valley pairs, ie, the number of peak-to-valley changes, is detected as a respiration rate (RR).

Respiratory volume can be calculated in two ways.

The first method detects the amplitude of a pair of consecutive peaks and valleys in the respiration signal, that is, the amplitude of the signal changing from peak to valley, as the respiration volume of the cycle.

In the second method, one respiration cycle is taken from one peak to the next consecutive peak in the respiration waveform, and the respiration signal of each respiration cycle is subtracted from the respiration signal of all consecutive respiration cycles to obtain the change amount for each respiration cycle, and for each respiration cycle Respiratory volume is calculated by adding up the changes. In some cases, the respiratory cycle may be a time interval from one peak-valley pair to a successive previous peak-valley pair.

That is, the amount of change for each respiration cycle is obtained by subtracting the respiration signal (each sample) of the current respiration cycle from each respiration signal (each sample) corresponding to the previous respiration cycle in succession. For example, the first respiration signal (sample) of the current respiration cycle is subtracted from the first respiration signal (sample) of the current respiration cycle to obtain the first respiration change of the current respiration cycle, and thus, from the start sample of each respiration cycle to the last sample By applying it, the amount of change in respiration in each cycle is calculated. Respiratory volume is obtained by adding up the amount of change in respiration of each cycle obtained in this way for each time period. Here, summing the respiration variation of each cycle by time period means that the first respiration variation signal of each cycle is summed as the first respiration volume signal, and the second respiration variation signal of each cycle is summed as the second respiration volume signal, and in this way, each cycle Respiratory volume is calculated by summing the respiratory change signals of

The memory unit 220 stores the respiration amount and respiration rate received from the calculation processing unit 210 .

The output unit 230 outputs the respiration amount and respiration rate received from the calculation processing unit 210 .

The key input unit 250 includes a start/stop switch and the like.

9 is a flowchart schematically illustrating a method of detecting a respiration volume by using a depth image received from a depth camera in the image analysis unit 200 of FIG. 2 and applying a level set method, and FIG. It is a flowchart explaining the step of separating the respiratory zone in 9.

In the initialization step (S10), the counter and the like are initialized.

In the depth image receiving step (S20), the operation processing unit 210 of the image analysis unit 200 receives the depth image of the subject's upper body from the depth camera 100 and temporarily stores it in the memory unit 220 . In the depth image receiving step S10, a depth image is received for each frame, but a depth image may be received one frame at a time, or a depth image of frames for a preset time may be received.

In the spatiotemporal median filtering step ( S110 ), median value filtering is performed on the time-dependent depth values of each pixel in successive depth images using a median filter, and also in the space of each pixel in successive depth images. The median value filtering is performed on the depth values according to the median value filter using the median value filter, and median value filtering is performed in the temporal and spatial domains.

In the spatio-temporal clipping step (S120), in the successive depth images that have been median-filtered in the temporal and spatial domains in the spatiotemporal median filtering step, the depth values according to time of each pixel are clipped based on the time domain clipping threshold, In addition, clipping is performed based on the spatial domain clipping threshold in the depth values according to the space of each pixel of the successive depth images, and as a result, the clipping is performed in space and time.

That is, in the spatio-temporal clipping step S120, in successive depth images that have been median-filtered in the temporal and spatial domains in the spatiotemporal median filtering step, the depth values in the temporal domain and the spatial domain are the temporal domain clipping threshold and the spatial domain clipping threshold. In the case of more than one, the depth value is set to 0.

Here, instead of the spatiotemporal median filtering step S110 and the spatiotemporal clipping step S120 , the temporal domain median value filtering and clipping step and the spatial domain median value filtering and clipping step may be performed.

In the temporal domain median filtering and clipping step, the median value filtering is performed on the time-dependent depth values of each pixel in successive depth images by using the median filter, and, in the successive depth images, the median value filtering is performed. Depth values according to ? are clipped based on the time domain clipping threshold. Here, in the temporal domain median filtering, pixel values of a specific region are read from successive images, arranged according to time, and median value filtering is performed on the listed pixel values. In the time domain clipping, pixel values of a specific area are read from successive images, arranged according to time, and clipping is performed from the listed pixel values according to a time domain clipping threshold.

In the step of filtering and clipping the median value in the spatial domain, the median value filtering is performed on the depth values according to the space of each pixel in the continuous depth images using the median value filter, and the space of each pixel in the continuous depth images is performed by using the median value filter. The depth values according to ? are clipped based on the spatial domain clipping threshold. Here, in the spatial domain median filtering, pixel values of a specific region of each frame are read from successive images, arranged according to the space of the image for each frame, and from the pixel values listed in each frame (that is, each frame within) median filtering. And clipping of the spatial domain reads the pixel values of a specific region of each frame from successive images and arranges them according to the space of the image for each frame, and the spatial domain from the pixel values listed in each frame (that is, within each frame) Clipping is performed according to the clipping threshold.

In the present invention, the spatiotemporal median filtering step and the spatiotemporal clipping step (or the temporal domain median filtering and clipping and the spatial median filtering and clipping step) may be referred to as a preprocessing step S100 .

In the difference image arrangement step (S150), in the images (frames) output from the spatiotemporal median value filtering step, the difference in the depth value of a predetermined pixel in temporally consecutive images, that is, in the depth value of the pixel in the current image, A difference image for each frame is obtained by subtracting a depth value of a corresponding pixel of a successive previous image, and the obtained difference image for each frame is sequentially arranged and stored in the memory unit 220 .

In the level setting step ( S210 ) using the shape prior information, a level set for separating the rib cage and the abdomen region from the depth image is performed using pre-stored shape dictionary information (ie, a shape of a body of a certain shape). This isolates only the respiratory-related areas, i.e., the thoracic and abdominal areas.

In the region information function acquisition step (S220), the image of the previous respiration-related region is read in succession, the weight is calculated using this, and the weight is reflected in the shape prior information to which the region information function is applied, and the adaptive region information function is detected do. Here, the application of the area information function is that when a signed distance function (SDF) is applied to the area separated using shape dictionary information, the boundary line is set as 0, and the further away from the reference is, the more The inner region has a positive value and the outer region has a negative value. In the region to which the SDF function is applied, a value exceeding a certain threshold (or a value within a predetermined size unit) is treated as the same value. Using the unit step function (Heaviside) function, the outer region is transformed so that a value that is constantly decreasing is decreased with a steep slope.

As a level set step (S230) using a region information function, a level set (S230) for separating the respiratory region using a region information function and a CVM (Chan-Vese model) technique on the difference image obtained in the difference image acquisition step ( area-based level set).

In the determination step (S250) of whether or not the breathing zone separation process is finished, it is checked whether the breathing zone separation process is repeated as many as the preset number of difference image sequences, and if not, the counter is incremented by 1 and the level setting step using the shape advance information (S210) Return, if the breathing area separation process is repeated as many as the preset number of difference image arrangement, it goes to the breathing waveform detection step (S310).

The level setting step (S210) using the shape prior information, the area information function acquisition step (S220), the level set step using the area information function (S230), and the respiratory area separation process end determination step (S250) are the breathing area separation steps ( S200).

In the respiration waveform detection step (S310), the depth value of the respiration region in each image (each frame) is summed to obtain a respiration-related signal for each frame, and the preceding pre-em respiration-related signal is subtracted from the current frame respiration-related signal. Respiratory change for each frame is obtained as a per-frame change amount, and the obtained frame-by-frame respiration change amount is multiplied by the unit volume value of a preset pixel to obtain a respiration change amount for each frame converted to the actual size, and the converted frame-by-frame respiration conversion amount is stored in the memory unit 220 They are stored sequentially, and the respiration conversion amount for each frame is read in chronological order and displayed as a graph to indicate the respiratory waveform.

In the respiration volume calculation step (S320), the respiration rate and respiration rate are calculated by finding a peak and a valley in the respiration waveform.

11 shows an example of the breathing zone separation of the existing method and an example of the breathing zone separation of the present invention, and FIG. 12 is a breathing zone separation method of the existing method in a normal environment, the breathing zone separation method of the present invention, and the respirator (ventilator) is a comparative example of the respiratory volume waveform when each is applied, Figure 13 is the breathing volume in the case of applying the breathing zone separation method of the existing method, the breathing zone separation method of the present invention and the ventilator in an environment with arm movement, respectively It is a comparative example of a waveform.

11 (a) is a case in which the breathing region is separated by applying the existing region separation method, and the breathing region is separated by positioning the upper body of a person in a fixed square box in the depth image. The existing method has a limitation in that the respiration rate error increases when there is movement of the upper body or arm because the separation area is fixed.

11 (b) is a case in which the breathing region is separated by applying the present invention, and by using shape dictionary information and region-based information, only a portion in the depth image where there is a change due to respiration is separated into the breathing region, and the existing method limitations can be overcome.

Since the respiration zone separation method according to the present invention calculates the respiration volume by separating only the part that is actually changed by respiration, it can measure the respiration volume more accurately than the existing method of calculating respiration volume using the overall upper body area.

In a normal (momal) environment and an environment with arm movement, an experiment was conducted to compare the breathing zone separation method of the existing method, the breathing zone separation method of the present invention, and a ventilator, respectively. As a result of comparing the accuracy by calculating the respiration volume for the existing method and the proposed method for 10 subjects, the respiration volume error of the existing method is 14.10% with respect to the actual respiration volume, the method of the present invention is 8.41%, the method of the present invention is the existing method It was confirmed that the respiratory volume was measured more accurately.

16 is a flowchart schematically illustrating a method of detecting a respiration volume by using a depth image received from a depth camera in the image analysis unit 200 of FIG. 2 and applying a ResU-net (Residual U-net) method.

Except for the breathing area division step (S200), the rest is mostly the same as in FIG.

In the initialization step (S10), the counter and the like are initialized.

In the depth image receiving step (S20), the operation processing unit 210 of the image analysis unit 200 receives the depth image of the subject's upper body from the depth camera 100 and temporarily stores it in the memory unit 220 . In the depth image receiving step S10, a depth image is received for each frame, but a depth image may be received one frame at a time, or a depth image of frames for a preset time may be received.

In the spatiotemporal median value filtering step S110 , the operation processing unit 210 performs median value filtering on the time-dependent depth values of each pixel in the continuous depth images using the median filter, and also, the continuous depth images. Median filtering is performed on the depth values according to the space of each pixel in the field using the median filter, and median filtering is performed in the temporal and spatial domains.

In the spatio-temporal clipping step S120 , the operation processing unit 210 sets the time-dependent depth values of each pixel in the temporal and spatial domain median-filtered successive depth images in the spatiotemporal median value filtering step as a time domain clipping threshold. Clipping is performed based on , and clipping is performed based on the spatial domain clipping threshold at the depth values according to the space of each pixel of the successive depth images, and as a result, the clipping is performed in space and time.

That is, in the spatio-temporal clipping step S120, in successive depth images that have been median-filtered in the temporal and spatial domains in the spatiotemporal median filtering step, the depth values in the temporal domain and the spatial domain are the temporal domain clipping threshold and the spatial domain clipping threshold. In the case of more than one, the depth value is set to 0.

In the present invention, the spatiotemporal median filtering step and the spatiotemporal clipping step (or the temporal domain median filtering and clipping and the spatial median filtering and clipping step) may be referred to as a preprocessing step S100 .

In the depth image arrangement step S130 , the operation processing unit 210 sequentially arranges and stores the output of the spatio-temporal clipping step S120 , that is, the preprocessed continuous depth images in the memory unit 220 .

In the difference image arrangement step ( S150 ), the operation processing unit 210 , in the images (frames) output from the spatiotemporal median value filtering step, the difference in depth values of predetermined pixels in temporally consecutive images, that is, the current image. From the depth value of the pixel, the depth value of the corresponding pixel of the consecutive image is subtracted to obtain a difference image for each frame, and the obtained difference image for each frame is sequentially arranged and stored in the memory unit 220 .

In the respiration region division step (S215) in the spatial domain using ResU-Net, the operation processing unit 210 receives the preprocessed depth image stored in the depth image arrangement step (S130) from the memory unit 220, and the second 1 Input to ResU-Net, and receive a segmentation map in which only the human body image is separated from the first ResU-Net.

In the temporary storage step (S225) of the segmentation map in which only the body image is separated, the operation processing unit 210 temporarily stores the segmentation map in which only the body image of the person received from the first ResU-Net is separated in the memory unit 220 .

The artificial neural network (1st ResU-Net) applied to the respiration region segmentation process in the spatial domain using ResU-Net labels pre-processed depth images and images that separate only the human body region from the depth images. It is a machine-learning artificial neural network using

A label used in machine learning may generate a label image using the level set method described above, that is, a level set using shape prior information.

In the multiplication step (S227) of the segmentation map in which only the body image is separated and the difference image, the output of the breathing region segmentation process in the spatial domain using ResU-Net, that is, in the segmentation map in which only the human body image is separated, the memory unit The difference image received from 220, that is, the difference image obtained in the difference image arrangement step S150, is multiplied in units of pixels.

In the respiratory region segmentation step (S235) in the time domain using ResU-Net, as an input of the second ResU-Net, the output of the multiplication step (S227) between the segmentation map in which only the body image is separated and the difference image (that is, the human A segmentation map obtained by separating only the body image and the product of the difference image) is input, and as the output of ResU-Net, a segmentation map in which only the region related to respiration is separated from the difference image is output.

Here, the arithmetic processing unit 210 multiplies the difference image received from the memory unit 220 and the image in which only the body region is separated, in units of pixels, and again ResU-Net (ie, the second ResU-Net) of It is used as an input, which means that the region corresponding to the body region in the car image enters the ResU-Net input. And the image output from ResU-Net (ie, the second ResU-Net) is an image in which only the region related to respiration is separated from the difference image.

ResU-Net (Second ResU-Net) of the respiration region segmentation process in the time domain using ResU-Net is, in advance, the image corresponding to the body region from the difference image (that is, the difference image and the body region only). It is machine-learning using an image obtained by multiplying an image by pixel) and an image in which only a region related to respiration is separated from a difference image as a label.

A label used in machine learning may generate a label image using the level set method described above, that is, a level set using the area information function.

As a step of determining whether the respiratory zone separation process is complete, it is checked whether the respiratory zone separation process is repeated as many as the preset number of secondary image sequences (S250), and if not, the counter is incremented by 1 (S260), and in the spatial domain using ResU-Net returns to the breathing region division step (S215) of, and if the breathing region separation process is repeated as many as the preset number of difference image sequences, it goes to the breathing waveform detection step (S310).

Respiratory region segmentation step in the spatial domain using ResU-Net (S215), temporary storage step of segmentation map separated only torso image (S225), multiplication step between segmentation map separated only torso image and difference image (S227), ResU -Respiratory region dividing step (S235) in the time domain using Net and determining whether or not the respiratory region separation process is finished (S250) can be referred to as breathing region separation step (S200).

In the respiration waveform detection step (S310), the depth value of the respiration area in each image (each frame) (that is, the depth value of the respiration area in the image in which only the respiration-related area is separated from the difference image) is added to the respiration-related signal for each frame It is obtained by subtracting the preceding pre-M respiration-related signal from the current frame respiration-related signal to obtain the respiration change for each frame, and by multiplying the obtained respiration change per frame by the unit volume value of the preset pixel, converted to the actual size Respiratory change for each frame is obtained, the converted amount of respiration for each frame is sequentially stored in the memory unit 220, and the respiration conversion amount for each frame is read in chronological order and displayed as a graph to indicate a respiration waveform.

In the respiration volume calculation step (S320), the respiration rate and respiration rate are calculated by finding a peak and a valley in the respiration waveform.

As described above, although the present invention has been described with reference to the limited examples and drawings, the present invention is not limited to the above examples, which are various modifications and variations from these descriptions by those skilled in the art to which the present invention pertains. Transformation is possible. Therefore, the spirit of the present invention should be understood only by the claims described below, and all equivalents or equivalent modifications thereof will fall within the scope of the spirit of the present invention.

50: subject 100: depth camera
200: image analysis unit 210: calculation processing unit
220: memory unit 230: output unit
250: key input unit

Claims (24)

a pre-processing step in which an operation processing unit receives a depth image array of the subject's upper body from a depth camera, and removes noise including impulse noise from the received depth image arrays;
The arithmetic processing unit obtains a difference image by subtracting the difference between the depth values of the corresponding pixels from the temporally consecutive previous frames from the depth values of each pixel of each frame of the depth image array from which the noise has been removed in the preprocessing step, and the memory unit stored sequentially in . difference image arrangement step;
The operation processing unit, in the difference image obtained in the difference image arrangement step, proceeds to level set using the shape prior information and the region information function, to separate the breathing region of the difference image, a breathing region separation step;
In the method for measuring respiration volume using a depth camera comprising:
Respiratory zone separation step,
a level setting step using the shape dictionary information, in which the operation processing unit separates the rib cage and the abdomen region from the depth image by using the shape dictionary information for separating only the torso region, which is the rib cage and abdomen region;
The operation processing unit calculates a weight having a value between 0 and 1 by using the image of the respiration-related region of the previous frame in succession, and applies the weight to the shape dictionary information to which the region information function is applied, thereby adaptive region information an area information function obtaining step of detecting a function;
A level set using a region information function, where the operation processing unit separates the breathing region of the difference image by using a region information function and a CVM (Chan-Vese model) technique on the difference image obtained in the difference image arrangement step step;
Respiratory volume measurement method using a depth camera, comprising: a pre-processing step in which the arithmetic processing unit receives a depth image array of the subject's upper body from a depth camera, and performs preprocessing for removing noise including impulse noise from the received depth image arrays;
The operation processing unit may include: a depth image arranging step of sequentially storing continuous depth images from which noise has been removed in the preprocessing step in the memory unit;
The arithmetic processing unit obtains a difference image by subtracting the difference between the depth value of the pixel in the temporally consecutive previous frames from the depth value of each pixel of each frame of the continuous depth image array from which the noise has been removed in the preprocessing step, stored sequentially in the memory unit. difference image arrangement step;
The calculation processing unit applies the difference image obtained in the difference image arrangement step and the continuous depth image stored in the memory unit in the depth image arrangement step to the pre-learned ResU-net (Residual U-net) to determine the breathing region of the difference image. Separating, breathing zone separation step;
Respiratory volume measurement method using a depth camera, comprising: delete 3. The method according to any one of claims 1 to 2, wherein the pretreatment step comprises:
a spatiotemporal median filtering step in which the arithmetic processing unit performs median value filtering on the depth values of each pixel according to the time domain in successive depth images, and performs median value filtering on the depth values of each pixel according to the spatial domain;
The operation processing unit performs clipping based on the temporal domain clipping threshold at the depth values of each pixel according to the temporal domain in successive depth images that have been median-filtered in the temporal and spatial domains in the spatiotemporal median filtering step, and a spatiotemporal clipping step of performing clipping based on a spatial domain clipping threshold at depth values of each pixel;
Respiratory volume measurement method using a depth camera, comprising: The method of claim 1, wherein the obtaining of the area information function comprises:
When the operation processing unit applies a signed distance function (SDF) to the region separated using the shape dictionary information, and the boundary line is set as 0, the further away from the criterion, the more the separated region The inner region has a positive value, the outer region of the separated region has a negative value, and the operation processing unit returns a value exceeding a certain threshold or within a predetermined size unit in the region to which the SDF function is applied. Respiratory volume measurement method using a depth camera, characterized in that it includes applying a unit step function (Heaviside) function to the external area, The method of claim 2, wherein the step of separating the breathing area
The arithmetic processing unit receives the preprocessed depth image stored in the depth image arrangement step (S130) from the memory unit, inputs it to the pre-learned first ResU-Net, and outputs from the first ResU-Net, ribcage and abdomen region Respiratory region segmentation step in the spatial region using ResU-Net, receiving a segmentation map in which only the in-body region is separated;
The arithmetic processing unit may include: a segmentation map arrangement step in which only the torso image is separated, in which the segmentation map obtained by separating only the torso image of the person received from the first ResU-Net is sequentially temporarily stored in the memory unit;
The arithmetic processing unit separates only the torso image by multiplying the segmentation map in which only the torso image is separated, output in the breathing region segmentation step in the spatial domain using ResU-Net, by the difference image obtained in the difference image arrangement step in units of pixels a multiplication step between a segmentation map and a difference image;
The arithmetic processing unit inputs, as an input of the previously learned second ResU-Net, the product of the segmentation map in which only the body image is separated and the segmentation map in which only the torso image is separated, which is output in the multiplication step of the difference image, and the difference image, 2 As the output of ResU-Net, receiving an image in which only a respiration-related area is separated from the difference image, which is a segmentation map in which only a respiration-related area is separated from the second image, is received, a respiration area segmentation step in the time domain using ResU-Net;
Respiratory volume measurement method using a depth camera, comprising: 7. The method of claim 6,
The first ResU-Net is a machine-learning artificial neural network using, as labels, depth images from which noise has been removed in the pre-processing step, and images obtained by separating only the body region from the depth images from which noise has been removed. Respiratory volume measurement method using a depth camera, 8. The method of claim 7,
The second ResU-Net is an image obtained by multiplying the segmentation map in which only the body image is separated and the difference image output in the multiplication step of the difference image in advance, and the image in which only the body region is separated, in units of pixels, and the difference image Respiratory volume measurement method using a depth camera, characterized in that it is a machine-learned artificial neural network by using an image that isolates only the region related to respiration in the label. 9. The method of claim 8,
The label used in machine learning of the first ResU-Net is a respiration volume measurement method using a depth camera, characterized in that the label image is generated using a level set using shape prior information, 10. The method of claim 9,
The label used in machine learning of the second ResU-Net is a respiration volume measurement method using a depth camera, characterized in that the label image is generated using a level set using the area information function; 7. The method of claim 6,
The operation processing unit checks whether the value of the counter is equal to the preset number of difference image sequences, and if not, increments the counter by 1, and returns to the breathing region division step in the spatial region using ResU-Net, breathing region separation determining whether the process has been completed;
Respiratory volume measurement method using a depth camera, characterized in that it further comprises, 6. The method of claim 5,
The calculation processing unit checks whether the value of the counter is equal to the preset number of difference image arrays, and if not, increments the counter by 1, and returns to the level set step using the shape prior information, determining whether the breathing area separation process ends ;
Respiratory volume measurement method using a depth camera, characterized in that it further comprises, 13. The method of any one of claims 11 or 12,
In the step of determining whether the respiration zone separation process ends, if the value of the counter is equal to the preset number of difference image sequences, the calculation processing unit,
In the frames of the images obtained in the respiration region separation step, in which only the breathing-related region is separated from the secondary image, the depth value of the breathing region of each frame is summed to obtain a respiration related signal for each frame, and the current frame respiration related signal Subtracts the previous pre-M respiration-related signal in succession to obtain the respiration change for each frame, multiply the respiration change per frame by the unit volume value of the preset pixel to obtain the respiration change for each frame converted to the actual size, and for each converted frame Respiratory waveform detection step, representing the respiratory waveform by displaying the respiration conversion amount in a graph according to the time sequence;
Respiratory volume measurement method using a depth camera, characterized in that it further comprises, 14. The method of claim 13,
Respiratory rate and respiration rate detecting step, wherein the operation processing unit detects a peak and a valley in the respiration waveform, and calculates the respiration rate and respiration rate using the detected peak and valley;
Respiratory volume measurement method using a depth camera, characterized in that it further comprises, 15. The method of claim 14, Respiratory rate and respiration volume detection step
A peak and valley detection step in which an operation processing unit first differentiates the respiration signal obtained in the respiratory waveform detection step to obtain an inflection point, and detects a peak and a valley using the inflection point;
In the peak and valley detected in the peak and valley detection step, the number of pairs of peaks and valleys in which the peaks and valleys appear one after another during a predetermined time interval, as the respiration rate Detecting, respiratory rate detection step;
Respiratory volume measurement method using a depth camera, comprising: According to claim 15, Respiratory rate and respiration volume detection step,
a respiration amount detecting step in which the calculation processing unit detects, by the operation processing unit, the amplitude of the pair of peaks and valleys in which the peaks and valleys appear sequentially in succession as the respiration volume of the respiration cycle including the pair of peaks and valleys;
Respiratory volume measurement method using a depth camera, comprising: According to claim 15, Respiratory rate and respiration volume detection step,
The calculation processing unit detects the respiration cycle as a pair of peaks and valleys in which peaks and valleys appear sequentially in succession, and subtracts the respiration signal of each respiration cycle from the respiration signal of the previous respiration cycle in succession to obtain the change amount for each respiration cycle, , Respiratory volume detection step of calculating the amount of change for each respiration cycle over time to obtain the respiration volume;
Respiratory volume measurement method using a depth camera, comprising: 10. The method of claim 9,
Respiratory volume measurement method using a depth camera, characterized in that when the amount of change for each respiration cycle is summed over time, the calculation processing unit adds up based on the starting point of each respiration cycle, [Claim 13] The recording medium storing a computer program source for the method for measuring respiration volume using the depth camera according to any one of claims 1 to 2, 5 to 12. In a respiration measurement system using a depth camera comprising an image analysis unit comprising a depth camera, an operation processing unit for detecting a respiration signal by analyzing the depth image of the subject's upper body received from the depth camera,
arithmetic processing unit,
receiving the depth image array of the subject's upper body from the depth camera, and performing preprocessing for removing noise including impulse noise from the received depth image arrays;
In the preprocessed depth image array, the difference image array is obtained by subtracting the difference between the depth values of the corresponding pixels in the previous temporally successive frames from the depth value of each pixel of each frame, and sequentially stored in the memory unit,
In the difference image arrangement of the depth image, the breathing region is separated by performing level set using the shape prior information and the region information function,
In order to separate the breathing region by performing level set, the operation processing unit,
In the difference image arrangement of the depth image, the body region, which is the rib cage and the abdomen region, is separated from the depth image by using the shape dictionary information for separating only the rib cage and the abdomen region.
Calculating a weight having a value between 0 and 1 using the image of the respiration-related region of the previous frame in succession, applying the weight to the shape dictionary information to which the region information function is applied, and detecting the adaptive region information function,
A respiration measurement system using a depth camera, characterized in that the respiration area of the difference image is separated using a region information function and a CVM (Chan-Vese model) technique in the difference image arrangement. In a respiration measurement system using a depth camera comprising an image analysis unit comprising a depth camera, an operation processing unit for detecting a respiration signal by analyzing the depth image of the subject's upper body received from the depth camera,
arithmetic processing unit,
Receives the depth image array of the subject's upper body from the depth camera, performs preprocessing for removing noise including impulse noise from the received depth image arrays, and sequentially stores the preprocessed continuous depth image array in the memory unit, ,
In the preprocessed depth image array, the difference image array is obtained by subtracting the difference between the depth values of the corresponding pixels in the previous temporally successive frames from the depth value of each pixel of each frame, and sequentially stored in the memory unit,
Using a depth camera, characterized in that the difference image arrangement and the pre-processed continuous depth image received from the memory unit are applied to the pre-learned ResU-net (Residual U-net) to separate the breathing region of the difference image Respiratory measurement system. delete 22. The method of claim 21,
In order to separate the breathing region of the difference image by applying the difference image arrangement and the preprocessed continuous depth image to the pre-learned ResU-net (Residual U-net), the operation processing unit,
The preprocessed depth image received from the memory unit is input to the pre-learned first ResU-Net, and the segmentation map that is output from the first ResU-Net, in which only the torso image, which is the thorax and abdomen region, is separated, is received, and the second 1 Temporarily store the segmentation map in which only the body image received from ResU-Net is separated in the memory unit,
Multiplying the segmentation map in which only the body image is separated, the difference image, in units of pixels,
As an input of the second ResU-Net that has already been learned, the product of a segmentation map and a difference image that separates only the body image is input, and as an output of the second ResU-Net, a segmentation map that separates only the respiration-related region from the difference image, difference Respiratory measurement system using a depth camera, characterized in that receiving an image in which only a region related to respiration is separated from the image. The method of any one of claims 21 or 23, wherein the arithmetic processing unit
Respiratory volume measurement using a depth camera, characterized in that a peak and a valley are detected from the separated respiration waveform, and the respiration volume and respiration rate are calculated using the detected peak and valley system.

Images