KR102577759B1 - System for measuring biomedical signal and method for measuring biomedical signal therof - Google Patents

Abstract

The present disclosure relates to a biosignal measurement system and a method of measuring biosignals thereof. More specifically, the biosignal measurement system according to an embodiment of the present disclosure includes an image acquisition module for acquiring an image of a user, from the image acquisition module. An adaptive block module that combines bio-signals extracted from the face area based on the provided image and estimates the user's bio-signals, and a bio-signal output module that outputs the bio-signals estimated from the adaptive block module to the outside. Includes.

Description

Biosignal measurement system and its biosignal measurement method {SYSTEM FOR MEASURING BIOMEDICAL SIGNAL AND METHOD FOR MEASURING BIOMEDICAL SIGNAL THEROF}

This disclosure relates to healthcare technology, and more specifically, to an image-based bio-signal measurement system using an adaptive block module and a bio-signal measurement method thereof.

In accordance with the global aging trend, the importance of healthcare technology used in medical fields such as various rehabilitation and health examination fields is increasing. In particular, as the need for in vitro diagnostic services and home health care services has increased in the aftermath of COVID-19, interest in monitoring technology using biological signals is increasing. Human biosignal data can be used in various fields, and in particular, among biosignals, heart rate data can be used to analyze the user's momentum, emotion analysis, etc.

Existing heart rate measurement methods project light of a specific wavelength onto the user's fingers or earlobes and detect changes in blood volume based on reflected or transmitted light intensity to derive heart rate. Since heart rate measuring devices based on this method must be implemented as contact devices that can be in close contact with the user's skin, they have been mainly applied to wearable devices such as wristwatches and earphones. However, when measuring the user's heart rate through a wearable device, various noises may be generated due to the user's movement, tension, etc., making it difficult to measure the heart rate more accurately.

In order to overcome these shortcomings, a non-contact heart rate measurement method using camera images was proposed, as in Patent Publication No. 10-2015-0093036. The non-contact heart rate measurement method recognizes the user's face through camera images, sets an area of interest to analyze biosignals, obtains color change data based on heart rate in the area, and processes the data to derive the user's heart rate. did. When measuring a user's heart rate using a non-contact heart rate measurement method, the user's face is required to be fixed during the time of measuring the heart rate in order to accurately measure the heart rate. However, since noise may be generated due to the user's subtle movements and facial expression changes that may occur in daily life environments, there are limitations in measuring heart rate more accurately.

The purpose of the present disclosure is to provide an image-based non-contact bio-signal measurement system using an adaptive block module and a bio-signal measurement method thereof.

A biometric signal measurement system according to an embodiment of the present disclosure combines an image acquisition module for acquiring an image of a user, biosignals extracted from the facial area based on the image provided from the image acquisition module, and biosignals of the user. It includes an adaptive block module that estimates and a biosignal output module that externally outputs the biosignal estimated from the adaptive block module.

As an example, the image acquisition module includes at least one photographing device, an image acquisition unit that acquires an image of the user, and an image storage unit that stores the acquired image of the user.

For example, when the image acquisition unit includes two or more photographing devices, the image acquisition unit performs a calibration operation to match the position, resolution, and angle of view of the photographing devices.

For example, the adaptive block module may include a face area detection unit that detects the face area using an image of the user, an area weight calculation unit that calculates a weight for each of the face areas detected from the face area detection unit, and the user. a background area detection unit that detects a background area excluding the face area from the image, an illumination change detection unit that detects an illumination change for each of the background areas, and the face area and the illumination change for each of the face areas based on the face area and the illumination change. It includes a biometric signal extraction unit that extracts a biosignal, and an ensemble correction unit that estimates the user's biosignal by combining the biosignal extracted from each of the face regions and the weight calculated for each of the face regions.

For example, the face area detector performs a face detection operation and a facial landmark extraction operation, and the face detection operation is performed to return image coordinates where the user's face is located in the form of a bounding box, and the facial landmark extraction The operation is performed to estimate the user's face model based on a standard face model and to separate the face area based on the estimated face model.

For example, the face detection operation is performed by template matching or deep learning-based face search techniques.

For example, the biometric signal output module includes a biometric signal information display unit that displays information about the user's biosignal estimated from the adaptive block module.

For example, the biometric signal output module further includes a communication unit that transmits information about the user's biosignal to the outside.

In the biosignal measurement method of the non-contact biosignal measurement system for deriving the user's heart rate according to an embodiment of the present disclosure, the biosignal measurement method uses an image of the user acquired from an image acquisition module to measure the user's heart rate. Detecting a face area, detecting a background area using the image of the user, calculating a weight for each face area of the user, detecting an illumination change for each of the background areas, extracting a biosignal from each of the user's face regions based on a change in illumination, and combining the weight for each of the user's face regions and the biosignal extracted from each of the user's face regions to obtain the user's biosignal. It includes the step of estimating.

As an example, the method of measuring the biological signal further includes outputting information about the estimated biological signal through a display device.

As an example, the method of measuring biological signals further includes transmitting information about the estimated biological signals to the outside.

As an example, detecting the user's face area may include returning image coordinates where the user's face is located, estimating the user's face model based on a standard face model, and based on the user's face model. It includes the step of separating the face area.

According to the bio-signal measurement system and its bio-signal measurement method according to the present disclosure, various environmental noises can be removed through an adaptive block module, and thus the user's bio-signal data can be derived more accurately.

1 is a block diagram showing a biosignal measurement system according to an embodiment of the present disclosure.
Figure 2 is a diagram for explaining the operation of an image acquisition module according to an embodiment of the present disclosure.
Figure 3 is a block diagram for explaining the operation of an adaptive block module according to an embodiment of the present disclosure.
Figure 4 is a diagram for explaining the operation of a biological signal output module according to an embodiment of the present disclosure.
Figure 5 is a flowchart for explaining a method of measuring biological signals according to an embodiment of the present disclosure.
FIG. 6 is a flowchart illustrating a method of estimating a biological signal in the method of measuring a biological signal according to an embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described clearly and in detail so that a person skilled in the art can easily practice the present disclosure.

The terms used in this specification are for describing embodiments and are not intended to limit the present disclosure. As used herein, the singular forms also include the plural forms, unless specifically stated otherwise in the context. As used in the specification, “comprises and/or comprising” means the presence or addition of one or more other components, steps, operations and/or elements to the mentioned elements, steps, operations and/or elements. does not rule out

Terms such as “first and/or second” used in this specification may be used to describe various components, but are only used for the purpose of distinguishing one component from another component, and the terms It is not intended to limit the components referred to as . For example, without departing from the scope of the present disclosure, a first component may be referred to as a second component, and the second component may also be referred to as a first component.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings that can be commonly understood by those skilled in the art to which this disclosure pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined. In this specification, the same reference numerals may refer to the same elements throughout.

FIG. 1 is a block diagram showing a biological signal measurement system 10 according to an embodiment of the present disclosure. Referring to FIG. 1, the biosignal measurement system 10 according to an embodiment of the present disclosure may include an image acquisition module 100, an adaptive block module 200, and a biosignal output module 300.

The image acquisition module 100 can acquire a user's face image. The image acquisition module 100 may store the acquired face image of the user. The image acquisition module 100 may transmit stored facial image data (DATA_IMG) to the adaptive block module 200. The detailed configuration of the image acquisition module 100 will be described in FIG. 2 to be described later.

The adaptive block module 200 may estimate the user's biosignal (SIG_B) based on the facial image data (DATA_IMG) received from the image acquisition module 100. As an example, the user's biosignal (SIG_B) may be heart rate. The adaptive block module 200 can remove environmental noise that may occur in daily life from face image data (DATA_IMG). As an example, environmental noise may be noise resulting from the user's movement, facial expression changes, self-occlusion, changes in surrounding lighting, etc. After removing environmental noise, the adaptive block module 200 can estimate the user's biological signal (SIG_B) and transmit the biological signal (SIG_B) to the biological signal output module 300. The detailed configuration of the adaptive block module 200 will be described in FIG. 3 to be described later.

The biometric signal output module 300 may output information about the user's biosignal (SIG_B) received from the adaptive block module 200. The detailed configuration of the biosignal output module 300 will be described in FIG. 4 to be described later.

The biosignal measurement system 10 according to an embodiment of the present disclosure removes various environmental noises that may occur in daily life through the adaptive block module 200 and extracts the user's biosignal (SIG_B), thereby Biological signals (SIG_B) can be derived more accurately. The biosignal measurement system 10 according to an embodiment of the present disclosure can be used to independently monitor the user's physical condition. Additionally, the biosignal measurement system 10 according to an embodiment of the present disclosure can be used in a remote medical system by transmitting the measured biosignal (SIG_B) information to the outside. In addition, the biometric signal measurement system 10 according to an embodiment of the present disclosure can be used as a system to analyze the user's emotional state based on the measured biosignal (SIG_B) and provide various contents based on the analysis results. there is.

FIG. 2 is a diagram for explaining the operation of the image acquisition module 100 according to an embodiment of the present disclosure. The image acquisition module 100 may generate facial image data (DATA_IMG) for measuring the biosignals of the user (USER). Referring to FIG. 2 , the image acquisition module 100 may include an image acquisition unit 110 and an image storage unit 120.

The image acquisition unit 110 may acquire a face image (IMG) of the user (USER). For example, the face image (IMG) of the user (USER) may be a two-dimensional color image of the user (USER) or a three-dimensional image of the user (USER).

The image acquisition unit 110 may include at least one photographing device. For example, at least one imaging device may be a device for acquiring two-dimensional color images, such as a camera module such as a web cam or a mobile device, or a depth camera for acquiring three-dimensional images, such as a stereo camera or Kinect. . When the image acquisition unit 110 includes a plurality of photographing devices, a calibration operation may be performed to match the position, resolution, and angle of view of the photographing devices. The calibration operation can be performed by affine transforming the coordinates of other photographing devices based on one photographing device. Meanwhile, if the image acquisition unit 110 includes a single imaging device, the calibration operation may be omitted. The image acquisition unit 110 may transmit the acquired facial image (IMG) of the user (USER) to the image storage unit 120.

The image storage unit 120 may store the face image (IMG) of the user (USER) received from the image acquisition unit 110. The image storage unit 120 may transmit facial image data (DATA_IMG) based on the stored facial image (IMG) of the user (USER) to the adaptive block module 200 (see FIG. 1).

Figure 3 is a block diagram for explaining the operation of the adaptive block module 200 according to an embodiment of the present disclosure. The adaptive block module 200 may remove environmental noise from the facial image data (DATA_IMG) received from the image acquisition module 100 and then derive the user's biometric signal (SIG_B). Referring to FIG. 3, the adaptive block module 200 includes a face area detection unit 210, an area weight calculation unit 220, a background area detection unit 230, an illumination change detection unit 240, a biometric signal extraction unit 250, and It may include an ensemble correction unit 260.

The face area detection unit 210 may receive face image data (DATA_IMG) and detect the face area of the user (USER, see FIG. 2) to measure the biosignal (SIG_B) from the face image data (DATA_IMG). Face area detection can be performed through a face detection operation and a facial landmark extraction operation. The face detection operation is a process of setting a region of interest for facial landmark extraction, and can be performed by returning image coordinates where the face is located in the form of a bounding box. For example, face detection operations can be performed by various methods, such as template matching and deep learning-based face search techniques.

The facial landmark extraction operation can detect changes in the user's movement or expression and set the calculation area for configuring ensemble data. The facial landmark extraction operation may proceed as a process of estimating the user's face model based on a standard face model and then separating the face area. Facial landmarks can be output as coordinate values based on a standard face model. For example, when using a two-dimensional standard face model, two-dimensional landmark coordinate values may be output, and when using a three-dimensional standard face model, three-dimensional landmark coordinate values may be output. In an embodiment according to the present disclosure, the facial landmark extraction operation may extract each facial region after estimating the user's facial model, rather than recognizing facial landmarks for each frame.

Therefore, it is possible to minimize the generation of noise due to facial scale changes due to forward and backward movement of the imaging device and facial movements due to face rotation up and down and left and right. The face area detection unit 210 may generate first data D1 representing the user's face area obtained after performing the face detection operation and the facial landmark extraction operation, and transmit it to the area weight calculation unit 220.

The area weight calculation unit 220 may receive the first data D1 from the face area detection unit 210 and use it to calculate weights for each time. More specifically, the area weight calculation unit 220 may calculate a time-specific weight for the first data D1 for changes in the facial area that change according to movements that may occur in daily life. Time-specific weight refers to the reliability of each area at the current frame stage. For example, movements that may occur in daily life include the most visible facial area at the current shooting point, momentary occlusion situations such as hair, facial movements, and hand movements, or changes in facial expressions due to emotions, and changes in the area around the mouth during conversation. It may include changes in facial expressions, etc., and when movements that may occur in daily life occur, the weight may be expressed low. The second data D2 on the time-dependent weight for changes in the face region generated by the region weight calculation unit 220 may be provided to the biosignal extraction unit 250 and the ensemble correction unit 260.

When measuring the biological signal (SIG_B), the background area detector 230 may extract areas other than the face image to remove noise caused by changes in surrounding light intensity. For example, when using a two-dimensional image obtained from a color camera, the user's face and the background area can be segmented based on machine learning techniques. Alternatively, when using a 3D image obtained from a depth camera, the background area can be extracted using the depth value. The background area detection unit 230 does not extract all areas other than the face area, but can set highly reflective objects that well reflect the surrounding illumination environment and areas of the objects according to color or reflectivity. The third data D3 representing the area of the object generated by the background area detection unit 230 may be transmitted to the lighting change detection unit 240.

The lighting change detection unit 240 may obtain the lighting intensity for each area of the object based on the third data D3 received from the background area detection unit 230. In this case, because the area of the object is set, changes in the intensity of surrounding lighting can be detected even when movement of the imaging device occurs. The illumination change detection unit 240 may extract a brightness change trend of the entire image based on the acquired illumination intensity and remove noise due to the illumination change based on this. The illumination change detection unit 240 may generate fourth data D4 from which noise caused by the illumination change is removed and provide the fourth data D4 to the biological signal extraction unit 250.

The biosignal extraction unit 250 receives the second data D2 and the fourth data D4, and extracts the biosignal by filtering signals related to the biosignal from the brightness change signal over time for each face region. You can. As an example, the biosignal extractor 250 converts the brightness change signal in each face area into the frequency domain through Fourier transform, takes the peak value in the frequency range of the heart rate, and converts the peak value into the time domain to provide the user ( User)'s heart rate can be extracted. Alternatively, data filtered for the frequency domain of the heart rate can be converted to the time domain through inverse Fourier transform to extract the heart rate signal in the time domain, and the user's heart rate can be extracted by taking the peak value. The fifth data D5 representing the biological signal extracted from the biological signal extraction unit 250 may be provided to the ensemble correction unit 260.

The ensemble correction unit 260 may receive the second data D2 and the fifth data D5 and combine the biometric signals extracted from each facial region by reflecting the weight for each facial region. Since the biometric signals extracted from each face region have a unique signal pattern for each facial region, the ensemble correction unit 260 can estimate the biosignal (SIG_B) for the user (USER) by combining them. The estimated biological signal (SIG_B) may be provided to the biological signal output module 300.

FIG. 4 is a diagram for explaining the operation of the biosignal output module 300 according to an embodiment of the present disclosure. The biosignal output module 300 may externally output the biosignal (SIG_B) estimated from the adaptive block module 200 (see FIG. 1). Referring to FIG. 4 , the biometric signal output module 300 may include a biometric signal information display unit 310 and, in some cases, may further include a communication unit 320.

The biosignal information display unit 310 may include a device that outputs a biosignal (SIG_B) measured from a face image of a user (USER, see FIG. 2). For example, the biometric signal information display unit 310 may include a display device such as a projector or monitor. By displaying bio-signal information based on the bio-signal (SIG_B) measured through the display device, the user (USER) can perform self-monitoring of his/her physical condition.

The communication unit 320 may transmit to the outside the biometric signal (SIG_B) measured from the face image of the user (USER, see FIG. 2). For example, an external receiver (RCV) can receive the biosignal (SIG_B) transmitted from the communication unit 320 and use it in a remote medical system.

Figure 5 is a flowchart for explaining a method of measuring biological signals according to an embodiment of the present disclosure.

In step S110, the biosignal measurement system 10 (see FIG. 1) according to an embodiment of the present disclosure may acquire a face image of the user (USER, see FIG. 2). The biosignal measurement system 10 according to an embodiment of the present disclosure can store the acquired face image of the user.

In step S120, the biosignal measurement system 10 according to an embodiment of the present disclosure may estimate the biosignal of the user (USER) from the face image of the user (USER). The biosignal measurement system 10 according to an embodiment of the present disclosure can estimate biosignals by removing noise caused by the user's movement, facial expression change, or change in surrounding illumination, and the biosignal estimation process will be described later. This will be explained in detail in Figure 6.

In step S130, the bio-signal measurement system 10 according to an embodiment of the present disclosure may output bio-signal information to the outside based on the estimated user's bio-signal. For example, biometric signal information may be displayed to the user by a display device or transmitted to an external server.

FIG. 6 is a flowchart illustrating a method of estimating a biological signal in the method of measuring a biological signal according to an embodiment of the present disclosure.

In step S121, the adaptive block module 200 (see FIG. 1) constituting the biosignal measurement system 10 according to an embodiment of the present disclosure acquires the user (USER) obtained from the image acquisition module 100 (see FIG. 1). The video (see FIG. 2) can be received.

In step S122, the adaptive block module 200 may detect the face area and background area of the user (USER) based on the acquired image of the user (USER). Detection of the user's face area can be performed through a face detection operation and a facial landmark extraction operation. Background area detection can be performed through a highly reflective object that well reflects the surrounding illumination environment and an object area extraction operation based on color or reflectivity.

In step S123, the adaptive block module 200 may calculate a weight for the face region using the face region detected in step S122. More specifically, the adaptive block module 200 can calculate time-specific weights for changes in facial areas that change according to movements that may occur in daily life.

In step S124, the adaptive block module 200 may detect a change in lighting in the background area using the background area detected in step S122. The adaptive block module 200 can extract a brightness change trend of the entire image based on the acquired lighting intensity and remove noise due to lighting changes based on this.

In step S125, the adaptive block module 200 may extract a biological signal by filtering a signal related to the biological signal from a brightness change signal over time for each face region.

In step S126, the adaptive block module 200 may estimate the biometric signal of the user (USER) by combining the time-specific weight calculated in step S123 and the biometric signals extracted from each facial region extracted in step S125.

In some embodiments, the method of measuring biological signals according to an embodiment of the present disclosure described based on FIGS. 5 and 6 described above is implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. It can be. Computer-readable media may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on a computer-readable medium may be specially designed and configured for the present disclosure, or may be known and usable by those skilled in the art of computer software.

For example, computer-readable media includes magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -May include hardware devices configured to store and execute program instructions, such as magneto-optical media and ROM, RAM, flash memory, etc. As an example, program instructions may include machine language code such as that generated by a compiler as well as high-level language code that can be executed by a computer using an interpreter, etc., and the hardware device may include one or more devices for performing the operations of the present disclosure. It can be configured to operate by a software module.

The above-described contents are specific embodiments for carrying out the present disclosure. The present disclosure will include not only the above-described embodiments, but also embodiments that are simply designed or can be easily changed. In addition, the present disclosure will also include techniques that can be easily modified and implemented using the embodiments. Therefore, the scope of the present disclosure should not be limited to the above-described embodiments, but should be determined by the claims and equivalents of the present disclosure as well as the claims described later.

10: Biosignal measurement system
100: Image acquisition module
200: Adaptive block module
300: Biosignal output module

Claims (12)

An image acquisition module that acquires an image of a user;
an adaptive block module that combines biometric signals extracted from the face area based on the image provided from the image acquisition module and estimates the user's biosignal; and
A bio-signal output module that outputs the bio-signal estimated from the adaptive block module to the outside,
The adaptive block module:
a face area detection unit that detects the face area using the image of the user;
an area weight calculation unit that calculates a weight for each of the face areas detected by the face area detection unit;
a background area detection unit that detects a background area excluding the face area from the image of the user;
an illumination change detection unit that detects a change in illumination for each of the background areas;
a biometric signal extraction unit that extracts the biosignal for each of the face areas based on the face area and the illumination change; and
An ensemble correction unit that estimates the user's biosignal by combining the biosignal extracted from each of the face regions and the weight calculated for each of the face regions,
The detection of the background area is performed through a highly reflective object that well reflects the surrounding illumination environment and an object area extraction operation according to color or reflectivity,
The calculation of the weight includes calculating a time-dependent weight for a change in the face area that changes according to movement. According to claim 1,
The image acquisition module:
an image acquisition unit including at least one photographing device and acquiring an image of the user; and
A biometric signal measurement system including an image storage unit that stores the acquired image of the user. According to claim 2,
When the image acquisition unit includes two or more photographing devices, the image acquisition unit performs a calibration operation to match the position, resolution, and angle of view of the photographing devices. delete According to claim 1,
The face area detection unit performs a face detection operation and a facial landmark extraction operation,
The face detection operation is performed to return image coordinates where the user's face is located in the form of a bounding box,
The facial landmark extraction operation is performed to estimate the user's face model based on a standard face model and to separate the face area based on the estimated face model. According to claim 5,
A biometric signal measurement system in which the face detection operation is performed by template matching or deep learning-based face search techniques. According to claim 1,
The biological signal output module includes a biological signal information display unit that displays information about the user's biological signal estimated from the adaptive block module. According to claim 7,
The biological signal output module further includes a communication unit that transmits information about the user's biological signals to the outside. In the biosignal measurement method of the non-contact biosignal measurement system for deriving the user's heart rate:
Detecting the user's face area using the image of the user acquired from an image acquisition module;
detecting a background area using the image of the user;
calculating a weight for each face area of the user;
detecting an illumination change for each of the background areas;
extracting biosignals from each facial area of the user based on the lighting change; and
Comprising the step of estimating the user's biosignal by combining the weight for each face area of the user and the biosignal extracted from each of the user's face areas,
The step of detecting the background area includes extracting an object area based on a highly reflective object and color or reflectivity that well reflects the surrounding illumination environment,
The step of calculating the weight includes calculating a time-dependent weight for a change in the face area that changes according to movement. According to clause 9,
A method of measuring biological signals further comprising outputting information about the estimated biological signals through a display device. According to clause 9,
A method of measuring biological signals further comprising transmitting information about the estimated biological signals to the outside. According to clause 9,
The steps for detecting the user's face area are:
Returning image coordinates where the user's face is located;
estimating a face model of the user based on a standard face model; and
A bio-signal measurement method comprising separating the face area based on the user's face model.