KR20220072714A - System for measuring biomedical signal and method for measuring biomedical signal therof - Google Patents

Abstract

The present disclosure relates to a biosignal measuring system and a biosignal measuring method thereof, and more particularly, the biosignal measuring system according to an embodiment of the present disclosure includes an image acquisition module for acquiring an image of a user, and an image acquisition module An adaptive block module for combining biosignals extracted from a face region based on the provided image and estimating the user's biosignals, and a biosignal output module for outputting the biosignals estimated from the adaptive block module to the outside; include

Description

Biosignal measuring system and biosignal measuring method thereof

The present disclosure relates to health care technology, and more particularly, to an image-based biosignal measuring system using an adaptive block module and a biosignal measuring method thereof.

With the global aging trend, the importance of health care technology used in medical fields such as various rehabilitation fields and health examination fields is increasing. In particular, as the need for in vitro diagnostic services and home health care services increases in the aftermath of COVID-19, interest in monitoring technology using biosignals is increasing. Human biosignal data may be utilized in various fields, and in particular, heart rate data among biosignals may be utilized for exercise amount analysis and emotion analysis of a user.

The existing heart rate measurement method derives the heart rate by projecting light of a specific wavelength to the user's finger or earlobe, and detecting a change in blood volume based on the reflected or transmitted light intensity. Since the heart rate measuring device based on this method should be implemented as a contact-type device that can be in close contact with the user's skin, it has been mainly applied to wearable devices such as wrist watches and earphones. However, when measuring the heart rate of the user through the wearable device, various noises may be generated due to the user's movement, tension, etc., and thus, it is difficult to measure the heart rate more accurately.

In order to overcome these shortcomings, a non-contact heart rate measurement method using a camera image has been proposed. The non-contact heart rate measurement method recognizes a user's face through a camera image, sets an area of interest to analyze biosignals, obtains color change data based on heart rate in the area, and processes the data to derive the user's heart rate did. When measuring a user's heart rate by a non-contact heart rate measuring method, it is required that the user's face is fixed for a time for measuring the heart rate in order to accurately measure the heart rate. However, since noise may be generated due to a user's minute movements and facial expression changes that may occur in a daily living environment, there is a limit to more accurate heart rate measurement.

An object of the present disclosure is to provide an image-based non-contact biosignal measuring system using an adaptive block module and a biosignal measuring method thereof.

A biosignal measuring system according to an embodiment of the present disclosure combines an image acquisition module for acquiring an image of a user, a biosignal extracted from a face region based on the image provided from the image acquisition module, and the biosignal of the user an adaptive block module for estimating , and a biosignal output module for outputting the biosignal estimated from the adaptive block module

For example, the image acquisition module includes at least one photographing device, and includes an image acquisition unit configured to acquire an image of the user, and an image storage unit configured to store the acquired image of the user.

For example, when the image acquisition unit includes two or more photographing devices, the image acquisition unit performs a calibration operation to match the positions, resolutions, and angles of view of the photographing devices.

For example, the adaptive block module may include a face region detector configured to detect the face region using the image of the user, a region weight calculator configured to calculate a weight for each of the face regions detected by the face region detector, and the user A background area detection unit for detecting a background area excluding the face area from the image of and a biosignal extractor for extracting a biosignal, and an ensemble corrector for estimating the biosignal of the user by combining the biosignal extracted from each of the face regions and the weight calculated for each of the face regions.

For example, the face region detector performs a face detection operation and a face landmark extraction operation, and the face detection operation is performed to return image coordinates where the user's face is located in the form of a bounding box, and extract the facial landmark The operation is performed to estimate the face model of the user based on a standard face model, and to separate the face region based on the estimated face model.

For example, the face detection operation is performed by template matching or a deep learning-based face search technique.

For example, the biosignal output module includes a biosignal information display unit that displays information about the user's biosignal estimated from the adaptive block module.

For example, the biosignal output module further includes a communication unit that transmits information on the user's biosignal to the outside.

In the biosignal measuring method of a non-contact biosignal measuring system for deriving a user's heart rate according to an embodiment of the present disclosure, the biosignal measuring method includes using an image of the user obtained from an image acquisition module to determine the user's detecting a face region, detecting a background region using the image of the user, calculating a weight for each of the user's face regions, detecting a change in lighting for each of the background regions, the extracting a biosignal from each of the user's face regions based on a change in lighting, and combining the weight for each of the user's face regions and the biosignals extracted from each of the user's face regions to obtain the user's biosignal estimating.

For example, the biosignal measuring method may further include outputting information on the estimated biosignal through a display device.

For example, the biosignal measuring method further includes transmitting information on the estimated biosignal to the outside.

For example, the detecting of the user's face region may include returning image coordinates where the user's face is located, estimating the user's face model based on a standard face model, and based on the user's face model and separating the face region.

According to the biosignal measuring system and its biosignal measuring method according to the present disclosure, various environmental noises can be removed through an adaptive block module, so that biosignal data of a user can be more accurately derived.

1 is a block diagram illustrating a biosignal measuring system according to an embodiment of the present disclosure.
2 is a diagram for explaining an operation of an image acquisition module according to an embodiment of the present disclosure.
3 is a block diagram illustrating an operation of an adaptive block module according to an embodiment of the present disclosure.
4 is a diagram for explaining an operation of a biosignal output module according to an embodiment of the present disclosure.
5 is a flowchart illustrating a method for measuring a biosignal according to an embodiment of the present disclosure.
6 is a flowchart illustrating a method of estimating a biosignal in the biosignal measuring method according to an embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described clearly and in detail to the extent that those of ordinary skill in the art can easily practice the present disclosure.

The terminology used herein is for the purpose of describing the embodiments, and is not intended to limit the present disclosure. As used herein, the singular also includes the plural unless the phrase specifically dictates otherwise. As used herein, “comprises and/or comprises” refers to the presence or addition of one or more other components, steps, acts and/or elements to a stated element, step, operation and/or element. do not exclude

As used herein, terms such as “first and/or second” may be used to describe various components, but these terms are only used for the purpose of distinguishing one component from other components, and the term It is not intended to limit the components referred to as . For example, without departing from the scope of the present disclosure, a first component may be referred to as a second component, and the second component may also be referred to as a first component.

Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used in the meaning commonly understood by those of ordinary skill in the art to which this disclosure belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly. In this specification, the same reference numerals may refer to the same elements throughout.

1 is a block diagram illustrating a biosignal measuring system 10 according to an embodiment of the present disclosure. Referring to FIG. 1 , a biosignal measurement system 10 according to an embodiment of the present disclosure may include an image acquisition module 100 , an adaptive block module 200 , and a biosignal output module 300 .

The image acquisition module 100 may acquire a face image of the user. The image acquisition module 100 may store the acquired user's face image. The image acquisition module 100 may transmit the stored face image data DATA_IMG to the adaptive block module 200 . A detailed configuration of the image acquisition module 100 will be described with reference to FIG. 2 to be described later.

The adaptive block module 200 may estimate the user's biosignal SIG_B based on the face image data DATA_IMG received from the image acquisition module 100 . For example, the user's biosignal SIG_B may be a heart rate. The adaptive block module 200 may remove environmental noise that may occur in daily life from the face image data DATA_IMG. For example, the environmental noise may be noise caused by a user's movement, a change in facial expression, self-occlusion, a change in ambient lighting, and the like. After removing environmental noise, the adaptive block module 200 may estimate the user's bio-signal SIG_B and transmit the bio-signal SIG_B to the bio-signal output module 300 . A detailed configuration of the adaptive block module 200 will be described with reference to FIG. 3 to be described later.

The biosignal output module 300 may output information about the user's biosignal SIG_B received from the adaptive block module 200 . A detailed configuration of the biosignal output module 300 will be described with reference to FIG. 4 to be described later.

The biosignal measurement system 10 according to an embodiment of the present disclosure removes various environmental noises that may occur in daily life through the adaptive block module 200 and then extracts the user's biosignal SIG_B. The biosignal SIG_B may be derived more accurately. The biosignal measuring system 10 according to an embodiment of the present disclosure may be used to self-monitor a user's physical condition. In addition, the biosignal measuring system 10 according to an embodiment of the present disclosure may transmit the measured biosignal SIG_B information to the outside to be used in a remote medical system. In addition, the biosignal measuring system 10 according to an embodiment of the present disclosure may be used in a system for analyzing a user's emotional state based on the measured biosignal SIG_B and providing various contents based on the analysis result. have.

2 is a diagram for explaining an operation of the image acquisition module 100 according to an embodiment of the present disclosure. The image acquisition module 100 may generate face image data DATA_IMG for measuring a biosignal of the user USER. Referring to FIG. 2 , the image acquisition module 100 may include an image acquisition unit 110 and an image storage unit 120 .

The image acquisition unit 110 may acquire a face image IMG of the user USER. For example, the face image IMG of the user USER may be a 2D color image of the user USER or a 3D image of the user USER.

The image acquisition unit 110 may include at least one photographing device. For example, the at least one imaging device may be a device for acquiring a two-dimensional color image, such as a camera module such as a web cam or a mobile device, or a depth camera for acquiring a three-dimensional image, such as a stereo camera or Kinect. . When the image acquisition unit 110 includes a plurality of photographing apparatuses, a calibration operation may be performed to match the position, resolution, and angle of view of the photographing apparatus. The calibration operation may be performed by affine transforming the coordinates of other imaging devices with respect to one imaging device. Meanwhile, when the image acquisition unit 110 includes a single photographing device, the calibration operation may be omitted. The image acquisition unit 110 may transmit the acquired face image IMG of the user USER to the image storage unit 120 .

The image storage unit 120 may store the face image IMG of the user USER received from the image acquisition unit 110 . The image storage unit 120 may transmit the stored face image data DATA_IMG based on the stored user's face image IMG to the adaptive block module 200 (refer to FIG. 1 ).

3 is a block diagram illustrating an operation of the adaptive block module 200 according to an embodiment of the present disclosure. The adaptive block module 200 may derive the user's biosignal SIG_B after removing environmental noise from the face image data DATA_IMG received from the image acquisition module 100 . Referring to FIG. 3 , the adaptive block module 200 includes a face region detector 210 , a region weight calculator 220 , a background region detector 230 , a lighting change detector 240 , a biosignal extractor 250 and The ensemble correction unit 260 may be included.

The face region detector 210 may receive the face image data DATA_IMG and detect the face region of the user (USER (refer to FIG. 2 ) for measuring the biosignal SIG_B from the face image data DATA_IMG). The face region detection may be performed through a face detection operation and a face landmark extraction operation. The face detection operation is a process of setting a region of interest for extracting facial landmarks, and may be performed by returning image coordinates where the face is located in the form of a bounding box. For example, the face detection operation may be performed by various methods, such as template matching, a deep learning-based face search technique, and the like.

The facial landmark extraction operation may detect a change in movement or expression of a user (USER) and set an operation area for ensemble data configuration. The facial landmark extraction operation may be performed by estimating the user's (USER) face model based on the standard face model and then separating the face regions. The face landmark may be output as a coordinate value based on a standard face model. For example, when a two-dimensional standard face model is used, a two-dimensional landmark coordinate value may be output, and when a three-dimensional standard face model is used, a three-dimensional landmark coordinate value may be output. In an embodiment according to the present disclosure, the facial landmark extraction operation is not a method of recognizing a facial landmark for every frame, but may extract each facial region after estimating the user's face model.

Accordingly, it is possible to minimize noise generation due to a face scale change according to a front-back movement of the photographing apparatus and a face movement due to vertical/left/right face rotation. The face region detector 210 may generate the first data D1 representing the user's face region obtained after performing the face detection operation and the facial landmark extraction operation, and transmit it to the region weight calculator 220 .

The region weight calculator 220 may receive the first data D1 from the face region detector 210 and calculate a weight for each time using the first data D1 . In more detail, the region weight calculator 220 may calculate a weight for each change in the face region that changes according to movements that may occur in daily life with respect to the first data D1 . The weight by time means the reliability of each region in the current frame stage. For example, movements that can occur in daily life include the facial region that is most visible at the time of shooting, hair, facial movements, hand movements, etc. that occur instantaneously, changing facial expressions according to situations or emotions, and changes in the area around the mouth during conversation. It may include a change in facial expression including the back, etc., and when a movement that may occur in daily life occurs, the weight may be expressed low. The second data D2 for the weight for each time of the face region change generated by the region weight calculator 220 may be provided to the biosignal extractor 250 and the ensemble corrector 260 .

When measuring the biosignal SIG_B, the background region detector 230 may extract a region other than the face image in order to remove noise caused by a change in ambient luminance. For example, when using a two-dimensional image obtained from a color camera, a user's face and a background region may be segmented based on a machine learning technique. Alternatively, when using a 3D image obtained from a depth camera, the background area may be extracted using the depth value. The background area detection unit 230 may not extract all areas other than the face area, but may set an object having high reflectivity that reflects the surrounding illuminance environment well, and an object area according to color or reflectivity. The third data D3 representing the area of the object generated by the background area detection unit 230 may be transmitted to the illumination change detection unit 240 .

The illumination change detection unit 240 may obtain the illumination intensity for each area of the object based on the third data D3 received from the background area detection unit 230 . In this case, since the area of the object is set, a change in ambient light intensity can be detected even when a motion of the photographing device occurs. The illumination change detection unit 240 may extract a brightness change trend of the entire image based on the obtained illumination intensity, and may remove noise according to the illumination change based thereon. The illumination change detection unit 240 may generate fourth data D4 from which noise according to the illumination change is removed, and may provide it to the biosignal extractor 250 .

The biosignal extractor 250 receives the second data D2 and the fourth data D4, and filters the biosignal-related signal from the time-dependent brightness change signal for each face region to extract the biosignal. can For example, the biosignal extractor 250 converts a brightness change signal in each face region into a frequency domain through Fourier transform, takes a peak value in a frequency range of heart rate, and converts the peak value into a time domain to convert the USER)'s heart rate can be extracted. Alternatively, the filtered data in the frequency domain of the heart rate may be transformed into the time domain through inverse Fourier transform, the heart rate signal may be extracted from the time domain, and the heart rate of the user may be extracted by taking the peak value. The fifth data D5 representing the biosignal extracted from the biosignal extractor 250 may be provided to the ensemble corrector 260 .

The ensemble corrector 260 may receive the second data D2 and the fifth data D5 , reflect the weights for each face region, and combine the biosignals extracted from each face region. Since the biosignal extracted from each face region has a unique signal pattern for each face region, the ensemble corrector 260 may combine them to estimate the biosignal SIG_B for the user USER. The estimated biosignal SIG_B may be provided to the biosignal output module 300 .

4 is a diagram for explaining an operation of the biosignal output module 300 according to an embodiment of the present disclosure. The biosignal output module 300 may output the biosignal SIG_B estimated from the adaptive block module 200 (refer to FIG. 1 ) to the outside. Referring to FIG. 4 , the biosignal output module 300 may include a biosignal information display unit 310 , and may further include a communication unit 320 in some cases.

The biosignal information display unit 310 may include a device that outputs the biosignal SIG_B measured from the face image of the user (see FIG. 2 ). For example, the biosignal information display unit 310 may include a display device such as a projector or a monitor. By displaying biosignal information based on the biosignal SIG_B measured through the display device, the user USER may perform self-monitoring of the body state.

The communication unit 320 may transmit the biosignal SIG_B measured from the face image of the user (see FIG. 2 ) to the outside. For example, the external receiver RCV may receive the biosignal SIG_B transmitted from the communication unit 320 and use it in the remote medical system.

5 is a flowchart illustrating a method for measuring a biosignal according to an embodiment of the present disclosure.

In step S110 , the biosignal measuring system 10 (see FIG. 1 ) according to an embodiment of the present disclosure may acquire a face image of a user ( USER, see FIG. 2 ). The biosignal measuring system 10 according to an embodiment of the present disclosure may store the acquired user's face image.

In operation S120 , the biosignal measuring system 10 according to an embodiment of the present disclosure may estimate the biosignal of the user USER from the face image of the user USER. The biosignal measuring system 10 according to an embodiment of the present disclosure may estimate a biosignal from which noise generated according to a user's movement, facial expression change, or ambient illumination change is removed, and the biosignal estimation process will be described later. 6 will be described in detail.

In step S130 , the biosignal measuring system 10 according to an embodiment of the present disclosure may output biosignal information to the outside based on the estimated user's biosignal. For example, the biosignal information may be displayed to the user by the display device or transmitted to an external server.

6 is a flowchart illustrating a method of estimating a biosignal in the biosignal measuring method according to an embodiment of the present disclosure.

In step S121, the adaptive block module 200 (refer to FIG. 1) constituting the biosignal measurement system 10 according to an embodiment of the present disclosure provides a user (USER, 2) may be received.

In step S122 , the adaptive block module 200 may detect a face region and a background region of the user USER based on the obtained user USER image. The detection of the user's face region may be performed through a face detection operation and a facial landmark extraction operation. The background area detection may be performed through an object area extraction operation according to a color or reflectivity of an object with high reflectivity that reflects the surrounding illuminance environment well.

In step S123 , the adaptive block module 200 may calculate a weight for the face region using the face region detected in step S122 . In more detail, the adaptive block module 200 may calculate a weight for each change in the face region that changes according to movement that may occur in daily life.

In step S124 , the adaptive block module 200 may detect a change in illumination of the background area using the background area detected in step S122 . The adaptive block module 200 may extract a brightness change trend of the entire image based on the obtained light intensity, and based on this, may remove noise according to the lighting change.

In step S125 , the adaptive block module 200 may extract a biosignal by filtering a signal related to the biosignal from the time-dependent brightness change signal for each face region.

In step S126 , the adaptive block module 200 may estimate the biosignal of the user USER by combining the weight for each time calculated in step S123 and the biosignals extracted from each face region extracted in step S125 .

In some embodiments, the biosignal measuring method according to an embodiment of the present disclosure described based on FIGS. 5 and 6 described above is implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. can be The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the computer readable medium may be specially designed and configured for the present disclosure, or may be known and available to those skilled in the art of computer software.

For example, the computer readable medium includes a hard disk, a floppy disk, a magnetic medium such as a magnetic tape, an optical media such as a CD-ROM, a DVD, and a magnetic medium such as a floppy disk. - May include magneto-optical media and hardware devices configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. For example, the program instructions may include not only machine code such as generated by a compiler but also high-level language code that can be executed by a computer using an interpreter or the like, and the hardware device includes one or more for performing the operations of the present disclosure. It may be configured to be operated by a software module.

The above are specific embodiments for carrying out the present disclosure. The present disclosure will include not only the above-described embodiments, but also simple design changes or easily changeable embodiments. In addition, the present disclosure will also include techniques that can be easily modified and implemented using the embodiments. Accordingly, the scope of the present disclosure should not be limited to the above-described embodiments, but should be defined by the claims and equivalents of the claims of the present disclosure as well as the claims to be described later.

10: biosignal measurement system
100: image acquisition module
200: Adaptive block module (Adaptive block module)
300: biosignal output module

Claims (12)

an image acquisition module for acquiring an image of the user;
an adaptive block module for combining bio-signals extracted from a face region based on the image provided from the image acquisition module and estimating the bio-signals of the user; and
and a biosignal output module for outputting the biosignal estimated from the adaptive block module to the outside. The method of claim 1,
The image acquisition module includes:
an image acquisition unit including at least one photographing device and acquiring an image of the user; and
and an image storage unit configured to store the acquired image of the user. 3. The method of claim 2,
When the image acquisition unit includes two or more photographing devices, the image acquisition unit performs a calibration operation to match the positions, resolutions, and angles of view of the photographing devices. The method of claim 1,
The adaptive block module includes:
a face region detector configured to detect the face region using the image of the user;
a region weight calculator configured to calculate a weight for each of the face regions detected by the face region detector;
a background area detection unit for detecting a background area excluding the face area from the image of the user;
an illumination change detector configured to detect a change in illumination for each of the background regions;
a biosignal extractor configured to extract the biosignal from each of the face regions based on the change in the face region and the illumination; and
and an ensemble corrector for estimating the biosignal of the user by combining the biosignal extracted from each of the face regions and the weight calculated for each of the face regions. 5. The method of claim 4,
The face region detection unit performs a face detection operation and a face landmark extraction operation,
The face detection operation is performed to return image coordinates where the user's face is located in the form of a bounding box,
The facial landmark extraction operation is performed to estimate the user's face model based on a standard face model, and to separate the face region based on the estimated face model. 6. The method of claim 5,
The face detection operation is a biosignal measurement system performed by template matching or a deep learning-based face search technique. The method of claim 1,
and the biosignal output module includes a biosignal information display unit configured to display information on the user's biosignal estimated from the adaptive block module. 8. The method of claim 7,
The biosignal output module further comprises a communication unit for transmitting information on the user's biosignal to the outside. A method for measuring a biosignal of a non-contact biosignal measuring system for deriving a user's heart rate, the method comprising:
detecting a face region of the user using an image of the user obtained from an image acquisition module;
detecting a background area using the image of the user;
calculating a weight for each of the user's face regions;
detecting an illumination change for each of the background regions;
extracting a biosignal from each of the face regions of the user based on the change in illumination; and
and estimating the user's bio-signal by combining the weight for each of the user's face regions and the bio-signals extracted from each of the user's face regions. 10. The method of claim 9,
and outputting the estimated biosignal information through a display device. 10. The method of claim 9,
The method of measuring a biosignal further comprising the step of transmitting information on the estimated biosignal to the outside. 10. The method of claim 9,
The detecting of the user's face region includes:
returning image coordinates where the user's face is located;
estimating the user's face model based on a standard face model; and
and separating the face region based on the face model of the user.

Images