KR20210060246A - The arraprus for obtaining biometiric data and method thereof - Google Patents

Abstract

An electronic device according to various embodiments may include a memory for storing instructions and a processor connected to the memory. The processor obtains image data captured by using a camera, analyzes the image data to calculate a heart rate value, determines whether the heart rate value is normally obtained, determines a correction value for a section where the heart rate value is not normally obtained when the heart rate value is not normally obtained, and stores biometric information data including the correction value in the memory. It is possible to monitor and manage the health status in daily life by taking into account individual differences.

Description

TECHNICAL FIELD The apparatus and method for acquiring biometric data TECHNICAL FIELD

Various embodiments relate to a method of measuring a biosignal and an apparatus for acquiring biometric data using the same.

In order to manage health and prevent diseases, there are technologies that collect biometric data by measuring an individual's vital signs through electronic devices. Vital signs refer to biological signals that show elements for maintaining life in the form of signals. Several international medical standards define heart rate (heart rate), respiratory rate, blood pressure, and body temperature as essential vital signs. Technologies that utilize medical big data including information related to the human body such as vital signs have been proposed.

Heart rate refers to the heart rate per unit time. For example, the heart rate value can be expressed as the number of pulses per minute. The heart rate changes according to the body's need for oxygen absorption or carbon dioxide excretion caused by physical activity. The heart rate may be measured at a position in the body where the pulse rate is sensed. That is, the heart rate may be measured at a location where an artery through which blood flows is located by the heart rate. For example, the heart rate may be measured in the neck where the carotid artery is located, the groin where the high artery is located, or the wrist. For precise heart rate measurement, an electrocardiogram (ECG) can be used. Continuous heart rate monitoring can be used to diagnose and treat disease.

Alternatively, photoplethysmograph (PPG) may be used to measure heart rate. Biometric information such as blood flow rate, blood oxygen saturation, or heart rate may be obtained by using the optical volume pulse wave signal. The photovoltaic pulse wave may be detected using a photoelectric conversion device that converts light into electric charge. An optical sensor including a photodiode may be used as an optical sensor for detecting a volumetric pulse wave. When light reaches the photoelectric conversion device through the lens, the light is converted into electric charges. Light in a designated band is irradiated to the body, and a light volume pulse wave can be obtained based on a signal that detects light reflected from the body or transmitted through the body.

Alternatively, there is a method and apparatus for measuring non-contact biometric information using a Doppler radar. In a method of measuring biometric information using a Doppler radar, a Doppler radar radiates a designated frequency signal and measures a reflected signal according to the movement of an organ. When the measured signal is converted into a high-frequency signal and transmitted to a digital signal processor, the digital signal processor can calculate the user's heart rate and respiration rate from the high-frequency signal using digital signal processing that performs frequency domain analysis.

There is a problem in which it is difficult to select meaningful data to be used in the treatment process among the vast and diverse medical big data. Due to the increasing complexity of medical data, it is difficult to utilize the data effectively by the existing approach. Even for patients with the same disease, symptoms are different depending on the individual's health status, lifestyle, and genomic information, so universal medical care may not be effective or may cause side effects. Therefore, there is a need for a health management monitoring and management system in daily life considering individual differences.

Cardiovascular diseases, such as tachycardia among arrhythmia, have a high frequency of occurrence of a specific heartbeat during exercise. In order to measure the heart rate of a user who is exercising, the user attaches contact equipment to the body during exercise load test and performs a test, or the user attaches the equipment to the body for a long period of time (e.g., 3 days) in daily life. You need to measure your heart rate. In this case, it causes inconvenience to the user or makes daily life uncomfortable.

As one of the methods for measuring heart rate in a non-contact manner, by recognizing and tracking the user's face from the image sequence collected through the camera, the electronic device can measure the user's heart rate from the change in pixel values for the same part of the face. have. As a feature point for recognizing a face, a method of identifying two eyes in an image may be used. The electronic device may recognize a region in which a face is photographed in an image based on the identified two eyes. In the case of recognizing the face and measuring the heart rate based on this, the electronic device can recognize the face and measure the heart rate normally only when the user is staring at the camera. If some of the facial feature points are lost, the electronic device may not be able to recognize the face because a correlation between the feature points cannot be obtained. In addition, when the user measures the heart rate while exercising, the user's head moves, so the facial recognition rate is lowered, and the normal heart rate is not measured, so that the normal heart rate is not provided to the user.

In addition, a method of measuring biometric information using a Doppler radar is not suitable as a method of measuring a heart rate for a moving body because it uses reflected waves according to movement of an organ.

Accordingly, there is a need for a technology to increase the facial recognition rate in order to normally provide the heart rate of a user who is exercising. In addition, there is a need for a technology for correcting and providing information on heart rate even when the face is not recognized.

In addition, there is a need for a method and apparatus for providing more suitable exercise-related information to a user using biometric information measured during exercise as well as biometric information measured in a stationary state.

The technical problems to be achieved in this document are not limited to the technical problems mentioned above, and other technical problems that are not mentioned can be clearly understood by those of ordinary skill in the technical field to which the present invention belongs from the following description. There will be.

An electronic device according to an embodiment may include a memory and a processor connected to the memory. According to an embodiment, the processor acquires image data captured using a camera, analyzes the image data to calculate a heart rate value, determines whether the heart rate value is normally acquired, and the heart rate value is not normally acquired. In this case, it may be configured to determine a correction value for a section in which the heart rate value is not normally obtained, and to store biometric information data including the correction value in a memory.

The method of operating the electronic device includes an operation of acquiring image data captured using a camera, an operation of calculating a heart rate value by analyzing the image data, an operation of determining whether the heart rate value is normally obtained, and an operation of obtaining a heart rate value normally. If not, an operation of determining a correction value for a section in which the heart rate value is not normally obtained and an operation of generating biometric information data including the correction value may be included.

A computer-readable recording medium recording a program according to an embodiment includes an operation of acquiring image data captured using a camera when executed, an operation of calculating a heart rate value by analyzing image data, and the heart rate value To perform an operation of determining whether or not the heart rate value is normally acquired, an operation of determining a correction value for a section in which the heart rate value is not normally obtained, and an operation of generating biometric information data including the correction value when the heart rate value is not normally obtained It may be a record of a program that does.

An electronic device according to an embodiment may include a memory for storing heart rate data of an arrhythmia patient and a processor connected to the memory. The processor acquires biometric information data including a heart rate value, and if the biometric information data does not contain a heart rate value or there is a section other than the normally calculated value, the value of the section is corrected to a correction value, and the corrected If the biometric information data and the heart rate data of the arrhythmia patient are compared, and the number of times the corrected biometric information data includes the abnormal pattern included in the heart rate data of the arrhythmia patient is greater than or equal to the threshold value, information on the abnormal heart rate pattern is It can be configured to output.

According to an embodiment disclosed in the present document, a method and apparatus capable of increasing a facial recognition rate for measuring a heart rate from an image may be provided.

According to an embodiment disclosed in this document, a method and apparatus capable of providing heart rate information to a user based on information acquired from an image may be provided even when a user's face is not normally recognized from an image. Accordingly, even when the user moves, a method and apparatus capable of providing normal heart rate information to the user without attaching or wearing the device to the body can be provided.

Depending on the exercise skill level, even if the user performs an exercise with the same exercise intensity, muscle fatigue and the like occur differently. According to an embodiment disclosed in this document, a method and an apparatus capable of inducing a user to easily achieve an exercise target value by presenting an appropriate exercise intensity based on a user's bio-signal may be provided.

The effects obtainable in the present disclosure are not limited to the effects mentioned above, and other effects not mentioned may be clearly understood by those of ordinary skill in the technical field to which the present disclosure belongs from the following description. will be.

1 is a block diagram of an electronic device in a network environment, according to various embodiments.
2 is a block diagram illustrating a program according to various embodiments.
3 is a diagram illustrating a configuration of an electronic device according to an exemplary embodiment.
4 is a flowchart illustrating a process of generating biometric information data by an electronic device according to an exemplary embodiment.
5 illustrates an example of an electronic device according to an embodiment.
6 illustrates an example of a system for acquiring biometric information data, according to an embodiment.
7 illustrates another example of an electronic device for obtaining biometric information data, according to an exemplary embodiment.
8 is a flowchart illustrating a process of calculating a heart rate value by an electronic device according to an exemplary embodiment.
9 is a diagram conceptually illustrating a method of recognizing a face area for calculating a heart rate value by an electronic device, according to an exemplary embodiment.
10 is a flowchart illustrating a process of determining whether a heart rate value is normally calculated by an electronic device, according to an exemplary embodiment.
11 is a flowchart illustrating a process of determining a correction value when a heart rate value is not normally calculated by an electronic device, according to an exemplary embodiment.
12 conceptually illustrates an operation of correcting a heart rate value by using candidate data by an electronic device according to an exemplary embodiment.
13 is a flowchart illustrating a process of additionally correcting the corrected biometric information data when a normal heart rate value is calculated by the electronic device, according to an exemplary embodiment.
14 conceptually illustrates an operation of recalibrating a corrected value by an electronic device according to an exemplary embodiment.
15 is a flowchart illustrating a process in which an electronic device generates candidate data and biometric information data based on a feature point, according to an exemplary embodiment.
16 illustrates an example of a screen displayed by an electronic device according to an embodiment.
17 is a flowchart illustrating a process for an electronic device to provide biometric information data together with guide image content, according to an exemplary embodiment.
18 is a flowchart illustrating a process of providing recommended exercise intensity information based on biometric information data by an electronic device according to an exemplary embodiment.
19 is a flowchart illustrating a process of providing recommended exercise information based on biometric information data by an electronic device according to an exemplary embodiment.
20 is a diagram illustrating an electronic device in a module unit according to an exemplary embodiment.

1 is a block diagram of an electronic device 101 in a network environment 100 according to various embodiments. Referring to FIG. 1, in a network environment 100, the electronic device 101 communicates with the electronic device 102 through a first network 198 (for example, a short-range wireless communication network), or a second network 199 It is possible to communicate with the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108. According to an embodiment, the electronic device 101 includes a processor 120, a memory 130, an input device 150, an audio output device 155, a display device 160, an audio module 170, and a sensor module ( 176, interface 177, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196, or antenna module 197 ) Can be included. In some embodiments, at least one of these components (eg, the display device 160 or the camera module 180) may be omitted or one or more other components may be added to the electronic device 101. In some embodiments, some of these components may be implemented as one integrated circuit. For example, the sensor module 176 (eg, a fingerprint sensor, an iris sensor, or an illuminance sensor) may be implemented while being embedded in the display device 160 (eg, a display).

The processor 120, for example, executes software (eg, a program 140) to implement at least one other component (eg, a hardware or software component) of the electronic device 101 connected to the processor 120. It can be controlled and can perform various data processing or operations. According to an embodiment, as at least a part of data processing or operation, the processor 120 may transfer commands or data received from other components (eg, the sensor module 176 or the communication module 190) to the volatile memory 132. It is loaded into, processes commands or data stored in the volatile memory 132, and the result data may be stored in the nonvolatile memory 134. According to an embodiment, the processor 120 includes a main processor 121 (eg, a central processing unit or an application processor), and a secondary processor 123 (eg, a graphic processing unit, an image signal processor) that can be operated independently or together. , A sensor hub processor, or a communication processor). Additionally or alternatively, the coprocessor 123 may be set to use lower power than the main processor 121 or to be specialized for a designated function. The secondary processor 123 may be implemented separately from the main processor 121 or as a part thereof.

The co-processor 123 is, for example, in place of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 is active (eg, executing an application). ) While in the state, together with the main processor 121, at least one of the components of the electronic device 101 (for example, the display device 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the functions or states associated with it. According to an embodiment, the coprocessor 123 (eg, an image signal processor or a communication processor) may be implemented as a part of other functionally related components (eg, the camera module 180 or the communication module 190). have.

The memory 130 may store various data used by at least one component of the electronic device 101 (eg, the processor 120 or the sensor module 176 ). The data may include, for example, software (eg, the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a nonvolatile memory 134.

The program 140 may be stored as software in the memory 130, and may include, for example, an operating system 142, middleware 144, or an application 146.

The input device 150 may receive a command or data to be used for a component of the electronic device 101 (eg, the processor 120) from outside (eg, a user) of the electronic device 101. The input device 150 may include, for example, a microphone, a mouse, a keyboard, or a digital pen (eg, a stylus pen).

The sound output device 155 may output an sound signal to the outside of the electronic device 101. The sound output device 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback, and the receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part of the speaker.

The display device 160 may visually provide information to the outside of the electronic device 101 (eg, a user). The display device 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to an embodiment, the display device 160 may include a touch circuitry set to sense a touch, or a sensor circuit (eg, a pressure sensor) set to measure the strength of a force generated by the touch. have.

The audio module 170 may convert sound into an electrical signal, or conversely, may convert an electrical signal into sound. According to an embodiment, the audio module 170 acquires sound through the input device 150, the sound output device 155, or an external electronic device (eg: Sound can be output through the electronic device 102) (for example, a speaker or headphones).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101, or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do. According to an embodiment, the sensor module 176 is, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface 177 may support one or more specified protocols that may be used for the electronic device 101 to connect directly or wirelessly with an external electronic device (eg, the electronic device 102 ). According to an embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102). According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus that a user can perceive through tactile or motor sensations. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture a still image and a video. According to an embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 includes a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). It is possible to support establishment and communication through the established communication channel. The communication module 190 operates independently of the processor 120 (eg, an application processor) and may include one or more communication processors supporting direct (eg, wired) communication or wireless communication. According to an embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg : A local area network (LAN) communication module, or a power line communication module) may be included. Among these communication modules, a corresponding communication module is a first network 198 (for example, a short-range communication network such as Bluetooth, WiFi direct or IrDA (infrared data association)) or a second network 199 (for example, a cellular network, the Internet, or It can communicate with external electronic devices through a computer network (for example, a telecommunication network such as a LAN or WAN). These various types of communication modules may be integrated into a single component (eg, a single chip), or may be implemented as a plurality of separate components (eg, multiple chips). The wireless communication module 192 uses subscriber information stored in the subscriber identification module 196 (eg, International Mobile Subscriber Identifier (IMSI)) in a communication network such as the first network 198 or the second network 199. The electronic device 101 can be checked and authenticated.

The antenna module 197 may transmit a signal or power to the outside (eg, an external electronic device) or receive from the outside. According to an embodiment, the antenna module may include one antenna including a conductor formed on a substrate (eg, a PCB) or a radiator formed of a conductive pattern. According to an embodiment, the antenna module 197 may include a plurality of antennas. In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is, for example, provided by the communication module 190 from the plurality of antennas. Can be chosen. The signal or power may be transmitted or received between the communication module 190 and an external electronic device through the at least one selected antenna. According to some embodiments, other components (eg, RFIC) other than the radiator may be additionally formed as part of the antenna module 197.

At least some of the components are connected to each other through a communication method (e.g., a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI))) between peripheral devices and a signal ( E.g. commands or data) can be exchanged with each other.

According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199. Each of the electronic devices 102 and 104 may be a device of the same or different type as the electronic device 101. According to an embodiment, all or part of the operations executed by the electronic device 101 may be executed by one or more of the external electronic devices 102, 104, or 108. For example, when the electronic device 101 needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device 101 In addition or in addition, it is possible to request one or more external electronic devices to perform the function or at least part of the service. One or more external electronic devices receiving the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit a result of the execution to the electronic device 101. The electronic device 101 may process the result as it is or additionally and provide it as at least a part of a response to the request. For this, for example, cloud computing, distributed computing, or client-server computing technology may be used.

An electronic device according to various embodiments disclosed in this document may be a device of various types. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. The electronic device according to the embodiment of the present document is not limited to the above-described devices.

Various embodiments of the present document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutes of the corresponding embodiment. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the above items unless clearly indicated otherwise in a related context. In this document, “A or B”, “at least one of A and B”, “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “A Each of the phrases such as "at least one of, B, or C" may include any one of the items listed together in the corresponding one of the phrases, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish the component from other Order) is not limited. Some (eg, first) component is referred to as “coupled” or “connected” to another (eg, second) component, with or without the terms “functionally” or “communicatively”. When mentioned, it means that any of the above components may be connected to the other components directly (eg by wire), wirelessly, or via a third component.

The term "module" used in this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic blocks, parts, or circuits. The module may be an integrally configured component or a minimum unit of the component or a part thereof that performs one or more functions. For example, according to an embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document include one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) that can be read by a machine (eg, electronic device 101). It may be implemented as software (for example, the program 140) including them. For example, the processor (eg, the processor 120) of the device (eg, the electronic device 101) may call and execute at least one command among one or more commands stored from a storage medium. This enables the device to be operated to perform at least one function according to the at least one command invoked. The one or more instructions may include code generated by a compiler or code executable by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here,'non-transitory' only means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term refers to the case where data is semi-permanently stored in the storage medium. It does not distinguish between temporary storage cases.

According to an embodiment, a method according to various embodiments disclosed in the present document may be provided by being included in a computer program product. Computer program products can be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a device-readable storage medium (e.g. compact disc read only memory (CD-ROM)), or through an application store (e.g. Play Store™) or two user devices (e.g., compact disc read only memory (CD-ROM)). It can be distributed (e.g., downloaded or uploaded) directly between, e.g. smartphones). In the case of online distribution, at least a part of the computer program product may be temporarily stored or temporarily generated in a storage medium that can be read by a device such as a server of a manufacturer, a server of an application store, or a memory of a relay server.

According to various embodiments, each component (eg, module or program) of the above-described components may include a singular number or a plurality of entities. According to various embodiments, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components in the same or similar to that performed by the corresponding component among the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component may be sequentially, parallel, repeatedly, or heuristically executed, or one or more of the operations may be executed in a different order or omitted. Or one or more other actions may be added.

2 is a block diagram 200 illustrating a program 140 according to various embodiments. According to an embodiment, the program 140 includes an operating system 142 for controlling one or more resources of the electronic device 101, middleware 144, or an application 146 executable in the operating system 142. Can include. The operating system 142 may include, for example, Android™, iOS™, Windows™, Symbian™, Tizen™, or Bada™. At least some of the programs 140 are, for example, preloaded on the electronic device 101 at the time of manufacture, or when used by a user, an external electronic device (eg, electronic device 102 or 104), or a server ( 108)).

The operating system 142 may control management (eg, allocation or retrieval) of one or more system resources (eg, process, memory, or power) of the electronic device 101. Operating system 142, additionally or alternatively, other hardware devices of the electronic device 101, for example, the input device 150, the sound output device 155, the display device 160, the audio module 170 , Sensor module 176, interface 177, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196, or One or more driver programs for driving the antenna module 197 may be included.

The middleware 144 may provide various functions to the application 146 so that a function or information provided from one or more resources of the electronic device 101 can be used by the application 146. The middleware 144 is, for example, an application manager 201, a window manager 203, a multimedia manager 205, a resource manager 207, a power manager 209, a database manager 211, and a package manager 213. ), a connectivity manager 215, a notification manager 217, a location manager 219, a graphic manager 221, a security manager 223, a call manager 225, or a voice recognition manager 227. I can.

The application manager 201 may manage the life cycle of the application 146, for example. The window manager 203 may manage one or more GUI resources used on a screen, for example. The multimedia manager 205, for example, identifies one or more formats required for playback of media files, and performs encoding or decoding of a corresponding media file among the media files by using a codec suitable for the selected corresponding format. You can do it. The resource manager 207 may manage the source code of the application 146 or a memory space of the memory 130, for example. The power manager 209 manages the capacity, temperature, or power of the battery 189, for example, and may determine or provide related information necessary for the operation of the electronic device 101 by using the corresponding information. . According to an embodiment, the power manager 209 may interwork with a basic input/output system (BIOS) (not shown) of the electronic device 101.

The database manager 211 may create, search, or change a database to be used by the application 146, for example. The package manager 213 may manage installation or update of an application distributed in the form of, for example, a package file. The connectivity manager 215 may manage, for example, a wireless connection or a direct connection between the electronic device 101 and an external electronic device. The notification manager 217 may provide a function for notifying the user of the occurrence of a designated event (eg, incoming call, message, or alarm), for example. The location manager 219 may manage location information of the electronic device 101, for example. The graphic manager 221 may manage, for example, one or more graphic effects to be provided to a user or a user interface related thereto.

The security manager 223 may provide, for example, system security or user authentication. The telephony manager 225 may manage, for example, a voice call function or a video call function provided by the electronic device 101. The voice recognition manager 227 transmits, for example, a user's voice data to the server 108, and a command corresponding to a function to be performed in the electronic device 101 based at least in part on the voice data, Alternatively, text data converted based at least in part on the voice data may be received from the server 108. According to an embodiment, the middleware 244 may dynamically delete some of the existing components or add new components. According to an embodiment, at least a part of the middleware 144 may be included as a part of the operating system 142 or implemented as separate software different from the operating system 142.

The application 146 is, for example, a home 251, a dialer 253, an SMS/MMS 255, an instant message (IM) 257, a browser 259, a camera 261, and an alarm 263. , Contacts (265), voice recognition (267), email (269), calendar (271), media player (273), album (275), watch (277), health (279) (e.g. Biometric information measurement), or environmental information 281 (eg, air pressure, humidity, or temperature information measurement) application may be included. According to an embodiment, the application 146 may further include an information exchange application (not shown) capable of supporting information exchange between the electronic device 101 and an external electronic device. The information exchange application may include, for example, a notification relay application configured to deliver specified information (eg, a call, a message, or an alarm) to an external electronic device, or a device management application configured to manage an external electronic device. have. The notification relay application, for example, transmits notification information corresponding to a specified event (eg, mail reception) generated by another application (eg, email application 269) of the electronic device 101 to an external electronic device. I can. Additionally or alternatively, the notification relay application may receive notification information from an external electronic device and provide it to the user of the electronic device 101.

The device management application includes, for example, an external electronic device that communicates with the electronic device 101 or some components thereof (for example, the display device 160 or the camera module 180). -Off) or a function (eg, brightness, resolution, or focus of the display device 160 or the camera module 180) may be controlled. The device management application may additionally or alternatively support installation, deletion, or update of an application operating in an external electronic device.

3 illustrates a configuration of an electronic device 300 (eg, the electronic device 101 of FIG. 1) according to an exemplary embodiment.

The electronic device 300 according to an embodiment may include a processor 310 (eg, the processor 120 of FIG. 1) and a memory 320 (eg, the memory 130 of FIG. 1 ). The processor 310 may be configured to be operatively connected to the memory 320. The processor 310 may control components of the electronic device 300 or process data by executing instructions stored in the memory 320. According to an embodiment, the electronic device 300 includes a camera 330 (eg, the camera module 180 of FIG. 1 ), a display 340 (eg, the display device 160 of FIG. 1 ), and a communication module 350. It may further include at least one of (eg, the communication module of FIG. 1).

The processor 310 may acquire image data including an image photographed of the user 1 through the camera 330. According to an embodiment, the processor 310 may obtain image data including an RGB value and a depth value. For example, the camera 330 may detect a distance value to a subject based on a Time of Flight (TOF) method. The processor 310 may recognize the motion of the subject photographed in the image by using the depth value. The processor 310 may track the user's face photographed in the image by using the recognized motion information. According to an embodiment, the processor 310 may acquire image data captured using the camera 330 included in the electronic device 300. According to another embodiment, the processor 310 may acquire image data captured through a camera 330 included in another device. For example, the processor 310 is located near the electronic device 300 and may receive image data from a device equipped with a camera through a communication connection (eg, short-range wireless communication or wired communication) using the communication module 350. I can. As another example, the processor 310 may obtain image data downloaded from an external server (eg, the server 108 of FIG. 1) by the electronic device 300. The method of obtaining the image data by the processor 310 may be variously modified.

The processor 310 that has obtained the image data may analyze the image data. The processor 310 according to an embodiment may recognize a user's face or skin in an image and track a location of the recognized face or skin in the image. In order to measure a user's biometric signal (eg, a pulse wave), a pixel value included in an area in which the user's skin is photographed may be required among information included in the image data. The processor 310 may identify a location of an area in which the user's face is photographed in the image. The processor 310 convolves the image before and after the image sequence included in the image data to identify the location of the region where the user's face is photographed, and determines the region where the user's face is photographed based on the result. I can.

The processor 310 may divide the image channels included in the area where the face of the image data is captured into color channels (eg, RGB channels), and select a channel necessary for measuring a biosignal from among the image channels. The processor 310 may remove color values that are not suitable for extracting a bio-signal from among color values of an image channel. For example, the processor 310 may exclude color values of a region in which the reflected light is photographed or a region other than the skin (eg, eyebrows) photographed from the analysis target. The processor 310 recognizes a region in which the user's face has been photographed in the image, and the pixel value of pixels within a range in which it can be determined that the skin has been photographed among regions recognized as photographing the user's face A biosignal may be obtained based on.

The processor 310 may amplify the values of the selected color channels and remove noise to extract the biosignal. For example, the processor 310 may adjust a signal to noise ratio (SNR) by applying a nonlinear scale factor to each RGB channel. The processor 310 may calculate a biosignal based on a signal to which a scale factor is applied.

The processor 310 may image-process the preprocessed image signal and then calculate the heart rate based on the time domain. Alternatively, the processor 310 may convert the image signal into the frequency domain and then calculate the heart rate based on the frequency domain. The processor 310 may control the display 340 to display a screen including information on the calculated heart rate. Alternatively, the processor 310 may store biometric information data including the calculated heart rate in the memory 320 or other device (eg, the electronic device 104 or server 108 of FIG. 1) through the communication module 350. Can be transferred to. In the present specification, the biometric information data may refer to information generated by collecting biometric information on the user 1. The biometric information data may be used to diagnose the health status of the user 1 or to determine the characteristics of the user 1's body.

Due to the movement of the user 1, a heart rate value may not be calculated from a bio-signal extracted from image data or a heart rate value that is out of a normal range may be calculated. For example, if the face of the user 1 moves outside the area photographed by the camera 330 due to the movement of the user 1, the heart rate value cannot be calculated from the image data. When the normal heart rate value is not calculated, the processor 310 may determine a correction value for a section in which the normal heart rate value is not calculated. The determined correction value may be output through the display 340 in place of the calculated heart rate value, or may be included in the biometric information data.

The processor 310 may determine a correction value to replace the measured biosignal (biosignal extracted from image data) using the candidate data. In the present disclosure, candidate data may mean data created by predicting how a biosignal of an unmeasured section will be formed. For example, the candidate data is a feature point for predicting how the biosignal of the correction target section will be formed from the biosignal last measured before the section in which the biosignal is not normally measured (hereinafter, "correction target section"). May include information about. As another example, the candidate data may include an artificial biosignal (eg, artificial heartbeat model data (AHD)) created based on a feature point extracted from a previously measured biosignal.

Since the candidate data is generated based on a previously normally measured biosignal, it may have characteristics of a normally measured biosignal. Accordingly, according to an embodiment, the processor 310 may determine whether the calculated value is a normal value by comparing a value calculated from the image data (eg, a heart rate value) with candidate data. For example, the processor 310 may compare the R-R peak interval of data (measured biosignal) associated with the heart rate value and the R-R peak interval of the candidate data. As a result of the comparison, when the R-R peak intervals are similar to each other, the processor 310 may determine that the calculated heart rate value is a normal value.

According to an embodiment, the processor 310 may determine a correction value in consideration of information on an exercise being performed by the user 1. For example, when a guide image content (eg, 1620 in FIG. 16) for guiding a user's exercise is being played through the display 340, the user performs an exercise according to the guide image content (eg, 1620 in FIG. 16). It can be seen as being performed. In this case, the processor 310 may acquire the amount of change in heart rate associated with the guide image content being played (eg, 1620 in FIG. 16 ). The amount of change in heart rate may be included in the guide image content (eg, 1620 in FIG. 16) or may be searched using an indicator indicating an exercise provided by the guide image content (eg, 1620 in FIG. 16 ). The processor 310 may determine a correction value such that the biosignal within the correction target section becomes a signal indicating the acquired heart rate change amount.

According to an embodiment, the processor 310 may determine a correction value according to the length of the region to be corrected. For example, if the correction target section is short (e.g. 3 seconds), the correction value is determined based on the candidate data, and if the correction target section is longer than a threshold value (e.g. 5 seconds), a pattern database (not shown) The correction value may be determined based on the data pattern stored in (not). The data pattern may include user information (eg, gender, age, weight, or height) of the user 1 and data defining an amount of change in heart rate according to the type of exercise that the user 1 is performing. For example, the processor 310 may search for a heart rate change amount from the pattern database based on user information and the type of exercise, and determine a correction value for the correction target section based on the searched heart rate change amount and a previous heart rate value. User information may refer to information indicating a user's attributes (eg, gender, age, weight, height).

When the normal biosignal is measured again after the correction target section occurs, the processor 310 according to an embodiment may recalibrate the correction value of the correction target section among the biometric information data based on the measured biosignal again. . Since the correction value input to the correction target section is not actually a measured value, non-linear data may be generated at the point when the correction target section ends. The processor 310 may correct the value of the correction target section again based on the biosignal normally measured after the time point to which the correction target section ends.

The processor 310 may control the display 340 to display at least a part of the corrected biometric information data. Alternatively, the processor 310 may control the display 340 to determine recommendation information to be provided to the user based on the biometric information data and display the determined recommendation information. The display 340 may be a device that is hardware coupled to the electronic device 300, but may be a device to which communication is connected through the communication module 350. Alternatively, the display 340 may be a device controlled through an external server in the IoT system.

According to an embodiment, the recommended information may include information related to exercise of the user 1. For example, the processor 310 may collect biometric information data while outputting guide image content (eg, 1620 in FIG. 16) for guiding a user's exercise through the display 340. Since the guide image content (e.g., 1620 in FIG. 16) is an image that guides the user to perform a specified exercise, the processor 310 may use biometric information data based on information included in the guide image content (e.g., 1620 in FIG. 16). It is possible to determine which exercise the user is performing at the time when is collected. However, the method of determining which exercise the user is performing by the electronic device is not limited thereto. For another example, the processor 310 may detect the user's movement and determine the user's exercise progress using an exercise matching algorithm corresponding to the detected movement. The processor 310 may track a user's movement using the image and depth information acquired through the 3D camera, and determine which exercise the user is performing based on exercise information matched with the user's movement. The processor 310 may compare the exercise performed by the user at the time the biometric information data is collected and reference data including average biometric information of other users while performing the same type of exercise. Alternatively, the reference data may be biometric information data measured when the user 1 previously performed the same exercise. The reference data may be stored in the memory 320 or may be downloaded through the communication module 350. Alternatively, the processor 310 may transmit the biometric information data to an external server, and may receive a comparison result of the biometric information data and the reference data from the external server. For example, the processor 310 may compare the change amount of the heart rate value included in the biometric information data with the change amount of the reference heart rate. The processor 310 may output recommended exercise intensity information for increasing exercise intensity when the increase in the heart rate value is smaller than the value of the reference heart rate change amount. Conversely, when the increase in the heart rate value is greater than the value of the reference heart rate change amount, the processor 310 may output recommended exercise intensity information for lowering the exercise intensity.

For another example, the processor 310 may output recommended exercise information recommending an exercise to be performed by the user based on a comparison result of biometric information of users having the same or similar user information as the user 1 in the same exercise program. have. Alternatively, the processor 310 may analyze the biometric information data of the user 1 for each exercise program and determine the ability value for each body part. The processor 310 may provide the capability value for each body part to the user 1 or may output recommended exercise information according to the capability value for each body part. For a more specific example, when the measured heart rate increases significantly and the number of failures is high when the exercise program for training the thigh muscles is provided, the processor 310 may output the exercise program for training the thigh muscles as recommended exercise information. have.

The electronic device 300 according to an embodiment may easily track a change in heart rate even during exercise of the user 1, and thus provide data that facilitates early detection of arrhythmia findings according to a change in heart rate during exercise.

4 is a flowchart 400 illustrating a process of generating biometric information data by an electronic device (eg, the electronic device 101 of FIG. 1 and the electronic device 300 of FIG. 3) according to an embodiment. In the operation of the electronic device disclosed in FIG. 4 and related description, the processor of the electronic device (eg, the processor 120 of FIG. 1, the processor 310 of FIG. 3) is a memory (eg, the memory 130 of FIG. 1, FIG. It can be understood that it is executed by executing instructions stored in the memory 320 of 3.

In operation S410, the electronic device may acquire image data photographing a user. According to an embodiment, the electronic device may acquire image data including an image captured using a camera (eg, the camera module 180 of FIG. 1 and the camera 330 of FIG. 3) provided in the electronic device. have. According to another embodiment, the electronic device may receive image data acquired through an external device.

In operation S420, the electronic device may analyze the acquired image data. According to an embodiment, the electronic device may recognize a user from an image included in image data and classify an image channel to be used to extract a biosignal. In operation S430, the electronic device may calculate a heart rate value from the biosignal extracted from the classified image channel.

In operation S440, the electronic device may determine whether the heart rate value is normally calculated. If the user's face is not recognized in operation 420 or the biosignal that is the basis of the heart rate value is out of the normal range in operation S430, the electronic device may determine that the heart rate value has not been normally calculated. According to an embodiment, the electronic device may not capture a shape of the user by the camera while the user performs other actions (eg, drinking water, talking on the phone, or resting) during exercise. If the user is not photographed, the electronic device may repeatedly perform operation S410 until the user is photographed. Alternatively, when the user is not recognized in the image (eg, when the user is not photographed for a specified time or longer), the electronic device may terminate a program for generating biometric information data. When it is determined in operation S440 that the heart rate value is normally calculated, biometric information data including the heart rate calculated in operation S450 may be generated. The generated biometric information data may be stored in a memory (eg, the memory 130 of FIG. 1 or the memory 320 of FIG. 3 ), or may be transmitted to an external server.

If the heart rate value is not normally calculated, the electronic device may determine a correction value in operation S445. According to an embodiment, the electronic device may determine a correction value based on at least one of a last measured value of a normal biosignal, a feature point extracted from existing data, information on an exercise being performed by a user, and a data pattern. . In operation S455, the electronic device may generate biometric information data in which a value of the biosignal associated with the heart rate value is replaced with the determined correction value.

5 illustrates an example of an electronic device 500 according to an embodiment.

According to an embodiment, the electronic device 500 (for example, the electronic device 101 of FIG. 1 and the electronic device 300 of FIG. 3) that measures a biosignal may be configured as a stand-alone device. . Referring to FIG. 5, the electronic device 500 may photograph a user 1 using a camera 530 provided on one surface, and obtain biometric information data from the captured image. The electronic device 500 includes at least a portion of the biometric information data or biometric data through an output device (for example, the display device 160 of FIG. 1, the sound output device 155, and the display 340 of FIG. 3) provided in the electronic device. Information related to data can be output.

6 shows an example of a system 600 for acquiring biometric information data, according to an embodiment.

According to an embodiment, an apparatus for acquiring and processing a bio-signal may be configured as a system 600 including a plurality of hardware devices connected through a network 699. For example, the plurality of devices 601, 630, and 640 may be devices connected to an Internet of Things (IoT) system controlled by the server 610. Referring to FIG. 6, while the user 1 is exercising through the exercise device 601, the camera device 630 may transmit image data including an image of the user 1 to the server 610. The server 610 may analyze image data to obtain biometric information data. The server 610 may display at least part of the biometric information data or information related to the biometric information data through the display device 640. According to an embodiment, the server 610 displays information on the exercise state of the user 1 required to correct the biometric information data or determine information related to the biometric information data. It can be obtained based on information related to the content being played through the device 640. For example, when an image guiding the lower body exercise is being played through the display device 640, or the exercise device 601 in motion is a device for lower body exercise, the server 610 provides the user 1 to exercise the lower body. It can be judged as being on the job. According to another embodiment, the server 610 may analyze an image acquired from the camera device 630 to obtain information on a user's exercise state.

In the IoT environment, while the user makes daily movements by using a camera located in the user's living space, heart rate information can be obtained while continuously monitoring the user's heart rate. The system 600 may be configured to automatically transmit a message to a designated contact (eg, an emergency agency or a guardian's communication device) when a singularity occurs in the heart rate information acquired during the user's daily life.

According to an embodiment, the server 610 may receive image data obtained from the camera device 630. The server 610 may generate biometric information data including a heart rate value calculated from the received image data. Alternatively, the server 610 may obtain biometric information data including a heart rate value calculated based on image data obtained from the camera device 630. The server 610 may compare the heart rate data of the arrhythmia patient and the biometric information data stored in the memory of the server 610. Here, the biometric information data may be corrected data. The arrhythmia patient heart rate data may include information on a heart rate value measured while the arrhythmia patients perform exercise. As a result of the comparison, the server 610 may determine the number of times that a characteristic (eg, an abnormal pattern) included in the heart rate data of the arrhythmia patient occurs in the biometric information data. In order to compare the arrhythmia patient heart rate data and the biometric information data, the server 610 may extract a feature point (data representing the feature) from the arrhythmia patient heart rate data and the biometric information data through a designated process, and compare the extracted feature points. When the number of times the determined characteristic occurs in the biometric information data is equal to or greater than a specified threshold value, the server 610 may output information on the abnormal heartbeat pattern to the display device 640 to display information. However, FIG. 6 is for explaining the structure of an embodiment, and the shape of the device may be differently modified according to the embodiment. For example, the system 600 of FIG. 6 may be implemented in the form of a healthcare device having functions of a display 640, a camera 630, and a server 610 without a separate server 610 being built. have.

7 illustrates another example of an electronic device 700 (eg, the electronic device 101 of FIG. 1) that acquires biometric information data, according to an exemplary embodiment.

The electronic device 700 according to an embodiment receives image data from the camera device 730 and processes the image data to transmit at least part of the biometric information data or information related to the biometric information data through the display device 740. Can be printed. The electronic device 700 may be connected to the camera device 730 or the display device 740 through short-range wireless communication (eg, Bluetooth™, Wi-Fi™).

The embodiments illustrated in FIGS. 5 to 7 are for explaining several embodiments, and the hardware configuration is not limited thereto, and the hardware configuration may be variously changed according to the embodiments.

8 is a flowchart 800 illustrating a process of calculating a heart rate value by an electronic device (eg, the electronic device 101 of FIG. 1 and the electronic device 300 of FIG. 3) according to an embodiment. In the operation of the electronic device disclosed in FIG. 8 and related description, the processor of the electronic device (eg, the processor 120 of FIG. 1, the processor 310 of FIG. 3) is a memory (eg, the memory 130 of FIG. 1, FIG. It can be understood that it is executed by executing instructions stored in the memory 320 of 3.

In operation S810, the electronic device may convolve the before and after images in the image sequence included in the image data. Based on the convolution result, the electronic device may compare the previous image frame and the current image frame in operation S820 to determine a first area within the current image frame in which the user's face is photographed. The user's face includes skin, eyes, lips, or eyebrows, but since only information on the user's skin color is required for heart rate detection, the remaining elements can be removed.

In operation 830, the electronic device may divide an image channel in the first region. In operation S840, the electronic device may remove an unnecessary color value from among values in the first area. For example, the electronic device may select color values excluding channels having values within a range similar to that of a skin value from among the divided image channels.

In operation S850, the electronic device may amplify the selected image channel values. That is, the electronic device can increase the size of the signal by amplifying the intensity of each pixel. For example, the electronic device may assign a scale factor to each color channel. By amplifying the value of the video channel, a signal having a large value becomes relatively larger, thereby improving a signal-to-noise ratio. In operation S860, the electronic device may calculate a heart rate value based on the amplified color channel value.

9 is a conceptual diagram illustrating a method of recognizing a face region for calculating a heart rate value by an electronic device (eg, the electronic device 101 of FIG. 1 and the electronic device 300 of FIG. 3 ), according to an exemplary embodiment It is a drawing for.

The electronic device may determine the first area 910 in the image 900 in which the user's face is photographed. Also, the electronic device may find a skin color in the first area 910 and determine a center point 920 thereof. The electronic device may determine the second area 931 in which pixels determined to be skin color values are located while amplifying the pixel value radially from the center point 920. The area 941 in which the eyebrows other than the color value of the skin are photographed may not be included in the second area 931.

Similarly, the electronic device determines a plurality of second regions 931, 932, 933, 934, and values of pixels included in an area combining the plurality of second regions 931, 932, 933, 934 By extracting the bio-signal using, the bio-signal can be extracted based on areas having a color value of the skin excluding the eyebrow area 941, the eye area 942, and the area 943 in which the reflected light is photographed. have.

10 is a flowchart 1000 illustrating a process of determining whether a heart rate value is normally calculated by an electronic device (eg, the electronic device 101 of FIG. 1 and the electronic device 300 of FIG. 3 ), according to an exemplary embodiment. . In the operation of the electronic device disclosed in FIG. 10 and related description, the processor of the electronic device (eg, the processor 120 of FIG. 1, the processor 310 of FIG. 3) is a memory (eg, the memory 130 of FIG. 1, FIG. It can be understood that it is executed by executing instructions stored in the memory 320 of 3.

In operation S1010, the electronic device may detect an object in the image data. According to an embodiment, the electronic device may determine whether an object corresponding to a user exists among the detected objects. If an object corresponding to the user does not exist, the electronic device may output a message inducing the user to be photographed in the image. Alternatively, the electronic device may terminate the process.

In operation S1020, the electronic device may determine whether the user's face is detectable within the image data. When the user's face is not detected in the image data, the electronic device may determine that the heart rate is not normally calculated.

When the user's face is detected in the image data, in operation S1030, the electronic device may calculate a heart rate value based on information in the detected face area. For example, the electronic device may extract a pulse wave as a bio-signal from a skin color value in the facial area, and determine a heart rate generated per unit time from the pulse wave.

In operation S1040, the electronic device may determine whether the heart rate value is normally calculated. For example, when the R-R peak interval value of the biosignal is included in the normal range, the electronic device may determine that the heart rate value is normally calculated. Conversely, when the R-R peak interval value of the bio-signal is out of the normal range, the electronic device may determine that the heart rate value is not normally calculated.

11 is a diagram illustrating a process of determining a correction value when a heart rate value is not normally calculated by an electronic device (eg, the electronic device 101 of FIG. 1 or the electronic device 300 of FIG. 3 ), according to an embodiment One is a flow chart 1100. In the operation of the electronic device disclosed in FIG. 11 and related description, the processor of the electronic device (eg, the processor 120 of FIG. 1, the processor 310 of FIG. 3) is a memory (eg, the memory 130 of FIG. 1, FIG. It can be understood that it is executed by executing instructions stored in the memory 320 of 3.

In operation S1110, the electronic device may determine whether a heart rate value that is not normally calculated can be corrected using candidate data. According to an embodiment, when the size of a section in which the heart rate value is not normally calculated is less than or equal to the threshold value, the electronic device may determine that the heart rate value can be corrected based on candidate data. Conversely, when the size of the section in which the heart rate value is not normally calculated is greater than the threshold value, the electronic device may determine that the heart rate value cannot be corrected as candidate data. That is, data may be corrected based on candidate data for a biosignal lost for a short period of time, and data may be corrected based on a different method when the period in which the biosignal is lost becomes longer.

Here, the candidate data may include data modeled to have characteristics similar to those of the measured heart rate data by comparing with the previously measured heart rate data. The candidate data may be referred to as a heart rate data prediction model. The electronic device may update candidate data based on a result of learning measured data using machine learning using a genetic algorithm.

If it is determined that data can be corrected based on the candidate data, in operation S1120, the electronic device may determine a correction value for correcting a heart rate value that is not normally calculated based on the candidate data. If it is determined that data cannot be corrected based on the candidate data, in operation S1125, the electronic device may determine a correction value based on the pattern similarity. In operation S1125, in order to determine the correction value based on the pattern similarity, the electronic device may determine the correction value based on user information and a normally measured last heart rate value. Here, the user information may include information indicating a user's attributes (eg, gender, age, weight, height) and information related to an exercise that the user is performing (eg, type of exercise, intensity of exercise). The electronic device may search for a data pattern corresponding to user information from the pattern database. The data pattern may include information on an amount of change in heart rate corresponding to user information. The electronic device may determine a correction value for correcting the heart rate value based on the searched heart rate change amount and the last heart rate value. According to an embodiment, the electronic device may search for a data pattern from a pattern database stored in the electronic device. According to another embodiment, the electronic device may provide user information to an external server, and the external server may retrieve a data pattern from a pattern database and transmit it to the electronic device.

In operation S1130, the electronic device may generate biometric information data including a correction value. The biometric information data including the correction value may include a heart rate value calculated based on the correction value. According to an embodiment, when a value included in the biometric information data is a corrected value, the electronic device may generate the biometric information data including an indicator indicating that the value has been corrected.

The flowchart 1100 illustrated in FIG. 11 is for explaining a method of correcting data that is not normally measured based on an embodiment, and a process of correcting data may be applied differently from the flowchart 1100.

12 conceptually illustrates an operation of correcting a heart rate value by using candidate data by an electronic device (eg, the electronic device 101 of FIG. 1 and the electronic device 300 of FIG. 3) according to an embodiment.

The electronic device may calculate a normal heart rate value from the pulse wave extracted in the normal section 1210. However, in the abnormal section 1220, since the pulse wave is not normally extracted, the heart rate value cannot be normally extracted. Therefore, it is necessary to correct the data of the abnormal section 1220. Referring to the abnormal section 1220 of FIG. 12, since the data of the abnormal section 1220 does not have a normal RR peak interval, the electronic device according to an exemplary embodiment uses the normal section 1210 based on the RR peak interval of the data. And the abnormal section 1220 can be identified.

In order to correct the data of the abnormal section 1220, the electronic device may replace the data of the abnormal section 1220 with candidate data 1200. According to an embodiment, the candidate data 1200 may be data created by predicting data to occur in the abnormal section 1220 by learning feature points extracted from data in the normal section 1210.

13 illustrates an electronic device (eg, the electronic device 101 of FIG. 1, the electronic device 300 of FIG. 3) additionally correcting the corrected biometric information data when a normal heart rate value is calculated, according to an exemplary embodiment. A flowchart 1300 depicting the process. In the operation of the electronic device disclosed in FIG. 13 and related descriptions, the processor of the electronic device (eg, the processor 120 of FIG. 1, the processor 310 of FIG. 3) is a memory (eg, the memory 130 of FIG. 1, FIG. It can be understood that it is executed by executing instructions stored in the memory 320 of 3.

When the heart rate value is normally calculated from the image data, in operation S1310, the electronic device may determine whether data corresponding to the heart rate value before the calculated heart rate value in the biometric information data is a corrected value. The previous data may be a heart rate value prior to the calculated heart rate value, or may mean another biosignal value for calculating a previous heart rate value. In a state in which the biometric information data is configured based on the corrected value because the heart rate value is not normally calculated, when a normal heart rate value is input, since the corrected value of the previous data is not an actual value, an error may occur with the actual heart rate value. In this case, a non-linear section may occur between the previous data and the calculated heart rate value. Data may be discontinuously formed because the corrected value of the previous data does not match the actual value.

In order to use the biometric information data more usefully for diagnosing the user's condition, it is necessary to correct the discrepancy between the corrected value and the actual value. Therefore, if the previous data for the calculated heart rate value is a corrected value, the electronic device evaluates the correction value of the previous data based on the heart rate value calculated in operation S1320, and if the correction value is not valid, the correction value of the previous data Can be recalibrated. For example, if the correction value of previous data and the calculated heart rate value do not have a linear relationship, the electronic device may determine that the correction value is not valid. The electronic device may recalibrate the previous data so that the corrected value of the previous data has a linear value with the calculated heart rate value.

In operation S1330, the electronic device may generate biometric information data including the calculated heart rate value.

14 conceptually illustrates an operation of recalibrating a corrected value by an electronic device (eg, the electronic device 101 of FIG. 1 and the electronic device 300 of FIG. 3) according to an exemplary embodiment.

Referring to FIG. 14, a biosignal between a first time point 1410 at which the biosignal of the normal section 830 starts and a second time point 1420 at which the corrected section 821 before the normal section 830 ends. A point 1430 at which is formed non-linearly may occur.

In order to remove the point 1430 where the nonlinearity occurs, the electronic device converts data of at least a portion of the section 1425 including the second point of view 1420 of the corrected section 821 into the data of the first point of view 1410. It can be corrected to have a linear relationship with.

However, FIG. 14 is merely a description of an exemplary embodiment of recalibrating a bio-signal, but is not limited thereto. According to another embodiment, the electronic device corrects the value of the corrected section based on the heart rate values normally calculated before and after the corrected data to a value that linearly connects the heart rate value normally calculated before and after the corrected data. You may. For example, the heart rate value was calculated as 80 at the end point of the first section, the heart rate value was corrected to 90 in the second section after the first section, and the heart rate value at the point in the third section after the second section When 105 is calculated, the electronic device may recalibrate the heart rate value of the second section to a heart rate value that linearly increases so that the heart rate value of the second section starts at 80 at the time point of the second section and becomes 105 at the end point.

15 is a diagram illustrating a process in which an electronic device (eg, the electronic device 101 of FIG. 1, the electronic device 300 of FIG. 3) generates candidate data based on a feature point and generates biometric information data according to an embodiment. One is a flow chart 1500. In the operation of the electronic device disclosed in FIG. 15 and related description, the processor of the electronic device (eg, the processor 120 of FIG. 1, the processor 310 of FIG. 3) is a memory (eg, the memory 130 of FIG. 1, FIG. It can be understood that it is executed by executing instructions stored in the memory 320 of 3.

In operation S1510, the electronic device may generate candidate data. In the case of generating candidate data for the first time, since the electronic device cannot know an optimal value or pattern, candidate data may be configured by estimating a target suitability. Target fit may mean an individual's average normal heart rate. That is, the electronic device may configure initial candidate data to indicate the user's normal heart rate. When candidate data is generated while the feature point data is accumulated, in operation S1510, the electronic device may generate candidate data based on the feature point data. According to an embodiment, the candidate data may be artificially generated heartbeat data.

In operation S1520, the electronic device may determine whether input data exists. The input data may mean a calculated heart rate value or a biosignal for calculating a heart rate value. When the user's face is not recognized from the image data, since the biosignal is not extracted and the heart rate value is not calculated, the electronic device may determine that the input signal does not exist. If the input data does not exist, since the heart rate value cannot be calculated, in operation S1525, the electronic device may generate biometric information data based on the candidate data.

When there is input data, in operation S1530, the electronic device may determine whether the amount of feature point data is sufficient. For example, in an electronic device, if the feature point data contains more than 500 frames and candidate data is created based on the feature point data, there is no change in the result at each run, and the candidate data is stably created. It can be judged that the amount is sufficient. The feature point data may mean data including information on feature points extracted from a bio-signal so that the device can learn by using a machine learning model. If the amount of feature point data is not sufficient, since candidate data generated in operation S1510 is not reliable, the electronic device may not use the candidate data to generate a correction value. In this case, the electronic device may extract the feature point from the input data in operation S1550, update the feature point data, and generate biometric information data based on a heart rate value included in the input data or calculated from the input data.

When the amount of feature point data is sufficient, the electronic device may compare a heart rate value based on the input data and candidate data in operation S1540. In order to compare the heart rate value and the candidate data, the electronic device may compare the R-R peak interval for the heart rate value and the R-R peak interval for the candidate data. Since the candidate data is a biosignal that is expected to be measured when the biosignal is normally measured, the biosignal normally measured has an R-R peak interval similar to the R-R peak interval of the candidate data. Accordingly, it may be determined whether the measured bio-signal or heart rate value is a value requiring correction through comparison of the R-R peak interval. According to an embodiment, the R-R peak interval with respect to a heart rate value may mean an R-R peak interval of a pulse wave curve for calculating a heart rate value. If the difference between the RR peak interval for the calculated heart rate value and the RR peak interval for the candidate data is within the specified range, the electronic device extracts and updates feature points from information related to heart rate calculation in operation S1550, and updates the calculated heart rate value. Biometric information data can be generated on the basis of it. When the difference between the RR peak interval for the calculated heart rate value and the RR peak interval for the candidate data is outside the specified range (or greater than the value in the specified range), the electronic device determines the heart rate value based on the candidate data in operation S1555. It is possible to generate biometric information data replaced (corrected) with the corrected value.

In operation S1560, when the process of measuring the heart rate value is not finished, the electronic device may repeatedly perform the operation of operation S1510.

According to an embodiment, the electronic device may allow only candidate data having a similar R-R peak interval value to remain among the generated candidate data. The electronic device may store the R-R peak interval value of the surviving candidate data and apply the stored R-R peak interval value to the next generation of candidate data.

16 illustrates an example of a screen 1600 displayed by an electronic device (eg, the electronic device 101 of FIG. 1 and the electronic device 300 of FIG. 3) according to an embodiment.

The screen 1600 displayed on the display (for example, the display 340 of FIG. 3) by the electronic device includes an image 1610 captured by a camera (eg, the camera 330 of FIG. 3) and the guide image content ( 1620). The guide image content 1620 may include, for example, a video stored in an electronic device or streamed from an external server.

According to an embodiment, the screen 1600 may include a heart rate display area 1630 indicating a heart rate value measured based on image data. When the user's body is not in contact with the contact-type measurement device, it is difficult to measure the heart rate value of the user during exercise, but according to embodiments disclosed in this document, the heart rate value of the user during exercise may be provided only by image analysis.

According to an embodiment, the screen 1600 displays a reference heart rate display area (not shown) representing a range of normal heart rate values for users having the same attributes as user information (eg, gender, age, height, or weight). Can include. The reference heart rate display area may be expressed in various ways according to exemplary embodiments. For example, the reference heart rate display area may be displayed on the heart rate display area 1630 to be compared with the measured heart rate value.

According to an embodiment, the screen 1600 may include a calorie display area 1640 that displays a calorie consumption amount calculated based on a heart rate value. The electronic device may calculate a more accurate amount of heat consumption by using the heart rate value measured during exercise. According to an embodiment, the electronic device may calculate the amount of heat consumed based on a type of exercise and a heart rate value corresponding to the guide image content 1620. For example, the electronic device may calculate the amount of heat consumed based on Equation 1 below.

M _count is the oxygen consumption coefficient for each exercise, weight is the user's weight, HR is the heart rate, HR _REST is the resting heart rate, and k _AGE is the age correlation coefficient.

17 illustrates an electronic device (eg, the electronic device 101 of FIG. 1, the electronic device 300 of FIG. 3) according to an embodiment together with a guide image content (eg, the guide image content 1620 of FIG. 16 ). A flow chart showing a process 1700 for providing biometric information data. In the operation of the electronic device disclosed in FIG. 17 and related description, the processor of the electronic device (eg, the processor 120 of FIG. 1, the processor 310 of FIG. 3) is a memory (eg, the memory 130 of FIG. 1, FIG. It can be understood that it is executed by executing instructions stored in the memory 320 of 3.

In operation S1710, the electronic device displays image data (eg, the image (eg, the image of FIG. 16)) including an image captured through a camera (eg, the camera 330 of FIG. 3) on the display (eg, the display 340 of FIG. 1610)) and guide video content can be output. Also, the electronic device may calculate a heart rate value based on image data including the captured image.

In operation S1720, the electronic device may determine whether the heart rate value is normally calculated. When the heart rate value is normally calculated, the electronic device may generate biometric information data in operation S1750 based on the calculated heart rate value.

When the heart rate value is not normally calculated, in operation S1730, the electronic device may acquire the normal heart rate change amount from the heart rate change amount data of the exercise program. According to an embodiment, the heart rate change amount data may be big-data in which measured data is accumulated. For example, the electronic device may acquire information about an exercise guided by the guide image content (eg, exercise intensity, exercise portion, and type of exercise such as strength exercise or aerobic exercise). For another example, the electronic device may analyze an image acquired through a 3D camera to obtain information on exercise (eg, type of exercise, number of repetitions of exercise). The electronic device may search for information on the amount of change in heart rate measured when performing the same exercise from the data on the amount of change in heart rate, based on the information on the exercise. When the heart rate change amount is acquired, in operation S1740, the electronic device may determine a correction value based on the heart rate change amount and the previous heart rate value. For example, in operation S1720, the electronic device may determine a correction value as a changed value based on a heart rate change amount from a heart rate value normally acquired before a heart rate value that is a target of determination in operation S1720. When the correction value is determined, in operation S1750, the electronic device may generate the corrected biometric information data based on the correction value.

The process illustrated by flow chart 1700 may be executed repeatedly while measuring heart rate values.

18 is a flowchart illustrating a process of providing recommended exercise intensity information based on biometric information data by an electronic device (eg, the electronic device 101 of FIG. 1 and the electronic device 300 of FIG. 3) according to an embodiment (1800). In the operation of the electronic device disclosed in FIG. 18 and related description, the processor of the electronic device (eg, the processor 120 of FIG. 1, the processor 310 of FIG. 3) is a memory (eg, the memory 130 of FIG. 1, FIG. It can be understood that it is executed by executing instructions stored in the memory 320 of 3.

In operation S1810, the electronic device may acquire exercise type information of the user. Here, the exercise type information may mean information indicating what kind of exercise the user is performing (eg, exercise intensity, exercise part, and type of exercise such as strength exercise or aerobic exercise).

In operation S1820, the electronic device may acquire a reference heart rate change amount corresponding to the exercise type information. For example, the electronic device may search for a reference heart rate change amount related to exercise type information from the database. The reference heart rate change amount may mean information on the amount of change in heart rate measured when a user or another person performs an exercise corresponding to the exercise type information. The electronic device may compare the obtained reference heart rate change amount with the change amount of the heart rate value included in the generated biometric information data.

In operation S1830, the electronic device may determine recommended exercise intensity information based on the comparison result. For example, when the amount of change in the heart rate value is smaller than the reference amount of change in heart rate by a threshold value or more, the electronic device may determine information for increasing the exercise intensity as the recommended exercise intensity information. Conversely, when the amount of change in the heart rate value is larger than the reference amount of change in heart rate by a threshold value or more, the electronic device may determine information to reduce the exercise intensity or recommend rest as the recommended exercise intensity information. In operation S1840, the electronic device may output a notification for delivering recommended exercise intensity information to the user in a designated form (eg, video or sound).

According to an embodiment, the electronic device may determine recommended exercise intensity information by further considering information on the user's exercise state according to the user's movement in the amount of change in the heart rate value. For example, if the user's movement is not matched with the reference data specified in the electronic device's program (that is, the user's exercise movement is not appropriate), and the change in the heart rate value is less than the threshold value, the electronic device determines the exercise intensity. You can determine the recommended exercise intensity information that you do not want to change. When the user's movement matching the specified reference data in the program of the electronic device is detected above the specified threshold, and the increase in the user's heart rate value is less than the specified threshold, the electronic device is recommended to increase the user's exercise intensity. Strength information can be determined. Conversely, if the heart rate increases rapidly, but the detected user's movement does not match the reference data specified in the electronic device's program, the user's exercise ability cannot digest the set exercise intensity, so the electronic device reduces the exercise intensity. You can determine the recommended exercise intensity information.

FIG. 19 is a flowchart illustrating a process of providing recommended exercise information based on biometric information data by an electronic device (eg, the electronic device 101 of FIG. 1 and the electronic device 300 of FIG. 3) according to an embodiment ( 1900). In the operation of the electronic device disclosed in FIG. 19 and related description, the processor of the electronic device (eg, the processor 120 of FIG. 1, the processor 310 of FIG. 3) is a memory (eg, the memory 130 of FIG. 1, FIG. It can be understood that it is executed by executing instructions stored in the memory 320 of 3.

In operation S1910, the electronic device may acquire user information on the user. Here, the user information may mean information indicating an attribute of a user. For example, the user information may include information on an exercise performed by the user, information on at least one of sex, age, weight, and height.

In operation S1920, the electronic device may determine recommended exercise information based on user information and biometric information data. According to an embodiment, the electronic device may compare an average body signal of a user having the same or similar condition as the user information with biometric information data. For example, if the amount of change in heart rate of the biometric data data is smaller than the amount of change in average heart rate information corresponding to user information, the electronic device may determine that the user's exercise ability is high. The electronic device may output information on the determined exercise capability of the user and determine an exercise program according to the determined exercise capability value. For example, if the heart rate included in the measured biometric information data increases significantly while performing an exercise program for training the thigh, the electronic device determines that the lower body exercise ability is low and recommends an exercise program that can improve the lower body exercise ability Can be determined as exercise information. In operation S1930, the electronic device may output the determined recommended exercise information.

FIG. 20 is a diagram illustrating an electronic device (eg, the electronic device 300 of FIG. 3) according to an exemplary embodiment in units of modules.

An electronic device according to an embodiment includes a camera input unit 2010, an image preprocessor 2020, a biosignal processing unit 2030, a correction module 2040, a recalibration module 2050, and an output unit 2060. Can be.

The camera input unit 2010 may acquire an image by photographing a user's movement. The image preprocessor 2020 may recognize whether a user is captured in an image input through the camera input unit 2010. When an area in which the user is photographed is recognized in the image, the image preprocessor 2020 may separate an area necessary for measuring a biosignal from the image.

The biosignal processing unit 2030 may include an image segmentation unit 2031, a signal generation unit 2032, a signal amplification unit 2033, and a biosignal extraction unit 2034. The biosignal processing unit 2030 may extract a biosignal from an area separated by the image preprocessor 2020. The image segmentation unit 2031 may separate and classify an image channel required for extracting a bio-signal, and select a color channel having a high effective value. The signal preprocessor 2032 may exclude a color value unnecessary for extracting the bio-signal (eg, a color value in which the reflected light is photographed) and select a color channel necessary for extracting the bio-signal. The signal amplifying unit 2033 may amplify a value of the selected color channel. The biosignal extractor may remove noise from the amplified color channel and extract a heart rate value.

When the image preprocessor 2020 determines that the area where the user is photographed is not detected or the biosignal extracted by the biosignal processing unit 2030 is determined to be not a normal value, the correction module 2040 corrects the biosignal. can do. The correction module 2040 may correct the biosignal based on the data model correction unit 2041 or the pattern information correction unit 2042. The data model correction unit 2041 may correct the biosignal based on the data model modeled based on the feature points. The pattern information correction unit 2042 may correct the biosignal based on the data pattern stored in the pattern database 2043. According to an embodiment, the correction module 2040 may perform correction using the data model correction unit 2041 for a section in which the biosignal is not normally extracted for a short period (eg, 2 seconds). Conversely, the correction module 2040 may perform correction using the pattern information correction unit 2042 for a section in which the biosignal is not normally extracted for a long period (eg, 2 minutes).

After the biosignal has been corrected by the correction module 2040, when the measurement of the biosignal by the biosignal processing unit 2030 is resumed, the recalibration module 2050 performs the correction of the biosignal corrected by the correction module 2040. At least some can be recalibrated. The output unit 2060 may output information based on the measured or corrected biosignal.

According to an embodiment, an electronic device including a memory (eg, the memory 130 of FIG. 1, the memory 320 of FIG. 3) and a processor connected to the memory (eg, the processor 120 of FIG. 1) The electronic device 101 of FIG. 1, the electronic device 300 of FIG. 3, the electronic device 500 of FIG. 5, the server 610 of FIG. 6, and the electronic device 700 of FIG. 7 ). The processor is a camera (eg, the camera module 180 of FIG. 1, the camera 330 of FIG. 3, the camera 530 of FIG. 5, the camera device 630 of FIG. 6, the camera device 730 of FIG. 7) Acquire the image data captured by using, analyze the image data to calculate a heart rate value, determine whether the heart rate value is normally acquired, and if the heart rate value is not normally acquired, the heart rate value is normally acquired. It may be configured to determine a correction value for an unchanged section and to store biometric information data including the correction value in the memory.

According to an embodiment, the processor may extract a feature point based on heart rate data calculated before the section, generate modeled candidate data based on the feature point, and determine a correction value based on the candidate data.

According to an embodiment, the processor may compare the calculated heart rate value with candidate data, and determine the correction value based on the comparison result.

According to an embodiment, the processor uses the calculated heart rate value when the difference between the RR peak interval for the calculated heart rate value and the RR peak interval for the candidate data is within a specified range as the normally acquired heart rate value, and the image data is It can be determined by the biometric information data at the time of acquisition. The processor may include the candidate data as the correction value in the biometric information data when the difference between the R-R peak interval for the calculated heart rate value and the R-R peak interval for the candidate data is greater than a specified range.

According to an embodiment, the electronic device may further include a pattern database storing a plurality of heart rate data patterns. When the size of the section is less than or equal to the threshold value, the processor determines a correction value based on the candidate data, and when the size of the section is greater than the threshold value, the processor is based on the similarity between the heart rate data and the plurality of heart rate data patterns. The correction value may be determined based on the selected heart rate data pattern.

According to an embodiment, when the heart rate value is normally acquired and the biometric information data for a second time point prior to the first time point at which the heart rate value is acquired is a correction value, the biometric information data at the second time point is the heart rate value. The biometric information data can be corrected to have a linear relationship with.

According to an embodiment, the processor may determine the first region captured by the user in the image data by convolving images included in the image sequence of the image data.

According to an embodiment, the processor determines a second region including pixels within a specified range, among pixels included in the first region, and a heart rate value based on the pixel value included in the second region. Can be calculated.

According to an embodiment, the electronic device may further include a display (eg, the display device 160 of FIG. 1 and the display 340 of FIG. 3 ). The processor may be configured to control the display to display an image included in the image data and guide image content, and to determine the correction value based on a heart rate change amount associated with the guide image content.

According to an embodiment, the processor may control the display to output recommended exercise intensity information determined based on a comparison result of a change amount of a heart rate value included in the biometric information data and a reference heart rate change amount.

According to an embodiment, the processor may control the display to obtain user information related to image data and to output biometric information data and recommended exercise information determined based on the user information.

According to an embodiment, the processor may control the display to calculate the calorie consumption amount based on the biometric information data and output the calorie consumption amount.

An electronic device according to an embodiment (for example, the electronic device 101 of FIG. 1, the electronic device 300 of FIG. 3, the electronic device 500 of FIG. 5, the server 610 of FIG. 6, and the electronic device of FIG. 7) The method of operating (700)) is a camera (e.g., the camera module 180 of FIG. 1, the camera 330 of FIG. 3, the camera 530 of FIG. 5, the camera device 630 of FIG. 6, and 7 The operation of acquiring the image data captured using the camera device 730) of, the operation of calculating the heart rate value by analyzing the image data, the operation of determining whether the heart rate value is normally acquired, and the heart rate value not being normally acquired. In this case, an operation of determining a correction value for a section in which the heart rate value is not normally obtained and an operation of generating biometric information data including the correction value may be included.

According to an embodiment, an operation of extracting a feature point based on heart rate data calculated before the section and an operation of generating modeled candidate data based on the feature point are further included, and the operation of determining the correction value is performed on the candidate data. It may include an operation of determining a correction value based on it.

According to an embodiment, the determining of the correction value based on the candidate data may include comparing the calculated heart rate value with the candidate data, and determining the correction value based on the comparison result.

According to an embodiment, the comparing operation may include comparing an R-R peak interval for the calculated heart rate value and an R-R peak interval for the candidate data. The operation of determining the correction value is a comparison result, when the difference between the two values is within a specified range, the calculated heart rate value is determined as the biometric information data at the time the image data is acquired as a normally acquired heart rate value, and the difference between the two values is specified. If it is out of the range, an operation of including the candidate data as the correction value in the biometric information data may be included.

According to an embodiment, the operation of calculating a heart rate value by analyzing image data includes an operation of determining a first region photographed by a user in the image data by convolving images included in an image sequence of the image data. An operation of determining a second region including pixels whose pixel value is within a specified range among pixels included in the first region, and an operation of calculating the heart rate value based on the pixel value included in the second region. I can.

According to an embodiment, the method of operating the electronic device further includes an operation of displaying an image and guide image content included in the image data, and the operation of determining a correction value includes acquiring a heart rate change amount associated with the guide image content. It may include an operation and an operation of determining the correction value based on the amount of change in heart rate.

A computer-readable recording medium (eg, nonvolatile memory 134, memory 320) in which one or more programs including a plurality of instructions executed by at least one processor according to an embodiment are recorded, is executed. At the time, a camera (e.g., the camera module 180 of FIG. 1, the camera 330 of FIG. 3, the camera 530 of FIG. 5, the camera device 630 of FIG. 6, the camera device 730 of FIG. 7) Acquiring the image data captured by using, calculating the heart rate value by analyzing the image data, determining whether or not the heart rate value is normally acquired, and when the heart rate value is not acquired normally, the heart rate value is normally acquired. A program for determining a correction value for an unchanged section and an operation for generating biometric information data including the correction value may be recorded.

According to an embodiment, the electronic device may include a memory for storing heart rate data of an arrhythmia patient and a processor connected to the memory. The processor acquires biometric information data including a heart rate value, and if the biometric information data does not contain a heart rate value or there is a section other than the normally calculated value, the value of the section is corrected to a correction value, and the corrected When the biometric information data and the heart rate data of the arrhythmia patient are compared, and the number of times the corrected biometric information data includes the abnormal pattern included in the heart rate data of the arrhythmia patient is greater than or equal to the threshold value, information on the abnormal heart rate pattern Can be configured to output.

The methods according to the embodiments described in the claims or specification of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

When implemented in software, a computer-readable storage medium storing one or more programs (software modules) may be provided. One or more programs stored in a computer-readable storage medium are configured to be executable by one or more processors in an electronic device (device). The one or more programs include instructions that cause the electronic device to execute methods according to embodiments described in the claims or specification of the present disclosure.

These programs (software modules, software) include random access memory, non-volatile memory including flash memory, read only memory (ROM), and electrically erasable programmable ROM. (EEPROM: electrically erasable programmable read only memory), magnetic disc storage device, compact disc-ROM (CD-ROM), digital versatile discs (DVDs), or other types of It may be stored in an optical storage device or a magnetic cassette. Alternatively, it may be stored in a memory composed of a combination of some or all of them. In addition, a plurality of configuration memories may be included.

In addition, the program is a communication network such as the Internet, an intranet, a local area network (LAN), a wide LAN (WLAN), or a storage area network (SAN), or a communication network composed of a combination thereof. It may be stored in an accessible storage device. Such a storage device may access a device performing an embodiment of the present disclosure through an external port. In addition, a separate storage device on the communication network may access a device performing an embodiment of the present disclosure.

In the above-described specific embodiments of the present disclosure, components included in the disclosure are expressed in the singular or plural according to the presented specific embodiments. However, the singular or plural expression is selected appropriately for the situation presented for convenience of description, and the present disclosure is not limited to the singular or plural constituent elements, and even constituent elements expressed in plural are composed of singular or singular. Even the expressed constituent elements may be composed of pluralities.

Meanwhile, although specific embodiments have been described in the detailed description of the present disclosure, various modifications may be made without departing from the scope of the present disclosure. Therefore, the scope of the present disclosure is limited to the described embodiments and should not be determined, and should be determined by the scope of the claims and equivalents as well as the scope of the claims to be described later.

Claims (20)

In the electronic device,
Memory; And
Including a processor connected to the memory,
The processor,
Acquire image data taken using a camera,
Analyzing the image data to calculate a heart rate value,
It is determined whether the heart rate value is normally acquired,
When the heart rate value is not normally acquired, a correction value for a section in which the heart rate value is not normally acquired is determined,
The electronic device, configured to store biometric information data including the correction value in the memory. The method according to claim 1,
The processor,
Extracting feature points based on the heart rate data calculated before the section,
Generate modeled candidate data based on the feature points,
Determining the correction value based on the candidate data,
Electronic device. The method according to claim 2,
The processor,
Compare the calculated heart rate value and the candidate data,
Determining the correction value based on the comparison result. The method of claim 3,
The processor,
When the difference between the RR peak interval for the calculated heart rate value and the RR peak interval for the candidate data is within a specified range, the calculated heart rate value is used as the normally acquired heart rate value, and the living body at the time the image data is acquired. Determined by information data,
When a difference between an RR peak interval for the calculated heart rate value and an RR peak interval for the candidate data is greater than a specified range, the candidate data is included in the biometric information data as the correction value. The method according to claim 2,
The electronic device further includes a pattern database storing a plurality of heart rate data patterns,
The processor,
When the size of the section is less than or equal to the threshold value, the correction value is determined based on the candidate data,
When the size of the section is greater than a threshold value, determining the correction value based on a heart rate data pattern selected based on a similarity between the heart rate data and each of the plurality of heart rate data patterns. The method according to claim 1,
The processor,
When the heart rate value is normally obtained and the biometric information data for a second time point before the first time point at which the heart rate value is obtained is a correction value, the biometric information data at the second time point has a linear relationship with the heart rate value. The electronic device for correcting the biometric information data to have. The method according to claim 1,
The processor,
An electronic device for determining a first area in the image data in which a user is photographed by convolving images included in an image sequence of the image data. The method of claim 7,
The processor,
Determine a second region including pixels whose pixel value falls within a specified range among the pixels included in the first region, and
The electronic device that calculates the heart rate value based on a pixel value included in the second area. The method according to claim 1,
Further comprising a display,
The processor,
Controlling the display to display an image and guide image content included in the image data,
The electronic device, configured to determine the correction value based on an amount of change in heart rate associated with the guide image content. The method according to claim 1,
Further comprising a display,
The processor,
The electronic device comprising: controlling the display to output recommended exercise intensity information determined based on a comparison result of a change amount of a heart rate value included in the biometric information data and a reference heart rate change amount. The method according to claim 1,
Further comprising a display,
The processor,
Acquire user information related to the image data,
The electronic device, wherein the display is controlled to output recommended exercise information determined based on the biometric information data and the user information. The method according to claim 1,
Further comprising a display,
The processor,
Calorie consumption is calculated based on the biometric information data,
Controlling the display to output the calorie consumption amount. In a method of operating an electronic device,
Obtaining image data photographed using a camera;
Analyzing the image data to calculate a heart rate value;
Determining whether the heart rate value is normally acquired;
When the heart rate value is not normally acquired, determining a correction value for a section in which the heart rate value is not normally acquired; And
Generating biometric information data including the correction value. The method of claim 13,
Extracting feature points based on heart rate data calculated before the section; And
Further comprising an operation of generating modeled candidate data based on the feature points,
Determining the correction value comprises determining the correction value based on the candidate data. The method of claim 14,
The operation of determining the correction value based on the candidate data,
Comparing the calculated heart rate value with the candidate data, and
Determining the correction value based on the comparison result. The method of claim 15,
The comparing operation includes comparing an RR peak interval for the calculated heart rate value and an RR peak interval for the candidate data,
The operation of determining the correction value,
As a result of the comparison, when the difference between the two values is within a specified range, the calculated heart rate value is determined as the biometric information data at the time the image data is acquired as a normally acquired heart rate value, and the difference between the two values is outside the specified range Including the candidate data in the biometric information data as the correction value. The method of claim 11,
The operation of analyzing the image data to calculate a heart rate value,
Determining a first region captured by a user in the image data by convolving images included in the image sequence of the image data,
Determining a second region including pixels whose pixel value is within a specified range among the pixels included in the first region, and
And calculating the heart rate value based on the pixel value included in the second area. The method of claim 11,
Further comprising an operation of displaying the image and guide image content included in the image data,
The operation of determining the correction value,
An operation of acquiring an amount of change in heart rate associated with the guide image content, and
And determining the correction value based on the amount of change in heart rate. In a computer-readable recording medium in which one or more programs including a plurality of instructions executed by at least one processor are recorded,
When the program is executed, the operation of acquiring image data photographed using a camera;
Analyzing the image data to calculate a heart rate value;
Determining whether the heart rate value is normally acquired;
When the heart rate value is not normally acquired, determining a correction value for a section in which the heart rate value is not normally acquired; And
To perform an operation of generating biometric information data including the correction value. In the electronic device,
A memory for storing heart rate data of a patient with arrhythmia; And
Including a processor connected to the memory,
The processor,
Acquiring biometric information data including a heart rate value,
When the biometric information data does not contain a heart rate value or there is a section other than the normally calculated value, the value of the section is corrected with a correction value,
Compare the corrected biometric information data with the heart rate data of the arrhythmia patient,
The electronic device configured to output information on the abnormal heart rate pattern based on the comparison result, when the number of times the corrected biometric information data includes the abnormal pattern included in the arrhythmia patient heart rate data is equal to or greater than a threshold value.

Images