KR102593722B1 - Deep learning-based In-person Customized Metabolic Rate Prediction System and Prediction Method - Google Patents

Abstract

The present invention is a deep learning-based occupant-customized metabolic rate prediction system performed by a computer, and includes a metabolic rate acquisition unit (100) that receives the resident's real-time heart rate measured by a wearable device (10) capable of measuring heart rate and calculates the resident's real-time metabolic rate. ); An image acquisition unit 200 that receives real-time activity images of occupants acquired through the camera 20; a data pre-processing unit 300 that pre-processes the image of the image acquisition unit 200; A deep learning learning unit 400 that generates a real-time metabolic rate calculation model of the occupant through deep learning using the real-time metabolic rate of the metabolic rate acquisition unit 100 and the pre-processed activity image as input values; and a deep learning prediction unit 500 that predicts the real-time metabolic rate of the occupant by inputting the image of the occupant's activity acquired in real time from the image acquisition unit 200 into the real-time metabolic rate calculation model.

Description

Deep learning-based In-person Customized Metabolic Rate Prediction System and Prediction Method}

The present invention relates to a metabolic prediction system and prediction room. Specifically, it concerns a deep learning-based metabolic rate prediction system and prediction method tailored to occupants.

Metabolism refers to the amount of heat generated by the occupant’s activities. Metabolic rate is one of the factors used to evaluate the thermal comfort of occupants.

In general, metabolic rate is difficult to predict due to the unexpected activities of occupants, so in the construction field, metabolic rate is assumed to be a fixed value when controlling air conditioning and heating. Therefore, there was a problem that the metabolic amount according to the actual activities of the occupants was not accurately reflected.

To improve this problem, Korean Patent Publication No. 10-2233157 proposed a method for calculating occupant activity using deep learning-based occupant pose classification. In this prior art, there was a difference in that it was an attempt to calculate the activity amount (metabolic amount) according to the actual activity of the occupant.

However, in the case of Korean Patent Publication No. 10-2233157, the classification model is used to specifically classify the occupant's activities and calculate the metabolic value set for the activity. Therefore, deep learning here is merely used to classify indoor activity poses from occupant images, and after classifying the occupants' poses, a specific activity amount (metabolism) is applied as a fixed value to a specific pose. Therefore, fixing the real-time behavioral changes of occupants to a specific value still has the problem of not being able to predict natural changes in dialogue due to actual behavioral changes of occupants.

(Document 1) Korean Patent Publication No. 10-2233157 (2021.03.23)

The deep learning-based occupant-customized metabolic rate prediction system and prediction method according to the present invention has the following problems to solve.

First, we want to calculate real-time metabolic rate using the resident's real-time heart rate.

Second, we want to calculate the real-time metabolism according to the real-time activity images of the occupants by deep learning the real-time activity images and real-time metabolism of the occupants.

Third, through data preprocessing that removes the background excluding the occupants, we want to focus on analyzing the activity images of the occupants.

Fourth, we aim to predict real-time metabolic rate by inputting only real-time activity images without heart rate information using a real-time activity change metabolic rate calculation model derived through deep learning learning.

The problems to be solved by the present invention are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

The present invention is a deep learning-based occupant-customized metabolic rate prediction system performed by a computer, comprising: a metabolic rate acquisition unit that receives the resident's real-time heart rate measured by a wearable device capable of measuring heart rate and calculates the resident's real-time metabolic rate; An image acquisition unit that receives real-time activity images of occupants acquired through a camera; a data pre-processing unit that pre-processes the image of the image acquisition unit; A deep learning learning unit that generates a real-time metabolic rate calculation model of the occupant through deep learning using the real-time metabolic rate of the metabolic rate acquisition unit and the pre-processed activity image as input values; and a deep learning prediction unit that predicts the real-time metabolism of the occupant by inputting the image of the occupant's activity acquired in real time from the image acquisition unit into the real-time metabolic mass calculation model. .

In the present invention, the wearable device includes a watch-type device, a band-type device, a glasses-type device, a necklace-type device, a helmet-type device, and a clothing-type device, and the camera may include a Kinect camera.

In the present invention, the metabolic rate acquisition unit can calculate the real-time metabolic rate of the occupant using the transmitted real-time heart rate information of the occupant and the previously entered gender, weight, and age information of the occupant.

In the present invention, the data pre-processing unit may delete the portion excluding the occupants from the activity image acquired by the image acquisition unit.

In the present invention, the camera uses a Kinect camera, acquires an RGB image and a depth image displaying perspective through the Kinect camera, detects occupants from the input image based on the acquired depth image, and , the part excluding the occupants can be removed by masking the RGB image.

In the present invention, the real-time metabolic rate calculation model of the deep learning learning unit is

ReLU is adopted as the activation function in Equation 1, batch normalization and SGD optimizer are adopted, and the initial target function of the real-time metabolic rate calculation model may have Equation 2.

In the present invention, the real-time metabolic rate calculation model may have a final target function of Equation 4 by adding the normalization function of Equation 3 below.

The present invention is a deep learning-based occupant-customized metabolic rate prediction method performed by a computer, comprising the steps of S100, where a metabolic rate acquisition unit receives the resident's real-time heart rate measured by a wearable device capable of measuring heart rate, and calculates the resident's real-time metabolic rate; Step S200, where the image acquisition unit receives the real-time activity image of the occupant acquired through the camera; Step S300 in which a data pre-processing unit pre-processes the image of the image acquisition unit; Step S400 in which the deep learning learning unit generates a real-time metabolic amount calculation model of the occupant through deep learning learning using the real-time metabolic amount of the metabolic amount acquisition unit and the pre-processed activity image as input values; And it may include a step S500 in which the deep learning prediction unit predicts the real-time metabolic rate of the occupant by inputting the image of the occupant's activity acquired in real time from the image acquisition unit into the real-time metabolic rate calculation model.

In step S100 according to the present invention, the metabolic rate acquisition unit may calculate the real-time metabolic rate of the occupant using the transmitted real-time heart rate information of the occupant and the previously entered gender, weight, and age information of the occupant.

In step S300 according to the present invention, the data pre-processing unit may delete the portion excluding the occupants from the activity image acquired by the image acquisition unit.

In the present invention, the camera uses a Kinect camera, obtains an RGB image and a depth image that displays perspective through the Kinect camera, and detects the occupant from the image input based on the obtained depth image. can be detected and masked to the RGB image to remove the part excluding the occupants.

The present invention can be combined with hardware and implemented as a computer program stored in a computer-readable recording medium to execute the deep learning-based occupant-customized metabolic rate prediction method according to claim 8 by a computer.

The deep learning-based occupant-customized metabolic rate prediction system and prediction method according to the present invention has the following effects.

First, it has the effect of calculating real-time metabolic rate rather than fixed metabolic rate by using the occupant's real-time heart rate.

Second, it has the effect of calculating the real-time metabolism according to the real-time activity image by deep learning the real-time activity image and real-time metabolism of the occupant.

Third, through data preprocessing that deletes the background excluding the occupants, there is an effect of focusing and analyzing the activity images of the occupants.

Fourth, the real-time activity change metabolic rate calculation model derived through deep learning is effective in predicting real-time metabolic rate by inputting only real-time activity images without heart rate information.

The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

Figure 1 shows an example of a deep learning-based occupant-customized metabolic rate prediction system according to the present invention.
Figure 2 is a schematic diagram of the deep learning learning process and deep learning prediction process.
Figure 3 shows a flowchart of the deep learning-based occupant-customized metabolic rate prediction method according to the present invention.
Figure 4 shows an example of data preprocessing according to the present invention.
Figures 5A to 5C show an example of removing the background excluding the occupants in the data pre-processing unit. Figure 5A shows the input image, Figure 5B shows the occupant image, and Figure 5C shows the background removal image.
6A to 6H show an example of various actual activities of occupants.
Figure 7 is an example showing the actual metabolic value (ground truth) and artificial intelligence predicted value (Predicted value) according to various changes in the occupants' activities.
Figure 8 shows an example comparing the image of Figure 5 before and after background exclusion.
Figure 9 shows an example of a residual block of an artificial neural network.
Figure 10 shows one embodiment of the METNet model structure.

Hereinafter, with reference to the attached drawings, embodiments of the present invention will be described so that those skilled in the art can easily implement the present invention. As can be easily understood by those skilled in the art to which the present invention pertains, the embodiments described below may be modified into various forms without departing from the concept and scope of the present invention. As much as possible, identical or similar parts are indicated using the same reference numerals in the drawings.

The terminology used herein is only intended to refer to specific embodiments and is not intended to limit the invention. As used herein, singular forms include plural forms unless phrases clearly indicate the contrary.

As used herein, the meaning of "comprising" is to specify a specific characteristic, area, integer, step, operation, element, and/or component, and to specify other specific characteristic, area, integer, step, operation, element, component, and/or This does not exclude the presence or addition of the military.

All terms, including technical and scientific terms, used in this specification have the same meaning as those generally understood by those skilled in the art in the technical field to which the present invention pertains. Terms defined in the dictionary are further interpreted as having meanings consistent with the related technical literature and currently disclosed content, and are not interpreted in ideal or very formal meanings unless defined.

Expressions related to direction used in this specification, for example, front/back/left/right, top/bottom, and vertical/lateral directions, can be interpreted with reference to the directions shown in the drawings.

In the case of the present invention, it is a customized model that selects the metabolic rate value using the heart rate of the occupant, and presents a value that is closer to the metabolic rate of the actual occupant.

Occupants perform various activities indoors, and artificial intelligence deep learning technology is used to learn, predict metabolic value for each action based on the learned content, and finally, real-time customized metabolic rate prediction is possible.

The present invention proposes a metabolic rate detection technology tailored to occupants by utilizing deep learning computer vision technology and a wearable device capable of measuring heart rate.

Using the present invention, it is possible to derive a metabolic value customized to the occupant rather than a generalized tabular average metabolic value, which can help create a better thermal environment.

The Kinect camera that can be used in the present invention has built-in technology to exclude the background except for the occupants, thereby ensuring high precision without being affected by changes in the occupants' space (surrounding environment).

Figure 2 is a schematic diagram of the deep learning learning process and deep learning prediction process. Figure 3 shows a flowchart of the deep learning-based occupant-customized metabolic rate prediction method according to the present invention.

The entire process is as follows. Metabolism (MET) is calculated in real time through a wearable device, and the corresponding images are simultaneously captured through a camera. Deep learning is performed on captured images and metabolic rate values according to the images. Afterwards, customized metabolic rate can be calculated in real time according to the occupant's activities without the need for a wearable device.

Hereinafter, the present invention will be described with reference to the drawings. For reference, the drawings may be somewhat exaggerated in order to explain the features of the present invention. In this case, it is preferable to interpret in light of the entire purpose of this specification.

Figure 1 shows an example of a deep learning-based occupant-customized metabolic rate prediction system according to the present invention.

As shown in Figure 1, the deep learning-based occupant-customized metabolic rate prediction system according to the present invention includes a metabolic rate acquisition unit 100, an image acquisition unit 200, a data preprocessor 300, a deep learning learning unit 400, and Includes a deep learning prediction unit 500.

The present invention is a deep learning-based occupant-customized metabolic rate prediction system performed by a computer, and includes a metabolic rate acquisition unit (100) that receives the resident's real-time heart rate measured by a wearable device (10) capable of measuring heart rate and calculates the resident's real-time metabolic rate. ); An image acquisition unit 200 that receives real-time activity images of occupants acquired through the camera 20; a data pre-processing unit 300 that pre-processes the image of the image acquisition unit 200; A deep learning learning unit 400 that generates a real-time metabolic rate calculation model of the occupant through deep learning using the real-time metabolic rate of the metabolic rate acquisition unit 100 and the pre-processed activity image as input values; and a deep learning prediction unit 500 that predicts the real-time metabolic rate of the occupant by inputting the image of the occupant's activity acquired in real time from the image acquisition unit 200 into the real-time metabolic rate calculation model.

Wearable devices 10 according to the present invention include watch-type devices, band-type devices, glasses-type devices, necklace-type devices, helmet-type devices, and clothing-type devices.

The metabolic rate acquisition unit 100 according to the present invention receives the resident's real-time heart rate measured by the wearable device 10 capable of measuring heart rate, and calculates the resident's real-time metabolic rate.

The metabolic rate acquisition unit 100 according to the present invention can calculate the real-time metabolic rate of the occupant using the transmitted real-time heart rate information of the occupant and the previously entered gender, weight, and age information of the occupant.

'Metabolic rate' according to the present invention refers to the total metabolic rate including basal metabolic rate and active metabolic rate.

In the present invention, metabolic rate conversion using heart rate follows the metabolic rate evaluation method Level 3 standards of the international standard organization ISO 8996.

The metabolic rate evaluation formula is as follows.

here, is the metabolic rate value. is the metabolic rate at rest, and the unit is W/ am. is the heart rate. is the heart rate in thermal neutral state at rest. RM is the number representing the increase in heart rate.

The RM calculation formula is as follows.

Here, MWC refers to maximum working capacity. At this time, gender, age, and weight are calculated. A is age. P is body weight.

When the metabolic value is calculated in W/㎡, 1 MET is converted proportionally to 58.2 W/㎡ according to ANSI/ASHRAE Standard 55.

The image acquisition unit 200 according to the present invention can receive real-time activity images of occupants acquired through the camera 20.

The camera 20 according to the present invention includes a Kinect camera. The Kinect sensor is a low-cost depth camera that can provide not only depth information but also RGB images and joint tracking information in real time.

The data preprocessing unit 300 according to the present invention can preprocess the image of the image acquisition unit 200.

Figure 4 shows an example of data preprocessing according to the present invention.

Data preprocessing is an operation to minimize the impact of previous activities on the MET of the next activity. For the example of Figure 4, data for the first 1 minute and the last 2 minutes per activity were removed.

Figures 5A to 5C show an example of removing the background excluding the occupants in the data pre-processing unit. Figure 5A shows the input image, Figure 5B shows the occupant image, and Figure 5C shows the background removal image.

Figure 8 shows an example comparing the image of Figure 5 before and after background exclusion.

The data pre-processing unit 300 according to the present invention can delete the portion excluding the occupants from the activity image acquired by the image acquisition unit 200.

The present invention uses visual attention to remove the background part of the image, which causes deterioration in action recognition performance. The performance of action recognition technology can be improved by using images with background removed using visual attention in the deep learning model developed in this study.

In the present invention, Microsoft's Kinect camera was used as an example in order to remove background parts containing unnecessary information for recognizing occupant behavior. As shown in Figure 5, an RGB image and a depth image displaying perspective are acquired through a Kinect camera, occupants are detected in the input image based on the acquired depth image, and the portion excluding the occupant is masked to the RGB image. can be removed.

For reference, in this specification and drawings, the term 'background' of the image refers to the part of the image excluding the 'occupant'.

The Kinect camera used in the present invention can secure high precision without being affected by changes in the space (surrounding environment) of the occupants by applying technology to exclude the background except the occupants.

As a learning step of the deep learning learning unit 400 according to the present invention, the occupants perform various activities, and the camera acquires this as image data. In addition, the real-time metabolic rate (MET) is calculated by calculating the occupant's real-time heart rate data obtained through the wearable device and the occupant's gender, weight, and age. Afterwards, the image and the amount of metabolism are used as input values, and the amount of metabolism is learned from the image using deep learning technology.

In the case of the prediction step of the deep learning prediction unit 500 according to the present invention, it is possible to calculate the occupant's customized metabolic rate using only real-time images obtained through a camera, even without a wearable device.

Figure 2 is a schematic diagram of the deep learning learning process and deep learning prediction process.

The MET prediction method of the present invention is shown in Figure 2. First, in the sensing stage, various life activities of the occupants are observed using the KikiKinect camera and at the same time, information about the occupants is obtained from a wearable device. In AI (artificial intelligence), deep learning is used to learn image information and metabolic information about behavior. Based on AI's accumulated database, AI makes it possible to predict metabolic equivalent task (MET) using only the Kinect camera, a non-contact sensor, without the help of wearable devices.

Figure 3 shows a flowchart of the deep learning-based occupant-customized metabolic rate prediction method according to the present invention.

The specific MET prediction method is shown in Figure 3. First, the Kinect camera is used to sense the occupants' behavior and obtain 5 images per second. At the same time, heart rate data is collected through a wearable device. Using the collected heart rate data and the experimenter's unchanging information such as gender, weight, and age as input data, the MET value is calculated according to the international metabolic standard ISO 8996 Level 3 method. The MET value calculated in this way and the image file obtained through the Kinect camera are learned (deep learning) in the artificial intelligence system (AI System). Ultimately, a person's MET can be predicted using only the Kinect camera, a non-contact sensor, without information from a wearable device.

The deep learning learning unit 400 according to the present invention can generate a real-time metabolic amount calculation model of the occupant through deep learning using the real-time metabolic amount of the metabolic amount acquisition unit 100 and the pre-processed activity image as input values.

Figure 9 shows an example of a residual block of an artificial neural network. Figure 10 shows one embodiment of the METNet model structure.

To improve the stability of the metabolic calculation model, Rectified Linear Unit (ReLU) was adopted as the activation function, and batch normalization and Stochastic Gradient Descent (SGD) optimizers were adopted.

Because ReLU has nonlinearity, it can alleviate gradient vanishing and gradient exploding problems. Additionally, by using batch normalization, the covariance shift problem within the artificial neural network is reduced as proven in batch normalization. Finally, the SGD optimizer was adopted.

This is very effective in learning models reliably and quickly. The initial objective function of the entire model is L1 Loss (Equation 2)ld, and the SGD optimizer can solve the optimization problem of reducing the objective function to the minimum.

Here, f(x) is the activation function and x is the input value.

Here, L1Loss is the initial target function, x is the input value, n is the number of input values (batchsize), y is the label, is the ith input value among n input values x.

However, the metabolic calculation model still has a bias in making predictions that are most similar to the data used for learning. The output model is trained to minimize the average absolute error by predicting the median value of the learning data using the objective function, which may cause results different from expectations. Therefore, the normalization function of Equation 3 is added. This function penalizes models that simultaneously predict similar values during the learning process.

That is, the real-time metabolic rate calculation model according to the present invention can have the final target function of Equation 4 below by adding the normalization function of Equation 3 below.

Here, REG is the normalization function, n is the number of input values (batchsize), is the ith input value among n input values x, is the n average of x.

Therefore, the final goal function according to the present invention is expressed as Equation 4 below.

Here, O is the final goal function, L1Loss is the initial goal function, and REG is the regularization function.

In the case of METNet according to the present invention, all CNN layers have ReLU as an activation function. The first CNN layer of each residual block sets the filter movement interval to 2 pixels for downsampling. Then, batch normalization is applied. In the present invention, an SGD optimizer with a learning rate of 5e-5, momentum of 0.9, and variable reduction rate of 5e-4 is used.

Meanwhile, the present invention may also be implemented as a metabolic rate prediction method. The prediction method invention has the same substantive technical composition as the above-mentioned prediction system invention, but the invention category is different. The overlapping details can be borrowed from the prediction system invention, and the following will mainly explain the technical composition.

The present invention is a deep learning-based occupant-customized metabolic rate prediction method performed by a computer, wherein the metabolic rate acquisition unit 100 receives the resident's real-time heart rate measured by a wearable device 10 capable of measuring heart rate, and calculates the resident's real-time metabolic rate. S100 step of calculating; Step S200, where the image acquisition unit 200 receives the real-time activity image of the occupant acquired through the camera 20; Step S300 in which the data pre-processing unit 300 pre-processes the image of the image acquisition unit 200; Step S400 in which the deep learning learning unit 400 generates a real-time metabolic amount calculation model of the occupant through deep learning using the real-time metabolic amount of the metabolic amount acquisition unit 100 and the pre-processed activity image as input values; And a step S500 in which the deep learning prediction unit 500 predicts the real-time metabolic rate of the occupant by inputting the image of the occupant's activity acquired in real time from the image acquisition unit 200 into the real-time metabolic rate calculation model.

Since steps S100 and S200 acquire real-time information, it is preferable that they are performed simultaneously.

In step S100, the metabolic rate acquisition unit 100 may calculate the real-time metabolic rate of the occupant using the transmitted real-time heart rate information of the occupant and the previously entered gender, weight, and age information of the occupant.

In step 300, the data pre-processing unit 300 may delete the portion excluding the occupants from the activity image acquired by the image acquisition unit 200.

The camera uses a Kinect camera, obtains an RGB image and a depth image that displays perspective through the Kinect camera, detects occupants from the input image based on the obtained depth image, and RGB By masking the video, you can remove parts excluding the occupants.

The real-time metabolic rate calculation model of the deep learning learning unit 400 adopts ReLU as the activation function of Equation 1, batch normalization and SGD optimizer, and the initial target function of the real-time metabolic rate calculation model is mathematical We can have equation 2.

The real-time metabolic rate calculation model can have a final target function of Equation 4 below by adding the normalization function of Equation 3 below.

Meanwhile, the present invention may be implemented as a computer program.

Specifically, the present invention can be combined with hardware and implemented as a computer program stored in a computer-readable recording medium to execute the deep learning-based occupant-customized metabolic rate prediction method according to the present invention by a computer.

Methods according to embodiments of the present invention may be implemented in the form of programs readable through various computer means and recorded on a computer-readable recording medium. Here, the recording medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the recording medium may be those specifically designed and constructed for the present invention, or may be known and available to those skilled in the art of computer software. For example, recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CDROMs and DVDs, and magneto-optical media such as floptical disks. optical media), and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions may include machine language, such as that created by a compiler, as well as high-level languages that can be executed by a computer using an interpreter, etc. These hardware devices may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

The embodiments described in this specification and the accompanying drawings merely illustratively illustrate some of the technical ideas included in the present invention. Accordingly, the embodiments disclosed in this specification are not intended to limit the technical idea of the present invention, but rather to explain it, and therefore, it is obvious that the scope of the technical idea of the present invention is not limited by these embodiments. All modifications and specific embodiments that can be easily inferred by a person skilled in the art within the scope of the technical idea included in the specification and drawings of the present invention should be construed as being included in the scope of the present invention.

10: Wearable device
20: Camera
100: Metabolism acquisition unit
200: Image acquisition unit
300: Data preprocessing unit
400: Deep learning learning department
500: Deep learning prediction unit

Claims (14)

A deep learning-based occupant-customized metabolic rate prediction system performed by a computer, comprising: a metabolic rate acquisition unit that receives the resident's real-time heart rate measured by a wearable device capable of measuring heart rate and calculates the resident's real-time metabolic rate; An image acquisition unit that receives real-time activity images of occupants acquired through a camera; a data pre-processing unit that pre-processes the image of the image acquisition unit; A deep learning learning unit that generates a real-time metabolic rate calculation model of the occupant through deep learning using the real-time metabolic rate of the metabolic rate acquisition unit and the pre-processed activity image as input values; And a deep learning prediction unit that predicts the real-time metabolic rate of the occupant by inputting the image of the occupant's activity acquired in real time from the image acquisition unit into the real-time metabolic rate calculation model,
The real-time metabolic calculation model of the deep learning unit adopts ReLU as the activation function of Equation 1, batch normalization and SGD optimizer, and the initial target function of the real-time metabolic calculation model is Equation 2 below. have,
The real-time metabolic rate calculation model is a deep learning-based, occupant-customized metabolic rate prediction system, characterized in that it has a final target function of Equation 4 below by adding the normalization function of Equation 3 below.
[Equation 1]
(Here, f(x) is the activation function, and x is the input value.)

[Equation 2]
(Here, L1Loss is the initial target function, x is the input value, n is the number of input values (batchsize), y is the label, is the ith input value among n input values x.)
[Equation 3]
(Here, REG is the normalization function, n is the number of input values (batchsize), is the ith input value among n input values x, is the n average of x.)
[Equation 4]
(Here, O is the final goal function, L1Loss is the initial goal function, and REG is the regularization function.) In claim 1,
The wearable devices include watch-type devices, band-type devices, glasses-type devices, necklace-type devices, helmet-type devices, and clothing-type devices,
The camera is a deep learning-based occupant-customized metabolic rate prediction system, characterized in that it includes a Kinect camera. In claim 1,
The metabolic rate acquisition unit calculates the real-time metabolic rate of the resident using the transmitted real-time heart rate information of the resident and the previously entered gender, weight, and age information of the resident. A deep learning-based personalized metabolic rate prediction system. The method of claim 1, wherein the data preprocessing unit
A deep learning-based occupant-customized metabolic rate prediction system characterized in that the portion excluding the occupant is deleted from the activity image acquired by the image acquisition unit. In claim 4,
The camera uses a Kinect camera,
It acquires an RGB image and a depth image that displays perspective through a Kinect camera, detects occupants in the input image based on the acquired depth image, and removes the part excluding the occupant by masking the RGB image. A deep learning-based metabolic rate prediction system tailored to occupants. delete delete A deep learning-based occupant-customized metabolic rate prediction method performed by a computer, comprising: S100 step in which a metabolic rate acquisition unit receives the resident's real-time heart rate measured by a wearable device capable of measuring heart rate and calculates the resident's real-time metabolic rate; Step S200, where the image acquisition unit receives the real-time activity image of the occupant acquired through the camera; Step S300 in which a data pre-processing unit pre-processes the image of the image acquisition unit; Step S400 in which the deep learning learning unit generates a real-time metabolic amount calculation model of the occupant through deep learning learning using the real-time metabolic amount of the metabolic amount acquisition unit and the pre-processed activity image as input values; And a step S500 in which the deep learning prediction unit inputs the resident's activity image acquired in real time from the image acquisition unit into the real-time metabolic rate calculation model to predict the resident's real-time metabolic rate,
The real-time metabolic calculation model of the deep learning unit adopts ReLU as the activation function of Equation 1, batch normalization and SGD optimizer, and the initial target function of the real-time metabolic calculation model is Equation 2 below: ,
The real-time metabolic rate calculation model is a deep learning-based user-customized metabolic rate prediction method characterized by adding a normalization function in Equation 3 below and having a final target function in Equation 4 below.
[Equation 1]
(Here, f(x) is the activation function, and x is the input value.)

[Equation 2]
(Here, L1Loss is the initial target function, x is the input value, n is the number of input values (batchsize), y is the label, is the ith input value among n input values x.)
[Equation 3]
(Here, REG is the normalization function, n is the number of input values (batchsize), is the ith input value among n input values x, is the n average of x.)
[Equation 4]
(Here, O is the final goal function, L1Loss is the initial goal function, and REG is the regularization function.) In claim 8,
In step S100, the metabolic rate acquisition unit calculates the real-time metabolic rate of the resident using the transmitted real-time heart rate information of the resident and the previously input gender, weight, and age information of the resident. A deep learning-based, personalized metabolic rate prediction method. . In claim 8,
In step S300, the data pre-processing unit deletes the portion excluding the occupant from the activity image acquired by the image acquisition unit. A deep learning-based occupant-customized metabolic rate prediction method. In claim 10,
The camera uses a Kinect camera,
Through the Kinect camera, an RGB image and a depth image that displays perspective are acquired, occupants are detected in the input image based on the obtained depth image, and the part excluding the occupant is removed by masking the RGB image. A deep learning-based user-customized metabolic rate prediction method characterized by: delete delete A computer program combined with hardware and stored on a computer-readable recording medium to execute the deep learning-based occupant-customized metabolic rate prediction method according to claim 8 by a computer.