KR102611458B1 - Method for Providing Augmented Reality by Baby's Emotional Sate using Baby's Peripheral Region - Google Patents

Abstract

The present invention relates to a method for enhancing a baby's emotional state using the area around the baby, and is performed through an operation server equipped with a sound input unit and a camera module and communicating with a camera device installed at a designated base, comprising: Among the F(F≥2) feature points present in the baby, E(E≥2) baby expression pattern information corresponding to the geometric relationship between feature points associated with the baby's emotional expression, and among the designated N(N≥2) baby emotion items, the baby Matching relationship information between N1 (1≤N1≤N) baby emotion items that can be distinguished based on facial expression patterns, and facial expression-based correlation numerical relationship that quantifies the correlation between the E baby facial expression pattern information and N1 baby emotional items Based on facial expression-based emotional state determination information containing information, C (C≥2) baby cry pattern information related to the baby's emotional expression among the baby's cry patterns, and baby cry pattern among the designated N baby emotion items. Matching relationship information between N2 (1≤N2≤N) baby emotion items that can be distinguished by crying sound-based correlation numerical relationship information that quantifies the correlation between the C baby cry pattern information and N2 baby emotion items. S (S ≥ 2) sensing data of a designated sensing target related to the baby's emotional expression among sensing data sensed through a sensor module provided or connected to the camera device and emotional state determination information based on crying sounds, including Matching relationship information between value pattern information and N3 (1≤N3≤N) baby emotion items that can be distinguished based on the sensing value pattern of a designated sensing object among the specified N baby emotion items, the S sensing value pattern information, and N3 Register and store sensing-based emotional state determination information, including numerical relationship information, in the operating D/B, and augment the area around the baby's face by corresponding it to the baby's emotional state. Register and store R (R≥2) virtual object data in the operation D/B, and register virtual object matching information that matches the baby emotional state information for each designated emotion item with one or more designated virtual object data in the operation D/B. stores, receives video data encoded in a designated manner by filming a baby in the base through a camera module of the camera device, and audio data encoded in a designated manner input through a sound input unit of the camera device, and receives the received video Synchronizes the time of data and audio data, stores and manages them in conjunction with the storage area of the designated storage medium, receives sensing data sensed through a sensor module provided or linked to the camera device, and links the video data to the designated storage medium. Store and manage in the storage area, read the designated video data, and recognize f (1≤f≤F) feature points of the baby's facial area within the base. Among the recognized f feature points, feature points associated with the baby's emotional expression Recognize i (i ≥ 1) baby facial expression recognition information corresponding to geometric relationships, and read audio data of a certain section linked to the video data to provide j (j ≥ 1) baby cry recognition information for the baby. Recognize, read the sensing data associated with the video data, recognize k (k ≥ 1) sensing value recognition information for the sensing data of the designated sensing target associated with the emotional expression of the baby, and determine the facial expression-based emotional state. Based on the discrimination information, identify the e (1≤e≤E) baby expression pattern information that matches the recognized i baby facial expression recognition information and a preset standard similarity level or higher, and identify the baby facial expression pattern information that has a matching relationship with the e baby facial expression pattern information. n1 (1≤n1≤N1) baby emotion items are determined, and c (1≤c) is matched with the recognized j baby cry recognition information and a preset standard similarity level or higher based on the crying sound-based emotional state determination information. Confirm ≤C) baby cry pattern information, determine n2 (1≤n2≤N2) baby emotion items that have a matching relationship with the confirmed c baby cry pattern information, and provide the sensing-based emotional state determination information. Based on this, check s (1≤s≤S) sensing value pattern information that matches the recognized k sensing value recognition information and a preset standard similarity or higher, and have a matching relationship with the confirmed s sensing value pattern information. n3 (1≤n3≤N3) baby emotion items are determined, and based on the matching ratio between the recognized i baby expression recognition information and the e baby expression pattern information, the e baby expression pattern information and the n1 babies The correlation and numerical relationships between emotional items are read to calculate the baby emotional state value for each n1 emotional item, which quantifies the baby emotional state for each n1 emotional item corresponding to the recognized i baby facial expression recognition information, and the recognized j Based on the matching ratio between the c baby cry recognition information and the c baby cry pattern information, the correlation and numerical relationship between the c baby cry pattern information and the n2 baby emotion items are read, and the recognized j baby cries are read. Calculate the baby's emotional state value for each n2 emotional items that quantifies the baby's emotional state for each n2 emotional items corresponding to the sound recognition information, and match the ratio between the recognized k sensing value recognition information and the s sensing value pattern information. Based on this, the correlation and numerical relationship between the s sensing value pattern information and the n3 baby emotional items are read, and the n3 emotions are quantified for each of the n3 emotional items corresponding to the recognized k sensing value recognition information. Calculate baby emotional state values for each item, and use the calculated baby emotional state values for n1 emotional items, baby emotional state values for n2 emotional items, and baby emotional state values for n3 emotional items to specify n (1≤ Generate baby emotional state information for n≤N) emotional items, and r (1≤r≤R) virtual objects that match the baby emotional state information for each n emotional items generated based on the virtual object matching information. q(1≤q≤r) virtual object data that checks the data and displays a string of sentences expressing the baby's emotional state corresponding to the baby's emotional state information for each of the n emotional items in the form of a speech bubble in the area around the baby's face is displayed overlaid on the area around the baby's face on the video data, and the area around the baby's face is augmented to correspond to the baby's emotional state information for each of the n emotion items, and the area around the baby's face is augmented to correspond to the baby's emotional state information for each of the n emotional items. Transmission data including video data is provided to an app or IPTV on the user's wireless terminal.

Description

Method for Providing Augmented Reality by Baby's Emotional Sate using Baby's Peripheral Region}

The present invention provides one or more data acquired through the camera device when transmitting data including video data captured of a specific baby within the designated base is provided to an app or IPTV of the designated user's wireless terminal through a camera device provided at the designated base. Based on this, generate baby emotional state information for each designated emotional item for the specific baby and superimpose virtual object data corresponding to the baby emotional state information for each generated emotional item in the area around the baby's face on the video data of the transmitted data. The area around the baby's face is augmented to correspond to the baby's emotional state information for each emotional item.

In the past, hospitals, midwifery centers, or postpartum care centers equipped with multiple web PC cameras provide a service that provides members with video or still images of babies and newborn information (e.g., fetal dreams, ultrasound diagnosis photos, etc.) via the Internet. (Patent Publication No. Special 2001-0091236 (October 23, 2001)).

However, the conventional service simply provides baby videos, still images, and newborn information through the Internet. It is difficult to automatically identify and provide real-time information related to the baby's current condition in addition to the basic baby video, still images, and newborn information. It has technical difficulties.

The purpose of the present invention to solve the above problems is to, in a method executed through an operation server equipped with a sound input unit and a camera module and communicating with a camera device installed at a designated base, F Based on E(E≥2) baby expression pattern information corresponding to the geometric relationship between feature points associated with the baby's emotional expression among (F≥2) feature points and the baby's expression pattern among the designated N(N≥2) baby emotion items. Contains matching relationship information between N1 (1≤N1≤N) baby emotion items that can be distinguished by expression-based correlation numerical relationship information that quantifies the correlation between the E baby expression pattern information and N1 baby emotion items. Facial expression-based emotional state discrimination information, C (C≥2) baby cry pattern information related to the baby's emotional expression among the baby's cry patterns, and N2 that can be distinguished based on the baby cry pattern among the designated N baby emotion items. A cry containing matching relationship information between (1≤N2≤N) baby emotion items and numerical relationship information based on crying sounds that quantifies the correlation between the C baby cry pattern information and N2 baby emotion items. Based emotional state determination information, S (S ≥ 2) sensing value pattern information for sensing data of a designated sensing target related to the baby's emotional expression among the sensing data sensed through a sensor module provided or connected to the camera device, and Matching relationship information between N3 (1≤N3≤N) baby emotion items that can be distinguished based on the sensing value pattern of the designated sensing object among the designated N baby emotion items, the S sensing value pattern information, and N3 baby emotion items Sensing-based emotional state determination information, which quantifies the inter-correlation relationship, is registered and stored in the operation D/B, and R (R) is used to augment the area around the baby's face by matching it to the baby's emotional state. ≥2) virtual object data are registered and stored in the operation D/B, and virtual object matching information that matches the baby emotional state information for each designated emotion item with one or more designated virtual object data is registered and stored in the operation D/B, Video data encoded in a designated manner is received by photographing a baby in the base through a camera module of the camera device, and audio data input through a sound input unit of the camera apparatus and encoded in a designated manner are received, and the received video data and audio data are received. Synchronizes the time of the device and manages linked storage in the storage area of the designated storage medium, receives sensing data sensed through a sensor module provided in or linked to the camera device, and stores it in the storage area of the designated storage medium in connection with the video data. Manages, reads designated video data, recognizes f (1≤f≤F) feature points of the baby's facial area within the base, and corresponds to the geometric relationship between feature points associated with the baby's emotional expression among the recognized f feature points. Recognize i (i ≥ 1) baby facial expression recognition information, read audio data of a certain section associated with the video data, and recognize j (j ≥ 1) baby cry recognition information for the baby, Reading the sensing data linked to the video data to recognize k (k≥1) sensing value recognition information for the sensing data of the designated sensing target associated with the baby's emotional expression, and based on the facial expression-based emotional state determination information Confirm e (1 ≤ e ≤ E) baby expression pattern information that matches the recognized i baby facial expression recognition information and a preset standard similarity or higher, and n1 (1 ≤ E) that has a matching relationship with the e baby facial expression pattern information. n1≤N1) baby emotion items are determined, and c (1≤c≤C) items are matched with the j recognized baby cry recognition information based on the crying sound-based emotional state determination information and a preset standard similarity level or higher. Check the baby cry pattern information, determine n2 (1≤n2≤N2) baby emotion items that have a matching relationship with the confirmed c baby cry pattern information, and based on the sensing-based emotional state determination information. Confirm s (1≤s≤S) sensing value pattern information that matches the recognized k sensing value recognition information and a preset standard similarity or higher, and n3 (1≤ n3≤N3) baby emotion items are determined, and the correlation between the e baby expression pattern information and the n1 baby emotion items is based on the matching ratio between the recognized i baby facial expression recognition information and the e baby facial expression pattern information. The numerical relationship is read to calculate the baby emotional state value for each n1 emotional item, which quantifies the baby emotional state for each n1 emotional item corresponding to the recognized i baby facial expression recognition information, and the recognized j baby crying sounds. Based on the matching ratio between the recognition information and the c baby cry pattern information, the correlation and numerical relationship between the c baby cry pattern information and the n2 baby emotion items are read, and the recognized j baby cry sound recognition information is read. Calculate the baby's emotional state value for each of the n2 emotional items, which quantifies the baby's emotional state for each of the n2 corresponding emotional items, based on the matching ratio between the recognized k sensing value recognition information and the s sensing value pattern information. Baby emotions for each of the n3 emotion items, which quantifies the baby's emotional state for each of the n3 emotion items corresponding to the recognized k sensing value recognition information by reading the numerical correlation between the s sensing value pattern information and the n3 baby emotion items. A state value is calculated, and n (1≤n≤N) is specified using the calculated baby emotional state value for each n1 emotional item, the baby emotional state value for each n2 emotional item, and the baby emotional state value for each n3 emotional item. Generate baby emotional state information for each emotional item, check r (1≤r≤R) virtual object data that matches the baby emotional state information for each n emotional item based on the virtual object matching information, and , q (1 ≤ q ≤ r) virtual object data that displays a sentence string expressing the baby's emotional state corresponding to the baby's emotional state information for each of the n emotional items in the form of a speech bubble in the area around the baby's face, and the video data. The area around the baby's face is augmented to correspond to the baby's emotional state information for each of the n emotional items by overlapping the display on the area around the baby's face, and includes augmented video data to correspond to the baby's emotional state information for each of the n emotional items. The aim is to provide a method of enhancing the baby's emotional state using the area around the baby, which provides transmission data through an app on the user's wireless terminal or IPTV.

The method for enhancing the baby's emotional state using the area around the baby according to the present invention is a method that is executed through an operation server that has a sound input unit and a camera module and communicates with a camera device installed at a designated base, Among the F (F≥2) existing feature points, E (E≥2) baby expression pattern information corresponding to the geometric relationship between feature points associated with the baby's emotional expression, and the baby's expression among the designated N (N≥2) baby emotion items Matching relationship information between N1 (1≤N1≤N) baby emotion items that can be distinguished based on patterns, and facial expression-based correlation numerical relationship information that quantifies the correlation between the E baby expression pattern information and N1 baby emotion items. Based on facial expression-based emotional state discrimination information, C (C≥2) baby cry pattern information related to the baby's emotional expression among the baby's cry patterns, and baby cry pattern among the designated N baby emotion items. Contains matching relationship information between N2 (1≤N2≤N) distinguishable baby emotion items and numerical relationship information based on crying sounds that quantifies the correlation between the C baby cry pattern information and N2 baby emotion items. S (S ≥ 2) sensing values for the sensing data of a designated sensing target related to the baby's emotional expression among the emotional state determination information based on the crying sound and the sensing data sensed through the sensor module provided or linked to the camera device Matching relationship information between pattern information and N3 (1≤N3≤N) baby emotion items that can be distinguished based on the sensing value pattern of the designated sensing object among the designated N baby emotion items, and the S sensing value pattern information and the N3 Sensing-based correlation that quantifies the correlation between the baby's emotional items. Sensing-based emotional state determination information, including numerical relationship information, is registered and stored in the operation D/B, and the area around the baby's face is used to augment the area around the baby's face in response to the baby's emotional state. Register and store R (R≥2) virtual object data in the operation D/B, and register and store virtual object matching information that matches baby emotional state information for each designated emotion item with one or more designated virtual object data in the operation D/B. The first step is to photograph the baby in the base through the camera module of the camera device and receive video data encoded in a specified manner and audio data encoded in a specified manner input through the sound input unit of the camera device, and receive. Synchronizes the time of video data and audio data, stores and manages them in conjunction with the storage area of a designated storage medium, receives sensing data sensed through a sensor module provided or linked to the camera device, and stores designated data in conjunction with the video data. A second step of storing and managing the storage area of the medium and reading the designated video data to recognize f (1≤f≤F) feature points of the baby's facial area within the base, and the baby's emotion among the recognized f feature points Recognize i (i ≥ 1) baby facial expression recognition information corresponding to the geometric relationship between feature points associated with the expression, read the audio data of a certain section linked to the video data, and j (j ≥ 1) baby facial expression recognition information for the baby. A device that recognizes baby crying sound recognition information, reads sensing data associated with the video data, and recognizes k (k≥1) sensing value recognition information for the sensing data of a designated sensing target associated with the emotional expression of the baby. Based on step 3 and the facial expression-based emotional state determination information, e (1≤e≤E) baby facial expression pattern information that matches the recognized i baby facial expression recognition information and a preset standard similarity level or higher is confirmed, and the e baby Determine n1 (1≤n1≤N1) baby emotion items that have a matching relationship with facial expression pattern information, and determine the similarity between the recognized j baby cry recognition information and a preset standard based on the crying sound-based emotional state determination information. Confirm the c (1≤c≤C) baby cry pattern information matched above, and determine n2 (1≤n2≤N2) baby emotion items that have a matching relationship with the confirmed c baby cry pattern information. , Based on the sensing-based emotional state determination information, check the s (1≤s≤S) sensing value pattern information that matches the recognized k sensing value recognition information and a preset standard similarity or higher, and check the confirmed s sensing values. Based on the fourth step of determining n3 (1≤n3≤N3) baby emotion items that have a matching relationship with the value pattern information and the matching ratio between the recognized i baby facial expression recognition information and the e baby facial expression pattern information, The baby's emotional state for each of the n1 emotion items corresponding to the recognized i baby facial expression recognition information is quantified by reading the numerical correlation between the e baby facial expression pattern information and the n1 baby emotion items. An emotional state value is calculated, and a correlation value between the c baby cry pattern information and the n2 baby emotion items is based on the matching ratio between the j recognized baby cry recognition information and the c baby cry pattern information. By reading the relationship, the baby's emotional state values are calculated for each of the n2 emotional items, which quantifies the baby's emotional state for each of the n2 emotional items corresponding to the recognized j baby cry recognition information, and the recognized k sensing value recognition information. Based on the matching ratio between the s sensing value pattern information and the s sensing value pattern information, the correlation and numerical relationship between the s sensing value pattern information and the n3 baby emotion items are read, and n3 emotions corresponding to the recognized k sensing value recognition information are read. A fifth step of calculating the baby emotional state values for each of the n3 emotional items that quantify the baby's emotional state for each item, and the calculated baby emotional state values for the n1 emotional items, the baby emotional state values for the n2 emotional items, and the n3 emotions. A sixth step of generating baby emotional state information for each n (1≤n≤N) emotional items specified using the baby emotional state values for each item, and the baby for each n emotional items generated based on the virtual object matching information. A seventh step of checking r (1 ≤ r ≤ R) virtual object data matching the emotional state information and a sentence string expressing the baby's emotional state corresponding to the baby's emotional state information for each of the n emotional items is sent to the baby's face. q (1≤q≤r) virtual object data displayed in the form of a speech bubble in the surrounding area are overlaid on the area around the baby's face on the video data, and the area around the baby's face is included in the baby's emotional state information for each of the n emotion items. An eighth step of augmenting to correspond, and a ninth step of providing transmission data including video data augmented to correspond to the baby emotional state information for each of the n emotion items to an app or IPTV of the user's wireless terminal. do.

delete

delete

According to the present invention, the geometric relationship between the feature points is characterized in that it includes one or more relationships among a distance relationship between each feature point, an angle relationship between each feature point, and a combination relationship of a distance relationship and an angle relationship between each feature point.

According to the present invention, the facial expression-based correlation numerical relationship information may include correlation numerical information calculated through a relational expression proportional to the matching ratio between baby facial expression recognition information that recognizes the baby's facial expression and pre-registered baby facial expression pattern information. You can.

According to the present invention, the facial expression-based correlation numerical relationship information may include correlation numerical information for each matching ratio section between baby facial expression recognition information that recognizes the baby's facial expression and pre-registered baby facial expression pattern information.

delete

delete

delete

delete

delete

delete

According to the present invention, the transmission data further includes audio data stored in the storage area of the storage medium.

According to the present invention, the designated video data may include at least one of video data periodically selected for each designated playback time unit and video data periodically selected for each designated number of playback frames.

According to the present invention, it further includes the step of confirming whether transmission data including video data stored in the storage area of the storage medium is provided to the app or IPTV of the designated user wireless terminal, and the third step is performed by the designated user wireless terminal. When providing transmission data including video data to an app or IPTV, designated video data among the video data of the transmission data is read, and f feature points of the baby's facial area in the base are recognized, and the recognized f It is characterized in that it includes the step of recognizing i baby facial expression recognition information corresponding to the geometric relationship between feature points related to the baby's emotional expression among the feature points.

According to the present invention, the third step reads designated video data from among the video data stored in the storage area of the designated storage medium, recognizes f feature points of the baby's facial area in the base, and selects f feature points among the recognized f feature points. It includes the step of recognizing i baby facial expression recognition information corresponding to the geometric relationship between feature points associated with the baby's emotional expression, wherein the sixth step is to combine the baby emotional state information for each of the generated n emotional items with the video data. It is characterized by further including the step of linked storage.

According to the present invention, the sixth step may include storing the baby emotional state information for each of the generated n emotional items in association with a time value (or timestamp) synchronized with a designated playback point of the video data. there is.

delete

delete

According to the present invention, the C baby cry pattern information is characterized in that it includes range information of cry characteristics that make up each baby cry pattern associated with the baby's emotional expression.

According to the present invention, the cry characteristic is a frequency band feature, a placement relationship feature of the frequency band feature, a periodicity feature of the frequency band feature, a sound intensity feature, a placement relationship feature of the sound intensity feature, a periodicity feature of the sound intensity feature, and a frequency band feature. It may include one or more features among a combination feature of a feature and a loudness feature, a placement relationship feature of a frequency band feature and a loudness feature, and a periodicity feature of a frequency band feature and a loudness feature.

According to the present invention, the crying sound-based correlation numerical relationship information is a correlation calculated through a relational expression proportional to the matching ratio between the baby crying sound recognition information that recognizes the baby's crying sound and the already registered baby crying sound pattern information. It is characterized by including numerical information.

According to the present invention, the crying sound-based correlation numerical relationship information includes correlation numerical information for each matching ratio section between the baby crying sound recognition information that recognizes the baby's crying sound and the already registered baby crying sound pattern information. It is characterized by being accomplished by doing so.

delete

delete

According to the present invention, the sensor module is at least one of a wearable sensor module that is worn on the baby's body and senses a designated sensing target, and a radar sensor module that senses a designated sensing target of the baby using designated electromagnetic waves. It is characterized by including one sensor module.

According to the present invention, the sensing object is characterized in that it includes at least one of the baby's heartbeat, the baby's body temperature, the humidity around the baby, and whether the baby is apnea.

According to the present invention, the S sensing value pattern information is characterized in that it includes range information of sensing value features that constitute each sensing value pattern for sensing data of a designated sensing target associated with the baby's emotional expression. .

According to the present invention, the sensing value feature may include at least one or a combination of two or more of a sensing value change rule feature, a sensing value change amount feature, a sensing value change rate feature, and a sensing value change periodicity feature.

According to the present invention, the sensing-based correlation numerical relationship information is a correlation calculated through a relational expression proportional to the matching ratio between sensing value recognition information recognized through sensing data of a designated sensing target and pre-registered sensing value pattern information. It is characterized by including numerical information.

According to the present invention, the sensing-based correlation numerical relationship information includes correlation numerical information for each matching ratio section between sensing value recognition information recognized through sensing data of a designated sensing target and pre-registered sensing value pattern information. It is characterized by being accomplished by doing so.

According to the present invention, the baby emotional state information for each of the n emotional items may include n emotional item information and numerical information about the baby's emotional state for each emotional item.

delete

delete

According to the present invention, the method for enhancing the baby's emotional state using the area around the baby further includes the step of recognizing the orientation direction of the baby's face on the video data, wherein the eighth step corresponds to the orientation direction of the baby's face. It may include the step of overlapping and displaying the q virtual object data in the area around the baby's face.

According to the present invention, when transmission data including video data captured of a specific baby within the designated base is provided to an app or IPTV of the designated user's wireless terminal through a camera device provided at the designated base, one or more items obtained through the camera device Based on the data, generate baby emotional state information for each emotion item specified for the specific baby, and virtual object data corresponding to the baby emotional state information for each emotion item generated in the area around the baby's face on the video data of the transmitted data. By overlapping the display and augmenting the area around the baby's face to correspond to the baby's emotional state information for each emotional item, a user watching the transmitted data through an app of a wireless terminal or IPTV corresponds to the video data of the transmitted data. In addition to seeing the baby's video screen and listening to the baby's sounds corresponding to the audio data of the transmitted data, you can intuitively determine the baby's current emotional state in real time through virtual object data displayed overlaid on the area around the baby's face on the video data. There are benefits to this.

1 is a diagram illustrating the configuration of a system for enhancing a baby's emotional state according to an implementation method of the present invention.
Figure 2 is a diagram illustrating the information storage/registration process according to the implementation method of the present invention.
Figure 3 is a diagram showing how acquisition data obtained through a camera device at a designated base is transmitted and stored and managed in a designated storage medium according to the implementation method of the present invention.
Figure 4 is a diagram illustrating providing transmission data including video data augmented with a baby's emotional state according to a first implementation method of the present invention.
Figure 5 is a diagram illustrating providing transmission data including video data augmented with a baby's emotional state according to a second implementation method of the present invention.
Figure 6 is a diagram illustrating providing transmission data including video data augmented with a baby's emotional state according to a third implementation method of the present invention.

Hereinafter, the operating principle of a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings and description. However, the drawings shown below and the description below are for preferred implementation methods among various methods for effectively explaining the characteristics of the present invention, and the present invention is not limited to the drawings and description below.

That is, the following examples correspond to examples of preferred union forms among the numerous embodiments of the present invention, and examples in which specific configurations (or steps) are omitted in the following examples, or specific configurations (or steps) are omitted. An embodiment of dividing the function implemented in a specific configuration (or step), or an embodiment of integrating the function implemented in two or more configurations (or steps) into one configuration (or step), a specific configuration (or step) It is clearly stated that embodiments that replace the operation sequence of all fall within the scope of the present invention, even if not separately mentioned in the following examples. Therefore, based on the examples below, it is clearly stated that various embodiments corresponding to subsets or complements can be divided retroactively to the filing date of the present invention.

Additionally, in the following description of the present invention, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. The terms described below are terms defined in consideration of functions in the present invention, and may vary depending on the intention or custom of the user or operator. Therefore, the definition should be made based on the overall content of the present invention.

As a result, the technical idea of the present invention is determined by the claims, and the following examples are a means to efficiently explain the advanced technical idea of the present invention to those skilled in the art to which the present invention pertains. It's just that.

Figure 1 is a diagram showing the configuration of a system for enhancing the emotional state of a baby according to the implementation method of the present invention.

In more detail, Figure 1 shows transmission data including video data captured of a specific baby within the designated base through a camera device 190 provided at the designated base to the app or IPTV 185 of the designated user wireless terminal 180. When provided, baby emotional state information is generated for each designated emotional item for the specific baby based on one or more data acquired through the camera device 190, and the generated baby emotional state information is generated in the area around the baby's face on the video data of the transmitted data. This shows the configuration of a system that displays virtual object data corresponding to the baby's emotional state information for each emotional item and augments the area around the baby's face to correspond to the baby's emotional state information for each of the n emotional items. Those of ordinary skill in the art to which the invention pertains can refer to and/or modify Figure 1 to provide various implementation methods for the configuration of the system (e.g., implementation methods in which some components are omitted, subdivided, or combined). ) can be inferred, but the present invention includes all of the above inferred implementation methods, and its technical features are not limited to only the implementation method shown in Figure 1.

The system of the present invention includes a camera device 190 provided at a designated base to film a subject and connect to a communication network, and an image signal obtained by photographing a specific baby in the base through the camera module of the camera device 190. A function of receiving acquired data containing video data encoded in a designated manner and storing it in a specific storage area of a designated storage medium, transmitting data containing video data stored in a specific storage area of the storage medium to a designated user wireless terminal (180) ) function provided by the app or IPTV 185, the function of generating baby emotional state information for each designated emotional item for the specific baby based on one or more data acquired through the camera device 190, the video A function of overlapping and displaying virtual object data corresponding to the baby's emotional state information for each emotion item generated above in the area around the baby's face in the data to augment the area around the baby's face to correspond to the baby's emotional state information for each of the n emotional items. It includes an operation server 100 that performs one or more functions. The operating server 100 may be implemented in the form of at least one or a combination of two or more of the following: an independent server, a combination of two or more servers, and software running through a pre-equipped server. The present invention is not limited by the examples or the name of the operation server 100.

The camera device 190 is a general term for a terminal or device equipped at a designated location and equipped with a camera to photograph a subject. It is preferably installed in a space where a baby resides at a postpartum care center or at home and captures a specific baby present in the space. It may include a terminal or device equipped with a camera that takes pictures.

According to the implementation method of the present invention, the camera device 190 includes a camera module that photographs a designated subject within the base, a communication module that connects to a designated communication network, and a video encoding the image signal acquired through the camera module in a designated manner. It is provided with a control module that performs a process of generating acquired data including data and transmitting it to the designated operation server 100 through the communication module.

The camera module includes one or more lens units that receive optical signals reflected from a subject, an image sensor unit that detects optical signals input through the lens unit and converts them into electrical image signals, and an automatic connection between the lens unit and the image sensor unit. It includes an actuator that implements designated functional operations, including focus functions and zoom functions. Meanwhile, depending on the implementation method, the camera module may further include a filter unit between the lens unit and the image sensor unit including an infrared filter for photographing a subject even at night. Meanwhile, the camera module is equipped with a PCB for providing the image signal generated through the image sensor unit to the control module. Meanwhile, depending on the implementation method, the camera module may further be equipped with an image signal processor (ISP) that processes the image signal according to a specified procedure to improve the quality of the image signal generated through the image sensor unit. .

The communication module is a general term for modules that connect to a packet-based communication network, and preferably includes at least one or both of a communication module that connects to a communication network using a wire such as a UTP cable and a communication module that connects to a communication network through a wireless LAN. It can be included. Meanwhile, depending on the implementation method, the communication module may further (or replace) include a communication module that connects to the communication network through a mobile communication network or a separately designated wireless communication in addition to the wireless LAN, but the present invention is not limited thereto.

The control module may generate video data by encoding the video signal generated through the camera module in a designated manner. Meanwhile, depending on the implementation method, the video data may be generated through the camera module and transmitted to the control module. Meanwhile, the video data may further include a time value (or timestamp) at the time the video signal was captured.

The control module may perform a procedure of generating acquisition data including the video data and transmitting it to the designated operation server 100 through the communication module. Meanwhile, depending on the implementation method, the acquired data may further include audio data in addition to the video data and/or may further include sensing data sensed through a designated sensor module.

According to the method of implementing the present invention, the camera device 190 further includes a sound input unit that receives a sound signal within the base, and in this case, the control module encodes the sound signal obtained through the sound input unit in a designated manner. A procedure can be performed to generate acquired data further including audio data and transmit it to the designated operation server 100 through the communication module.

The sound input unit includes a microphone that converts air vibration into a sound signal in the form of an electrical signal, and an interface unit that interfaces the same with a control module, wherein the control module converts the sound signal obtained through the sound input unit into a sound signal in a designated manner. Audio data can be generated by encoding. Meanwhile, when the sound input unit is implemented in the form of a separate module according to the implementation method, the audio data can be generated through the camera module and transmitted to the control module through the sound input unit. Meanwhile, the audio data may further include a time value (or timestamp) at the time the sound signal is input.

When the camera device 190 is provided with the sound input unit and audio data encoded by the sound signal acquired through the sound input unit is generated, the control module generates acquired data including the video data and audio data. The transmission procedure to the designated operation server 100 can be performed through the communication module.

According to an implementation method of the present invention, the acquired data may be generated in the form of video data (or multimedia data) including the video data and audio data in a designated data format.

According to another implementation method of the present invention, the acquired data includes the video data and audio data in each designated data format, and the viewpoint is determined through a time value (or timestamp) included in the video data and audio data. It can be created in each data format that can be synchronized.

According to the implementation method of the present invention, the camera device 190 may further be provided with a sound output unit that outputs a designated sound signal at a designated signal strength, and in this case, the control module outputs the sound through the sound output unit according to a designated procedure. Generate audio data corresponding to the sound signal to be output and/or extract it from a designated storage area and/or extract audio data corresponding to the sound signal to be output through the sound output unit from the operation server 100 designated through the communication module. A procedure may be performed to receive and process a sound signal corresponding to the generated/extracted/received audio data to be output through the sound output unit.

The sound output unit includes a speaker that outputs a sound signal in the form of an electrical signal decoded from audio data, and an interface unit that interfaces the speaker with a control module. The control module generates audio data to be output through the sound output unit, and /or extract from a designated storage area and/or receive from the designated operation server 100 through the communication module, and output a sound signal corresponding to the audio data through the sound output unit with a designated signal strength (e.g., camera device 190 The sound signal output through the speaker can be output at a signal intensity set to be transmitted at a specified sound volume to a baby spaced within a specified distance.

According to the implementation method of the present invention, the camera device 190 may be provided with or linked to a sensor module that senses a designated sensing target of the baby through a designated sensor, and the control module may be configured to detect the baby's designated sensing object through a designated sensor. A procedure may be performed to generate acquired data further including sensing data encoded by a sensing signal in a designated manner and transmit it to the designated operation server 100 through the communication module. Preferably, the baby's designated sensing object sensed through the sensor module may include at least one of the baby's heartbeat, the baby's body temperature, the humidity around the baby, and whether the baby is apnea.

The sensor module may include a wearable sensor module that is worn on a designated part of the baby's body to sense the baby's designated sensing target. In this case, the control module uses established short-range communication (e.g., Bluetooth, etc.). In connection with the wearable sensor module, a sensing signal corresponding to a sensing value sensed through the sensor module may be received, or sensing data encoded by the sensing signal in a designated manner may be received through the sensor module.

Meanwhile, the sensor module may include a radar sensor module that senses the designated sensing target of the baby using designated electromagnetic waves. In this case, the control module may generate a sensing signal corresponding to the sensing value received through the radar sensor module. Sensing data can be generated by checking and encoding in a specified manner. Meanwhile, the radar sensor module can generate sensing data encoded by the sensing signal in a designated manner and provide it to the control module. In this case, the control module can receive the sensing data from the radar sensor module.

According to the implementation method of the present invention, the sensing data includes a sensing value obtained by sensing a designated sensing target through the sensor module, and preferably includes a time value (or timestamp) at the time of sensing the sensing value through the sensor module. ) may further be included.

When the camera device 190 is equipped with or linked to the sensor module to generate sensing data that senses the baby's designated sensing target, the control module adds the sensing data to the acquired data and detects the designated sensing target through the communication module. The transmission procedure to the operation server 100 can be performed. Preferably, the sensing data may be included in the acquired data in a form included in a designated data format, and may be generated in a form of data whose viewpoint can be synchronized through a time value (or timestamp) included in the video data and the sensing data. You can.

According to the implementation method of the present invention, the camera device 190 further includes a driving module that drives the camera module in the shooting direction in the up/down/left/right directions within a specified angle range, and the control module controls the camera. By reading the video data captured through the module, you can attempt to recognize an object that corresponds to the baby (or the baby's face) or the cradle or blanket on which the baby is lying. If the object is not recognized, the control module moves the shooting direction of the camera module through the drive module by a specified angle range in a specified direction and repeats the process of recognizing the object until the entire angle range is reached. can do. If the object is recognized through the above recognition process, the camera device 190 controls the shooting direction so that the recognized facial area of the baby is within a designated area on the video data, and moves the facial area of the baby. By tracking the baby's facial area in response to movement, the shooting direction of the camera module can be controlled to exist within a designated area on the video data. Meanwhile, the process of controlling the shooting direction of the camera module or recognizing the object through the drive module may be performed through the server, but the present invention is not limited thereto.

According to the implementation method of the present invention, the camera device 190 may further include a projector unit that projects a designated shape or image onto the ceiling or wall through a lamp. In this case, the control module controls the projector unit according to a designated procedure. The lamp can be turned on/off, or the control module can receive control information for turning on/off the lamp of the projector unit from the designated operation server 100 through the communication module, and the control module can receive the control information. Based on this, the lamp of the projector unit can be turned on/off. Meanwhile, when it is possible to select a shape or image to be projected by selecting one of a plurality of films through the projector unit, the control module selects the film of the projector unit and/or turns on/off the lamp according to a designated procedure. Alternatively, the control module may receive control information for selecting the film of the projector unit and/or turning on/off the lamp from the designated operation server 100 through the communication module, and may receive the control information. Based on this, the film of the projector unit can be selected and/or the lamp can be turned on/off. Meanwhile, when an electronic shape or electronic image can be projected through the projector unit, the control module extracts the electronic shape or electronic image stored in a designated storage area according to a designated procedure and processes it into a shape or image to be projected through the projector unit, and/ Alternatively, the lamp can be turned on/off, or the control module receives the stored electronic shape or electronic image from the designated operation server 100 through the communication module and/or control information for turning the lamp on/off. Based on the control information, the received electronic shape or electronic image can be processed into a shape or image to be projected through the projector unit, and/or the lamp can be turned on/off.

The user wireless terminal 180 is a general term for wireless communication devices used by the user who will receive transmission data including video data captured through the camera device 190, such as a mobile phone, smartphone, or tablet PC used by the user. It may include various wireless communication devices such as.

According to the implementation method of the present invention, the user wireless terminal 180 registers specified information in the operation server 100 and includes video data captured through the camera device 190 from the operation server 100. An application is installed and executed to receive transmission data. Hereinafter, unless otherwise stated, the description of the function implemented using the user wireless terminal 180 as the subject or object will be made clear that it is a function implemented through the app.

The operating server 100 is a general term for a server capable of communicating with the camera device 190 provided at a designated base and communicating with the app of the user wireless terminal 180 that installs and executes the designated app, and is the camera of the camera device 190. A function to receive acquisition data including video data encoded in a designated manner by filming a specific baby in the base through a module and store it in a specific storage area of a designated storage medium, video data stored in a specific storage area of the storage medium It performs at least one function of providing transmission data including to an app of the designated user wireless terminal 180 or IPTV 185, and one or more data acquired through the camera device 190 according to the present invention. Based on this, a function to generate baby emotional state information for each emotional item specified for the specific baby, and superimposing virtual object data corresponding to the baby emotional state information for each generated emotional item in the area around the baby's face on the video data. At least one of the functions of displaying and augmenting the area around the baby's face to correspond to the baby's emotional state information for each of the n emotional items is performed.

According to an implementation method of the present invention, the transmission data includes, in addition to video data stored in a specific storage area of the storage medium, audio data stored in a specific storage area of the storage medium in connection with the video data, or the video data. Audio data of the video data may be further included.

Referring to Figure 1, the operation server 100, in order to determine the emotional state of the baby, matches the pattern information of the recognition target designated to be recognized and each baby emotional item, and the pattern information of the designated recognition target. It includes an information management unit 105 that registers, stores, and manages emotional state determination information for each designated recognition target, including numerical correlation information, which quantifies the correlation between the baby's emotional items, in a designated operation D/B.

The information management unit 105 provides E (E ≥ 2) baby expression pattern information corresponding to the geometric relationship between feature points associated with the baby's emotional expression among the F (F ≥ 2) feature points present in the baby's facial area and the designated N. It includes matching relationship information between N1 (1≤N1≤N) baby emotion items that can be distinguished based on baby expression patterns among (N≥2) baby emotion items, and includes the E baby expression pattern information and N1 baby emotions. Expression-based emotional state determination information, which includes facial expression-based correlation and numerical relationship information that quantifies the correlation between items, is registered and stored in the designated operation D/B.

According to the implementation method of the present invention, the information management unit 105 receives E baby expression pattern information corresponding to the geometric relationship between feature points associated with the baby's emotional expression among the F feature points present in the baby's facial area from the designated operating terminal. Receive registration, and receive matching relationship information that matches the E baby expression pattern information with N1 baby emotion items that can be distinguished based on the baby expression pattern among the designated N baby emotion items, and register the E baby expression pattern information and N1 baby emotion items. The facial expression-based correlation information, which quantifies the correlation between the baby's emotional items, can be registered and stored in the designated operating D/B. For example, the information management unit 105 includes the E baby expression pattern information and N1 baby emotion items as row or column titles, and stores the facial expression-based correlation numerical relationship information in cells corresponding to each row and column. The facial expression-based emotional state determination information in the form of a table included as information can be registered and stored in the designated operation D/B.

According to the method of implementing the present invention, the N baby emotion items include 'joy', 'anger', 'sadness', 'joy', 'love', Among the basic emotions such as 'hatred' and 'greed', there is one or more emotions that even babies who are not socialized or are less advanced than adults can feel (e.g. 'joy', 'anger', 'sadness', It may include emotional items corresponding to 'joy', etc.), and in addition to the basic emotions above, various emotions that the baby can feel (e.g., 'annoyed', 'lonely', 'frustrated', 'displeasure', 'fear', etc.) ) and/or an emotion that embodies at least one of the basic emotions above to suit the baby's situation (e.g., 'sleepiness' corresponding to the desire to sleep among 'greed', 'hunger' corresponding to appetite, etc. ) may further include emotion items corresponding to.

According to one implementation method of the present invention, the E baby facial expression pattern information may include geometric relationship information between each feature point corresponding to each baby facial expression pattern associated with the baby's emotional expression. Preferably, the geometric relationship between the feature points may include one or more of a distance relationship between each feature point, an angle relationship between each feature point, and a combination relationship of a distance relationship and an angle relationship between each feature point. For example, the geometric relationship information between feature points may include one or more of a distance value between each feature point based on a specified distance value between reference feature points, an angle value between each feature point, and a combination of the distance value and angle value between each feature point. You can.

Meanwhile, according to another implementation method of the present invention, the E baby facial expression pattern information may include range information of the geometric relationship between each feature point corresponding to each baby facial expression pattern associated with the baby's emotional expression. Preferably, the geometric relationship between the feature points may include one or more of a distance relationship between each feature point, an angle relationship between each feature point, and a combination relationship of a distance relationship and an angle relationship between each feature point. For example, the range information of the geometric relationship between feature points may be a combination of the distance value range between each feature point based on the specified reference feature point distance value, the angle value range between each feature point, and the distance value range and angle value range between each feature point. May contain one or more numerical ranges.

According to one implementation method of the present invention, the facial expression-based correlation value relationship information is a correlation value calculated through a relationship equation proportional to the matching ratio between baby facial expression recognition information that recognizes the baby's facial expression and pre-registered baby facial expression pattern information. May contain information. For example, the facial expression-based correlation numerical relationship information may be calculated as small numerical information as the matching ratio between baby facial expression recognition information and pre-registered baby facial expression pattern information decreases, and as the matching ratio increases, it may be calculated as large numerical information. It can be.

Meanwhile, according to another implementation method of the present invention, the facial expression-based correlation numerical relationship information includes correlation numerical information for each matching ratio section between the baby facial expression recognition information that recognizes the baby's facial expression and the already registered baby facial expression pattern information. It can be included. For example, the facial expression-based correlation numerical relationship information may be calculated as '0' or numerical information of a specified minimum value when the matching ratio between baby facial expression recognition information and pre-registered baby facial expression pattern information is a pre-set minimum ratio range. , if the matching ratio is a preset maximum ratio section, it can be calculated as numerical information of the specified maximum value.

Meanwhile, according to the implementation method of the present invention, after the expression-based emotional state determination information is registered and stored in the operation D/B through a designated operation terminal, when the designated baby facial expression recognition information is recognized through the information recognition unit 140 below. , the information management unit 105 recognizes an object corresponding to the baby's guardian by reading video data received from the camera device 190 and stored in a specific storage area of a designated storage medium after the baby's facial expression recognition information is recognized. At the same time, the audio data associated with the video data is read, and among the voices uttered by the baby's guardian to the baby, voices related to the recognition of the baby's emotions (e.g., 'My baby is hungry', 'Is my baby poop?', You can perform a recognition procedure (e.g., ‘My baby is sleepy’). If a voice related to the guardian's recognition of the baby's emotion is recognized after the baby's facial expression recognition information is recognized, the information management unit 105 patterns the recognized baby's facial expression recognition information and simultaneously responds to the voice related to the baby's emotion recognition. By repeating the process of identifying voice recognition-based baby emotion items and learning a combination of baby expression pattern information patterned from the recognized baby expression recognition information and the identified voice recognition-based baby emotion items, the expression-based The big data-based accuracy and reliability of emotional state determination information can be continuously improved.

Meanwhile, according to an extended embodiment of the present invention, the information management unit 105 recognizes the baby's cry based on audio data acquired through the sound input unit of the camera device 190 in addition to the facial expression-based emotional state determination information. Register and store crying sound-based emotional state determination information in the operating D/B to determine the emotional state of the baby, and/or the camera device 190 is equipped with a sensor module that senses a designated sensing object related to the baby's emotional expression. Sensing-based emotional state determination to determine the baby's emotional state using the sensing data when sensing data obtained by sensing the baby's designated sensing target is received from the camera device 190 through the sensor module. Information can be registered and saved in the operation database.

Meanwhile, according to another embodiment of the present invention, the information management unit 105 operates the crying sound-based emotional state determination information and/or sensing-based emotional state determination information in place of (or replacing) the facial expression-based emotional state determination information. It can be registered and stored in D/B, but the present invention is not limited thereto. Hereinafter, for convenience, the present invention will be described through an embodiment in which facial expression-based emotional state determination information is used as basic emotional state determination information and the crying sound-based emotional state determination information and/or sensing-based emotional state determination information are used as secondary emotional state determination information. The features of the present invention will be described, but the technical features of the present invention are not limited thereto, and according to the description of the embodiments and claims of the present invention, the crying sound-based emotional state determination information and/or the sensing-based emotional state determination information are also basic. We make it clear that it can be used as emotional state determination information.

According to an extension/another embodiment of the present invention, the information management unit 105 provides C (C≥2) baby cry pattern information related to the baby's emotional expression among the baby's cry patterns and N designated baby emotion items. A cry that contains matching relationship information between N2 (1≤N2≤N) baby emotion items that can be distinguished based on baby cry patterns, and quantifies the correlation between the C baby cry pattern information and N2 baby emotion items. Crying-based emotional state determination information, including sound-based correlation and numerical relationship information, can be registered and stored in the operation D/B.

According to the implementation method of the present invention, the information management unit 105 registers C baby cry pattern information related to the baby's emotional expression among the baby cry patterns from the designated operating terminal, and receives the C baby cry pattern information. Register matching relationship information that matches N2 baby emotion items that can be distinguished based on baby cry patterns among the designated N baby emotion items, and calculate the correlation between the C baby cry pattern information and the N2 baby emotion items. Numerical relationship information based on crying sounds can also be registered and stored in a designated operational D/B. For example, the information management unit 105 includes C baby cry pattern information and N2 baby emotion items as row or column titles, and converts the cry sound-based correlation numerical relationship information into information of cells corresponding to each row and column. Crying-based emotional state determination information in a table format can be registered and stored in the operation D/B.

According to one implementation method of the present invention, the C baby cry pattern information may include cry characteristic information constituting each baby cry pattern associated with the baby's emotional expression. Preferably, the cry characteristic includes a frequency band feature, a placement relationship feature of the frequency band feature, a periodicity feature of the frequency band feature, a sound intensity feature, a placement relationship feature of the sound intensity feature, a periodicity feature of the sound intensity feature, and a frequency band feature. It may include one or more features among a combination feature of sound intensity features, an arrangement relationship feature of frequency band features and sound intensity features, and a periodicity feature of frequency band features and sound intensity features.

Meanwhile, according to another implementation method of the present invention, the C baby cry pattern information may include range information of cry characteristics constituting each baby cry pattern associated with the baby's emotional expression. Preferably, the cry characteristic includes a frequency band feature, a placement relationship feature of the frequency band feature, a periodicity feature of the frequency band feature, a sound intensity feature, a placement relationship feature of the sound intensity feature, a periodicity feature of the sound intensity feature, and a frequency band feature. It may include one or more features among a combination feature of sound intensity features, an arrangement relationship feature of frequency band features and sound intensity features, and a periodicity feature of frequency band features and sound intensity features. For example, the range information of the crying sound characteristics may include an allowable range of frequency bands for each feature and/or an allowable range of sound intensity for each feature.

According to an implementation method of the present invention, the crying sound-based correlation numerical relationship information is calculated through a relationship equation proportional to the matching ratio between the baby crying sound recognition information that recognizes the baby's crying sound and the already registered baby crying sound pattern information. The correlation may also include numerical information. For example, the crying sound-based correlation numerical relationship information may be calculated as smaller numerical information as the matching ratio between the baby crying sound recognition information and the already registered baby crying sound pattern information is smaller, and the larger the matching ratio, the larger the numerical information. It can be calculated as information.

Meanwhile, according to another implementation method of the present invention, the crying sound-based correlation numerical relationship information is the correlation for each matching ratio section between the baby crying sound recognition information that recognizes the baby's crying sound and the already registered baby crying sound pattern information. It may also include numerical information. For example, the crying sound-based correlation numerical relationship information is calculated as '0' or numerical information of the specified minimum value when the matching rate between the baby crying sound recognition information and the registered baby crying sound pattern information is within the preset minimum ratio range. It can be, and if the matching ratio is a preset maximum ratio section, it can be calculated as numerical information of the specified maximum value.

Meanwhile, according to the implementation method of the present invention, after the crying sound-based emotional state determination information is registered and stored in the operation D/B through a designated operation terminal, the designated baby cry recognition information is recognized through the following information recognition unit 140. In this case, the information management unit 105 reads the video data received from the camera device 190 and stored in a specific storage area of the designated storage medium after the baby's crying sound recognition information is recognized to respond to the baby's guardian. At the same time as recognizing the object, the audio data associated with the video data is read, and among the voices uttered by the baby's guardian to the baby, voices related to the recognition of the baby's emotions (e.g., 'My baby is hungry', 'My baby's poop?' ?', 'My baby is sleepy', etc.) can be performed. If a voice related to the guardian's recognition of the baby's emotion is recognized after the baby's cry recognition information is recognized, the information management unit 105 patterns the recognized baby's cry recognition information and simultaneously creates a voice related to the baby's emotion recognition. Repeat the process of identifying the corresponding voice recognition-based baby emotion item and learning a combination of the identified baby cry pattern information patterned from the recognized baby cry recognition information and the identified voice recognition-based baby emotion item. By doing so, the big data-based accuracy and reliability of the crying sound-based emotional state determination information can be continuously improved.

Meanwhile, according to an expansion/another embodiment of the present invention, the information management unit 105 detects a designated sensing object related to the baby's emotional expression among the sensing data sensed through a sensor module provided or linked to the camera device 190. Matching relationship between S (S≥2) sensing value pattern information for data and N3 (1≤N3≤N) baby emotion items that can be distinguished based on the sensing value pattern of the designated sensing object among the designated N baby emotion items. Sensing-based emotional state determination information including information and sensing-based correlation numerical relationship information that quantifies the correlation between the S sensing value pattern information and N3 baby emotional items can be registered and stored in the operation D/B.

According to the method of implementing the present invention, the sensor module is a wearable sensor module that is worn on the baby's body to sense a designated sensing target, and a radar sensor module that senses the baby's designated sensing target using designated electromagnetic waves. It may include at least one sensor module, and the designated sensing object may include at least one of the baby's heartbeat, the baby's body temperature, the humidity around the baby, and whether the baby is apnea.

According to the implementation method of the present invention, the information management unit 105 detects a designated sensing object related to the baby's emotional expression among the sensing data sensed through a sensor module provided or linked to the camera device 190 from a designated operating terminal. Matching relationship information between N3 baby emotion items that can register S sensing value pattern information for data and distinguish the S sensing value pattern information based on the sensing value pattern of the designated sensing target among the N designated baby emotion items. can be registered, and sensing-based correlation numerical relationship information, which quantifies the correlation between the S sensing value pattern information and N3 baby emotion items, can be registered and stored in a designated operation D/B. For example, the information management unit 105 includes the S sensing value pattern information and N3 baby emotion items as row or column titles, and includes sensing-based correlation numerical relationship information as information in cells corresponding to each row and column. Sensing-based emotional state determination information in the form of a table can be registered and stored in the operation D/B.

According to one implementation method of the present invention, the S sensing value pattern information may include sensing value characteristic information constituting each sensing value pattern for sensing data of a designated sensing target associated with the baby's emotional expression. Preferably, the sensing value feature may include at least one or a combination of two or more of a sensing value change rule feature, a sensing value change amount feature, a sensing value change rate feature, and a sensing value change periodicity feature.

Meanwhile, according to another implementation method of the present invention, the S sensing value pattern information may include range information of sensing value features that constitute each sensing value pattern for sensing data of a designated sensing target associated with the baby's emotional expression. You can. Preferably, the sensing value feature may include at least one or a combination of two or more of a sensing value change rule feature, a sensing value change amount feature, a sensing value change rate feature, and a sensing value change periodicity feature. For example, the range information of the sensing value features may include an allowable range of change rules for each feature, an allowable range of the amount of change in sensing value for each feature, and an allowable range of the rate of change of the sensing value for each feature.

According to an implementation method of the present invention, the sensing-based correlation numerical relationship information is calculated through a relational expression proportional to the matching ratio between sensing value recognition information recognized through sensing data of a designated sensing target and pre-registered sensing value pattern information. The correlation may also include numerical information. For example, the sensing-based correlation numerical relationship information may be calculated as small numerical information as the matching ratio between sensing value recognition information and pre-registered sensing value pattern information is small, and as the matching ratio is large, it may be calculated as large numerical information. It can be.

Meanwhile, according to another implementation method of the present invention, the sensing-based correlation numerical relationship information is the correlation for each matching ratio section between sensing value recognition information recognized through sensing data of a designated sensing target and pre-registered sensing value pattern information. It may also include numerical information. For example, the sensing-based correlation numerical relationship information may be calculated as numerical information of '0' or a specified minimum value when the matching ratio between sensing value recognition information and pre-registered sensing value pattern information is a preset minimum ratio range, , if the matching ratio is a preset maximum ratio section, it can be calculated as numerical information of the specified maximum value.

Meanwhile, according to the implementation method of the present invention, after the sensing-based emotional state determination information is registered and stored in the operation D/B through a designated operation terminal, when the designated sensing value recognition information is recognized through the information recognition unit 140 below , the information management unit 105 reads video data received from the camera device 190 and stored in a specific storage area of a designated storage medium after the sensing value recognition information is recognized to recognize an object corresponding to the baby's guardian. At the same time, the audio data associated with the video data is read, and among the voices uttered by the baby's guardian to the baby, voices related to the recognition of the baby's emotions (e.g., 'My baby is hungry', 'Is my baby poop?', You can perform a recognition procedure (e.g., ‘My baby is sleepy’). If a voice related to the guardian's recognition of the baby's emotion is recognized after the sensing value recognition information is recognized, the information management unit 105 patterns the recognized sensing value recognition information and simultaneously responds to the voice related to the recognition of the baby's emotion. The sensing is performed by repeating the process of identifying voice recognition-based baby emotion items and learning a combination of baby cry pattern information patterned from the recognized sensing value recognition information and the identified voice recognition-based baby emotion items. The big data-based accuracy and reliability of emotional state determination information can be continuously improved.

Referring to Figure 1, the operation server 100 is a virtual object that registers and stores R (R≥2) virtual object data in the operation D/B to augment the area around the baby's face in correspondence to the baby's emotional state. It includes a management unit 110 and an information matching unit 115 that registers and stores virtual object matching information matching baby emotional state information for each designated emotional item and one or more designated virtual object data in the operation D/B.

The virtual object management unit 110 overlaps the area around the baby's face existing in the video data acquired through the camera module of the camera device 190 with the area around the baby's face in order to augment it in response to the baby's emotional state. Register and store R virtual object data that can be displayed in the operation D/B. For example, the virtual object management unit 110 may register R virtual object data that can be displayed by overlapping the area around the baby's face from a designated operation terminal and store them in the operation database.

According to the implementation method of the present invention, at least some of the virtual object data among the R virtual object data is a sentence string expressing the baby's emotional state corresponding to the baby's emotional state information for each of the n emotional items in the area around the baby's face. It may contain virtual object data displayed in the form of a speech bubble.

The information matching unit 115 determines the baby's emotional state for each designated emotional item generated based on at least one of the facial expression-based emotional state information, crying sound-based emotional state information, and sensing-based emotional state information. Virtual object matching information that matches the information with one or more virtual object data among the R virtual object data is registered and stored in the operation DB. For example, the information matching unit 115 may register virtual object matching information that matches baby emotional state information for each designated emotional item and one or more designated virtual object data from a designated operating terminal, and register and store it in the operating DB. .

According to the implementation method of the present invention, the baby emotional state information for each designated emotional item generated based on at least one emotional state determination information includes numerical information about the baby's emotional state for each designated emotional item, and the virtual object data is the designated emotional state. It can be structured in such a way that it is possible to intuitively express the strength (or intensity) of the emotion corresponding to the numerical information about the baby's emotional state for each emotional item. For example, if the virtual object data is virtual object data in the form of tear stream marks that are displayed overlapping on the 'lower part of both insides of the baby' to enhance the emotional state of 'sadness', the virtual object data is 'sadness'. It may be structured so that the width, length, or color of the tear flow marks varies depending on the numerical information about the baby's emotional state in the emotional item.

Referring to Figure 1, the operation server 100 is provided with a target to receive transmission data including video data captured of a specific baby in the designated base through the camera device 190 provided at the designated base and the camera device ( It includes an information register 120 that registers information mapping 190) and stores it in a designated management DB.

The user who purchases or receives the camera device 190 enters subscription application information through the user terminal used or submits a paper subscription application form (e.g., an application for combining the service using the camera device 190 and IPTV 185, When an application (such as an application combining a service using the camera device 190 and the user wireless terminal 180) is created, the information register 120 provides information corresponding to the subscription application information or subscription application through the user terminal or a designated registration terminal. User information and camera device 190 information that uniquely identifies the camera device 190 provided/supplied to the user can be registered and stored in the management DB. Meanwhile, the information registration unit 120 provides ID/PW information for providing transmission data including video data captured through the camera device 190 to the IPTV 185 used by the user through the user terminal or a designated registration terminal. can be registered and stored in the management DB in connection with the user information and/or camera device 190 information. Meanwhile, according to the implementation method of the present invention, the ID/PW information is used to provide transmission data including video data captured through the camera device 190 to the wired terminal or wireless terminal 180 used by the user through the web. It can be used, and the present invention can also include these embodiments within its scope of rights.

Meanwhile, when the user installs the designated app on the user wireless terminal 180 and runs it for the first time (or before signing up), the information register 120 provides information about the user from the app of the user wireless terminal 180. User information and camera device 190 information that uniquely identifies the camera device 190 provided/supplied to the user can be registered and stored in the management database. Meanwhile, the information registration unit 120 uniquely configures the app of the user wireless terminal 180 to provide transmission data including video data captured through the camera device 190 to the app of the user wireless terminal 180. Unique identification information may be generated or provided from the app of the user wireless terminal 180 and stored in the management DB in connection with the user information and/or camera device 190 information, and the unique identification information may be generated. In one case, it can be provided as an app for the user's wireless terminal 180 and stored in a designated storage area. Meanwhile, the information registration unit 120 provides an ID/PW for providing transmission data including video data captured through the camera device 190 from the app of the user wireless terminal 180 to the IPTV 185 used by the user. Information can be registered and stored in the management DB in connection with the user information and/or camera device 190 information. Meanwhile, according to the implementation method of the present invention, the ID/PW information is used to provide transmission data including video data captured through the camera device 190 to the wired terminal or wireless terminal 180 used by the user through the web. It can be used, and the present invention can also include these embodiments within its scope of rights.

Meanwhile, according to the implementation method of the present invention, the app of the user wireless terminal 180 provides unique information (e.g., , mobile phone number, etc.) can be entered/registered, and in this case, the information register 120 stores the unique information of the other wireless terminal 180 in the management database, and the unique information of the other wireless terminal 180 is stored in the management database. Guide the specified app to be installed on the other wireless terminal 180 through the information and/or check the app installed and executed on the other wireless terminal 180, and uniquely identify the app of the other wireless terminal 180. By generating unique identification information or receiving it from the app of the other wireless terminal 180 and storing it in the management DB in connection with the registered camera device 190 information, the user other than the app of the wireless terminal 180 Transmission data including video data captured through the camera device 190 can be processed so that it can be received through an app of another wireless terminal 180 that is registered through an app of the user wireless terminal 180.

Referring to Figure 1, the operation server 100 is obtained through the communication management unit 125, which manages the communication connection with the camera device 190 provided at the designated base, and the camera module of the camera device 190. It includes a data receiver 130 that receives acquired data including video data encoded in a designated manner, and a data storage unit 135 that stores and manages the received acquired data in a specific storage area of a designated storage medium.

When power is input to the camera device 190 provided at the designated base, the control module of the camera device 190 exchanges one or more designated information with the designated operation server 100 through the communication module according to the designated procedure to establish communication. Requesting, the communication management unit 125 authenticates the validity of the camera device 190 based on the exchange of the specified information and connects communication with the camera device 190.

When communication is connected with the camera device 190, the communication management unit 125 dynamically generates disposable address information for receiving acquired data obtained through the camera device 190 and provides it to the camera device 190. In this case, the control module of the camera device 190 receives the disposable address information, sets the disposable address information as a destination address to transmit the acquired data, and then corresponds the acquired data to the disposable address. It can be sent to the destination address. Here, the disposable address information may include address information composed of dynamically generated parameter values so that the address domain information is the same but parameter values after the domain do not overlap.

According to the implementation method of the present invention, the communication management unit 125 can periodically dynamically generate new disposable address information for the camera device 190 and provide it to the camera device 190, in this case. The control module at 190 receives the new disposable address information, interrupts the process of transmitting the acquired data to the destination address corresponding to the previous disposable address, and simultaneously transmits previous transmission information that transmitted the acquired data to the previous disposable address. After confirming and setting the new disposable address information as a new destination address to which the acquired data will be subsequently transmitted, based on the previous transmission information, the acquired data transmitted to the previous disposable address is transferred to the new destination address corresponding to the new disposable address. It can then be transmitted, and the above process can be repeated periodically. Hereinafter, it is specified that the camera device 190 can transmit acquired data to the disposable address even if not otherwise stated. Meanwhile, according to another implementation method of the present invention, the process of generating or applying disposable address information of the communication management unit 125 can be omitted, and the present invention is not limited thereto.

According to the implementation method of the present invention, when the control module of the camera device 190 transmits acquired data including video data obtained through the camera module and encoded in a specified manner, the data receiver 130 receives the camera module. Receives acquired data including video data acquired through and encoded in a designated manner, and the data storage unit 135 stores the video data of the received acquired data in a specific storage area of a designated storage medium for a certain period of time. It can be managed.

Meanwhile, according to the implementation method of the present invention, the control module of the camera device 190 includes video data obtained through the camera module and encoded in a designated manner and audio data acquired through the sound input unit and encoded in a designated manner. When the acquired data is transmitted, the data receiving unit 130 receives acquired data including video data acquired through the camera module and encoded in a designated manner and audio data acquired through the sound input unit and encoded in a designated manner. , the data storage unit 135 stores the video data and audio data of the received acquisition data in a specific storage area of a designated storage medium and manages them for a certain period of time.

According to the implementation method of the present invention, the control module of the camera device 190 may transmit acquired data in the form of moving image data including the video data and audio data, and in this case, the data receiver 130 may transmit the acquired data to the camera device 190. Acquired data in the form of video data is received from 190, and the data storage unit 135 can store and manage the received video data in a specific storage area of a designated storage medium.

Meanwhile, according to the implementation method of the present invention, the control module of the camera device 190 can transmit acquired data including the video data and the audio data in separate data forms, and in this case, the data receiver 130 Acquired data including video data and audio data are received from the camera device 190, and the data storage unit 135 links the video data and audio data respectively received from the camera device 190. It can be managed by storing it in a specific storage area of a designated storage medium. Preferably, the data storage unit 135 may synchronize the times of the video data and audio data based on the time values (or timestamps) of the video data and audio data and store them in a specific storage area of a designated storage medium. .

Meanwhile, according to the implementation method of the present invention, the control module of the camera device 190 is used in addition to video data obtained through the camera module and encoded in a designated manner and/or audio data acquired through the sound input unit and encoded in a designated manner. , when acquired data further including sensing data sensed through a designated sensor module and encoded in a designated manner is transmitted, the data receiving unit 130 receives acquired data including the sensing data, and the data storage unit 135 ) can store the video data and/or audio data of the received acquisition data and the sensing data in a specific storage area of a designated storage medium and manage them for a certain period of time. Preferably, the data storage unit 135 stores the video data and/or audio data based on the time value (or timestamp) of the video data and/or audio data and the time value (or timestamp) of the sensing data. The time of the sensing data can be synchronized with the time of the sensing data and stored in a specific storage area of a designated storage medium.

Referring to Figure 1, the operation server 100 reads at least one of the acquired data obtained through the camera device 190 at the designated base and provides recognition information of the designated recognition target for determining the baby's emotion. An information recognition unit 140 that recognizes, and on the basis of designated emotional state determination information, identifies pattern information of a designated recognition target that matches the recognized recognition information and a preset standard similarity level or higher, and has a matching relationship with the confirmed pattern information. An information determination unit 145 that determines one or more baby emotion items, and one corresponding to the recognition information by reading a numerical correlation between the pattern information and the baby emotion items based on the matching ratio between the recognition information and the pattern information. An emotional state calculation unit 150 that calculates a baby emotional state value for each emotional item that quantifies the baby emotional state for each emotional item, and baby emotional state information for each emotional item specified using the calculated one or more baby emotional state values. It includes an information generation unit 155 that generates.

According to the first baby emotional state generation embodiment of the present invention, the operation server 100 includes video data stored in a specific storage area of the storage medium through an app of the designated user wireless terminal 180 or IPTV 185. At the time of providing the transmission data, the baby's emotions for each emotion item according to a designated procedure based on the video data included in the transmission data and/or data associated with the video data (e.g., audio data and/or sensing data, etc.) Status information can be generated. Here, the baby's emotional state information for each emotional item may include numerical information (or rating information) about the baby's emotional state for each emotional item.

Meanwhile, according to the second embodiment of the baby emotional state generation of the present invention, the operation server 100 stores the acquired data received from the camera device 190 in a specific storage area of the designated storage medium, and then stores the acquired data received from the camera device 190 in a specific storage area of the designated storage medium. Based on the acquired data stored in the storage area, baby emotional state information for each emotional item can be generated according to a designated procedure and stored in conjunction with the acquired data. In this case, video data stored in a specific storage area of the storage medium can be processed. When provided through an app of the designated user's wireless terminal 180 or IPTV 185, information on the baby's emotional state for each emotional item associated with the video data of the transmitted data can be extracted. Here, the baby's emotional state information for each emotional item may include numerical information (or rating information) about the baby's emotional state for each emotional item.

The information recognition unit 140 reads the designated video data and recognizes f (1≤f≤F) feature points for the facial area of a specific baby within the base, and among the recognized f feature points, the information is associated with the baby's emotional expression. Recognize i (i≥1) baby facial expression recognition information corresponding to the geometric relationship between feature points.

According to the implementation method of the present invention, the designated video data may include at least one of video data periodically selected for each designated playback time unit and video data periodically selected for each designated number of playback frames.

In the case of the first baby emotional state generation embodiment of the present invention, the information recognition unit 140 stores an app of the user wireless terminal 180 or an app of the user wireless terminal 180 corresponding to the unique identification information stored in the management D/B. It is checked whether transmission data including video data stored in a specific storage area of the storage medium is provided to the IPTV 185 identified/authenticated through ID/PW information. If the transmission data is provided to an app of the user wireless terminal 180 or IPTV 185, the information recognition unit 140 reads designated video data among the video data of the transmission data and detects a specific video data in the base. While recognizing f feature points for the facial area, i baby facial expression recognition information corresponding to the geometric relationship between feature points related to the baby's emotional expression among the recognized f feature points can be recognized.

Meanwhile, in the case of the second baby emotional state generation embodiment of the present invention, the information recognition unit 140 stores the acquired data received from the camera device 190 in a specific storage area of the storage medium and then stores the acquired data in a specific storage area of the storage medium. Among the video data stored in the area, designated video data is read, and f feature points for the facial area of a specific baby in the base are recognized. Among the recognized f feature points, i are corresponding to the geometric relationship between feature points associated with the baby's emotional expression. Baby facial expression recognition information can be recognized.

When i baby facial expression recognition information is recognized through the information recognition unit 140, the information determination unit 145 determines the i baby facial expressions based on the facial expression-based emotional state determination information stored in the operation D/B. Confirm e (1≤e≤E) baby facial expression pattern information that matches the recognition information and a preset standard similarity level or higher. If e baby expression pattern information that matches the recognized i baby facial expression recognition information and a preset standard similarity level or higher is confirmed, the information determination unit 145 has a matching relationship with the confirmed e baby facial expression pattern information. Determine n1 (1≤n1≤N1) baby emotion items.

When n1 baby emotion items corresponding to i baby facial expression recognition information are determined through the information determination unit 145, the emotional state calculation unit 150 determines the i baby facial expression recognition information and the e baby facial expression recognition information. Based on the matching ratio between facial expression pattern information, the correlation and numerical relationship between the e baby facial expression pattern information and the n1 baby emotional items are read, and the baby emotional state for each n1 emotional item corresponding to the recognized i baby facial expression recognition information. Calculates the baby's emotional state values for each n1 emotional items in numerical form, and the information generator 155 uses the calculated baby emotional state values for each n1 emotional items to specify n (1≤ Generate baby emotional state information for each n≤N) emotional item. Here, the baby's emotional state information for each emotional item may include numerical information (or rating information) about the baby's emotional state for each emotional item.

Meanwhile, when baby emotional state information for each of the n emotional items is generated after the acquired data is stored in a specific storage area of the storage medium according to the second baby emotional state generation embodiment of the present invention, the information generator 155 may store the baby's emotional state information for each of the generated n emotional items in a specific storage area of the storage medium in connection with the video data.

According to an extended/different embodiment of the present invention, when the crying sound-based emotional state determination information is stored in the operating D/B and video data and audio data are linked and stored in a specific storage area of the storage medium, the information recognition unit (140) can recognize j (j ≥ 1) pieces of baby crying sound recognition information for the specific baby by reading audio data of a certain section associated with the video data.

Meanwhile, when j pieces of baby crying sound recognition information are recognized through the information recognition unit 140, the information determination unit 145 recognizes j based on the crying sound-based emotional state determination information stored in the operation D/B. Confirm c (1≤c≤C) baby cry pattern information that matches the baby cry recognition information and a preset standard similarity level or higher. If c baby cry pattern information that matches the recognized j baby cry recognition information and a preset standard similarity level or higher is confirmed, the information determination unit 145 matches the confirmed c baby cry pattern information. Determine n2 (1≤n2≤N2) baby emotion items with relationships.

When n2 baby emotion items corresponding to j baby cry recognition information are determined through the information determination unit 145, the emotional state calculation unit 150 determines the j baby cry recognition information and the c. Based on the matching ratio between the baby cry pattern information, the correlation and numerical relationship between the c baby cry pattern information and the n2 baby emotion items are read, and the n2 emotions corresponding to the recognized j baby cry recognition information are read. The baby's emotional state values for each n2 emotion items are calculated, which are quantifiable values for the baby's emotional state for each item, and the information generator 155 uses the calculated baby emotional state values for each n2 emotional items (additionally) to specify n designated baby emotional state values. Baby emotional state information can be generated for each emotional item. Here, the baby's emotional state information for each emotional item may include numerical information (or rating information) about the baby's emotional state for each emotional item.

According to the implementation method of the present invention, when the baby emotional state value for each n1 emotion item is already calculated based on the i baby facial expression recognition information, the information generator 155 uses the i baby facial expression recognition information. By combining the baby emotional state values for n1 emotional items calculated with the baby emotional state values for n2 emotional items calculated based on the j baby crying sound recognition information, baby emotional state information for each n designated emotional items is generated. can do. Preferably, the baby emotional state information for each n emotional item may be a union relationship of 'baby emotional state value for n1 emotional items ∩ baby emotional state value for n2 emotional items', or a subset relationship of the union. .

Meanwhile, when generating baby emotional state information for each n emotional item by combining the baby emotional state value for each n1 emotional item and the baby emotional state value for each n2 emotional item, the information generator 155 generates the baby emotional state information for each n1 emotional item. For overlapping emotional items among the n2 emotional items, a specified statistical operation (e.g., average value calculation operation) for each baby emotional state value is assigned to a designated recognition target according to the correlation between the emotional item and the facial expression or cry. Baby emotional state information of the overlapping emotional items can be generated through weighted statistical calculations.

According to an extended/another embodiment of the present invention, the sensing-based emotional state determination information is stored in the operating D/B, and video data and/or audio data and the camera device 190 are stored in a specific storage area of the storage medium. When sensing data sensed through a provided or linked sensor module is linked and stored, the information recognition unit 140 reads the sensing data linked to the video data ??/or audio data and identifies information related to the emotional expression of the specific baby. K (k≥1) pieces of sensing value recognition information for sensing data of a designated sensing target can be recognized.

Meanwhile, when k sensing value recognition information is recognized through the information recognition unit 140, the information determination unit 145 recognizes the k sensing values based on the sensing-based emotional state determination information stored in the operation D/B. Check s (1≤s≤S) sensing value pattern information that matches the value recognition information and a preset standard similarity level or higher. If the s sensing value pattern information that matches the recognized k sensing value recognition information and a preset standard similarity or higher is identified, the information determination unit 145 determines the s sensing value pattern information and the identified s sensing value pattern information. Determine n3 (1≤n3≤N3) baby emotion items.

When n3 baby emotion items corresponding to k sensing value recognition information are determined through the information determination unit 145, the emotional state calculation unit 150 determines the k sensing value recognition information and the s sensing values. Based on the matching ratio between value pattern information, the correlation and numerical relationship between the s sensed value pattern information and the n3 baby emotion items are read to determine the baby emotional state for each n3 emotion items corresponding to the recognized k sensed value recognition information. The baby's emotional state values for each of the n3 emotional items are calculated as numbers, and the information generator 155 uses the calculated baby emotional state values for each of the n3 emotional items to (add) the baby's emotional state for each of the n designated emotional items. Status information can be generated. Here, the baby's emotional state information for each emotional item may include numerical information (or rating information) about the baby's emotional state for each emotional item.

According to the implementation method of the present invention, when the baby emotional state value for each n1 emotion item is already calculated based on the i baby facial expression recognition information, the information generator 155 uses the i baby facial expression recognition information. By combining the baby emotional state values for n1 emotional items calculated with the baby emotional state values for n3 emotional items calculated based on the k sensing value recognition information, baby emotional state information for each n designated emotional items can be generated. You can. Preferably, the baby emotional state information for each n emotional item may be a union relationship of 'baby emotional state value for n1 emotional items ∩ baby emotional state value for n3 emotional items', or a subset relationship of the union. .

Meanwhile, when generating baby emotional state information for each n emotional item by combining the baby emotional state value for each n1 emotional item and the baby emotional state value for each n3 emotional item, the information generator 155 generates baby emotional state information for each n1 emotional item. For overlapping emotion items among the n3 emotion items, a specified statistical operation (e.g., average value calculation operation) for each baby emotional state value is specified according to the correlation between the emotion item and the facial expression or the sensing value of the designated sensing object. Baby emotional state information for the overlapping emotional items can be generated through statistical calculations that apply weights designated to the recognition target.

Meanwhile, according to the implementation method of the present invention, based on the i baby facial expression recognition information, the baby emotional state value for each n1 emotional item and the j baby crying sound recognition information are used to calculate the baby emotional state value for each n2 emotional item. In this case, the information generator 155 generates baby emotional state values for each n1 emotional items calculated based on the i baby facial expression recognition information and n2 emotional items calculated based on the j baby crying sound recognition information. Baby emotional state information for each n designated emotional items can be generated by combining the baby emotional state values for each baby and the baby emotional state values for each n3 emotional items calculated based on the k sensing value recognition information. Preferably, the baby emotional state information for each n emotional item is a union relationship of 'baby emotional state value for n1 emotional items ∩ baby emotional state value for n2 emotional items ∩ baby emotional state value for n3 emotional items', or Or, it may be a subset relationship of the union.

Meanwhile, when generating baby emotional state information for n emotional items by combining the baby emotional state values for n1 emotional items, the baby emotional state values for n2 emotional items, and the baby emotional state values for n3 emotional items, the information The generator 155 performs a designated statistical operation (e.g., average value calculation operation) on each baby's emotional state value for overlapping emotional items among the n1 emotional items, n2 emotional items, and n3 emotional items, or the corresponding emotional items. The baby's emotional state information of the overlapping emotional items can be generated through a statistical operation that assigns a designated weight to the designated recognition object according to the correlation between the facial expression or crying sound or the sensing value of the designated sensing object.

Referring to Figure 1, the operation server 100 generates r (1 ≤ r ≤ R) virtual object data that matches the baby emotional state information for each of the n emotional items generated based on the virtual object matching information. A virtual object confirmation unit 160 that checks, and q (1 ≤ q ≤ r) virtual object data that can be displayed overlapping in the area around the baby's face are overlapped and displayed in the area around the baby's face on the video data, thereby defining the area around the baby's face. An emotional state enhancer 165 for augmenting the baby's emotional state information for each of the n emotional items, and transmitting data including video data augmented to correspond to the baby's emotional state information for each of the n emotional items, to a user wireless terminal ( It includes a data provider 170 that provides data through the app 180 or IPTV 185.

When the baby's emotional state information for n emotional items including numerical information (or rating information) about the baby's emotional state for each emotional item is generated through the information generation unit 155, the virtual object confirmation unit 160 Based on the virtual object matching information, confirms r virtual object data that matches the baby emotional state information for each of the generated n emotional items among the R virtual objects stored in the operation D/B.

When r virtual object data matching the baby's emotional state information for each of the n emotional items are confirmed through the virtual object confirmation unit 160, the emotional state enhancer 165 selects the r virtual object data around the baby's face. Select q virtual object data that can be displayed overlapping in the area. For example, the r virtual object data may include two or more virtual object data that can be displayed overlapping in the same area around the baby's face. In this case, the emotional state enhancer 165 is configured to operate the area around the baby's face according to a designated procedure. q pieces of virtual object data can be selected through the process of selecting which virtual object data to actually overlap among two or more virtual object data that can be displayed overlapping in the same area.

According to the implementation method of the present invention, the q virtual object data is a virtual object that displays a sentence string expressing the baby's emotional state corresponding to the baby's emotional state information for each of the n emotional items in the form of a speech bubble in the area around the baby's face. Can contain data.

When q virtual object data are selected to be overlaid in the area around the baby's face, the emotional state enhancer 165 displays the selected q virtual object data in the area around the baby's face on the video data. The area around the baby's face is augmented to correspond to the baby's emotional state information for each of the n emotional items.

According to the implementation method of the present invention, the emotional state enhancer 165 recognizes the orientation direction of the baby's face on the video data, and adds the q pieces of virtual object data to the area around the baby's face corresponding to the orientation direction of the baby's face. can be displayed overlapping. For example, virtual object data containing a string in the form of a speech bubble can be placed in the direction in which the baby's face is facing and simultaneously displayed in the form of a speech balloon lowering the tail toward the baby's face, thereby allowing the video data to be viewed. The user feels as if the baby is expressing emotions corresponding to the emotional expression strings inside the speech bubble (e.g., strings such as 'I'm hungry', 'I'm annoyed', 'I'm lonely', etc.).

The data provider 170 displays q virtual object data overlaid on the area around the baby's face and transmits transmission data including augmented video data to correspond to the baby's emotional state information for each of the n emotional items to a designated user wireless terminal ( It is provided through an app or IPTV 185 of the user wireless terminal 180, and the app or IPTV 185 of the user wireless terminal 180 can receive and output the transmission data. Preferably, the transmission data may further include audio data in addition to the video data, but the present invention is not limited thereto.

Figure 2 is a diagram illustrating the information storage/registration process according to the implementation method of the present invention.

In more detail, Figure 2 shows a designated recognition target for generating baby emotional state information for each designated emotional item (e.g., baby's facial expression, baby's cry, sensing value obtained by sensing the baby's designated sensing line through a built-in sensor module, etc.) Each emotional state determination information is stored in the operation D/B, or a target to receive transmission data including video data captured of a specific baby in the base through the camera device 190 provided at the designated base and the camera device ( This shows the process of registering information mapping 190) and storing it in a designated management DB. Those skilled in the art will refer to Figure 2 and/or modify the above process. Although various implementation methods can be inferred (e.g., implementation methods in which some steps are omitted or the order is changed), the present invention includes all of the above inferred implementation methods, and the implementation shown in Figure 2 The technical features are not limited to the method alone.

Referring to Figure 2, the operation server 100 uses the matching relationship information between the pattern information of the recognition object designated to be recognized and each baby emotional item in order to determine the emotional state of the baby, and the pattern information of the designated recognition object and each baby. Emotional state determination information for each designated recognition target, including numerical relationship information and a quantified correlation between emotional items, is registered and stored in the designated operation DB and managed (200).

According to the implementation method of the present invention, the operation server 100 provides E baby expression pattern information corresponding to the geometric relationship between feature points associated with the baby's emotional expression among the F feature points present in the baby's facial area and the designated N baby facial expression pattern information. Among the emotion items, it includes matching relationship information between N1 baby emotion items that can be distinguished based on the baby's facial expression patterns, and numerical expression-based correlation numerical relationship information that quantifies the correlation between the E baby facial expression pattern information and the N1 baby emotional items. The facial expression-based emotional state determination information including is registered and stored in the designated operation D/B.

According to an extension/another embodiment of the present invention, the operation server 100 collects C baby cry pattern information related to the baby's emotional expression among the baby cry patterns and baby cry pattern information among the designated N baby emotion items. A crying sound that contains matching relationship information between N2 baby emotion items that can be distinguished based on the basis, and numerical relationship information based on a cry that quantifies the correlation between the C baby cry pattern information and the N2 baby emotion items. The basic emotional state determination information can be registered and stored in the operation DB.

Meanwhile, according to an expansion/another embodiment of the present invention, the operation server 100 detects a designated sensing object related to the baby's emotional expression among the sensing data sensed through a sensor module provided or linked to the camera device 190. Contains matching relationship information between S sensing value pattern information for data and N3 baby emotion items that can be distinguished based on the sensing value pattern of a designated sensing object among the designated N baby emotion items, and the S sensing value pattern information and Sensing-based emotional state determination information, including numerical relationship information and sensing-based correlation that quantifies the correlation between N3 baby emotional items, can be registered and stored in the operation D/B.

Referring to Figure 2, the operation server 100 registers and stores R (R≥2) virtual object data in the operation D/B to augment the area around the baby's face in response to the baby's emotional state (205). , Baby emotional state information for each designated emotional item generated based on at least one of the facial expression-based emotional state discrimination information, crying sound-based emotional state discrimination information, and sensing-based emotional state discrimination information, and one or more designated virtual objects. Virtual object matching information matching data is registered and stored in the operation D/B (210).

Referring to Figure 2, the operation server 100 is configured to receive transmission data including video data captured of a specific baby in the designated base through the camera device 190 provided at the designated base and the camera device 190. ), user information about the user from the app of the user wireless terminal 180, camera device 190 information uniquely identifying the camera device 190 provided/supplied to the user, and the user wireless terminal 180 The unique identification information that uniquely identifies the app and the ID/PW information set by the user may be requested for mapping registration (215) according to a designated procedure, and the operation server 100 may register the app of the user's wireless terminal (180). Receive (220) the user information, camera device 190 information, unique identification information, and ID/PW information from the user according to a designated procedure, and the user information, camera device 190 information, unique identification information, and ID/PW Information, etc. is linked and stored in the designated management Dd/B (225). Meanwhile, depending on the implementation method, the user information, camera device 190 information, ID/PW information, etc. are stored in a user terminal used by the user who purchases or receives the camera device 190 or a designated registration terminal that receives the user's application for membership. It can be registered through .

Figure 3 is a diagram showing how acquired data acquired through the camera device 190 at a designated base is transmitted and stored and managed in a designated storage medium according to the implementation method of the present invention.

In more detail, Figure 3 shows that the acquired data, including video data acquired through the camera module of the camera device 190 provided at the designated base, is transmitted to the operation server 100 and stored in a specific storage area of the designated author media. As a illustration of the management process, those of ordinary skill in the art to which the present invention pertains can refer to Figure 3 and/or modify various implementation methods for the process (e.g., some steps are omitted, or However, the present invention includes all of the above inferred implementation methods, and its technical features are not limited to only the implementation method shown in Figure 3.

Referring to Figure 3, when the power supply for operation is confirmed (300), the camera device 190 establishes a communication connection by exchanging one or more designated information with the designated operation server 100 through a communication module according to a designated procedure. A request is made (305), and when the operation server 100 authenticates the validity of the camera device 190 by exchanging specified information with the camera device 190 (310), a communication connection is made with the camera device 190. (315), and correspondingly, communication is established with the operation server 100 through the camera device 190 (315).

Meanwhile, the operation server 100 dynamically generates disposable address information for receiving acquired data obtained through the camera device 190 (320), and provides the generated disposable address information to the camera device 190. (325), the camera device 190 may receive the disposable address information from the operation server 100 (330), and set the received disposable address information as a destination address to transmit the acquired data (335). ), the above process of generating disposable address information or setting it as a destination address may be repeated periodically (e.g., every 30 seconds).

Meanwhile, the camera device 190 can acquire video data of a baby in a designated base through the camera module (340a), and transmit the acquired data including the video data to the designated destination address (345).

Alternatively, the camera device 190 acquires video data of a baby within a designated base through the camera module (340a), acquires audio data within the base through the sound input unit (340b), and then acquires the video data (340b). Acquired data including audio data can be transmitted to a designated destination address (345).

Alternatively, the camera device 190 acquires video data of a baby within a designated base through the camera module (340a), and acquires sensing data of a designated sensing target of the baby through the sensor module (340c). ), the acquired data including the video data and sensing data can be transmitted to a designated destination address (345).

Alternatively, the camera device 190 acquires video data of a baby within a designated base through the camera module (340a), acquires audio data within the base through the sound input unit (340b), and uses the sensor module. After obtaining sensing data for sensing the baby's designated sensing target (340c), the acquired data including the video data, audio data, and sensing data can be transmitted to the designated destination address (345).

The operation server 100 receives acquired data including at least video data from the camera device 190 (350), and stores the received acquired data (or each data included in the acquired data) in a designated storage medium. It is managed by storing it in a specific storage area (355).

Figure 4 is a diagram illustrating providing transmission data including video data augmented with a baby's emotional state according to a first implementation method of the present invention.

In more detail, Figure 4 shows that baby emotional state information for each n designated emotional items is generated using baby facial expression recognition information recognized by reading designated video data, and then the area around the baby's face on the video data is divided into the n emotional items. It illustrates the process of providing transmission data including video data augmented to correspond to emotional state information to the app of the user wireless terminal 180 or the IPTV 185, which is within the technical field to which the present invention pertains. Those who have this will be able to infer various implementation methods for the above process (e.g., implementation methods in which some steps are omitted or the order is changed) by referring to and/or modifying Figure 4, but the present invention does not apply to the above analogy. It includes all possible implementation methods, and its technical features are not limited to only the implementation methods shown in Figure 4.

Referring to Figure 4, the point at which transmission data including video data stored in a specific storage area of the designated storage medium is provided to the app or IPTV 185 of the designated user wireless terminal 180 through the process of Figure 3, or After acquired data including video data acquired through the camera device 190 is stored in a specific storage area of the storage medium designated through the process of Figure 3, the operation server 100 reads the designated video data and While recognizing f feature points for the facial area of a specific baby, among the recognized f feature points, i baby facial expression recognition information corresponding to the geometric relationship between feature points associated with the baby's emotional expression is recognized (400).

If i baby facial expression recognition information is recognized based on the video data, the operation server 100 recognizes i based on the facial expression-based emotional state determination information stored in the operation D/B through the process of Figure 2. e baby expression pattern information that matches the baby facial expression recognition information and a preset standard similarity level or higher is confirmed (405), and n1 baby emotion items that have a matching relationship with the e baby facial expression pattern information are determined (410).

If n1 baby emotion items corresponding to the recognized i baby facial expression recognition information are determined, the operation server 100 determines the matching ratio between the recognized i baby facial expression recognition information and the e baby facial expression pattern information. As a basis, the correlation and numerical relationship between the e baby facial expression pattern information and the n1 baby emotional items are read, and the n1 emotional items quantify the baby's emotional state for each of the n1 emotional items corresponding to the recognized i baby facial expression recognition information. Calculate the star baby emotional state value (415).

If the baby's emotional state values for each of the n1 emotional items are calculated, the operation server 100 uses the calculated baby's emotional state values for each of the n1 emotional items to display the baby's emotional state for each of the n1 emotional items. Generates emotional state information (420).

If the baby emotional state information for each of the n emotion items is generated, the operation server 100 generates a baby for each of the n emotion items based on the virtual object matching information stored in the operation D/B through the process of Figure 2. Check r(1≤r≤R) virtual object data matching the emotional state information (425), and q(1≤q≤r) virtual object data that can be displayed overlapping in the area around the baby's face. The area around the baby's face is augmented to correspond to the baby's emotional state information for each of the n emotional items by overlapping the display on the area around the face (430), and the video data is augmented to correspond to the baby's emotional state information for each of the n emotional items. The operation server 100 transmits the transmission data containing the app of the user wireless terminal 180 corresponding to the unique identification information stored in the management D/B or the ID/PW stored in the management D/B through the process of Figure 2. Information is provided through identified/authenticated IPTV (185) (435). The app or IPTV 185 of the user wireless terminal 180 receives and outputs transmission data including video data augmented to correspond to the baby emotional state information for each of the n emotional items (440). Preferably, the transmission data may further include audio data associated with the video data.

Figure 5 is a diagram illustrating providing transmission data including video data augmented with a baby's emotional state according to a second implementation method of the present invention.

In more detail, Figure 5 shows that the baby's emotional state information for each n designated emotional items is generated by (additionally) using the baby's cry recognition information recognized by reading the audio data associated with the designated video data, and then generating the baby's emotional state information for each of the n emotional items. This shows a process of providing transmission data including video data augmented by the area around the face to correspond to the baby's emotional state information for each of the n emotional items to an app of the user wireless terminal 180 or an IPTV 185. Those of ordinary skill in the art to which the invention pertains can infer various implementation methods for the above process (for example, implementation methods in which some steps are omitted or the order is changed) by referring to and/or modifying Figure 5. However, the present invention includes all the implementation methods inferred above, and its technical features are not limited to only the implementation method shown in Figure 5.

Referring to Figure 5, the point at which transmission data including video data stored in a specific storage area of the storage medium designated through the process of Figure 3 is provided to the app or IPTV 185 of the designated user wireless terminal 180, or After acquired data including video data acquired through the camera device 190 is stored in a specific storage area of the storage medium designated through the process of Figure 3, or n1 emotions based on baby facial expression recognition shown in Figure 4 In conjunction with the process of calculating the baby's emotional state value for each item, the operation server 100 reads the audio data of a certain section associated with the designated video data and recognizes j baby crying sound recognition information for the specific baby ( 500).

If j pieces of baby cry recognition information are recognized based on the audio data, the operation server 100 recognizes the baby cry based on emotional state determination information stored in the operation D/B through the process of Figure 2. Confirm c baby cry pattern information that matches the j baby cry recognition information and a preset standard similarity level or higher (505), and n2 baby emotion items that have a matching relationship with the confirmed c baby cry pattern information. Determine (510).

If n2 baby emotion items corresponding to the recognized j baby cry recognition information are determined, the operation server 100 determines the relationship between the recognized j baby cry recognition information and the c baby cry pattern information. Based on the matching ratio, the correlation and numerical relationship between the c baby cry pattern information and n2 baby emotion items are read to quantify the baby emotional state for each n2 emotion items corresponding to the recognized j baby cry recognition information. The baby's emotional state value is calculated for each n2 emotional items (515).

If the baby's emotional state values for each of the n2 emotional items are calculated, the operation server 100 uses the calculated baby's emotional state values for each of the n2 emotional items to display the baby's emotional state for each of the n2 emotional items. Emotional state information may be generated (520).

Meanwhile, when the baby emotional state value for each n1 emotional item is calculated through the process of Figure 4 at a point in time before, during, or after the n2 emotional state value is calculated, the operation server 100 By combining the calculated baby emotional state values for each n1 emotional items and the baby emotional state values for each n2 emotional items, baby emotional state information for each n designated emotional items that can be displayed on video data can be generated (520). .

If the baby emotional state information for each of the n emotion items is generated, the operation server 100 generates a baby for each of the n emotion items based on the virtual object matching information stored in the operation D/B through the process of Figure 2. Check r(1≤r≤R) virtual object data matching the emotional state information (525), and q(1≤q≤r) virtual object data that can be displayed overlapping in the area around the baby's face. The area around the baby's face is augmented to correspond to the baby's emotional state information for each of the n emotional items by overlapping the display on the area around the face (530), and the video data is augmented to correspond to the baby's emotional state information for each of the n emotional items. The operation server 100 transmits the transmission data containing the app of the user wireless terminal 180 corresponding to the unique identification information stored in the management D/B or the ID/PW stored in the management D/B through the process of Figure 2. Information is provided through identified/authenticated IPTV (185) (535). The app or IPTV 185 of the user wireless terminal 180 receives and outputs transmission data including video data augmented to correspond to the baby emotional state information for each of the n emotional items (540). Preferably, the transmission data may further include audio data associated with the video data.

Figure 6 is a diagram illustrating providing transmission data including video data augmented with a baby's emotional state according to a third implementation method of the present invention.

In more detail, Figure 6 shows that the baby's emotional state information for each of the n designated emotional items is generated by (additionally) using the sensing value recognition information recognized by reading the sensing data associated with the designated video data and/or audio data. Shows a process of providing transmission data including video data in which the area around the baby's face on the video data is augmented to correspond to the baby's emotional state information for each of the n emotion items to an app of the user wireless terminal 180 or IPTV 185. As such, those of ordinary skill in the technical field to which the present invention pertains can refer to and/or modify Figure 6 to provide various implementation methods for the above process (e.g., an implementation method in which some steps are omitted or the order is changed). ) can be inferred, but the present invention includes all of the above inferred implementation methods, and its technical features are not limited to only the implementation method shown in Figure 6.

Referring to Figure 6, the point at which transmission data including video data stored in a specific storage area of the designated storage medium is provided to the app or IPTV 185 of the designated user wireless terminal 180 through the process of Figure 3, or After acquired data including video data acquired through the camera device 190 is stored in a specific storage area of the storage medium designated through the process of Figure 3, or n1 emotions based on baby facial expression recognition shown in Figure 4 In conjunction with the process of calculating the baby emotional state value for each item, or in conjunction with the process for calculating the baby emotional state value for each n2 emotional items based on baby crying sound recognition shown in Figure 5, the operation server 100 stores the designated video data. And/or the sensing data associated with the audio data is read to recognize k sensing value recognition information for the sensing data of the designated sensing target associated with the emotional expression of the specific baby (600).

If k sensing value recognition information is recognized based on the sensing data, the operation server 100 recognizes the k based on the sensing-based emotional state determination information stored in the operation D/B through the process of Figure 2. Confirm s sensing value pattern information that matches the s sensing value recognition information and a preset standard similarity or higher (605), and determine n3 baby emotion items that have a matching relationship with the confirmed s sensing value pattern information (610). ).

If n3 baby emotion items corresponding to the recognized k sensing value recognition information are determined, the operation server 100 determines a matching ratio between the recognized k sensing value recognition information and the s sensing value pattern information. As a basis, the correlation and numerical relationship between the s sensing value pattern information and the n 3 baby emotional items are read, and the n 3 emotional items corresponding to the recognized k sensing value recognition information are quantified to determine the baby's emotional state. Calculate the star baby emotional state value (615).

If the baby's emotional state values for each of the n3 emotional items are calculated, the operation server 100 uses the calculated baby's emotional state values for each of the n3 emotional items to display the baby's emotional state for each of the designated n emotional items on the video data. Emotional state information can be generated (620).

Meanwhile, when the baby emotional state value for each n1 emotional item is calculated through the process of Figure 4 at a point in time before, during, or after the n3 emotional state values are calculated, the operation server 100 By combining the calculated baby emotional state values for each n1 emotional items and the baby emotional state values for each n3 emotional items, baby emotional state information for each n designated emotional items that can be displayed on video data can be generated (620). .

Or, at a point in time before, during, or after the baby emotional state values for the n3 emotional items are calculated, the baby emotional state values for the n1 emotional items are calculated through the process of Figure 4, and the baby emotional state values for the n1 emotional items are calculated through the process of Figure 5. When the baby emotional state value for each emotion item is calculated, the operation server 100 generates the baby emotional state value for each n1 emotional item, the baby emotional state value for each n2 emotional item, and the baby emotional state value for each n3 emotional item. By combining the state values, baby emotional state information for each n designated emotional items that can be displayed on the video data can be generated (620).

If the baby emotional state information for each of the n emotion items is generated, the operation server 100 generates a baby for each of the n emotion items based on the virtual object matching information stored in the operation D/B through the process of Figure 2. Check r(1≤r≤R) virtual object data matching the emotional state information (625), and q(1≤q≤r) virtual object data that can be displayed overlapping in the area around the baby's face. The area around the baby's face is augmented to correspond to the baby's emotional state information for each of the n emotional items by overlapping the display on the area around the face (630), and the video data is augmented to correspond to the baby's emotional state information for each of the n emotional items. The operation server 100 transmits the transmission data containing the app of the user wireless terminal 180 corresponding to the unique identification information stored in the management D/B or the ID/PW stored in the management D/B through the process of Figure 2. Information is provided through identified/authenticated IPTV (185) (635). The app or IPTV 185 of the user wireless terminal 180 receives and outputs transmission data including video data augmented to correspond to the baby emotional state information for each of the n emotional items (640). Preferably, the transmission data may further include audio data associated with the video data.

100: Operation server 105: Information management department
110: Virtual object management unit 115: Information matching unit
120: Information Register 125: Communication Management Department
130: data receiving unit 135: data storage unit
140: Information recognition unit 145: Information determination unit
150: emotional state calculation unit 155: information generation unit
160: Virtual object confirmation unit 165: Emotional state enhancement unit
170: data provider 180: wireless terminal
185: IPTV 190: Camera device

Claims (28)

In a method executed through an operating server equipped with a sound input unit and a camera module and communicating with a camera device provided at a designated base,
Among the F (F≥2) feature points present in the baby's facial area, E (E≥2) baby expression pattern information corresponding to the geometric relationship between feature points associated with the baby's emotional expression and the designated N (N≥2) baby Matching relationship information between N1 (1≤N1≤N) baby emotion items that can be distinguished based on baby facial expression patterns among emotional items, and expression-based quantification of the correlation between the E baby facial expression pattern information and N1 baby emotional items. Expression-based emotional state discrimination information including correlation and numerical relationship information, C (C≥2) baby cry pattern information related to the baby's emotional expression among the baby's cry patterns, and baby cry among the designated N baby emotion items. Matching relationship information between N2 (1≤N2≤N) baby emotion items that can be distinguished based on sound patterns, and a cry-based correlation chart that quantifies the correlation between the C baby cry pattern information and N2 baby emotion items. S (S ≥ 2) Matching relationship information between sensing value pattern information and N3 (1≤N3≤N) baby emotion items that can be distinguished based on the sensing value pattern of a designated sensing object among the designated N baby emotion items, and the S sensing values Sensing-based emotional state determination information, including numerical relationship information, is registered and stored in the operation D/B, and the area around the baby's face is compared to the baby's emotional state. Register and store R (R≥2) virtual object data for matching and augmentation in operation D/B, and operate virtual object matching information that matches baby emotional state information for each designated emotion item with one or more designated virtual object data. The first step of registering and storing in /B;
Video data encoded in a designated manner is received by filming a baby in the base through a camera module of the camera device, and audio data input through a sound input unit of the camera apparatus and encoded in a designated manner is received, and the received video data and audio are received. Synchronizes the time of the data, stores and manages it in conjunction with the storage area of the designated storage medium, receives sensing data sensed through a sensor module provided or linked to the camera device, and links it with the video data to the storage area of the designated storage medium. A second step of storage management;
By reading the designated video data, f (1≤f≤F) feature points of the baby's facial area within the base are recognized, and among the recognized f feature points, i ( Recognize i≥1) pieces of baby facial expression recognition information, read audio data of a certain section associated with the video data, recognize j(j≥1) pieces of baby cry recognition information for the baby, and recognize the video data. A third step of reading the sensing data associated with and recognizing k (k≥1) sensing value recognition information for the sensing data of the designated sensing target associated with the baby's emotional expression;
Based on the facial expression-based emotional state determination information, identify e (1≤e≤E) baby facial expression pattern information that matches the recognized i baby facial expression recognition information and a preset standard similarity level or higher, and identify the e baby facial expression pattern information. and determine n1 (1≤n1≤N1) baby emotion items that have a matching relationship, and match the j recognized baby cry recognition information with a preset standard similarity level or higher based on the crying sound-based emotional state determination information. Confirm c (1≤c≤C) baby cry pattern information, determine n2 (1≤n2≤N2) baby emotion items that have a matching relationship with the confirmed c baby cry pattern information, and perform the sensing. Based on the emotional state determination information, check s (1≤s≤S) sensing value pattern information that matches the recognized k sensing value recognition information and a preset standard similarity or higher, and obtain the confirmed s sensing value pattern information. A fourth step of determining n3 (1≤n3≤N3) baby emotion items that have a matching relationship with;
Based on the matching ratio between the recognized i baby facial expression recognition information and the e baby facial expression pattern information, the correlation and numerical relationship between the e baby facial expression pattern information and the n1 baby emotion items are read, and the recognized i baby Calculate the baby emotional state value for each n1 emotional item, which quantifies the baby emotional state for each n1 emotional item corresponding to the facial expression recognition information, and calculate the difference between the recognized j baby cry recognition information and the c baby cry pattern information. Based on the matching ratio, the correlation and numerical relationship between the c baby cry pattern information and n2 baby emotion items are read to quantify the baby emotional state for each n2 emotion items corresponding to the recognized j baby cry recognition information. Calculate the baby emotional state value for each n2 emotional items, and calculate the s sensing value pattern information and the n3 baby emotional items based on the matching ratio between the recognized k sensing value recognition information and the s sensing value pattern information. A fifth step of calculating the baby emotional state value for each of the n3 emotional items, which is a numerical value of the baby emotional state for the n3 emotional items corresponding to the recognized k sensing value recognition information, by reading the numerical relationship between correlations;
Baby's emotions for each n (1≤n≤N) emotion items specified using the calculated baby emotional state values for n1 emotional items, baby emotional state values for n2 emotional items, and baby emotional state values for n3 emotional items. A sixth step of generating status information;
A seventh step of confirming r (1≤r≤R) virtual object data matching the baby emotional state information for each of the generated n emotional items based on the virtual object matching information;
Q (1≤q≤r) virtual object data that displays a sentence string expressing the baby's emotional state corresponding to the baby's emotional state information for each of the n emotional items in the form of a speech bubble in the area around the baby's face is displayed on the video data. An eighth step of overlapping display on the area around the baby's face and augmenting the area around the baby's face to correspond to the baby's emotional state information for each of the n emotional items; and
A ninth step of providing transmission data including video data augmented to correspond to the baby's emotional state information for each of the n emotional items to an app or IPTV of the user's wireless terminal; augmenting the baby's emotional state using the area around the baby, including; method.
delete The method of claim 1, wherein the geometric relationship between the feature points is:
A method of enhancing the baby's emotional state using the area around the baby, characterized in that it includes one or more of the following relationships: a distance relationship between each feature point, an angle relationship between each feature point, and a combination relationship of a distance relationship and an angle relationship between each feature point.
The method of claim 1, wherein the facial expression-based correlation numerical relationship information is,
The baby's emotional state using the area around the baby, characterized in that it includes correlation numerical information calculated through a relationship that is proportional to the matching ratio between the baby's facial expression recognition information and the baby's facial expression pattern information that is already registered. Augmentation method.
The method of claim 1, wherein the facial expression-based correlation numerical relationship information is,
A method of enhancing the baby's emotional state using the area around the baby, characterized in that it includes correlation numerical information for each matching ratio section between the baby's facial expression recognition information and the already registered baby's facial expression pattern information.
delete delete delete The method of claim 1, wherein the transmitted data is:
A method of enhancing the baby's emotional state using the area around the baby, further comprising audio data stored in the storage area of the storage medium.
The method of claim 1, wherein the designated video data is:
Video data periodically selected by specified playback time units,
A method of enhancing the emotional state of a baby using the area around the baby, comprising at least one video data among video data periodically selected for a specified number of playback frames.
According to clause 1,
It further includes the step of confirming whether transmission data including video data stored in the storage area of the storage medium is provided to the app or IPTV of the designated user's wireless terminal,
In the third step, when transmission data including video data is provided to an app or IPTV of a designated user wireless terminal, designated video data among the video data of the transmission data is read and f Recognizing the baby's facial expression recognition information corresponding to the geometric relationship between the characteristic points associated with the baby's emotional expression among the recognized f feature points while recognizing the baby's characteristic points. How to enhance your emotional state.
According to clause 1,
The third step is to read designated video data from among the video data stored in the storage area of the designated storage medium, recognize f feature points of the baby's facial area within the base, and identify the baby's emotional expressions among the recognized f feature points. It includes the step of recognizing i baby facial expression recognition information corresponding to the geometric relationship between associated feature points,
The sixth step further includes storing the baby's emotional state information for each of the generated n emotional items in conjunction with the video data.
The method of claim 12, wherein the sixth step is:
Baby using the area around the baby, comprising the step of storing the baby emotional state information for each of the generated n emotional items in association with a time value (or timestamp) synchronized with a designated playback point of the video data. Method for enhancing emotional state.
delete The method of claim 1, wherein the C baby cry pattern information is:
A method of enhancing a baby's emotional state using the area around the baby, comprising information on the range of cry characteristics that make up each baby's cry pattern associated with the baby's emotional expression.
The method of claim 15, wherein the cry characteristic is,
Frequency band feature, placement relationship feature of frequency band feature, periodicity feature of frequency band feature, sound intensity feature, placement relationship feature of sound intensity feature, periodicity feature of sound intensity feature, combination feature of frequency band feature and sound intensity feature, frequency A method for enhancing the baby's emotional state using the area around the baby, comprising one or more of the following features: the arrangement relationship between the band feature and the sound intensity feature, and the periodicity feature of the frequency band feature and the sound intensity feature.
The method of claim 1, wherein the crying sound-based correlation numerical relationship information is,
A baby using the area around the baby, characterized in that it includes correlation numerical information calculated through a relationship proportional to the matching ratio between the baby cry recognition information that recognizes the baby's cry and the already registered baby cry pattern information. Method for enhancing emotional state.
The method of claim 1, wherein the crying sound-based correlation numerical relationship information is,
Augmentation of the baby's emotional state using the area around the baby, characterized in that it includes correlation numerical information for each matching ratio section between the baby's crying sound recognition information and the already registered baby's crying sound pattern information. method.
delete The method of claim 1, wherein the sensor module:
Wearable sensor module that is worn on the baby's body to sense a designated sensing target,
A method of enhancing the baby's emotional state using the area around the baby, comprising at least one sensor module among radar sensor modules that senses the baby's designated sensing target using designated electromagnetic waves.
The method of claim 1 or 20, wherein the sensing object is,
A method of enhancing the baby's emotional state using the area around the baby, comprising at least one of the baby's heartbeat, the baby's body temperature, the humidity around the baby, and whether the baby is apnea.
The method of claim 1, wherein the S sensing value pattern information is:
A method for enhancing a baby's emotional state using the area around the baby, characterized in that it includes range information of sensing value features that constitute each sensing value pattern for sensing data of a designated sensing object associated with the baby's emotional expression.
The method of claim 22, wherein the sensing value characteristic is:
A method of enhancing the baby's emotional state using the area around the baby, comprising at least one or a combination of two or more of the following: a sensing value change rule feature, a sensing value change amount feature, a sensing value change rate feature, and a periodicity feature of the sensing value change. .
The method of claim 1, wherein the sensing-based correlation numerical relationship information is,
Using the area around the baby, characterized in that it includes correlation numerical information calculated through a relational expression proportional to the matching ratio between sensing value recognition information recognized through sensing data of a designated sensing target and pre-registered sensing value pattern information. How to enhance your baby's emotional state.
The method of claim 1, wherein the sensing-based correlation numerical relationship information is,
The baby's emotional state using the area around the baby, characterized in that it includes correlation numerical information for each matching ratio section between sensing value recognition information recognized through sensing data of a designated sensing target and pre-registered sensing value pattern information. Augmentation method.
The method of claim 1, wherein the baby emotional state information for each of the n emotional items is:
A method of enhancing a baby's emotional state using the area around the baby, comprising n emotional item information and numerical information of the baby's emotional state for each emotional item.
delete According to clause 1,
Further comprising recognizing the orientation of the baby's face on the video data,
The eighth step is a method of enhancing the baby's emotional state using the area around the baby, comprising the step of overlapping and displaying the q virtual object data on the area around the baby's face corresponding to the orientation direction of the baby's face. .