KR102239223B1 - Method and system for automatic activation of machine - Google Patents

Abstract

A method and system for automatic activation of a machine is disclosed. A computer-implemented artificial intelligence conversation method includes the steps of: extracting emotion information of a user from information collected while a conversation function is deactivated; And activating the conversation function by using the emotion information as a conversation activation trigger.

Description

METHOD AND SYSTEM FOR AUTOMATIC ACTIVATION OF MACHINE

The description below relates to an artificial intelligence dialogue system.

In general, artificial intelligence conversation systems used in personal assistant systems, chatbot platforms, and artificial intelligence (AI) speakers are a way to understand the intention of a person's command and provide a response phrase corresponding thereto.

Mainly, an artificial intelligence conversation system is a method in which a person delivers a functional request and a machine provides an answer to a person's request. It receives the user's voice input through a microphone and uses the device based on the received voice input. You can control the operation or content provision.

For example, in Korean Patent Publication No. 10-2011-0139797 (published on December 30, 2011), it is possible to provide a home network service using a second communication network such as Wi-Fi in addition to a mobile communication network in the home network service. A technology is disclosed in which a user can multi-control a plurality of multimedia devices in a home through a voice command without a separate button operation.

In general, artificial intelligence conversation systems use predetermined keywords (eg, the name of a machine) as a conversation activation trigger. The machine recognizes when the user calls the machine's name and can activate it in command collection mode, but the machine cannot activate the conversation by itself.

It provides a method and system that can trigger the activation of a machine for artificial intelligence conversation by itself.

Provides a method and system that enables a machine to activate itself by using at least one of a user's facial expression information and voice information, and to initiate a conversation to the user.

A computer-implemented artificial intelligence conversation method, comprising: extracting emotion information of a user from information collected while a conversation function is deactivated; And activating the conversation function by using the emotion information as a conversation activation trigger.

According to an aspect, the extracting may include receiving the user's voice input and extracting the emotion information from the sentence of the voice input.

According to another aspect, in the extracting, the emotion information may be extracted from the facial expression by receiving the user's face image.

According to another aspect, the extracting may include activating an image input device when motion is detected through a motion sensor; Receiving a face image of the user from the image input device; And extracting the emotion information from a facial expression based on the facial image.

According to another aspect, a timer for a motion detection period of the motion sensor may be set.

According to another aspect, the extracting includes: receiving the user's voice input and extracting first emotion information from the sentence of the voice input; Receiving the user's face image and extracting second emotion information from the facial expression; And determining the user's emotion information by synthesizing the first emotion information and the second emotion information.

According to another aspect, the extracting may extract emotion information including types and intensity of emotions from the collected information.

According to another aspect, in the activating step, when the intensity of the emotion included in the emotion information exceeds a set range, the conversation function may be activated.

According to another aspect, the setting range may be set to a value converged through learning about the user's emotion pattern.

According to another aspect, the activating may include determining whether the intensity of the emotion included in the emotion information exceeds a set range; Determining whether context information related to the user satisfies a preset activation condition when the intensity of the emotion exceeds the set range; And activating the conversation function when the context information meets the activation condition.

According to another aspect, the activation condition is set to a value converged through learning about the user's emotion pattern or activity pattern, and the activation condition is one of time, type or size of utterance, type of emotion, and size of motion. At least one condition may be included.

It is combined with a computer to provide a computer program stored in a computer-readable recording medium to execute the artificial intelligence conversation method on the computer.

It provides a computer-readable recording medium, characterized in that a program for executing the artificial intelligence conversation method on a computer is recorded.

A computer-implemented artificial intelligence conversation system, comprising at least one processor implemented to execute a command readable by a computer, wherein the at least one processor determines a user's conversation intention, and a conversation function is deactivated. Provides an artificial intelligence dialogue system, characterized in that the user's emotion information is extracted from the information collected at and the emotion information is used as a dialogue activation trigger to activate the dialogue function.

According to embodiments of the present invention, activation of a machine for artificial intelligence conversation may be triggered by itself using at least one of the user's facial expression information and voice information. Therefore, it does not require a separate keyword or action to activate the machine, and even if the user does not communicate the intention of the conversation first, the machine activates itself according to the user's emotions and can initiate a conversation with the user, further enhancing the emotional connection between the user and the machine. You can increase it.

1 is a diagram illustrating an example of a service environment using a voice-based interface according to an embodiment of the present invention.
2 is a diagram illustrating another example of a service environment using a voice-based interface according to an embodiment of the present invention.
3 is a diagram illustrating an example of a cloud artificial intelligence platform according to an embodiment of the present invention.
4 is a block diagram illustrating an internal configuration of an electronic device and a server according to an embodiment of the present invention.
5 to 7 are flowcharts illustrating an artificial intelligence conversation method according to an embodiment of the present invention.
8 is a flowchart illustrating an example of a conversation activation process according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The artificial intelligence conversation system according to embodiments of the present invention may be implemented through an electronic device that provides an interface that operates based on a conversation with a user. At this time, the artificial intelligence dialogue system can trigger the activation of the machine for dialogue with the user by itself and initiate dialogue with the user first.

The artificial intelligence conversation method according to the embodiments of the present invention may be performed through the above-described electronic device. In this case, a computer program according to an embodiment of the present invention may be installed and driven in the electronic device, and the electronic device may perform an artificial intelligence conversation method according to an embodiment of the present invention according to the control of the driven computer program. have. The above-described computer program may be combined with an electronic device implemented by a computer and stored in a computer-readable recording medium in order to execute an artificial intelligence conversation method on a computer.

1 is a diagram illustrating an example of a service environment using a voice-based interface according to an embodiment of the present invention. In the embodiment of FIG. 1, in a technology for connecting and controlling in-house devices such as a smart home or a home network service, the electronic device 100 providing an interface operating based on voice An example of recognizing and analyzing a voice input "Turn off" received according to an utterance and controlling the light power of the in-house lighting device 120 connected to the electronic device 100 through an internal network in the home is shown.

For example, in-house devices include home appliances such as televisions, personal computers (PCs), peripheral devices, air conditioners, refrigerators, robot cleaners, etc., in addition to the above-described in-house lighting devices 120, and energy consumption such as water, electricity, air conditioning and heating devices. It may include a variety of devices that can be connected and controlled online, such as security devices such as devices, door locks, and surveillance cameras. In addition, the internal network includes wired network technologies such as Ethernet, HomePNA, and IEEE 1394, and wireless network technologies such as Bluetooth, Ultra Wide Band (UWB), ZigBee, Wireless 1394, and Home RF. Can be utilized.

The electronic device 100 may be one of in-house devices. For example, the electronic device 100 may be one of devices such as an artificial intelligence speaker or a robot cleaner provided in the home. In addition, the electronic device 100 may be a mobile device of the user 110 such as a smart phone, a mobile phone, a notebook computer, a terminal for digital broadcasting, a personal digital assistant (PDA), a portable multimedia player (PMP), a tablet PC, and the like. have. As such, the electronic device 100 is not particularly limited as long as it includes a function capable of connecting with in-house devices in order to control in-house devices by receiving a voice input from the user 110. Also, according to an embodiment, the mobile devices of the user 110 described above may be included as in-house devices.

2 is a diagram illustrating another example of a service environment using a voice-based interface according to an embodiment of the present invention. FIG. 2 shows that the electronic device 100 providing an interface operating based on voice recognizes and analyzes the voice input “today weather” received according to the user's utterance from the external server 210 through an external network. It shows an example of acquiring information about today's weather and outputting the acquired information by voice, such as "Today's weather is ...".

For example, external networks include personal area network (PAN), local area network (LAN), campus area network (CAN), metropolitan area network (MAN), wide area network (WAN), broadband network (BBN), and the Internet. May include any one or more of the networks of.

In the embodiment of FIG. 2, the electronic device 100 may be one of the devices in the home or one of the mobile devices of the user 110, and provides a function for receiving and processing the user's 110 voice input and an external network. A device including a function for providing a service or content provided by the external server 210 to the user 110 by accessing the external server 210 through the external server 210 is not particularly limited.

As such, the electronic device 100 according to the embodiments of the present invention is not particularly limited as long as it is a device capable of processing a user command including at least a voice input received according to the user's utterance through a voice-based interface. I can. For example, the electronic device 100 may directly recognize and analyze the user's voice input and perform an operation suitable for the voice input to process a user command. However, according to an embodiment, the user's voice input is recognized or recognized. Processing such as analysis of voice input and synthesis of voice to be provided to the user may be performed through an external platform connected with the electronic device 100.

3 is a diagram illustrating an example of a cloud artificial intelligence platform according to an embodiment of the present invention. 3 shows electronic devices 310, a cloud artificial intelligence platform 320, and a content/service 330.

For example, the electronic devices 310 may refer to devices provided in a home, and may include at least the electronic device 100 described above. These electronic devices 310 or applications (hereinafter referred to as apps) installed and driven on the electronic devices 310 may be linked to the cloud artificial intelligence platform 320 through the interface connect 340. Here, the interface connect 340 may provide developers with software development kits (SDKs) and/or development documents for developing apps installed and driven on the electronic devices 310 or the electronic devices 310. In addition, the interface connect 340 provides an API (Application Program Interface) that allows apps installed and driven on the electronic devices 310 or the electronic devices 310 to utilize functions provided by the cloud artificial intelligence platform 320. Can provide. As a specific example, developers may use a software development kit (SDK) and/or a development document provided by the interface connect 340 to develop a device or app using an API provided by the interface connect 340 to a cloud artificial intelligence platform ( 320) can be used.

Here, the cloud artificial intelligence platform 320 may provide a function for providing a voice-based service. For example, the cloud artificial intelligence platform 320 is a voice processing module 321 for recognizing a received voice and synthesizing a voice to be output, and a vision processing module 322 for analyzing and processing a received image or video. , A conversation processing module 323 for determining an appropriate conversation in order to output an appropriate voice according to the received voice, a recommendation module 324 for recommending a function suitable for the received voice, and a sentence unit through data learning by artificial intelligence It may include various modules for providing voice-based services, such as Neural Machine Translation (NMT, 325), which supports translation of languages to and from.

For example, in the embodiments of FIGS. 1 and 2, the electronic device 100 may transmit the voice input of the user 110 to the cloud artificial intelligence platform 320 using an API provided by the interface connect 340. have. In this case, the cloud artificial intelligence platform 320 may recognize and analyze the received voice input using the modules 321 to 325 described above, and synthesize and provide an appropriate answer voice according to the received voice input, or Can recommend appropriate actions.

In addition, the extension kit 350 may provide a development kit that enables third-party content developers or companies to implement new voice-based functions based on the cloud artificial intelligence platform 320. For example, in the embodiment of FIG. 2, the electronic device 100 may transmit the received voice input of the user 110 to the external server 210, and the external server 210 is Voice input may be transmitted to the cloud artificial intelligence platform 320 through the provided API. In this case, similar to the foregoing, the cloud artificial intelligence platform 320 recognizes and analyzes the received voice input, synthesizes and provides an appropriate answer voice, or provides recommended information on a function to be processed through the voice input to the external server 210 ) Can be provided. As an example, in FIG. 2, the external server 210 may transmit the voice input “today weather” to the cloud artificial intelligence platform 320, and through recognition of the voice input “today weather” from the cloud artificial intelligence platform 320 The extracted keywords "today" and "weather" can be received. In this case, the external server 210 generates text information such as "Today's weather is ..." through the keywords "today" and "weather" and transmits the generated text information to the cloud artificial intelligence platform 320 again. I can. In this case, the cloud artificial intelligence platform 320 may synthesize text information into speech and provide it to the external server 210. The external server 210 may transmit the synthesized voice to the electronic device 100, and the electronic device 100 outputs the synthesized voice "Today's weather ..." through a speaker, from the user 110 The received voice input "weather today" may be processed.

In this case, the electronic device 100 may perform an artificial intelligence conversation method according to an embodiment of the present invention to operate a device or provide content based on a conversation with a user.

4 is a block diagram illustrating an internal configuration of an electronic device and a server according to an embodiment of the present invention. The electronic device 410 of FIG. 4 may correspond to the electronic device 100 described above, and the server 420 may correspond to a single computer device implementing the external server 210 or the cloud artificial intelligence platform 320 described above. I can.

The electronic device 410 and the server 420 may include memories 411 and 421, processors 412 and 422, communication modules 413 and 423, and input/output interfaces 414 and 424. The memories 411 and 421 are computer-readable recording media, and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, a non-destructive large-capacity recording device such as a ROM and a disk drive may be included in the electronic device 410 or the server 420 as a separate permanent storage device separated from the memories 411 and 421. In addition, the memory 411 and 421 stores an operating system and at least one program code (for example, a code for an application installed in the electronic device 410 and driven by the electronic device 410 to provide a specific service). I can. These software components may be loaded from a computer-readable recording medium separate from the memories 411 and 421. Such a separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, software components may be loaded into the memories 411 and 421 through communication modules 413 and 423 rather than a computer-readable recording medium. For example, at least one program is electronic based on a computer program (for example, the above-described application) installed by files provided by the file distribution system for distributing the installation files of the developers or applications through the network 430 It may be loaded into the memory 411 of the device 410.

The processors 412 and 422 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. The instructions may be provided to the processors 412 and 422 by the memories 411 and 421 or the communication modules 413 and 423. For example, the processors 412 and 422 may be configured to execute a command received according to a program code stored in a recording device such as the memories 411 and 421.

The communication modules 413 and 423 may provide a function for the electronic device 410 and the server 420 to communicate with each other through the network 430, and the electronic device 410 and/or the server 420 A function for communicating with an electronic device or other server may be provided. As an example, a request generated by the processor 412 of the electronic device 410 according to a program code stored in a recording device such as the memory 411 is sent to the server 420 through the network 430 under the control of the communication module 213. ) Can be delivered. Conversely, control signals, commands, contents, files, etc. provided under the control of the processor 422 of the server 420 are transmitted through the communication module 223 and the network 430 to the communication module 213 of the electronic device 410. ) Through the electronic device 410. For example, a control signal, command, content, file, etc. of the server 420 received through the communication module 213 may be transmitted to the processor 412 or memory 411, and the content or file may be transmitted to an electronic device ( It may be stored in a storage medium (permanent storage device described above) that 410 may further include.

The input/output interface 414 may be a means for an interface with the input/output device 415. For example, the input device may include a device such as a microphone, a keyboard, or a mouse, and the output device may include a device such as a display or a speaker. As another example, the input/output interface 414 may be a means for interfacing with a device in which input and output functions are integrated into one, such as a touch screen. The input/output device 415 may be configured with the electronic device 410 and one device. In addition, the input/output interface 424 of the server 420 may be a means for interface with a device (not shown) for input or output that is connected to the server 420 or that the server 420 may include.

In addition, in other embodiments, the electronic device 410 and the server 420 may include fewer or more components than the components of FIG. 4. However, there is no need to clearly show most of the prior art components. For example, the electronic device 410 may be implemented to include at least some of the input/output devices 415 described above, or other components such as a transceiver, a global positioning system (GPS) module, a camera, various sensors, a database, etc. May include more. As a more specific example, when the electronic device 410 is a smartphone, in general, an acceleration sensor or a gyro sensor, a motion sensor, a camera module, various physical buttons, a button using a touch panel, an input/output port, and a vibration included in the smartphone Various components, such as a vibrator for, may be implemented to be further included in the electronic device 410.

In the present embodiments, the electronic device 410 may basically include a microphone for receiving a user's voice input and/or a camera for receiving a user's face image as the input/output device 415, and the user's voice input And/or a speaker for outputting a sound such as an answer voice or audio content corresponding to a face image may be further included as the input/output device 415.

The artificial intelligence conversation method according to embodiments of the present invention may be performed by a computer device such as the electronic device 410 described above. In this case, the processor 412 of the electronic device 410 may be implemented to execute a control instruction according to a code of an operating system included in the memory 411 or a code of at least one program. Here, the processor 412 operates the electronic device 410 so that the electronic device 410 performs steps included in the artificial intelligence conversation method to be described below according to a control command provided by the code stored in the electronic device 410. Can be controlled.

5 is a flowchart illustrating an example of an artificial intelligence conversation method according to an embodiment of the present invention.

In step S511, the electronic device 410 may receive a user's voice input through a conversation-based interface while the conversation function is deactivated. For example, the electronic device 410 may receive a voice input according to a user's utterance through a voice input device such as a microphone included in the electronic device 410 or a microphone interlocked with the electronic device 410 as a conversation-based interface. I can. In other words, the electronic device 410 may collect the user's voice information through the voice input device while the conversation function is deactivated.

In step S521, the electronic device 410 may extract emotion information (hereinafter, referred to as “sentence emotion information”) from the sentence of the voice input received in step S511. In this case, the sentence emotion information may include an emotion type and an emotion intensity (feeling degree). Terms representing emotions, that is, emotion terms, are determined in advance and are classified into a plurality of emotion types (e.g., joy, sadness, surprise, trouble, suffering, anxiety, fear, disgust, anger, etc.) according to a predetermined criterion. According to, it may be classified into a plurality of strength classes (eg, 1 to 10). For example, the electronic device 410 may convert a user's voice input into a text-type sentence through speech to text (STT) conversion. Further, the electronic device 410 may extract a morpheme from a sentence corresponding to a voice input, and then extract a predetermined emotion term from the extracted morpheme, and classify an emotion type and an emotion intensity corresponding to the extracted emotion term. When a plurality of emotion terms are included in a sentence of the voice input, the electronic device 410 may calculate a weight according to the emotion type and the emotion intensity to which the emotion term belongs. Through this, the emotion vector for the emotion information of the sentence is calculated and the corresponding sentence It is possible to extract the emotional information representing the. The technique for extracting the sentence emotion information is illustrative and is not limited thereto, and other well-known techniques may also be used.

In step S530, the electronic device 410 may check whether the intensity of the emotion included in the sentence emotion information exceeds a predetermined setting range by using the sentence emotion information extracted in step S521. The setting range for the intensity of emotion may be set in advance to a specific value by the user or the system, or may be determined as a value converged through a learning process for the user's emotion pattern.

In step S540, the electronic device 410 may automatically activate the conversation function when the intensity of the emotion included in the sentence emotion information exceeds a set range. The electronic device 410 may activate a conversation by itself when an emotion exceeding a set range is detected from a user's voice input.

Unlike the conventional method in which the conversation is activated only when the user first communicates the conversation intention through a specific keyword or action, in this embodiment, the user's emotional intensity may be used as a conversation activation trigger in place of a specific keyword or action. For example, the types of emotions are divided into joy, sadness, surprise, anxiety, suffering, anxiety, fear, disgust, anger, etc., and the intensity of emotion can be divided into a plurality of grades (e.g., 1-10). . The electronic device 410 may set the level of emotion in a predetermined range, for example, in a stable range from 1 to 6. In addition, the electronic device 410 may automatically activate a conversation when the user's emotion appears as an intensity exceeding a stable range, for example, [sad, 8], [anger, 9], [joy, 7], and the like. At this time, the electronic device 410 is included in the electronic device 410 when the conversation is activated, or information through an output device that can be interlocked with the electronic device 410, for example, turning on the display or Sound effect can be output.

In step S550, the electronic device 410 may automatically activate a conversation with respect to a user's emotion exceeding a set range, and then output conversation information corresponding to the sentence emotion information. The conversation information is information that can be output through a conversation-based interface, and may include at least one of a voice message, facial expression information, and motion information. For example, the electronic device 410 may output a voice message according to conversation information through an audio output device such as a speaker included in the electronic device 410 or a speaker interlocked with the electronic device 410. In addition, the electronic device 410 may output facial expression information according to conversation information through an image output device such as a display included in the electronic device 410 or a display interlocked with the electronic device 410. For example, the image output device may display a predetermined character corresponding to a conversation-based interface, and may reflect facial expression information corresponding to the user emotion exceeding a set range to the corresponding character. In addition, the electronic device 410 may output motion information according to conversation information through a motor control operating device included in the electronic device 410 or a motor control operating device interlocked with the electronic device 410. For example, when the electronic device 410 is a conversation robot, an operation related to the corresponding conversation content may be implemented according to conversation information corresponding to the user's emotion. If the user's emotion appears as [Sad, 8] that exceeds the setting range, the robot's sad gesture and expression can be output and a voice message such as <Are there any bad things?> can be output together.

Therefore, in this embodiment, the type and intensity of the user's current emotion can be grasped from the user's voice input, and the user emotion can be used as a conversation activation trigger to automatically activate the conversation when the intensity of the emotion exceeds the set range. have.

6 is a flowchart illustrating another example of an artificial intelligence conversation method according to an embodiment of the present invention.

In step S612, the electronic device 410 operates within a certain distance (eg, 3m) using a motion sensor (eg, an infrared sensor) that is included in the electronic device 410 or can be interlocked with the conversation function disabled. Can be detected. In other words, the electronic device 410 may detect a user's motion through a motion sensor such as an infrared sensor. If motion detection occurs too often, it may cause unnecessary current use, and so on, the electronic device 410 may apply a motion detection period by setting a timer for the motion sensor. For example, the electronic device 410 may deactivate the motion sensor for 12 hours after motion is detected by the motion sensor, and activate it again after 12 hours elapse.

In step S613, when a motion is detected, the electronic device 410 activates an image input device such as a camera included in the electronic device 410 or a camera interlocked with the electronic device 410, Face images can be received. In other words, when a user motion is detected while the conversation function is deactivated, the electronic device 410 may activate the camera and collect face information of the user whose motion is detected through the camera.

In step S622, the electronic device 410 may extract emotion information (hereinafter referred to as “face emotion information”) from the face image received in step S613. The electronic device 410 may extract facial emotion information including an emotion type and an emotion intensity from a facial expression based on an image. Facial expressions are expressed by contraction of facial muscles that occur when facial elements such as eyebrows, eyes, nose, mouth, and skin are deformed, and the intensity of facial expressions can be determined according to geometric changes in facial features or density of muscle expressions. have. As an example, the electronic device 410 extracts a region of interest (eg, an eye region, an eyebrow region, a nose region, a mouth region, etc.) for extracting a feature according to an expression, and then extracts a feature point from the region of interest. Can be used to determine a certain feature value. The feature value corresponds to a specific value representing a person's facial expression based on the distance between feature points, etc. In order to apply the determined feature value to the emotion sensitive value model, the electronic device 410 determines a certain intensity value according to the degree of the value for the feature value displayed in the image, and uses a mapping table prepared in advance to apply the determined value to the value of each specific value. Determine the matching constant intensity value. The mapping table is prepared in advance according to the emotion sensitivity model. The electronic device 410 may map the emotion sensitive value model and the intensity value, and extract the determined emotion type and intensity according to a result of applying the corresponding intensity value to the emotion sensitive value model. The technique for extracting the facial emotion information is illustrative and is not limited thereto, and other well-known techniques may also be used.

In step S630, the electronic device 410 may check whether the intensity of the emotion included in the facial emotion information exceeds a predetermined setting range by using the facial emotion information extracted in step S622.

In step S640, the electronic device 410 may automatically activate the conversation function when the intensity of the emotion included in the facial emotion information exceeds a set range. The electronic device 410 may activate a conversation by itself when an emotion exceeding a set range is detected from the user's face image.

Since the details of automatically activating the conversation based on the intensity of emotion are the same as those described with reference to FIG. 5, detailed descriptions will be omitted.

In step S650, the electronic device 410 may automatically activate a conversation for a user emotion exceeding a set range, and then output conversation information corresponding to the facial emotion information. The conversation information may include at least one of a voice message, facial expression information, and motion information, and the electronic device 410 includes at least one of an audio output device such as a speaker, an image output device such as a display, and a motor control operation device such as a robot. You can output conversation information by using. Details of outputting conversation information are also the same as those described with reference to FIG. 5, and thus detailed descriptions will be omitted.

Therefore, in this embodiment, the type and intensity of the user's current emotion can be grasped from the expression in the user's face image, and the user emotion is used as a conversation activation trigger to automatically activate the conversation when the intensity of the emotion exceeds the set range. I can make it.

7 is a flowchart illustrating another example of an artificial intelligence conversation method according to an embodiment of the present invention.

In step S711, the electronic device 410 may receive a user's voice input through a conversation-based interface while the conversation function is deactivated. For example, the electronic device 410 may receive a voice input according to a user's utterance through a voice input device such as a microphone included in the electronic device 410 or a microphone interlocked with the electronic device 410 as a conversation-based interface. I can.

In step S712, the electronic device 410 may detect a motion within a certain distance (eg, 3m) using a motion sensor (eg, an infrared sensor) included in the electronic device 410. In other words, the electronic device 410 may detect a user's motion through a motion sensor such as an infrared sensor.

In step S713, when a motion is detected, the electronic device 410 may activate a conversation-based interface and receive a face image of the user through the interface. For example, when a user's motion is detected, the electronic device 410 may receive a face image of the user by activating an image input device such as a camera included in the electronic device 410 or a camera interlocked with the electronic device 410. have.

In step S721, the electronic device 410 may extract emotion information (ie, sentence emotion information) from the sentence of the voice input received in step S711. In this case, the sentence emotion information may include an emotion type and an emotion intensity (feeling degree). Since the method of extracting the sentence emotion information is the same as in step S521 described with reference to FIG. 5, a detailed description will be omitted.

In step S722, the electronic device 410 may extract emotion information (ie, facial emotion information) from the face image received in step S713. The electronic device 410 may extract facial emotion information including an emotion type and an emotion intensity from a facial expression based on an image. Since the method of extracting facial emotion information is the same as in step S622 described with reference to FIG. 6, a detailed description will be omitted.

In step S723, the electronic device 410 determines emotion information (hereinafter, referred to as'user emotion information') by combining the sentence emotion information extracted in step S721 and the facial emotion information extracted in step S722. can do. For example, the electronic device 410 may calculate a weight according to sentence emotion information and face emotion information, and determine comprehensive emotion information through this. User emotion information may also include emotion type and emotion intensity. Accordingly, the electronic device 410 may determine the final user emotion by synthesizing the emotion expressed in the sentence according to the user's voice input and the emotion expressed in the facial expression of the user's face image.

In step S730, the electronic device 410 may check whether the intensity of the emotion included in the user emotion information exceeds a predetermined setting range by using the user emotion information extracted in step S723.

In step S740, the electronic device 410 may automatically activate the conversation function when the intensity of the emotion included in the user emotion information exceeds a set range. The electronic device 410 may activate a conversation by itself when the strength of the determined emotion exceeds a set range by synthesizing the emotion indicated by the user's voice input and the emotion indicated by the facial expression of the facial image.

Since the details of automatically activating the conversation based on the intensity of emotion are the same as those described with reference to FIG. 5, detailed descriptions will be omitted.

In operation S750, the electronic device 410 may automatically activate a conversation for a user emotion exceeding a set range, and then output conversation information corresponding to the user emotion information. The conversation information may include at least one of a voice message, facial expression information, and motion information, and the electronic device 410 includes at least one of an audio output device such as a speaker, an image output device such as a display, and a motor control operation device such as a robot. You can output conversation information by using. Details of outputting conversation information are also the same as those described with reference to FIG. 5, and thus detailed descriptions will be omitted.

Therefore, in this embodiment, the type and intensity of the user's current emotion can be grasped by using the user's voice input and the face image together, and when the intensity of the emotion exceeds the set range, the user's emotion is used as a conversation activation trigger. Can be activated automatically.

Furthermore, in the present embodiments, it is also possible to use user emotion as a conversation activation trigger, but to apply context information related to the user as an additional activation condition.

8 is a flowchart illustrating an example of a conversation activation process according to an embodiment of the present invention.

In step S830, the electronic device 410 may check whether the intensity of the emotion determined from the user's voice input and/or the face image exceeds a predetermined setting range.

In step S831, when the determination result in step S830 that the intensity of emotion exceeds a set range, the electronic device 410 may determine whether context information related to the user satisfies a preset activation condition. The activation condition may include at least one of various situational conditions such as time, type or size of utterance, type of emotion, size of motion, and so on. The above-described activation condition may be set in advance to a specific value by the user or the system, or may be determined as a value converged through a learning process for the user's emotion pattern or activity pattern.

In step S840, the electronic device 410 may automatically activate the conversation function when the user's context information satisfies the activation condition. As an example, the electronic device 410 may use a time in which the deactivation state of the conversation function is continuously maintained as an activation condition, and when the maintenance time in the deactivation state exceeds the time set as the activation condition, the intensity exceeding the set range You can automatically activate the conversation about your emotions. As another example, the electronic device 410 may use a user's usual activity time as an activation condition, and may automatically activate a conversation when an emotion having an intensity exceeding a set range is detected at a time other than the user's usual activity time. . As another example, the electronic device 410 may use the user's usual activity time and the size of sigh as an activation condition, and when a sigh of more than a set size is detected at a time other than the user's usual activity time, the conversation may be automatically activated. have. As another example, the electronic device 410 may use the type of emotion normally displayed to the user as an activation condition, and automatically activate the conversation when the emotion having an intensity exceeding the set range does not correspond to the user's usual emotion type. have. As another example, the electronic device 410 may use the size of the motion as an activation condition, and automatically initiate a conversation when the size of the motion detected through the motion sensor exceeds the set range along with the emotion of the intensity exceeding the set range. Can be activated.

Accordingly, in the present embodiment, an emotion having an intensity exceeding the set range is used as a conversation activation trigger, but various context information such as time, type or size of utterance, type of emotion, size of motion, etc. can be applied as an additional activation condition. .

As described above, according to embodiments of the present invention, activation of a machine for artificial intelligence conversation may be triggered by itself using at least one of the user's facial expression information and voice information. Therefore, it does not require a separate keyword or action to activate the machine, and even if the user does not communicate the intention of the conversation first, the machine activates itself according to the user's emotions and can initiate a conversation with the user, further enhancing the emotional connection between the user and the machine. You can increase it.

The apparatus described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices and components described in the embodiments include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and a programmable gate array (PLU). It may be implemented using one or more general purpose or special purpose computers, such as a logic unit), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. Further, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be embodyed in any type of machine, component, physical device, computer storage medium or device to be interpreted by the processing device or to provide instructions or data to the processing device. have. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. In this case, the medium may continue to store a program executable by a computer or temporarily store a program for execution or download. Further, the medium may be a variety of recording means or storage means in a form in which a single or several pieces of hardware are combined, but is not limited to a medium directly connected to a computer system, but may be distributed on a network. Examples of media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, And there may be ones configured to store program instructions, including ROM, RAM, flash memory, and the like. In addition, examples of other media include an app store that distributes applications, a site that supplies or distributes various software, and a recording medium or a storage medium managed by a server.

Although the embodiments have been described by the limited embodiments and drawings as described above, various modifications and variations can be made from the above description to those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as systems, structures, devices, circuits, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and those equivalent to the claims also fall within the scope of the claims to be described later.

Claims (5)

In the artificial intelligence conversation method implemented by a computer,
Extracting emotion information of a user from information collected in a state in which the conversation function is deactivated; And
Activating the conversation function by using the emotion information as a conversation activation trigger
Including,
The activating step,
Determining whether the intensity of the emotion included in the emotion information extracted while the conversation function is deactivated exceeds a set range;
Determining whether context information related to the user satisfies a preset activation condition when the intensity of the emotion exceeds the set range; And
Activating the conversation function for an emotion having an intensity exceeding the set range when the context information meets the activation condition
Including,
The setting range is set to a value converged through learning about the user's emotion pattern,
The activation condition includes at least one of time, type or size of utterance, type of emotion, and size of motion,
The activating step,
Activating the conversation function when the type of emotion included in the emotion information does not correspond to the type of emotion normally displayed to the user and the intensity of the emotion included in the emotion information exceeds the setting range
Artificial intelligence conversation method comprising a. The method of claim 1,
The activating step,
Activating the conversation function for an emotion having an intensity exceeding the setting range when the retention time of the inactive state of the conversation function exceeds a set time
Artificial intelligence conversation method further comprising a. The method of claim 1,
The activating step,
Activating the conversation function when an emotion having an intensity exceeding the set range is detected at a time other than the user's usual activity time
Artificial intelligence conversation method further comprising a. The method of claim 1,
The activating step,
Activating the conversation function when a sigh of more than a set size is detected at a time other than the user's usual activity time
Artificial intelligence conversation method further comprising a. The method of claim 1,
The activating step,
Activating the conversation function when the magnitude of the motion detected through the motion sensor exceeds the set range, together with the emotion of the intensity exceeding the set range.
Artificial intelligence conversation method further comprising a.

Images