KR20200006738A - Dialogue system, and dialogue processing method - Google Patents

Abstract

Provided are a dialogue system which correctly grasps a user′s intention based on various kinds of information, such as a user′s dialogue, vehicle condition information, driving environment information, user information and the like, in a vehicle driving environment to provide a service fit to the user′s intention or the most necessary service to the user, a vehicle including the same, and a dialogue processing method thereof. According to one embodiment of the present invention, the dialogue system comprises: a storage unit storing context information including at least one among vehicle condition information related to a condition of a vehicle, user information related to a driver of the vehicle, and driving environment information related to a driving environment of the vehicle; and an input processor determining whether to be in a first speaking situation for determining the depression degree of the user based on the context information, acquiring an action for performing first speaking when the first speaking situation is determined, and determining the depression degree of the user based on a user′s response to the first speaking.

Description

Dialog system and how to handle it {DIALOGUE SYSTEM, AND DIALOGUE PROCESSING METHOD}

The disclosed invention relates to a conversation system for identifying a user's intention through a conversation with a user and providing information or services required by the user, and a conversation processing method.

In a vehicle AVN or most mobile devices, a small screen and a small button may cause inconvenience in providing visual information to a user or receiving a user input.

In particular, in order for a user to check visual information or to operate a device while driving, moving eyes and releasing the steering wheel pose a threat to safe driving.

Therefore, it is expected to provide a safer and more convenient service when a conversation system that grasps a user's intention through a dialogue with a user and provides a service required by the user is applied to a vehicle.

Korean Laid-Open Patent Document 2013-0035983

One aspect of the disclosed invention is to accurately grasp the user's intention based on various information such as dialogue with the user and vehicle status information, driving environment information, user information, etc. Provides a chat system and a chat processing method that can provide the services most needed by the user.

Another aspect of the disclosed invention is to determine the degree of depression of the user on the basis of various information such as dialogue with the user and vehicle status information, driving environment information, user information in the vehicle driving environment, it is possible to prevent an accident caused by depression It provides a conversation system and a method of processing a conversation.

As a technical means for achieving the above-described technical problem, the dialogue system according to an aspect includes at least one of vehicle state information related to a state of a vehicle, user information related to a driver of the vehicle, and driving environment information related to a driving environment of the vehicle. A storage unit for storing the situation information including one; And determining whether the user is in a starting situation for determining a degree of depression of the user based on the situation information, and when determined to be the starting situation, acquiring an action for performing the starting process, and the user for the selection. It includes; an input processor for determining the degree of depression of the user based on the response.

The input processor may further include a tone analyzer configured to receive a voice of a user and to perform tone analysis on the input voice of the user, wherein the user information of the context information may include the tone analysis result. have.

The storage unit may store at least one question for determining a degree of depression of the user, at least one answer thereto, and at least one depression index corresponding to the at least one answer. When the selection condition is satisfied, an action for performing the selection based on the stored at least one question may be obtained.

The input processor may determine the degree of depression of the user based on the stored at least one response of the user's response to the selection based on the stored at least one question.

The input processor may acquire the depression index of the user based on the at least one depression index corresponding to the stored at least one answer, and determine the degree of depression of the user based on the obtained depression index. have.

The apparatus may further include a risk situation manager configured to determine whether the user is in a dangerous situation based on the degree of depression of the user and the situation information, and obtain an action corresponding to the dangerous situation.

The storage unit may store an accident response manual including at least one action corresponding to a dangerous situation in which an accident due to depression is expected, and the risk situation manager is configured to perform the dangerous situation based on the stored accident response manual. An action corresponding to may be obtained.

In addition, if it is determined that the risk situation manager is the dangerous situation, the risk situation manager may determine a risk situation step and obtain an action corresponding to the risk situation based on the determined risk situation step.

The apparatus may further include a conversation manager configured to obtain at least one candidate action related to the action acquired by the input processor from the storage.

The storage unit may store vehicle function information related to a supportable function of the vehicle, and the conversation manager determines whether to perform the at least one candidate action based on the stored vehicle function information, and determines whether to perform the at least one candidate action. The priority of the at least one candidate action may be determined based on the result.

According to another aspect of the present invention, there is provided a method for processing a conversation, the method comprising: storing situation information including at least one of vehicle state information related to a state of a vehicle, user information related to a driver of the vehicle, and driving environment information related to a driving environment of the vehicle; Determining whether the user is in a starting situation for determining a degree of depression of the user based on the situation information; If it is determined that the starting situation is obtained, acquire an action for performing the starting method; And determining a degree of depression of the user based on the user's response to the selection.

The method may further include receiving a voice of the user and executing tone analysis on the input voice of the user, wherein the user information of the context information may include the tone analysis result.

The method may further include storing at least one question, at least one answer thereto, and at least one depression index corresponding to the at least one answer, for determining a degree of depression of the user. If it is determined that the action to perform the starter can be obtained, the action for performing the starter based on the stored at least one question when the starter condition is satisfied.

In addition, determining the degree of depression of the user based on the user's response to the selection, the user's response to the selection based on the stored at least one question based on the stored at least one response And determining the degree of depression of the user.

The determining of the degree of depression of the user based on the response of the user to the selection may include obtaining the depression index of the user based on at least one depression index corresponding to the stored at least one answer, And determining the degree of depression of the user based on the obtained depression index.

In addition, based on the degree of depression of the user and the situation information determines whether or not a dangerous situation; And acquiring an action corresponding to the dangerous situation when determined to be the dangerous situation.

The method may further include storing an accident response manual including at least one action corresponding to a dangerous situation in which an accident due to depression is expected to occur, and if determined to be a dangerous situation, take an action corresponding to the dangerous situation. Acquiring may include acquiring an action corresponding to the dangerous situation based on the stored accident response manual.

In addition, acquiring an action corresponding to the dangerous situation when determined to be the dangerous situation may include determining a dangerous situation step if determined to be the dangerous situation and based on the determined dangerous situation step, It may include acquiring.

The method may further include acquiring at least one candidate action related to the obtained action.

And storing vehicle function information related to the supportable function of the vehicle; And determining whether to perform the at least one candidate action based on the stored vehicle function information, and determining the priority of the at least one candidate action based on the determination result.

According to the dialogue system and the dialogue processing method according to an aspect, the user's intention is accurately determined by accurately understanding the user's intention based on various information such as dialogue with the user and vehicle status information, driving environment information, and user information in the vehicle driving environment. It can provide a service that meets the intention or a service that is most needed by the user.

According to the dialogue system and the dialogue processing method according to another aspect, it is possible to determine the degree of depression of the user, it is possible to prevent the accident caused by depression by providing a function corresponding thereto.

1 is a control block diagram of a conversation system according to an embodiment.
2 is a diagram illustrating a configuration of a vehicle interior.
3 to 5 illustrate examples of conversations that can be exchanged between the conversation system and the driver.
6 and 7 are control block diagrams schematically illustrating the connection relationship between the dialogue system and the components of the vehicle.
8 and 9 are control block diagrams schematically illustrating the connection relationship between the components of the dialogue system and the components of the vehicle.
10 is a control block diagram of a vehicle standalone system in which a conversation system is provided in a vehicle.
11 and 12 are control block diagrams for a vehicle gateway method in which a conversation system is provided in a remote server and the vehicle serves only as a gateway connecting the user and the conversation system.
FIG. 13 is a control block diagram of a case in which a vehicle may perform some input processing and output processing in a vehicle gateway method.
FIG. 14 is a control block diagram of a hybrid scheme in which both a remote conversation system server and a vehicle can perform conversation processing.
15 and 16 are control block diagrams of a mobile gateway method in which a mobile device connected to a vehicle connects a user with a remote conversation system server.
17 is a control block diagram of a mobile standalone system in which a dialogue system is mounted on a mobile device.
18, 19A, and 19B are control block diagrams in which the configuration of the input processor of the configuration of the conversation system is subdivided.
20A and 20B illustrate examples of information stored in a situation understanding table.
FIG. 21 is a control block diagram of a conversation system applicable when the conversation system first outputs a utterance before a user input is received.
22A, 22B, 22C, 22D, 22E and 22F are diagrams showing examples of information stored in the starting condition table.
23 is a control block diagram of the configuration of the conversation manager.
24 is a diagram illustrating an example of information stored in the association action DB.
25 is a diagram illustrating an example of information stored in an action execution condition DB.
26 is a diagram illustrating an example of information stored in an action factor DB.
27 is a diagram illustrating an example of information stored in the ambiguity resolution information DB.
28A and 28B are tables listing various examples in which the ambiguity resolver resolves ambiguities and extracts actions to control a vehicle by referring to the ambiguity resolution information DB.
Fig. 29 is a control block diagram of the configuration of the result processor.
30 to 42 are diagrams illustrating specific examples in which a conversation system processes an input, manages a conversation, and outputs a result when a user inputs a speech related to a road guidance.
43 is a flowchart illustrating a method of processing user input in a conversation processing method according to an embodiment.
44A and 44B are flowcharts illustrating a method of managing a conversation using an output of an input processor according to an embodiment.
45 is a flowchart illustrating a result processing method for generating a response corresponding to a result of conversation management in the conversation processing method according to an exemplary embodiment.
46 to 48 are flowcharts illustrating a case in which a conversation system outputs a selection prior to a user's speech input in a conversation processing method according to an exemplary embodiment.
49 is a flowchart illustrating a process of processing a duplicate task when the conversation system outputs a selection prior to inputting a speech by a user according to an embodiment.

Like reference numerals refer to like elements throughout. This specification does not describe all elements of the embodiments, and overlaps between the general contents or the embodiments in the technical field to which the present invention belongs. As used herein, the term 'part, module, member, block' may be implemented in software or hardware. According to embodiments, a plurality of 'part, module, member, block' may be embodied as one component. It is also possible that one 'part, module, member, block' includes a plurality of components.

Throughout the specification, when a part is said to be "connected" with another part, it includes not only a case where the part is directly connected, but also an indirect connection, and the indirect connection includes connecting through a wireless communication network. do.

In addition, when a part is said to "include" a certain component, this means that it may further include other components, except to exclude other components unless otherwise stated.

Singular expressions include plural expressions unless the context clearly indicates an exception.

The symbols attached to the steps are used to identify the steps. These signs do not indicate the order between the steps. Each step is performed in a different order than the stated order unless the context clearly indicates a specific order. Can be.

Hereinafter, an embodiment of a conversation system, a vehicle including the same, and a conversation processing method will be described in detail with reference to the accompanying drawings.

According to an exemplary embodiment, a conversation system is a device that grasps a user's intention by using a voice and a non-voice input of a user and provides a service suitable for the user's intention or a service required by the user. As a means of clarifying the problem, the system utterance can be output to communicate with the user.

The services provided to the user in this embodiment are all operations performed to meet the needs of the user or the intention of the user, such as providing information, controlling the vehicle, executing audio / video / navigation functions, and providing content obtained from an external server. It may include.

In addition, the conversation system according to an exemplary embodiment may accurately understand a user's intention in a special environment of a vehicle by providing a conversation processing technology specialized for a vehicle environment.

The gateway connecting the conversation system and the user may be a vehicle or a mobile device connected to the vehicle. As will be described later, the conversation system may be provided in a vehicle, or may be provided in a remote server outside the vehicle to exchange data through communication with the vehicle or a mobile device connected to the vehicle.

In addition, some of the components of the conversation system may be provided in the vehicle and some may be provided in the remote server to partially perform operations of the conversation system in the vehicle and the remote server.

1 is a control block diagram of a conversation system according to an embodiment.

Referring to FIG. 1, the conversation system 100 according to an exemplary embodiment may include an input processor 110 that processes a user input including a voice and a non-speech input of a user, an input related to a vehicle, or an input including information related to a user. ), Using the processing result of the input processor 110 to determine the user's intention or the state of the vehicle, and determine the action corresponding to the user's intention or the state of the vehicle. The result processor 130 for outputting a system utterance for providing a specific service or continuing the conversation according to the output result, and the storage unit 140 for storing various information necessary for performing the operation described later by the dialog system 100. It includes.

The input processor 110 may receive two types of inputs, a user voice and a non-voice input. The non-speech input may include a user's gesture recognition, a non-speech input of a user input through manipulation of an input device, vehicle state information indicating a state of a vehicle, driving environment information related to a driving environment of a vehicle, and user information indicating a state of a user. And the like. In addition to the information, any information related to the vehicle and the user may be input to the input processor 110 as long as the information may be used to determine the user's intention or provide a service required by the user or the vehicle. The user can include both a driver and a passenger.

The input processor 110 recognizes the input user's voice and converts it into a text-type speech, and applies a natural language understanding algorithm to the user's speech to grasp the user's intention.

In addition, the input processor 110 collects information related to the state of the vehicle or the driving environment in addition to the user's voice, and understands the situation using the collected information.

The input processor 110 transmits information related to the intention and the situation of the user identified through the understanding of natural language, to the conversation manager 120.

The conversation manager 120 determines an action corresponding to the user's intention or the current situation based on the intention of the user, information related to the situation, etc. transmitted from the input processor 110, and manages the factors necessary to perform the action. .

In this embodiment, the action may mean all operations performed to provide a specific service, and the type of action may be predefined. In some cases, providing a service and performing an action may have the same meaning.

For example, in the domain / action inference rule DB 141 (see FIG. 19A), actions such as road guidance, vehicle status check, gas station recommendation, speaking, depression index determination, risk avoidance, etc. may be predefined and stored inference According to the rule, an action corresponding to the user's speech, that is, an action intended by the user, may be extracted from the predefined actions. In addition, an action associated with an event occurring in the vehicle may be predefined and stored in the association action DB 146b (see FIG. 21).

There is no restriction on the type of action, and the conversation system 100 is performed as the vehicle 200 or the mobile device 400, and is predefined, and the inference rule and the relationship with other actions / events are stored. If it is, it can be an action.

The conversation manager 120 transmits the information about the determined action to the result processor 130.

The result processor 130 generates and outputs a conversation response and a command necessary to perform the delivered action. The conversation response may be output as text, image, or audio, and when a command is output, services such as vehicle control and external content corresponding to the output command may be performed.

The storage unit 140 stores various types of information necessary for conversation processing and service provision. For example, information related to domains, actions, acts, and entity names used to understand natural language can be stored in advance, and a situation understanding table used to understand a situation from inputted information can be stored, and a sensor provided in a vehicle can be detected. One data, information related to a user, and information necessary for performing an action may be stored in advance. A more detailed description of the information stored in the storage 140 will be described later.

As described above, the conversation system 100 provides a conversation processing technique specific to the vehicle environment. All of the components of the dialogue system 100 may be included in the vehicle, or only some of them may be included. The conversation system 100 may be provided on a remote server and the vehicle may serve only as a gateway between the conversation system 100 and the user. In either case, the conversation system 100 may be connected to a user through a vehicle or a mobile device connected to the vehicle.

2 is a diagram illustrating a configuration of a vehicle interior.

Referring to FIG. 2, the center fascia 203, which is a central area of the dashboard 201 inside the vehicle 200, is required to perform control of a vehicle including audio, video, navigation, and dialing functions. A display 231 displaying a screen and an input button 221 for receiving a user's control command may be provided.

In addition, an input button 223 may also be provided on the steering wheel 207 for the driver's convenience, and serves as an input button in the center console area 202 between the driver's seat 254a and the passenger seat 254b. A jog shuttle 225 may be provided.

A module including a display 231, an input button 221, and a processor for controlling various functions in general may be referred to as an audio video navigation (AVN) terminal or a head unit.

The display 231 may be implemented as one of various display devices such as a liquid crystal display (LCD), a light emitting diode (LED), a plasma display panel (PDP), an organic light emitting diode (OLED), and a cathode ray tube (CRT). have.

As shown in FIG. 2, the input button 221 may be provided as a hard key type in an area adjacent to the display 231. When the display 231 is implemented as a touch screen type, the display 231 may be an input button. The function of 221 may also be performed.

The vehicle 1 may receive a user's command by voice through the voice input device 210. The voice input device 210 may include a microphone that receives sound and converts the sound into an electrical signal.

The voice input device 210 may be provided in the headlining 205 as shown in FIG. 2 for effective voice input. However, the exemplary embodiment of the vehicle 200 is not limited thereto. It may be provided or provided in the steering wheel 207. In addition, there is no limit to any location suitable for receiving a user's voice input.

A speaker 232 may be provided inside the vehicle 200 to output a sound required to communicate with a user or to provide a service desired by the user. For example, the speaker 232 may be provided inside the driver's seat door 253a and the passenger seat door 253b.

The speaker 232 may output a voice for navigation path guidance, a sound or voice included in audio / video content, a voice for providing information or service desired by the user, a system speech generated in response to the user's speech, or the like. Can be.

The conversation system 100 according to an embodiment provides a service optimized for a user's lifestyle by using a conversation processing technology specialized for a vehicle environment, and connects a connected car, an IoT, an artificial intelligence ( New services can be implemented using technologies such as AI).

When the conversation processing technology specialized for the vehicle environment is applied, such as the conversation system 100 according to an exemplary embodiment, the driver may easily recognize and respond to the main context in the direct driving situation. Services can be provided by weighting factors that affect driving, such as lack of flow, drowsy driving, or factors affecting the degree of driver's depression, and in most cases, driving time and destination information based on conditions moving to the destination. Etc., information necessary for providing a service can be easily obtained.

In addition, it is possible to easily implement an intelligent service that grasps the driver's intention and proposes a function. This is because real-time information and action are considered first in the driver's direct driving situation. For example, if the driver searches for a gas station while driving, this may be interpreted as the driver's intention to go to the gas station. However, when a gas station is searched in a non-vehicle environment, in addition to the intention of going to a gas station now, it may be interpreted as various intentions such as location information inquiry, telephone number inquiry, price inquiry, and the like.

In addition, the vehicle is a limited space, but there may be various situations therein. For example, the conversation system 100 may be utilized in driving situations of an unfamiliar interface such as a car, a surrogate driving situation, a vehicle management situation such as washing a car, a situation in which an infant is boarded, and a situation in which a specific destination is found.

In addition, various service opportunities and conversation situations occur at each stage configuring the driving of the vehicle and before and after the vehicle checking stage, the departure preparation stage, the driving stage, and the parking stage. In particular, the situation of not knowing how to deal with vehicle problems, the situation of interlocking vehicles and various external devices, the situation of checking driving habits such as fuel economy, the use of safety support functions such as Smart Cruise Control, the navigation operation situation, drowsy driving situation, The conversation system 100 may be utilized in a situation in which the same route is repeatedly driven every day, a situation in which it is necessary to check whether parking is possible.

3 to 5 illustrate examples of conversations that can be exchanged between the conversation system and the driver.

Referring to FIG. 3, even if the driver does not input a ignition for asking the current remaining amount or requesting gas station guidance, the conversation system 100 determines the current remaining amount by itself, and when the determined remaining amount is less than the set reference value, Informational ignition (S1: can travel 43km with the remaining oil) can be output first.

In response, the driver may enter an ignition (U1: Tell a nearby gas station) asking for a nearby gas station to get directions to the gas station, and the conversation system 100 may provide information about the gas station closest to the current location. It is possible to output ignition (S2: gas station near the current location, A oil forest gas station, B oil pole gas station, C oil Pacific gas station).

The driver may additionally enter a ignition (U2: where is the cheapest?) For requesting information on the oil price, and the dialogue system 100 may provide a ignition (S3: gasoline) that provides information on prices by oil type. The lowest price is B oil pole gas station at 1,294 won per liter and the lowest price at diesel oil is A oil Sunglim gas station at 985 won per liter.

The driver can enter a utterance (U3) to guide the B oil pole gas station, and the dialogue system 100 outputs a utterance (S4: starts guiding to the B oil pole gas station) that the driver starts to guide the selected gas station. can do.

That is, the conversation system 100 may determine that the currently required service is a gas station guide service based on the state information of the vehicle received through the input processor 110, and output a starter for providing the service. In addition, the driver may receive a route guidance to a nearby gas station that sells the current type of vehicle at the lowest price through the dialogue with the conversation system 100. In this embodiment, the speech output by the conversation system 100 before the user speaks will be referred to as starting speech.

Meanwhile, in selecting the gas station in the example of FIG. 3 described above, the conversation system 100 may shorten the steps and time of the conversation by omitting some queries and providing information immediately.

For example, the conversation system 100 may know in advance that the type of the current vehicle is gasoline and the driver's gas station selection criterion is price. Information about the type of the vehicle may be obtained from the vehicle, and the driver's gas station selection criteria may be obtained by inputting the driver in advance, or by learning the conversation history or the gas station selection history of the driver. Such information may be stored in advance in the storage 140 of the conversation system 100.

In this case, as shown in FIG. 4, even if the driver does not input the ignition U2 for requesting information on the oil price (omitted U3), the conversation system 100 may provide information on the oil price, in particular of the current vehicle. It is possible to preemptively output utterances (S2 + S3 = S3 ') that provide information on the price of oil, which is a type of oil.

The driver may omit the ignition U2 for requesting information on the oil price, and the response of the dialogue system 100 may include the ignition S2 for guiding the nearby gas station and the ignition S3 for guiding the oil price. By combining, you can shorten the dialogue step and time.

In addition, the conversation system 100 may determine for itself that the driver's intention is to search for a gas station based on the fact that the driver has asked the current residual amount.

In this case, as shown in FIG. 5, even if the driver does not input the ignition U1 asking for a nearby gas station (omit U1), the utterance system S2 + S3 = provides information on the oil price. S3 ") can be preemptively outputted.

In addition, if there is one gas station that is closest to the current location and has the lowest oil price to the ignition (S3 ") that provides information on the oil price, the user can also query whether to guide the gas station. Rather than inputting a specific speech for guiding to a gas station, the user may request directions to the corresponding gas station by simply inputting an utterance (U3 ': yeah) agreeing to the query of the conversation system 100.

As described above, the conversation system 100 may consider the contents not uttered by the user by using the previously obtained information, thereby grasping the actual intention of the user and preemptively providing information corresponding thereto. This can shorten the dialogue step and time until the user provides the desired service.

6 and 7 are control block diagrams briefly illustrating the connection relationship between the dialogue system and the components of the vehicle.

Referring to FIG. 6, a voice of a user input to the conversation system 100 may be input through a voice input device 210 provided in the vehicle 200. As described above with reference to FIG. 2, the voice input device 210 may include a microphone provided in the vehicle 200.

Inputs other than voices among the user inputs may be input through the non-speech input device 220. The non-voice input device 220 may include input buttons 221 and 223 and a jog shuttle 225 for receiving a command through a user's manipulation.

In addition, the non-voice input device 220 may include a camera for capturing a user. The user's gesture, facial expression or gaze direction used as a means of command input may be recognized through the image photographed by the camera. Alternatively, the user's state (drowsiness state, emotional state, etc.) may be determined through an image captured by the camera.

Information about the vehicle may be input to the conversation system 100 through the vehicle controller 240. The information about the vehicle may include vehicle state information or surrounding situation information obtained through various sensors provided in the vehicle 200, and may also include information basically stored in the vehicle 200, such as the type of the vehicle.

The conversation system 100 may use a user's voice input through the voice input device 210, a non-voice input of the user input through the non-voice input device 220, and various information input through the vehicle controller 240. It grasps the intention and the situation of the user and outputs a response for performing an action corresponding to the intention of the user.

The speaker output device 230 is a device that provides a visual, audio, or tactile output to the speaker. The speaker output device 230 may include a display 231 and a speaker 232 provided in the vehicle 200. The display 231 and the speaker 232 may visually or audibly output a response to the user's speech, a query for the user, or information requested by the user. Alternatively, it is possible to mount a vibrator on the steering wheel 207 to output the vibration.

In addition, the vehicle controller 240 may control the vehicle 200 to perform an action corresponding to the user's intention or the current situation according to the response output from the conversation system 100.

On the other hand, the vehicle 200 is not only data obtained through the sensor provided in the vehicle 200, but also information obtained from the external content server 300 or an external device through the communication device 280, for example, traffic conditions, weather Also, driving environment information such as temperature, accident-prone area information, dangerous area information, passenger information, driver personal information, and family information, schedule information, biorhythm information, sleep information, and the like are collected and transmitted to the conversation system 100. Can be. In addition, the user information may include educational information, residence information, work information, occupation information, preference information, and the like.

As shown in FIG. 7, information obtained from sensors provided in a vehicle such as residual flow rate, rainfall, rainfall speed, ambient obstacle information, speed, engine temperature, tire pressure, current position, and the like, reproduced sound source information, sound source reproduction time, etc. Vehicle state information including information obtained from a device provided in the vehicle may be input to the conversation system 100 through the internal signal controller 241.

In addition to the vehicle status information, driving environment information including weather information, traffic information, destination information, road situation information, and the like may also be input to the conversation system 100 through the internal signal controller 241.

The driving environment information acquired from the outside through V2X (Vehicle to Everything) may be input to the conversation system 100 through the external signal controller 242. V2X means that vehicles can drive and communicate with road infrastructure and other vehicles, exchanging and sharing useful information such as traffic conditions.

V2X communication provides vehicle-to-infrastructure (V2I) communications, vehicle-to-vehicle (V2V) communications, and vehicle-to-Nomadic devices (V2N). It may include. Therefore, using the V2X communication, it is possible to directly communicate with each other, or to communicate with the infrastructure installed on the street to send and receive traffic information in front of the vehicle, the possibility of approaching or colliding with other vehicles, and inform the driver.

Accordingly, the driving environment information input to the conversation system 100 through the external signal controller 242 may include traffic information in front of the vehicle, access information of surrounding vehicles, collision warning with other vehicles, real time traffic situation, breakout situation, and traffic flow control. State, weather information, and the like.

Although not shown in the drawing, a signal obtained through V2X may also be input to the vehicle 200 through the communication device 280.

The vehicle controller 240 may include a memory for storing a program for performing the above-described operation and the operation described later, and a processor for executing the stored program. At least one memory and a processor may be provided, and in the case of a plurality of processors, they may be integrated on one chip or physically separated.

In addition, the internal signal controller 241 and the external signal controller 242 may be implemented by the same processor and memory, or may be implemented by separate processors and memory.

8 and 9 are control block diagrams schematically illustrating the connection relationship between the components of the dialogue system and the components of the vehicle.

Referring to FIG. 8, the user voice transmitted from the voice input device 210 is input to the voice input processor 111 in the input processor 110, and the non-voice input of the user transmitted from the non-voice input device 220 is input. It is input to the situation information processor 112 in the input processor 110.

In addition, the information input through the internal signal controller 241 and the information input through the external signal controller 242 are input to the situation information processor 112 in the input processor 110.

The contextual information input to the contextual information processor 112 includes vehicle state information, driving environment information, user information, and the like input from the non-voice input device 220 and the vehicle controller 240. The situation information processor 112 understands the situation based on the input situation information. The conversation system 100 can more accurately grasp the user's intention or effectively find the service required by the current user through understanding the situation.

For example, the context information input to the context information processor 112 includes a driving duration, a distance traveled, a sleep state of the user, biorhythm information of the user, playlist information of the AVN 255 device, play sound source information, and play time information. Destination information, and the like.

The response output from the result processor 130 is input to the talker output device 230 or the vehicle controller 240 to enable the vehicle 200 to provide the necessary services to the user. Or, it may be transmitted to the external content server 300 to request a necessary service.

Vehicle state information, driving environment information, and user information transmitted from the vehicle controller 240 are stored in the storage 140.

Referring to FIG. 9, the storage 140 may include a long term memory 143 and a short term memory 144. The data stored in the storage 140 may be divided into short-term and long-term memories according to the importance, durability, and intention of the designer.

Short-term memory 144 may store a previously performed conversation. The previously performed conversation may be a conversation performed within a reference time from the current time point. Alternatively, the conversation may continue to be stored until the capacity of the utterance content between the user and the conversation system 100 becomes a reference value.

For example, when the current time is a meal time, the vehicle 200 may output a speech asking whether to guide the restaurant through the speaker 232. Whether or not it is a meal time may be determined based on whether the current time is included in a preset time range as a meal time. If the user utters “let me know the restaurant near Gangnam Station” or the user utters “let me know the restaurant” and the current location of the vehicle 200 is near Gangnam Station, the conversation system 100 may be near the Gangnam Station via an external content server 300. Search for restaurants in the city, and provide information on restaurants near the found Gangnam station to the user. As an example of providing information, the list of restaurants may be displayed on the display 231, and when the user utters "first", the conversation contents from the inquiry about the restaurant to the selection may be stored in the short-term memory 144. .

Alternatively, not only the entire conversation contents may be stored, but also specific information included in the conversation contents may be stored. For example, it is also possible to store the first restaurant in the restaurant list in the short term memory 144 or the long term memory 143 as the restaurant selected by the user.

After a conversation about a restaurant near the Gangnam station, when the user asks the conversation system 100, “How is the weather?”, The conversation system 100 estimates that the user's location of interest is the Gangnam station from the conversation stored in the short-term memory 144. The user can output the response "Raining in Gangnam Station."

Subsequently, when the user utters, "Recommend the restaurant menu.", The conversation system 100 estimates that "the restaurant" is a restaurant near Gangnam station from the conversation stored in the short-term memory, and from the external content server 300 The service can be used to obtain information about the restaurant's recommended menu, and output a response such as, "The restaurant has delicious kalguksu."

The long term memory 143 may store data depending on whether data is persistent. For example, a phone number of a family member or friend, POI information such as a home or a company, a user's preference for a specific factor, and the like may be determined to ensure data permanence, and may be stored in the long-term memory 143. On the other hand, data that is determined to not be durable may be stored in the short term memory 144.

For example, since the user's current location is temporary data, the user's current location may be stored in the short-term memory 144, and the user's preference for the restaurant may be stored in the long-term memory 144 since the user's preference for the restaurant may be viewed as available data.

When the user speaks, "Is there no restaurant nearby?", The conversation system 100 can determine the user's current location from the short-term memory 144 and the user's preferred restaurant from the long-term memory 143. . Therefore, the user can recommend a list of preferred restaurants around the current location using the external content.

In addition, the long term memory 143 may store the vehicle state information, the driving environment information, and the user information acquired by the situation information processor 112. For example, the long-term memory 143 may include driving duration, driving distance, sleep state of the user, biorhythm information of the user, playlist information of the AVN 255 device, playback sound source information, playback time information, destination information, and the like. Can be stored.

In addition, the conversation system 100 may preemptively provide a service or information to a user by using data stored in the long term memory 143 and the short term memory 144.

For example, information about a user's home may be stored in the long term memory 143. The conversation system 100 may obtain information related to the user's house from the external content server 300, and provide the user with the information, "I am planning to clean the apartment complex this Friday."

Alternatively, information about the battery state of the vehicle may be stored in the short term memory 144. The conversation system (100) analyzes the vehicle's battery status stored in the short-term memory (1440) and says, “The vehicle's battery status is not good. Have it repaired before winter. ”

As another example, information about the biorhythm and the sleep state of the user may be stored in the long term memory 143. The conversation system 100 obtains information related to the user's biorhythm and sleep from the external content server 300 and the external device, and provides the user with a query service, "I'm not enough sleep recently. Are you worried?" Can give

As another example, the tone analysis value of the user's voice may be stored in the long term memory 143. When the tone analysis value of the user's voice differs from the tone analysis value of the user's voice by more than a predetermined value, the dialog system 100 asks the user, "Your voice is not bright. Do you feel bad?" 10 is a control block diagram of a vehicle-only method in which a conversation system is provided in a vehicle.

In the vehicle only method, as illustrated in FIG. 10, the vehicle 200 includes a conversation system 100 including an input processor 110, a conversation manager 120, a result processor 130, and a storage 140. Can be.

When the conversation system 100 is included in the vehicle 200, the vehicle 200 may process a conversation with the user by itself and provide a service necessary for the user. However, information necessary for processing a conversation and providing a service may be obtained from an external content server 300.

Vehicle status information or destination information such as residual flow amount, rainfall, rainfall speed, surrounding obstacle information, tire pressure, current position, engine temperature, vehicle speed, reproduced sound source information, sound source playback time, road detected by vehicle detector 260 Driving environment information such as situation information is input to the conversation system 100 through the vehicle controller 240.

In addition, the vehicle controller 240 may control the air conditioner 251, the window 252, the door 253, the seat 254, or the AVN 255 provided in the vehicle 200 according to the response output by the dialogue system 100. Etc. can be controlled.

For example, when the conversation system 100 determines that a user's intention or a service required by the user lowers the temperature inside the vehicle 200, and generates and outputs a command corresponding thereto, the vehicle controller 240 controls the air conditioning apparatus. The temperature inside the vehicle 200 may be lowered by controlling 251.

As another example, when the conversation system 100 determines that the user's intention or the service required by the user raises the driver's seat window 252a, and generates and outputs a command corresponding thereto, the vehicle controller 240 opens the window 252. The driver's seat window 252a can be raised by controlling.

As another example, when the conversation system 100 determines that a service corresponding to a user's intention or a service required by the user is a route guidance to a specific destination, and generates and outputs a command corresponding thereto, the vehicle controller 240 displays the AVN. Route control may be performed by controlling 255. If necessary, the communication device 280 may obtain map data, POI information, and the like from the external content server 300 and use the same to provide a service.

As another example, when the conversation system 100 determines that the user is in a depressed state, it is determined that a service corresponding to the user's intention or a service required by the user is provided upward, and generates a command to play a corresponding sound source. When output, the vehicle controller 240 may control the AVN 255 to reproduce the sound source. If necessary, the communication device 280 may obtain a sound source or the like from the external content server 300 and use it to provide a service.

As another example, when the conversation system 100 determines that the user is in a depressed state, when the user determines that the user needs fresh air, the vehicle controller 240 controls the window 252 to induce ventilation or to provide an air conditioning device ( 251 may be controlled to lower the temperature inside the vehicle 200 or to convert to the outdoor mode.

As another example, when the conversation system 100 determines that the user is in a depressed state or a dangerous situation, the dialog system 100 determines that the user needs to use the autonomous driving function, generates and outputs a command for guiding the autonomous driving function, and outputs the vehicle controller. The 240 may control the AVN 255 to display the autonomous driving function.

11 and 12 are control block diagrams for a vehicle gateway method in which a conversation system is provided in a remote server and the vehicle serves only as a gateway connecting the user and the conversation system.

In the vehicle gateway method, as shown in FIG. 11, a remote conversation system server 1 is provided outside the vehicle 200, and the vehicle 200 is connected to the remote conversation system server 1 through a communication device 280. Conversation system client 270 is provided. The communication device 280 serves as a gateway connecting the vehicle 200 and the remote conversation system server 1.

The conversation system client 270 functions as an interface connected to an input / output device, and may perform data collection and transmission and reception.

When the voice input device 210 and the non-voice input device 220 provided in the vehicle 200 receive a user input and transmit the user input to the chat system client 270, the chat system client 270 communicates through the communication device 280. The input data can be sent to the remote conversation system server 1.

The vehicle controller 240 may also transmit data sensed by the vehicle detection unit 260 to the conversation system client 270, and the conversation system client 270 may communicate with the remote conversation system server 1 through the communication device 280. The data detected by the vehicle detector 260 may be transmitted.

The remote conversation system server 1 may be provided with the above-described conversation system 100 to perform both input data processing, dialog processing based on the processing result of the input data, and result processing based on the conversation processing result.

In addition, the remote chat system server 1 may import information or content necessary for processing input data, chat management, or result processing from the external content server 300.

The vehicle 200 may also import content required to provide a service required by the user from the external content server 300 according to a response transmitted from the remote conversation system server 1.

Referring to FIG. 12, the communication device 280 may include one or more communication modules that enable communication with an external device, for example, a short range communication module 281, a wired communication module 282, and a wireless communication module. (283).

The short range communication module 281 uses a wireless communication network in a short distance such as a Bluetooth module, an infrared communication module, a radio frequency identification (RFID) communication module, a wireless local access network (WLAN) communication module, an NFC communication module, and a Zigbee communication module. It may include at least one of a variety of short-range communication module for transmitting and receiving a signal.

The wired communication module 282 may be connected to a variety of wired communication modules such as a local area network (LAN) module, a wide area network (WAN) module, or a value added network (VAN) module, as well as universal USB. It may include at least one of various cable communication modules such as a serial bus (HDMI), high definition multimedia interface (HDMI), digital visual interface (DVI), reduced standard232 (RS-232), power line communication, or plain old telephone service (POTS). have.

The wireless communication module 283 includes a Wi-Fi module and a wireless broadband module, as well as a global system for mobile communication (GSM), code division multiple access (CDMA), wideband code division multiple access (WCDMA), and universal communication system (UMTS). It may include at least one of a variety of wireless communication modules that can be connected to the Internet network in a wireless communication scheme such as Mobile Telecommunications System (TDMA), Time Division Multiple Access (TDMA), Long Term Evolution (LTE), 4G, 5G.

Meanwhile, the communication device 280 may further include an internal communication module (not shown) for communication between electronic devices inside the vehicle 200. As an internal communication protocol of the vehicle 200, a controller area network (CAN), a local interconnection network (LIN), a flexray, an Ethernet, or the like may be used.

The conversation system 100 may exchange data with the external content server 300 or the remote conversation system server 1 using the wireless communication module 283. In addition, V2X communication may be performed using the wireless communication module 283. In addition, the local area communication module 281 or the wired communication module 282 may exchange data with a mobile device connected to the vehicle 200.

FIG. 13 is a control block diagram of a case in which a vehicle may perform some input processing and output processing in a vehicle gateway method.

As described above, the conversation system client 270 of the vehicle 200 may perform only the functions of collecting and transmitting data, but as shown in FIG. 13, an input processor 271 is input to the conversation system client 270. ), The result processor 273 and the storage unit 274 may be included, and the vehicle 200 may perform processing related to the processing of data input from the user or the vehicle or the provision of a service determined to be necessary for the user. That is, the operations of the input processor 110 and the result processor 130 may be performed in the vehicle 200 as well as the remote conversation system server 1.

In this case, the above-described operation of the input processor 110 may be performed by the chat system client 270 only partially or entirely. In addition, the conversation system client 270 may perform only part of the operation of the processor 130 as described above, or may perform all of the operations.

The task sharing between the remote conversation system server 1 and the conversation system client 270 may be determined in consideration of the capacity of the data to be processed, the data processing speed, and the like.

FIG. 14 is a control block diagram of a hybrid scheme in which both a remote conversation system server and a vehicle can perform conversation processing.

In the hybrid method, as illustrated in FIG. 14, an input processor 110, a conversation manager 120, a result processor 130, and a storage 140 may also be provided in the remote conversation system server 1 to perform conversation processing. In addition, the vehicle 200 may also be provided with a terminal conversation system 290 including an input processor 291, a conversation manager 292, a result processor 293, and a storage 294.

However, the processor or memory provided in the vehicle 200 may have a difference in capacity or performance from the processor or memory provided in the remote conversation system server 1. Therefore, if the terminal conversation system 290 can process all input data and manage the conversation and output the result, the terminal conversation system 290 performs the whole process, otherwise the remote conversation system server 1 ) Can be processed.

Before performing the conversation processing in the terminal conversation system 290, it is determined whether the terminal conversation system 290 is processed based on the type of input data, and is directly processed or processed by the remote conversation system server 1 according to the determination result. You can request

Alternatively, when a situation in which the terminal conversation system 290 is unable to process a conversation while performing a conversation process, the terminal conversation system 290 may transmit the result of the processing together while requesting the process to the remote conversation system server 1.

For example, when the high performance computing power and the long-term data processing are required, the remote conversation system server 1 may process, and the terminal conversation system 290 may process what should be processed in real time. For example, when an instance requiring immediate processing occurs and needs to process data before synchronization, the terminal conversation system 290 may be set to be preferentially processed.

In addition, when there is an unregistered talker in the vehicle and needs to confirm the user, the remote conversation system server 1 may process the conversation.

In addition, when the terminal conversation system 290 cannot complete the conversation process by itself in a situation where it is impossible to connect with the remote conversation system server 1 through the communication device 280, the user through the speaker output device 230. It may be informed that the conversation process cannot be performed.

Data stored in the terminal conversation system 290 and data stored in the remote conversation system server 1 may be determined according to criteria such as the capacity of the data or the type of the data. For example, data that may be personally identified and may cause privacy infringement may be stored in the storage unit 294 of the terminal conversation system 290. In addition, a large amount of data may be stored in the storage 140 of the remote conversation system server 1, and a small amount of data may be stored in the storage 294 of the terminal conversation system 290. Alternatively, a small amount of data may be stored in both the storage 140 of the remote conversation system server 1 and the storage 294 of the terminal conversation system 290.

15 and 16 are control block diagrams of a mobile gateway method in which a mobile device connected to a vehicle connects a user with a remote conversation system server.

In the mobile gateway method, as illustrated in FIG. 15, the mobile device 400 receives vehicle state information, driving environment information, and the like from the vehicle 200, and transmits user input and vehicle state information to the remote conversation system server 1. send. That is, the mobile device 400 serves as a gateway connecting the user with the remote conversation system server 1 or the vehicle 200 and the remote conversation system server 1.

The mobile device 400 may be an electronic device that is portable such as a smart phone, a smart watch, a smart glass, a PDA, a tablet PC, and the like, and communicates with an external server and a vehicle to exchange data.

The mobile device 400 may include a voice input device 410 that receives a user voice, a non-voice input device 420 that receives a user's non-speech input, and an output device 430 that outputs a response visually, audibly, or tactilely. ), A communication device 480 that communicates with the remote conversation system server 1 and the vehicle 200 to transmit and receive data, and collects input data from the vehicle 200 and a user to communicate with the remote conversation system server through the communication device 480. (1) includes a conversation system client 470 to transmit.

The voice input device 410 may include a microphone that receives sound and converts the sound into an electrical signal.

The non-voice input device 420 may include an input button, a touch screen, or a camera provided in the mobile device 400.

The output device 430 may include a display, a speaker, or a vibrator provided in the mobile device 400.

In addition to using the voice input device 410, the non-voice input device 420, and the output device 430 provided in the mobile device 400 as an input / output interface for a user, the voice input device 210 provided in the vehicle 200 is provided. ), The non-voice input device 220 and the speaker output device 230 may be used.

When the vehicle 200 transmits the data and the user input detected by the vehicle detecting unit 260 to the mobile device 400, the chat system client 470 of the mobile device 400 sends the data to the remote chat system server 1. send.

In addition, the chat system client 470 may transmit a response or a command transmitted from the remote chat system server 1 to the vehicle 200. When using the dialogue output device 230 provided in the vehicle 200 as an input / output interface to the user, the dialogue output device 230 outputs the speech of the dialogue system 100 or outputs a response to the speech of the user. can do. When the output device 430 provided in the mobile device 400 is used, the speech of the conversation system 100 may be output through the output device 430 of the mobile device 400, or a response to the user's speech may be output. Can be.

The command for controlling the vehicle is transmitted to the vehicle 200, and the vehicle controller 240 may provide a service necessary for the user by performing a control corresponding to the transmitted command.

On the other hand, the conversation system client 470 not only collects and transmits input data to the remote conversation system server 1, but also performs some or all of the functions of the input processor 110 and the result processor 130 of the conversation system 100. It is also possible to do it all.

Referring to FIG. 16, the communication device 480 of the mobile device 400 may include one or more communication modules that enable communication with an external device, for example, a short range communication module 481 and a wired communication module ( 482 and wireless communication module 483.

The short range communication module 481 uses a wireless communication network at a short distance such as a Bluetooth module, an infrared communication module, a radio frequency identification (RFID) communication module, a wireless local access network (WLAN) communication module, an NFC communication module, and a Zigbee communication module. It may include at least one of a variety of short-range communication module for transmitting and receiving a signal.

The wired communication module 482 may be connected to a variety of wired communication modules such as a local area network (LAN) module, a wide area network (WAN) module, or a value added network (VAN) module, as well as universal USB. It may include at least one of various cable communication modules such as a serial bus (HDMI), high definition multimedia interface (HDMI), digital visual interface (DVI), reduced standard232 (RS-232), power line communication, or plain old telephone service (POTS). have.

The wireless communication module 483 includes a Wi-Fi module and a wireless broadband module, as well as a Global System for Mobile Communication (GSM), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), and UMTS (Universal). It may include at least one of a variety of wireless communication modules that can be connected to the Internet network in a wireless communication scheme such as Mobile Telecommunications System (TDMA), Time Division Multiple Access (TDMA), Long Term Evolution (LTE), 4G, 5G.

For example, the mobile device 400 may be connected to the vehicle 200 via the short range communication module 481 or the wired communication module 482, and may be connected to the remote conversation system server 1 through the wireless communication module 483. It may be connected to the external content server 300.

17 is a control block diagram of a mobile standalone system in which a dialogue system is mounted on a mobile device.

In the mobile only mode, as shown in FIG. 17, a dialogue system 100 is provided in the mobile device 400.

Accordingly, it is not necessary to be connected to the remote conversation system server 1 to perform the conversation processing, and the mobile device 400 may process the conversation with the user and provide the necessary service to the user. However, some of information necessary for processing a conversation and providing a service may be obtained from an external content server 300.

In addition, in any of the above-described manner, the components constituting the dialogue system 100 may be physically separated, and some of the components may be omitted. For example, even when the conversation system 100 is provided in the remote conversation system server 1, some of the components constituting the conversation system 100 may be provided in a separate server or a vehicle. In this case, the separate server may be the same as or different from the remote conversation system server 1 in an operation or management entity. For example, the speech recognizer or natural language reader to be described later may be provided in a separate server, and the conversation system 100 may receive a speech recognition result or natural language understanding result for user speech from a separate server. Alternatively, the storage 140 may be provided in a separate server.

Hereinafter, the detailed configuration of the conversation system 100 and the specific operations for each component will be described in detail. In the following embodiments, for convenience of description, it is assumed that the dialog system 100 is provided in the vehicle 200. Detailed configurations of the dialog system 100 to be described below are classified according to operations to be performed, and are not limited to whether they are implemented by the same processor and memory, physical locations of the processor and memory, and the like.

18, 19A, and 19B are control block diagrams in which the configuration of the input processor of the configuration of the conversation system is subdivided.

Referring to FIG. 18, the input processor 110 may include a voice input processor 111 that processes a voice input and a context information processor 112 that processes context information.

In addition, the input processor 110 may further include a risk situation manager 119 that determines whether the user is in a dangerous situation.

The user's voice input through the voice input device 210 is transmitted to the voice input processor 111, and the non-voice user input input through the non-voice input device 220 is transmitted to the situation information processor 112.

The vehicle controller 240 transmits the vehicle state information, the driving environment information, and the user information to the situation information processor 112. The driving environment information and the user information may be provided from the external content server 300 or the mobile device 400 connected to the vehicle 200.

All non-voice input may be included in the context information. That is, the situation information may include vehicle state information, driving environment information, and user information.

The vehicle state information may include information indicating the state of the vehicle as information obtained by a sensor provided in the vehicle 200, or may include information stored in the vehicle as information related to the vehicle, such as the type of the vehicle. have.

In addition, the vehicle state information may include information obtained from a device provided in the vehicle, such as sound source information to be reproduced, sound source reproduction time, and the like.

The driving environment information is information obtained by a sensor provided in the vehicle 200, and image information obtained by a front camera, a rear camera or a stereo camera, and a sensor such as a radar, a lidar, an ultrasonic sensor, and the like. It may include obstacle information, rainfall / rainfall speed information obtained by the rainfall sensor.

The driving environment information may further include traffic condition information, traffic light information, possibility of approaching or colliding with surrounding vehicles, and the like as information obtained through the V2X.

The user information includes information related to a user's state measured by a camera or a biosignal measuring device provided in a vehicle, information related to a user directly input by using an input device provided in a vehicle, and a user stored in an external content server 300. And information stored in the mobile device 400 connected to the vehicle.

In particular, the user information may include user personal information, biorhythm information, and sleep related information transmitted from the outside, and may include tone information including a tone analysis value of the user voice received from the tone analyzer 111d. can do.

In particular, the user information includes information related to the user's voice input by the voice input processor 111 as well as the aforementioned non-voice input, for example, the user's voice request, the tone of the user's voice analyzed by the tone analyzer 111d. Information may be further included.

The voice input processor 111 recognizes the inputted user's voice and outputs the spoken text in a text form, and the natural language understanding is applied to the spoken text so that the spoken text includes the user's speech. It may include a natural language reader 111b that grasps the intention, and a conversation input manager 111c that delivers the natural language understanding result and situation information to the conversation manager 120.

In addition, the voice input processor 111 may further include a tone analyzer 111d that receives a voice of a user and performs tone analysis to determine whether the user is in a bad mood or a depressed state than usual. .

In addition, the voice input processor 111 may further include a response analyzer 111e that determines whether the user is in a depressed state and a degree of depression based on the user's response to the question of the conversation system 100. 111a includes a speech recognition engine, and the speech recognition engine may apply a speech recognition algorithm to the input speech to recognize the speech spoken by the user and generate a recognition result.

At this time, the input voice may be converted into a more useful form for speech recognition, and the starting and ending points are detected from the speech signal to detect the actual speech section included in the input speech. This is called end point detection (EPD).

In the detected section, features such as Cepstrum, Linear Predictive Coefficient (LPC), Mel Frequency Cepstral Coefficient (MFCC), Filter Bank Energy, etc. A vector extraction technique may be applied to extract a feature vector of an input voice.

The recognition result may be obtained by comparing the extracted feature vector with the trained reference pattern. To this end, an acoustic model for modeling and comparing signal characteristics of speech and a language model for modeling linguistic order relations such as words or syllables corresponding to a recognized vocabulary may be used. To this end, the storage unit 140 may store the acoustic model / language model DB.

The acoustic model may be further divided into a direct comparison method for setting a recognition object as a feature vector model and comparing it with a feature vector of speech data and a statistical method for statistically processing the feature vector of the recognition object.

The direct comparison method is a method of setting a unit of a word, a phoneme, or the like to be recognized as a feature vector model and comparing how similar the input speech is. As a representative example, a vector quantization method is used. According to the vector quantization method, a feature vector of input speech data is mapped with a codebook, which is a reference model, and encoded into a representative value to compare the code values.

The statistical model method is a method of constructing a unit for a recognition object into a state sequence and using a relationship between state columns. The status column may consist of a plurality of nodes. Methods using the relationship between the state columns again include Dynamic Time Warping (DTW), Hidden Markov Model (HMM), and neural networks.

Dynamic time warping is a method of compensating for differences in the time axis when considering the dynamic characteristics of speech whose length varies over time even when the same person pronounces the same. Assuming a Markov process with probability and observed probability of nodes (output symbols) in each state, we estimate state transition probability and observed probability of the node from the training data, and calculate the probability that the input voice will occur in the estimated model. Is a recognition technology.

On the other hand, a language model that models linguistic ordinal relationships such as words or syllables may apply acoustic relations between units constituting language to units obtained in speech recognition, thereby reducing acoustic ambiguity and reducing recognition errors. The linguistic model has a model based on a statistical language model and a finite state automata (FSA), and the statistical linguistic model uses a chain probability of words such as Unigram, Bigram, and Trigram.

The voice recognizer 111a may use any of the above-described methods in recognizing the voice. For example, an acoustic model with a hidden Markov model may be used, or an N-best search method that integrates the acoustic model and the speech model. The N-best search method can improve recognition performance by selecting up to N recognition result candidates using acoustic models and language models, and then re-evaluating the ranks of these candidates.

The speech recognizer 111a may calculate a confidence value in order to secure the reliability of the recognition result. The confidence value is a measure of how reliable the result is for a speech recognition result. For example, a phoneme or a word that is a recognized result may be defined as a relative value of the probability that the word is spoken from other phonemes or words. Therefore, the confidence value may be represented by a value between 0 and 1, and may be represented by a value between 0 and 100.

If the confidence value exceeds a preset threshold, the recognition result may be output so that an operation corresponding to the recognition result may be performed. If the confidence value is less than or equal to the threshold value, the recognition result may be rejected. have.

The spoken text in the form of text that is a result of the recognition of the speech recognizer 111a is input to the natural language recognizer 111b.

The natural language reader 111b may grasp the intention of the user included in the speech by applying the natural language understanding technology. Accordingly, a user may input a command through a natural dialog, and the dialog system 100 may also induce input of a command through a dialog or provide a service required by the user.

First, the natural language reader 111b performs morphological analysis on the spoken text in a text form. A morpheme is the smallest unit of meaning and represents the smallest semantic element that can no longer be subdivided. Thus, morphological analysis is the first step in understanding natural language, converting input strings into morphological sequences.

The natural language recognizer 111b extracts a domain from the spoken text based on the morphological analysis result. Domains can identify the subject of the language spoken by the user. For example, domains representing various topics such as route guidance, weather search, traffic search, scheduling, refueling guidance, air conditioning control, diversion, and accident prevention. This is a database.

The natural language reader 111b can recognize the individual name from the spoken text. The entity name is a proper noun such as a person's name, place name, organization name, time, date, currency, etc. The entity name recognition is the task of identifying the entity name in the sentence and determining the type of the entity name identified. By recognizing individual names, we can extract important keywords from sentences to understand the meaning of sentences.

The natural language reader 111b may analyze the speech act of the spoken text. Speech act analysis is to analyze the intention of the user's speech, and to grasp the intention of the speech regarding whether the user asks a question, makes a request, makes a response, or expresses a simple emotion.

The natural language reader 111b extracts an action corresponding to the user's intention to speak. A user's speech intent may be grasped based on information on a domain, an entity name, and a speech act corresponding to the speech text, and an action corresponding to the speech intent may be extracted. An action may be defined by an object and an operator.

In addition, the natural language reader 111b may extract a factor related to performing an action. The arguments associated with performing the action may be valid arguments directly required to perform the action, or may be invalid arguments used to extract such valid arguments.

For example, when the user's utterance is "Let's go in Seoul," the natural language reader 111b may extract "navigation" into a domain corresponding to the utterance and extract "direction" as an action. A speech act corresponds to a "request".

The entity name "Seoul Station" corresponds to [Factor: Destination] associated with performing an action, but may require a specific exit number or GPS information for navigating through actual navigation. In this case, [Factor: Destination: Seoul Station] extracted by the natural language reader 111b may be a candidate factor for searching for Seoul Station desired by a real user among a plurality of Seoul Station POIs.

In addition, the natural language interpreter 111b may extract a means for expressing a mathematical relationship between a word, a word, a sentence, and a sentence, such as a parse-tree.

The morphological analysis result, domain information, action information, speech act information, extracted factor information, entity name information, pastry, and the like, which are the processing results of the natural language understander 111b, are transmitted to the conversation input manager 111c.

The tone analyzer 111d may analyze the tone of the input voice and determine the height of the input voice tone by comparing with the user's usual tone. The tone analysis information generated by the tone analyzer 111d may be used as a control basis for determining the degree of depression of the user by the response analyzer 111e to be described later.

Also, the analysis information of the user's usual tone, which is the reference information when the tone analyzer 111d determines the height of the voice tone, may be stored in the long-term memory 143.

The response analyzer 111e may compare the user's response to the question for determining the degree of depression of the user with previously stored data, and determine the degree of depression of the user based on the comparison result. In this case, the previously stored data may be Q & A data stored in the Q & A list 146g to be described later.

Specifically, the response analyzer 111e may receive the processing result from the natural language reader 111b, compare the processing result with pre-stored Q & A data, and determine the degree of depression of the user based on the comparison result. have.

In this case, the response analyzer 111e may quantify the degree of depression of the user by using the depression index indicating the degree of depression of the user, and the depression index information of the user analyzed by the response analyzer 111e may be a conversation input manager ( 111c).

The response analyzer 111e may obtain a question item from the Q & A list 146g which is a database including at least one question data for determining a degree of depression of the user. At this time, the Q & A list (146g) stores a list of [questions, answers, answers], the depression index for [questions, answers, answers], or the depression index information for the 'word' included in the 'answers' It may include. In addition, the response analyzer 111e may receive natural language information through the natural language reader 111b with respect to the user's answer to the question.

For example, the response analyzer 111e responds to the question "How are you feeling today?" Provided by the conversation system 100 to the user. The depression index for the user may be determined based on previously stored depression index information.

In addition, the response analyzer 111e may determine the depression index of the user based on the depression index information for the word stored in the Q & A list 146g of the word included in the user's 'answer' to the 'question'.

The response analyzer 111e may transmit the depression index information obtained based on the user's answer to the question from the Q & A list 146g to the conversation input manager 111c.

In addition, the response analyzer 111e may store the obtained depression index information in a separate storage space, and may store the acquired depression index information in the short term memory 144. In addition, the response analyzer 111e may store, in the long-term memory 143, the average value of the depression index of the user, which is obtained as information representing the degree of depression of the user.

The contextual information processor 112 includes a contextual information collector 112a that collects information from the non-voice input device 220 and the vehicle controller 240, a contextual information collection manager 112b that manages the collection of contextual information, and a natural language understanding result. And a context comparator 112c for understanding the context based on the collected context information.

In addition to the above-described information, the situation information collector 112a may further collect tone analysis information about the voice of the user analyzed by the tone analyzer 111d. The tone analysis information acquired by the situation information collector 112a from the tone analyzer 111d may be included in the user information and stored in the situation information DB 142. At this time, the analysis information of the user's usual tone, which is the reference information when the tone analyzer 111d determines the height of the voice tone, may be stored in the long-term memory 143.

When the situation information collector 112a collects tone analysis information from the tone analyzer 111d, the situation information collector 112a may transmit a confirmation signal to the situation information collection manager 112b.

In addition, the situation information collection manager 112b also sends a confirmation signal to the situation understanding 112c, and the situation understanding 112c sends necessary information to the situation information DB 142, the long-term memory 143, or the short-term memory 144. ) Is transferred to the conversation input manager 111c.

The dangerous situation manager 119 may determine whether the dangerous situation is based on the collected situation information. At this time, the dangerous situation refers to a situation in which a high degree of depression of the user is high, and an accident due to depression is likely to occur.

The dangerous situation manager 119 may determine whether a predetermined dangerous situation condition is satisfied based on the collected situation information, and may determine whether the user is in a dangerous situation based on whether the condition is satisfied.

To this end, the dangerous situation manager 119 may transmit a request signal of situation information necessary for determining whether the dangerous situation condition is satisfied to the situation information collection manager 112b.

For example, the dangerous situation manager 119 may determine whether the dangerous condition condition 'when the user travels to a far place without a passenger' is satisfied, and for this, the passenger information, destination information, You can collect notice information.

As another example, the risk situation manager 119 may determine whether the risk condition condition 'when a user is located in an accident-prone area without a passenger' is met, and for this purpose, the passenger information and accident-prone area information may be used. Can be collected.

As another example, the risk situation manager 119 may be configured to 'when the user travels far away without a passenger, and the site is far away', 'when the user is located without a passenger, in an accident-prone area', 'when driving without a destination', ' At least one of the following conditions: "If your average sleep time during the last month is less than your average sleep time," "I sigh at least once every 30 minutes," and "I am talking to an unknown self." It is possible to determine whether or not the user is in a dangerous situation based on whether or not it is satisfied.

In addition, the risk situation manager 119 may determine whether it is a risk situation based on the depression index information of the user acquired by the response analyzer 111e, and based on the depression index information of the user together with the above-described situation information. You can also determine whether it is a dangerous situation.

For example, the dangerous situation manager 119 may determine that the user is in a dangerous situation when the obtained depression index information of the user is larger than a predetermined reference value or is out of the predetermined reference range.

As another example, the risk situation manager 119 is a user with a high index of depression usually 'when the user is driving without a passenger, the connection is far away', 'user is located without a passenger, in an accident-prone area', 'destination destination Driving without "," if your average sleep during the last month is less than your average sleep, "" if you sigh at least once every 30 minutes, "and" if you speak unintelligible loneliness. " If at least one of the conditions is satisfied, it may be determined that the user is in danger. In this case, the user who has a high depression index usually means a user whose average value of the stored depression index is higher than a predetermined reference value.

When the dangerous situation manager 119 determines that the dangerous situation is satisfied and the user is in a dangerous situation, the dangerous situation manager 119 may transmit an action trigger signal to the situation understander 112c so that the selection for the actual situation determination is performed.

At this time, the content of the selection that is performed to determine the actual situation may be a question stored in the Q & A repository 146, and the risk situation manager 119 may understand the action trigger signal associated with the user's answer to the question. To device 112c.

In addition, the risk situation manager 119 may determine the risk situation step based on the number of conditions that are satisfied among the predetermined risk situation conditions, and transmits an action trigger signal corresponding to the determined risk situation step to the situation understanding 112c. Can transmit

For example, the risk situation manager 119 may distinguish the risk situation stage into five stages, and may transmit an action trigger signal for preventing an accident if the risk stage is greater than three stages, and the risk situation stage has three stages. In the following case, an action trigger signal for recommending accident prevention may be transmitted to the user.

The input processor 110 may include a memory for storing a program for performing the above-described or later operations, and a processor for executing the stored program. At least one memory and a processor may be provided, and in the case of a plurality of processors, they may be integrated on one chip or physically separated.

In addition, the voice input processor 111 and the context information processor 112 included in the input processor 110 may be implemented by the same processor and memory, or may be implemented by separate processors and memories.

Hereinafter, with reference to FIGS. 19A and 19B, how the components of the input processor 110 process the input data using the information stored in the storage 140 will be described in detail.

Referring to FIG. 19A, the natural language interpreter 111b may use the domain / action inference rule DB 141 for domain extraction, entity name recognition, speech act analysis, and action extraction.

The domain / action inference rule DB 141 may store domain extraction rules, speech act analysis rules, entity name conversion rules, action extraction rules, and the like.

Other information such as non-voice user input, vehicle status information, driving environment information, and user information is input to the situation information collector 112a and stored in the situation information DB 142, the long term memory 143, or the short term memory 144. Can be.

For example, raw data detected by the vehicle detecting unit 260 may be classified into a sensor type and a sensor value in the situation information DB 142.

The short-term memory 144 and the long-term memory 143 may store data meaningful to the user, such as the current state of the user, the user's preference / propensity, or data that can determine the user's preference.

As described above, the long-term memory 143 may store information that can be used for a long period of time, such as a user's phone book, schedule, preference, education, personality, occupation, family related information, and the like. In addition, the long-term memory 143 may store information related to the tone of the usual user voice.

The short-term memory 144 may store information that is used in the short term because of the non-persistence or uncertainty such as the current / previous location, the today's schedule, the contents of the previous conversation, the participants of the conversation, the surrounding situation, the domain, the driver's status, and the like. Depending on the type of data, there may be data that is duplicated and stored in two or more storages of the situation information DB 142, the short term memory 144, and the long term memory 143. In addition, the short-term memory 144 may store tone information of the input voice at the time of the user's speech.

In addition, the information stored in the short-term memory 144, which is determined to be durable, may be sent to the long-term memory 143.

It is also possible to obtain information to be stored in the long term memory 143 using the information stored in the short term memory 144 or the contextual information DB 142. For example, the user's preference may be obtained by analyzing destination information or conversation contents accumulated for a certain period, and the obtained user's preference may be stored in the long-term memory 143.

Acquiring information to be stored in the long-term memory 143 using the information stored in the short-term memory 144 or the contextual information DB 142 may be performed in the dialogue system 100, or may be performed in a separate external system. It is also possible.

In the former case, the memory manager 135 of the result processor 130 to be described later may be performed. In this case, among the data stored in the short-term memory 144 or the contextual information DB 142, data used to obtain meaningful or persistent information, such as user preference or disposition, may be stored in the long-term memory 143 in the form of a log file. Can be stored.

The memory manager 135 analyzes the data accumulated for a predetermined period of time to obtain data having persistentness and to store the data in the long term memory 143 again. In the long term memory 143, a location where persistent data is stored and a location where data stored in a log file form are stored may be different from each other.

Alternatively, the memory manager 135 may determine persistent data among the data stored in the short-term memory 144 and transfer the determined data to the long-term memory 143 for storage.

When performed in a separate external system, as illustrated in FIG. 19B, a data management system 800 including a communication unit 810, a storage unit 820, and a control unit 830 may be used.

The communication unit 810 receives data stored in the situation information DB 142 or the short term memory 144. All of the stored data may be transmitted to the communication unit 810, or only data used to acquire meaningful information or persistent information, such as a user's preference or disposition, may be selected and transmitted. The received data is stored in the storage 820.

The controller 830 analyzes the accumulated data to obtain persistent data, and transmits the acquired data to the conversation system 100 through the communication unit 810. The transmitted data is stored in the long term memory 143 of the conversation system 100.

In addition, the conversation input manager 111c may transfer the output result of the natural language reader 111b to the situation understander 112c to obtain context information related to performing an action.

The context comparator 112c may determine what context information is associated with performing an action corresponding to the user's intention of speaking, with reference to the contextual information for each action stored in the context understanding table 145.

20A and 20B illustrate examples of information stored in a situation understanding table.

Referring to the example of FIG. 20A, the context understanding table 145 may store context information and context information types related to performing an action for each action.

For example, when the road guidance is an action, the current location is needed as the context information, and the context information type may be GPS information. When the vehicle condition check is an action, a moving distance is required as the situation information, and the situation information type may be an integer. When the gas station recommendation is an action, the remaining oil amount and the distance to empty (DTE) are required as the situation information, and the situation information type may be an integer.

As another example, when the depression index determination is an action in the starting situation for determining the depression index of the user, sleep time is required as the situation information, and the situation information type may be an integer. In addition, when the dangerous situation determination is an action in the starting situation for determining the dangerous situation, destination information is needed as the situation information, and the situation information type may be GPS information.

When the situation information related to the action performed corresponding to the user's utterance intention is already stored in the situation information DB 142, the long term memory 143, or the short term memory 144, the situation reader 112c may request the situation information DB 142. ), The information is taken from the long term memory 143 or the short term memory 144 and transferred to the conversation input manager 111c.

When the situation information related to performing an action corresponding to the user's utterance intention is not stored in the situation information DB 142, the long term memory 143, or the short term memory 144, the situation understanding unit 112c may use the situation information collection manager ( Request the information required in 112b). The contextual information collection manager 112b allows the contextual information collector 112a to collect necessary information.

The situation information collector 112a may periodically collect data, may collect at a specific event occurrence, or may additionally collect at a specific event occurrence. In addition, data may be collected when a data collection request is input from the situation information collection manager 112b.

The contextual information collector 112a collects the necessary information, stores it in the contextual information DB 142 or the short-term memory 144, and transmits a confirmation signal to the contextual information collection manager 112b.

The situation information collection manager 112b also transmits a confirmation signal to the situation understanding 112c, and the situation understanding 112c sends necessary information from the situation information DB 142, the long-term memory 143, or the short-term memory 144. Take it and pass it to the dialog input manager 111c.

As a specific example, when the action corresponding to the user's uttering intention is a road guide, the situation understanding unit 112c may search the situation understanding table 145 to know that the situation information related to the road guide is the current location.

If the current position is already stored in the short term memory 144, the situation understander 112c takes the current position from the short term memory 144 and transfers it to the conversation input manager 111c.

If the current location is not stored in the short-term memory 144, the current location request is requested from the situation information collection manager 112b, and the situation information collection manager 112b causes the situation information collector 112a to execute the vehicle controller 240. ) To get the current position.

The contextual information collector 112a obtains the current location and stores the current location in the short-term memory 144, and transmits a confirmation signal to the contextual information collection manager 112b. The contextual information collection manager 112b also transmits a confirmation signal to the contextual reader 112c, which obtains the current position information from the short-term memory 144 and delivers it to the conversation input manager 111c.

The conversation input manager 111c may transmit the output of the natural language reader 111b and the output of the situation understanding 112c to the conversation manager 120, and manage the duplicate input to prevent the input from entering the conversation manager 120. have. In this case, the output of the natural language reader 111b and the output of the situation reader 112c may be combined into one and delivered to the conversation manager 120, or may be delivered independently of each other.

Meanwhile, the situation information collection manager 112b may transmit an action trigger signal to the situation understander 112c when it is determined that a specific event occurs because the data collected by the situation information collector 112a satisfies a preset condition.

The context comparator 112c searches the context comprehension table 145 to search for context information related to the corresponding event, and if the searched context information is not stored, transmits a request signal of the context information to the context information collection manager 112b. do.

As illustrated in FIG. 20B, the context understanding table 145 may store context information related to an event and types of context information for each event.

For example, when the generated event is an engine temperature warning, the engine type of the integer type may be stored as related context information. When the generated event is driver drowsiness detection, an integer type driver drowsiness step may be stored as related context information. If the generated event is a tire inflation pressure, the tire type air pressure of the integer type may be stored as related context information. If the generated event is a fuel warning, the integer type of driving distance that can be stored may be stored as related context information. If the generated event is a sensor error, the sensor name of the character type may be stored as related context information. In addition, when the generated event is a user depression index determination, an integer type depression index may be stored as related situation information. When the generated event is a risk situation determination, an integer type dangerous situation step is stored as related situation information. Can be.

The contextual information collecting manager 112b collects the necessary contextual information through the contextual information collector 112a and transmits a confirmation signal to the contextual understanding 112c. The situation understanding unit 112c takes necessary situation information from the situation information DB 142, the long term memory 143, or the short term memory 144, and transfers the necessary situation information to the conversation input manager 111c together with the action information.

The conversation input manager 111c inputs the output of the situation understanding 112c to the conversation manager 120.

Hereinafter, a description will be given of a case in which the dialogue system 100 outputs the starting speech by itself before the user's speech is input.

FIG. 21 is a control block diagram of a conversation system applicable when the conversation system first outputs a utterance before a user input is received, and FIGS. 22A, 22B and 22C illustrate examples of information stored in the selection condition table. The figure shown.

Referring to FIG. 21, the input processor 110 of the conversation system 100 may further include a starter determiner 151 and a duplicate task processor 152 that determine whether the starter is in a starting state. 140 may further include a starting condition table 145a for storing the starting condition and a task processing DB 145b.

Data stored in the contextual information DB 142, the long term memory 143, and the short term memory 144 may be transferred to the starter determiner 151. The starter determiner 151 may analyze the transmitted data to determine whether the starter condition stored in the starter condition table 145a is satisfied.

Referring to the example of FIG. 22A, the starting condition table 145a may store the starting condition related to the situation information and the starting message output when the corresponding starting condition is satisfied for each situation information.

The starter determination unit 151 determines that the starter situation is generated when the situation information transmitted from the situation information DB 142 satisfies the starter condition and generates a starter trigger signal.

The starter determiner 151 may transmit the starter trigger signal to the situation understander 112c together with the starter message corresponding to the starter situation. In addition, it is possible to convey the information on the starting situation. The information on the starting situation may include a starting condition corresponding to the starting situation or an action corresponding to the starting situation to be described later.

For example, when the situation information is tire inflation pressure, the selection condition may be satisfied when the tire inflation pressure is equal to or less than a preset reference value. When the tire pneumatic pressure satisfies the starting condition, the starter judging unit 151 may determine that the starting condition is due to insufficient tire pressure, and generate a starting trigger signal.

The starter determiner 151 may transfer the starter trigger signal to the situation understander 112c together with the starter message. For example, in the starting situation due to insufficient tire pressure, a starting message that informs that the tire pressure is insufficient, such as “tire air pressure is too low,” may be transmitted to the situation understanding unit 112c.

In addition, when the situation information is the engine temperature, the starting condition may be satisfied when the engine temperature is equal to or greater than a preset reference value. When the engine temperature satisfies the starting condition, the starter determiner 151 may determine that the starting condition is caused by the engine temperature or more, and generate the starting trigger signal.

The starter determiner 151 may transfer the starter trigger signal to the situation understander 112c together with the starter message. For example, in a starting situation due to an engine temperature or more, a starting message that informs the engine that the engine is overheated, such as “engine temperature is too high,” may be transmitted to the situation understanding unit 112c.

In addition, when the situation information is the residual amount, the selection condition may be satisfied when the residual amount is less than or equal to the reference value. When the user sets the destination using the vehicle's navigation service, the reference value may be set based on the distance from the current location to the set destination. When the destination is not set, a reference value set as a default may be used regardless of the destination. For example, a value smaller than the reference value indicating the low fuel warning lamp may be set as the reference value of the starting condition due to the insufficient amount of residual oil. When the remaining amount of water satisfies the starting condition, the starter determiner 151 may determine that the starting state is due to the lack of the remaining amount, and may generate a starting trigger signal.

The starter determiner 151 may transfer the starter trigger signal to the situation understander 112c together with the starter message. For example, in the starting situation due to insufficient residual amount, a starting message that informs that the residual amount is insufficient, such as "the amount of residual amount to the destination" is insufficient, may be transmitted to the situation understanding unit 112c.

In addition, when the situation information is sleep information in the selection situation for determining the depression index of the user, the selection condition may be satisfied when the sleep time is less than a preset reference value (5 hours). When the running time satisfies the starting condition, the starter judging unit 151 lacks the sleep time such as, "I'm having sleep lately. Are you worried?" With the starting trigger signal for determining the depression index. In addition to the information, the initiation message for the indirect query for determining the depression index of the user may be transmitted to the situation understandr 112c.

In addition, when the risk situation step is less than or equal to a predetermined level in the selection situation for the user to determine the risk situation, the selection condition for the actual situation determination may be satisfied. The starter determination unit 151 selects a destination message such as "How do you set the destination?" Together with the starter trigger signal for determining the actual situation when the dangerous situation step satisfies the starter condition. May be passed to the situation understanding 112c.

However, the starting condition and the starting message shown in FIGS. 22A and 22E are merely examples that may be applied to the conversation system 100. In the above-described example, the case where the starting message corresponding to the starting situation is a content indicating the current situation has been described as an example. However, the dialogue system 100 first suggests execution of a specific function or service required for the starting situation. It is possible.

Referring to the example of FIG. 22B, when the starting situation is caused by insufficient tire pressure or the starting situation is caused by an abnormal engine temperature, a starting point reservation service such as "Should I reserve a repair shop?" You can save what you offer proactively.

In addition, in the case of the starter situation due to the insufficient amount of residual oil, a content for proactively proposing a gas station guide service, such as "Do you want to guide a gas station?", May be stored as a starter message corresponding thereto.

In addition, when the situation information is the vehicle internal temperature, the starting condition may be satisfied when the vehicle internal temperature is out of a preset reference range. When the vehicle internal temperature satisfies the starting condition, the situation understanding unit 112c may determine that the starting condition is due to the vehicle internal temperature or higher and generate a starting trigger signal.

In a starting situation due to an abnormal temperature inside the vehicle, a preliminary suggestion of an internal temperature control function may be stored as a starting message such as "will the air conditioner be operated?".

In addition, when the situation information is a microphone input, the start condition may be satisfied when the microphone input value is equal to or less than a preset reference value. When the microphone input 112 satisfies the starting condition, the situation understanding unit 112c may determine that the starting condition for changing the atmosphere is generated and generate the starting trigger signal. As the corresponding starter message, contents which preemptively propose a multimedia playback service may be stored, such as "play music?"

In addition, when the situation information is whether the window is opened or closed, the start condition may be satisfied when the window is open and the precipitation situation. The situation understanding unit 112c may determine that the starting situation is due to the window opening when the window is open and the precipitation situation, and generate the starting trigger signal.

In the starting situation due to the opening of the window, the contents of preemptively suggesting the window closing function, such as "Do you want to close the window?", May be stored as the starting message.

In the example of FIGS. 22A and 22B described above, the case in which the selection message corresponding to the selection situation is previously stored in the selection condition table 145a is taken as an example. However, the example of the dialogue system 100 is not limited thereto, and an action corresponding to the selection situation may be stored in advance.

As described above, when the user's utterance is input, the natural language interpreter 111b extracts an action corresponding to the user's utterance with reference to the domain / action inference rule DB 141. When the conversation system 100 outputs the starter, as shown in FIG. 22C, an action corresponding to the starter may be stored in advance.

For example, in the case of a starter situation due to an abnormal tire inflation pressure and a starter situation due to an engine temperature abnormality, the shop guide may be stored as a corresponding action, and in a starter situation due to a lack of residual oil, a gas station guide may respond to it. Can be stored as an action.

In addition, the air conditioner operation may be stored as an action corresponding to the starter situation due to an abnormal temperature inside the vehicle, and multimedia playback may be stored as an action corresponding to the starter situation for changing the atmosphere. have. In the case of the starting situation due to the window opening, opening / closing the window may be stored as an action corresponding thereto. As described above, when an action corresponding to the starting situation is previously stored, the window opening and closing action may be performed together with the starting trigger signal. The action is transmitted to the situation understanding unit 112c, and the conversation input manager 111c may input the start trigger signal and the action corresponding thereto to the conversation manager 120. In this case, the conversation manager 120 may perform the same operation as when the user's speech is input.

As another example, a virtual user utterance corresponding to each starting situation may be matched and stored in the starting condition table 145a, or the starting decision 151 may generate a virtual user utterance corresponding to the starting situation. It is possible. The starter determiner 151 may transfer the user speech stored in the starter condition table 145a or generated by the starter determiner 151a in text form to the natural language reader 111b. For example, in the case of starting shoes due to abnormal tire pressure, virtual user utterances such as "Please check tire pressure" or "Guide me the repair shop" may be stored or generated. Alternatively, in the case of a start-up situation due to an abnormal temperature inside the vehicle, a virtual user utterance such as “turn on the air conditioner” may be stored or generated.

Alternatively, in the mobile gateway method in which the mobile device 400 serves as a gateway between the vehicle 100 and the conversation system 100, the conversation system client 470 of the mobile device 400 selects the starter determiner 151. Some of the operations of) can be performed. In this case, the conversation system client 470 may generate a virtual user speech corresponding to the selection situation and deliver the virtual user speech to the natural language reader 111b.

22D to 22F are diagrams showing another example of information stored in the starting condition table.

Referring to FIG. 22D, in the selection condition table 145a, the selection condition related to the situation information for estimating that the user needs to determine the depression and the selection message output when the selection condition is satisfied for each situation are described. Can be stored.

For example, when the situation information is the vehicle speed, the starting condition may be satisfied when the vehicle is stopped. When the vehicle speed satisfies the starting condition, the starter judging unit 151 determines that the user's depression is necessary, and the user's feeling of depression such as "How are you feeling today" with the starting trigger signal. The starter message for judgment may be forwarded to the situation understanding 112c.

In addition, if the situation information is radio / TV / newspaper content, if the output radio / TV / newspaper content satisfies the selection condition related to depression, it is accompanied by the selection trigger signal. Can be delivered. "

In addition, when the situation information is biorhythm information and the biorhythm index is lower than the reference value, the start message may be transmitted along with the start trigger signal, "Biorhythm is not good. What is happening?"

In addition, when the situation information is sleep information and the recent sleep time is lower than the reference value, a starter message, such as, "You have recently gone to sleep. What is happening?" May be transmitted with the starter trigger signal.

In addition, when the situation information is a playback sound source, when the BPM (Beats Per Minute) of the played music sound is lower than a predetermined reference value, a start message may be output, "The music is sad. Do you have a sad day?" .

In addition, when the situation information is user facial expression information, when the user cries, a starting message "Why cry?" May be output. When the situation information is a microphone input, when the user sighs, "a lot of sighs" may be output. The ground will go off due to rest. Is there a bad thing? "

In addition, when the situation information is weather information, a starter message, such as rainy weather, "the mood falls on a rainy day. How are you feeling today?" May be output.

In addition, when the situation information is a control command of the user information, when a depression index determination command is input, a starting message "Are there any bad things?" Is output and a conversation for determining the depression index of the user may be started. have.

Referring to FIG. 22E, the starting condition table 145a may store, in advance, a starting condition related to situation information for estimating that the user needs to determine depression and an action performed when the starting condition is satisfied. have.

At this time, [depression index determination] may be stored as an action corresponding to the case where the selection condition related to the situation information for estimating that the user needs to determine the depression is satisfied.

For example, when the vehicle is stopped, the radio / TV / newspaper is related to depression, the biorhythm index is low, the sleep time is insufficient, the tempo of the playback source is slow, the user is crying, the user Depression index determination may be stored as a corresponding action in a situation where a sigh is detected, when the weather is rainy, and when a depression index determination command is input.

As such, one or more selection conditions corresponding to [depression index determination] may be stored in the selection condition table 145a.

If the starter conditioner 151 is satisfied with the starter condition related to the situation information for estimating that the above-described user's depression judgment is required, the starter determiner 151 determines the depression index as an action corresponding to the starter trigger signal. ] Can be passed to the situation understanding 112c. The situation understanding 112c may input such [depression index determination] action into the conversation manager 120.

The starting message for determining the depression index may be stored in the Q & A list 146g, and the conversation manager 120 extracts or generates the starting message from the question data stored in the Q & A list 146g based on the situation information. can do. In addition, as illustrated in FIG. 22F, the starting condition table 145a includes a starting condition for the situation information for determining whether the user is in a dangerous situation and an action performed when the starting condition is satisfied. The judgment can be stored. At this time, at least one situation information related to the starting condition for the risk situation determination may be stored.

When the starter judging unit 151 satisfies the starter condition of 'if the distant area without a passenger is the destination' based on the passenger information, the destination information and the connection site information among the situation information, the [dangerous situation determination] The action may be passed to the situation understanding 112c.

Also, when the selection condition of 'if no passenger is located in the accident occurrence area' is satisfied based on the passenger information and the accident occurrence area information among the situation information, the [dangerous situation determination] action is sent to the situation understanding unit 112c. I can deliver it.

In addition, when the starting condition 'when driving without a passenger and without a destination' is satisfied based on the passenger information and the destination information, the [dangerous situation determination] action may be transmitted to the situation understanding unit 112c.

In addition, when the selection condition, 'when the average sleep time during the last month is less than or equal to the reference value', is satisfied based on the sleep information in the situation information, the [risk situation determination] action may be transmitted to the situation understandr 112c. .

In addition, based on the microphone input of the situation information, if the user satisfies the starting condition of 'if the user sighs more than once every 30 minutes' or the case of 'if the user speaks a message of unknown meaning' If satisfactory, the [risk situation determination] action can be passed to the situation understanding (112c).

The situation understanding 112c may input such a [risk situation determination] action into the conversation manager 120, and the conversation manager 120 may determine whether the user is in a dangerous situation based on the data stored in the accident response manual DB 148b. The verbalization message for determining whether or not to be extracted may be generated or generated. The natural language reader 111b may extract a domain, an action, and the like corresponding to the delivered virtual user speech and transmit the extracted domain message to the conversation input manager 111c. The action extracted by the natural language reader 111b becomes an action corresponding to the selection situation. After the action corresponding to the starting situation is transmitted to the conversation manager 120, the process may be performed in the same manner as when the user first speaks.

The above-described situation information, starting condition, starting message and action are only examples applicable to the embodiment of the conversation system 100, and the embodiment of the conversation system 100 is not limited to the above examples. In addition, various situation information, starting conditions, starting messages and actions may be stored.

When the starter determiner 151 transmits the starter trigger signal and the information on the starter situation to the situation understander 112c, the situation understander 112c informs the duplicate task processor 152 about the starter situation. To pass.

The duplicate task processor 152 may determine whether a task related to the currently occurring starting situation is already processed or a duplicate task in progress based on the information stored in the task processing DB 145b.

The task processing DB 145b stores information about tasks processed or in progress in the conversation system 100. For example, the conversation history (including the conversation contents and each conversation time point), the vehicle state at the time of the conversation, and whether the task is completed may be stored. In addition, the progress and processing results of tasks such as driving directions using the navigation function may be stored regardless of the conversation.

As a specific example, in the starting situation due to insufficient residual amount, the redundant task processor 152 may determine whether the gas station guide task is in progress based on the information stored in the task processing DB 145b. When a conversation for a gas station guide is in progress or a gas station guide action is in progress, the duplicate task processor 152 may determine a task related to the current starting situation as a duplicate task and terminate the starting situation.

Alternatively, even when the speech for the gas station guide has been previously output and there is a conversation history in which the user rejects the gas station guidance, the duplicate task processor 152 determines the task related to the current starting situation as a duplicate task, and selects it. You can end the fire situation.

Alternatively, even when the gas station guide task using the navigation function is currently in progress, the duplicate task processor 152 determines the task related to the current starting situation as a duplicate task regardless of the conversation history for the gas station guidance. Can be terminated. The duplicate task processor 152 may know from the information stored in the task processing DB 14b that a gas station guide task using the navigation function is in progress.

Alternatively, if the reference time has not elapsed since the conversation related to the residual flow guidance, the user may be driving toward the gas station by himself even if the gas station guide task is not in progress. The related task may be determined as a duplicate task, and the starting situation may be terminated.

In addition, in the case of a selection situation for guiding a schedule based on information stored in the long-term memory 143 such as a user's birthday or a family's birthday, a conversation history of guiding the same schedule previously exists, and the conversation is performed. If the elapsed time has not elapsed by the reference time, the duplicate task processor 152 may determine that the task related to the current starting situation is a duplicate task and terminate the starting situation.

That is, the duplicate task processor 152 may determine whether the starting information is already output, the user's intention about the starting situation, and the like based on the conversation history stored in the task processing DB 145b. It may be determined whether a duplicate task is based on the intention of the user, the vehicle state, or whether the task is completed.

The duplicate task processor 152 may store a policy for determining whether the task is a duplicate task or not, based on the information stored in the task processing DB 145b. The duplicate task processor 152 may determine whether a task related to the current starting situation is a duplicate task according to the stored policy, and if the duplicate task processor 152 determines that the task is a duplicate task, the duplicate task processor 152 may terminate the starting situation.

On the other hand, the above-described example will be described for the case of including the components of the starter determination unit 151, the duplicate task processor 152, the starter condition table 145a, the task processing DB 145b for the starter processing. It was.

However, the example of the dialogue system 100 is not limited thereto, and it is of course possible that the components shown in FIGS. 19A and 19B perform the operations of the added components.

For example, the situation understander 112c may perform an operation of the starter determiner 151 that determines whether the starter condition is satisfied and an operation of the duplicate task process 152 that processes the duplicated task.

In addition, the information stored in the start condition table 145a may be stored in the situation understanding table 145, and the information stored in the task processing DB 145b may be stored in the conversation / action status DB 147 described later. It is also possible.

FIG. 23 is a block diagram illustrating a detailed configuration of a conversation manager, FIG. 24 illustrates an example of information stored in the associated action DB, FIG. 25 illustrates an example of information stored in the action performing condition DB, and FIG. 26. Is a diagram illustrating an example of information stored in the action factor DB.

Referring to FIG. 23, the conversation manager 120 may create / delete / update a conversation or action according to a request of the conversation flow manager 121 and the conversation flow manager 121 that request to create / delete / update a conversation or action. The dialogue action manager 122, the ambiguity resolver 123 to resolve the ambiguity of the situation and the ambiguity of the conversation ultimately to clarify the user's intention, the argument manager 124 for managing the factors necessary to perform the action, a plurality The action priority determiner 125 determines whether to perform actions on candidate actions of the user, and determines the priority of the actions, and an external information manager that manages an external content list and related information and manages factor information required for an external content query. 126 may include.

The conversation manager 120 may include a memory for storing a program for performing the above-described or later operation, and a processor for executing the stored program. At least one memory and a processor may be provided, and in the case of a plurality of processors, they may be integrated on one chip or physically separated.

In addition, each component included in the conversation manager 120 may be implemented by a single processor, or may be implemented by a separate processor.

In addition, the conversation manager 120 and the input processor 110 may also be implemented by a single processor or may be implemented by separate processors.

When the user's utterance is input or when the user utterance matched to the starting situation is transmitted to the natural language understanding 111b, the conversation input manager 111c transmits the natural language understanding result (output of the natural language understanding) and the situation information ( Output of the context comparator) to the conversation flow manager 121. In addition, when a starting situation occurs, the starting trigger signal may be transmitted together.

The output of the natural language recognizer 111b includes information about the user's speech content, such as the result of the morpheme analysis, in addition to information such as a domain and an action. The output of the situation understandr 112c may include an event determined by the situation information collection manager 112b in addition to the situation information.

The conversation flow manager 121 searches whether the conversation task or the action task corresponding to the input from the conversation input manager 111c exists in the conversation / action DB 147.

The conversation / action DB 147 is a storage space for managing the state of the conversation and the state of the action. The conversation / action DB 147 may store a conversation state and an action state of a conversation in progress, actions, and preliminary actions in the future. For example, the status of a conversation / action that has ended, a conversation / action that has been stopped, a conversation / action in progress, or a conversation / action to be processed may be stored.

In addition, whether the action is switched / nested, the switched action index, the action change time, and the last output state such as the screen / voice / command can be stored.

For example, when a domain and an action corresponding to a user's utterance are extracted, if a dialogue or an action corresponding to the domain and the action exists among the most recently stored conversations, it corresponds to an input from the dialog input manager 111c. It can be determined as a dialog task or an action task.

If the domain and the action corresponding to the user's utterance are not extracted, the task may be generated to request the dialogue action generator 122 to generate an arbitrary task or to refer to the most recently stored task.

If there is no chat task or action task corresponding to the output of the input processor 110 in the chat / action DB 147, the chat flow manager 121 creates a new chat task and action task in the chat action manager 122. Ask.

In addition, when the start trigger signal is transmitted from the input processor 110, even if a conversation task or action task already in progress exists, it is temporarily stopped and a dialog task or action task corresponding to the starting situation is preferentially generated. Can be. Alternatively, the priority may be determined according to a predetermined rule.

When the action corresponding to the start trigger signal is input from the conversation input manager 111c and the corresponding action is input, the conversation flow manager 121 performs a new conversation task on the conversation action manager 122 similarly to the case where the action is extracted from the user's speech. And request to create an action task.

Alternatively, when the selection message corresponding to the selection trigger signal is input together with the selection trigger signal from the conversation input manager 111c, the conversation input manager 111c may request to generate a new conversation task for outputting the input selection message.

The conversation flow manager 121 may refer to the conversation policy DB 148 in managing the flow of the conversation. The conversation policy DB 148 stores a policy for deploying a conversation and specifically stores a policy for selecting / starting / proposing / stopping / ending a conversation.

In addition, the dialog policy DB 148 may store a policy on when and how the system outputs a response, and a policy for creating a response by linking multiple services and a policy for deleting an existing action and replacing it with another action. Can be saved.

"B액션을 수행할까요?"와 같이 하나의 액션에 대한 응답을 생성한 이후에, B 액션에 대해서도 별도의 응답을 생성하는 정책이 모두 가능하다. For example, if there are multiple candidate actions, or multiple actions corresponding to the user's intent or situation (A actions, B actions), do you want to perform two actions such as "Do you want to perform A action and B action?" A policy that generates a response all at once, and performs "A action"

After generating a response for one action, such as "Would you like to perform a B action?", It is possible to create a separate response for a B action.

The conversation policy DB 147 may also store a policy that determines the priority among candidate actions. The prioritization policy will be described later.

The conversation action manager 122 allocates storage space to the conversation / action DB 147 to generate a conversation task and an action task corresponding to the output of the input processor 110.

Meanwhile, when the domain and the action cannot be extracted from the user's speech, the conversation action manager 122 may generate an arbitrary conversation state. In this case, as described below, the ambiguity resolver 123 may determine the user's intention based on the user's utterance, surrounding conditions, vehicle status, user information, and the like, and determine an appropriate action corresponding thereto.

If there is a dialogue task and an action task corresponding to the output of the input processor 110 in the conversation / action DB 147, the dialogue flow manager 121 causes the dialogue action manager 122 to refer to the conversation task and the action task. request.

In addition, the dialog flow manager 121 extracts or generates a dialog task and an action task for preventing the accident that may occur in a dangerous situation by the dialog action manager 122 using the output result of the dialog input manager 111a as an input value. can do.

In this case, an accident that may occur in a dangerous situation may mean an accident due to a user's depression, and may include all accidents that may occur when the degree of depression of the user is large.

At this time, if the dialog flow manager 121 has a dialog task and an action task corresponding to the output of the input processor 110 in the incident response manual DB 148b, the dialog action manager 122 performs the dialog task and the action task. You may request to refer to

The accident response manual DB 148b refers to a space for storing a manual for preventing an accident caused by a depressed state of a user.

The accident response manual DB 148b may store a dialog task and an action task as an action for responding to a dangerous situation. In addition, the accident response manual DB 148b may store a conversation task and an action task for each dangerous situation step as an action for responding to a dangerous situation. In addition, the incident response manual DB 148b may store a conversation task and an action task to be provided according to a user's depression index.

For example, the accident response manual DB 148b may store a response action for a dangerous situation step divided into five steps, and up to step 3, a response action for recommending a specific function is stored in response to a fourth step. The accident prevention action can be stored in response.

The conversation action manager 122 may take an appropriate incident response action to prevent an accident that may occur due to the user's depression in consideration of at least one of the user's depression index and the risk state stage received from the conversation input manager 111c. can do.

The action priority determiner 125 retrieves a list of actions associated with the action or event included in the output of the input processor 110 in the associated action DB 146b and extracts the candidate action. As in the example of FIG. 24, associative action DB 146b may represent interrelated actions and relationships between them and actions associated with events and relationships between them. For example, actions such as directions, vehicle condition checks, gas station recommendations, and the like may be classified as associated actions, and their relationships may correspond to each other.

Therefore, in performing the road guidance action, actions such as vehicle status check and gas station recommendation may be performed together. Here, to be performed together may include both before and after the road guide action and during the execution of the road guide action (added as a stop).

Warning light output events may be stored as event-actions associated with workshop guidance actions, and their relationships may also be correlated.

Therefore, when the warning light output event occurs, the workshop guidance action can be performed depending on the type of warning light or whether maintenance is necessary.

In addition to the action corresponding to the user utterance from the input processor 110, when the event determined by the situation collection manager 112b is also transmitted, the action associated with the event may be a candidate action together with the action associated with the action corresponding to the user utterance. Can be.

The extracted candidate action list is transferred to the dialogue action manager 122, and the dialogue action manager 122 adds the candidate action list to update the action state of the dialogue / action state DB 147.

The action priority determiner 125 retrieves a condition for performing each candidate action in the action performing condition DB 146c. As illustrated in FIG. 25, the action performing condition DB 146c may store a condition necessary for performing an action and a factor used to determine whether the corresponding condition is satisfied for each action.

For example, the execution condition of the vehicle condition checking action may be a case where the destination distance is 100 km or more, and a factor used for determining the condition corresponds to the destination distance. The performance condition of the gas station recommendation action may be a case where the destination distance is farther than the driving distance (DTE), and the factors used for determining the condition correspond to the destination distance and the driving distance (DTE).

The action priority determiner 125 transmits the execution condition of the candidate action to the conversation action manager 122, and the dialog action manager 122 adds the action execution condition for each candidate action to the conversation / action state DB 147. Update the action state.

The action priority determiner 125 is a factor necessary for determining an action performing condition in the context information DB 142, the long term memory 143, the short term memory 144, or the conversation / action state DB 147 (hereinafter, the condition determining factor). Search), and it is possible to determine whether to perform each candidate action by using the found factor.

If the factor used for determining the action execution condition is not stored in the situation information DB 142, the long term memory 143, the short term memory 144, or the conversation / action state DB 147, the external information manager 126 The necessary factor may be obtained from the external content server 300.

In addition, the action priority determiner 125 may search for a factor necessary for determining an action execution condition in the vehicle function DB 146f and determine whether to perform each candidate action using the found factor.

The vehicle function DB 146f may store information about a function that the vehicle can provide to the user, and store information such as types of devices and setting values provided by devices inside or outside the vehicle and functions provided by the vehicle.

The action priority determiner 125 may refer to the vehicle function DB 146f to determine whether each candidate action may be provided to a user in the vehicle, that is, whether to perform each candidate action. The action priority determiner 125 may determine whether to perform each candidate action by using a factor necessary for determining an action execution condition. In addition, the priority of each candidate action may be determined based on whether to perform each candidate action and the prioritization rule stored in the dialogue policy DB 148.

The score for each candidate action may be calculated according to the current situation. The higher the priority, the higher the calculated score is. For example, as shown in [Equation 1] below, the action corresponding to the user's utterance, safety score, convenience score, processing time, processing time (whether it should be processed right now), user preference (user at the time of service proposal) Acceptance level or user-defined preference), manager score, vehicle status related score, and action success rate (conversation success rate) may be used as factors of score calculation. w1, w2, w3, w4, w5, w6, w7, w8 and w9 represent weights for each factor.

[Equation 1]

Priority Score = w1 * User Spoken Action + w2 * Safety Score + w3 * Comfort Score + w4 * Processing Time + w5 * Processing Point + w6 * User Preference + w7 * Manager Score + w8 * Vehicle Status Association Score + w9 * Action success rate * Action can be performed (1: possible, tentatively 0: impossible) * Action completed (completed: 1, incomplete: 0)

As described above, the action priority determiner 125 may provide not only the actions directly related to the user's utterance or context information, but also the action list associated with the user and determine the priority among them to provide the service most needed by the user. Will be.

The action priority determiner 125 transmits the performance and priority of each candidate action to the dialogue action manager 122, and the dialogue action manager 122 adds the transmitted information to the conversation / action state DB 147. Update the action state.

The argument manager 124 may search for an argument (hereinafter, referred to as an action factor) used to perform each candidate action in the action factor DB 146a.

As illustrated in FIG. 26, the action factor DB 146a may store a mandatory factor, a selection factor, an initial value of the factor, and a reference position from which the factor can be obtained for each action. When the initial value of the argument is stored, the argument value corresponding to the corresponding factor is not included in the user's utterance or situation information output from the input processor 110, and the corresponding argument value is also stored in the situation information DB 142. If not, the user may check whether the action is performed according to the stored initial value or whether the action is performed according to the stored initial value.

For example, a mandatory argument used to perform a directions action may include a current location and a destination, and an optional argument may include a route type. Fast paths can be stored as initial values of the selection parameters. The current location and the destination may be obtained by sequentially searching the conversation / action status DB 147, the situation information DB 142, the short term memory 144, or the long term memory 143.

The essential factor used to perform the vehicle condition checking action may include vehicle condition information, and the selection factor may include an inspection part. The entire part may be stored as an initial value of the selection factor. Vehicle state information may be obtained from the situation information DB 142.

The selection factor used to perform the gas station recommendation action may include a preferred gas station, and A oil may be stored as an initial value of the selection factor. The preferred gas station may be obtained from the long term memory 143. It is also possible to further include the type of vehicle, oil price, etc. in the selection factor.

As described above, the argument manager 124 obtains the argument value of the argument retrieved from the action argument DB 146a at the corresponding reference position. The reference position from which the argument value may be obtained may be at least one of the situation information DB 142, the long term memory 143, the short term memory 144, the conversation / action state DB 147, and the external content server 300.

When the argument manager 124 obtains the argument value from the external content server 300, it may be via the external information manager 126. The external information manager 126 may determine where to get information by referring to the external service set DB 146d.

The external service set DB 146d stores information about an external content server associated with the conversation system 100. For example, information about an external service name, a description of the external service, a type of information provided by the external service, a method of using the external service, and an entity providing the external service may be stored.

The argument value obtained by the argument manager 124 is transferred to the dialogue action manager 122, and the dialogue action manager 122 updates the dialogue / action state DB 147 by adding argument values for each candidate action to the action state. do.

The argument manager 124 may obtain argument values of all candidate actions, or may obtain only argument values of candidate actions determined to be performed by the action priority determiner 125.

In addition, the parameter manager 124 may selectively use various kinds of factor values indicating the same information. For example, a Seoul station in a text form representing a destination may be converted into a Seoul station in POI form using a destination search service of navigation.

If there is no ambiguity in a conversation or situation, it is possible to obtain necessary information and manage conversations and actions according to the operations of the action priority determiner 125, the factor manager 124, and the external information manager 126 described above. However, when there is ambiguity in a conversation or situation, it is difficult to provide a proper service necessary for a user by only the actions of the action priority determiner 125, the argument manager 124, and the external information manager 126.

In this case, the ambiguity resolver 123 may resolve the ambiguity about the conversation or the ambiguity about the situation. For example, if the conversation includes what people, yesterday, dad, mom, grandmother, daughter-in-law, and the like are ambiguous, the ambiguity resolver 123 is the situation information DB 142, the long-term memo ( 143) or short-term memory 144 may provide a guide to resolve this ambiguity or to resolve it.

For example, an ambiguous word included in a conversation, such as “Yesterday there”, “A-mart near home”, “Seoul station between yesterday”, and the like may correspond to an argument value of an action argument or an argument value of a condition determination factor. However, in this case, the word itself alone cannot perform the action or determine the action execution condition due to its ambiguity.

The ambiguity resolver 123 may solve the ambiguity of the factor value by referring to the information stored in the context information DB 142, the long term memory 143, or the short term memory 144. Alternatively, if necessary, the external information manager 126 may be used to import necessary information from the external content server 300.

For example, by referring to the short-term memory 144, a user may search for a place where he or she went yesterday, thereby converting "yesterday there" into information that can be used as a destination for a road guide action. Alternatively, by referring to the long-term memory 143, the user's home address is retrieved, and the location information about the A-mart near the home address is obtained from the external content server 300, thereby making the "A-mart near the home" a destination of the road guide action. Can be converted into usable information.

In addition, when an action (object or operator) is not clearly extracted from the input processor 110 or the user's intention is ambiguous, the ambiguity solver 123 refers to the ambiguity resolution information DB 146e to determine the user's intention. It is also possible to determine the action corresponding thereto.

27 is a diagram illustrating an example of information stored in the ambiguity resolution information DB.

The ambiguity resolution information DB 146e may match and store the utterance door and the corresponding action based on the vehicle state information and the surrounding situation information. The utterance text stored in the ambiguity resolution information DB 146e may be a utterance statement in which an action cannot be extracted through natural language understanding. In the example of FIG. 27, a description will be given of a case in which the utterance content according to the morphological analysis result is a hand cold or hand cold.

The surrounding situation information may include the vehicle external temperature and rainfall, and the vehicle state information may include whether the air conditioner / heater is on / off and whether the air volume, the wind direction, and the steering heating wire are on / off.

As a specific example, when the outside temperature is more than 20 degrees of rain, if the air conditioner is On, the air conditioner temperature is set too low, so it can be seen that the hand is cold, and the corresponding vehicle control action can store an increase of the air conditioner temperature 3 degrees. have.

If the outside temperature is more than 20 degrees of rain, when the air conditioner is off, it can be seen that it feels cold due to the rain, and the heater On can be stored as a corresponding vehicle control action.

If the outside temperature is over 20 degrees and there is no rain, if the air conditioner is turned on and the air conditioner wind direction is set to Up, the air conditioner wind can be regarded as the hand directly hitting the hand. You can save changing the wind direction to Down.

If the outside temperature is over 20 degrees and there is no rain, if the air conditioner is turned on and the air conditioner wind direction is set downward and the air volume is set to the middle or higher, the air conditioner wind direction is too strong to feel cold, and the corresponding vehicle control action It can save to lower air conditioner air volume.

If the rain does not rain when the outside temperature is more than 20 degrees, when the air conditioner is turned on, the air conditioner wind direction is set downward, and the air conditioner air volume is set low (lower), it is possible to store raising the air conditioner temperature by 3 degrees with the corresponding vehicle control action.

When the heater is turned off while the outside temperature is lower than 20 degrees, it can be seen that the hand is cold due to cold weather, and the on-state can be stored by the corresponding vehicle control action.

When the external temperature is lower than 20 degrees and the heater is turned on and the steering heating wire is turned off, the warmth is not transmitted to the hand, so it can be seen that the hand is cold, and the corresponding vehicle control action can be stored as the steering heating wire on.

When the outside temperature is lower than 20 degrees and both the heater and the steering heating wire are turned on, and the heater wind direction is set downward, the wind of the heater is not transmitted to the hand and it can be seen that the hand is cold, and the corresponding vehicle control action is the heater wind direction. It can be saved as a bidirectional change of.

If the outside temperature is lower than 20 degrees and both the heater and the steering heating wire are on, and the heater wind direction is set upward, if the heater temperature is set lower than the maximum, the corresponding vehicle control action can be stored as the heater temperature increaser. have.

If the outside temperature is lower than 20 degrees and both the heater and the steering heating wire are on, the heater wind direction is set upward, the heater temperature is set to the maximum value, and the heater air flow rate is not set to the maximum value, the corresponding vehicle The control action can be saved as increasing the heater wind volume.

If both the heater and the steering heating wire are turned on while the outside temperature is lower than 20 degrees, the heater wind direction is set upward, and both the heater temperature and the heater wind direction are set to the maximum value, the corresponding vehicle control is performed. You can save heated sheet on as an action.

When the outside temperature is lower than 20 degrees and both the heater and the steering heating wire are turned on, the heater wind direction is set upward, both the heater temperature and the heater air flow are set to the maximum value, and the heating wire sheet is also turned on, the corresponding vehicle control As the action, the heater is currently running, so you can output a guide to wait for a while.

28A and 28B are tables listing various examples in which the ambiguity resolver resolves ambiguities and extracts actions to control a vehicle by referring to the ambiguity resolution information DB.

For example, as illustrated in FIGS. 28A and 28B, when the content of ignition according to the morphological analysis results in cold or cold hands, the wind condition of the air conditioner is the head of the passenger while the surrounding situation is summer. If the air conditioner set temperature is 19 degrees, and the air volume is a situation, it can be seen that the hand is cold due to the air conditioner wind toward the hand. Accordingly, the air conditioner control action for reducing the air volume intensity while changing the wind direction toward the foot side (downward) may be extracted as an action corresponding to the corresponding ignition, and the vehicle may be controlled according to the extracted action.

In addition, in the ignition of the same content, when the surrounding situation is winter and the vehicle condition is the wind direction of the heater toward the passenger's foot, the air conditioner setting temperature is 25, and in a situation where the air volume is strongly set, the warmth is not delivered to the user's hand. It can be seen that the hand is cold. Therefore, the hot wire on action of the steering wheel may be extracted as an action corresponding to the utterance, and the vehicle may be controlled according to the extracted action.

In the case where the content of the utterance according to the morphological analysis results in frustration, when the vehicle condition is less than 30 km / h and the distance between the front and rear of the car is less than 30 cm, it may be determined that the frustration is felt by traffic jams. Accordingly, an action of changing a route option (fast route guidance), playing multimedia content such as music, or turning on a chat function in an action guide action as an action corresponding to the corresponding speech is extracted and extracted action. Can control the vehicle.

When the content of the utterance according to the morphological analysis results in sleepiness, when the vehicle state is in the betting mode, it can be understood that drowsiness is caused by the lack of air circulation. Accordingly, the action of changing to the outdoor mode may be extracted as an action corresponding to the utterance, and the vehicle may be controlled according to the extracted action.

In addition, in the utterance of the same content, when the vehicle state is in the outdoor mode and the heater is on (On), it can be seen that the drowsiness caused by the warmth emitted from the heater. Therefore, the window opening may be extracted as an action corresponding to the corresponding speech, and the vehicle may be controlled according to the extracted action.

 In case that the ignition content according to the morphological analysis results in sweat or hotness, if the surrounding condition is winter and the vehicle condition is the heater on state, it can be understood that the heat is caused by the warmth of the heater. . Therefore, the lowering of the temperature of the heater or the reduction of the amount of air may be stored as an action corresponding to the ignition.

In addition, when the surrounding situation is winter and the heater is off (Off) in the same utterance, it can be seen that the heat is caused by the user's body heat. Therefore, the window opening or the window opening proposal may be extracted as an action corresponding to the corresponding speech, and the vehicle may be controlled according to the extracted action.

In addition, when the surrounding condition is summer and the condition of the vehicle is the air conditioner is turned off in the same ignition, it can be seen that the heat caused by the high temperature inside the vehicle. Accordingly, the air conditioner execution On may be extracted as an action corresponding to the corresponding speech, and the vehicle may be controlled according to the extracted action.

In addition, when the surrounding condition is summer and the condition of the vehicle is in the condition that the air conditioner is turned on, the air conditioner is set to have a high temperature, so it can be understood that heat is caused. Therefore, the air conditioner temperature decrease or the air volume increase may be extracted as an action corresponding to the utterance, and the vehicle may be controlled according to the extracted action.

If the ignition according to the result of the morphological analysis is cold, if the surrounding situation is summer and the condition of the air conditioner is on, the air condition is set too low or the air conditioner wind is too strong. It can be seen that. Accordingly, the air conditioner temperature increase or air volume decrease may be extracted as an action corresponding to the utterance, and the vehicle may be controlled according to the extracted action.

In addition, when the surrounding condition is summer and the condition of the vehicle is the air conditioner is turned off in the same utterance, it can be determined that the user is in a cold condition due to a problem in the physical condition of the user. Accordingly, the execution of the heater or the biorhythm check of the user may be extracted as an action corresponding to the utterance, and the vehicle may be controlled according to the extracted action.

In addition, when the surrounding condition is winter and the vehicle is in a state in which the heater is turned on in the same ignition, it can be understood that the cold is caused because the temperature of the heater is set low or the air volume is weak. Therefore, the heater temperature increase or the air volume increase may be extracted as an action corresponding to the ignition, and the vehicle may be controlled according to the extracted action.

In addition, in the ignition of the same content, if the surrounding situation is the situation that the heater is off in winter, it can be seen that the cold caused by the non-execution of the heater. Accordingly, the heater execution may be extracted as an action corresponding to the utterance, and the vehicle may be controlled according to the extracted action.

If the ignition according to the morphological analysis results in the head hurts, if the surrounding situation is winter and the vehicle is in a state in which the heater is turned on, it can be understood that the headache is caused by the lack of air circulation. Therefore, the air mode change or window opening may be extracted as an action corresponding to the corresponding speech, and the vehicle may be controlled according to the extracted action.

In addition, when the surrounding situation is winter and the vehicle state is the heater is turned off in the same ignition, it can be seen that the headache caused by the cold. Accordingly, the heater execution may be extracted as an action corresponding to the utterance, and the vehicle may be controlled according to the extracted action.

In addition, if the surrounding condition is summer and the vehicle condition is off the air conditioner in the same ignition, it can be seen that the headache caused by the heat. Therefore, the air conditioner execution may be extracted as an action corresponding to the utterance, and the vehicle may be controlled according to the extracted action.

In addition, when the surrounding condition is summer and the vehicle condition is on (On) in the same ignition, it can be seen that the headache caused by the cooling disease. Accordingly, it is possible to extract a change in the wind direction or the air volume of the air conditioner as an action corresponding to the utterance, and control the vehicle according to the extracted action.

In the case that the content of the utterance according to the morphological analysis results are awful, if the surrounding situation is winter and rainy conditions, it can be understood that the high humidity causes unpleasantness. Therefore, the execution of the defog function or the execution of the dehumidification function can be extracted as an action corresponding to the utterance, and the vehicle can be controlled according to the extracted action.

In addition, if the surrounding situation is summer and no rain in the same ignition, it can be seen that the discomfort caused by seasonal characteristics and heat. Therefore, running the air conditioner at the lowest temperature may extract the action corresponding to the utterance, and control the vehicle according to the extracted action.

In addition, when the surrounding situation is summer and rainy conditions in the same ignition, it can be seen that the discomfort caused by the heat and high humidity. Therefore, running the air conditioner in the dehumidification mode may extract the action corresponding to the utterance, and control the vehicle according to the extracted action.

According to the operation of the ambiguity solver 123 described above, even when there is an ambiguity in the user's speech or situation, the surrounding situation information and the vehicle state information are considered together with the user's speech to actually give the user the desired action or action. Accurately identify and provide the required action.

Information about the action determined by the ambiguity resolver 123 may be transmitted to the conversation action manager 122, and the conversation action manager 122 may update the conversation / action DB 147 based on the transmitted information. .

The action priority determiner 125 and the argument manager 124 determine an action execution condition, determine a priority, and obtain an argument value as described above with respect to the action determined by the ambiguity resolver 123. do.

If all of the argument values used to perform each action are available through the current situation and the dialogue, the dialogue action manager 122 sends a signal to the dialogue flow manager 121.

On the other hand, the parameter values necessary for performing the action or determining the condition do not exist in the dialogue / action DB 147, the external content server 300, the long term memory 143, the short term memory 144, and the context information DB 142. In the case where it can be obtained only through the user, the result processor 130 may generate a conversation response for querying the user with the argument value.

The conversation flow manager 121 transmits information on the action corresponding to the first priority and the conversation state to the result processor 130. It is also possible to convey information about a plurality of candidate actions according to the dialogue policy.

When the conversation system 100 outputs the starter, that is, when the starter trigger signal is generated in the input processor 110, the starter trigger signal may be included in the conversation state transmitted to the result processor 130. However, the start state trigger signal does not necessarily need to be included in the conversation state, and any type of information that may indicate that the start situation may be included in the conversation state. When the conversation state includes information indicating that the starting situation is included, the result processor 130 may output the conversation response in preference to the other type of response or may be output together with the other type of response.

In addition, when the conversation system 100 outputs the starter, when the starter message corresponding to the starter situation is input from the dialog input manager 111c, the above-mentioned ambiguity resolution, factor management, action ranking, and the like are described. The selection message may be transmitted to the result processor 130 without going through the process of.

In addition, in the case where the conversation system 100 outputs the starter, when an action corresponding to the starter situation is input from the dialog input manager 111c, the above-described ambiguity resolution, factor management, and action ranking are determined. It is also possible to pass through, and it is also possible to deliver the action corresponding to the starting situation to the result processor 130 without going through this process.

Fig. 29 is a control block diagram of the configuration of the result processor.

Referring to FIG. 29, the result processor 130 may generate a text according to a request of the response generation manager 131 and the response generation manager 131 that manage generation of a response required to perform an action input from the conversation manager 120. A conversation response generator 132 for generating a response, an image response or an audio response, a command generator 136 for generating a command for controlling a vehicle or a command for providing a service using external content according to a request of the response generation manager 131. ), A service editor 134 for executing a plurality of services sequentially or sporadically and collecting result values to provide a service desired by a user, and outputting a generated text response, image response or audio response, or the command generator 136 An output manager 133 and a response generation manager 131 which output the generated command and determine the output order when there are a plurality of outputs; It based on the output of the output manager 133, and includes a long-term memory 143 and the short-term memory 144, the corresponding manual DB accident (148b), a memory manager 135 to manage the vehicle functions DB (146f).

The result processor 130 may include a memory for storing a program for performing the above-described or later operations, and a processor for executing the stored program. At least one memory and a processor may be provided, and in the case of a plurality of processors, they may be integrated on one chip or physically separated.

In addition, each component included in the result processor 130 may be implemented by a single processor or may be implemented by a separate processor.

In addition, the result processor 130, the conversation manager 120, and the input processor 110 may also be implemented by a single processor, or may be implemented by separate processors.

The response output in response to the user's speech or situation may include a conversation response, vehicle control, and external content provision. The conversation response may be in the form of an initial conversation, a query, an answer including providing information, and the like, and may be stored in a database in the response template 149.

The response generation manager 131 requests the conversation response generator 132 and the command generator 136 to generate a response required to perform the action determined by the conversation manager 120. To this end, information about an action to be performed may be transmitted to the conversation response generator 132 and the command generator 136, and the information about the action to be performed may include an action name, an argument value, and the like. In generating the response, the conversation response generator 132 and the command generator 136 may refer to the current conversation state and the action state.

The conversation response generator 132 may search the response template 149 to extract a conversation response form, and generate a conversation response by filling in argument values necessary for the extracted conversation response form. The generated conversation response is transmitted to the response generation manager 131. If a parameter value required to generate a chat response is not delivered from the chat manager 120 or an instruction to use external content is received from the external content server 300 or the long-term memory 143, the short-term memory 144, or the like. The situation information DB 142 can be searched.

For example, if the action determined in the conversation manager 120 corresponds to the directions, search the response template 149 and search for "time required:-" from "[current location:-] to [destination:-]". Is expected. "Can we start the announcement?"

Among the factors to be filled in the dialogue response form, the argument values of [the current position] and the [destination] are transmitted from the dialogue manager 120, and the argument values of the [time required] may not be transmitted. In this case, the conversation response generator 132 may request the external content server 300 from the current location to the destination.

In addition, when the Q & A list 146g has a question or answer related to a user's speech or response to a situation, the conversation response generator 132 may extract a question form by referring to the corresponding question or answer in the Q & A list 146g. You can also create

For example, when the action determined by the conversation manager 120 corresponds to the depression index determination, the conversation response generator 132 may extract, “How are you feeling today?” As the conversation response form by selecting the command. 136 generates a command for executing the response to the user's utterance or the situation includes vehicle control or external content provision. For example, when the action determined by the conversation manager 120 is a control of an air conditioner, a window, a sheet, or an AVN, a command for executing the control is generated and transmitted to the response generation manager 131.

Alternatively, when the action determined by the conversation manager 120 requires the provision of external content, a command for receiving the corresponding content from the external content server 300 is generated and transmitted to the response generation manager 131.

In addition, if there is a question or answer related to the user's speech or response to the situation in the Q & A list 146g, the command generator 136 may generate a command by referring to the question or answer.

For example, when the action determined by the conversation manager 120 corresponds to the depression index determination, and the depression index determination action requires biorhythm information, a command for receiving the information is generated to generate a situation information collection manager 112b. To pass).

In addition, when the action determined by the conversation manager 120 is related to at least one of the accident response manual DB 148b and the vehicle function DB 146f, the command generator 136 may refer to the related information stored in the DB to execute the command. Can be generated.

For example, if the action determined by the conversation manager 120 is to call a family in response to accident prevention, a command for performing a call function may be generated and transmitted to the AVN 255.

When there are a plurality of commands generated by the command generator 136, the service editor 134 determines the method and order of executing the plurality of commands and transmits them to the response generation manager 131.

The response generation manager 131 transfers the response received from the conversation response generator 132, the command generator 136, or the service editor 134 to the output manager 133.

The output manager 133 determines an output timing, an output order, an output position, and the like of the dialogue response generated by the dialogue response generator 132 and the instruction generated by the instruction generator 136.

The output manager 133 outputs the response by transmitting the conversation response generated by the response generator 132 and the command generated by the command generator 136 to an appropriate output location in an appropriate order at an appropriate timing. The text to speech (TTS) response may be output through the speaker 232, and the text response may be output through the display 231. When outputting a conversation response in a TTS form, the TTS module provided in the vehicle 200 may be used or the output manager 133 may include a TTS module.

The command may be transmitted to the vehicle controller 240 according to the control target, or may be transmitted to the communication device 280 for communicating with the external content server 300.

The response generation manager 131 may also transmit the response received from the conversation response generator 132, the command generator 136, or the service editor 134 to the memory manager 135.

In addition, the output manager 133 may also transmit a response outputted by the output manager 133 to the memory manager 135.

The memory manager 135 manages the long term memory 143 and the short term memory 144 based on the contents received from the response generation manager 131 and the output manager 133. For example, the memory manager 135 may update the short-term memory 144 by storing the conversation contents between the user and the system based on the generated and outputted conversation response, and may obtain user-related information obtained through the conversation with the user. The long term memory 143 may be updated by storing the same.

In addition, information stored in the short-term memory 144 may be stored in the long-term memory 143 such as meaningful and persistent information such as a user's disposition or preference, or information that may be used to obtain such information.

In addition, the user's preference or vehicle control history stored in the long-term memory 143 may be updated based on the vehicle control or the external content request corresponding to the generated and output command.

On the other hand, when the conversation system 100 outputs the selection prior to the user's speech input, when the action corresponding to the selection situation is input from the conversation input manager 111c, the information regarding the action is received. The conversation response generator 132 may search for the response template 149 to extract a conversation response form, and generate a conversation response by filling in argument values required for the extracted conversation response form. The generated conversation response is transmitted to the response generation manager 131. The conversation response here is the start of the conversation system 100.

The response generation manager 131 transfers the conversation response received from the conversation response generator 132 to the output manager 133.

The output manager 133 may output the conversation response generated by the conversation response generator 132 through the speaker 232.

When the result processor 130 receives the selection message itself corresponding to the selection situation from the conversation flow manager 121, the input selection message is a conversation response, and the input selection message is an output manager 133. Can be delivered.

The output manager 133 may output the delivered starter message through the speaker 232.

In addition, if the user's utterance is input after the conversation system 100 outputs the starter, the operation of processing the user's utterance described above may be performed in the same manner.

According to the dialogue system 100 according to the above-described embodiment, it is possible to provide an optimal service necessary for a user in consideration of various situations occurring inside the vehicle. In particular, even if the user's utterance is not input, it is possible to determine a service necessary for the user based on the situation information or the driver information collected by the conversation system 100 and provide it preemptively.

For example, it is possible to vary the evaluation criteria of the vehicle state according to the situation at the time of starting the vehicle and to provide feedback about this preemptively. The starting point of driving is defined as when the vehicle is started, when the EPB (Electronic Parking Brake) is released, or when a navigation destination is set. Weights are applied to the vehicle condition evaluation system and the individual device for calculating the driving possible score, and the variable weight applied to the individual device is changed according to the situation factor. If it is determined that there is a problem with the vehicle condition, it may be possible to provide a solution for the individual device, such as a workshop guidance.

In addition, when the vehicle is authorized to start, it may be determined whether the flow rate is insufficient compared to the destination, and if the flow rate is insufficient, the user may add the user's preferred gas station as an automatic waypoint in the route to the destination, and guide the user. In addition, the gas station to be added to the automatic waypoint according to the user's response may be changed.

In addition, even if the current state of the vehicle does not indicate a lack of flow rate, it is possible to preemptively provide a fueling location or time point in consideration of the user's future schedule, major movement records, remaining amount, and the like.

In addition, it is possible to obtain information regarding the physical condition of the driver and the sleep record, and conditionally permit the vehicle starting authorization based on the obtained information. For example, non-driving may be recommended if the driver may recognize drowsy driving by recognizing a physical condition and a sleep record outside the vehicle. Or, depending on your physical condition or sleep record, you can provide information about recommended driving times.

In addition, when a trigger indicating drowsy driving possibility repeatedly occurs, the drowsy driving possibility may be detected, and a feedback may be provided to output a warning or automatically change (rest stop) according to the degree of the detected possibility. The trigger indicating drowsy driving possibility may be acquired by manually measuring the driver's state or the state of the vehicle, such as when the heartbeat decreases, when the distance to the preceding vehicle is greater than or equal to the reference distance, and when the speed is less than or equal to the reference speed. It is also possible to proactively acquire through conversations, such as firing a question to the driver and measuring the driver's response speed.

In addition, when a user inputs a speech that expresses emotion, a specific domain or action cannot be extracted from the user's speech, but the conversation system 100 uses the surrounding situation information, the vehicle status information, the user status information, and the like. Identify intentions and develop conversations. This example may be performed by solving the ambiguity of user speech in the ambiguity resolver 230 as described above.

Hereinafter, an example of a specific conversation process using the conversation system 100 according to an embodiment will be described in detail.

30 to 42 illustrate a specific example in which the conversation system 100 processes an input, manages a conversation, and outputs a result when a user inputs a utterance related to a road guidance.

As shown in FIG. 30, when the user inputs a speech of "Let's go to Seoul Station yesterday", the voice recognizer 111a outputs the user's voice as a text-type speech (let's go to Seoul Station yesterday).

The natural language understander 111b analyzes the morphemes, referring to the domain / action inference rule DB 141, from the morphological analysis results (yesterday / NNG, liver / VV, Seoul station / NNP, Gaza / VV). ], [Action: directions], [speech act: request], [factor: NLU destination: Seoul Station] can be extracted and input to the conversation input manager (111c).

Referring to FIG. 31, the conversation input manager 111c transmits a natural language understanding result of the natural language reader 111b to the contextual reader 112c and requests to send additional information.

The context comparator 112c may search the context comprehension table 145 to extract that context information related to [domain: navigation] and [action: directions] is a current location and a context information type is a GPS value.

The context comparator 112c searches the context information DB 142 and extracts the GPS value of the current location. When the GPS value of the current position is not stored in the situation information DB 142, the GPS information of the current position is requested to the situation information collection manager 112b.

The situation information collection manager 112b sends a signal to the situation information collector 112a to collect the GPS value of the current location, and the situation information collector 112a collects the GPS value from the vehicle controller 240 and collects the situation information DB ( In step 142, the GPS value collection confirmation signal is transmitted to the situation information collection manager 112b. When the contextual information collection manager 112b transmits the GPS value collection confirmation signal to the contextual understanding 112c, the contextual understanding 112c extracts the GPS value of the current location from the contextual information DB 142 and uses the dialog input manager ( 111c).

Conversation input manager 111c is the result of understanding natural language [Domain: Navigation], [Action: Directions], [Action: Request], [Factor: NLU: Destination: Seoul Station] and [Status information: Current location: Uiwang Station (GPS) Value)] to the conversation manager 120.

Referring to FIG. 32, the conversation flow manager 121 searches the conversation / action state DB 147 to determine whether there is a conversation task or an action task in progress. At this time, the dialogue policy DB 148 may be referred to. In this example, it is assumed that there is no conversation task or action task in progress.

The dialog flow manager 121 requests the dialog action manager 122 to generate an action task and a dialog task corresponding to the output of the input processor 110. Creating an action task and a conversation task means allocating a storage space for storing and managing information about the action state and the conversation state.

Accordingly, the dialogue action manager 122 allocates a storage space in the dialogue / action state DB 147 to store information about the action state and information about the conversation state.

The conversation action manager 122 communicates the action state and the conversation state to the action priority determiner 125.

The action priority determiner 125 retrieves a vehicle status check and a gas station recommendation, which are actions associated with road guidance, in the associated action DB 146b. The road guide action and the association action become candidate actions.

The action priority determiner 125 may determine the priority of each candidate action according to a pre-stored rule. The priority may be determined before the execution condition for each candidate action is determined, or may be determined only for candidate actions satisfying the execution condition after the execution condition is determined.

The candidate action list is passed back to the conversation action manager 122, which adds the retrieved association actions to update the action state.

Referring to FIG. 33A, the action priority determiner 125 searches for an execution condition for each candidate action and a factor for determining an execution condition in the action execution condition DB 146c. It may also determine the priority between each candidate action.

For example, the condition for performing the vehicle condition check may be the case where the destination distance is 100 km or more, and a factor required for determining the execution condition is the destination distance.

The condition for performing the gas station recommendation may be a case where the destination distance is farther than the driving distance (DTE), and the factors necessary for determining the execution condition are the destination distance and the driving distance.

The dialog action manager 122 updates the action state by adding a condition for performing each candidate action and a factor necessary for condition determination to the dialog / action state DB 147.

The action priority determiner 125 determines the argument values necessary for determining whether the candidate action satisfies the execution condition. The conversation / action state DB 147, the situation information DB 142, the long-term memory 143, or the short-term memory 144. Can be retrieved from).

If the argument value is included in the previous conversation content, the context information related to the conversation content, or included in the redemption information related to the generated event, the argument value may be obtained from the conversation / action status DB 147.

If the argument value cannot be obtained from the conversation / action status DB 147, the situation information DB 142, the long term memory 143, or the short term memory 144, the external information manager 126 may be requested.

For example, the destination distance may be obtained from the external content server 300 providing the navigation service through the external information manager 126, and the DTE may be obtained from the context information DB 142. Meanwhile, in order to search for a destination distance, accurate destination information used in a navigation service is required. In this example, the destination input from the user's speech corresponds to "Seoul Station", which may represent various kinds of places starting with Seoul Station or may be Seoul Station having a specific meaning. Therefore, it is difficult to search the exact destination distance only by "Seoul station".

In addition, it is possible to obtain the print value from the mobile device 400 connected to the vehicle 200 as needed. For example, if user information, such as a contact or schedule, which is not stored in the long-term memory 143, is required as a parameter value, the external information manager 126 requests necessary information from the mobile device 400 to obtain necessary parameter values. can do.

In addition, when the factor value cannot be obtained through the storage 140, the external content server 300, and the mobile device 400, the user may obtain a necessary factor value by querying the user.

The action priority determiner 125 determines an execution condition of each candidate action using the factor value. Since the destination distance has not been retrieved, the determination of the execution condition is suspended for the vehicle condition check action and the gas station recommendation action.

33B illustrates a method of determining priorities for actions retrieved from the diversion domain.

For example, referring to FIG. 33B, the action priority determiner 125 determines an execution condition and an execution condition for each candidate action in the action execution condition DB 146c and the vehicle function DB 146f. Search. It may also determine the priority between each candidate action.

For example, the condition for performing the window opening is when the vehicle internal temperature is out of range, and a factor necessary for determining the execution condition may be the internal temperature.

The condition for performing the rest area guidance should be that the vehicle should be capable of autonomous driving, and the necessary factor for determining the performance condition should be when the autonomous driving function is installed.

The dialog action manager 122 updates the action state by adding a condition for performing each candidate action and a factor necessary for condition determination to the dialog / action state DB 147.

The action priority determiner 125 determines the argument values necessary for determining whether the candidate action satisfies the execution condition. The conversation / action state DB 147, the situation information DB 142, the long-term memory 143, or the short-term memory 144. Can be retrieved from).

When the argument value is included in the previous conversation content, the context information related to the conversation content, or the context information related to the generated event, the argument value may be obtained from the conversation / action status DB 147.

If the argument value cannot be obtained from the conversation / action status DB 147, the situation information DB 142, the long term memory 143, or the short term memory 144, the external information manager 126 may be requested.

Therefore, the action priority determiner 125 may suspend the determination of the execution condition for the action recommending the rest area guidance if the vehicle is not equipped with the autonomous driving function.

As illustrated in FIG. 34, the dialogue action manager 122 updates the action state by adding the acquired argument value to the conversation / action state DB 147 and whether the action execution condition determined by using the corresponding argument value is satisfied. .

The dialog action manager 122 requests the argument manager 124 for a list of factors used to perform each candidate action.

The parameter manager 124 may extract the current location and the destination as an essential factor used to perform the road guidance action from the action factor 146a, and may extract a path type (initial value: fast path) as an optional factor.

An inspection part (initial value: all parts) may be extracted as an optional parameter used to perform a vehicle condition inspection action, and a preferred gas station (initial value: A oil) may be extracted as an optional parameter used to perform a gas station recommendation action. .

The extracted argument list can be passed to dialog action manager 122 and used to update the action state.

The argument manager 124 may be configured to acquire the argument values corresponding to the required and optional arguments of each candidate action, the dialogue / action status DB 147, the situation information DB 142, the long term memory 143, and the short term memory 144. In the reference position of each factor, the relevant factor value is searched out, and if the factor value is to be provided through an external service, an external factor manager 126 may request a necessary factor value from the external content server 300.

The factors used to determine the execution condition of the candidate action and the factors used to perform the candidate action may overlap each other. If any of the argument values obtained by the action priority determiner 125 and stored in the conversation / action state DB 147 correspond to the factors (mandatory and optional factors) used to perform the candidate action, they may be used.

Referring to FIG. 35, the dialogue action manager 122 updates the action state by adding a factor value obtained by the argument manager 124.

As described above, there is an ambiguity that the destination (Seoul station) extracted from the user's speech is used as a factor of the road guide action. Therefore, the values of the factors of the road guidance action (destination), the vehicle condition checking action (destination distance) and the parameters of the gas station recommended action (destination distance) have not yet been obtained.

The ambiguity resolver 123 checks whether there is ambiguity in converting [Factor: NLU: Destination: Seoul Station] into a destination factor suitable for the road guidance action. As described above, "Seoul Station" may mean another kind of place starting with Seoul Station, or may be Seoul Station of a specific location that the user means.

The ambiguity resolver 123 determines that there is a modifier of Seoul station during user utterance with reference to the result of the morpheme analysis. The ambiguity resolver 123 searches for a schedule, a moving location, a contact point, etc. in the long-term memory 143 and the short-term memory 144 in order to determine the location of “Seoul Station in Seoul yesterday”.

For example, the ambiguity resolver 123 may confirm that "Seoul station between yesterday" is "Seoul station exit 4" from the user's moving position yesterday. Check that there is a POI at "Exit 4 of Seoul Station" and get the value.

The destination information obtained by the ambiguity resolver 123 is transmitted to the dialogue action manager 122, and the dialogue action manager 122 adds "Seoul Station 4 exit" to the destination factor of the candidate action to update the action state.

The argument manager 124 gets destination information (Seoul Station 4 exit) from the conversation / action DB 147 and provides a destination distance value to an external content server 300 that provides a navigation service through the external information manager 126. request.

Referring to FIG. 36, when the external information manager 126 obtains a destination distance value (80 km) from the external content server 300 and transmits the destination distance value (80 km) to the parameter manager 124, the parameter manager 124 may communicate this to the conversation action manager 122. ) So that the action status can be updated.

The action priority determiner 125 may determine whether the candidate action may be performed with reference to the action state and adjust the priority. The road guidance action may be performed because the current location and the parameter values of the destination, which are essential factors, are acquired, and the vehicle condition checking action may be determined to be impossible because the destination distance (70 km) is less than 100 km. The gas station recommendation action may be determined to be possible because the destination distance (80 km) is farther than the driving distance (DTE).

The vehicle condition check action cannot be performed and therefore is excluded from prioritization. Therefore, the road guidance action is adjusted to priority 1 and the gas station recommendation action is adjusted to priority 2.

The dialog action manager 122 updates the action state according to whether the candidate action is performed and the adjusted priority.

The conversation flow manager 121 confirms the conversation state and action state stored in the conversation / action state DB 147, and develops a conversation strategy for conducting a conversation with reference to the conversation policy DB 148. For example, the response generation manager 131 may be selected to select an action having the highest priority among the actions that can be performed, and generate a response for conducting a conversation according to the dialogue policy DB 148.

The conversation state and action state stored in the conversation / action DB 147 may be updated with [Status: Direction Start Confirmation].

Referring to FIG. 37, the response generation manager 131 requests the response response generator 132 to generate a response in response to a request of the conversation flow manager 121.

The conversation response generator 132 searches the response template 149 to generate a TTS response and a text response. For example, you can generate a conversation response that can be printed in TTS and text: "It is expected to take 30 minutes from Uiwang Station to Exit 4 of Seoul Station.

The response generation manager 131 transmits the TTS response and the text response generated by the conversation response generator 132 to the output manager 133 and the memory manager 135, and the output manager 133 transmits the TTS response to the speaker 232. And a text response to the display 231. At this time, the output manager 133 may transmit the TTS response to the speaker 232 via the TTS module for synthesizing the text into voice.

The memory manager 135 may store in the short term memory 144 or the long term memory 143 that the user requested a directions.

Through the display 231 and the speaker 232, a dialog response is output, asking, "It is expected to take 30 minutes from Uiwang Station to Exit 4 of Seoul Station. Do you want to start the guidance?" As shown in FIG. 38, when the user speaks "Yes", the user's speech is inputted to the speech recognizer 111a and output as [text: yes], and the natural language reader 111b is read from the user's speech. Domain:-], [action:-], [speech act:-], [morphological analysis result: yes / IC].

The natural language understanding result is transmitted to the conversation input manager 111c, and the conversation input manager 111c delivers it to the conversation manager 120. FIG.

Referring to FIG. 39, the conversation flow manager 121 searches the conversation / action state DB to analyze a previous conversation state and requests the conversation action manager 122 to update an ongoing [direction of directions] related conversation / action. .

The dialogue action manager 122 updates the dialogue state and the action state to [state: start of directions].

The dialog flow manager 121 requests the result processor 130 to generate a response for starting the directions.

Referring to FIG. 40, the dialogue action manager 122 updates the dialogue state to [state: next conversation progress], and updates the action state to [state: execute].

The dialog flow manager 121 requests the response generation manager 131 to generate a response for guiding a road.

The response generator 132 may generate a conversation response capable of outputting the TTS and the text as “starts the directions” and transmits the conversation response to the response generation manager 131.

The command generator 136 may generate a command (target: navigation, command: directions, destination: Seoul Station Exit 4, departure place: Uiwang Station) for performing the road guidance and transmit the generated command to the response generation manager 131.

The response generation manager 131 transfers the generated conversation response and command to the output manager 133. The output manager 133 outputs a conversation response through the display 231 and the speaker 232, and the directions guidance command is transmitted to the AVN 230 of the vehicle 200 or provides a navigation service through the control unit 240. It may transmit to the external content server 300.

Referring to FIG. 41, the conversation flow manager 121 next selects a gas station recommendation as an executable action, and requests the response generation manager 131 to generate a response inquiring the user of the gas station recommendation.

The conversation state and the action state may be updated to [Status: Confirmation of Linked Service Recommendation].

The response generation manager 131 requests the conversation response generator 132 to generate a TTS response and a text response, and the conversation response generator 132 runs out of fuel for moving to the destination. Would you like to add an oil station A to the stop? "Can be generated and transmitted to the response generation manager 131 to generate a conversation response that can be output as a TTS and text.

The response generation manager 131 transmits the TTS response and the text response generated by the conversation response generator 132 to the output manager 133 and the memory manager 135, and the output manager 133 transmits the TTS response to the speaker 232. And a text response to the display 231.

Through the display 231 and the speaker 232, a dialogue response is output, asking, "There is not enough fuel to move to the destination. Do you want to add A oil service station to the waypoint?" As shown in FIG. 42, when the user speaks "no", the user's speech is inputted to the speech recognizer 111a and output as [text: no], and the natural language interpreter 111b is read from the user's speech. Outputs domain:-], [action:-], [speech act:-], [morphological analysis result: no / IC].

The dialogue flow manager 121 requests the dialogue action manager 122 to update the dialogue state and the action state.

The conversation action manager 122 may update the conversation state to [state: next conversation progress], and update the action state to [state: CANCEL].

The conversation flow manager 121 requests the response generation manager 131 to generate a response indicating that the service station recommendation service has been canceled, and checks whether there is a conversation that can proceed next. If no further progress is available, update the conversation status to [Status: IDLE] and wait for user input.

The flow of data processing described above is just one example applied to the dialogue system 100. Therefore, the order in which the components of the conversation system 100 process data is not limited to the above-described example, and it is also possible for a plurality of components to process the data at the same time. It is also possible.

Hereinafter, a conversation processing method according to an embodiment will be described. In the conversation processing method according to an exemplary embodiment, the above-described conversation system 100 or the vehicle 200 including the same may be applied. Therefore, the description of FIGS. 1 to 42 may be equally applied to the method of processing a conversation according to an exemplary embodiment.

43 is a flowchart illustrating a method of processing user input in a conversation processing method according to an embodiment. The method of processing the user input may be performed by the input processor 110 of the conversation system 100.

Referring to FIG. 43, when a user's speech is input (Yes of 500), the tone analyzer 111d analyzes the user's tone (505). In addition, when the user's speech is input (Yes of 500), the voice recognizer 111a recognizes the input user's speech (510). The user's speech may be input through the voice input device 210 provided in the vehicle 200 or the voice input device 410 provided in the mobile device 400.

The tone analyzer 111d may analyze the tone of the user to determine whether a tone lower than usual is input, and output tone analysis information including the determination result.

The voice recognizer 111a recognizes the input speech of the user and outputs the speech in a text form.

The natural language reader 111b applies the natural language understanding technique to the text-type spoken text (520) and outputs the result.

Specifically, the natural language understanding process 520 performs a morphological analysis 521 based on the spoken text in a text form, extracts a domain 522 based on the morphological analysis result, recognizes an entity name 523, Analyzing the speech act 524, extracting the action 525, and extracting the factor 526.

The domain / action inference rule DB 141 may be referred to for domain extraction, entity name recognition, and action extraction.

The output of the natural language comparator 111b, that is, the natural language comprehension result, may include a domain, an action, a speech act, a morphological analysis result, etc. corresponding to the user's speech. The situation information related to the extracted action may be stored in the situation understanding table 145, and the situation understanding 112c retrieves the situation information related to the action extracted from the situation understanding table 145, and the information of the retrieved situation information. The value is taken from the situation information DB 142, the long term memory 143 or the short term memory 144.

If additional situation information is needed (example of 530), that is, if there is situation information that cannot be obtained from the situation information DB 142, the long term memory 143, or the short term memory 144, the relevant situation information may be requested by collecting the situation information. The situation information is collected (540). Non-voice input such as vehicle status information, surrounding environment information, driver information, and the like may be input through the situation information collector 111a independently of the user's speech input.

The natural language understanding result and the collected situation information about the user's answer may be transmitted to the response analyzer 111e (550). The response analyzer 111e may compare the processing result received from the natural language reader 111b with the Q & A data stored in the Q & A list 146g, and analyze the user's response by determining the degree of depression of the user based on the comparison result. Can be. The response analyzer 111e may determine the depression index by quantifying the degree of depression of the user (560), and the depression index is transmitted to the conversation input manager 111c.

Such information may be input periodically, or may be input when a specific event occurs, or may be additionally input when the specific event occurs periodically. In either case, when the collection of information is requested, the information can be actively collected.

Therefore, when the situation information related to the action has already been collected, the information is taken from the situation information DB 142, the long term memory 143, or the short term memory 144. Otherwise, the situation information collector 111a is taken. Collect this information through.

When the situation information collector 111a, which is requested to collect the situation information, collects the relevant situation information and stores the situation information in the situation information DB 142, the situation understanding unit 112c may obtain the situation information from the situation information DB 142. have.

Meanwhile, the situation information collection manager 112b may transmit an action trigger signal to the situation understander 112c when it is determined that a specific event occurs because the data collected by the situation information collector 112a satisfies a preset condition.

The context comparator 112c searches the context comprehension table 145 to search for context information related to the corresponding event, and if the searched context information is not stored, transmits a request signal of the context information to the context information collection manager 112b. do.

When the collection of the necessary situation information is completed, the natural language understanding result and the situation information is transmitted to the conversation manager 120. When an event occurs, information related to the event (what event occurred) and situation information related to the generated event may be transmitted together. 44A and 44B are flowcharts illustrating a method of managing a conversation using an output of an input processor according to an embodiment. The method of managing a conversation may be performed in the conversation manager 120 of the conversation system 100.

Referring to FIG. 44A, the conversation flow manager 121 searches 600 for a conversation history in a conversation / action status DB 147.

In this example, a case in which the domain and the action can be extracted from the user's speech will be described as an example. However, in some cases, the domain and the action may not be extracted from the user's speech because the content or situation of the speech is ambiguous. In this case, the conversation action manager 122 generates an arbitrary conversation state, and the ambiguity solver 123 grasps the user's intention based on the user's speech contents, surrounding conditions, vehicle status, user information, and the like, Determine the appropriate action.

If the relevant conversation history exists (YES of 600), it is referenced 690, and if it does not exist (NO of 600), a new conversation task and action task are created (610).

The association action DB 146b generates a candidate action list by searching for an association action list associated with an action extracted from the user's speech (hereinafter referred to as an input action) (620). The input action and the actions associated with it become a candidate action list.

The execution condition for each candidate action is searched for in the action execution condition DB 146c (620). The execution condition is a condition necessary to perform the action. If the corresponding condition is satisfied, it is determined that the action can be performed. The action execution condition DB 146c also stores information on the type of the factor used for determining the action execution condition.

An argument value used to determine an action execution condition is obtained (640). A factor used to determine an action execution condition will be referred to as a condition judgment factor. The condition value DB 142, the long term memory 143, the short term memory 144, or the conversation / action state DB 147 may be searched to obtain a factor value of the condition determination factor. When the argument value of the condition determination factor is to be provided through an external service, the necessary factor value may be provided from the external content server 300 through the external information manager 126.

In addition, when a situation or speech is ambiguous and a necessary factor value cannot be obtained, the ambiguity resolver 230 may be used to resolve the ambiguity to obtain a necessary factor value.

In addition, even if the acquired factor is an ineffective factor that is difficult to use in determining the action execution condition, the ambiguity solver 230 may obtain an effective factor from the ineffective factor.

Based on the obtained condition determination factor, it is determined whether to perform each candidate action (650), and the priority of candidate actions is determined (660). The rule for determining the priority of candidate actions may be stored in advance. After determining whether to perform an action, the action priority determiner 125 may determine the priority by considering only candidate actions capable of performing the action. Alternatively, after determining the priority regardless of whether the actions are performed, the priority may be adjusted based on the action execution.

The action factor DB 146a retrieves a list of factors used to perform candidate actions (670). An argument used to perform an action is called an action argument. Action arguments can include required and optional arguments.

An argument value of an action argument used to perform a candidate action is obtained (680). The argument value of the action factor may be obtained by searching the context information DB 142, the long term memory 143, the short term memory 144, or the conversation / action state DB 147. When the argument value of the action factor is to be provided through an external service, the necessary argument value may be provided from the external content server 300 through the external information manager 126.

In addition, when a situation or speech is ambiguous and a necessary factor value cannot be obtained, the ambiguity resolver 230 may be used to resolve the ambiguity to obtain a necessary factor value.

In addition, even when the acquired factor is an ineffective factor that is difficult to use in performing an action, the ambiguity solver 230 may obtain a valid factor from the ineffective factor.

The conversation state and action state managed by the conversation action manager 122 may be updated whenever the state changes as the above-described steps are performed.

When all of the obtainable argument values are acquired, the conversation flow manager 121 transmits information on the candidate action and the conversation state to the result processor 130. According to the dialogue policy, only information on actions corresponding to the first rank may be delivered, and information on a plurality of candidate actions may be delivered.

On the other hand, if the necessary parameter value does not exist in the external content server 300, the long term memory 143, the short term memory 144, and the context information DB 142, and can be obtained only through the user, It is also possible to output a dialogue response to query.

Referring to FIG. 44B, the dangerous situation manager 119 may determine whether the dangerous situation, that is, whether the user is in a dangerous situation (850). In detail, the dangerous situation manager 119 may determine whether the dangerous situation is based on at least one of the collected situation information and the user's depression index.

If it is determined that the dangerous situation, the conversation flow manager 121 may search (851) whether there is an associated accident response manual in the accident response manual DB 148b.

If there is an associated accident response manual (Yes of 851), the associated accident response manual stored in the accident response manual DB 148b, that is, a dialogue task and an action task for responding to a dangerous situation may be referred to (852).

If the associated incident response manual does not exist (NO in 852), a new conversation task and action task may be created (853). In this case, new conversation tasks and action tasks may be generated based on the data stored in the accident response manual DB 148b.

In the accident response manual DB 148b, a list of related actions associated with the determined risk situation may be retrieved, and a candidate action list may be generated (854). In this case, the accident response manual DB 148b may further store an action list including at least one action for responding to a dangerous situation, and the actions associated with the dangerous situation become a candidate action list.

The execution condition for each candidate action may be searched for in the action execution condition DB 146c (855). At this time, the vehicle function DB 146f may also search for the execution condition for each candidate action. The vehicle function DB 146f may store information on a function that can be provided by the vehicle, and also store information on the type of a factor used to determine an execution condition for the function that can be provided.

An argument value used for determining an action execution condition may be acquired (856), whether to perform execution for each candidate action based on the obtained condition determination factor (857), and the priority of candidate actions may be determined (858).

In the action factor DB 146a, the argument list used to perform the candidate action may be retrieved (859), and an argument value of the action factor used to perform the candidate action may be obtained (860).

45 is a flowchart illustrating a result processing method for generating a response corresponding to a result of conversation management in the conversation processing method according to an exemplary embodiment. The result processing method may be performed by the result processor 130 of the conversation system 100.

Referring to FIG. 45, when generation of a chat response is required (Yes of 700), the chat response generator 132 searches for a response template 149 (710). The response template 149 extracts a dialogue response form corresponding to the current dialogue state and action state, and generates a dialogue response by filling in argument values necessary for the dialogue response form (720).

If a parameter value required to generate a chat response is not delivered from the chat manager 120 or an instruction to use external content is received from the external content server 300 or the long-term memory 143, the short-term memory 144, or the like. The situation information DB 142 can be searched. If the necessary argument values do not exist in the external content server 300, the long term memory 143, the short term memory 144, and the context information DB 142, and can only be obtained through the user, query the user with the argument values. It is also possible to generate a conversation response for the user.

If command generation is required (760), the command generator 136 generates a command for vehicle control or external content use (770).

The generated conversation response or command is input to the output manager 133, and the output manager 133 may determine an output order between the conversation response and the command or an output order between the plurality of commands (730).

The memory is updated based on the generated conversation response or command (740). The memory manager 135 may update the short-term memory 144 by storing the conversation contents between the user and the system based on the generated and outputted dialogue response, and may store user-related information obtained through the dialogue with the user for a long term. The memory 143 may be updated. In addition, the user's preference or vehicle control history stored in the long-term memory 143 may be updated based on the generated and output vehicle control or an external content request.

The output manager 133 outputs the response by sending the conversation response and the command to an appropriate output location (750). The TTS response may be output through the speaker 232 and the text response may be output through the display 231. The command may be transmitted to the vehicle controller 240, the external content server 300, or the communication device 280 for communicating with the external content server 300 according to the control target. .

46 to 48 are flowcharts illustrating a case in which a conversation system outputs a selection prior to a user's speech input in a conversation processing method according to an exemplary embodiment.

Referring to FIG. 46, the situation collector 112a and the situation collection manager 112b collect situation information (810). In detail, the vehicle controller 240 may display the vehicle status information and the surrounding environment information obtained from a sensor provided in the vehicle, such as residual flow rate, rainfall, rainfall speed, ambient obstacle information, speed, engine temperature, tire pressure, and current location. The user information input to the non-speech input device 220 or the information obtained from the external content server 300 or an external device may also be input to the situation processor 112. The collected situation information may be stored in the situation information DB 142, the long term memory 143, or the short term memory 144.

The starter determination unit 151 determines the starter condition based on the situation information (S811). The starting conditions may be stored in the starting conditions table 145a. As illustrated in FIGS. 22A to 22C described above, the starting condition related to the situation information may be stored for each situation information in the starting condition table 145a.

The starter determination unit 151 determines that the starter situation is satisfied when the situation information transmitted from the situation information DB 142, the long term memory 143, or the short term memory 144 satisfies the starter condition (Yes of 812). In operation 813, the start trigger signal is generated.

The starter determiner 151 extracts an action corresponding to the starter situation (814). As illustrated in FIG. 22C described above, the action corresponding to the selection situation may be previously stored in the selection condition table 145a. The starter determiner 151 may extract an action corresponding to the starter situation from the starter condition table 145a. Alternatively, the starter determiner 151 may generate an action corresponding to the starter situation according to a predetermined rule.

When the starter determiner 151 transmits the start trigger signal and the action corresponding to the starter situation to the conversation input manager 111c, the conversation input manager 111c corresponds to the starter situation to the conversation manager 120. Pass the action (815). At this time, it is also possible to transmit a start trigger signal or a signal indicating a start situation.

After the action corresponding to the starting situation is transferred to the conversation manager 120, a series of processes for generating a conversation task and an action task and obtaining an action factor may be performed as illustrated in FIG. 44. At this time, when another chat task or action task is being performed, the chat flow manager 121 may preferentially generate and process a task for the starting situation, and determine the priority according to a predetermined rule. .

When the conversation manager 120 first transmits information about an action to be processed to the result processor 130, the response generator 132 may retrieve the response template 149 and extract a conversation response form. The response generator 132 may generate a conversation response by filling in a factor value required for the extracted response form, and the generated conversation response may be delivered to the output manager 133 through the response generation manager 131. The output manager 133 may output the generated conversation response through a speaker provided in the vehicle 200 or the mobile device 400.

Alternatively, it is also possible to extract or generate the starting message itself corresponding to the starting situation. Referring to FIG. 47, the situation collector 112a and the situation collection manager 112b collect situation information (820), and the starter determiner 151 determines the starting condition based on the situation information (821). .

The starter determination unit 151 determines that the starter situation is satisfied when the situation information transmitted from the situation information DB 142, the long term memory 143, or the short term memory 144 satisfies the starter condition (YES in 822). In operation 823, the starter determiner 151 extracts a starter message corresponding to the starter situation. As illustrated in FIGS. 22A and 22B, the starting condition table 145a may store a starting message corresponding to each starting situation in advance. The starter message stored in advance may be content indicating a current situation, or may be content that first suggests execution of a specific function or service required for the starter situation. Alternatively, the starter determiner 151 may generate the starter message according to a predetermined rule.

When the starter determiner 151 transfers the starter trigger signal and the starter message to the conversation input manager 111c, the conversation input manager 111c transfers the starter message to the conversation manager 120 (825). At this time, it is also possible to transmit a start trigger signal or a signal indicating a start situation.

The conversation manager 120 may generate a conversation task for outputting the delivered starter message, and deliver the starter message to the result processor 130. The result processor 130 may output the input starter message through the speaker 232.

Alternatively, the speech of the virtual user corresponding to the starting situation may be extracted or generated. Referring to FIG. 48, the situation collector 112a and the situation collection manager 112b collect situation information (830), and the starter determiner 151 determines the start condition based on the situation information (831). .

The situation understanding 112c determines that the situation information DB 142, the long-term memory 143, or the short-term memory 144 satisfies the selection condition (Yes of 832), and is a selection situation. A start trigger signal is generated (833).

The starter determiner 151 extracts a virtual user speech corresponding to the starter situation (834). Although not shown in the drawing, a virtual user speech corresponding to the starting condition may be stored in advance in the starting condition table 145a. The starter determiner 151 may extract a virtual user speech corresponding to the starter situation from the starter condition table 145a. Alternatively, the starter determiner 151 may generate a virtual user speech corresponding to the starter situation according to a predetermined rule.

When the starter determination unit 151 transmits the virtual user utterance in text form to the natural language reader 111b (835), the natural language reader 111b receives the domain from the virtual user utterance as if the user actually uttered it. , Extract actions, etc.

The conversation input manager 111c transmits the start trigger signal and the natural language understanding result to the conversation manager 120 (836). Here, the natural language understanding result may include a domain, an action, and the like extracted from the virtual user speech, and the extracted domain and the action may be a domain or action corresponding to the selection situation.

For example, in the mobile gateway method in which the mobile device 400 serves as a gateway between the vehicle 100 and the conversation system 100, the conversation system client 470 of the mobile device 400 may select a starter determiner. Some of operations of 151 may be performed. In this case, the conversation system client 470 may generate a virtual user speech corresponding to the selection situation and transmit the virtual user speech to the natural language reader 111b.

After the start trigger signal and the natural language understanding result are transmitted to the conversation manager 120, a series of processes for generating a conversation task and an action task and obtaining an action factor may be performed as illustrated in FIG. 44. At this time, when another chat task or action task is being performed, the chat flow manager 121 may preferentially generate and process a task for the starting situation, and determine the priority according to a predetermined rule. .

When the conversation manager 120 first transmits information about an action to be processed to the result processor 130, the response generator 132 may retrieve the response template 149 and extract a conversation response form. The response generator 132 may generate a conversation response by filling in a factor value required for the extracted response form, and the generated conversation response may be delivered to the output manager 133 through the response generation manager 131. The output manager 133 may output the generated conversation response through a speaker provided in the vehicle 200 or the mobile device 400.

49 is a flowchart illustrating a process of processing a duplicate task when the conversation system outputs a selection prior to inputting a speech by a user according to an embodiment.

Referring to FIG. 49, the situation collector 112a and the situation collection manager 112b collect situation information (840), and the starter determiner 151 determines the start condition based on the situation information (841). .

The starter determination unit 151 determines whether the situation information transmitted from the situation information DB 142, the long term memory 143, or the short term memory 144 satisfies the starting condition, and if the starting condition is satisfied ( In operation 842, the duplicate task processor 152 determines whether a task related to the current starting situation is duplicated (843).

In detail, the duplicate task processor 152 may have already completed a task such as a conversation, an action, etc. related to the current starting situation based on the information about the task processed or in progress in the conversation system 100 stored in the task processing DB 145b. It can be determined whether it has been processed or is in progress.

For example, if a conversation related to the current starting situation has already been performed and the reference time has not elapsed since the execution of the conversation, the task related to the current starting situation may be determined as a duplicate task. Alternatively, a duplicate task may be determined even when a conversation or an action related to the current starting situation occurs.

That is, the duplicate task processor 152 may determine whether the starting information is already output, the user's intention about the starting situation, and the like, based on the conversation history stored in the task processing DB 145b and whether the task is performed. It may be determined whether a duplicate task is based on the time of the conversation, the intention of the user, or whether the task is performed.

If it is determined that the task related to the current starting situation is a duplicate task (YES in 843), the duplicate task processor 152 ends the starting situation.

If it is determined that the task is not a duplicate task (NO in 843), the start operation may be performed as in the above-described embodiment (844). For example, the starter trigger signal and the action or starter message corresponding to the starter situation may be transmitted to the conversation manager 120. Alternatively, the virtual user utterance corresponding to the starting situation may be transmitted to the natural language reader 111b, and the natural language understanding result and the starting trigger signal may be transmitted to the conversation manager 120.

In the above-described embodiment, separate components such as the starter determination unit 151 and the redundant task processor 152, the starter condition table 145a, the task processing DB It is assumed that a separate storage such as 145b) is used. However, the embodiment of the conversation processing method is not limited thereto, and the situation understander 112c performs the operations of the selection determiner 151 and the duplicate task processor 152 or selects the situation understanding table 145. Information stored in the condition table 145a and the task processing DB 145b may be stored.

The conversation processing method according to an embodiment is not limited by the order in the above-described flowchart. 41 to 49 are only examples of flows that can be applied to the conversation processing method, a plurality of steps may be performed simultaneously, and the order of the steps may be changed.

The above description is merely illustrative of the technical idea, and various modifications, changes, and substitutions may be made by those skilled in the art without departing from the essential characteristics. Accordingly, the above-described embodiments and the accompanying drawings are not intended to limit the technical spirit, but rather to describe it, and the scope of the technical spirit is not limited by the embodiments and the accompanying drawings. The scope of protection shall be interpreted by the following claims, and all technical ideas within the scope of equivalent shall be interpreted as being included in the scope of rights.

According to the dialog system described above, the vehicle and the conversation processing method including the same, it is possible to provide an optimal service that meets the intention of the user or is necessary for the user by using the conversation processing technology specialized for the vehicle environment.

In addition, in consideration of various situations occurring inside the vehicle, it is possible to provide an optimal service required by the user. In particular, regardless of the user's utterance, it is possible to determine the service required by the user based on the situation information or the driver information collected by the conversation system 100 and provide it preemptively.

100: conversation system
110: input handler
120: conversation manager
130: Result handler
200: vehicle
210: voice input device
220: non-voice input device
230: dialog output device
280: communication device

Claims (20)

A storage unit for storing situation information including at least one of vehicle state information related to a state of a vehicle, user information related to a driver of the vehicle, and driving environment information related to a driving environment of the vehicle; And
Determining whether the user is in a starting situation for determining a degree of depression of the user based on the situation information, and if determined to be the starting situation, acquiring an action for performing the starting, and And an input processor that determines a degree of depression of the user based on a response. The method of claim 1,
The input processor,
And a tone analyzer configured to receive a voice of a user and perform tone analysis on the input voice of the user.
And the user information of the situation information includes the tone analysis result. The method of claim 1,
The storage unit,
At least one question for determining a degree of depression of the user, at least one answer thereto, and at least one depression index corresponding to the at least one answer;
The input processor,
And if the starting condition is satisfied, acquiring an action for performing the starting based on the stored at least one question. The method of claim 3,
The input processor,
And the user's response to the selection based on the stored at least one question to determine the degree of depression of the user based on the stored at least one answer. The method of claim 4,
The input processor,
And acquire a depression index of the user based on at least one depression index corresponding to the stored at least one answer, and determine a degree of depression of the user based on the obtained depression index. The method of claim 1,
And a risk situation manager for determining whether the user is in a dangerous situation based on at least one of the user's depression and the situation information, and obtaining an action corresponding to the dangerous situation. . The method of claim 6,
The storage unit,
Save an incident response manual that includes at least one action that responds to a dangerous situation in which an accident due to depression is expected,
The risk situation manager,
And a dialog system for obtaining an action corresponding to the dangerous situation based on the stored accident response manual. The method of claim 7, wherein
The risk situation manager,
And if determined to be a dangerous situation, determine a dangerous situation step and obtain an action corresponding to the dangerous situation based on the determined dangerous situation step. The method of claim 1,
And a conversation manager for obtaining from the storage at least one candidate action related to the action obtained by the input processor. The method of claim 9,
The storage unit,
Store vehicle feature information related to the vehicle's supportable features,
The conversation manager,
And determine whether to perform the at least one candidate action based on the stored vehicle function information, and determine the priority of the at least one candidate action based on the determination result. Store state information including at least one of vehicle state information related to a state of a vehicle, user information related to a driver of the vehicle, and driving environment information related to a driving environment of the vehicle;
Determining whether the user is in a starting situation for determining a degree of depression of the user based on the situation information;
If it is determined that the starting situation is obtained, acquire an action for performing the starting method; And
And determining the degree of depression of the user based on the user's response to the selection. The method of claim 11,
Receiving a voice of a user and executing tone analysis on the input voice of the user;
And the user information of the situation information includes the tone analysis result. The method of claim 11,
Storing at least one question for determining the degree of depression of the user, at least one answer thereto, and at least one depression index corresponding to the at least one answer;
If it is determined that the starting situation is to obtain the action for performing the starting,
And if the selection condition is satisfied, obtaining an action for performing the selection based on the stored at least one question. The method of claim 13,
Determining the degree of depression of the user based on the user's response to the selection,
And determining the degree of depression of the user based on the stored at least one answer by the user's answer to the selection based on the stored at least one question. The method of claim 14,
Determining the degree of depression of the user based on the user's response to the selection,
And obtaining the depression index of the user based on at least one depression index corresponding to the stored at least one answer, and determining the degree of depression of the user based on the obtained depression index. The method of claim 11,
Determining whether the user is at risk based on at least one of the degree of depression of the user and the situation information; And
If it is determined that the dangerous situation, acquiring an action corresponding to the dangerous situation. The method of claim 16,
Storing an accident response manual that includes at least one action corresponding to a dangerous situation in which an accident due to depression is anticipated;
If it is determined that the dangerous situation is to obtain an action corresponding to the dangerous situation,
And obtaining an action corresponding to the dangerous situation based on the stored accident response manual. The method of claim 17,
If it is determined that the dangerous situation is to obtain an action corresponding to the dangerous situation,
If it is determined that the dangerous situation is determined, the step of determining a dangerous situation, and obtaining an action corresponding to the dangerous situation based on the determined dangerous situation step. The method of claim 11,
Acquiring at least one candidate action related to the obtained action. The method of claim 19,
Store vehicle function information related to a supportable function of the vehicle; And
And determining whether to perform the at least one candidate action based on the stored vehicle function information, and determining the priority of the at least one candidate action based on the determination result.

Images