KR102403355B1 - Vehicle, mobile for communicate with the vehicle and method for controlling the vehicle - Google Patents

Abstract

The vehicle of the present invention includes a voice input device for receiving a user's voice; an image input device for photographing a road and its surroundings and outputting the photographed image; Receives and stores the captured image, converts the received user's voice into text, recognizes the user's intention and notification situation using the converted text and natural language understanding technology, and based on the recognized user's intention and notification situation, the user a dialogue system for controlling conversation with and transmission of stored images; an output device for outputting utterances for conversation with the user; and a communication device for transmitting situation information of a notification situation and a stored image.

Description

A vehicle, a mobile device communicating therewith, and a method of controlling the vehicle

The present invention relates to a vehicle, a mobile device that communicates with the vehicle, and a method for controlling a vehicle by identifying a user's intention through a conversation with the user and providing information or service necessary for the user.

An in-vehicle audio video navigation device (AVN) or most mobile devices may experience inconvenience in providing visual information to a user or receiving a user's input due to a small screen and small buttons.

In particular, moving the user's gaze and taking their hands off the steering wheel in order to check visual information or operate devices while driving acts as a threat to safe driving.

Therefore, it is expected that the conversation system that understands the user's intention through conversation with the user and provides the necessary service to the user is applied to the vehicle to provide the service more safely and conveniently.

(Patent Document 1) KR2013-0035983 A

One aspect is a vehicle that recognizes the user's voice and if the recognized voice is a voice for a notification situation, acquires road situation information and transmits the obtained road situation information to a server of a responsible agency corresponding to the situation information, and communicates with the vehicle A mobile device and a vehicle control method are provided.

In the other aspect, by accurately identifying the user's intention based on various information such as conversations with the user in the vehicle driving environment, vehicle status information, driving environment information, and user information, the service that meets the user's actual intention or the most necessary service for the user A vehicle capable of providing a service, a mobile device communicating therewith, and a vehicle control method are provided.

A vehicle according to one aspect includes a voice input device for receiving a user's voice; Conversation system that converts the received user's voice into text, recognizes the user's intention and notification situation using the converted text and natural language understanding technology, and controls conversation performance with the user based on the recognized user's intent and notification situation ; an output device for outputting utterances for conversation with the user; and a communication device that transmits context information about the notification state.

The vehicle according to an aspect further includes an image input device for photographing a road and its surroundings, and a controller for receiving and storing the captured image, and controlling the communication device to transmit situation information of a notification situation and the stored image.

The notification situation of the vehicle according to one aspect includes at least one of a traffic accident situation and an illegal action situation, and the dialog system determines whether the notification situation is a traffic accident situation or an illegal action, and provides an agency in charge of responding to the determined situation. and controlling the communication device to check and transmit the situation information and the image to the server of the verified authority.

The dialogue system of a vehicle according to one aspect determines whether a notification situation has occurred based on a voice, and when it is determined that a notification situation has occurred, recognizes the vehicle number of a traffic accident vehicle or illegal act vehicle in the image, and transmits the recognized vehicle number includes controlling the

The dialogue system of a vehicle according to an aspect includes performing a confirmation query for confirming the coincidence of the recognized vehicle number when the vehicle number recognition of the traffic accident vehicle or the illegal act vehicle is successful.

The dialogue system of a vehicle according to an aspect includes performing a vehicle number query when the vehicle number recognition of a traffic accident vehicle or an illegal vehicle is unsuccessful.

The dialogue system of a vehicle according to an aspect determines whether a notification situation has occurred based on a voice, and when it is determined that a notification situation has occurred, performs a query for the license plate number of a traffic accident vehicle or illegally engaged vehicle, and receives the input voice device and recognizing the vehicle number based on the voice.

A communication device for a vehicle according to an aspect includes a GPS receiver, and the conversation system includes matching and storing location information when storing an image, and controlling transmission of the matched image and location information when controlling image transmission.

A conversation system for a vehicle according to an aspect includes matching and storing date and time information when storing an image, and controlling transmission of the matched image, date and time information when controlling image transmission.

The vehicle conversation system according to one aspect identifies a time point at which a notification voice corresponding to a notification situation is input, and a time point before a first predetermined time as the confirmed time point and a time point after the second predetermined time has elapsed from the confirmed time point confirming, acquiring only images captured between a time before and after a time point at which a notification voice is input among stored images, and controlling transmission of the acquired image.

According to another aspect, a vehicle may include a voice input device configured to receive a user's voice; an image input device for photographing a road and its surroundings and outputting the photographed image; a controller for storing the captured image and controlling the output of the received voice and the stored voice; It communicates with the controller, converts the user's voice into text, uses the converted text and natural language understanding technology to determine whether it is a notification situation, and if it is a notification situation, performs a conversation with the user to recognize the situation information of the notification situation and recognize a dialogue system that transmits context information to the controller; an output device for outputting an utterance for a conversation with a user based on a command of the conversation system; and a communication device for transmitting situation information of a notification situation and a stored image on the basis of a command from a controller.

The notification situation of the vehicle according to another aspect includes at least one of a traffic accident situation and an illegal action situation, and the dialog system determines whether the notification situation is a traffic accident situation or an illegal action, and sends an agency in charge of responding to the determined situation. includes checking

The controller of the vehicle according to another aspect includes recognizing the vehicle number of the traffic accident vehicle or the illegal act vehicle in the image when a notification situation occurs, and controlling the transmission of the recognized vehicle number.

A communication device for a vehicle according to another aspect includes a GPS receiver, and the controller includes matching and storing location information when storing an image, and controlling transmission of the matched image and location information when controlling image transmission.

A vehicle controller according to another aspect includes matching and storing date and time information when storing an image, and controlling transmission of the matched image, date and time information when controlling image transmission.

The controller of the vehicle according to another aspect checks the time when the notification voice corresponding to the notification situation is input, and determines the time before the first predetermined time as the confirmed time and the time after the second predetermined time has elapsed from the confirmed time Checking includes acquiring only images captured between a time before and after a time point at which a notification voice is input among stored images and controlling transmission of the acquired image.

A mobile device according to another aspect includes a communication device for communicating with a vehicle and a server; a voice input device for receiving a user's voice; It converts the user's voice into text, uses the converted text and natural language understanding technology to determine whether it is a notification situation, and if it is a notification situation, conducts a conversation with the user and recognizes the situation information of the notification situation based on the voice received during the conversation a conversation system for requesting an image to be provided, and controlling to transmit the recognized situation information and the image transmitted from the vehicle to the server; an output device for outputting utterances for a conversation with the user based on a command of the dialogue system, wherein the communication device transmits situation information and an image of a notification situation based on a command of the dialogue system, and the notification situation is a traffic accident situation and including at least one of the circumstances of the tort.

A conversation system of a mobile device according to another aspect includes recognizing the license plate number of a vehicle in a traffic accident or illegal activity in an image and controlling transmission of the recognized license plate number.

A conversation system of a mobile device according to another aspect includes receiving at least one of location information, time information, and date information when an image is received from a vehicle.

A vehicle control method according to another aspect includes capturing a road and its surroundings, storing the captured image together with time information and location information, and converting the user's voice into text when a user's voice is input to a voice input device Then, using the converted text and natural language understanding technology, it is determined whether it is a notification situation, and when it is determined that it is a notification situation, an utterance corresponding to the notification situation is output through an output device, and a notification situation is based on the voice input to the voice input device. , and transmits the acquired context information, location information, time information, and the stored image.

Determining the notification situation includes determining whether the notification situation is a traffic accident situation or an illegal act.

Transmitting the acquired context information and the stored image includes identifying an institution in charge corresponding to the determined context, and transmitting the context information and the image to a server of the identified institution in charge.

Acquiring the situation information includes recognizing and acquiring the license plate number of the traffic accident vehicle or illegal act vehicle in the image when it is determined that the notification situation has occurred.

Transmission of the stored video is performed by checking the time when the notification voice corresponding to the notification situation is input, and the time point before the first predetermined time as the confirmed time point and the time point after the second predetermined time has elapsed from the confirmed time point checking, acquiring only images captured between a time before and after a time point at which a notification voice is input among stored images, and controlling transmission of the acquired image.

According to the vehicle, the mobile device communicating with the vehicle, and the vehicle control method according to one aspect, when an accident or illegal activity is witnessed by the user, the image of the accident site or the image of the illegal activity vehicle is handled through conversation with the user By transmitting to the server of the institution, operations such as video shooting, editing, and transmission that require manual work can be omitted, and thus, convenience for users can be improved.

In addition, in one aspect, it is possible to increase the convenience of reporting accidents and illegal activities, and to encourage participation in reporting accidents and illegal activities, thereby increasing the safety of the road environment.

According to another aspect, by accurately identifying the user's intentions based on various information such as conversations with the user in the vehicle driving environment, vehicle status information, driving environment information, and user information, the best service or service that meets the user's actual intention is provided. We can provide you with the services you need.

1 is a control block diagram of a conversation system according to an embodiment.
2 is a view showing the configuration of the interior of the vehicle.
3 to 5 are diagrams illustrating examples of conversations that can be exchanged between a conversation system and a driver.
6 and 7 are control block diagrams schematically illustrating the connection relationship between the dialogue system and the components of the vehicle.
8 and 9 are control block diagrams schematically illustrating a connection relationship between components of a dialogue system and components of a vehicle.
10 is a control block diagram for a vehicle-only method in which a conversation system is provided in a vehicle.
11 and 12 are control block diagrams of a vehicle gateway method in which a dialogue system is provided in a remote server and the vehicle serves only as a gateway connecting the user and the dialogue system.
13 is a control block diagram illustrating a case in which a vehicle may partially perform input processing and output processing in a vehicle gateway method.
14 is a control block diagram of a hybrid method in which both a remote conversation system server and a vehicle can perform conversation processing.
15 and 16 are control block diagrams of a mobile gateway method in which a mobile device connected to a vehicle connects a user and a remote conversation system server.
17 is a control block diagram of a mobile-only method in which a conversation system is mounted on a mobile device.
18, 19A, and 19B are control block diagrams in which the configuration of the input processor is subdivided in the configuration of the conversation system.
20A and 20B are diagrams illustrating examples of information stored in a context understanding table.
21 is a control block diagram in which the configuration of a conversation manager is subdivided.
22 is a diagram illustrating an example of information stored in a related action DB.
23 is a diagram illustrating an example of information stored in an action execution condition DB.
24 is a diagram illustrating an example of information stored in an action factor DB.
25 is a diagram illustrating an example of information stored in an ambiguity resolution information DB.
26A and 26B are tables summarizing various examples in which the ambiguity resolver refers to the ambiguity resolving information DB to resolve ambiguity and extract actions to control the vehicle.
27 is a control block diagram in which the configuration of the result processor is subdivided.
28 to 40 are diagrams illustrating specific examples in which the dialog system processes the input, manages the dialog, and outputs a result when the user inputs an utterance related to road guidance.
41 is a flowchart illustrating a method of processing a user input in a conversation processing method according to an embodiment.
42 is a flowchart illustrating a method of managing a conversation using an output of an input processor in a conversation processing method according to an embodiment.
43 is a flowchart illustrating a result processing method for generating a response corresponding to a result of conversation management in the conversation processing method according to an embodiment.
44 is a configuration diagram of a vehicle according to another exemplary embodiment.
45 is a detailed configuration diagram of the in-vehicle conversation system shown in FIG. 44 .
46 is a configuration diagram of a vehicle according to another exemplary embodiment.
47 is a detailed configuration diagram of the in-vehicle conversation system shown in FIG. 46 .
48 is a control configuration diagram of an in-vehicle conversation system according to another exemplary embodiment.
49 to 53 are diagrams illustrating conversations between an in-vehicle conversation system and a user according to another embodiment.
54 is a configuration diagram of a vehicle and a mobile device communicating with the vehicle according to another embodiment.

Like reference numerals refer to like elements throughout. This specification does not describe all elements of the embodiments, and general content in the technical field to which the present invention pertains or content that overlaps between the embodiments is omitted. The term 'part, module, member, block' used in this specification may be implemented in software or hardware, and according to embodiments, a plurality of 'part, module, member, block' may be implemented as one component, It is also possible for one 'part, module, member, block' to include a plurality of components.

Throughout the specification, when a part is "connected" to another part, it includes not only direct connection but also indirect connection, and indirect connection includes connection through a wireless communication network. do.

In addition, when a part "includes" a certain component, this means that other components may be further included, rather than excluding other components, unless otherwise stated.

The singular expression includes the plural expression unless the context clearly dictates otherwise.

The signs attached to each step are used to identify each step, and these signs do not indicate the order between the steps, and each step is performed differently from the stated order unless the context clearly indicates a specific order. can be

Hereinafter, an embodiment of a conversation system, a vehicle including the same, and a conversation processing method will be described in detail with reference to the accompanying drawings.

A conversation system according to an embodiment is an apparatus for identifying a user's intention by using the user's voice and non-voice input and providing a service suitable for the user's intention or a service necessary to the user, as a means of providing a service or the user's intention As one means for clearly grasping , a conversation with the user may be performed by outputting a system utterance.

In this embodiment, the service provided to the user includes all operations performed to meet the user's needs or intentions, such as providing information, controlling a vehicle, executing an audio/video/navigation function, and providing content obtained from an external server. may include

In addition, the dialog system according to an exemplary embodiment may accurately identify the user's intention in the special environment of the vehicle by providing a dialog processing technology specialized for the vehicle environment.

The gateway connecting the user with this conversation system may be a vehicle or a mobile device connected to the vehicle. As will be described later, the conversation system may be provided in the vehicle, or it may be provided in a remote server outside the vehicle and exchange data through communication with the vehicle or a mobile device connected to the vehicle.

In addition, it is also possible that some of the components of the chat system are provided in the vehicle and some are provided in the remote server to partially perform the operation of the chat system in the vehicle and the remote server.

1 is a control block diagram of a conversation system according to an embodiment.

Referring to FIG. 1 , the dialog system 100 according to an exemplary embodiment includes an input processor 110 that processes a user input including a user's voice and non-voice input or an input including vehicle-related information or user-related information. ), the dialog manager 120 for determining the user's intention using the processing result of the input processor 110, and determining an action corresponding to the user's intention or the state of the vehicle, according to the output result of the dialog manager 120 The result processor 130 for providing a specific service or outputting a system utterance for continuing a conversation, and a storage unit 140 for storing various types of information necessary for the dialog system 100 to perform an operation to be described later are included.

The input processor 110 may receive two types of input, a user's voice and a non-voice input. The non-voice input includes a user's non-voice input input through gesture recognition or manipulation of an input device, vehicle state information indicating the state of the vehicle, driving environment information related to the driving environment of the vehicle, and user information indicating the user's state and the like. In addition to these information, any information related to the vehicle and the user may be input to the input processor 110 as long as it is information that can be used to identify a user's intention or provide a service to the user. Users may include both drivers and passengers.

The input processor 110 recognizes the inputted user's voice, converts it into a text-type utterance, and applies a natural language understanding technology to the user's utterance to understand the user's intention.

In addition, the input processor 110 collects information related to the state of the vehicle or driving environment in addition to the user's voice, and understands the situation by using the collected information.

The input processor 110 transmits information related to the user's intention and situation identified through natural language understanding to the conversation manager 120 .

The dialog manager 120 determines an action corresponding to the user's intention and the current situation based on the user's intention and situation-related information transmitted from the input processor 110 , and manages factors necessary to perform the corresponding action. .

In this embodiment, an action may mean all actions performed to provide a specific service, and the type of action may be predefined. In some cases, provision of a service and execution of an action may have the same meaning.

For example, actions such as route guidance, vehicle condition check, gas station recommendation, etc. may be predefined in the domain/action inference rule DB (141, see FIG. 19A), and actions corresponding to the user's utterance according to the stored reasoning rules; That is, an action intended by the user may be extracted from predefined actions. In addition, an action related to an event occurring in the vehicle may be predefined and stored in the related action DB 146b (refer to FIG. 21 ).

There is no limitation on the type of action, and the dialog system 100 is predefined as what can be performed through the vehicle 200 or the mobile device 400, and the reasoning rules and relationships with other actions/events are stored. If it is, it can be an action.

The conversation manager 120 transmits information about the determined action to the result processor 130 .

The result processor 130 generates and outputs a dialogue response and a command necessary to perform the delivered action. The dialogue response may be output as text, image, or audio, and when the command is output, a service such as vehicle control and external content provision corresponding to the output command may be performed.

The storage unit 140 stores various types of information required for conversation processing and service provision. For example, information related to a domain, action, dialogue act, and entity name used for understanding natural language may be stored in advance, a situation understanding table used to understand a situation from input information may be stored, and a sensor provided in the vehicle may detect One data, information related to the user, and information necessary to perform an action may be stored in advance. A more detailed description of the information stored in the storage unit 140 will be described later.

As described above, the conversation system 100 provides a conversation processing technology specialized for a vehicle environment. All of the components of the dialogue system 100 may be included in the vehicle, or only some of the components may be included. The conversation system 100 may be provided on a remote server and the vehicle may only serve as a gateway between the conversation system 100 and the user. In either case, the conversation system 100 may be connected to the user through a vehicle or a mobile device connected to the vehicle.

2 is a view showing the configuration of the interior of the vehicle.

Referring to FIG. 2 , in the center fascia 203 , which is the central area of the dashboard 201 inside the vehicle 200 , it is necessary to control the vehicle including an audio function, a video function, a navigation function, or a dialing function. A display 231 for displaying a screen and an input button 221 for receiving a user's control command may be provided.

In addition, an input button 223 may be provided on the steering wheel 207 for the driver's convenience in operation, and serves as an input button in the center console area 202 between the driver's seat 254a and the passenger seat 254b. A jog shuttle 225 may be provided.

A module including the display 231 , the input button 221 , and a processor for generally controlling various functions may be referred to as an AVN (Audio Video Navigation) terminal or a head unit.

The display 231 may be implemented as one of various display devices such as a liquid crystal display (LCD), a light emitting diode (LED), a plasma display panel (PDP), an organic light emitting diode (OLED), and a cathode ray tube (CRT). have.

The input button 221 may be provided as a hard key type in an area adjacent to the display 231 as shown in FIG. 2 , and when the display 231 is implemented as a touch screen type, the display 231 is an input button The function of (221) can also be performed together.

The vehicle 1 may receive a user's command as a voice through the voice input device 210 . The voice input device 210 may include a microphone that receives sound, converts it into an electrical signal, and outputs it.

For effective voice input, the voice input device 210 may be provided on the headlining 205 as shown in FIG. 2 , but the embodiment of the vehicle 200 is not limited thereto. It is also possible to be provided or provided on the steering wheel (207). In addition to this, there is no limitation wherever it is suitable for receiving the user's voice input.

A speaker 232 for outputting a sound necessary to perform a conversation with a user or provide a service desired by the user may be provided inside the vehicle 200 . For example, the speaker 232 may be provided inside the driver's door 253a and the passenger's door 253b.

The speaker 232 may output a voice for guiding a navigation route, a sound or voice included in audio/video content, a voice for providing information or service desired by the user, a system utterance generated as a response to the user's utterance, etc. can

The dialog system 100 according to an embodiment provides a service optimized for a user's lifestyle by using a dialog processing technology specialized for a vehicle environment, and provides a connected car (Connected Car), Internet of Things (IoT), artificial intelligence (AI) A new service can be configured using technologies such as AI).

When a dialog processing technology specialized for a vehicle environment is applied, such as the dialog system 100 according to an embodiment, it is easy to recognize and respond to a main context in a driver's direct driving situation. The service can be provided by giving weight to factors that affect driving, such as insufficient flow and drowsy driving, and in most cases, information necessary for service provision, such as driving time and destination information, can be easily provided based on the condition of moving to the destination. can be obtained

In addition, it is possible to easily implement an intelligent service that identifies the driver's intention and suggests a function. This is because real-time information and actions are prioritized in the driver's direct driving situation. For example, if the driver searches for a gas station while driving, this may be interpreted as the driver's intention to go to the gas station now. However, when searching for a gas station in a non-vehicle environment, it is possible to interpret various other intentions such as location information inquiry, phone number inquiry, and price inquiry in addition to the intention to go to the gas station now.

Also, although a vehicle is a limited space, various situations may exist within it. For example, the dialog system 100 may be utilized in a driving situation of an unfamiliar interface such as a rental car, a situation in which a substitute driving is entrusted, a vehicle management situation such as car washing, a situation in which a child is boarded, a situation in which a specific destination is visited, and the like.

In addition, opportunities for various services and dialogue situations occur in each stage constituting the driving of the vehicle and before and after, such as the vehicle inspection stage, the departure preparation stage, the driving stage, and the parking stage. In particular, the situation where you do not know how to deal with vehicle problems, the situation where the vehicle and various external devices are interlocked, the driving habits such as fuel economy, the use of safety support functions such as smart cruise control, the navigation operation situation, the drowsy driving situation, The dialogue system 100 may be utilized in a situation in which the same route is repeatedly driven every day or in a situation where it is necessary to check whether parking or stopping is possible.

3 to 5 are diagrams illustrating examples of conversations that can be exchanged between a conversation system and a driver.

Referring to FIG. 3 , when the driver inputs an utterance asking for the current residual oil amount (U1: how much oil is left?), the dialogue system 100 provides information on the current residual oil amount (S1: to the current remaining oil) 43 km driving is possible).

The driver may input an utterance asking for a nearby gas station to receive route guidance to a gas station (U2: tell me about a nearby gas station), and the conversation system 100 provides information about the nearest gas station from the current location (S2: presently) You can print out A Oil Seongrim Gas Station, B Oil Jangdae Gas Station, and C Oil Pacific Gas Station as gas stations close to the location.

The driver may additionally input an utterance to request information on oil price (U3: where is the cheapest place?), and the dialogue system 100 provides information on the price by oil type (S3: gasoline) The lowest price is at B Oil Jangdae gas station, which is 1,294 won per liter, and the place with the lowest diesel price is A Oil Seongrim gas station, which is 985 won per liter) can be printed.

The driver may input an utterance requesting guidance to the B Oil Pole gas station (U4), and the dialog system 100 outputs an utterance that the driver starts guiding to the selected gas station (S4: Start guiding to the B Oil Pole gas station) can do.

That is, the driver can receive route guidance to a nearby gas station that sells the current vehicle type at the lowest price through conversation with the conversation system 100 .

Meanwhile, when the dialog system 100 selects a gas station in the example of FIG. 3 , it is possible to shorten the steps and time of the conversation by omitting some queries and providing information directly.

For example, the conversation system 100 may know in advance the information that the fuel type of the current vehicle is gasoline and that the driver's gas station selection criterion is price. Information on the fuel type of the vehicle may be obtained from the vehicle, and the driver's gas station selection criteria may be obtained by receiving input from the driver in advance or by learning the driver's conversation history or gas station selection history. Such information may be stored in advance in the storage unit 140 of the conversation system 100 .

In this case, as shown in FIG. 4 , even if the driver does not input the utterance U3 for requesting information on the oil price (U3 is omitted), the dialogue system 100 displays the information on the oil price, especially the information on the current vehicle. It is possible to preemptively output an utterance (S2+S3=S3') that provides information on the price of gasoline, which is a type of oil.

The driver can omit the utterance (U3) for requesting information on the oil price, and the response of the dialogue system 100 is an utterance (S2) for guiding a nearby gas station and an utterance (S3) for guiding the oil price into one. By merging, you can shorten conversation steps and time.

Also, the dialog system 100 may determine by itself that the driver's intention is to search for a gas station based on the fact that the driver has asked for the current remaining fuel level.

In this case, as shown in FIG. 5 , even if the driver does not input the utterance U2 asking for a nearby gas station (U2 is omitted), the dialog system 100 provides information on the oil price (S2 + S3 = S3") can be output preemptively.

In addition, if there is only one gas station that is closest to the current location and has the cheapest oil price to the speech (S3") that provides information on the price of oil, it is possible to inquire whether or not to guide to the gas station. Therefore, the user may request directions to a corresponding gas station by simply inputting an utterance (U4 ′: yes) agreeing to the query of the dialogue system 100 without inputting a specific utterance for guiding to a certain gas station.

In this way, the dialog system 100 can recognize the user's actual intention and preemptively provide information corresponding thereto by considering the content that the user has not uttered using the previously acquired information. Through this, it is possible to shorten the conversation step and time until the user provides the desired service.

6 and 7 are control block diagrams schematically illustrating a connection relationship between an interactive system and components of a vehicle.

Referring to FIG. 6 , the user's voice input to the conversation system 100 may be input through the voice input device 210 provided in the vehicle 200 . As described above with reference to FIG. 2 , the voice input device 210 may include a microphone provided inside the vehicle 200 .

An input other than a voice among user inputs may be input through the non-voice input device 220 . The non-voice input device 220 may include input buttons 221 and 223 for receiving a command through a user's manipulation and a jog shuttle 225 .

In addition, the non-voice input device 220 may include a camera for photographing the user. Through the image captured by the camera, a user's gesture, facial expression, or gaze direction used as a means of inputting a command may be recognized. Alternatively, it is also possible to determine the user's state (drowsiness state, etc.) through the image captured by the camera.

Information about the vehicle may be input to the conversation system 100 through the vehicle controller 240 . The information about the vehicle may include vehicle state information or surrounding situation information obtained through various sensors provided in the vehicle 200 , and may also include information basically stored in the vehicle 200 , such as the type of vehicle.

The dialog system 100 uses the user's voice input through the voice input device 210 , the user's non-voice input input through the non-voice input device 220 , and various information input through the vehicle controller 240 . It identifies the user's intention and situation, and outputs a response for performing an action corresponding to the user's intention.

The talker output device 230 is a device that provides a visual, auditory, or tactile output to the talker, and may include a display 231 and a speaker 232 provided in the vehicle 200 . The display 231 and the speaker 232 may visually or audibly output a response to a user's utterance, a query to the user, or information requested by the user. Alternatively, it is possible to output vibration by mounting a vibrator on the steering wheel 207 .

In addition, the vehicle controller 240 may control the vehicle 200 to perform an action corresponding to a user's intention or a current situation according to a response output from the dialogue system 100 .

Meanwhile, the vehicle 200 includes not only data obtained through a sensor provided in the vehicle 200 , but also information obtained from the external content server 300 or an external device through the communication device 280 , for example, traffic conditions and weather conditions. , driving environment information, such as temperature, passenger information, and driver personal information, and user information may also be collected and transmitted to the dialogue system 100 .

As shown in FIG. 7 , information obtained from sensors provided in the vehicle, such as residual oil amount, rainfall amount, rainfall speed, surrounding obstacle information, speed, engine temperature, tire air pressure, current location, etc. (100) can be entered.

Driving environment information obtained from the outside through Vehicle to Everything (V2X) may be input to the dialogue system 100 through the external signal controller 242 . V2X refers to the exchange and sharing of various useful information such as traffic conditions while communicating with road infrastructure and other vehicles while the vehicle is driving.

V2X communication includes vehicle-to-infrastructure (V2I: Vehicle-to-Infrastructure, hereinafter V2I) communication, vehicle-to-vehicle (V2V: Vehicleto-Vehicle), and vehicle-to-mobile device (V2N: Vehicle-to-Nomadic devices) communication. can Therefore, by using V2X communication, it is possible to communicate with each other directly or with infrastructure installed on the street to exchange information such as traffic information in front and the possibility of approach or collision of another vehicle and inform the driver.

Therefore, the driving environment information input to the dialogue system 100 through the external signal controller 242 includes forward traffic information, approach information of surrounding vehicles, collision warning with other vehicles, real-time traffic conditions, unexpected room situations, and traffic flow control. status and the like.

Although not shown in the drawing, a signal obtained through V2X may also be input to the vehicle 200 through the communication device 280 .

The vehicle controller 240 may include a memory in which a program for performing the above-described operation and an operation to be described later is stored and a processor that executes the stored program. At least one memory and processor may be provided, respectively, and when a plurality of memory and processor are provided, they may be integrated on one chip or may be physically separated.

In addition, the internal signal controller 241 and the external signal controller 242 may be implemented by the same processor or by separate processors.

8 and 9 are control block diagrams schematically illustrating a connection relationship between components of a dialogue system and components of a vehicle.

Referring to FIG. 8 , the user's voice transmitted from the voice input device 210 is input to the voice input processor 111 in the input processor 110 , and the user's non-voice input transmitted from the non-voice input device 220 is It is input to the context information processor 112 in the input processor 110 .

The context information processor 112 understands the context based on vehicle state information, driving environment information, user information, and the like. The dialog system 100 may more accurately identify the user's intention or effectively find a service required by the current user through this context understanding.

The response output from the result processor 130 is input to the dialog output device 230 or the vehicle controller 240 so that the vehicle 200 can provide a service required to the user. Alternatively, it may be transmitted to the external content server 300 to request a service required by the user.

Vehicle state information, driving environment information, user information, etc. transmitted from the vehicle controller 240 are stored in the storage unit 140 .

Referring to FIG. 9 , the storage unit 140 may include a long-term memory 143 and a short-term memory 144 . Data stored in the storage unit 140 may be divided into short-term and long-term memory according to the importance and permanence of data and the intention of a designer.

The short-term memory 144 may store previously conducted conversations. The previously performed conversation may be a conversation performed within a reference time from the current time point. Alternatively, the conversation may be continuously stored until the capacity of the utterance content between the user and the conversation system 100 becomes a reference value.

As an example, if the user utters "Tell me about restaurants near Gangnam Station", the dialog system 100 searches for a restaurant near Gangnam Station through the external content server 300, and provides information about the found restaurant near Gangnam Station to the user. . As an example of providing information, a list of restaurants may be displayed on the display 231 , and when the user utters “first”, the short-term memory 144 may store conversation contents from inquiry to selection of restaurants. .

Alternatively, it is possible not only to store the entire conversation content, but also to store specific information included in the conversation content. For example, it is also possible to store the first restaurant in the list of restaurants in the short-term memory 144 or the long-term memory 143 as a restaurant selected by the user.

After a conversation about a restaurant near Gangnam Station, when the user asks "what is the weather?" to the conversation system 100, the conversation system 100 estimates that the user's interest location is Gangnam Station from the conversation stored in the short-term memory 144, You can output the response "It's raining at Gangnam Station" to the user.

Afterwards, when "recommend a menu for that restaurant" is uttered, the dialog system 100 estimates that "the restaurant" is a restaurant near Gangnam Station from the conversation stored in the short-term memory, and provides Through the service, information about the recommended menu of the restaurant can be obtained, and a response such as "The restaurant has delicious kalguksu" can be output.

The long-term memory 143 may store data according to the persistence of the data. For example, phone numbers of family or friends, POI information such as home or work, user's preference for specific factors, etc. may be determined to ensure data persistence and stored in the long-term memory 143 . On the other hand, data for which it is determined that the persistence of data is not guaranteed may be stored in the short-term memory 144 .

For example, since the user's current location is temporary data, it may be stored in the short-term memory 144 , and the user's preference for a restaurant may be stored in the long-term memory 144 because it can be viewed as persistent data that can be used afterward.

When the user utters "Isn't there a restaurant nearby?", the dialog system 100 may determine the user's current location from the short-term memory 144 and determine from the long-term memory 143 that the user's preferred restaurant is a Chinese restaurant. . Accordingly, it is possible to recommend a list of Chinese restaurants preferred by the user in the vicinity of the current location by using the external content.

Also, the chat system 100 may preemptively provide services or information to the user by using the data stored in the long-term memory 143 and the short-term memory 144 .

For example, information about the user's house may be stored in the long-term memory 143 . The conversation system 100 may obtain information related to the user's house from the external content server 300 and provide the user with information that "the water supply is scheduled to be cut off due to cleaning of the apartment complex this Friday."

Alternatively, information on the battery state of the vehicle may be stored in the short-term memory 144 . The dialog system 100 may analyze the battery state of the vehicle stored in the short-term memory 1440 and provide information such as "The battery state of the vehicle is not good. Get it repaired before winter."

10 is a control block diagram for a vehicle-only method in which a conversation system is provided in a vehicle.

In the vehicle-only method, as shown in FIG. 10 , a conversation system 100 including an input processor 110 , a conversation manager 120 , a result processor 130 , and a storage unit 140 is included in the vehicle 200 . can

When the dialog system 100 is included in the vehicle 200 , the vehicle 200 may itself process a conversation with the user and provide a necessary service to the user. However, information necessary for conversation processing and service provision may be obtained from the external content server 300 .

Vehicle state information or driving environment information such as residual oil amount, rainfall amount, rainfall speed, surrounding obstacle information, tire pressure, current location, engine temperature, vehicle speed, etc. detected by the vehicle sensor 260 is transmitted through the vehicle controller 240 through the dialogue system (100) is entered.

In addition, the vehicle controller 240 includes the air conditioner 251 , the window 252 , the door 253 , the seat 254 or the AVN 255 provided in the vehicle 200 according to the response output from the dialogue system 100 . etc can be controlled.

For example, if the dialog system 100 determines that the user's intention or a service required by the user is to lower the temperature inside the vehicle 200 , and generates and outputs a command corresponding thereto, the vehicle controller 240 controls the air conditioner By controlling 251 , the temperature inside the vehicle 200 may be lowered.

As another example, if the dialog system 100 determines that the user's intention or a service required by the user is to raise the driver's seat window 252a, and generates and outputs a corresponding command, the vehicle controller 240 opens the window 252 It is possible to raise the driver's seat window 252a by controlling it.

As another example, if the dialogue system 100 determines that the service corresponding to the user's intention or the service required by the user is a route guidance to a specific destination, and generates and outputs a corresponding command, the vehicle controller 240 is the AVN 255 may be controlled to perform route guidance. If necessary, the communication device 280 may obtain map data, POI information, and the like from the external content server 300 and use it to provide a service.

11 and 12 are control block diagrams of a vehicle gateway method in which the chat system is provided in a remote server and the vehicle only serves as a gateway connecting the user and the chat system.

In the vehicle gateway method, as shown in FIG. 11 , the remote dialogue system server 1 is provided outside the vehicle 200 , and the vehicle 200 is connected to the remote dialogue system server 1 and the communication device 280 through the communication device 280 . An interactive system client 270 is provided. The communication device 280 serves as a gateway connecting the vehicle 200 and the remote conversation system server 1 .

The dialog system client 270 functions as an interface connected to the input/output device, and may perform data collection and transmission/reception.

When the voice input device 210 and the non-voice input device 220 provided in the vehicle 200 receive the user input and transmit it to the dialog system client 270 , the dialog system client 270 communicates with the communication device 280 . It is possible to transmit input data to the remote chat system server 1 .

The vehicle controller 240 may also transmit the data detected by the vehicle detection unit 260 to the chat system client 270 , and the chat system client 270 transmits the data detected by the vehicle sensor 260 to the remote chat system server 1 through the communication device 280 . Data detected by the vehicle detection unit 260 may be transmitted.

The remote dialogue system server 1 is provided with the above-described dialogue system 100 to perform all of the processing of input data, the dialogue processing based on the processing result of the input data, and the result processing based on the dialogue processing result.

In addition, the remote conversation system server 1 may obtain information or content necessary for processing input data, conversation management, or result processing from the external content server 300 .

The vehicle 200 may also bring from the external content server 300 content necessary to provide a service required to the user according to a response transmitted from the remote conversation system server 1 .

Referring to FIG. 12 , the communication device 280 may include one or more communication modules that enable communication with an external device, for example, a short-range communication module 281 , a wired communication module 282 , and a wireless communication module. (283) may be included.

The short-distance communication module 281 uses a wireless communication network in a short distance such as a Bluetooth module, an infrared communication module, a Radio Frequency Identification (RFID) communication module, a Wireless Local Access Network (WLAN) communication module, an NFC communication module, and a Zigbee communication module. to include at least one of various short-range communication modules for transmitting and receiving signals.

The wired communication module 282 includes various wired communication modules such as a local area network (LAN) module, a wide area network (WAN) module, or a value added network (VAN) module, as well as a universal USB (USB) module. Serial Bus), HDMI (High Definition Multimedia Interface), DVI (Digital Visual Interface), RS-232 (recommended standard232), power line communication, or POTS (plain old telephone service) may include at least one of various cable communication modules. have.

The wireless communication module 283 is a Wi-Fi (Wifi) module, a WiBro (Wireless broadband) module, in addition to GSM (Global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (Universal) Mobile Telecommunications System), Time Division Multiple Access (TDMA), Long Term Evolution (LTE), 4G, 5G, etc. may include at least one of various wireless communication modules that can be connected to the Internet network.

Meanwhile, the communication device 280 may further include an internal communication module (not shown) for communication between electronic devices inside the vehicle 200 . As an internal communication protocol of the vehicle 200 , a controller area network (CAN), a local interconnection network (LIN), FlexRay, Ethernet, or the like may be used.

The chat system 100 may send and receive data with the external content server 300 or the remote chat system server 1 using the wireless communication module 283 . In addition, V2X communication may be performed using the wireless communication module 283 . In addition, data may be exchanged with a mobile device connected to the vehicle 200 by using the short-range communication module 281 or the wired communication module 282 .

13 is a control block diagram illustrating a case in which a vehicle may partially perform input processing and output processing in a vehicle gateway method.

As described above, it is also possible for the dialog system client 270 of the vehicle 200 to perform only the functions of data collection and transmission/reception, but as shown in FIG. 13 , the input processor 271 to the dialog system client 270 ), the result processor 273 and the storage unit 274 are included, so that it is possible to process data input from a user or vehicle, or to perform a process related to the provision of a service determined to be necessary for the user in the vehicle 200 . That is, the operation of the input processor 110 and the result processor 130 may be performed not only in the remote conversation system server 1 but also in the vehicle 200 .

In this case, it is also possible for the dialog system client 270 to perform only a part or all of the operations of the input processor 110 described above. In addition, it is also possible for the dialog system client 270 to partially or completely perform the operation of the above-described result processor 130 .

Task sharing between the remote chat system server 1 and the chat system client 270 may be determined in consideration of the capacity of data to be processed, data processing speed, and the like.

14 is a control block diagram of a hybrid method in which both a remote conversation system server and a vehicle can perform conversation processing.

In the hybrid method, as shown in FIG. 14 , an input processor 110 , a dialog manager 120 , a result processor 130 , and a storage unit 140 are provided in the remote dialog system server 1 to perform dialog processing. Also, the vehicle 200 may be provided with a terminal conversation system 290 including an input processor 291 , a conversation manager 292 , a result processor 293 , and a storage unit 294 to perform conversation processing.

However, the processor or memory provided in the vehicle 200 may be different from the processor or memory provided in the remote conversation system server 1 in terms of capacity or performance. Accordingly, if the terminal dialog system 290 processes all input data and manages the dialog to output the results, the entire process is performed in the terminal dialog system 290; otherwise, the remote dialog system server 1 ) can be requested for processing.

Before the dialog processing in the terminal dialog system 290 is performed, it is determined whether the terminal dialog system 290 is processed based on the type of input data, and directly processed or processed in the remote dialog system server 1 according to the determination result. can request

Alternatively, when a situation arises that the terminal conversation system 290 performs conversation processing and the processing is impossible, it is also possible to transmit the result of its own processing while requesting the processing to the remote conversation system server 1 .

For example, when high-performance computing power and long-term data processing is required, the remote conversation system server 1 may process it, and the terminal conversation system 290 may process it in real time. For example, when an instance requiring immediate processing occurs and data before synchronization needs to be processed, it may be set to be processed preferentially in the terminal conversation system 290 .

In addition, when there is an unregistered speaker in the vehicle and user confirmation is required, the remote conversation system server 1 may process the conversation.

In addition, when the terminal conversation system 290 cannot complete the conversation process by itself in a situation in which connection with the remote conversation system server 1 through the communication device 280 is impossible, the user through the dialogue output device 230 may inform that dialog processing cannot be performed.

Data stored in the terminal conversation system 290 and data stored in the remote conversation system server 1 may be determined according to criteria such as data capacity or data type. For example, in the case of data that may be personally identifiable and may infringe privacy, it may be stored in the storage unit 294 of the terminal conversation system 290 . Also, a large amount of data may be stored in the storage unit 140 of the remote conversation system server 1 , and a small amount of data may be stored in the storage unit 294 of the terminal conversation system 290 . Alternatively, it is also possible that a small amount of data is stored in both the storage unit 140 of the remote conversation system server 1 and the storage unit 294 of the terminal conversation system 290 .

15 and 16 are control block diagrams of a mobile gateway method in which a mobile device connected to a vehicle connects a user and a remote conversation system server.

In the mobile gateway method, as shown in FIG. 15 , the mobile device 400 receives vehicle state information and driving environment information from the vehicle 200 , and transmits user input and vehicle state information to the remote conversation system server 1 . do. That is, the mobile device 400 serves as a gateway connecting the user and the remote chat system server 1 or the vehicle 200 and the remote chat system server 1 .

The mobile device 400 may be an electronic device that is portable, such as a smart phone, a smart watch, smart glasses, a PDA, or a tablet PC, and can communicate with an external server and vehicle to exchange data.

The mobile device 400 includes a voice input device 410 for receiving a user's voice, a non-voice input device 420 for receiving a user's non-voice input, and an output device 430 for outputting a response visually, auditory or tactilely. ), a communication device 480 that communicates with the remote conversation system server 1 and the vehicle 200 to transmit and receive data, and a remote conversation system server through the communication device 480 by collecting input data from the vehicle 200 and the user. (1) includes a dialog system client 470 that sends to.

The voice input device 410 may include a microphone that receives sound, converts it into an electrical signal, and outputs it.

The non-voice input device 420 may include an input button provided in the mobile device 400 , a touch screen, or a camera.

The output device 430 may include a display, a speaker, or a vibrator provided in the mobile device 400 .

In addition to using the voice input device 410 , the non-voice input device 420 , and the output device 430 provided in the mobile device 400 as an input/output interface for the user, the voice input device 210 provided in the vehicle 200 . ), it is also possible to use the voice input device 220 and the dialogue output device 230 .

When the vehicle 200 transmits the data detected by the vehicle detection unit 260 and the user input to the mobile device 400 , the chat system client 470 of the mobile device 400 sends it to the remote chat system server 1 . send.

Also, the chat system client 470 may transmit a response or command transmitted from the remote chat system server 1 to the vehicle 200 . When the dialogue output device 230 provided in the vehicle 200 is used as the input/output interface for the user, a response to the user's utterance may be output through the dialogue output device 230 . When the output device 430 provided in the mobile device 400 is used, a response to the user's utterance may be output through the output device 430 of the mobile device 400 .

A command for vehicle control is transmitted to the vehicle 200 , and the vehicle controller 240 may provide a necessary service to the user by performing a control corresponding to the transmitted command.

On the other hand, the dialog system client 470 not only collects input data and transmits it to the remote dialog system server 1 , but also partially or partially functions of the input processor 110 and the result processor 130 of the dialog system 100 . It is also possible to do all of them.

Referring to FIG. 16 , the communication device 480 of the mobile device 400 may include one or more communication modules that enable communication with an external device, for example, a short-range communication module 481, a wired communication module ( 482 ) and a wireless communication module 483 .

The short-distance communication module 481 uses a wireless communication network in a short distance such as a Bluetooth module, an infrared communication module, a radio frequency identification (RFID) communication module, a wireless local access network (WLAN) communication module, an NFC communication module, and a Zigbee communication module. to include at least one of various short-range communication modules for transmitting and receiving signals.

The wired communication module 482 includes various wired communication modules, such as a Local Area Network (LAN) module, a Wide Area Network (WAN) module, or a Value Added Network (VAN) module, as well as a Universal USB (USB) module. Serial Bus), HDMI (High Definition Multimedia Interface), DVI (Digital Visual Interface), RS-232 (recommended standard232), power line communication, or POTS (plain old telephone service) may include at least one of various cable communication modules. have.

The wireless communication module 483 is a Wifi module, a WiBro (Wireless broadband) module, in addition to GSM (Global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (Universal) Mobile Telecommunications System), Time Division Multiple Access (TDMA), Long Term Evolution (LTE), 4G, 5G, etc. may include at least one of various wireless communication modules that can be connected to the Internet network.

For example, the mobile device 400 may be connected to the vehicle 200 through the short-range communication module 481 or the wired communication module 482 , and the remote conversation system server 1 or It may be connected to the external content server 300 .

17 is a control block diagram of a mobile-only method in which a conversation system is mounted on a mobile device.

In the mobile-only method, as shown in FIG. 17 , the conversation system 100 is provided in the mobile device 400 .

Accordingly, there is no need to be connected to the remote conversation system server 1 in order to perform conversation processing, and the mobile device 400 can process a conversation with the user by itself and provide a necessary service to the user. However, information necessary for conversation processing and service provision may be obtained from the external content server 300 .

Hereinafter, a detailed configuration of the dialogue system 100 and a specific operation for each component will be described in detail. In the embodiment to be described below, for convenience of description, it is assumed that the conversation system 100 is provided in the vehicle 200 .

18, 19A, and 19B are control block diagrams in which the configuration of the input processor is subdivided in the configuration of the conversation system.

Referring to FIG. 18 , the input processor 110 may include a voice input processor 111 for processing a voice input and a context information processor 112 for processing context information.

The user's voice input through the voice input device 210 is transmitted to the voice input processor 111 , and the user input other than the voice input through the non-voice input device 220 is transmitted to the context information processor 112 .

The vehicle controller 240 transmits vehicle state information, driving environment information, and user information to the context information processor 112 . The driving environment information and user information may be provided from the external content server 300 or the mobile device 400 connected to the vehicle 200 .

Any input other than voice may be included in the context information. That is, the context information may include vehicle state information, driving environment information, and user information.

The vehicle state information is information obtained by a sensor provided in the vehicle 200 and may include information indicating the state of the vehicle, information related to the vehicle such as the type of vehicle, and information stored in the vehicle.

Driving environment information is information acquired by a sensor provided in the vehicle 200, and image information acquired by a front camera, a rear camera, or a stereo camera, and a sensor such as a radar, lidar, and ultrasonic sensor acquired. It may include obstacle information, rainfall amount/rain speed information obtained by the rainfall sensor, and the like.

In addition, the driving environment information is information obtained through V2X, and may further include traffic situation information, traffic light information, approach or collision possibility information of nearby vehicles, and the like.

The user information includes information related to a user's status measured through a camera or biosignal measuring device provided in the vehicle, information related to a user directly input by the user using an input device provided in the vehicle, and a user stored in the external content server 300 . It may include information related to , information stored in the mobile device 400 connected to the vehicle, and the like.

The voice input processor 111 is a voice recognizer 111a that recognizes the input user's voice and outputs it as a utterance in text form, and applies Natural Language Understanding technology to the utterance of the user in which the utterance is implied. It may include a natural language interpreter 111b that detects an intention, and a dialog input manager 111c that transmits the natural language understanding result and context information to the dialog manager 120 .

The speech recognizer 111a may include a speech recognition engine, and the speech recognition engine may recognize a speech uttered by a user by applying a speech recognition algorithm to the input speech, and may generate a recognition result.

At this time, since the input voice can be converted into a more useful form for voice recognition, a start point and an end point are detected from the voice signal to detect an actual voice section included in the input voice. This is called EPD (End Point Detection).

And, within the detected section, features such as Cepstrum, Linear Predictive Coefficient (LPC), Mel Frequency Cepstral Coefficient (MFCC) or Filter Bank Energy A feature vector of the input speech can be extracted by applying a vector extraction technique.

Then, the recognition result can be obtained by comparing the extracted feature vector with the trained reference pattern. To this end, an acoustic model that models and compares the signal characteristics of speech and a language model that models a linguistic order relationship such as words or syllables corresponding to a recognized vocabulary may be used. To this end, the acoustic model/language model DB may be stored in the storage unit 140 .

The acoustic model can be divided into a direct comparison method in which the recognition target is set as a feature vector model again and compares it with the feature vector of speech data, and a statistical method in which the feature vector of the recognition target is statistically processed and used.

The direct comparison method is a method of setting a unit such as a word or phoneme to be recognized as a feature vector model and comparing how similar the input speech is. A representative vector quantization method is used. According to the vector quantization method, a feature vector of input speech data is mapped with a codebook, which is a reference model, and encoded as a representative value, thereby comparing the code values with each other.

The statistical model method is a method of composing a unit for a recognition target into a state sequence and using the relationship between the state sequences. The status column may consist of a plurality of nodes. Methods using the relationship between the state columns again include Dynamic Time Warping (DTW), Hidden Markov Model (HMM), and a method using a neural network.

Dynamic time warping is a method of compensating for differences in the time axis when compared with the reference model in consideration of the dynamic characteristics of speech in which the length of a signal varies with time even if the same person pronounces the same pronunciation, and the Hidden Markov model converts speech into state transition After assuming as a Markov process with the probability and the observation probability of the node (output symbol) in each state, the state transition probability and the observation probability of the node are estimated through the training data, and the probability that the input voice occurs in the estimated model is calculated It is a recognition technology that

On the other hand, a language model modeling a linguistic order relationship such as words or syllables can reduce acoustic ambiguity and reduce recognition errors by applying the order relationship between units constituting language to units obtained in speech recognition. The language model includes a statistical language model and a model based on a Finite State Automata (FSA). In the statistical language model, chain probabilities of words such as Unigram, Bigram, and Trigram are used.

The voice recognizer 111a may use any of the above-described methods in recognizing a voice. For example, an acoustic model to which a hidden Markov model is applied may be used, or an N-best search method in which the acoustic model and the speech model are integrated may be used. The N-best search method can improve recognition performance by selecting up to N recognition result candidates using an acoustic model and a language model, and then re-evaluating the ranks of these candidates.

The voice recognizer 111a may calculate a confidence value in order to secure the reliability of the recognition result. The confidence value is a measure of how reliable the voice recognition result is. For example, a phoneme or word, which is a recognized result, may be defined as a relative value with respect to a probability that the word is uttered from other phonemes or words. Accordingly, the confidence value may be expressed as a value between 0 and 1 or as a value between 0 and 100.

When the confidence value exceeds a preset threshold, an operation corresponding to the recognition result can be performed by outputting the recognition result, and when the confidence value is less than the threshold value, the recognition result can be rejected have.

The speech recognition result of the speech recognizer 111a in the form of text is input to the natural language understander 111b.

The natural language understander 111b may recognize the intention of the user contained in the utterance by applying the natural language understanding technology. Accordingly, the user may input a command through a natural dialogue, and the dialogue system 100 may also induce input of the command through the dialogue or provide a service required by the user.

First, the natural language understander 111b performs morpheme analysis on an utterance in a text form. A morpheme is the smallest unit of meaning and represents the smallest element of meaning that cannot be further subdivided. Therefore, morpheme analysis is the first step in understanding natural language, and converts an input string into a morpheme sequence.

The natural language interpreter 111b extracts a domain from the utterance based on the morpheme analysis result. The domain can identify the subject of the language spoken by the user. For example, domains representing various topics such as route guidance, weather search, traffic search, schedule management, fueling guide, and air conditioning control are databased.

The natural language understander 111b may recognize an entity name from an utterance. An entity name is a proper noun such as a person's name, a place name, an organization name, time, date, money, etc., and entity name recognition is the task of identifying the entity name in a sentence and determining the type of the identified entity name. It is possible to understand the meaning of the sentence by extracting important keywords from the sentence by recognizing the entity name.

The natural language interpreter 111b may analyze the dialog act of the utterance. Dialog act analysis is the task of analyzing the intention of the user's utterance, and it is to identify the intention of the utterance regarding whether the user asks a question, makes a request, responds, or expresses a simple emotion.

The natural language understander 111b extracts an action corresponding to the user's utterance intention. Based on information such as domain, entity name, and dialogue act corresponding to the speech, the user's speech intention is identified, and an action corresponding to the speech intent is extracted. An action may be defined by an object and an operator.

In addition, the natural language interpreter 111b may extract factors related to action performance. The factor related to the action execution may be an effective factor directly required to perform the action, or may be an ineffective factor used to extract such an effective factor.

For example, when the user's utterance is "Let's go to Seoul Station", the natural language understander 111b may extract "navigation" as a domain corresponding to the utterance, and extract "navigation" as an action. A dialogue act corresponds to a "request".

The entity name “Seoul Station” corresponds to [factor: destination] related to action, but specific exit numbers or GPS information may be required for actual navigation through navigation. In this case, the [factor: destination: Seoul Station] extracted by the natural language interpreter 111b may be a candidate factor for searching for a Seoul station desired by the actual user from among a plurality of Seoul Station POIs.

In addition, the natural language interpreter 111b may also extract a means for expressing a mathematical relationship between words and sentences, such as a parse-tree.

The processing results of the natural language understander 111b, such as a morpheme analysis result, domain information, action information, dialogue act information, extracted factor information, entity name information, pastries, etc., are transmitted to the dialog input manager 111c.

The contextual information processor 112 includes a contextual information collector 112a that collects information from the non-voice input device 220 and the vehicle controller 240, a contextual information collection manager 112b that manages the collection of contextual information, and a natural language understanding result and a context understanding unit 112c for understanding the context based on the collected context information.

The input processor 110 may include a memory in which a program for performing an operation to be described above or below is stored and a processor for executing the stored program. At least one memory and processor may be provided, respectively, and when a plurality of memory and processor are provided, they may be integrated on one chip or may be physically separated.

Also, the voice input processor 111 and the context information processor 112 included in the input processor 110 may be implemented by the same processor or may be implemented by separate processors.

Hereinafter, how the components of the input processor 110 process input data using information stored in the storage 140 will be described in detail with reference to FIGS. 19A and 19B .

Referring to FIG. 19A , the natural language interpreter 111b may use the domain/action inference rule DB 141 for domain extraction, entity name recognition, dialogue act analysis, and action extraction.

The domain/action inference rule DB 141 may store domain extraction rules, dialogue act analysis rules, entity name conversion rules, action extraction rules, and the like.

Other information such as user input, vehicle status information, driving environment information, and user information other than voice is input to the context information collector 112a, and is stored in the context information DB 142, long-term memory 143 or short-term memory 144 can be

For example, raw data detected by the vehicle detection unit 260 may be stored in the context information DB 142 after being divided into a sensor type and a sensor value.

The short-term memory 144 and the long-term memory 143 may store data meaningful to the user, such as the user's current state, the user's preference/inclination, or data capable of determining it.

As described above, the long-term memory 143 may store information that can be used for a long time, such as a user's phone book, schedule, preference, academic background, personality, occupation, family-related information, etc., with permanence guaranteed. The short-term memory 144 may store information that is used in the short term because its permanence is not guaranteed or uncertain, such as a current/previous location, today's schedule, previous conversation contents, conversation participants, surrounding circumstances, domain, driver status, and the like. Depending on the type of data, there may be data stored in duplicate in two or more storages of the context information DB 142 , the short-term memory 144 , and the long-term memory 143 .

Also, among the information stored in the short-term memory 144 , it is determined that the permanence is guaranteed may be transmitted to the long-term memory 143 .

Also, it is possible to obtain information to be stored in the long-term memory 143 by using the information stored in the short-term memory 144 or the context information DB 142 . For example, the user's preference may be obtained by analyzing destination information or conversation contents accumulated for a certain period of time, and the obtained user preference may be stored in the long-term memory 143 .

Acquisition of information to be stored in the long-term memory 143 by using the information stored in the short-term memory 144 or the context information DB 142 may be performed inside the dialogue system 100 or performed in a separate external system. It is also possible to be

The former case can be performed in the memory manager 135 of the result processor 130 to be described later. In this case, data used to acquire meaningful information or persistent information such as user preference or tendency among data stored in the short-term memory 144 or the context information DB 142 is stored in the long-term memory 143 in the form of a log file. can be saved. The memory manager 135 analyzes the data accumulated over a certain period of time to obtain data with persistence and stores it again in the long-term memory 143 . A location in which persistent data is stored in the long-term memory 143 and a location in which data stored in the form of a log file are stored may be different from each other.

Alternatively, the memory manager 135 may determine persistent data among data stored in the short-term memory 144 , and move the determined data to the long-term memory 143 and store it.

When performing in a separate external system, a data management system 800 including a communication unit 810 , a storage unit 820 , and a control unit 830 may be used as shown in FIG. 19B .

The communication unit 810 receives data stored in the context information DB 142 or the short-term memory 144 . All of the stored data may be transmitted to the communication unit, or only data used to acquire meaningful information or persistent information such as user preference or tendency may be selected and transmitted. The received data is stored in the storage unit 820 .

The controller 830 obtains persistent data by analyzing the accumulated data, and transmits the obtained data to the conversation system 100 again through the communication unit 810 . The transmitted data is stored in the long term memory 143 of the conversation system 100 .

The dialog input manager 111c may transfer the output result of the natural language interpreter 111b to the context interpreter 112c to obtain context information related to performing an action.

The context understandr 112c may refer to context information for each action stored in the context understanding table 145 to determine what context information is related to performing an action corresponding to the user's utterance intention.

20A and 20B are diagrams illustrating examples of information stored in a context understanding table.

Referring to the example of FIG. 20A , in the context understanding table 145, context information related to action execution and a context information type may be stored for each action.

For example, when the route guidance is an action, the current location is required as context information, and the context information type may be GPS information. When the vehicle condition check is an action, a moving distance is required as context information, and the context information type may be an integer. When the gas station recommendation is an action, residual fuel amount and a distance to empty (DTE) are required as context information, and the context information type may be an integer.

When context information related to performing an action corresponding to the user's utterance intention is already stored in the context information DB 142 , the long-term memory 143 , or the short-term memory 144 , the context understandr 112c is the context information DB 142 . ), the long-term memory 143 or the short-term memory 144 fetches the corresponding information and transmits it to the dialog input manager 111c.

If the context information related to performing the action corresponding to the user's utterance intention is not stored in the context information DB 142, the long-term memory 143, or the short-term memory 144, the context understandr 112c is the context information collection manager ( 112b) to request the necessary information. The context information collection manager 112b allows the context information collector 112a to collect necessary information.

The contextual information collector 112a may periodically collect data, may collect data when a specific event occurs, or periodically collect data and additionally collect additional data when a specific event occurs. In addition, data may be collected when a data collection request is input from the context information collection manager 112b.

The contextual information collector 112a collects necessary information, stores it in the contextual information DB 142 or the short-term memory 144, and transmits a confirmation signal to the contextual information collection manager 112b.

The contextual information collection manager 112b also transmits a confirmation signal to the contextual understander 112c, and the contextual understander 112c retrieves necessary information from the contextual information DB 142, the long-term memory 143, or the short-term memory 144. It is brought and delivered to the dialog input manager 111c.

As a specific example, when the action corresponding to the user's utterance intention is navigation, the context understanding unit 112c searches the context understanding table 145 to recognize that the context information related to the route guidance is the current location.

If the current location is already stored in the short-term memory 144, the context understandr 112c fetches the current location from the short-term memory 144 and transmits it to the dialog input manager 111c.

When the current location is not stored in the short-term memory 144, the current location is requested from the context information collection manager 112b, and the context information collection manager 112b causes the context information collector 112a to cause the vehicle controller 240 ) to get the current location.

The contextual information collector 112a acquires the current location, stores it in the short-term memory 144, and transmits a confirmation signal to the contextual information collection manager 112b. The contextual information collection manager 112b also transmits a confirmation signal to the contextual understandr 112c, and the contextual understander 112c brings the current location information from the short-term memory 144 and transmits it to the dialog input manager 111c.

The dialog input manager 111c transmits the output of the natural language understander 111b and the output of the situation understander 112c to the dialog manager 120, and manages so that the duplicate input does not enter the dialog manager 120 have. In this case, the output of the natural language interpreter 111b and the output of the context interpreter 112c may be combined into one and transmitted to the conversation manager 120 , or may be independently transmitted.

Meanwhile, the contextual information collection manager 112b may transmit an action trigger signal to the contextual understandingr 112c when it is determined that a specific event has occurred because the data collected by the contextual information collector 112a satisfies a preset condition.

The context understandr 112c searches the context understanding table 145 to retrieve context information related to the corresponding event, and if the retrieved context information is not stored, transmits a request signal for context information to the context information collection manager 112b again do.

As in the example of FIG. 20B , context information related to an event and a type of context information may be stored in the context understanding table 145 for each event.

For example, when the generated event is an engine temperature warning, an integer type engine temperature may be stored as related context information. When the generated event is driver drowsiness detection, an integer type driver drowsiness level may be stored as related context information. When the generated event is insufficient tire air pressure, an integer type of tire air pressure may be stored as related context information. When the generated event is a fuel warning, an integer type of drivable distance may be stored as related context information. When the generated event is more than a sensor, a text-type sensor name may be stored as related context information.

The context information collection manager 112b collects necessary context information through the context information collector 112a, and transmits a confirmation signal to the context understand unit 112c. The context understanding unit 112c fetches necessary context information from the context information DB 142 , the long-term memory 143 or the short-term memory 144 , and delivers it to the dialog input manager 111c together with the action information.

The dialog input manager 111c inputs the output of the situation understander 112c to the dialog manager 120 .

21 is a control block diagram in which the configuration of the dialog manager is subdivided, FIG. 22 is a diagram showing an example of information stored in the related action DB, FIG. 23 is a diagram showing an example of information stored in the action execution condition DB, and FIG. 24 is a diagram showing an example of information stored in the action factor DB.

Referring to FIG. 21 , the dialog manager 120 creates/deletes/updates the dialog or action according to the request of the dialog flow manager 121 and the dialog flow manager 121 to request to create/delete/update a dialog or action. conversation action manager 122, ambiguity resolver 123 that ultimately clarifies the intention of the user by resolving ambiguity about the situation and ambiguity about dialogue, factor manager 124 that manages factors necessary for action execution, multiple An action priority determiner 125 that determines whether to perform an action for candidate actions of , and determines their priorities, and an external information manager that manages an external content list and related information and manages factor information required for external content query (126).

The conversation manager 120 may include a memory in which a program for performing an operation described above or to be described later is stored and a processor that executes the stored program. At least one memory and processor may be provided, respectively, and when a plurality of memory and processor are provided, they may be integrated on one chip or may be physically separated.

In addition, each of the components included in the conversation manager 120 may be implemented by a single processor or may be implemented by a separate processor.

In addition, the conversation manager 120 and the input processor 110 may also be implemented by a single processor or may be implemented by a separate processor.

The natural language understanding result (output of the natural language interpreter) and context information (output of the context interpreter) output from the dialog input manager 111a are input to the dialog flow manager 121 . The output of the natural language interpreter 111b includes information on the content of the user's utterance, such as a morpheme analysis result, in addition to information such as a domain and an action. The output of the context understanding unit 112c may include events determined by the context information collection manager 112b in addition to the context information.

The dialog flow manager 121 searches whether a dialog task or action task corresponding to the input from the dialog input manager 111a exists in the dialog/action DB 147 .

The conversation/action DB 147 is a storage space for managing the state of the dialog and the state of the action, and may store the dialog state and the action state for the current conversation, actions, and preliminary actions to be performed in the future. For example, the state of the finished conversation/action, the stopped conversation/action, the ongoing conversation/action, and the ongoing conversation/action may be stored.

In addition, it is possible to store whether an action is switched/nested, a switched action index, an action change time, a last output state of a screen/voice/command, and the like.

For example, when a domain and an action corresponding to a user's utterance are extracted, if a dialog or action corresponding to the corresponding domain and action exists among the most recently stored conversations, it corresponds to the input from the dialog input manager 111a. It can be judged as a conversation task or an action task.

When the domain and action corresponding to the user's utterance are not extracted, an arbitrary task may be created or a request may be made to the dialog action generator 122 to refer to the most recently stored task.

If a dialog task or action task corresponding to the output of the input processor 110 does not exist in the dialog/action DB 147 , the dialog flow manager 121 creates a new dialog task and an action task in the dialog action manager 122 . ask to do

When the conversation flow manager 121 manages the conversation flow, the conversation policy DB 148 may be referred to. The dialog policy DB 148 stores a policy for developing a conversation, and specifically stores a policy for selecting/starting/suggesting/stopping/ending a conversation.

In addition, the dialog policy DB 148 may also store a policy on the timing and methodology at which the system outputs a response, and a policy for creating a response by linking a plurality of services and a policy for deleting an existing action and replacing it with another action can be saved.

For example, when the candidate action is plural, or the action corresponding to the user's intention or situation is plural (action A, action B), the response for both actions is "Do action A and then action B?" A policy that generates a response at once, and a policy that generates a separate response for action B after generating a response for one action such as “I will perform action A” i. “Do I want to perform action B?” All policies are possible.

In addition, the dialogue policy DB 147 may also store a policy for determining priority among candidate actions. The prioritization policy will be described later.

The dialog action manager 122 allocates a storage space to the dialog/action DB 147 to generate a dialog task and an action task corresponding to the output of the input processor 110 .

Meanwhile, when the domain and action cannot be extracted from the user's utterance, the dialog action manager 122 may generate an arbitrary dialog state. In this case, as will be described later, the ambiguity resolver 123 may determine the user's intention based on the user's utterance, surrounding circumstances, vehicle state, user information, and the like, and determine an appropriate action corresponding thereto.

If a dialog task and an action task corresponding to the output of the input processor 110 exist in the dialog/action DB 147, the dialog flow manager 121 allows the dialog action manager 122 to refer to the dialog task and the action task. request.

The action prioritization determiner 125 extracts a candidate action by searching the action list associated with the action or event included in the output of the input processor 110 from the related action DB 146b.

As in the example of FIG. 22 , the related action DB 146b may indicate mutually related actions and a relationship between them, and an event related action and a relationship between them. For example, actions such as directions, vehicle condition check, and gas station recommendation may be classified as related actions, and their relationship may correspond to mutual linkage.

Accordingly, in performing the navigation action, actions such as vehicle condition check and gas station recommendation may be performed together. Here, being performed together may include both being performed before and after the navigation action and during the performance of the navigation action (added as a waypoint).

The warning light output event may be stored as an event-action associated with the workshop guide action, and their relationship may also correspond to a correlation.

Accordingly, when a warning light output event occurs, a repair shop guidance action may be performed according to the type of the warning light or whether maintenance is required.

When the event determined by the situation collection manager 112b as well as the action corresponding to the user's speech is transmitted from the input processor 110, the action related to the event along with the action corresponding to the user's speech will also be a candidate action. can

The extracted candidate action list is transmitted to the dialogue action manager 122 , and the dialogue action manager 122 updates the action status of the dialogue/action status DB 147 by adding the candidate action list.

The action priority determiner 125 searches for conditions for performing each candidate action in the action execution condition DB 146c.

As in the example of FIG. 23 , the action execution condition DB 146c may store a condition necessary for performing an action and a factor used to determine whether the corresponding condition is satisfied for each action.

For example, the condition for performing the vehicle condition check action may be a case where the destination distance is 100 km or more, and a factor used for determining the condition corresponds to the destination distance. A condition for performing the gas station recommendation action may be a case where the destination distance is greater than the drivable distance DTE, and a factor used for determining the condition corresponds to the destination distance and the drivable distance DTE.

The action prioritization determiner 125 transmits the execution condition of the candidate action to the dialogue action manager 122, and the dialogue action manager 122 adds an action execution condition for each candidate action to the dialogue/action state DB 147. Update the action state.

The action prioritization determiner 125 is a factor necessary for determining the action execution condition in the context information DB 142, the long-term memory 143, the short-term memory 144, or the conversation/action state DB 147 (hereinafter, the condition determination factor). ), and it is possible to determine whether to perform each candidate action using the searched factors.

When the factor used for determining the action execution condition is not stored in the context information DB 142, long-term memory 143, short-term memory 144, or conversation/action state DB 147, the external information manager 126 is Through this, a necessary factor may be obtained from the external content server 300 .

The action priority determiner 125 may determine whether to perform each candidate action by using a factor necessary for determining an action execution condition. In addition, the priority of each candidate action may be determined based on whether each candidate action is performed and a prioritization rule stored in the conversation policy DB 148 .

A score for each candidate action may be calculated according to the current situation. The highest priority is assigned in the order of the highest calculated score. For example, as shown in [Equation 1] below, the action corresponding to the user's utterance, the safety score, the convenience score, the processing time, the processing time (whether it should be processed immediately), the user preference (the user when the service is proposed) acceptance degree or user's pre-defined preference), manager's score, vehicle status-related score, and action success rate (conversation success rate) may be used as factors of score calculation.

[Equation 1]

Priority score = w1*user utterance action + w2*safety score + w3*convenience score + w4*processing time + w5*processing time+ w6*user preference + w7*administrator score + w8*vehicle condition related score + w9* Action success rate * Whether the action can be performed (1: possible, undecided 0: impossible) * Whether the action is completed (complete: 1, incomplete: 0)

As described above, the action prioritization determiner 125 searches not only the actions directly related to the user's utterance or contextual information, but also the list of actions related thereto, and determines the priority between them, thereby providing the most necessary service to the user. there will be

The action prioritization determiner 125 transmits the execution status and priority of each candidate action to the dialog action manager 122, and the dialog action manager 122 adds the transmitted information to the dialog/action state DB 147. Update the action state.

The factor manager 124 may search for factors (hereinafter, referred to as action factors) used to perform each candidate action from the action factor DB 146a.

As in the example of FIG. 24 , the action factor DB 146a may store essential factors, optional factors, initial values of factors, and reference positions from which factors can be retrieved for each action. When the initial value of the factor is stored, the factor value corresponding to the factor is not included in the user's utterance or context information output from the input processor 110 , and the factor value is also stored in the context information DB 142 . If not, the user can check whether to perform the action according to the stored initial value or to perform the action according to the stored initial value.

For example, essential factors used to perform the navigation action may include a current location and destination, and optional factors may include a route type. A fast path may be stored as an initial value of the selection parameter. The current location and destination may be acquired by sequentially searching the conversation/action state DB 147 , the context information DB 142 , the short-term memory 144 , or the long-term memory 143 .

The essential factor used to perform the vehicle condition check action may include vehicle condition information, and the optional factor may include an inspection part. As an initial value of the selection factor, the entire part may be stored. The vehicle state information may be obtained from the context information DB 142 .

The selection factor used to perform the gas station recommendation action may include a preferred gas station, and oil A may be stored as an initial value of the selection factor. The preferred gas station may be obtained from the long-term memory 143 . It is also possible that the selection factors further include the type of vehicle, oil price, and the like.

As described above, the factor manager 124 brings the factor value of the factor retrieved from the action factor DB 146a from the reference position. The reference location from which the factor value can be obtained may be at least one of the context information DB 142 , the long-term memory 143 , the short-term memory 144 , the conversation/action state DB 147 , and the external content server 300 .

When the factor manager 124 gets the factor value from the external content server 300 , it may go through the external information manager 126 . The external information manager 126 may refer to the external service set DB 146d to determine where the information is to be retrieved.

The external service set DB 146d stores information on an external content server associated with the dialog system 100 . For example, it is possible to store information on the name of the external service, the description of the external service, the type of information provided by the external service, the method of using the external service, and the subject of the external service.

The factor value obtained by the factor manager 124 is transmitted to the dialogue action manager 122, and the dialogue action manager 122 updates the dialogue/action status DB 147 by adding factor values for each candidate action to the action status. do.

The factor manager 124 may acquire factor values of all candidate actions, and it is also possible to acquire only factor values of the candidate actions determined to be executable by the action prioritization determiner 125 .

Also, the factor manager 124 may selectively use various kinds of factor values representing the same information. For example, Seoul Station in the form of text indicating the destination may be converted into Seoul Station in the form of POI using a destination search service of a navigation system.

When there is no ambiguity in the conversation or situation, necessary information may be obtained according to the operations of the above-described action prioritization determiner 125 , the factor manager 124 , and the external information manager 126 , and the conversation and actions may be managed. However, when there is ambiguity in a conversation or situation, it is difficult to provide an appropriate service necessary for the user only by the operations of the action prioritization determiner 125 , the factor manager 124 , and the external information manager 126 .

In this case, the ambiguity resolver 123 may resolve ambiguity about a conversation or ambiguity about a situation. For example, when it is ambiguous what the conversation refers to because it contains a corresponding word such as that person, yesterday there, dad, mom, grandmother, daughter-in-law, etc., the ambiguity resolver 123 provides the context information DB 142, 143) or short-term memory 144 to resolve this ambiguity or provide a guide for resolving it.

For example, an ambiguous word included in a conversation, such as "yesterday there", "a mart near my house", "yesterday to Seoul station", etc. may correspond to a factor value of an action factor or a factor value of a conditional determination factor. However, in this case, it is impossible to actually perform an action or determine an action execution condition due to the ambiguity of the word itself.

The ambiguity resolver 123 may resolve the ambiguity of the factor value by referring to information stored in the context information DB 142 , the long-term memory 143 , or the short-term memory 144 . Alternatively, it is also possible to obtain necessary information from the external content server 300 by using the external information manager 126 if necessary.

For example, by retrieving a place the user went yesterday with reference to the short-term memory 144 , “yesterday there” can be converted into information usable as a destination for a navigation action. Alternatively, by referring to the long-term memory 143 , searching for the user's home address and retrieving the location information about the A mart near the home address from the external content server 300 , "a mart near the house" as the destination of the navigation action It can be transformed into usable information.

In addition, in a situation where the action (object, operator) is not clearly extracted from the input processor 110 or the user's intention is ambiguous, the ambiguity resolver 123 refers to the ambiguity resolution information DB 146e to identify the user's intention. and it is also possible to determine an action corresponding thereto.

25 is a diagram illustrating an example of information stored in an ambiguity resolution information DB.

The ambiguity resolution information DB 146e may match and store an utterance statement and an action corresponding thereto, based on the vehicle state information and the surrounding context information. The utterances stored in the ambiguity resolution information DB 146e may be utterances from which an action cannot be extracted through natural language understanding. In the example of FIG. 25 , a case in which the content of the utterance according to the result of the morpheme analysis is the content that the hand is cold or that the hand is cold will be described.

The surrounding situation information may include external temperature of the vehicle and whether it is raining, and the vehicle status information may include whether the air conditioner/heater is on/off, the air volume, the wind direction, and whether the steering heating wire is on/off.

As a specific example, if it is raining while the outside temperature is over 20 degrees, if the air conditioner is on, the air conditioner temperature is set too low and your hands are cold, and the corresponding vehicle control action can save the air conditioner temperature increase by 3 degrees. have.

If it is raining while the outside temperature is over 20 degrees, if the air conditioner is off, it can be recognized as feeling cold due to the rain, and the heater On can be stored as a corresponding vehicle control action.

If the outside temperature is over 20 degrees and it doesn't rain, if the air conditioner is on and the air conditioner wind direction is set to Up, it can be recognized that the air conditioner blows directly to the hand and the hand is cold. Changes in wind direction to Down can be saved.

If the outside temperature is over 20 degrees and there is no rain, if the air conditioner is on, the wind direction of the air conditioner is set to downward, and the air volume is set to medium or higher, the wind direction of the air conditioner is too strong to feel cold, and the corresponding vehicle control action is taken. It can be saved to lower the air conditioner air volume.

When the outside temperature exceeds 20 degrees and it does not rain, if the air conditioner is on, the air conditioner wind direction is downward, and the air conditioner air volume is set to low (lower), raising the air conditioner temperature by 3 degrees can be saved as a corresponding vehicle control action.

When the outside temperature is lower than 20 degrees and the heater is off, it may be determined that the hand is cold due to cold weather, and the turn-on may be stored as a corresponding vehicle control action.

When the outside temperature is lower than 20 degrees and the heater is on and the steering heating wire is off, it can be recognized that the hand is cold because the heat is not transmitted to the hand, and a corresponding vehicle control action can be stored as the steering heating wire on.

If the outside temperature is lower than 20 degrees, both the heater and the steering heating wire are on, and the heater wind direction is set to the downward direction, the heater wind is not transmitted to the hand and the hand is cold, and the corresponding vehicle control action is set to the heater wind direction can be saved as a two-way change of .

If the outside temperature is lower than 20 degrees, both the heater and the steering heating wire are on, and the heater wind direction is set to upward, if the heater temperature is set lower than the maximum value, the corresponding vehicle control action can be saved as heating the heater temperature. have.

If the external temperature is lower than 20 degrees, both the heater and the steering heating wire are on, the heater wind direction is set to upward, the heater temperature is set to the maximum value, and the heater air volume is not set to the maximum value, the corresponding vehicle A control action can be saved as an increase in heater airflow.

When the outside temperature is lower than 20 degrees and both the heater and the steering heating wire are on, the heater wind direction is upward, and both the heater temperature and the heater wind direction are set to the maximum values, if the heating wire seat is off, the corresponding vehicle control You can save heated seat-on as an action.

When the outside temperature is lower than 20 degrees, both the heater and the steering heating wire are on, the heater wind direction is upward, the heater temperature and the heater air volume are both set to the maximum values, and the heating wire seat is also on, the corresponding vehicle control As an action, since the heater is currently in full operation, a notice to wait for a while can be output.

26A and 26B are tables summarizing various examples in which the ambiguity resolver refers to the ambiguity resolving information DB to resolve ambiguity and extract actions to control the vehicle.

For example, as shown in FIGS. 26A and 26B , when the content of the utterance according to the morpheme analysis result is that the hand is cold or the hand is cold, the surrounding situation is summer and the vehicle state is that the wind direction of the air conditioner is the head of the occupant. If the air conditioner faces toward the side (upward), the set temperature of the air conditioner is 19 degrees, and the air volume is the situation, it can be understood that the hands are cold due to the air conditioning wind directed to the hands. Accordingly, it is possible to extract the air conditioner control action for reducing the wind volume intensity while changing the wind direction toward the foot (downward) as an action corresponding to the corresponding utterance, and control the vehicle according to the extracted action.

Also, in the utterance of the same content, when the surrounding situation is winter and the vehicle is in a situation where the wind direction of the heater is toward the occupant's feet, the air conditioner set temperature is 25, and the air volume is set strongly, the warmth is not delivered to the user's hand. You can tell that your hand is cold. Accordingly, an action of heating the steering wheel on (On) may be extracted as an action corresponding to the corresponding ignition, and the vehicle may be controlled according to the extracted action.

If the content of the utterance according to the morpheme analysis result is stuffy, when the vehicle speed is 30 km/h or less, and the distance between the front and rear of the vehicle is less than 30 cm, it can be understood that it is the stuffiness felt due to traffic jam. Accordingly, an action for changing a route option (quick route guidance), playing multimedia content such as music, or turning on a chat function in the navigation action is extracted as an action corresponding to the corresponding utterance as an action, and the extracted action You can control the vehicle accordingly.

If the content of the utterance according to the result of the morpheme analysis is drowsy, if the vehicle is in the betting mode, it can be understood that drowsiness is induced due to lack of air circulation. Accordingly, the action of changing to the outdoor mode may be extracted as an action corresponding to the corresponding utterance, and the vehicle may be controlled according to the extracted action.

Also, in the utterance of the same content, when the vehicle state is in the outdoor mode and the heater is on, it can be understood that drowsiness is induced due to the heat emitted from the heater. Accordingly, opening the window may be extracted as an action corresponding to the corresponding utterance, and the vehicle may be controlled according to the extracted action.

If the utterance according to the result of the morpheme analysis is sweating or hot, if the surrounding situation is winter and the vehicle condition is a situation in which the heater is on, it can be understood that the heat is caused by the heater's warmth. . Accordingly, lowering the temperature of the heater or reducing the air volume may be stored as an action corresponding to the corresponding ignition.

Also, in the utterance of the same content, if the surrounding situation is winter and the heater is off, it can be determined that the heat is induced by the user's body heat. Accordingly, the window opening or the window opening suggestion may be extracted as an action corresponding to the corresponding utterance, and the vehicle may be controlled according to the extracted action.

In addition, in the utterance of the same content, if the surrounding situation is summer and the vehicle state is a situation in which the air conditioner is off, it may be determined that the heat is induced due to the high internal temperature of the vehicle. Accordingly, it is possible to extract the air conditioner execution (On) as an action corresponding to the corresponding utterance, and control the vehicle according to the extracted action.

In addition, in the utterance of the same content, if the surrounding situation is summer and the vehicle state is a situation in which the air conditioner is on, it can be understood that the heat is induced because the temperature of the air conditioner is set high. Accordingly, lowering the air conditioner temperature or increasing the air volume may be extracted as an action corresponding to the corresponding ignition, and the vehicle may be controlled according to the extracted action.

If the utterance according to the morpheme analysis result is cold, if the surrounding situation is summer and the vehicle is in a situation where the air conditioner is on, the temperature of the air conditioner is set too low or the wind of the air conditioner is too strong. can be found to have been Accordingly, increasing the air conditioner temperature or decreasing the air volume may be extracted as an action corresponding to the corresponding ignition, and the vehicle may be controlled according to the extracted action.

In addition, in the utterance of the same content, if the surrounding situation is summer and the vehicle state is a situation in which the air conditioner is off, it may be determined that the cold is induced due to a problem in the user's physical condition. Accordingly, the execution of the heater or the user's biorhythm check may be extracted as an action corresponding to the corresponding utterance, and the vehicle may be controlled according to the extracted action.

In addition, in the utterance of the same content, if the surrounding situation is winter and the vehicle state is a situation in which the heater is on, it can be understood that the cold is induced because the temperature of the heater is set low or the air volume is weak. Accordingly, increasing the heater temperature or increasing the air volume may be extracted as an action corresponding to the corresponding ignition, and the vehicle may be controlled according to the extracted action.

Also, in the utterance of the same content, if the surrounding situation is winter and the heater is off, it may be understood that the cold is induced due to the non-execution of the heater. Accordingly, the heater execution may be extracted as an action corresponding to the corresponding utterance, and the vehicle may be controlled according to the extracted action.

If the content of the utterance according to the morpheme analysis result is a headache, if the surrounding situation is winter and the vehicle is in a state where the heater is on, it can be understood that the headache is caused due to lack of air circulation. Accordingly, it is possible to extract outdoor mode conversion or window opening as an action corresponding to the corresponding utterance, and control the vehicle according to the extracted action.

Also, in the utterance of the same content, if the surrounding situation is winter and the vehicle state is a situation in which the heater is off, it may be understood that the headache is induced by the cold. Accordingly, the heater execution may be extracted as an action corresponding to the corresponding utterance, and the vehicle may be controlled according to the extracted action.

In addition, in the utterance of the same content, if the surrounding situation is summer and the vehicle is in a state in which the air conditioner is off, it can be understood that the headache is caused by the heat. Accordingly, the air conditioner may be extracted as an action corresponding to the utterance, and the vehicle may be controlled according to the extracted action.

In addition, in the utterance of the same content, if the surrounding situation is summer and the vehicle is in a state in which the air conditioner is on, it can be understood that the headache is caused by the air conditioner. Accordingly, a change in the wind direction or air volume of the air conditioner may be extracted as an action corresponding to the corresponding utterance, and the vehicle may be controlled according to the extracted action.

If the content of the utterance according to the morpheme analysis result is unpleasant, if the surrounding environment is winter and it is raining, it can be understood that the discomfort is caused by high humidity. Accordingly, the execution of the defogging function or the execution of the dehumidifying function may be extracted as an action corresponding to the corresponding utterance, and the vehicle may be controlled according to the extracted action.

Also, in the utterance of the same content, if the surrounding situation is summer and there is no rain, it can be understood that discomfort is induced due to seasonal characteristics and heat. Accordingly, running the air conditioner at the lowest temperature may be extracted as an action corresponding to the corresponding ignition, and the vehicle may be controlled according to the extracted action.

Also, in the utterance of the same content, if the surrounding situation is summer and it is raining, it can be understood that discomfort is induced by the heat and high humidity. Accordingly, running the air conditioner in the dehumidifying mode may extract an action corresponding to the corresponding ignition and control the vehicle according to the extracted action.

According to the operation of the ambiguity resolver 123 described above, even when there is ambiguity in the user's utterance or situation, the user's actual action or the user's actual action is actually given by considering the surrounding situation information and the vehicle state information together with the user's utterance. It is possible to accurately identify and provide necessary actions.

Information on the action determined by the ambiguity resolver 123 may be transmitted to the dialogue action manager 122 , and the dialogue action manager 122 may update the dialogue/action DB 147 based on the transmitted information. .

The action prioritization determiner 125 and the factor manager 124 also determine the action execution condition for the action determined by the ambiguity resolver 123, determine the priority, and obtain the factor value as described above. do.

When all values obtainable through the current situation and conversation among the factor values used to perform each action are obtained, the conversation action manager 122 transmits a signal to the conversation flow manager 121 .

On the other hand, the factor values necessary for action execution or condition determination do not exist in the dialogue/action DB 147 , the external content server 300 , the long-term memory 143 , the short-term memory 144 , and the context information DB 142 . , it is also possible for the result processor 130 to generate a dialogue response for querying the user for the factor value when it can be obtained only through the user.

The conversation flow manager 121 transmits information about the action corresponding to the first priority and the conversation state to the result processor 130 . It is also possible to deliver information about a plurality of candidate actions according to a conversation policy.

27 is a control block diagram in which the configuration of the result processor is subdivided.

Referring to FIG. 27 , the result processor 130 is a response generation manager 131 that manages generation of a response required to perform an action input from the conversation manager 120 , and a text according to a request from the response generation manager 131 . A command generator 136 that generates a command for vehicle control or a command for providing a service using external content according to a request of the dialog response generator 132 for generating a response, an image response, or an audio response, and the response generation manager 131 . ), a service editor 134 that sequentially or sporadically executes a number of services and collects the result values to provide the service desired by the user, outputs the generated text response, image response or audio response, or the command generator 136 is Long-term memory 143 and short-term memory ( and a memory manager 135 for managing 144 .

The result processor 130 may include a memory in which a program for performing an operation to be described above or below is stored and a processor for executing the stored program. At least one memory and processor may be provided, respectively, and when a plurality of memory and processor are provided, they may be integrated on one chip or may be physically separated.

In addition, each of the components included in the result processor 130 may be implemented by a single processor or may be implemented by a separate processor.

In addition, the result processor 130 , the conversation manager 120 , and the input processor 110 may also be implemented by a single processor or may be implemented by separate processors.

The response output in response to the user's utterance or situation may include a dialogue response, vehicle control, external content provision, and the like. The dialogue response may have the form of an initial dialogue, a question, an answer including information provision, etc., and may be stored in a database in the response template 149 .

The response generation manager 131 requests the dialogue response generator 132 and the command generator 136 to generate a response necessary to perform the action determined by the dialogue manager 120 . To this end, information about the action to be performed may be transmitted to the dialog response generator 132 and the command generator 136 , and the information about the action to be performed may include an action name, a factor value, and the like. In generating a response, the dialog response generator 132 and the command generator 136 may refer to the current dialog state and the action state.

The dialog response generator 132 may search the response template 149 to extract a dialog response format, and generate a dialog response by filling factor values necessary for the extracted dialog response format. The generated dialogue response is transmitted to the response generation manager 131 . When a factor value necessary for generating a dialog response is not delivered from the dialog manager 120 or an instruction to use external content is delivered, it is provided from the external content server 300 or long-term memory 143, short-term memory 144, or It can be searched in the context information DB 142 .

For example, if the action determined in the conversation manager 120 corresponds to a route guide, the response template 149 is searched and "[Required time:-] from [Current location:-] to [Destination:-] It is expected. Shall we start the guidance?" can be extracted as a dialog response format.

Among the factors to be filled in the dialog response form, the factor values of [Current Location] and [Destination] may be transmitted from the conversation manager 120, and the factor value of [Required Time] may not be delivered. In this case, the conversation response generator 132 may request the external content server 300 the time taken from the [current location] to the [destination].

When the response to the user's utterance or situation includes vehicle control or external content provision, the command generator 136 generates a command for executing the command. For example, when the action determined by the conversation manager 120 is control of an air conditioner, a window, a seat, an AVN, etc., a command for executing the corresponding control is generated and transmitted to the response generation manager 131 .

Alternatively, when the action determined by the conversation manager 120 requires the provision of external content, a command for receiving the corresponding content from the external content server 300 is generated and transmitted to the response generation manager 131 .

When there are a plurality of commands generated by the command generator 136 , the service editor 134 determines a method and order for executing the plurality of commands and transmits them to the response generation manager 131 .

The response generation manager 131 transmits the response received from the dialog response generator 132 , the command generator 136 , or the service editor 134 to the output manager 133 .

The output manager 133 determines the output timing, output order, output position, etc. of the dialog response generated by the dialog response generator 132 and the command generated by the command generator 136 .

The output manager 133 transmits the dialog response generated by the response generator 132 and the command generated by the command generator 136 to an appropriate output location at an appropriate timing and in an appropriate order to output a response. A Text to Speech (TTS) response may be output through the speaker 232 , and a text response may be output through the display 231 . In the case of outputting the dialogue response in the form of TTS, a TTS module provided in the vehicle 200 may be used or the output manager 133 may include a TTS module.

The command may be transmitted to the vehicle controller 240 or to the communication device 280 for communicating with the external content server 300 according to the control target.

The response generation manager 131 may also transfer the response received from the dialog response generator 132 , the command generator 136 , or the service editor 134 to the memory manager 135 .

Also, the output manager 133 may transmit a response output by the output manager 133 to the memory manager 135 .

The memory manager 135 manages the long-term memory 143 and the short-term memory 144 based on the content received from the response generation manager 131 and the output manager 133 . For example, the memory manager 135 may update the short-term memory 144 by storing the conversation content between the user and the system based on the generated and output dialog response, and user-related information obtained through the conversation with the user. may be stored to update the long-term memory 143 .

Also, among the information stored in the short-term memory 144 , meaningful and persistent information such as a user's disposition or preference or information that can be used to obtain such information may be stored in the long-term memory 143 .

In addition, the user's preference or vehicle control history stored in the long-term memory 143 may be updated based on a vehicle control corresponding to the generated and output command or an external content request.

According to the dialogue system 100 according to the above-described embodiment, it is possible to provide an optimal service required to the user in consideration of various situations occurring inside the vehicle. In particular, even if the user's utterance is not input, it is possible to determine a service necessary for the user by himself/herself based on the situation information or driver information collected by the dialogue system 100 and to preemptively provide it.

For example, it is possible to variably change the evaluation criteria of the vehicle state according to the situation at the time of starting the vehicle and preemptively provide feedback thereto. The driving start time is defined as when the vehicle is started, when the Electronic Parking Brake (EPB) is released, or when a navigation destination is set. A weight is given to the vehicle condition evaluation system that calculates the drivability score and individual devices, and the variable weight applied to the individual devices is changed according to situational factors. When it is determined that there is a problem in the vehicle state, it is possible to provide a problem solving method for individual devices, such as a guide to a repair shop.

In addition, it is possible to determine whether the flow rate is insufficient compared to the destination when the vehicle is started, and if the flow rate is insufficient, the user's preferred gas station can be added as an automatic stopover in the route to the destination as a feedback, and the user can be informed of this. In addition, the gas station added as an automatic waypoint may be changed according to the user's response.

In addition, even if the current vehicle state does not indicate insufficient flow rate, a refueling place or time may be preemptively provided by comprehensively considering the user's future schedule, major movement records, residual oil, and the like.

In addition, information on the driver's physical condition and sleep record may be obtained, and vehicle start authorization may be conditionally granted based on the obtained information. For example, when the possibility of drowsy driving is recognized by recognizing the physical condition and sleep record from outside the vehicle, non-driving may be recommended. Or, it can provide information about recommended driving times based on your physical condition or sleep history.

In addition, when a trigger indicating the possibility of drowsy driving occurs repeatedly, it is possible to detect the possibility of drowsy driving and output a warning or provide feedback for automatically changing a route (resting area) according to the degree of the detected possibility. The trigger indicating the possibility of drowsy driving can be obtained by passively measuring the driver's condition or the vehicle's condition, such as when a drop in heart rate appears, when the distance to the preceding vehicle is greater than or equal to the reference distance, and when the speed is less than or equal to the reference speed, It is also possible to actively acquire it through conversation, such as uttering a question to the driver and measuring the driver's response speed.

In addition, when the user inputs a speech expressing emotion, a specific domain or action cannot be extracted from the user's speech, but the dialog system 100 uses the surrounding situation information, vehicle status information, user status information, etc. Able to understand intentions and develop conversations. This example may be performed by resolving the ambiguity of the user's utterance in the ambiguity resolver 230 as described above.

Hereinafter, an example of specific dialog processing using the dialog system 100 according to an embodiment will be described in detail.

28 to 40 are diagrams illustrating specific examples in which the dialog system 100 processes the input, manages the dialog, and outputs a result when the user inputs an utterance related to road guidance.

As shown in FIG. 28 , when the user inputs an utterance “Let’s go to Seoul Station yesterday”, the voice recognizer 111a outputs the user’s voice as a text utterance (let’s go to Seoul Station yesterday).

The natural language interpreter 111b analyzes the morpheme, refers to the domain/action inference rule DB 141, and uses the morpheme analysis result (Yesterday/NNG, Gan/VV, Seoul Station/NNP, Gaja/VV) from [Domain: Navigation ], [Action: Directions], [Dialogue Act: Request], [Factor: NLU Destination: Seoul Station] may be extracted and input to the dialog input manager 111c.

Referring to FIG. 29 , the dialogue input manager 111c transmits the natural language understanding result of the natural language understander 111b to the context understander 112c and requests to send additional information, if any.

The context understanding unit 112c may search the context understanding table 145 to extract that context information related to [Domain: Navigation] and [Action: Directions] is the current location and the context information type is a GPS value.

The context understandr 112c searches the context information DB 142 to extract a GPS value of the current location. When the GPS value of the current location is not stored in the context information DB 142 , the GPS value of the current location is requested from the context information collection manager 112b.

The contextual information collection manager 112b sends a signal to the contextual information collector 112a to collect the GPS value of the current location, and the contextual information collector 112a collects the GPS value from the vehicle controller 240 to obtain a contextual information DB ( 142) while transmitting the GPS value collection confirmation signal to the context information collection manager 112b. When the context information collection manager 112b transmits a GPS value collection confirmation signal to the context understandr 112c, the context understandr 112c extracts the GPS value of the current location from the contextual information DB 142 to obtain a dialog input manager ( 111c).

The dialog input manager 111c provides natural language understanding results [Domain: Navigation], [Action: Directions], [Dialogue Act: Request], [Factor: NLU: Destination: Seoul Station] and [Situation Information: Current Location: Uiwang Station (GPS). value)] and transmitted to the conversation manager 120 .

Referring to FIG. 30 , the conversation flow manager 121 searches the conversation/action state DB 147 to determine whether there is an ongoing conversation task or action task. In this case, the conversation policy DB 148 may be referred to. In this example, it is assumed that there is no conversation task or action task in progress.

The dialog flow manager 121 requests the dialog action manager 122 to generate an action task and a dialog task corresponding to the output of the input processor 110 . Creating an action task and a dialog task means allocating a storage space for storing and managing information about an action state and a dialog state.

Accordingly, the dialog action manager 122 allocates a storage space in the dialog/action state DB 147 to store information on the action state and information on the dialog state.

The dialog action manager 122 transmits the action state and the dialog state to the action prioritization determiner 125 .

The action prioritization determiner 125 searches the related action DB 146b for actions related to road guidance, such as vehicle condition check and gas station recommendation. The navigation action and associated actions become candidate actions.

The action prioritization determiner 125 may determine the priority of each candidate action according to a pre-stored rule. The priority may be determined before the execution condition for each candidate action is determined, or may be determined only for candidate actions that satisfy the execution condition after the execution condition is determined.

The candidate action list is transmitted to the dialog action manager 122 again, and the dialog action manager 122 updates the action state by adding the searched associated actions.

Referring to FIG. 31 , the action priority determiner 125 searches for an execution condition for each candidate action and a factor for determining the execution condition from the action execution condition DB 146c. In addition, it is also possible to determine the priority between the respective candidate actions.

For example, the condition for performing the vehicle condition check may be a case where the destination distance is 100 km or more, and a factor necessary for determining the execution condition is the destination distance.

A condition for performing the gas station recommendation may be a case where the destination distance is greater than the drivable distance (DTE), and factors necessary for determining the execution condition are the destination distance and the drivable distance.

The dialog action manager 122 updates the action state by adding a condition for performing each candidate action and a factor necessary for determining the condition to the dialog/action state DB 147 .

The action prioritization determiner 125 sets a factor value necessary for determining whether the performance condition of each candidate action is satisfied in the conversation/action state DB 147, the context information DB 142, the long-term memory 143, or the short-term memory 144 ) can be retrieved from

When the factor value is included in the previous conversation content, included in context information related to the conversation content, or included in redemption information related to an event that occurred, the factor value may be retrieved from the conversation/action state DB 147 .

If the argument value cannot be retrieved from the conversation/action state DB 147 , the context information DB 142 , the long-term memory 143 , or the short-term memory 144 , it may be requested from the external information manager 126 .

For example, the destination distance may be obtained from the external content server 300 providing a navigation service through the external information manager 126 , and the DTE may be obtained from the context information DB 142 . Meanwhile, in order to search for a destination distance, accurate destination information used in a navigation service is required. In this example, the destination input from the user's utterance corresponds to "Seoul Station", and Seoul Station may indicate various types of places starting with Seoul Station or may be Seoul Station with a specific meaning. Therefore, it is difficult to search for an exact destination distance only with "Seoul Station".

In addition, it is also possible to bring the factor value from the mobile device 400 connected to the vehicle 200 if necessary. For example, when user information such as contact information and schedule not stored in the long-term memory 143 is required as a factor value, the external information manager 126 requests the mobile device 400 for necessary information to obtain the necessary factor value. can do.

In addition, when it is not possible to obtain a factor value through the storage 140 , the external content server 300 , and the mobile device 400 , a necessary factor value may be obtained by querying the user.

The action prioritization determiner 125 determines the execution condition of each candidate action by using the factor value. Since the destination distance is not searched, the determination of the execution condition is withheld for the vehicle condition check action and the gas station recommendation action.

As shown in FIG. 32 , the dialog action manager 122 updates the action state by adding the acquired factor value to the dialog/action state DB 147 and whether the action execution condition determined using the factor value is satisfied. .

The dialog action manager 122 requests the factor manager 124 for a list of factors used to perform each candidate action.

The factor manager 124 may extract the current location and destination as essential factors used for performing the route guidance action from the action factor 146a, and may extract a path type (initial value: fast path) as an optional factor.

An inspection part (initial value: all parts) can be extracted as a selection factor used to perform the vehicle condition check action, and a preferred gas station (initial value: Oil A) can be extracted as a selection factor used to perform the gas station recommendation action. .

The extracted factor list may be transmitted to the dialog action manager 122 and used to update the action status.

The factor manager 124 includes the conversation/action state DB 147, the context information DB 142, the long-term memory 143 and the short-term memory 144 to obtain factor values corresponding to the essential factors and optional factors of each candidate action. ), a corresponding factor value is retrieved from the reference position of each factor, and when the factor value is to be provided through an external service, a required factor value can be requested from the external content server 300 through the external information manager 126 .

A factor used to determine the execution condition of the candidate action and a factor used to perform the candidate action may overlap each other. Among the factor values obtained by the action prioritization determiner 125 and stored in the conversation/action state DB 147, if there is a factor corresponding to a factor (essential factor, optional factor) used to perform a candidate action, it may be used.

Referring to FIG. 33 , the dialog action manager 122 updates the action state by adding the factor value obtained by the factor manager 124 .

As described above, there is ambiguity in that the destination (Seoul Station) extracted from the user's utterance is used as a factor of the road guidance action. Accordingly, the factor values of the factor (destination) of the road guidance action, the factor of the vehicle condition check action (destination distance), and the factor (destination distance) of the gas station recommendation action have not yet been obtained.

The ambiguity resolver 123 checks whether there is an ambiguity in converting [argument:NLU:destination:Seoul Station] into a destination argument suitable for a navigation action. As described above, "Seoul Station" may mean another type of place starting with Seoul Station, or may be Seoul Station at a specific location that the user means.

The ambiguity resolver 123 refers to the morpheme analysis result and confirms that there is a modifier of Seoul Station among the user's utterances. The ambiguity resolver 123 searches the long-term memory 143 and the short-term memory 144 for schedules, moving locations, contacts, etc. in order to determine the location of "Seoul Station between yesterday".

For example, the ambiguity resolver 123 may confirm that "Seoul Station between yesterday" is "Seoul Station Exit 4" from the user's yesterday's movement location. Check that there is a POI of "Seoul Station Exit 4" and get the corresponding value.

The destination information obtained by the ambiguity resolver 123 is transmitted to the dialog action manager 122, and the dialog action manager 122 updates the action state by adding “Seoul Station Exit 4” to the destination factor of the candidate action.

The factor manager 124 obtains destination information (Seoul Station Exit 4) from the conversation/action DB 147, and provides a destination distance value to an external content server 300 that provides a navigation service through the external information manager 126. request.

Referring to FIG. 34 , when the external information manager 126 obtains a destination distance value (80 km) from the external content server 300 and transmits it to the factor manager 124, the factor manager 124 transmits it to the conversation action manager 122 ) so that the action state can be updated.

The action prioritization determiner 125 may determine whether a candidate action can be performed by referring to the action state, and may adjust the priority. It can be determined that the route guidance action can be performed because the essential factors of the current location and the destination are obtained, and the vehicle condition check action cannot be performed because the destination distance (70km) is less than 100km. It may be determined that the gas station recommendation action can be performed because the destination distance (80 km) is longer than the drivable distance (DTE).

Since the vehicle condition check action cannot be performed, it is excluded from prioritization. Accordingly, the navigation action is adjusted to priority 1, and the gas station recommendation action is adjusted to priority 2.

The dialog action manager 122 updates the action state according to whether the candidate action is performed or not and the adjusted priority.

The dialog flow manager 121 checks the dialog state and action state stored in the dialog/action state DB 147 , and develops a dialog strategy for conducting a dialog with reference to the dialog policy DB 148 . For example, the response generation manager 131 may be requested to select an action having the highest priority among the actions that can be performed, and to generate a response for conducting a conversation according to the conversation policy DB 148 .

The dialog state and the action state stored in the dialog/action DB 147 may be updated to [Status: Guidance start confirmation], respectively.

Referring to FIG. 35 , the response generation manager 131 requests the dialogue response generator 132 to generate a response according to the request of the dialogue flow manager 121 .

The dialogue response generator 132 retrieves the response template 149 to generate a TTS response and a text response. For example, you can create a dialog response that can be printed out in TTS and text, "It is estimated that it will take 30 minutes from Uiwang Station to Exit 4 of Seoul Station. Shall we start the guidance?"

The response generation manager 131 transmits the TTS response and the text response generated by the dialogue response generator 132 to the output manager 133 and the memory manager 135 , and the output manager 133 transmits the TTS response to the speaker 232 . and transmits a text response to the display 231 . In this case, the output manager 133 may transmit a TTS response to the speaker 232 through a TTS module that synthesizes text into voice.

The memory manager 135 may store in the short-term memory 144 or the long-term memory 143 that the user has requested directions.

Through the display 231 and the speaker 232, a dialogue response asking “It is expected that it will take 30 minutes from Uiwang Station to Exit 4 of Seoul Station. Shall we start the guidance?” is output. As shown in FIG. 36 , when the user utters "yes", the user's utterance is input to the voice recognizer 111a and output as [text: yes], and the natural language understander 111b receives [ Domain:-], [Action:-], [Dialogue act: -], [Result of morphological analysis: Yes/IC] are output.

The natural language understanding result is transmitted to the dialogue input manager 111c, and the dialogue input manager 111c transmits it to the dialogue manager 120.

Referring to FIG. 37 , the dialog flow manager 121 searches the dialog/action state DB, analyzes the previous dialog state, and requests the dialog action manager 122 to update the ongoing [direction]-related dialog/action. .

The dialog action manager 122 updates the dialog state and the action state to [Status: Guidance start].

The conversation flow manager 121 requests the result processor 130 to generate a response for starting the route guidance.

Referring to FIG. 38 , the dialog action manager 122 updates the dialog state to [Status: Next dialog progress] and updates the action state to [Status: execute].

The conversation flow manager 121 requests the response generation manager 131 to generate a response for guidance.

The response generator 132 may generate a dialogue response capable of outputting "start guidance" in TTS and text, and transmit it to the response generation manager 131 .

The command generator 136 may generate a command [target: navigation, command: route guidance, destination: Seoul Station Exit 4, departure point: Uiwang Station] for performing route guidance and transmit it to the response generation manager 131 .

The response generation manager 131 transmits the generated dialogue response and the command to the output manager 133 . The output manager 133 outputs a dialogue response through the display 231 and the speaker 232 , and the navigation command is transmitted to the AVN 230 of the vehicle 200 through the control unit 240 or provides a navigation service. It can be transmitted to the external content server 300 .

Referring to FIG. 39 , the conversation flow manager 121 selects a gas station recommendation as the next actionable action, and requests the response generation manager 131 to generate a response that inquires the user about whether to recommend a gas station.

The conversation state and the action state may be updated with [Status: Confirmation of linked service recommendation].

The response generation manager 131 requests the dialogue response generator 132 to generate a TTS response and a text response, and the dialogue response generator 132 says, "There is not enough fuel to travel to the destination. Shall we add A Oil gas station as a stopover? It is possible to generate a dialogue response capable of outputting " as TTS and text and deliver it to the response generation manager 131 .

The response generation manager 131 transmits the TTS response and the text response generated by the dialogue response generator 132 to the output manager 133 and the memory manager 135 , and the output manager 133 transmits the TTS response to the speaker 232 . and transmits a text response to the display 231 .

Through the display 231 and the speaker 232, a dialogue response asking “There is not enough fuel to move to the destination. Would you like to add oil A gas station as a stopover?” is output. As shown in FIG. 40 , when the user utters “no”, the user's utterance is input to the voice recognizer 111a and output as [text: no], and the natural language understander 111b receives [ Domain:-], [action:-], [dialog act: -], [result of morphological analysis: no/IC] are output.

The conversation flow manager 121 requests the conversation action manager 122 to update the conversation state and the action state.

The dialog action manager 122 may update the dialog state to [Status: Next dialog progress] and update the action state to [Status: CANCEL].

The conversation flow manager 121 requests the response generation manager 131 to generate a response indicating that the gas station recommendation service has been canceled, and checks whether there is a next available conversation. If there are no further conversations available, update the conversation state to [Status: IDLE] and wait for user input.

The flow of data processing described above is only an example applied to the dialog system 100 . Accordingly, the order in which each component of the dialog system 100 processes data is not limited to the above example, and it is also possible for a plurality of components to simultaneously process data, and to process data in a different order from the above example. It is also possible

Hereinafter, a conversation processing method according to an embodiment will be described. The dialog system 100 or the vehicle 200 including the same may be applied to the dialog processing method according to an embodiment. Accordingly, the description of FIGS. 1 to 40 may be equally applied to the conversation processing method according to an embodiment.

41 is a flowchart illustrating a method of processing a user input in a conversation processing method according to an embodiment. The method of processing user input may be performed in the input processor 110 of the dialog system 100 .

Referring to FIG. 41 , when the user's utterance is input (Yes in 500), the voice recognizer 111a recognizes the inputted user's utterance (510). The user's utterance may be input through the voice input device 210 provided in the vehicle 200 or the voice input device 410 provided in the mobile device 400 .

The voice recognizer 111a recognizes the input of the user's utterance and outputs it as a utterance in the form of text.

The natural language understander 111b applies the natural language understanding technology to the utterance in the text form ( 520 ), and outputs the result.

Specifically, in the natural language understanding 520 process, a morpheme analysis 521 is performed based on an utterance in a text form, a domain is extracted 522 based on the morpheme analysis result, an entity name is recognized 523, It may include analyzing (524) the dialogue act and extracting (525) the action.

The domain/action inference rule DB 141 may be referred to for domain extraction, entity name recognition, and action extraction.

The output of the natural language understander 111b, that is, the natural language understanding result, may include a domain corresponding to the user's utterance, an action, a dialogue act, a morpheme analysis result, and the like.

Context information related to the extracted action is retrieved ( 530 ). The extracted action-related contextual information may be stored in the contextual understanding table 145 , and the contextual understandr 112c searches for the extracted action-related contextual information from the contextual understanding table 145 , and information of the retrieved contextual information The value is retrieved from the context information DB 142 , the long-term memory 143 , or the short-term memory 144 .

When additional context information is needed (Yes of 540), that is, when there is context information that cannot be retrieved from the context information DB 142, the long-term memory 143, or the short-term memory 144, the collection of the context information is requested ( 550). A non-voice input such as vehicle state information, surrounding environment information, and driver information may be input through the context information collector 111a independently of the user's speech input.

Such information may be periodically input, may be input when a specific event occurs, may be input periodically and may be additionally input when a specific event occurs. In either case, if the collection of information is requested, the information can be actively collected.

Therefore, if the action-related context information has already been collected, the corresponding information is fetched from the context information DB 142, the long-term memory 143, or the short-term memory 144, otherwise, the context information collector 111a collect that information through

When the contextual information collector 111a requested to collect contextual information collects the corresponding contextual information and stores it in the contextual information DB 142 , the contextual understandr 112c may retrieve the corresponding contextual information from the contextual information DB 142 . have.

Meanwhile, the contextual information collection manager 112b may transmit an action trigger signal to the contextual understandingr 112c when it is determined that a specific event has occurred because the data collected by the contextual information collector 112a satisfies a preset condition.

The context understandr 112c searches the context understanding table 145 to retrieve context information related to the corresponding event, and if the retrieved context information is not stored, transmits a request signal for context information to the context information collection manager 112b again do.

When the collection of necessary context information is completed, the natural language understanding result and context information are transmitted to the conversation manager 120 (560). When an event is generated, information related to the event (which event has occurred) and context information related to the generated event may also be transmitted.

42 is a flowchart illustrating a method of managing a conversation using an output of an input processor in a conversation processing method according to an embodiment. The conversation management method may be performed in the conversation manager 120 of the conversation system 100 .

Referring to FIG. 42 , the conversation flow manager 121 searches for a related conversation history in the conversation/action state DB 147 ( 600 ).

In this example, a case in which a domain and an action can be extracted from a user's utterance is described as an example. However, there may be cases in which the domain and action cannot be extracted from the user's utterance because the content or situation of the utterance itself is ambiguous. In this case, the dialog action manager 122 generates an arbitrary dialog state, and the ambiguity resolver 123 recognizes the user's intention based on the user's utterance content, surrounding situation, vehicle state, user information, etc. Appropriate action can be determined.

If the related conversation history exists (Yes of 600), it is referred to (690), and if it does not exist (No of 600), a new conversation task and an action task are created (610).

A candidate action list is generated by searching the related action list associated with the action extracted from the user's utterance (hereinafter, referred to as input action) in the related action DB 146b ( 620 ). do.

An execution condition for each candidate action is searched for in the action execution condition DB 146c (620). The execution condition is a condition necessary to perform the action. If the corresponding condition is satisfied, it is determined that the action can be performed, and if the condition is not satisfied, it is determined that the action cannot be performed. The action execution condition DB 146c also stores information on the type of factor used to determine the action execution condition.

A factor value used to determine the action execution condition is acquired ( 640 ). A factor used to determine the action execution condition will be referred to as a condition determination factor. A factor value of the condition determination factor may be obtained by searching the context information DB 142 , the long-term memory 143 , the short-term memory 144 , or the conversation/action state DB 147 . When the factor value of the condition determination factor is to be provided through an external service, the required factor value may be provided from the external content server 300 through the external information manager 126 .

In addition, when a necessary factor value cannot be obtained because the situation or utterance is ambiguous, the necessary factor value can be obtained by resolving the ambiguity through the ambiguity resolver 230 .

Also, even when the obtained factor is an invalid factor that is difficult to be used for determining an action execution condition, the ambiguity resolver 230 may acquire a valid factor from the invalid factor.

Based on the obtained condition determination factors, it is determined whether each candidate action can be performed ( 650 ), and the priority of the candidate actions is determined ( 660 ). A rule for determining the priority of candidate actions may be stored in advance. The action prioritization determiner 125 may determine the priority in consideration of only candidate actions capable of performing the action after it is determined whether the action can be performed. Alternatively, after determining the priority regardless of whether the actions are performed or not, the priority may be adjusted based on whether the actions are performed.

A list of factors used to perform a candidate action is retrieved from the action factor DB 146a ( 670 ). An argument used to perform an action is referred to as an action argument. Action factors may include mandatory and optional factors.

A factor value of an action factor used to perform a candidate action is acquired ( 680 ). A factor value of an action factor may be obtained by searching the context information DB 142 , the long-term memory 143 , the short-term memory 144 , or the conversation/action state DB 147 . When the factor value of the action factor is to be provided through an external service, the required factor value may be provided from the external content server 300 through the external information manager 126 .

In addition, when a necessary factor value cannot be obtained because the situation or utterance is ambiguous, the necessary factor value can be obtained by resolving the ambiguity through the ambiguity resolver 230 .

Also, even when the obtained factor is an invalid factor that is difficult to be used for performing an action, the ambiguity resolver 230 may obtain a valid factor from the invalid factor.

The dialog state and action state managed by the dialog action manager 122 may be updated whenever the state is changed while the above-described steps are performed.

When all obtainable factor values are obtained, the conversation flow manager 121 transmits information about the candidate action and the conversation state to the result processor 130 . According to the dialogue policy, it is possible to deliver only information about the action corresponding to the first priority, and it is also possible to deliver information about a plurality of candidate actions.

On the other hand, when the required factor value does not exist in the external content server 300, the long-term memory 143, the short-term memory 144, and the context information DB 142, and can be obtained only through the user, the factor value is provided to the user. It is also possible to output dialog responses for querying.

43 is a flowchart illustrating a result processing method for generating a response corresponding to a result of conversation management in the conversation processing method according to an embodiment. The result processing method may be performed by the result processor 130 of the dialog system 100 .

Referring to FIG. 43 , when generation of a dialog response is required (YES in 700 ), the dialog response generator 132 searches for a response template 149 ( 710 ). A dialog response format corresponding to the current dialog state and action state is extracted from the response template 149, and a dialog response is generated by filling factor values necessary for the dialog response format (720).

When a factor value necessary for generating a dialog response is not delivered from the dialog manager 120 or an instruction to use external content is delivered, it is provided from the external content server 300 or long-term memory 143, short-term memory 144, or It can be searched in the context information DB 142 . When the required factor value does not exist in the external content server 300, the long-term memory 143, the short-term memory 144, and the context information DB 142, and can only be obtained through the user, query the user for the factor value It is also possible to create a dialog response for

When it is necessary to generate a command ( 760 ), the command generator 136 generates a command for vehicle control or external content use ( 770 ).

The generated dialogue response or command is input to the output manager 133 , and the output manager 133 may determine an output order between the dialogue response and the command or an output order between a plurality of commands ( 730 ).

The memory is updated based on the generated dialog response or command ( 740 ). The memory manager 135 may update the short-term memory 144 by storing the conversation content between the user and the system based on the generated and output dialog response, and store user-related information obtained through the conversation with the user to store the long-term The memory 143 may be updated. In addition, based on the generated and output vehicle control or external content request, the user's preference or vehicle control history stored in the long-term memory 143 may be updated.

The output manager 133 outputs the response by sending the dialog response and the command to an appropriate output location (750). The TTS response may be output through the speaker 232 , and the text response may be output through the display 231 . The command may be transmitted to the vehicle controller 240, the external content server 300, or the communication device 280 for communicating with the external content server 300 according to the control target. .

The conversation processing method according to an embodiment is not limited by the order in the above-described flowchart. The flow according to the flowchart of FIGS. 41 to 43 is only an example applicable to the conversation processing method, and a plurality of steps may be simultaneously performed, and the order of each step may be changed.

44 is a configuration diagram of a vehicle according to another exemplary embodiment, and FIG. 45 is a detailed configuration diagram of the dialogue system shown in FIG. 44 .

When the voice recognized by the dialog system 1100 is a voice corresponding to the notification situation, the vehicle 1200 according to another embodiment provides situation information about the notification situation to the server 1300 of the agency in charge corresponding to the notification situation. .

Here, the notification situation includes illegal behavior situations of other vehicles such as shoulder driving, retaliatory driving, signal violations and tail biting, and accident situations of other vehicles, and obstacles such as road flooding, icing on the road, and falling objects on the road Existing state conditions may be further included.

The server 1300 may include at least one of the servers of the National Police Agency 112 , the safety report center 119 , the Ministry of Land, Infrastructure and Transport, and the traffic information center.

The server 1300 may be a server for vehicle communication for performing V2X or V2V, a server of a mobile communication company to which the owner's mobile device is subscribed, and a server for conversation.

The server 1300 performs accident processing in response to a traffic accident, and punitive processing for illegal acts.

In addition, the server 1300 may provide road situation information to other vehicles located in the vicinity through V2X or V2V, and it is also possible to output road situation information through a mobile device.

As shown in FIG. 44 , a vehicle 1200 according to another embodiment includes a conversation system 1100 , an audio input device 1210 , an image input device 1220 , an output device 1230 , a controller 1240 , and a communication device. (1280).

The voice input device 1210 may receive the user's spoken voice.

The voice input device 1210 may include a microphone or a microphone array that receives sound, converts it into an electrical signal, and outputs it.

For effective voice input, the voice input device 1210 may be provided on the headlining 205 as shown in FIG. 2 , but the embodiment of the vehicle 200 is not limited thereto. It is also possible to be provided or provided on the steering wheel (207). In addition to this, there is no limitation wherever it is suitable for receiving the user's voice input.

The voice input device 1210 is always operable or operable in a starting state, and it is also possible to perform an operation when a user is present in the vehicle.

In the notification mode, the voice input device 1210 includes a user's command for instructing information collection and information output on notification situations such as accidents, illegal acts, caution, etc., and answers to notification situations and questions of the dialog system, The vehicle number of another vehicle can be input by voice.

The image input device 1220 captures a road and its surroundings, and transmits the captured image of the road to the controller 1240 .

The image input device 1220 may include a camera or an image sensor for photographing the front of the vehicle.

Here, the camera may be a camera for autonomous driving, a camera for a black box, a camera for detecting obstacles, or a rear camera.

In addition to the front of the vehicle, the image input device 1220 may also photograph a road in at least one of a rear and a side direction of the vehicle.

The image input device 1220 may include one non-rotatable camera, a plurality of non-rotatable cameras, or a single rotatable camera.

The image input device 1220 may be always operable or operable in a starting state, and may perform an operation when it is determined that a notification situation has occurred through voice recognition.

The image input device 1220 may always operate when the power of the power supply is equal to or greater than the reference power.

The vehicle 1200 may further include input buttons 221 and 223 and a jog shuttle 225 for receiving commands through a user's manipulation in addition to voice (refer to FIG. 2 ).

The input buttons 221 and 223 may receive an on/off operation of the notification mode for informing the outside of road situation information.

The on/off operation of the notification mode may be input using the jog shuttle 225 or may be inputted by voice using a voice input device.

The input buttons 221 and 223 may also receive the vehicle number of another vehicle.

In addition, the voice input device 1210 and the image input device 1220 may directly transmit input information to the chat system 1100 in the notification mode.

The output device 1230 is a device that provides a visual, auditory, or tactile output to a user, and may include a display 1231 and a speaker 1232 provided in the vehicle 1200 .

The output device 1230 may include a display 1231 and a speaker 1232 provided in the conversation system 1100 .

The display 1231 and the speaker 1232 may visually or aurally output a response to a user's utterance, a query to the user, or information requested by the user.

The display 1231 may output an image for guiding a navigation route, an image of content, an image for providing information or service desired by a user, a text image corresponding to a system utterance generated as a response to the user's utterance, and the like. .

In addition, the display 1231 may display an image corresponding to the context information on a portion of the screen in the notification mode.

For example, when it is necessary to display context information in the notification mode while displaying an image for guiding a navigation route, the display 1231 removes a portion of the image for guiding a navigation route and displays the status of the notification mode in the removed portion of the screen area Displays the image corresponding to the information.

The speaker 1232 outputs audio for guiding a navigation route, audio of content (including sound and voice), a voice for providing information or service desired by a user, a voice of a dialog system generated as a response to a user's utterance, etc. can do.

In addition, the speaker 1232 may preferentially output the voice of the dialog system corresponding to the context information over other audio in the notification mode.

For example, when it is necessary to output the voice generated by the dialog system 1100 during audio output for navigation route guidance, the speaker 1232 pauses the audio output for navigation route guidance, and output the voice

In addition, the output device 1230 may further include a vibrator provided on the steering wheel 207, and it is also possible to output a response to the user's utterance as vibration through the vibrator.

The output device 1230 may also output on/off information of the notification mode.

In addition, the output device 1230 may output the vehicle number of another vehicle in the notification mode, and it is also possible to output the name of the reporting center.

The controller 1240 controls the vehicle 1200 to perform an action corresponding to a user's intention or a current situation according to output information (eg, a response) output from the dialogue system 1100 .

The controller 1240 performs a notification mode for letting other users or agencies in charge know of a dangerous situation on a road, an illegal activity situation, an accident situation, etc. according to mutual conversation information with the conversation system 1100 .

The controller 1240 may always perform the notification mode, and may also perform according to the user's on/off operation input.

The controller 1240 transmits the voice information of the voice input device 1210 to the conversation system 1100 in the notification mode, processes the image information transmitted from the image input device 1220, and stores the image-processed image information do.

The controller 1240 matches and stores the image information of the image-processed road with at least one of date information, time information, and location information of the photographed road when storing the image information.

The controller 1240 may determine whether a notification situation has occurred based on the processing result of the conversation system 110 .

When a notification situation occurs, the controller 1240 may also transmit an image at the time the notification situation occurs to the chat system 1100 .

The controller 1240 checks the time point at which a voice corresponding to the notification situation is input to the voice input device 1210, and determines when a previous time point before the first predetermined time and a second predetermined time have elapsed from the confirmed time point. After checking the time point, the image information between the previous time point and the subsequent time point is obtained, and the obtained image information is transmitted to the server.

When the image information is transmitted to the server 1300 , the controller 1240 matches the metadata, which is data related to an attribute required for data management, and transmits the matched information.

Here, the metadata may include at least one of location information at which the image was captured, date information, time information, vehicle numbers of other vehicles in the image, and road situation information.

That is, when a notification situation occurs, the controller 1240 controls the communication device 1280 to transmit at least one of image information, result information of a conversation system, date information, time information, and location information to the server 1300 .

Here, the result information of the dialogue system may include information on the agency in charge, situation information such as traffic accidents and illegal acts, and whether or not to report them.

If the notification situation is a traffic accident situation, the controller 1240 may recognize the vehicle number of the traffic accident vehicle through image processing of the stored image and transmit the recognized vehicle number to the conversation system 1100 .

The controller 1240 controls at least one of the display 1231 and the speaker 1232 so that result information corresponding to the processing result of the conversation system 1100 is outputted as at least one of an image and an audio.

The controller 1240 may also recognize the road condition based on the output information and the image information output from the dialogue system 1100 .

For example, if the situation information recognized by the dialogue system 1100 is a situation for an illegal activity, the controller 1240 recognizes whether the situation of the illegal activity is tail biting, signal violation, shoulder driving or retaliatory driving based on the image information, and By transmitting the recognized illegal activity situation to the conversation system 1100 , the conversation system 1100 can output a query about the recognized reporting situation.

The controller 1240 may recognize the illegal activity based on the location of the intersection in the image, the color of the traffic light, the location of the lane, the location of another vehicle, and the speed of the vehicle.

As another example, if the situation information recognized by the dialogue system 1100 is an attention situation, the controller 1240 recognizes whether the surrounding situation is an attention situation due to icing, immersion, or an obstacle based on the image information, and sets the recognized attention situation to the dialogue system By transmitting to 1100 , it is possible to output a query about the attention situation recognized by the conversation system 1100 .

As another example, the controller 1240 recognizes whether an obstacle on the road is another vehicle based on the image information of the road, determines that it is a traffic accident if the obstacle on the road is another vehicle, and if the obstacle on the road is not a vehicle, It is also possible to determine that the object is an object. Based on the image information of the road, by identifying the lane where the road obstacle is located, and transmitting the road obstacle information and location information to the dialogue system 1100, the dialogue system 1100 It allows you to output queries about obstacles.

When a notification situation occurs, the controller 1240 may automatically perform deceleration control to acquire an image of a precise road, and it is also possible to automatically perform steering control.

When a notification situation occurs, the controller 1240 may output deceleration guidance information and steering guidance information through at least one of the display 1231 and the speaker 1232 to obtain a precise image of the road.

The controller 1240 may output road situation information as voice through the dialogue system 1100 .

The controller 1240 may receive the image information of the surrounding road received from another vehicle or server and output the image of the surrounding road through the display.

The controller 1240 may receive the surrounding road situation information received from another vehicle or server and output the surrounding road situation through the speaker 1232 or the display 1231 .

The controller 1240 may include a memory in which a program for performing the above-described operation and an operation to be described later is stored and a processor for executing the stored program. At least one memory and processor may be provided, respectively, and when a plurality of memory and processor are provided, they may be integrated on one chip or may be physically separated.

The communication device 1280 communicates with the controller 1240 and the server 1300 .

The communication device 1280 may include a GPS receiver for recognizing the current location of the vehicle.

The communication device 1280 transmits the GPS value of the current location received from the GPS receiver to the controller 1240 when capturing an image of the road.

The communication device 1280 may include one or more communication modules that enable communication with an external device, and may include, for example, a short-range communication module, a wired communication module, and a wireless communication module.

The short-distance communication module transmits signals using a wireless communication network in a short distance such as a Bluetooth module, an infrared communication module, an RFID (Radio Frequency Identification) communication module, a WLAN (Wireless Local Access Network) communication module, an NFC communication module, and a Zigbee communication module. It may include at least one of various short-distance communication modules for transmitting and receiving.

The wired communication module includes various wired communication modules such as a Local Area Network (LAN) module, a Wide Area Network (WAN) module or a Value Added Network (VAN) module, as well as a USB (Universal Serial Bus) module. , HDMI (High Definition Multimedia Interface), DVI (Digital Visual Interface), RS-232 (recommended standard232), power line communication, or POTS (plain old telephone service) may include at least one of various cable communication modules such as.

In addition to the Wifi module and the Wireless broadband module, the wireless communication module includes Global System for Mobile Communication (GSM), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), and Universal Mobile Telecommunications System (UMTS). ), Time Division Multiple Access (TDMA), Long Term Evolution (LTE), 4G, 5G, and the like, and may include at least one of various wireless communication modules that can be connected to the Internet network.

Meanwhile, the communication device 1280 may further include an internal communication module (not shown) for communication between electronic devices inside the vehicle 1200 . As an internal communication protocol of the vehicle 200 , a controller area network (CAN), a local interconnection network (LIN), FlexRay, Ethernet, or the like may be used.

The communication device 1280 may perform Vehicle to Everything (V2X).

That is, the communication device 1280 may transmit image information to another vehicle through Vehicle to Everything (V2X).

V2X refers to the exchange and sharing of various useful information such as traffic conditions while communicating with road infrastructure and other vehicles while the vehicle is driving.

V2X communication includes vehicle-to-infrastructure (V2I: Vehicle-to-Infrastructure, hereinafter V2I) communication, vehicle-to-vehicle (V2V: Vehicle to Vehicle), and vehicle-to-mobile device (V2N: Vehicle-to-Nomadic devices) communication. can

The communication device 1280 may also receive image information of another vehicle through V2X.

The image information of the other vehicle may be image information corresponding to the current location of the own vehicle and the condition of a road adjacent to the current location.

The dialog system 1100 recognizes the user's voice input through the voice input device 1210 to understand the user's intention and road conditions, and responds (ie, utterance) to perform an action corresponding to the user's intention. print out

Here, the user's intention is the user's utterance intention, and may be a specific response to a request, a question, a yes/no response, or a query of a conversation system.

In more detail, the dialog system 1100 recognizes the user's voice input to the voice input device 1210, checks whether a word corresponding to the notification situation exists among the recognized voices, and if there is a word corresponding to the notification situation, It is determined that a notification situation has occurred, and a query is performed to check whether the confirmed word is a word corresponding to the notification situation.

When a notification situation occurs, the conversation system 1100 performs a conversation with the user to obtain situation information about the notification situation.

That is, when a notification situation occurs, the dialog system 1100 may repeat the process of recognizing the voice input to the voice input device 1210 and performing a query corresponding to the recognized voice.

For example, when a notification situation occurs, the dialog system 1100 inquires whether a traffic accident situation or an illegal activity situation, and when a voice in response to the inquiry is recognized, the user's response is confirmed based on the recognized voice, , query for additional information about the alert status that has occurred.

As another example, the dialog system 1100 determines a notification situation generated based on the user's voice, and performs a query to confirm whether the determined notification situation is a traffic accident situation or an illegal action situation. .

As another example, the dialogue system 1100 may output an utterance for confirming the vehicle number of the recognized vehicle to the user when a traffic accident or illegal act occurs, and a request for vehicle number input when the vehicle number is not recognized It is also possible to output utterances.

As another example, the dialog system 1100 may check a responsible agency corresponding to the notification situation and inquire whether the confirmed agency is correct.

As another example, the dialogue system 1100 may inquire about what obstacles on the road are when an attention situation occurs, and it is also possible to inquire about the lanes of the road where the obstacles are located.

It is also possible for the dialog system 1100 to transmit the occurrence information of the notification situation to the controller 1240 so as to obtain the situation information about the notification situation by using the controller 1240 .

In addition, it is also possible to determine the notification situation in the conversation system 1100 and read image information at the time the notification situation occurs from the controller 1240 .

The dialog system 1100 may also receive image information, road location information, date information, and time information at the time the notification situation occurs, from the controller 1240 .

The dialogue system 1100 is also capable of receiving the vehicle number of the vehicle of a traffic accident or illegal activity from the controller 1240, and transmitting the vehicle number recognized by the user's voice to the controller 1240 together with context information It is possible, and it is also possible to check the authority in charge corresponding to the situation information and transmit the information to the server of the confirmed authority in charge.

As described above, the dialog system 1100 may perform input information processing, dialog processing based on a processing result of input information, and result processing based on a dialog processing result.

The conversation system 1100 may obtain information or content necessary for processing input information, conversation management, or result processing from a conversation server (not shown).

45 , the conversation system 1100 uses an input processor 1110 that processes a user's voice and a processing result of the input processor 1110 to determine the user's intention, and the user's intention (eg, , reporting or providing information) or a result of outputting utterances to provide a specific service or continue a conversation according to the output result of the dialog manager 1120 and the dialog manager 1120 for determining an action corresponding to the road situation The processor 1130 and the storage unit 1140 for storing various types of information necessary for the operation of the dialog system 1100 are included.

The input processor 1110 may receive a user's voice.

Here, the user may include all passengers, such as a driver and a passenger.

The input processor 1110 recognizes the input of the user's voice, converts it into a text-type utterance, and applies a natural language understanding technology to the user's utterance to understand the user's intention.

In addition, the input processor 1110 transmits information related to the user's intention and situation identified through natural language understanding to the conversation manager 1120 .

The voice input processor 1110 includes a voice recognizer 1111 that recognizes the input user's voice and outputs it as a text-type utterance, and a user who applies natural language understanding to the utterance to contain the utterance. It may include a natural language interpreter 1112 for understanding the intention of , and a dialog input manager 1113 for transmitting the natural language understanding result and context information to the dialog manager 1120 .

A detailed configuration of the voice input processor 1110 is the same as that of FIG. 18 , so a description thereof will be omitted.

The conversation manager 1120 determines an action corresponding to the user's intention and the current situation based on the user's intention and context information transmitted from the input processor 1110 , and manages factors necessary to perform the corresponding action.

Here, the action may include utterance for information confirmation and query, and information transmission to the server.

The conversation manager 1120 transmits information about the determined action to the result processor 1130 .

The result processor 1130 generates and outputs a dialog response and a command necessary to perform the delivered action. The dialogue response may be output as text, image, or audio, and when the command is output, driving control of the vehicle corresponding to the output command, information transmission operation to the controller, etc. may be performed.

In addition, the result processor 1130 may directly transmit information to the server 1300 .

The storage unit 1140 stores various types of information required for conversation processing and service provision. For example, information related to a domain, action, dialogue act, and entity name used for understanding natural language may be stored in advance, a situation understanding table used to understand a situation from input information may be stored, and a sensor provided in the vehicle may detect One data, information related to the user, and information necessary to perform an action may be stored in advance.

In this way, the dialog system 1100 may recognize the user's actual intention and preemptively provide information corresponding thereto by considering the content that the user has not uttered using the previously acquired information. Through this, it is possible to shorten the conversation step and time until the user provides the desired service.

As described above, the dialogue system 1100 provides a dialogue processing technology specialized for the environment of the vehicle.

All of the components of the conversation system 1100 may be included in the vehicle, or only some of the components may be included. The chat system 1100 may be provided in a chat server located remotely, and the vehicle may only serve as a gateway between the chat system 1100 and the user.

In either case, the conversation system 1100 may be connected to the user through a vehicle or a mobile device connected to the vehicle.

46 is a configuration diagram of a vehicle according to another embodiment, and FIG. 47 is a detailed configuration diagram of the in-vehicle conversation system shown in FIG. 46 .

As shown in FIG. 46 , the vehicle of another embodiment, like the vehicle of the other embodiment shown in FIG. 44 , a conversation system 1100a, a voice input device 1210 , an image input device 1220 , and an output device 1230 . , a controller 1240a and a communication device 1280a may be included.

The vehicle voice input device 1210 , the image input device 1220 , and the output device 1230 according to another embodiment include the vehicle voice input device 1210 , the image input device 1220 and the As it is the same as that of the output device 1230 , a description thereof will be omitted.

The communication device 1280a may include a GPS receiver for recognizing the current location of the vehicle.

The communication device 1280a may include one or more communication modules that enable communication with an external device, and may include, for example, a short-range communication module, a wired communication module, and a wireless communication module.

The communication device 1280a may further include an internal communication module (not shown) for communication between electronic devices inside the vehicle 1200a. As an internal communication protocol of the vehicle 200 , a controller area network (CAN), a local interconnection network (LIN), FlexRay, Ethernet, or the like may be used.

The communication device 1280a may perform Vehicle to Everything (V2X).

The controller 1240a transmits input information transmitted from the voice input device 1210 and the image input device 1220) to the conversation system 1100 in the notification mode.

The controller 1240a may store the image information of the image-processed road together with at least one of date information, time information, and location information of the photographed road when storing the image information, and the stored image information and location according to the request of the conversation system. Sends information, date information, and time information to the chat system.

The controller 1240a controls the vehicle 1200 to perform an action corresponding to a user's intention or a current situation according to output information (eg, a response) output from the dialog system 1100 .

When a notification situation occurs, the controller 1240a may automatically perform deceleration control to acquire an image of a precise road, and it is also possible to automatically perform steering control.

When a notification situation occurs, the controller 1240a may output deceleration guidance information and steering guidance information through at least one of the display 1231 and the speaker 1232 to obtain a precise image of the road.

The controller 1240a may receive image information of the surrounding road received from another vehicle or server and output the image of the surrounding road through a display.

It is also possible for the controller 1240a to receive the image information of the surrounding road received from another vehicle or server and output the condition of the surrounding road corresponding to the received image information through the speaker 1232 or the display 1231 .

The conversation system 1100a receives the user's voice inputted to the voice input device 1210 and the image inputted to the image input device 1220 , and stores the received image.

The conversation system 1100a may store information about the time when a voice is input, information on the date and time the image was captured, and information about the location at which the image was captured.

Here, the location information at which the image was captured may be provided from the controller 1240a.

Information on time at which a voice is input and information on a date and time at which an image is captured may be provided from a timer (not shown) in the conversation system.

In addition, it is also possible to receive information on the time at which the voice is input, and the date and time information at which the image was captured, from the controller 1240a.

The dialog system 1100a recognizes the user's voice input to the voice input device 1210, checks whether a word corresponding to the notification situation exists among the recognized voices, and when a word corresponding to the notification situation exists, a notification situation occurs It is determined that it has been done, and a query is performed to check whether the confirmed word is a word corresponding to the notification situation.

When a notification situation occurs, the conversation system 1100a performs a conversation with the user to obtain situation information about the notification situation.

That is, when a notification situation occurs, the conversation system 1100a may repeat the process of recognizing the voice input to the voice input device 1210 and performing a query corresponding to the recognized voice.

The conversation system 1100a may transmit information on the occurrence of a notification situation to the controller 1240a.

When a notification situation occurs, the dialog system 1100a checks an institution in charge corresponding to the context information, and transmits the context information and the stored image to the server 1300 to the server 1300 of the confirmed institution in charge.

Here, the situation information includes a traffic accident situation, an illegal activity situation, and may further include a state situation.

In more detail, when transmitting an image to the server 1300 , the conversation system 1100a checks a time point at which a voice corresponding to a notification situation is input to the voice input device 1210 , and transfers the confirmed time point before the first predetermined time Check the time point and the time point after the second predetermined time has elapsed from the confirmed time point, acquire the image taken between the previous time point and the later time point among the stored images, and transmit the obtained image and situation information to the server 1300 .

Here, the first predetermined time and the second predetermined time may be the same as or different from each other.

The first predetermined time and the second predetermined time may be 60 seconds.

The conversation system 1100a may recognize the vehicle number of the vehicle of the traffic accident or illegal act based on the obtained image information, and further transmit the vehicle number to the server 1300 of the verified agency.

When transmitting an image, the conversation system 1100a confirms the date and time that the acquired image was captured, confirms the location of the road in the acquired image, and transmits the date, time, and location together to the server 1300 .

The dialogue system 1100a controls the output device 1150 to output the processing result through the output device 1150 of the dialogue system so that the processing result is output as at least one of an image and an audio.

The dialogue system 1100a may also recognize the situation of the road based on the obtained image information and situation information.

The dialogue system 1100a is also capable of recognizing the vehicle number of a vehicle in a traffic accident or illegal activity based on the obtained image information, and it is also possible to transmit the recognized vehicle number to the controller 1240 together with the situation information, It is also possible to check the authority in charge corresponding to the context information and transmit the information to the server of the confirmed authority in charge.

The dialog system 1100a may recognize the illegal activity based on the location of the intersection in the acquired image, the color of traffic lights, the position of the lane, the position of another vehicle in the acquired image, and the speed of the other vehicle.

As shown in FIG. 47 , the dialogue system 1100a includes an audio input processor 1110a that processes a user's voice, an image input processor 1110b that processes an image of a road, and an audio input processor 1110a. A dialog manager 1120a that identifies the user's intention using the processing result and determines an action corresponding to the user's intention (eg, report or information provision) or road conditions, and stores the processed image and input the image The image manager 1120b that determines the condition of the road by using the processing result of the processor 1110b, and the video manager 1120a and the video manager 1120b to provide a specific service or to continue the conversation according to the output results of the It includes a result processor 1130a for outputting utterances and a storage unit 1140a for storing various types of information required for the operation of the dialogue system 1100a.

A detailed configuration of the voice input processor 1110a is the same as that of FIG. 18 , so a description thereof will be omitted.

The conversation system 1100a may further include an output device 1150 and a communication device 1160 .

The output device 1150 may include a display 1151 and a speaker 1152 provided in the conversation system 1100 .

The display 1151 and the speaker 1152 may visually or aurally output a response to a user's utterance, a query to the user, or information requested by the user.

The speaker 1152 outputs audio for guiding a navigation route, audio of content (including sound and voice), a voice for providing information or service desired by a user, a voice of a dialogue system generated as a response to a user's utterance, etc. can do.

In the notification mode, the speaker 1152 may preferentially output the voice of the dialogue system corresponding to the situation information over the audio of the speaker 1232 of the vehicle.

The output device 1150 of the dialogue system 1100a may be omitted.

The communication device 1160 communicates with the server 1300 and transmits the collected situation information, image, time, date, and location information to the server 1300 .

The communication device 1160 may include one or more communication modules that enable communication with an external device, and may include, for example, a short-range communication module, a wired communication module, and a wireless communication module.

48 is a flowchart of a method for controlling a conversation system of a vehicle according to another exemplary embodiment. This will be described with reference to FIGS. 49 to 53 .

49 to 53 are diagrams illustrating conversations between a vehicle conversation system and a user according to another embodiment.

When the operation of the voice input device and the image input device is turned on, the conversation system determines whether the user's voice is inputted while photographing an image around the vehicle using the image input device and storing the captured image.

Here, the state in which the operation of the voice input device and the image input device are turned on may be one of a state in which a notification mode is turned on, a state in which the vehicle is started, and a state in which the charge amount of the vehicle's battery is equal to or greater than the reference charge amount.

The captured image may be stored in the vehicle's controller.

When it is determined that the user's voice is input, the conversation system recognizes the input voice ( 1171 ).

The dialog system outputs the recognized voice as a text-type utterance, and applies natural language understanding technology to determine the user's intention and the road situation contained in the utterance.

When it is determined ( 1172 ) that the input voice is a notification voice for notifying the road condition, the conversation system outputs a situation query to inquire whether the user is uttered about the notification situation ( 1173 ).

When a voice for a situational query is input, the dialog system recognizes the input response voice ( 1174 ).

That is, when a voice is input within a preset time from the time the situation query is output, the dialogue system may determine that it is a voice response to the situation inquiry.

The dialog system outputs the recognized response voice as a utterance in the form of text, and applies natural language understanding technology to determine the user's intention and notification situation contained in the utterance.

The dialogue system determines whether the determined notification situation is an accident notification voice corresponding to a traffic accident (1175), and when it is determined that the determined notification situation is an accident notification voice corresponding to a traffic accident, acquires (1176) an image of the road where the traffic accident occurs.

Here, acquiring the image of the road on which the traffic accident occurred may include acquiring an image at the point in time when the user witnessed the traffic accident among images stored in a vehicle controller or images stored in a conversation system.

That is, acquiring an image of a road is editing a stored image, and includes acquiring an image based on a time point at which a notification voice is input.

More specifically, the dialogue system checks the time point at which the notification voice is input and confirms the time point before the first predetermined time as the confirmed time point and the time point after the second predetermined time has elapsed from the confirmed time point, and the entire video Only images captured between the time before and after the time when the notification voice is input are acquired.

The dialogue system confirms the location information of the road in the acquired image, and confirms the time and date when the acquired image was taken.

Next, the dialog system recognizes (1177) the license plate number of the vehicle in the traffic accident in the acquired image, and determines whether the vehicle number recognition is successful (1178).

When it is determined that the vehicle number recognition is successful, the dialogue system outputs a confirmation query for confirming the vehicle number of the vehicle recognized in the traffic accident to the user.

When the voice response to the confirmation query is input, the conversation system recognizes the input response voice.

The dialog system outputs the recognized response voice as a utterance in text form, and applies Natural Language Understanding technology to determine whether the vehicle number contained in the utterance matches or not.

That is, the dialogue system determines whether the vehicle number of the actual traffic accident vehicle matches the vehicle number recognized through image processing from the input response voice.

When it is determined that the vehicle number matches based on the recognized response voice, the dialogue system checks the authority in charge of handling traffic accidents, and transmits the information of the video along with the situation information of the traffic accident to the server of the confirmed authority in charge.

The conversation system collects ( 1179 ) information on the date, time, and location of the image when transmitting the information of the image, and transmits the collected information to the server ( 1180 ).

As shown in FIG. 49 , when it is determined that a notification voice of 'Report' is input by the user, the conversation system inquires about the situation the user wants to notify.

In other words, the dialogue system is 'Is it a car accident? Is it illegal?' If a query is output and it is determined that an accident notification voice of 'traffic accident' is input by the user, an image at the time the voice 'report' is input is obtained and the vehicle number of the accident vehicle in the acquired image is recognized.

The dialog system outputs the query 'Is the vehicle number oooo?' to check whether the recognized vehicle number is correct.

When it is determined that the voice of 'Yes' is input in response to the inquiry, the dialogue system checks the agency in charge of the traffic accident, situation information (i.e., traffic accident), vehicle number, obtained road video, and traffic accident information. The information on the date, time, and location of the video is transmitted to the server of the confirmed agency.

Here, the responsible authority may be the National Police Agency (112) and the Safety Report Center (119).

When the information transmission is completed, the dialog system outputs a report guide utterance, 'Traffic accident report has been completed.'

In addition, when it is determined that the dialogue system is in a traffic accident situation, it is possible to output an injured person query as to whether there are any injuries. Accordingly, the dialogue system can control the transmission of information to the server of the safety report center 119 according to the presence of an injured person.

If the dialog system fails to recognize the vehicle number, it queries the user for the vehicle number of the traffic accident vehicle.

When the voice response to the vehicle number query is input, the dialog system recognizes the input response voice (1181).

The dialogue system outputs the recognized response voice as a utterance in text form, and applies Natural Language Understanding technology to confirm the vehicle number contained in the utterance.

When the vehicle number is confirmed, the dialogue system checks the agency in charge of handling the accident and transmits it to the server of the confirmed agency in charge (1180).

That is, the dialogue system transmits the information of the image along with the situation information of the traffic accident to the server.

The conversation system collects ( 1179 ) information on the date, time, and location of the image when transmitting the information of the image, and transmits the collected information to the server ( 1180 ).

As shown in FIG. 50 , when it is determined that a notification voice of 'Report' is input by the user, the conversation system inquires about the situation the user wants to notify.

In other words, the dialogue system is 'Is it a car accident? Is it illegal?' If a query is output and it is determined that an accident notification voice of 'traffic accident' is input by the user, an image at the time the voice 'report' is input is obtained and the vehicle number of the accident vehicle in the acquired image is recognized.

The dialog system outputs the query 'Is the vehicle number oooo?' to check whether the recognized vehicle number is correct.

When it is determined that a voice of 'no' is input in response to the query, the conversation system queries the user for the vehicle number of the traffic accident vehicle.

That is, the dialogue system outputs a query 'Call the vehicle number'.

When it is determined that the voice corresponding to the vehicle number is input by the user, the dialog system determines the vehicle number of the traffic accident based on the input voice, identifies the authority responsible for the traffic accident, and provides context information (ie, traffic accident ), the vehicle number, the acquired image of the road, and the date, time, and location of the traffic accident image are transmitted to the server of the confirmed agency.

When the information transmission is completed, the dialog system outputs a report guide utterance, 'Traffic accident report has been completed.'

As shown in FIG. 51 , when it is determined that a notification voice of 'Report' is input by the user, the conversation system inquires about the situation the user wants to notify.

In other words, the dialogue system is 'Is it a car accident? Is it illegal?' If a query is output and it is determined that an accident notification voice of 'traffic accident' is input by the user, an image at the time the voice 'report' is input is obtained and the vehicle number of the accident vehicle in the acquired image is recognized.

If it is determined that the vehicle number has not been recognized as a result of image processing, the conversation system queries the user for the vehicle number of the traffic accident vehicle.

That is, the dialogue system outputs a query 'Call the vehicle number'.

When it is determined that the voice corresponding to the vehicle number is input by the user, the dialogue system determines the vehicle number of the traffic accident based on the input voice.

It is also possible that the dialogue system outputs a confirmation query for confirming whether the determined vehicle number is correct.

The dialogue system identifies the authority responsible for the traffic accident, and confirms the situation information (that is, the traffic accident), the vehicle number, the image of the obtained road, and the information of the date, time and location where the video of the traffic accident was taken. sent to the organization's server.

When the information transmission is completed, the dialog system outputs a report guide utterance, 'Traffic accident report has been completed.'

If it is determined that the notification voice is not the voice of a traffic accident, the dialogue system determines ( 1182 ) as a voice corresponding to the illegal action situation, and acquires ( 1183 ) an image corresponding to the illegal action situation from among the stored images of the road.

Here, the stored image of the road may include at least one of an image stored in a vehicle controller or an image stored in a conversation system.

Acquiring the image of the road includes editing only the stored image at the point in time when the notification voice is input among the stored images.

More specifically, the dialog system checks the time point at which the notification voice is input and confirms the time point before the first predetermined time as the confirmed time point and the time point after the second predetermined time has elapsed from the confirmed time point, the notification voice An image captured between a time before and a time after the input time is acquired.

The dialogue system identifies the agency in charge of handling illegal acts, and transmits the acquired image and situation information to the server of the verified agency.

Here, the responsible agency may be the National Police Agency.

The dialogue system confirms the location information of the road in the acquired image, confirms the time and date that the acquired image was taken, collects (1183) the confirmed time, date, location information and information on the image, and collects the collected information. It transmits (1180) to the server.

In addition, the following dialogue system can recognize the license plate number of the illegal vehicle, and it is also possible to query the user for the license plate number of the illegal vehicle.

As shown in FIG. 52 , when it is determined that a notification voice of 'Report' is input by the user, the conversation system inquires about the situation the user wants to notify.

In other words, the dialogue system is 'Is it a car accident?' It outputs a query, and if it is determined that an accident notification voice of 'illegal act' along with 'no response' is inputted by the user in response to the query, what kind of illegal act is inquired.

In addition, it is possible to directly receive the notification voice that notifies the situation, such as a traffic accident, tail biting, and driving on the shoulder, rather than a comprehensive 'report' voice.

The dialogue system asks, 'Is it tail biting?', and when it is determined that a notification voice for illegal activity such as 'drive on the shoulder' is input along with 'No answer' in response to the question, the time when the voice 'Report' is input Acquires an image of , recognizes the vehicle number of the illegal act based on the obtained image, and performs a confirmation query to confirm to the user whether the recognized vehicle number is correct.

That is, the conversation system asks 'Is the vehicle number oooo correct?', and if it is determined that a voice of 'no' is input in response to the query, it queries the user for the vehicle number of the illegal act.

The dialogue system outputs a query 'Call the vehicle number', and when the voice of the vehicle number of an illegal act is input in response to the query, the vehicle number of the traffic accident is determined based on the input voice.

It is also possible that the dialogue system outputs a confirmation query for confirming whether the determined vehicle number is correct.

The dialogue system identifies the authority responsible for the illegal activity, and provides contextual information (ie, illegal shoulder driving), license plate number, image of the road containing the image of the vehicle in the illegal activity, date, time, and date the image was taken. The location information is transmitted to the server of the verified agency.

When the information transmission is completed, the dialog system outputs a report guide utterance, 'Traffic accident report has been completed.'

In addition, if the dialog system determines that a notification voice of an illegal activity has been input, it is possible to skip the recognition process of the vehicle number of the illegal activity, and to directly query the user for the vehicle number of the vehicle in a traffic accident to receive the vehicle number of the illegal vehicle from the user do.

The dialogue system is also capable of transmitting information about the state of the road to the server of the Ministry of Land, Infrastructure and Transport and the Traffic Information Center.

The dialogue system acquires an image at the point in time when the notification voice is input when a notification voice for the caution condition of the road is input, determines the caution situation based on the acquired image, and transmits information on the determined caution situation to the server It is possible.

For example, as shown in FIG. 53 , when a voice 'stone' is input by the user, the conversation system determines that it is an attention situation, and performs a confirmation query to confirm the situation.

The dialogue system performs a query of 'Is there a stone on the road?', and when a voice 'uh' is inputted by the user as a response voice to the query, the location of the stone on the road is queried.

The dialogue system asks 'how many lanes?' and when '3 lanes' is input as a response to the query, 'I will provide information on falling rocks on the 3rd lane.'

The dialogue system identifies the agency in charge of handling rockfalls, and transmits situational information (ie, warning of rockfalls), the video of the road including the three-lane road, and the date, time and location of the video to the server of the verified agency. send.

In addition, the dialogue system may provide situational information about the attention situation to surrounding vehicles.

It is also possible that the server provides situation information about the attention situation to surrounding vehicles based on the location information of the road.

10 is a control block diagram for a vehicle-only method in which a conversation system is provided in a vehicle.

In the vehicle-only method, as shown in FIGS. 44 and 46 , the conversation systems 1100 and 1100a may be included in the vehicles 1200 and 1200a.

When the dialog systems 1100 and 1100a are included in the vehicles 1200 and 1200a, the vehicles 1200 and 1200a may themselves process a conversation with the user and provide a necessary service to the user. However, information necessary for conversation processing and service provision may be obtained from an external content server.

The conversation system may be provided in a mobile device, which may communicate with the vehicle. Accordingly, it is also possible to perform the notification mode in the vehicle using the chat system in the mobile device. This will be described with reference to FIG. 54 .

54 is a configuration diagram of a vehicle and a mobile device communicating with the vehicle according to another embodiment.

The vehicle 1200b includes an image input device 1220b, a controller 1240b, and a first communication device 1280b, and the mobile device 1400 includes an audio input device 1410, a conversation system 1420, and an output device ( 1430 ) and a second communication device 1440 .

The image input device 1220b of the vehicle 1200b photographs the road.

The controller 1240b stores the captured image, acquires an image at the time a notification voice is input according to a request of the mobile device, and controls transmission of the acquired image.

The controller 1240b may recognize the vehicle number of the vehicle in the image obtained according to the request of the mobile device, and it is also possible to transmit the recognized vehicle number to the mobile device.

When the captured image is stored, the controller 1240b stores information of the location, date, and time of the captured road together.

The first communication device 1280b communicates with the mobile device and transmits information to the mobile device based on a command from the controller 1240b.

The first communication device 1280b includes a GPS receiver, and transmits location information to the controller 1240b when capturing an image.

The mobile device 1400 may be an electronic device that is portable, such as a smart phone, a smart watch, smart glasses, a PDA, or a tablet PC, and can communicate with an external server and vehicle to exchange data.

The mobile device 1400 includes a voice input device 1410 that receives a user's voice, a vehicle 1200b, and a conversation system that collects input data from the user and transmits it to the server 1300 through the second communication device 1480 ( 1420), an output device 1430 that outputs a response visually, auditory or tactile, and a second communication device 1440 that communicates with the server 1300 and the vehicle 1200b to transmit and receive data.

The voice input device 1410 may include a microphone that receives sound, converts it into an electrical signal, and outputs it.

The dialogue system 1420 has the same configuration as the dialogue system shown in FIGS. 46 and 47 and will be briefly described.

The conversation system 1420 receives the user's voice input to the voice input device 1410 , receives an image of the image input device transmitted from the controller 1240b , and stores the received image.

The conversation system 1420 may store information about the time at which a voice is input, information on the date and time the image was captured, and information about the location at which the image was captured.

Accordingly, when a voice for a notification situation is input, it is possible to transmit the captured image information together with the conversation information about the notification situation.

Here, the location information at which the image was captured may be provided from the controller 1240b.

Information on time at which a voice is input and information on a date and time at which an image is captured may be provided from a timer (not shown) in the mobile device.

In addition, it is also possible to receive information on the time at which the voice is input and the date and time information at which the image was captured from the controller 1240b.

The dialog system 1420 recognizes the user's voice input to the voice input device 1410, checks whether a word corresponding to the notification situation exists among the recognized voices, and when a word corresponding to the notification situation exists, a notification situation occurs It is determined that it has been done, and a query is performed to check whether the confirmed word is a word corresponding to the notification situation.

When a notification situation occurs, the dialog system 1420 performs a conversation with the user to obtain context information about the notification situation.

That is, when a notification situation occurs, the conversation system 1420 may repeat the process of recognizing the voice input to the voice input device 1210 and performing a query corresponding to the recognized voice.

The dialog system 1420 may also transmit information on the occurrence of a notification situation to the controller 1240b.

When a notification situation occurs, the dialog system 1420 checks an institution in charge corresponding to the context information, and transmits the context information and the stored image to the server 1300 to the server 1300 of the confirmed institution in charge.

Here, the situation information includes a traffic accident situation, an illegal activity situation, and may further include a state situation.

When a notification situation occurs, the dialog system 1420 may request the vehicle controller to provide image information captured at the time the notification situation occurred. At this time, the controller of the vehicle may acquire image information from a time before the time the notification situation occurs to a time after the time when the notification situation occurs based on the time when the notification situation occurs, and provide the obtained image information to the conversation system.

The output device 1430 may include a display 1431 and a speaker 1432 provided in the conversation system 1420 .

The display 1431 and the speaker 1432 may visually or aurally output a response to the user's utterance, a query to the user, or information requested by the user.

The speaker 1432 may output a voice for providing information or service desired by a user, a voice of a dialogue system generated as a response to the user's utterance, and the like.

In the notification mode, the speaker 1432 may preferentially output the voice of the dialog system corresponding to the context information rather than the audio of the vehicle speaker.

As an input/output interface for a user, not only the voice input device 410 and the output device 430 provided in the mobile device 1400 may be used, but also the voice input device and the output device provided in the vehicle 1200b may be used.

The communication device 1440 communicates with the vehicle 1200b and the server 1300 , receives an image from the vehicle, and transmits situation information of the conversation system, image, time, date, and location information to the server 1300 . do.

The communication device 1440 may include one or more communication modules that enable communication with an external device, and may include, for example, a short-range communication module, a wired communication module, and a wireless communication module.

The above description is merely illustrative of the technical idea, and various modifications, changes and substitutions will be possible within the range that does not deviate from the essential characteristics by those of ordinary skill in the art of the present invention. Accordingly, the embodiments disclosed above and the accompanying drawings are for explanation rather than limiting the technical idea, and the scope of the technical idea is not limited by these embodiments and the accompanying drawings. The scope of protection should be construed by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of rights.

According to the dialog system described so far, the vehicle including the same, and the dialog processing method, it is possible to provide an optimal service that meets the user's intention or needs by using the dialog processing technology specialized for the vehicle environment.

In addition, it is possible to provide an optimal service required to the user in consideration of various situations occurring inside the vehicle. In particular, it is possible to self-determine a service required for a user based on the situation information or driver information collected by the conversation system regardless of the user's utterance, and to preemptively provide it.

1100, 1100a: dialog system 1110: input handler
1120: conversation manager 1130: result handler
1200, 1200a, 1200b: vehicle
1210: audio input device 1220: video input device
1230: dialog output device 1280: communication device
1300: server 1400: mobile device

Claims (24)

a voice input device for receiving a user's voice;
Converts the received user's voice into text, recognizes the user's intention and a notification situation for another vehicle using the converted text and natural language understanding technology, and recognizes the recognized user's intention and notification situation for the other vehicle a dialogue system for controlling conversation with the user based on
an output device for outputting utterances for conversation with the user; and
and a communication device for transmitting situation information about the notification situation for the other vehicle,
The notification situation for the other vehicle includes at least one of a traffic accident situation of the other vehicle and an illegal action situation,
The dialog system determines whether the notification situation for the other vehicle is a traffic accident situation of the other vehicle or an illegal action situation of the other vehicle, and identifies a responsible authority corresponding to the determined situation, A vehicle that controls the communication device to transmit information on the situation to a server of a responsible agency. The method of claim 1,
an image input device for photographing a road and a periphery of the road; and
and a controller configured to receive and store the image captured by the image input device, and control the communication device to transmit situation information of the notification situation and the stored image. delete 3. The method of claim 2, wherein the dialogue system comprises:
Determining whether a notification situation for the other vehicle has occurred based on the voice, and when it is determined that the notification situation has occurred, recognizing the vehicle number of another vehicle in the photographed image and controlling transmission of the recognized vehicle number vehicle that does. 5. The method of claim 4, wherein the dialog system comprises:
If the recognition of the vehicle number of the other vehicle is successful, the vehicle comprising performing a confirmation query to confirm the matching of the recognized vehicle number. 5. The method of claim 4, wherein the dialog system comprises:
If the recognition of the vehicle number of the other vehicle fails, the vehicle comprising performing a query for the vehicle number of the other vehicle. According to claim 1, wherein the dialogue system,
It is determined based on the voice whether a notification situation for the other vehicle has occurred, and when it is determined that the notification situation has occurred, a query is performed for the vehicle number of the other vehicle, and based on the voice received from the voice input device, the A vehicle comprising recognizing the license plate number of another vehicle. 3. The method of claim 2,
The communication device includes a GPS receiver,
The dialog system includes matching and storing the location information received in the GPS receiver when storing the video, and controlling transmission of the matched video and location information when controlling transmission of the video. 3. The method of claim 2, wherein the dialogue system comprises:
A vehicle comprising matching and storing date and time information when storing the image, and controlling transmission of the matched image and date and time information when controlling transmission of the image. 3. The method of claim 2, wherein the dialogue system comprises:
Check the time point at which the notification voice corresponding to the notification situation for the other vehicle is input, and check the time point before the first predetermined time and after the second predetermined time has elapsed from the confirmed time point and controlling to acquire only the images captured between the time points before and after the point in time when the notification voice is input from among the stored images, and to transmit only the obtained images. a voice input device for receiving a user's voice;
an image input device for photographing a road and a periphery of the road and outputting the photographed image;
a controller that stores the captured image and controls output of the received audio and the stored image;
It communicates with the controller, converts the user's voice into text, uses the converted text and natural language understanding technology to determine whether it is a notification situation for another vehicle, and talks with the user if it is a notification situation for the other vehicle a dialog system for recognizing context information of a notification situation for the other vehicle by performing , and transmitting the recognized context information to the controller;
an output device for outputting an utterance for a conversation with the user based on a command of the conversation system; and
and a communication device for transmitting situation information of the notification situation and the stored image based on a command from the controller,
The notification situation includes at least one of a traffic accident situation of another vehicle and an illegal action situation,
The dialogue system determines whether the notification situation for the other vehicle is a traffic accident situation of the other vehicle or an illegal act of the other vehicle, and identifies a responsible agency corresponding to the determined situation, A vehicle that controls the communication device to transmit information and an image on the determined situation to a server. delete 12. The method of claim 11, wherein the controller,
Recognizing the vehicle number of the other vehicle and controlling transmission of the recognized vehicle number of the other vehicle. 12. The method of claim 11,
The communication device includes a GPS receiver,
wherein the controller matches and stores the location information received in the GPS receiver when the image is stored, and controls transmission of the matched image and location information when controlling the transmission of the image. 12. The method of claim 11, wherein the controller,
A vehicle comprising matching and storing date and time information when storing the image, and controlling transmission of the matched image and date and time information when controlling transmission of the image. 12. The method of claim 11, wherein the controller,
A time point at which a notification voice corresponding to the notification situation is input is confirmed, and a time point before a first predetermined time and a time point after the lapse of a second predetermined time from the confirmed time point are checked, and the stored A vehicle comprising: acquiring only an image captured between a time point before and after a time point at which the notification voice is input, and controlling to transmit only the acquired image. a communication device for communicating with the vehicle and the server;
a voice input device for receiving a user's voice;
The user's voice is converted into text, the converted text and natural language understanding technology are used to determine whether it is a notification situation for another vehicle, and if it is determined that it is a notification situation for the other vehicle, a conversation with the user is performed and during the conversation a dialogue system for recognizing context information of a notification situation for the other vehicle based on the received voice, requesting the vehicle to provide an image, and controlling to transmit the recognized situation information and the image transmitted from the vehicle to a server; and
and an output device for outputting an utterance for a conversation with the user based on a command of the conversation system,
The communication device transmits situation information and the image of a notification situation for the other vehicle based on a command of the dialogue system,
The notification situation includes at least one of a traffic accident situation with respect to the other vehicle and a situation of illegal activity. 18. The method of claim 17, wherein the dialogue system comprises:
Recognizing the vehicle number of the other vehicle in the image and controlling transmission of the recognized vehicle number. 18. The method of claim 17, wherein the dialog system comprises:
Mobile device comprising receiving at least one of location information, time information, and date information of the vehicle when the image is received from the vehicle. Taking pictures of the road and the surroundings of the road,
storing the captured image along with time information and location information;
When the user's voice is input through the voice input device, the user's voice is converted into text,
Using the converted text and natural language understanding technology to determine whether it is a notification situation for another vehicle,
If it is determined that the notification condition for the other vehicle is, output an utterance corresponding to the notification condition for the other vehicle through an output device,
obtaining situation information of a notification situation for the other vehicle based on the voice input to the voice input device;
Transmitting the obtained situation information, the location information, the time information and the stored image,
The notification situation for the other vehicle includes at least one of a traffic accident situation and an illegal action situation of the other vehicle. delete The method of claim 20, wherein transmitting the obtained situation information and the stored image comprises:
Identify the agency in charge of responding to the determined situation,
and transmitting the situation information and the image to the identified server of the agency in charge. The method of claim 20, wherein obtaining the context information comprises:
and recognizing and acquiring the vehicle number of another vehicle in the image when it is determined that the notification situation has occurred. The method of claim 20, wherein transmitting the stored image comprises:
Check when the notification voice corresponding to the notification situation is input,
As the confirmed time point, a time before the first predetermined time and a time point after the lapse of a second predetermined time from the confirmed time point are confirmed,
Obtaining only the images taken between the time before and after the time point at which the notification voice is input among the stored images,
A control method of a vehicle comprising controlling transmission of the acquired image.

Images