JP7394674B2 - Display control device, display control method, and program - Google Patents

Description

The present invention relates to a display control device, a display control method, and a program.

Conventionally, technologies have been disclosed that display characters on a vehicle's display device and provide services such as driving support information, vehicle control, and other applications according to the user's requests while interacting with the user (e.g. , see Patent Document 1).

JP2006-335231A

In recent years, with the expansion of car sharing services, various users may use vehicles, but it is necessary to display characters that interact with each user in a more appropriate output format and provide various information. was not considered.

The present invention has been made in consideration of such circumstances, and one of the objects is to provide a display control device, a display control method, and a program that can provide information in a more appropriate output format. do.

A display control device, a display control method, and a program according to the present invention employ the following configuration.
(1): A display control device according to one aspect of the present invention is a display control device that displays an image of a character on an output device, and displays an accompanying image associated with the character on the output device together with the image of the character. an image setting unit that accepts settings to be displayed; and an information setting unit that accepts settings for incidental information to be output from the output device in association with displaying the accompanying image on the output device together with the image of the character. This is a display control device.

(2): In the aspect of (1) above, the information setting section enables setting of the incidental information for each user.

(3): In the aspect of (1) or (2) above, the accompanying image includes one or both of an image of an accessory equipped by the character and an image of another character different from the character.

(4): The aspect of (3) above, including a recognition unit that recognizes a user who uses the output device, and a setting to display the ornament in the incidental information for the user recognized by the recognition unit. In this case, the output device further includes an image control unit that causes the output device to display an image of the character equipped with the accessory.

(5): In the aspect of (4) above, the information setting unit accepts the setting of the incidental information when the user recognized by the recognition unit has authority to display the incidental image. be.

(6): In the aspect of (4) or (5) above, when the user recognized by the recognition unit has the authority to display the accompanying image, output that causes the output device to output the accompanying information. The apparatus further includes a control section.

(7): In any one of the above (1) to (6), the output device is a device that can be mounted on a vehicle, and the information setting unit is configured to set the incidental information on the vehicle. This allows settings to be made in association with the state.

(8): In the aspect of (6) above, the output control unit displays an image of the character equipped with the accessory on the output device, and displays the vehicle mounted on the output device at the time of the accessory. When the information is in a state associated with the information setting, the output device is caused to output the incidental information.

(9): In the aspect of (4) above, the information setting section accepts the setting of information regarding safe driving by the user as the incidental information when the ornaments include an amulet.

(10): In the aspect of (9) above, the image control unit is configured such that the recognition unit recognizes a plurality of users, and the image setting unit sets the output device for two or more users among the plurality of users. If it is set to display an image of the character equipped with the amulet, and if the types of amulets of each user are the same, the output device displays an image of the character equipped with the amulet. An image of the state is displayed, and when the types of amulets of the respective users are not the same, the image of the state in which the character is equipped with the amulet is not displayed on the output device.

(11): In the aspect of (9) above, in the case where a different character is set for each of the plurality of users recognized by the recognition unit, the image control unit controls the talisman of each user. If the types are the same, the output device displays an image of the character equipped with the amulet, and if the types of amulets of each user are not the same, the output device displays the character wearing the amulet. The image of the equipped state is not displayed.

(12): In the aspect of (9) above, when a plurality of users are recognized by the recognition unit and the output device displays an image of a single character, the image control unit provides a talisman for each user. If the types of the amulets are the same, the output device displays an image of the character equipped with the amulet, and if the types of amulets of each user are not the same, the output device displays the character equipped with the amulet. It does not display an image of the equipped state.

(13): In a display control method according to another aspect of the present invention, a display control device that displays an image of a character on an output device displays, on the output device, an accompanying image associated with the character together with the image of the character. The present invention is a display control method that receives a setting for causing the incidental image to be displayed on the output device together with an image of the character, and accepts a setting for incidental information that is caused to be output from the output device in association with the incidental image being displayed on the output device together with the image of the character.

(14): A program according to another aspect of the present invention includes a display control device that causes an output device to display a character image, and a display control device that causes the output device to display an accompanying image associated with the character together with the character image. The program causes the output device to accept the setting of incidental information to be outputted from the output device in association with the fact that the incidental image is displayed together with the image of the character.

According to aspects (1) to (14), information can be provided to the user in a more appropriate output format.

1 is a configuration diagram of an agent system 1 including an agent device 100. FIG. 1 is a diagram showing the configuration of an agent device 100 and equipment mounted on a vehicle M according to a first embodiment. FIG. 2 is a diagram showing an example of the arrangement of display/operation devices 20. FIG. 2 is a diagram showing a configuration of an agent server 200 and a part of the configuration of an agent device 100. FIG. 3 is a diagram showing an example of the configuration of an agent function unit 150. FIG. 3 is a diagram illustrating an example of the contents of user setting information 162. FIG. 7 is a diagram illustrating an example of the contents of accompanying image setting information 164. FIG. It is a figure which shows an example of the content of accompanying information related DB255. FIG. 6 is a diagram for explaining a procedure for displaying an agent image wearing a predetermined accessory. FIG. 3 is a diagram for explaining an output mode in a first scene. FIG. 7 is a diagram for explaining an output mode in a second scene. FIG. 7 is a diagram for explaining the output mode in a third scene. FIG. 7 is a diagram for explaining the output mode in a fourth scene. FIG. 7 is a diagram for explaining the output mode in a fifth scene. FIG. 7 is a diagram for explaining the output mode in a sixth scene. FIG. 12 is a diagram for explaining the output mode in a seventh scene. FIG. 12 is a diagram for explaining the output mode in an eighth scene. 5 is a diagram showing an example of agent output by the image control unit 152 and the output control unit 158 in a list format. FIG. 3 is a flowchart showing an example of the flow of processing by the agent device 100. FIG. 3 is a flowchart showing an example of the flow of processing by the agent device 100. FIG.

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of a display control device, a display control method, and a program according to the present invention will be described below with reference to the drawings. An agent device, which is one aspect of a display control device, is a device that implements part or all of an agent system. Below, as an example of an agent device, an agent device installed in a vehicle (hereinafter referred to as vehicle M) and provided with one or more types of agent functions will be described. The agent function is, for example, a function that provides various information based on requests (commands) included in the user's utterances while interacting with the user using the vehicle M, and mediates network services. be. When multiple types of agents are implemented, the multiple types of agents may have different functions, processing procedures, controls, output modes, and contents. Further, some of the agent functions may have a function of controlling devices in the vehicle (for example, devices related to driving control and vehicle body control).

Agent functions include, for example, a voice recognition function that recognizes the user's voice (a function that converts voice into text), a natural language processing function (a function that understands the structure and meaning of text), a dialogue management function, and a function that This is realized by integrally using a network search function that searches other devices or searches a predetermined database held by the device itself. Some or all of these functions may be realized by AI (Artificial Intelligence) technology. In addition, a part of the configuration for performing these functions (especially the voice recognition function and the natural language processing interpretation function) is an agent server that can communicate with the in-vehicle communication device of vehicle M or the general-purpose communication device brought into vehicle M. (an example of an external device). The following description assumes that a part of the configuration is installed in an agent server, and that an agent device and an agent server cooperate to realize an agent system. Further, a service providing entity (service entity) that is made to appear virtually through the cooperation of an agent device and an agent server is called an agent. The agent device 100 causes an output device (such as a display) to display an image of a character related to the agent.

<Overall configuration>
FIG. 1 is a configuration diagram of an agent system 1 including an agent device 100. The agent system 1 includes, for example, an agent device 100 and an agent server 200. The agent in the present invention is an agent realized by a service provider. Examples of service providers include automobile manufacturers, network service providers, e-commerce operators, mobile terminal sellers and manufacturers, and any entity (corporation, organization, individual, etc.) Can be a service provider.

Agent device 100 communicates with agent server 200 via network NW. The network NW includes, for example, some or all of the Internet, a cellular network, a Wi-Fi network, a WAN (Wide Area Network), a LAN (Local Area Network), a public line, a telephone line, a wireless base station, and the like. Various web servers 300 are connected to the network NW, and the agent server 200 or agent device 100 can acquire web pages from the various web servers 300 via the network NW.

The agent device 100 is a device that can be mounted on a vehicle M, for example. The agent device 100 provides the user with a service in which an agent, which is a virtual subject, provides information through audio guidance using a speaker or image display using a display device. For example, the agent device 100 interacts with the user of the vehicle M, transmits the voice from the user to the agent server 200, and presents the answer obtained from the agent server 200 to the user in the form of voice output or image display. .

[vehicle]
FIG. 2 is a diagram showing the configuration of the agent device 100 and equipment mounted on the vehicle M according to the embodiment. The vehicle M includes, for example, one or more microphones 10, a display/operation device 20, a speaker unit 30, a navigation device 40, a vehicle device 50, an in-vehicle communication device 60, an occupant recognition device 80, and an agent device. 100 is installed. Additionally, a general-purpose communication device 70 such as a smartphone may be brought into the vehicle interior and used as a communication device. These devices are connected to each other via multiplex communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, wireless communication networks, and the like. Note that the configuration shown in FIG. 2 is just an example, and a part of the configuration may be omitted, or another configuration may be added. The display/operation device 20 is an example of a "display device." Furthermore, a combination of the display/operation device 20 and the speaker unit 30 is an example of an "output device."

The microphone 10 is a sound collection unit that collects sounds emitted inside the vehicle. The display/operation device 20 is a device (or a group of devices) that can display images and accept input operations. The display/operation device 20 includes, for example, a display device configured as a touch panel. The display/operation device 20 may further include a HUD (Head Up Display) and a mechanical input device. The speaker unit 30 includes, for example, a plurality of speakers (sound output sections) arranged at different positions within the vehicle interior. The display/operation device 20 may be shared by the agent device 100 and the navigation device 40. Details of these will be described later.

The navigation device 40 includes a navigation HMI (Human Machine Interface), a positioning device such as a GPS (Global Positioning System), a storage device that stores map information, and a control device (navigation controller) that performs route searching and the like. . Some or all of the microphone 10, display/operation device 20, and speaker unit 30 may be used as a navigation HMI. The navigation device 40 searches for a route (navigation route) for traveling from the position of the vehicle M specified by the positioning device to the destination input by the user, and allows the vehicle M to travel along the route. , outputs guidance information using the navigation HMI. The route search function may be located in a navigation server accessible via the network NW. In this case, the navigation device 40 obtains a route from the navigation server and outputs guidance information. Note that the agent device 100 may be constructed based on a navigation controller, and in that case, the navigation controller and the agent device 100 are configured as one piece in terms of hardware.

The vehicle equipment 50 includes, for example, a driving force output device such as an engine or a driving motor, an engine starting motor, a door lock device, a door opening/closing device, a window, a window opening/closing device, a window opening/closing control device, a seat, and a seat position control device. Control devices, rearview mirrors and their angular position control devices, vehicle interior and exterior lighting devices and their control devices, wipers and defoggers and their respective control devices, turn signal lights and their control devices, air conditioning devices, mileage and tire pressure information This includes vehicle information devices such as information on the amount of fuel left and fuel remaining.

The in-vehicle communication device 60 is a wireless communication device that can access the network NW using, for example, a cellular network or a Wi-Fi network.

The occupant recognition device 80 includes, for example, a seating sensor, an in-vehicle camera, an image recognition device, and the like. The seating sensor includes a pressure sensor provided at the bottom of the seat, a tension sensor attached to the seat belt, and the like. The vehicle interior camera is a CCD (Charge Coupled Device) camera or a CMOS (Complementary Metal Oxide Semiconductor) camera provided in the vehicle interior. The image recognition device analyzes the image taken by the vehicle interior camera, recognizes the presence or absence of a user in each seat, the user's face, etc., and recognizes the user's seating position. Further, the occupant recognition device 80 may identify the user seated in the driver's seat, passenger seat, etc. included in the image by performing matching processing with a face image registered in advance.

FIG. 3 is a diagram showing an example of the arrangement of the display/operation device 20. As shown in FIG. The display/operation device 20 includes, for example, a first display 22, a second display 24, and an operation switch ASSY 26. The display/operation device 20 may further include a HUD 28.

The vehicle M includes, for example, a driver's seat DS provided with a steering wheel SW, and a passenger seat AS provided in the vehicle width direction (-Y direction in the figure) with respect to the driver's seat DS. The first display 22 is a horizontally elongated display device that extends from around the middle between the driver's seat DS and the passenger seat AS on the instrument panel to a position facing the left end of the passenger seat AS. The second display 24 is installed approximately midway in the vehicle width direction between the driver's seat DS and the passenger seat AS, and below the first display. For example, both the first display 22 and the second display 24 are configured as touch panels, and include an LCD (Liquid Crystal Display), an organic EL (Electroluminescence), a plasma display, or the like as a display section. The operation switch ASSY 26 is an integrated set of dial switches, button type switches, and the like. The display/operation device 20 outputs the contents of the operation performed by the user to the agent device 100. The content displayed on the first display 22 or the second display 24 may be determined by the agent device 100.

[Agent device]
Returning to FIG. 2, the agent device 100 includes, for example, a management section 110 and an agent function section 150. The management unit 110 includes, for example, an audio processing unit 112, a setting unit 116, and an audio control unit 118. The setting section 116 includes, for example, an image setting section 116A and an information setting section 116B. The software arrangement shown in FIG. 2 is shown simply for explanation, and in reality, it may be arbitrarily modified so that, for example, the management section 110 may be interposed between the agent function section 150 and the in-vehicle communication device 60. can do.

Each component of the agent device 100 is realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components are hardware (circuit parts) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit). (including circuitry), or may be realized by collaboration between software and hardware. The program may be stored in advance in a storage device (a storage device with a non-transitory storage medium) such as an HDD (Hard Disk Drive) or flash memory, or may be stored in a removable storage device such as a DVD or CD-ROM. It is stored in a medium (non-transitory storage medium), and may be installed by loading the storage medium into a drive device.

The management unit 110 functions by executing programs such as an OS (Operating System) and middleware.

In order to activate the agent, the audio processing unit 112 performs audio processing on the input sound so that the agent is in a state suitable for recognizing a wakeup word preset for each agent.

The image setting unit 116A accepts settings for displaying an accompanying image associated with a character on an output device such as the display/operation device 20 together with the character image. Specifically, the image setting unit 116A, under the control of some of the agent function units 150 in response to instructions from the agent function unit 150, sets an image of an anthropomorphic character that communicates with the user in the vehicle interior, for example. (hereinafter referred to as an agent image) settings are accepted, and the accepted agent image is displayed on the first display 22 or the like. The agent image is, for example, an image in which the agent speaks to the user. The agent image may include, for example, a facial image whose expression and facial orientation are recognized by at least the viewer (user). For example, the agent image may include parts imitating eyes and a nose in the face area, and facial expressions and facial orientation may be recognized based on the positions of the parts in the face area. In addition, the agent image feels three-dimensional, and the viewer can recognize the direction of the agent's face by including an image of the head in three-dimensional space, and by including an image of the agent's body (torso and limbs). The character's actions (including behavior, posture, facial expressions, etc.) may be recognized. Further, the agent image may be an animated image. Furthermore, the agent image may be any type of object that can be anthropomorphized, such as an animal, a plant, an insect, an anime character, or a game character.

Further, the agent image may be displayed together with an accompanying image associated with the character. The accompanying images include, for example, one or both of ornaments and other character images. Ornaments include religious tools such as amulets, talismans, amulets, sutras, prayer beads, bells, medals, rosaries, bonfins, plakluwans, statues, and various ornaments with religious patterns and designs (necklaces, pendants, bracelets, etc.) (rings, brooches, glasses, sunglasses, earrings, hair accessories, hats, gloves, ties, scarves, mufflers, bags, umbrellas, key chains, etc.). Furthermore, the accessories may include, for example, clothes worn by the character (eg, suits, dresses, swimsuits, ski wear), weapons, armor, and the like. Other character images include, for example, images of characters associated with fellow passengers of the vehicle (e.g., occupants other than the driver), images of characters with motifs of children, friends, lovers, family members, etc. It will be done. The image setting unit 116A allows, for example, setting of accompanying images for each user. Further, the accompanying image may be set by the agent server 200 or various web servers 300.

The information setting unit 116B accepts settings for incidental information to be outputted to an output device such as the display/operation device 20 in association with the fact that the incidental image is displayed together with the character image. The incidental information is, for example, information regarding output information (eg, images and sounds) and output mode associated with the incidental image. Further, the accompanying information may include information regarding the behavior of the character when the accompanying image is displayed. The output information includes, for example, recommendation information recommended to the user. For example, the information setting unit 116B may configure the first display 22, the second display 24, and the speaker included in the speaker unit 30 in association with the user's display of the agent image on the first display 22 together with a predetermined accompanying image. Settings for incidental information to be output from some or all of them are accepted by the microphone 10, the display/operation device 20, etc. Further, the information setting unit 116B may receive information regarding whether or not the user has the authority to display an accompanying image or set an output mode such as a character's movement, as setting of accompanying information. The information setting unit 116B allows setting of incidental information for each user who uses the agent device 100. Further, the information setting unit 116B may be able to set incidental information in association with the state of the vehicle. Thereby, when the state of the vehicle is in the state associated with the setting in the information setting unit 116B, the incidental information can be output from the output device. The state of the vehicle includes, for example, the number of users (passengers) of the vehicle M, the state of the interior of the vehicle such as who is sitting in the driver's seat, the vehicle being stopped, the vehicle being stopped for a predetermined time or more, This includes the running state of the vehicle, such as when the vehicle is running or traveling at a predetermined speed or higher, and the operation mode of the vehicle, such as when the vehicle is operating in manual operation or when the vehicle is operating in automatic operation.

The audio control unit 118 causes some or all of the speakers included in the speaker unit 30 to output audio in response to instructions from the agent function unit 150. The audio control unit 118 may use the plurality of speaker units 30 to perform control to localize the sound image of the agent's voice at a position corresponding to the display position of the agent image. The position corresponding to the display position of the agent image is, for example, the position where the user is predicted to feel that the agent image is speaking the agent voice. Specifically, the position corresponding to the display position of the agent image (for example, 2 ~3 [cm]). Furthermore, localizing a sound image means, for example, determining the spatial position of the sound source that the user perceives by adjusting the volume of the sound transmitted to the user's left and right ears.

The agent function unit 150 cooperates with the agent server 200 to make an agent appear based on various information acquired by the management unit 110, and provides services including voice responses in response to the utterances of the user present in the vehicle. I will provide a. For example, when the wakeup word is included in the audio processed by the audio processing unit 112, the agent function unit 150 activates an agent associated with the wakeup word. Further, the agent function unit 150 performs control such as terminating the agent after providing the service. Further, the agent function unit 150 transmits the audio stream processed by the audio processing unit 112 to the agent server 200, and provides information obtained from the agent server 200 to the user.

Furthermore, the agent function section 150 may include one that is given authority to control the vehicle equipment 50. Further, the agent function unit 150 may have a function of cooperating with the general-purpose communication device 70 and communicating with the agent server 200. In this case, the agent function unit 150 performs pairing with the general-purpose communication device 70 using Bluetooth (registered trademark), for example, to connect the agent function unit 150 and the general-purpose communication device 70. Further, the agent function unit 150 may be connected to the general-purpose communication device 70 by wired communication using a USB (Universal Serial Bus) or the like. The details of the functions of the agent function section 150 will be described later.

[Agent server]
FIG. 4 is a diagram showing a configuration of the agent server 200 and a part of the configuration of the agent device 100. The configuration of the agent server 200 and the operations of the agent function unit 150 and the like will be described below. Here, description of physical communication from the agent device 100 to the network NW will be omitted.

The agent server 200 includes a communication section 210. The communication unit 210 is, for example, a network interface such as a NIC (Network Interface Card). Further, the agent server 200 includes, for example, a speech recognition section 220, a natural language processing section 222, a dialogue management section 224, a network search section 226, and a response sentence generation section 228. These components are realized by, for example, a hardware processor such as a CPU executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as LSI, ASIC, FPGA, or GPU, or may be realized by collaboration between software and hardware. Good too. The program may be stored in advance in a storage device such as an HDD or flash memory (storage device equipped with a non-transitory storage medium), or may be stored in a removable storage medium (non-transitory storage medium) such as a DVD or CD-ROM. The software may be installed by attaching the storage medium to a drive device.

Agent server 200 includes a storage unit 250. The storage unit 250 is realized by the various storage devices described above. The storage unit 250 stores data and programs such as a personal profile 251, a dictionary DB (database) 252, a knowledge base DB 253, a response rule DB 254, and an accompanying information related DB 255, for example.

In the agent device 100, the agent function unit 150 transmits an audio stream or an audio stream that has undergone processing such as compression and encoding to the agent server 200. Furthermore, when the agent function unit 150 recognizes a voice command that can be processed locally (processing that does not involve the agent server 200), the agent function unit 150 may perform the process requested by the voice command. A voice command that can be locally processed is a voice command that can be answered by referring to a storage unit (not shown) included in the agent device 100, or, in the case of the agent function unit 150, a voice command that controls the vehicle equipment 50. (for example, a command to turn on an air conditioner). Therefore, the agent function unit 150 may have some of the functions included in the agent server 200.

When the audio stream is acquired, the audio recognition unit 220 performs audio recognition and outputs text information, and the natural language processing unit 222 performs semantic interpretation on the character information while referring to the dictionary DB 252. The dictionary DB 252 has abstracted semantic information associated with character information. The dictionary DB 252 may include list information of synonyms and similar words. The processing of the speech recognition unit 220 and the processing of the natural language processing unit 222 are not clearly separated into stages, and there are mutual interactions such as the speech recognition unit 220 correcting the recognition result in response to the processing result of the natural language processing unit 222. It may be done by influencing each other.

For example, if the meaning of "Today's weather is" or "Today's weather is" is recognized as a recognition result, the natural language processing unit 222 generates a command in which the standard character information is replaced with "Today's weather". Thereby, even if there is character fluctuation in the voice of the request, it is possible to facilitate the conversation that meets the request. Further, the natural language processing unit 222 may recognize the meaning of character information using artificial intelligence processing such as machine learning processing using probability, or may generate a command based on the recognition result, for example.

Based on the processing results (commands) of the natural language processing unit 222, the dialogue management unit 224 determines the content of the utterance to the user of the vehicle M while referring to the personal profile 251, knowledge base DB 253, response rule DB 254, and accompanying information related DB 255. decide. The personal profile 251 includes the user's personal information, hobbies and preferences, history of past interactions, etc. that are saved for each user. The knowledge base DB 253 is information that defines relationships between things. The response rule DB 254 is information that defines actions (responses, details of device control, etc.) that the agent should perform in response to commands. The accompanying information related DB 255 stores information regarding an output mode when an agent image is displayed in association with a predetermined accompanying image. The information regarding the output mode may include, for example, information regarding the behavior of the character or the like in the agent image in addition to the display contents of the character image and accompanying images. Details of the accompanying information related DB 255 will be described later.

Furthermore, the dialogue management unit 224 may identify the user by comparing it with the personal profile 251 using feature information obtained from the audio stream. In this case, in the personal profile 251, for example, voice characteristic information is associated with personal information. Voice feature information is, for example, information about speaking style features such as voice pitch, intonation, and rhythm (patterns of pitch and low pitches), and feature quantities such as Mel Frequency Cepstrum Coefficients. . The voice characteristic information is, for example, information obtained by having the user utter a predetermined word, sentence, etc. at the time of initial registration of the user, and recognizing the uttered voice.

If the command requests information that can be searched via the network NW, the dialogue management unit 224 causes the network search unit 226 to perform the search. The network search unit 226 accesses various web servers 300 via the network NW and obtains desired information. "Information that can be searched via the network NW" is, for example, the evaluation results of restaurants near the vehicle M by general users, the weather forecast according to the location of the vehicle M on that day, or the number of passengers. It may be recommendation information depending on the situation.

Further, the dialogue management unit 224 may generate recommendation information according to the output mode of the agent image, accompanying image, and the like. The recommendation information includes various contents such as images, videos, and audio. In this case, the dialogue management unit 224 refers to the accompanying information related DB 255 based on, for example, the type of agent image to be output to the user from the vehicle M, the type of accompanying image, etc., and outputs the output mode associated with the matching information. It performs control such as acquiring information (for example, recommendation information) and providing it to the user, giving it a higher priority than other content, etc. The recommendation information included in the recommendation information DB 256 may be stored in the storage unit 250 in advance, or may be obtained from various web servers 300 via the network NW.

The response sentence generation unit 228 generates a response sentence and transmits it to the agent device 100 so that the content of the utterance determined by the dialogue management unit 224 is conveyed to the user of the vehicle M. When the user is identified as a user registered in the personal profile, the response sentence generation unit 228 generates a response sentence that calls the user's name or speaks in a manner similar to the user's speaking style. Good too.

When the agent function unit 150 obtains the response sentence, it instructs the voice control unit 118 to perform voice synthesis and output voice. Further, the agent function unit 150 generates an image of the agent in accordance with the audio output, and instructs the management unit 110 to display the generated image. Further, the agent function unit 150 generates device control information for adjusting the volume output from the speaker unit 30 and controlling devices related to the vehicle device 50 (for example, air conditioner) in accordance with instructions from the agent server 200. Then, the generated device control information is output to the vehicle device 50 or the like. In this way, an agent function in which the virtually appearing agent responds to the user of the vehicle M and a service provided in an output mode associated with the display content of the accompanying image are realized.

[Agent function section]
Next, details of the agent function section 150 will be explained. FIG. 5 is a diagram showing an example of the configuration of the agent function section 150. The agent function section 150 includes, for example, an image control section 152, a user recognition section 154, an agent-related information setting section 156, an output control section 158, and a storage section 160. The storage unit 160 stores data and programs such as user setting information 162 and accompanying image setting information 164, for example. The user recognition unit 154 is an example of a “recognition unit”.

The image control unit 152 causes the display/operation device 20 to display an image of an agent that provides a service including voice guidance. For example, the image control unit 152 causes the display/operation device 20 to display an image of the agent equipped with a predetermined accessory or an image of the agent including a pet or other agent.

The user recognition unit 154 recognizes the user who uses the output device. For example, the user recognition unit 154 recognizes the user operating the display/operation device 20 or interacts with the agent based on the user's movements and mouth movements recognized by the occupant recognition device 80. Recognize users. Further, the user recognition unit 154 determines whether the user recognized by the occupant recognition device 80 is registered in advance as a user using the agent device 100 or is a user for whom an accompanying image is set for the agent image. recognize. Further, the user recognition unit 154 may recognize whether the user using the output device is a user who has the authority to display an accompanying image or set an output mode such as a character's movement.

The agent-related information setting unit 156 configures the settings when the user recognized by the user recognition unit 154 has the authority to set the agent image and accompanying images, and the output mode including the agent's operation. It cooperates with the unit 116 to receive various setting information from the user. For example, when the user recognized by the user recognition unit 154 has the authority to display an accompanying image, the agent-related information setting unit 156 displays the agent image set by the information setting unit 116B together with the accompanying image. Accepts the settings for the behavior of the agent (character) in the case. The agent-related information setting unit 156 stores information received from the user, such as the type of character, the type of accompanying image, and the output mode, in the storage unit 160 as user setting information 162.

FIG. 6 is a diagram showing an example of the contents of the user setting information 162. The user setting information 162 is information in which a user ID that identifies a user is associated with an agent ID that identifies an agent image associated with an agent, accompanying image information, and output mode information. The accompanying image information stores, for example, an accompanying image identification ID for identifying an accompanying image and a specific name of the accompanying image. The output mode information includes incidental information. The output mode information stores, for example, the output mode of the agent image identified by the agent ID and the output mode of the accompanying image displayed along with the agent image. The output mode may be set, for example, by selecting one from a plurality of output patterns set in advance or registered by the user. The output patterns include, for example, patterns related to facial expressions and actions (for example, putting hands together, waving hands, dancing, walking), etc. of the character in the agent image. Further, the user setting information 162 may store condition settings regarding whether or not to display an accompanying image. Condition settings include, for example, ``I do not want to display images of decorations when other people are riding the vehicle,'' ``I do not want to see images of decorations set by others (do not display them),'' and ``I do not want to display images of decorations set by others.'' Conditions regarding whether or not display is permitted, such as, "If a person is riding with the vehicle, the image of the accessory that has been set may be shown (displayed)."

Further, the agent-related information setting unit 156 stores information regarding the accompanying image set by the user in the accompanying image setting information 164. FIG. 7 is a diagram showing an example of the contents of the accompanying image setting information 164. The accompanying image setting information 164 is information in which accompanying image information is associated with image data and an image ID that identifies the image. The type of image may include, for example, classification information for identifying whether or not predetermined ornaments may be displayed at the same time when a plurality of users are riding in the vehicle M. For example, since ornaments provided by the same organization have a high affinity with each other and may be displayed at the same time, the same ornament type is set. In addition, if the religious organizations that provide the ornaments have a relationship between the head office (main shrine, main temple) and branch offices (sub-sha/setsu-sha, sub-temple), they may be displayed at the same time as there is a high degree of affinity. to the same decoration type. Furthermore, since non-religious ornaments may be displayed at the same time, they are set to the same ornament type.

Note that if a single user has set to display an image of the agent image equipped with multiple ornaments of different types (for example, Shinto ornaments and Buddhist ornaments) at the same time, , differences in the types of ornaments can be ignored.

When the user acquires a character image or an accompanying image, the agent-related information setting unit 156 may set information to be stored in the user setting information 162 or the accompanying image setting information 164 regarding the acquired character image or accompanying image. , may be set before or after acquiring the image. Further, the agent-related information setting unit 156 may change already set information according to the user's settings. Further, the agent-related information setting unit 156 may change the user setting information 162 and the accompanying image setting information 164 based on information acquired from the agent server 200. The user setting information 162 and the accompanying image setting information 164 may be stored in the storage unit 250 of the agent server 200.

For example, when the voice generated by the user includes a wake-up word, or when responding to an inquiry from the user, the output control unit 158 performs the output control based on the user setting information 162 and the accompanying image setting information 164. , generates an agent image and an accompanying image to be displayed on the display/operation device 20. Further, the output control unit 158 generates an agent image and an accompanying image, generates the agent's voice, and generates control information regarding vehicle equipment based on the response result and output mode acquired from the agent server 200.

Here, the dialogue management unit 224 of the agent server 200 refers to the accompanying information related DB 255 and determines the content to be recommended to the user, for example, based on the accompanying image displayed together with the agent image being displayed to the user. It is output to the output control section 158. FIG. 8 is a diagram showing an example of the contents of the accompanying information related DB 255. The accompanying information related DB 255 is, for example, information in which accompanying image information, output information, and recommendation content are associated with each other. The output information includes, for example, information regarding safe driving associated with a predetermined accessory, and information regarding display control details of accompanying images. For example, if a predetermined ornament is a "talisman" for traffic safety, the output information includes driving advice, omens from the gods and Buddha, fortune-telling about traffic luck, and the like. Additionally, if the specified decoration is a health-related ``talisman'' or ``accessory,'' the output information may include expert advice on diet, information on stretching exercises you can do in the car, and recommendations for restaurants and supermarkets near your destination. Includes information, etc. In addition, if a given accessory is an accessory with strong religious meaning, the output information may include information based on the religion that provided the accessory (for example, to promote faith in that religion, statements by religious leaders such as priests and clergy) may also be included.

Note that the output information may reflect the contents of a vow (determination, goal, etc.) registered in advance by the user. For example, when a user registers a predetermined accessory or changes settings, the oath is also registered, and, like other utterances by the user, is recognized by the voice recognition unit 220 of the agent server 200 and the natural voice recognition unit 220 of the agent server 200. It is analyzed by the language processing unit 222. Thereby, the response sentence generation unit 228 can read the information related to the oath from the accompanying information related DB 255 and transmit it to the agent device 100, and can output output information reflecting the content of the oath to the agent. For example, if a predetermined ornament is a traffic safety talisman and the user has registered in advance as a pledge of "no accidents and no violations," the output information may include a message that warns the vehicle M before it approaches an accident-prone area. This includes information on legal speed limits, etc.

Further, conditions for reflecting the content of the oath in the output information and conditions for not reflecting the content of the oath may also be set. For example, if you do not want your fellow passengers to know your personal oath, the user may set the condition for outputting information when equipped as ``If you are riding alone, output information that reflects the content of your oath.'' However, if someone else is on board, the content of the oath will not be reflected in the output information.

In addition, if the accompanying image is fashion such as clothes, headphones, etc., the output information includes information indicating whether the agent wears or not (equipment information), and the behavior of the character associated with the accompanying image, such as the user, etc. It may be set by For example, if the accompanying image is fashion for mountain climbing, the output information is set to show the character walking wearing mountain climbing clothes, or if the accompanying image is fashion for the sea, the output information is set to show the character walking while wearing mountain climbing clothes. Swimming behavior is set when wearing marine equipment. In addition, if the accompanying image is another character such as a pet, the output information includes information indicating the display mode and output pattern (for example, characters are displayed side by side), and information indicating the actions of the other character. etc. may be set.

The recommendation content is recommendation information provided to the user in relation to the accompanying image and the output information of the accompanying image. The recommendation information may include incidental information. The recommendation information includes, for example, information related to driving (eg, driving route, driving support information), a facility that is a vendor or supervisor of the product (article) shown in the accompanying image, and a related place. For example, if the predetermined accompanying image is an image of a traffic safety amulet (an example of an ornament), the recommended content will include a temple or shrine that provides the amulet, or a temple or shrine that is of the same sect as the source of the amulet. include. The accompanying information related DB 255 may be updated with user setting information obtained from the agent device 100.

In the example of FIG. 8, when the traffic safety amulet of XX Shrine is to be displayed as an accompanying image, and when "information regarding safety support (pattern 1)" is selected as the output information, the dialogue management unit 224 The route with the lowest number is prioritized and recommended, and the response statement generating unit 228 generates a response statement for recommending the temple/shrine that is the provider. The route with fewer accidents is acquired by the network search unit 226 from a management device or the like that manages traffic information via the network NW, for example. Further, when "information regarding safety support (pattern 2)" is selected as the output information, the dialogue management unit 224 causes the response sentence generation unit 228 to generate a response sentence for recommending information regarding driving support. The information related to driving support includes, for example, information for adjusting the distance between the vehicle and the vehicle ahead, information for suppressing lane departure, information for instructing the timing of changing lanes, and the like.

Further, in the example of FIG. 8, when the dialogue management unit 224 causes the agent image to wear a fashion image for the sea as an accompanying image, the dialogue management unit 224 recommends music related to the sea or recommends a seaside route. The response sentence generation unit 228 generates a response sentence for recommending a seafood restaurant.

Further, in the example of FIG. 8, when an output mode is set in which the agent image is displayed with headphones attached as an accompanying image, the dialogue management unit 224 recommends information regarding music. Furthermore, instead of (or in addition to) recommending information regarding music, the dialogue management unit 224 increases the volume of the sound output from the speaker unit 30 by a predetermined amount compared to when the agent image is not wearing headphones. The response sentence generation unit 228 may generate information regarding the output mode such as increasing the size. Furthermore, when an output mode is set in which the agent image is displayed with earplugs attached as an accompanying image, the dialogue management unit 224 may reduce the sound output from the speaker unit 30 (or do not output any sound). The response sentence generation unit 228 may generate information regarding the output mode.

The dialogue management unit 224 also recommends content when no other characters are displayed (when only the self-agent image is displayed), or according to the number and attributes of other characters to be displayed (for example, friends and family). may be changed.

For example, the output control unit 158 provides content (recommendation information, etc.) related to a predetermined accompanying image to the user with priority over other content. For example, when displaying an image of another character as an accompanying image on the output device together with the agent image, the output control unit 158 may prioritize content according to the number and type of characters to be displayed over other content. provided to the user. For example, when a family is displayed as an accompanying image, the output control unit 158 obtains a family-friendly restaurant or music from the agent server 200 and causes the display unit of the display/operation device 20 to display the information. Further, when the agent image is wearing an aloha shirt, which is an accompanying image, the output control unit 158 displays the recommended information of a restaurant in the Hawaiian area on the display/operation device 20 with priority over other recommended information. section. This makes it possible to provide information to the user in a more appropriate output format depending on the content such as the number and type of agent images and accompanying images.

Further, the output control unit 158 may output incidental information such as output mode information to an output device such as the display/operation device 20 when the user is a user who has the authority to display an incidental image. For example, when the user has set to display an image of a traffic safety charm, the output control unit 158 outputs preset information regarding safe driving (in the example of FIG. 8, pattern 1 and pattern 2). The information may be presented to the user as a candidate for mode information or output information, and the user may select from among the presented candidates. This allows the user to easily set incidental information.

[Output mode of incidental information]
Next, the output mode of incidental information in this embodiment will be specifically explained. FIG. 9 is a diagram illustrating a procedure for displaying an agent image wearing a predetermined accessory. For example, a user of vehicle M purchases a predetermined ornament in person at a temple or shrine. The accessories AC purchased by the user are accompanied by a two-dimensional code for image registration, and a dedicated application can be run using the terminal device T used by the user (for example, a smartphone or tablet) or the in-vehicle camera of the vehicle M. Read the two-dimensional code through the The dedicated application reads VR (Virtual Reality) information obtained by reading a two-dimensional code, and virtually equips the agent image A1 mounted on the vehicle with the accessory image IMA.

Note that even if a dedicated application uses an in-vehicle camera or the like to acquire image information that shows the actual appearance of a predetermined accessory, and generates an image of the accessory image IMA to be virtually equipped on the agent image A1. good. In addition, the user accesses a market site dedicated to images provided by various web servers 300 via the terminal device T, and purchases the right to use the image of the accessory image IMA that can be virtually equipped to the agent image A1. You may. In the following explanation, a user who has the authority to display a predetermined ornament is a user who has been authorized to display the ornament image IMA through purchase or transfer as described above, and whose usage settings of the agent device 100 are registered. Possible (or pre-registered) users.

The image control unit 152 displays an agent image equipped with a predetermined accessory on the display/operation device 20 when the user of the agent device 100 recognized by the user recognition unit 154 has the authority to display the predetermined accessory. Display. When the image control unit 152 causes the display/operation device 20 to display an agent image equipped with a predetermined accessory, the output control unit 158 refers to the accompanying information related DB 255 and provides output information to the user. “When displaying an agent image equipped with a predetermined accessory on the display/operation device 20,” in addition to the state where an agent image equipped with a predetermined accessory is already displayed on the display/operation device 20, , including providing the user with output information that is about to start displaying (at the same time as starting displaying the image on the display/operation device 20, or immediately before displaying the image).

For example, when the user has the authority to display a predetermined ornament, the output control unit 158 can accept settings for incidental information for each user, and the setting contents are stored in the user setting information 162. . Further, the output control unit 158 can set output information in association with the state of the vehicle M (which user is on board, whether there is one user or multiple users, etc.). Hereinafter, the output mode for each scene will be specifically explained.

[First scene]
FIG. 10 is a diagram for explaining the output mode in the first scene. The first scene shows a scene where the user UA is alone in the driver's seat DS of the vehicle M, and the accompanying image of the amulet is provided to the user UA along with the agent image A1. In the first scene, the image control unit 152 generates an image IM1 in which the agent image A1 is equipped with the accompanying image IMA as preset by the user UA, and displays the generated image IM1 on the display/operation device 20. Display it on the display. Further, the output control unit 158 causes the speaker unit 30 to output a voice such as "This is an announcement from the god of the XX shrine" as the output information (incidental information) OP1 in the first scene.

[Second scene]
FIG. 11 is a diagram for explaining the output mode in the second scene. In the second scene, when the user recognition unit 154 recognizes that the user UA who is seated in the driver's seat DS of the vehicle M and the user UB who is seated in the passenger seat AS are in the vehicle, This shows a scene in which output information including accompanying images is provided to users UA and UB. In the second scene, it is assumed that the users UA and UB have set the same agent image A1 and the same type of accompanying image IMA to be output in the same output pattern. In this case, the image control unit 152 generates an image IM1 in which the single agent image A1 is equipped with the accompanying image IMA, and displays the generated image IM1, as the users UA and UB have set the display in advance. -Display on the display section of the operating device 20. Further, the output control unit 158 causes the speaker unit 30 to output a sound such as "This is an announcement from the god of the XX shrine" as output information OP1 in the second scene, similar to the first scene.

Note that in the second scene, "the accompanying images are of the same type" means, for example, that the accompanying image identification IDs are the same. In addition, "the accompanying images are of the same type" means that when the accompanying images are images of decorative items such as amulets, the religious organization that sells the decorative items is the same, or that the religious organization that sells the decorative items It may also include that the religion or sect of the two people are the same, that the purpose of the ornaments is the same (for example, "safety", "health", etc.), and that the appearance of the ornaments is the same. Further, if users UA and UB have set different ornaments to be displayed, and none of the predetermined ornaments set by users UA and UB is religious, the image control unit 152 , display control may be performed assuming that the types of ornaments are the same.

Note that even if the users UA and UB have set to output different agent images and the same predetermined ornament, the image control unit 152 outputs the same predetermined ornaments for each of the agent images as in the second scene. An image with the accessory equipped may be displayed. Furthermore, if the users UA and UB have set to display the same agent image and the same type of predetermined ornaments, and the users UA and UB have registered different oaths, the output control unit 158 may output, as incidental information, output information based on the oaths of the users UA and UB, or output information based on the oaths of the user UA seated in the driver's seat DS among the users UA and UB. It may be output, or output information based on the oath may not be output. For example, when user UA has registered a pledge regarding traffic safety and user UB has registered a pledge regarding health, the output control unit 158 outputs output information related to traffic safety and output information related to health as accompanying information. Output both of the output information.

[Third scene]
FIG. 12 is a diagram for explaining the output mode in the third scene. The third scene is a scene in which the user recognition unit 154 recognizes that there are a user UA seated in the driver's seat DS of the vehicle M and a user UC seated in the passenger seat AS. In the third scene, it is assumed that users UA and UC are set to output the same agent image and different amulet images. In this case, the image control unit 152 generates an image IM2 in which the agent image A1 is not equipped with an amulet image, and displays the generated image on the display unit of the display/operation device 20. This is because the amulet image set by the user UA and the amulet image set by the user UC belong to different types of accessories in the accompanying image setting information 164. In this way, the agent device 100 controls the display of different ornaments so that the users UA and UC do not display images of ornaments of other religions than they expected. By seeing this, you can reduce the possibility of feeling uncomfortable.

Furthermore, in the third scene, the output control unit 158 generates output information OP2 that is not related to each ornament, and causes the speaker unit 30 to output the output information OP2. In the example of FIG. 12, a voice saying "Let's drive safely." is output. As a result, the agent device 100 may cause the users UA and UC to feel uncomfortable by hearing other religious remarks that they are not expecting, or if the user UA in the vehicle M after providing the output information OP2 It is also possible to reduce the possibility that the mutual atmosphere between the UB and the UB becomes bad.

[Scene 4]
FIG. 13 is a diagram for explaining the output mode in the fourth scene. The fourth scene is a scene in which the user recognition unit 154 recognizes that there are a user UA seated in the driver's seat DS of the vehicle M and a user UD seated in the passenger seat AS. In the fourth scene, users UA and UD are set to output different agent images A1 and A2, respectively, and user UA is set to display a predetermined ornament, but user UD is set to output a predetermined ornament. If no decorations are set to be displayed, the image control unit 152 generates an image IM3 in which agent images A1 and A2 are not equipped with predetermined decorations in the fourth scene. The image IM3 is displayed on the display section of the display/operation device 20. Furthermore, the output control section 158 causes the speaker unit 30 to output output information OP2 that is not related to the predetermined ornament.

Thereby, the agent device 100 can reduce the possibility that the user UA and the user UD will feel uncomfortable if they see an ornament of another religion that they are not expecting. In addition, the agent device 100 may make the user UA and the user UD feel uncomfortable by hearing other religious remarks that they do not expect, or may feel uncomfortable about the atmosphere in the vehicle M after providing the output information OP2. This can reduce the possibility of deterioration.

Note that if user UA and user UD allow other users to display ornaments of other religions, image control unit 152 displays predetermined images on agent image A1 and agent image A2, respectively. You may display an image equipped with accessories.

[Scene 5]
FIG. 14 is a diagram for explaining the output mode in the fifth scene. The fifth scene is a diagram showing a scene in which the agent image is caused to perform a predetermined operation when the agent image is displayed together with the accompanying image. For example, if the user recognition unit 154 recognizes that there is a user UA sitting in the driver's seat DS of the vehicle M, and if the user UA is alone in the driver's seat DS of the vehicle M, the talisman This shows a scene in which the accompanying image is provided to the user UA together with the agent image A1. In the fifth scene, the image control unit 152 generates an image IM4 in which the agent image A1 is equipped with the accompanying image IMA, as set in advance by the user UA, and displays the generated accompanying image IM4 on the display/operation device. 20 on the display section. Here, if the user UA has set an action in which the character of the agent image A1 joins hands together as incidental information, the image control unit 152 generates an image in which the character of the agent image A1 joins hands. . The accompanying image IM4 may be an animation image. Further, the output control unit 158 outputs a sound such as "This is an announcement from the god of the XX shrine" as the output information OP1 in the first scene. This allows the agent image to perform a predetermined action in accordance with the accompanying image displayed together with the agent image, thereby making the agent more familiar to the user.

[Scene 6]
FIG. 15 is a diagram for explaining the output mode in the sixth scene. The sixth scene is different from the fifth scene described above in that the user sitting in the driver's seat DS is the user UE instead of the user UA. In this case, when the user UE recognized by the user recognition unit 154 is recognized, the image control unit 152 configures the agent image A1 with the associated image IMA as set in advance by the user UE. An image IM5 is generated, and the generated image IM5 is displayed on the display section of the display/operation device 20. Here, if the user UE has set an action in which the character of the agent image A1 bows as incidental information, the image control unit 152 generates an image in which the character of the agent image A1 bows. Image IM5 may be an animation image in which the character of agent image A1 bows.

As shown in the fifth scene and the sixth scene, by setting the action as incidental information set for each user, it is possible to cause the agent to perform an appropriate action for each user.

Furthermore, when multiple users are recognized by the user recognition unit 154 and each user has set a different action, the output control unit 158 may control the character not to perform any action. good. Furthermore, when each user has set common or highly related actions, the output control unit 158 may display the movements of the respective characters in synchronization or coordination. For example, the output control unit 158 may cause a plurality of characters to wave their hands together or perform a walking motion, or have the characters shake hands with each other, or alternately jump. In this way, by having the characters move in synchronization or in cooperation with each other, it is possible to help the users deepen their friendship with each other.

[Scene 7]
FIG. 16 is a diagram for explaining the output mode in the seventh scene. The seventh scene shows a scene where a recommendation is made to the user according to incidental information associated with the agent. In the seventh scene, it is assumed that the user UA is seated in the driver's seat of the vehicle M. When the user UA recognized by the user recognition unit 154 is recognized, the image control unit 152 displays an image IM6 in which the agent image A1 is wearing fashion for the sea, as set in advance by the user UA. let Here, the output control unit 158 acquires recommendation information regarding the sea from the agent server 200, and outputs the acquired recommendation information to the user with priority over other recommendation information.

In the example of FIG. 16, when the user UA inquires from the agent about the driving route to the destination, among the routes obtained as a result of the inquiry, the agent image is wearing fashion for the sea; Display routes with priority. Further, the output control unit 158 outputs a voice such as "We will travel along the seaside route" as output information OP3 in the seventh scene. According to the scene in FIG. 7, the recommendation information associated with the accompanying image set by the user can be provided to the user with priority over other recommendation information.

[Scene 8]
FIG. 17 is a diagram for explaining the output mode in the eighth scene. The eighth scene shows a scene in which recommendation information is output when a plurality of users are recognized by the user recognition unit 154. In the example of FIG. 17, it is assumed that the user UA is seated in the driver's seat DS and the user UD is seated in the passenger seat AS, similarly to FIG. 13. In this case, the image control unit 152 displays agent images A1 and A2 corresponding to the respective users UA and UD, and also recommends sightseeing spots etc. that can be enjoyed by two people as recommendation information when two agents are displayed. Recommend. Further, the output control unit 158 outputs a voice such as "I will introduce a sightseeing spot that two people can enjoy" as output information OP4 in the eighth scene.

Thereby, more appropriate recommendation information can be provided preferentially to the user depending on the user riding in the vehicle M or the displayed character image. As a result, more appropriate information can be provided to the user.

FIG. 18 is a diagram showing examples of output of agent images by the image control unit 152 and the output control unit 158 in a list format. The information shown in FIG. 18 is used, for example, during control by the image control section 152 and the output control section 158. In the example of FIG. 18, the output image is associated with information indicating whether the agents are the same, information indicating a group of decorations, and information indicating whether to display other people's decorations. . For example, when there are multiple users recognized by the user recognition unit 154, the displayed agent images are the same, and the types of predetermined accessories are the same (case No. 1 shown in FIG. 18). , the image control unit 152 causes the display/operation device 20 to display an image in which the agent image A1 is equipped with a predetermined accessory image IMA. Note that even if there is only one user recognized by the user recognition unit 154 and the setting is made to display a predetermined ornament, the agent image A1 will similarly generate an image equipped with the predetermined ornament image IMA. , causes the display/operation device 20 to display the generated image. At this time, the output control unit 158 causes the agent to provide incidental information regarding the equipment.

Further, there is a case where there are multiple users recognized by the user recognition unit 154, the displayed agent images are the same, and the types of predetermined decorations are not the same, and at least one of the users recognizes the decorations of others. If the display of the accessory image is not permitted (case No. 2 shown in FIG. 18), the image control unit 152 displays an image in which the agent image A1 is not equipped with the predetermined accessory image IMA. Displayed on the display/operation device 20. At this time, the output control unit 158 does not allow the agent to provide the incidental information as a service.

Further, there is a case where there are multiple users recognized by the user recognition unit 154, the displayed agent images are the same, and the types of predetermined accessories are not the same, and all of the users are When the image display is set to be permitted (in the case of No. 3 shown in FIG. 18), the image control unit 152 controls whether the agent image A1 is a superimposed image of a predetermined accessory image IMA set by each user. The image is displayed on the display/operation device 20. At this time, the output control unit 158 may cause the agent to provide output information related to each of the predetermined accessories as a service, or may cause the agent to provide only the equipment information set by one of the users as a service. It's fine, and you don't have to provide information when equipped.

On the other hand, when there are multiple users recognized by the user recognition unit 154, the displayed agent images are different, and the types of predetermined accessories are the same (case No. 4 shown in FIG. 18), the image The control unit 152 causes the display/operation device 20 to display images in which each of the agent image A1 and the agent image A2 is equipped with a predetermined accessory image IMA. At this time, the output control unit 158 may cause the agent to provide output information related to each of the predetermined accessories as a service, or may cause the agent to provide only the equipment information set by one of the users as a service. It's fine, and you don't have to provide information when equipped.

Further, there is a case where there are multiple users recognized by the user recognition unit 154, the displayed agent images are different, and the types of predetermined accessories are not the same, and at least one of the users (in the case of No. 5 shown in FIG. 18), the image control unit 152 determines that the agent image A1 and the agent image A2 are not equipped with the predetermined accessory image IMA. The image is displayed on the display/operation device 20. At this time, the output control unit 158 does not allow the agent to provide the installation information as a service.

Further, if there are multiple users recognized by the user recognition unit 154, the agent images to be displayed are different, and all of the users are set to allow displaying images of other people's accessories, one agent is set to be equipped with a predetermined accessory, and the other agent is set to be equipped with no accessory (in the case of No. 6 shown in FIG. 18), the image control unit 152 determines that the agent image A1 is set by the user. The agent image A2 is equipped with a predetermined accessory image IMA that has been installed, and the display/operation device 20 displays an image in which the agent image A2 is not equipped with an accessory. At this time, the output control unit 158 may cause the agent to provide the equipment information as a service, or may not provide the equipment information.

Further, if there are multiple users recognized by the user recognition unit 154, the agent images to be displayed are different, and all of the users are set to allow the display of other people's ornament images, each agent When the images are set to be equipped with different predetermined decorations (in the case of No. 7 shown in FIG. 18), the image control unit 152 controls the agent image A1 and the agent image A2 to be equipped with different predetermined decorations set by the user. The image equipped with the product image IMA is displayed on the display/operation device 20. At this time, the output control unit 158 may cause the agent to provide output information related to each of the predetermined accessories as a service, or may cause the agent to provide only the equipment information set by one of the users as a service. It's fine, and you don't have to provide information when equipped.

In addition to the information shown in FIG. 18, the output example list format may also include output images associated with groups of attached images other than ornaments and information on whether to permit display. . Furthermore, the output image shown in FIG. 18 may include information regarding the agent's actions (for example, joining hands, bowing, waving, dancing, walking, etc.).

[Processing flow]
Next, the flow of processing in the agent device 100 will be explained. Note that, below, a setting process for accepting settings of an agent and an accompanying image by a user, and a display control process for an agent based on the set information will be explained separately.

FIG. 19 is a flowchart illustrating an example of the flow of setting processing by the agent device 100. The processing flow shown in FIG. 19 is performed, for example, before the user gets on the vehicle M or at the timing when the user purchases a predetermined accessory.

First, the agent device 100 receives display agent settings from the user (step S100). Next, the agent-related information setting unit 156 receives registration of accompanying information to be displayed together with the image of the agent displayed on the display/operation device 20 (step S102). The accompanying information includes, for example, registration of purchased articles such as ornaments and clothes, and accompanying images related to the registered articles. Furthermore, information regarding the user's oath may also be registered in the accompanying information. Next, the output control unit 158 refers to the incidental information related DB 255 and presents candidates for incidental information such as output information and output mode (step S104), accepts the user's selection of incidental information, and provides user setting information. 162 and the accompanying information related DB 255 (step S106). Thereby, the processing of this flowchart ends.

FIG. 20 is a flowchart illustrating an example of the flow of display control processing by the agent device 100. The processing flow shown in FIG. 20 may be performed, for example, when the agent function unit 150 is displaying an agent image to provide a service, when the agent image display is started, or when a new user is recognized by the occupant recognition device 80. will be carried out at the specified timing. The following description uses an image of an ornament as an example of an accompanying image.

First, the user recognition unit 154 calls the user setting information 162 of the user recognized by the occupant recognition device 80 (step S200). Next, the user recognition unit 154 determines whether there is only one recognized user (step S202). If it is determined that there is only one user, the image control unit 152 starts displaying an image in which the agent image is equipped with a predetermined accessory according to the user's presettings stored in the user setting information 162, The output control unit 158 provides information based on the output information settings (step S204). In the process of step S204, the output control unit 158 controls the agent to perform a predetermined action according to the accessory image displayed together with the agent image, or to output recommendation information preferentially according to the accessory image. You may do so.

Further, in the process of step S202, if it is determined that the number of recognized users is not one (there is a plurality of users), the image control unit 152 determines the types of accessories set by each of the recognized users. It is determined whether they are the same (step S206). Note that in the process of step S206, "the ornaments are the same" may include that the ornaments completely match or that the types of the ornaments are the same (or similar). Further, in the process of step S206, if at least one of the ornaments is not religious, it may be determined that "the ornaments are the same".

In the process of step S206, if it is determined that the accessories are the same, the image control unit 152 determines to display the agent image equipped with the accessory (step S208). In the process of step S206, if it is determined that the ornaments are not of the same type, the image control unit 152 refers to the user setting information 162, etc., and determines whether the user allows display of the ornament image. (Step S210). "Allow display" indicates a case where the output mode information of the user setting information 162 is set to display the accompanying image in some output pattern. If display is permitted, the image control unit 152 advances the process to step S208. If display is not permitted, the image control unit 152 determines to display an agent image without accessories (step S212).

After the processing in step S208 or step S212, the image control unit 152 determines whether the same agent is to be displayed (step S214). If it is determined that the same agent image is to be displayed, the image control unit 152 determines that a single agent image is to be displayed on the display/operation device 20 (step S216). If it is determined in step S214 that different agent images are to be displayed, the image control unit 152 determines that a plurality of agent images are to be displayed on the display/operation device 20 (step S218). After the processing in step S216 or step S218, the image control unit 152 advances the processing to step S204. This concludes the explanation of the processing of this flowchart.

According to the agent device 100 of the embodiment described above, information can be provided to the user in a more appropriate output mode. Further, according to the present embodiment, an image in which the agent image is equipped with accessories can be displayed. This allows the user to virtually carry accessories such as amulets. Furthermore, this frees the user from the inconvenience of forgetting to carry accessories or leaving accessories behind in the vehicle.

Although the mode for implementing the present invention has been described above using embodiments, the present invention is not limited to these embodiments in any way, and various modifications and substitutions can be made without departing from the gist of the present invention. can be added.

10 Microphone 20 Display/operation device 30 Speaker unit 40 Navigation device 50 Vehicle equipment 60 In-vehicle communication device 70 General-purpose communication device 80 Occupant recognition device 100 Agent device 110 Management section 112 Sound processing section 114 Display control section 116 Setting section 116A Image setting section 116B Information setting section 118 Voice control section 150 Agent function section 152 Image control section 154 User recognition section 156 Agent related information setting section 160, 250 Storage section 158 Output control section 200 Agent server 220 Speech recognition section 222 Natural language processing section 224 Dialogue management section 226 Network search unit 228 Response sentence generation unit

Claims (12)

A display control device that displays a character image on an output device,
an image setting unit that accepts settings for causing the output device to display an accompanying image associated with the character together with the image of the character;
an information setting unit that accepts settings for incidental information to be output from the output device in association with displaying the incidental image together with the image of the character on the output device;
a recognition unit that recognizes a plurality of users who use the output device;
When the incidental information for the plurality of users recognized by the recognition unit includes a setting for displaying an ornament included in the incidental image, a state in which the character is equipped with the ornament is displayed on the output device. an image control unit for displaying an image of the image ;
The ornaments include religious tools or ornaments containing religious patterns or designs,
The image control unit determines whether or not to display the ornament depending on whether the types of ornaments of each of the plurality of users are the same.
Display control device. The information setting unit enables setting of the incidental information for each user,
The display control device according to claim 1. The information setting unit accepts the setting of the incidental information when the user recognized by the recognition unit has authority to display the incidental image.
The display control device according to claim 1 . further comprising an output control unit that causes the output device to output the incidental information when the user recognized by the recognition unit has authority to display the incidental image;
The display control device according to claim 1 . The output device is a device that can be mounted on a vehicle,
The information setting unit is configured to set the incidental information in association with the state of the vehicle.
The display control device according to any one of claims 1 to 4 . The output control unit is configured to control the output control unit when an image of the character equipped with an accessory is displayed on the output device, and a vehicle equipped with the output device is in a state associated with the setting of the incidental information. causing the output device to output the incidental information;
The display control device according to claim 4 . The information setting unit receives settings of information regarding safe driving by the user as the incidental information when the accessories include a talisman.
The display control device according to claim 1 . The image control unit is configured to cause the recognition unit to recognize a plurality of users, and to cause the image setting unit to display an image of the character equipped with an amulet on the output device for two or more users among the plurality of users. If it is set to display,
If the types of amulets of the respective users are the same, displaying an image of the character equipped with the amulet on the output device;
not displaying an image of the character equipped with the amulet on the output device when the types of amulets of the respective users are not the same;
The display control device according to claim 7 . The image control unit may set different characters for each of the plurality of users recognized by the recognition unit,
When the types of amulets of the respective users are the same, displaying an image of the character equipped with the amulet on the output device;
If the types of amulets of the respective users are not the same, and if at least one of the plurality of users has set to disallow the display of images of other people's ornaments, the output device displays the Do not display an image of the character equipped with the amulet,
The display control device according to claim 7 . The image control unit may be arranged such that when a plurality of users are recognized by the recognition unit and the output device displays an image of a single character,
When the types of amulets of the respective users are the same, displaying an image of the character equipped with the amulet on the output device;
not displaying an image of the character equipped with the amulet on the output device when the types of amulets of the respective users are not the same;
The display control device according to claim 7 . A display control device that displays a character image on an output device is
receiving a setting for causing the output device to display an accompanying image associated with the character together with the image of the character;
accepting settings for incidental information to be output from the output device in association with the incidental image being displayed on the output device together with the image of the character;
recognizing multiple users using the output device;
When the incidental information for the plurality of recognized users includes a setting for displaying accessories included in the incidental image, displaying an image of the character equipped with the ornaments on the output device. ,
The ornaments include religious tools or ornaments containing religious patterns or designs,
determining whether to display the ornaments depending on whether the types of ornaments of each of the plurality of users are the same;
Display control method. A display control device that displays a character image on an output device,
causing the output device to accept settings for displaying an accompanying image associated with the character together with the image of the character;
causing the output device to accept the setting of incidental information to be output from the output device in association with the display of the incidental image together with the image of the character ;
recognizing a plurality of users using the output device;
If the incidental information for the plurality of recognized users includes a setting for displaying an accessory included in the incidental image, displaying an image of the character equipped with the accessory on the output device. let me,
The ornaments include religious tools or ornaments containing religious patterns or designs,
determining whether to display the ornaments depending on whether the types of ornaments of each of the plurality of users are the same;
program.

Images