WO2022254667A1 - 表示制御装置及び表示制御方法 - Google Patents

表示制御装置及び表示制御方法 Download PDF

Info

Publication number
WO2022254667A1
WO2022254667A1 PCT/JP2021/021229 JP2021021229W WO2022254667A1 WO 2022254667 A1 WO2022254667 A1 WO 2022254667A1 JP 2021021229 W JP2021021229 W JP 2021021229W WO 2022254667 A1 WO2022254667 A1 WO 2022254667A1
Authority
WO
WIPO (PCT)
Prior art keywords
display
animation
control device
display control
screen
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/021229
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
了輔 天野
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nissan Motor Co Ltd
Original Assignee
Nissan Motor Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nissan Motor Co Ltd filed Critical Nissan Motor Co Ltd
Priority to PCT/JP2021/021229 priority Critical patent/WO2022254667A1/ja
Priority to CN202180004774.XA priority patent/CN115943358A/zh
Priority to EP21848123.2A priority patent/EP4354266A4/en
Priority to US17/629,830 priority patent/US11893297B2/en
Priority to JP2022504682A priority patent/JP7323050B2/ja
Publication of WO2022254667A1 publication Critical patent/WO2022254667A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
    • G06F3/147—Digital output to display device ; Cooperation and interconnection of the display device with other functional units using display panels
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
    • B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
    • B60K35/20—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor
    • B60K35/21—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor using visual output, e.g. blinking lights or matrix displays
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
    • B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
    • B60K35/20—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor
    • B60K35/28—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16—Sound input; Sound output
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16—Sound input; Sound output
    • G06F3/167—Audio in a user interface, e.g. using voice commands for navigating, audio feedback
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G06T13/80—Two-dimensional [2D] animation, e.g. using sprites
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
    • G09G5/14—Display of multiple viewports
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60H—ARRANGEMENTS OF HEATING, COOLING, VENTILATING OR OTHER AIR-TREATING DEVICES SPECIALLY ADAPTED FOR PASSENGER OR GOODS SPACES OF VEHICLES
    • B60H1/00—Heating, cooling or ventilating devices
    • B60H1/00642—Control systems or circuits; Control members or indication devices for heating, cooling or ventilating devices
    • B60H1/00985—Control systems or circuits characterised by display or indicating devices, e.g. voice simulators
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
    • B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
    • B60K2360/16—Type of output information
    • B60K2360/165—Videos and animations
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2320/00—Control of display operating conditions
    • G09G2320/10—Special adaptations of display systems for operation with variable images
    • G09G2320/103—Detection of image changes, e.g. determination of an index representative of the image change
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2340/00—Aspects of display data processing
    • G09G2340/04—Changes in size, position or resolution of an image
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2340/00—Aspects of display data processing
    • G09G2340/04—Changes in size, position or resolution of an image
    • G09G2340/0464—Positioning
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2354/00—Aspects of interface with display user
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
    • G09G2380/00—Specific applications
    • G09G2380/10—Automotive applications
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
    • G10L2015/225—Feedback of the input speech

Definitions

  • the present invention relates to a display control device and a display control method.
  • a known user interface device is capable of transitioning from the display of a transition source screen, which is a transition source screen, to the display of a transition destination screen, which is a transition destination screen, by animation (for example, Patent Document 1).
  • This user interface device calculates the necessity of saving the screen information of the transition source screen based on the transition information of the transition source screen, acquires the information of the transition source screen corresponding to one acquisition stage based on the necessity, The acquired information of the transition source screen is saved as screen information. Then, when the screen information of the transition destination screen in this transition is saved, the saved screen information of the transition source screen is acquired as the screen information of the transition destination screen in this transition.
  • the problem to be solved by the present invention is to provide a display control device and a display control method that produce natural animation before and after screen transitions.
  • the first index and the second index correspond to the plurality of first still images and the plurality of second still images so that the characteristic points of periodic motion correspond between the plurality of first still images and the plurality of second still images.
  • the first animation included in the first display screen and the second animation included in the second display screen which are respectively assigned to the second still images, continuously reproduce the plurality of first still images and the plurality of second still images, respectively.
  • animation can be displayed in a natural way before and after screen transitions.
  • FIG. 1 is a block diagram showing a dialogue service system according to one embodiment of the present invention.
  • 2 is a front view of a dashboard with the display of FIG. 1;
  • FIG. 3A is a table for explaining the relationship between the operation of the steering wheel switch, the wakeup word, and the activation switching of the dialogue system.
  • FIG. 3B is a table for explaining the relationship between the operation of the steering wheel switch, the wakeup word, and the activation switching of the dialogue system in the modified example.
  • 4 is a diagram showing a display screen of the display in FIG. 1.
  • FIG. 5 is a diagram showing a display screen of the display in FIG. 1.
  • FIG. 6 is a diagram showing a display screen of the display in FIG. 1.
  • FIG. 7 is a diagram showing a display screen of the display in FIG.
  • FIG. 8 is a diagram showing a display screen of the display in FIG. 1.
  • FIG. 9 is an animation image and a graph for explaining the temporal transition of the animation "voice detection state”.
  • FIG. 10 is an animation image and a graph for explaining the time transition of the animation.
  • 11A and 11B are conceptual diagrams for explaining the image structure of the animation and indexes selected at the time of screen transition.
  • 12 is a diagram showing a display screen of the display in FIG. 1.
  • FIG. 13 is a diagram showing a display screen of the display in FIG. 1.
  • FIG. 14 is a diagram showing a display screen of the display in FIG. 1.
  • FIG. 15 is a diagram showing a display screen of the display in FIG. 1.
  • FIG. 16 is a diagram showing a display screen of the display in FIG. 1.
  • FIG. 17 is a diagram showing a display screen of the display in FIG. 1.
  • FIG. 18 is a diagram showing a display screen of the display in FIG. 1.
  • FIG. 19 is a table showing a model flow of dialogue in the dialogue system of FIG.
  • FIG. 20 is a table showing a model flow of dialogue in the dialogue system of FIG. 21 is a diagram showing a display screen of the display in FIG. 1.
  • FIG. 22 is a table showing a model flow of dialogue in the dialogue system of FIG.
  • FIG. 23 is a table showing a model flow of dialogue in the dialogue system of FIG.
  • FIG. 24 is a table showing a model flow of dialogue in the dialogue system of FIG.
  • FIG. 25 is a table showing a model flow of dialogue in the dialogue system of FIG.
  • FIG. 1 is a block diagram showing a dialogue service system according to one embodiment of the present invention.
  • the interactive service system is a system mounted on a vehicle, and includes a display 10, a communication device 20, a handle switch 30, an ECU 40, and a controller 100.
  • the dialogue service system according to the present embodiment includes a system for providing services to the crew by interacting with the crew using a voice processing function, and a system for controlling the display screen of the display.
  • the display 10 is a touch panel display and is mounted on the vehicle.
  • the touch panel type display 10 includes a pressure-sensitive sensor, and detects a touch operation by a user (occupant) by measuring the pressure applied to the display 10 from a resistance value, voltage, or the like during a touch operation.
  • the mechanism of the pressure sensor is not particularly limited, and any method known at the time of filing can be used as appropriate.
  • the touch panel is not limited to the pressure-sensitive type, and may be of another type such as an electrostatic type.
  • FIG. 2 is a front view of the vicinity of a dashboard equipped with a touch panel. As shown in FIG. 2, the display 10 is a display (center display) in the front part of the dashboard, and is arranged between the driver's seat and the passenger's seat.
  • the display 10 is provided at a position that can be touched by an occupant sitting on the driver's seat and by an occupant sitting on the front passenger seat. Moreover, the display 10 does not necessarily have to be placed at a position where both the driver and the front passenger can touch it, and for example, it may be placed at a position where only the driver can touch it. Further, the display 10 is not limited to the dashboard, and may be arranged, for example, at a position touched by an occupant in the rear seat.
  • a touch operation is a pointing operation performed by a gesture using a finger, such as touching an icon or button displayed on the display 10 with the finger or bringing the finger close to the icon or button.
  • Touch operations include tap (touching the screen once), double tap (touching the screen twice), long tap (pressing and holding the screen for a long time), and swipe (swiping the finger touching the screen as it is on the screen ( tracing a finger)), flick (moving a finger that touches the screen quickly on the screen), pinch-in/pinch-out (touching the screen with two fingers and bringing them closer/farther apart), etc. is.
  • the touch operation is not limited to the first gesture of touching one or more points on the screen with the finger, but may include the second gesture of moving the touch point of the finger while touching the screen.
  • the touch operation does not have to directly touch the display screen, and may be so-called hovering, in which a finger or the like is brought closer to the display screen. Note that the operation method of the touch operation is not limited to these, and other methods may be used.
  • the communication device 20 is a communication device capable of communicating with the server 2 outside the vehicle.
  • the communication device 20 transmits and receives signals between the controller 100 and the server 2 .
  • a steering wheel switch (steering switch) 30 is a switch for starting up (activating) the dialogue system, and is provided on the steering wheel as shown in FIG.
  • a passenger (driver) operates the handle switch 30 to activate the dialogue system when he/she wants to receive the service provided by the dialogue system.
  • the dialogue system mainly uses a speech recognition engine provided in the vehicle 1 to perform speech processing and provides services, and a speech recognition engine included in the server 2. It includes a second dialogue system 32 which is used to perform voice processing and to provide services from the server 2 to the occupants.
  • the handle switch 30 is assigned to activate one of the dialogue systems, and the operation of the handle switch 30 activates either the first dialogue system or the second dialogue system.
  • the ECU 40 is a control unit that controls objects to be controlled such as the air conditioner 41 and the sunroof 42, and controls the objects to be controlled and the controller 100 via a CAN communication network.
  • the number of ECUs 40 is not limited to one, but is a large number of control units, and the ECU 40 is connected to each control object.
  • the controller 100 is configured by a computer including hardware and software, and has a memory storing programs and a CPU etc. for executing the programs stored in this memory. Further, the controller 100 has a dialogue system control unit 110, an activation unit 120, and a display control unit 130 as functional blocks. A program for implementation is stored in the memory. A computer included in the controller 100 executes a program to realize each function of the functional blocks. Also, the controller 100 is connected to the display 10, the communication device 20, the handle switch 30, and the ECU 40 via a CAN communication network or the like. The controller 100 is not limited to the dialog system control unit 110, the activation unit 120, and the display control unit 130, but also has control blocks for controlling various systems in the vehicle interior, such as functions for controlling the audio system. Control other in-vehicle equipment. A device having the controller 100 corresponds to the "display control device" of the present invention.
  • the dialogue system control unit 110 has a function of recognizing the voice of the passenger and controls the dialogue system.
  • the dialogue system control section 110 has a speech recognition section 111 and a first dialogue system 112 .
  • the voice recognition unit 111 detects the voice input of the passenger, and uses the voice recognition engine included in the first dialogue system 112 to recognize the voice by performing voice processing on the detected voice.
  • the speech recognition unit 11 detects the input speech. Voice detection is performed from the start of voice input until a predetermined time elapses after the end of voice input.
  • the voice recognition unit 11 detects the voice from the beginning to the end of the message and recognizes the message. After a silent period of a predetermined time from the end, voice detection is terminated.
  • the speech recognition unit 11 recognizes the detected speech by performing speech processing using a speech recognition engine.
  • the speech recognition unit 111 when it recognizes a wakeup word for activating the dialogue system, it outputs a control signal indicating recognition of the wakeup word to the second interface 122, which will be described later.
  • the wakeup word is a pre-determined calling message such as "Hello XX (name of car manufacturer, compatibility with dialogue system, etc.)". Note that the wakeup word may be changed according to the setting of the crew member.
  • the first dialogue system 112 is an in-vehicle voice processing system and is mainly intended for cooperation with in-vehicle equipment. provide services to the crew.
  • the first dialogue system 112 is connected to the ECU 40 in order to cooperate with in-vehicle equipment such as the air conditioner 41, the sunroof 42, the navigation system, the audio system, and the hands-free system.
  • the first dialogue system 112 has a communication function for communicating with various ECUs. For example, when an occupant speaks to the dialogue system to change the temperature setting of the air conditioner 41, the voice recognition unit 111 performs voice processing, and based on the voice data recognized by the voice recognition unit 111, the object to be controlled is determined.
  • the air conditioner 41 After specifying the air conditioner 41 to be, a control command is generated, and the control command is output to the ECU 40 through the in-vehicle communication path (CAN communication network).
  • the in-vehicle communication path through which the control signal passes includes the communication path connecting between the controller 100 and the communication device 20.
  • the ECU 40 controls the air conditioner 41 according to the control command and changes the set temperature of the air conditioner 41 .
  • the first dialogue system 112 provides services to the passengers through dialogue.
  • the service provided by the first dialogue system 112 is not limited to the setting of the air conditioner 41, and may be adjustment of the wind direction, air volume, etc. of the air conditioner 41.
  • FIG. Examples of other services include navigation system route guidance, audio system settings, hands-free system phone operation and phone number search, short message service (SMS), opening and closing of sunroof 42, windows, back doors, sliding doors, etc. It is the operation of in-vehicle equipment such as operation.
  • SMS short message service
  • the second dialogue system 32 included in the server 2 is a system that performs speech processing using an external speech recognition engine.
  • the services provided by the second interactive system 32 are a wide variety of services such as shopping using the Internet and remote control of home electrical appliances.
  • the second dialogue system 32 is not intended to cooperate with in-vehicle equipment.
  • the first dialogue system and the second dialogue system are separate dialogue systems, and the first dialogue system transmits control commands to in-vehicle equipment through an in-vehicle communication path.
  • the second dialogue system has a speech recognition engine different from the speech recognition engine included in the first dialogue system.
  • the difference between the first dialogue system 112 and the second dialogue system 32 is that the first dialogue system 112 inputs or outputs a control command to the in-vehicle equipment through the in-vehicle communication path, whereas the second dialogue system 32 Alternatively, the system may be such that the control command is not input to or output from on-vehicle equipment other than the communication device 20 through the in-vehicle communication path.
  • the difference between the first dialogue system 112 and the second dialogue system 32 is that the first dialogue system communicates through an in-vehicle communication path other than the communication path connecting between the communication device 20 and the first dialogue system 112.
  • a system that outputs the control command to the vehicle-mounted device and does not output the control command through an in-vehicle communication path other than the communication path connecting the communication device 20 and the first dialogue system 112 may be used.
  • the speech recognition unit 111 detects the passenger's speech input and transmits the detected speech data to the server 2 via the communication device 20 .
  • the second dialog system 32 of the server 2 performs voice processing on the input voice data and transmits the processing result to the vehicle 1 .
  • the voice recognition unit 111 outputs the result of processing by the second dialogue system 32 through voice output from the in-vehicle speaker or the display screen of the display 10 .
  • the first dialogue system 112 may be connected not only to the ECU 40 inside the vehicle, but also to the server 2 outside the vehicle.
  • the first dialogue system 112 may connect to the server 2 outside the vehicle and perform speech recognition processing using a speech recognition engine outside the vehicle.
  • a speech recognition engine outside the vehicle.
  • the first dialogue system 112 performs speech recognition processing using a speech recognition engine outside the vehicle.
  • the activation unit 120 is a functional block for activating the first dialogue system 112 and the second dialogue system 32 and has a first interface 121 and a second interface 122 .
  • the first interface 121 transmits a start command to the dialogue system by operating the handle switch 30. 32 are connected to each other.
  • the first interface 121 transmits an activation command to either one of the first dialogue system 112 and the second dialogue system 32 .
  • the first interface 121 never sends control commands to both the first dialogue system 112 and the second dialogue system 32 at the same time.
  • the destination of the control command by the first interface 121 is determined by user settings and/or factory default settings.
  • the first interface 121 transmits an activation command to the first dialogue system 112 when receiving an operation command by operating the handle switch 30. do. If the handle switch 30 is assigned to activate the second dialogue system 32 , the second interface 122 sends an activation command to the second dialogue system 32 .
  • the activation command for the second dialogue system 32 is a command for enabling communication between the vehicle 1 and the server 2 and enabling the vehicle 1 to use the dialogue system of the server 2. .
  • the second interface 122 sends an activation command to the dialogue system by means of a wakeup word.
  • a second interface 122 is a connection between the speech recognizer 111 and the dialog system.
  • the second interface 122 sends an activation command to either one of the first dialogue system 112 and the second dialogue system 32 .
  • the first interface 121 never sends control commands to both the first dialogue system 112 and the second dialogue system 32 at the same time.
  • the second interface 122 sends an activation command to the first interactive system 112 .
  • the transmission destination of the control command by the second interface 122 is determined by factory default settings.
  • the wakeup word is assigned to the activation of the first dialogue system 112.
  • the speech recognition unit 111 recognizes the wakeup word
  • the second interface 122 recognizes the wakeup word of the first dialogue system 112. Send a start command to In other words, the speech recognition engine included in the first dialogue system 112 is on standby in a state where at least the wakeup word can be recognized. A wake-up command is sent to the first dialog system 112 and the first dialog system 112 wakes up the speech recognition engine so that various words other than the wake-up word can be recognized. It should be noted that the second interface 122 sends an activation command to the second dialogue system 32 when the wakeup word is assigned to the activation of the second dialogue system 32 .
  • the activation unit 120 sets the transmission destination of the activation command to either one of the first dialogue system 112 and the second dialogue system 32 based on the operation of the passenger.
  • the occupant's operation for setting the transmission destination of the activation command includes a touch operation on the display 10, a switch operation other than the handle switch 30, and the like.
  • the controller 100 displays a setting screen of the dialogue system on the display 10, and the passenger selects the first dialogue system 112 and the second dialogue system 32 from the displayed setting screen. Touch the selection menu for selecting one of the systems, and select the interactive system you want to use on the selection menu.
  • the activation unit 120 sets the transmission destination of the activation signal of the first interface 121 in order to transmit the activation signal to the dialogue system selected by the touch operation on the display 10 .
  • the activation unit 120 changes the first interface 121 is set to the second interactive system 32.
  • the passenger can select the dialogue system to be activated when the handlebar switch 30 is operated from the first dialogue system 112 and the second dialogue system 32 according to his/her preference.
  • the activation unit 120 activates the first interface 121 based on the voice recognized by the dialogue system control unit 110.
  • Either one of the first dialogue system 112 and the second dialogue system 32 may be set as the transmission destination of the command.
  • the passenger While the passenger is having a conversation with the controller 100 through the first dialogue system 112, the passenger wishes to change the assignment of the handlebar switch 30 from the first dialogue system 112 to the second dialogue system 32.
  • the dialog system control unit 110 recognizes the voice of the passenger and transmits a control command to the activation unit 120 to allocate the steering wheel switch 30 to activate the second dialog system 32 .
  • the activation unit 120 sets the transmission destination of the activation signal from the first interface 121 to the second dialogue system 32 based on the control command.
  • the passenger can select the first dialogue system 112 or the second dialogue system 32 as the target system to be activated when the handle switch 30 is operated through dialogue using the system.
  • the dialogue system that can be activated by operating the handle switch 30 is configured such that the passenger can select either the first dialogue system 112 or the second dialogue system 32, but it is activated by the wake-up word.
  • the available dialogue system is fixed to the first dialogue system 112 .
  • the first dialogue system 112 is mainly intended for cooperation with in-vehicle equipment, so it is used more frequently than the second dialogue system 32 . Therefore, by fixing the activation by the wakeup word to the first dialogue system 112, it is possible to improve the convenience for the passengers.
  • the allocation of the handlebar switches 30 can be selected by the passenger, it is possible to ensure the degree of freedom in selection of the dialogue system.
  • FIG. 3A is a table for explaining whether or not activation of the first/second dialogue system can be switched.
  • the "wakeup word" can activate the first dialog system 112 and cannot activate the second dialog system 32.
  • FIG. The "handle switch” can activate either the first dialogue system 112 or the second dialogue system 32 depending on the setting of the passenger. That is, activation of the dialogue system by the handle switch 30 can be selected exclusively.
  • the handle switch 30 is assigned to the first dialogue system 112
  • the second dialogue system 32 cannot be activated by the operation of the handle switch 30 and the wakeup word. It can be activated by selection on the menu screen displayed on 10 or by operating a switch other than the handle switch 30 .
  • FIG. 3B is a table for explaining the correlation with activation of the first/second dialogue systems in the dialogue service system according to the modified example of this embodiment.
  • the dialogue system that can be activated by the wakeup word can be exclusively selected by user setting.
  • the activation unit 120 sets one of the first dialogue system 112 and the second dialogue system 32 as the transmission destination of the activation command when the wakeup word is recognized.
  • the passenger can select the dialogue system to be activated from the first dialogue system 112 and the second dialogue system 32 by touching the display 10, operating switches other than the handle switch 30, and interacting with the controller 100 using the dialogue system. This avoids a situation in which a plurality of dialogue systems are activated when the wake-up word is issued. It can be selected from system 32 .
  • the dialogue system may have its own standard regarding system startup.
  • the original standard states, "When a specific dialogue system can be activated, other dialogue systems must be turned off (disabled)."
  • a dialogue system having such a standard is used as the first dialogue system 112
  • the activation of the dialogue system by the wakeup word can be changed by the user setting
  • the allocation of the activation by the wakeup word can be changed by the user setting as in the modified example, or the dialogue system can be exclusively can be selected. That is, according to the standard of the first dialogue system, the first dialogue system 112 and the second dialogue system 32 must be activated exclusively.
  • the degree of freedom in selection of the dialog system is maintained while complying with the standard of the dialog system. can also be ensured.
  • the interaction system control unit 110 may audibly output the changes to the passenger.
  • the display control unit 130 controls the display screen of the display 10. Further, when the display detects a touch operation, the display control unit 130 accepts an operation command according to the position of the touch operation on the display screen and the type of the touch operation.
  • the display control unit 130 causes the display 10 to display an operation screen for operating the dialogue system, a setting screen for the dialogue system, a screen for displaying information provided by the dialogue system, and the like.
  • the display control unit 130 outputs an operation command received by a touch operation on the display 10 to the dialogue system control unit 110 and the activation unit 120 .
  • the control of the dialog system by the dialog system control unit 110 and the control of the display screen by the display control unit 130 are associated by an application (software).
  • the operating state of the dialogue system controlled by the dialogue system control section 110 is displayed on the display 10 .
  • the display control unit 130 controls the display screen of the display 10 according to the dialogue by the dialogue system.
  • the dialogue system control unit 110 transmits a control command according to the contents of the dialogue to the display control unit 130, and the display control unit 130 responds to the control command. transition the display screen of
  • FIG. 4 is a display screen of the display 10 and shows the top screen of the entire system including the interactive service system.
  • the display control unit 130 displays the top screen on the display 10.
  • the top screen is a menu screen for selecting the navigation system, music, first dialogue system 112 and second dialogue system 32 .
  • Each menu screen contains icons representing contents, for example, the menu image of the first dialogue system 112 contains the first icon 101 and the menu image of the second dialogue system 32 contains the second icon 102 .
  • the first icon 101 is an icon that validates the touch operation for activating the first dialogue system 112
  • the second icon 102 is an icon that validates the touch operation for activating the second dialogue system 32. be.
  • the activation unit 120 transmits an activation signal to the first dialogue system 112 .
  • the activation unit 120 transmits an activation signal to the second dialogue system 32 .
  • the first dialogue system 112 and the second dialogue system 32 can be exclusively selected by touch operation on the display 10 .
  • the operation state of the first dialogue system 112 includes four states: "initial state”, “speech acceptable state”, “speech detection state”, and "speech processing state”.
  • the initial state is a state from when the voice recognition engine is activated based on the activation signal until it becomes ready to detect voice.
  • the speech recognition unit 111 performs initialization processing before speech processing.
  • the operation state of the first dialogue system 112 changes from the initial state to the voice reception enabled state.
  • the voice-acceptable state is a state in which voice can be accepted, and indicates a state in which the passenger does not speak.
  • the voice detection state is a state in which voice can be received, and is a state in which the passenger speaks and the voice is detected.
  • the operating state of the first dialog system 112 is the voice input acceptance state, and when voice is input, the first dialog system 112 is activated.
  • the operating state is the voice detection state.
  • the speech processing state is a state in which speech processing is being performed by the speech recognition engine.
  • the display screens in FIGS. 5 to 8 represent the main images in the service through the dialogue system, and include operating state images showing the operating state of the dialogue system.
  • the operation state image is an image representing what state (status) the dialogue system is in for the passenger, and corresponds to text, animation, patterns, and the like.
  • the main message 50 and/or the animation 70 correspond to the operating state image.
  • FIG. 5 shows a display screen displayed on the display 10 when the operating state of the first interactive system 112 is "initial state".
  • the “initial state” display screen includes a main message 50 , a sample message 60 and an animation 70 .
  • the main message 50 is a message directed from the first dialogue system 112 to the passenger, and is a notation prompting the passenger to input by voice. By confirming the main message 50, the passenger can confirm that the dialogue system has been activated. That is, the main message 50 corresponds to an operating state image indicating that the interactive system has been activated.
  • the sample message 60 represents an example of speech to be uttered by the passenger in order to use the first dialogue system 112 and includes commands 61 and setting values 62 .
  • the command 61 is a word representing the operation or state of the service that the passenger wants to receive through the dialogue system, and represents an operation command from the passenger to the dialogue system.
  • the set value 62 is a word or phrase representing a service target, and is represented by a noun such as a numerical value or a name. For example, in the message "Play station AAA Radio" on the second line of the sample message 60, "Play” indicates the command and "station AAA Radio” indicates the set value. The command “Play” indicates to operate the audio system, and "station AAA Radio” indicates the target (name of the radio station) to be played back on the audio system.
  • the sample message 60 also serves as a function button. For example, when the passenger touches the message "Enter Number" on the third row, the screen transitions to a screen for entering a telephone number.
  • the animation 70 is an image that expresses the status of a given function with periodic movements.
  • the predetermined function corresponds to the speech detection processing function in the dialogue system
  • the status corresponds to the operating state of the dialogue system.
  • Cyclic motion is represented by waveform motion.
  • animation 70 there are four animations 71 - 74 to represent the four operating states of first dialog system 112 .
  • the four animations 71 to 74 represent the progress of speech processing by the first dialogue system 112 .
  • the animation 70 is distinguished for each operation state by the shape of the wave, the color of the wave, the size of the wave, etc., so that the crew members who watch the animation can see it.
  • Animation 71 in FIG. 5 represents the "initial state”.
  • the animation 71 is a gray waveform with a smaller amplitude.
  • Animation 70 is displayed at a position between main message 50 and sample message 60 .
  • the operating state image indicating the operating state of the interactive system may include at least one image of the main message 50 and the animation 70 .
  • the operating state image may include at least one of the main message 50 and the animation 70.
  • FIG. 6 shows a display screen displayed on the display 10 when the operating state of the first dialogue system 112 is "speech acceptable state".
  • the display screen of the “audio reception enabled state” includes a main message 50, a sample message 60, and an animation 70.
  • the display positions and display images of the main message 50 and the sample message 60 are the same as in FIG.
  • An animation 72 in FIG. 6 represents a "speech acceptable state”.
  • Animation 72 is the blue waveform, and the amplitude of animation 72 is greater than the amplitude of the waves of animation 71 .
  • FIG. 7 shows a display screen displayed on the display 10 when the operating state of the first dialogue system 112 is "voice detection state".
  • the “voice detection state” display screen includes a main message 50 , a sample message 60 and an animation 70 .
  • the display positions and display images of the main message 50 and the sample message 60 are the same as in FIGS.
  • An animation 73 in FIG. 7 represents the "voice detection state”.
  • Animation 73 is a blue waveform, and the amplitude of waves in animation 73 is greater than the amplitude of waves in animation 72 .
  • the display control unit 130 may change the amplitude of the animation 73 according to the volume of the detected sound.
  • the display control unit 130 controls the amplitude of the animation 73 according to the voice input so that the amplitude of the animation 73 increases as the volume of the detected voice increases.
  • the passenger can confirm from the display screen that the voice is detected by the voice recognition system by watching the amplitude movement of the animation 73 while speaking.
  • the animation 73 may be an animation in which a plurality of waves are superimposed to give a sense of dynamism to the movement.
  • FIG. 8 shows a display screen displayed on the display 10 when the operating state of the first dialogue system 112 is "speech processing state".
  • the “speech processing status” display screen includes a main message 50, a sample message 60, and an animation 70.
  • FIG. The display positions and display images of the main message 50 and the sample message 60 are the same as in FIGS.
  • Animation 74 in FIG. 8 represents the "sound processing state".
  • Animation 74 is a red, white, and green waveform, separating the waves into areas by color. For example, at one time animation 74 is displayed in the order red, white, green, from left to right on the page of FIG. Then, the amplitude of the wave and the color-coded parts change with the passage of time.
  • the amplitude of animation 74 is smaller than the amplitude of the waves of animation 73 .
  • the crew can confirm on the display screen that voice processing has started by seeing that the color of the wave has changed from "blue” to "red, white, and green.”
  • the animation 74 may be an animation in which a plurality of waves are superimposed to give a sense of dynamism to the movement.
  • the animations 71 to 73 may be made by superimposing a plurality of waves.
  • the number of colors given to animation 74 is three, which is greater than the number of colors given to animations 71-73.
  • the passenger can see from the display form of the animation 70 that the operation state of the first dialogue system 112 is the "speech processing state" and that the operation state of the first dialogue system 112 is in a state other than the "speech processing state". can be identified more clearly.
  • four types of animations 40 are used in accordance with the operation state of the audio processing function, but the number of types of animations is not limited to four types, and may be two, three, or five or more types.
  • the movement displayed by the animation is not limited to waves, and may be, for example, a change in pattern, a change in the height of a bar graph, or the like, as long as it has at least periodicity.
  • FIG. 9 is a graph for explaining the time transition of the animation 73 representing the "voice detection state”.
  • Animation 73 changes the shape of the wave in one cycle (2 ⁇ ).
  • the wave shown in animation 73 contains periodic motion features.
  • a feature point is represented by a wave antinode/node position, a wave crest position, a color change point, or the like.
  • a feature point is a point that serves as an index for matching with other animations 71 , 72 , and 74 .
  • (a) represents the animation 73 at time (0)
  • (c), and (d) represent the animation 73 at times ( ⁇ /2, ⁇ , 3 ⁇ /2).
  • the left side of FIG. 9 is an image of waves displayed on the display screen.
  • the movement of the feature points may be represented by, for example, a wave (periodic function) representing changes in wave shape, color, color, coordinates, etc., and by superimposing the waves to represent a standing wave.
  • the feature points move in the same cycle as the animation 73 periodically moves.
  • the change in the feature point does not necessarily have to be represented by movement or color that can be visually confirmed by the occupant, and may be calculated as a point.
  • FIG. 10 is a graph for explaining the relationship between the still images of the animations 71 to 74 and the feature points in each of the animations 71 to 74 at a certain point in the cycle.
  • (a) represents a still image of animation 71
  • (b)-(d) represent still images of animations 72-74.
  • (e) is a graph of feature points in animations 71-74.
  • the animations 71 to 74 are at the same time in the cycle, the shape of the wave, the color of the wave, etc. are different according to each animation.
  • the animations 71-74 have the same period.
  • the waveform of animation 71 and the waveform of animation 72 at time ( ⁇ /2) have different colors and different shapes.
  • the feature points of the periodic motions of the animations 71 to 74 are represented by the same waveform if they are at the same time in the cycle.
  • FIG. 11 is a conceptual diagram for explaining the image configuration for displaying the animations 71 to 74 and the screen transition of the display 10.
  • the first and second display screens are display screens displayed according to each operation state of the first dialogue system 112.
  • the first display screen indicates the screen before transition
  • the second display screen indicates the screen after transition.
  • the operating state of the first dialogue system 112 switches from the "initial state” to the "speech acceptable state” and the display screen of the display 10 changes
  • the first display screen changes to the "initial state” display screen.
  • the second display screen corresponds to the display screen of the "voice reception possible state".
  • Animations 71 to 74 are displayed by continuously reproducing multiple still images.
  • the images displaying the animations 71 to 74 are composed of 60 still images, each of which is assigned an index from 0 to 59.
  • FIG. The number of still images is the same for each of the animations 71-74.
  • the index is a still image identifier and also represents the playback order. Indexes are given to the still images so that feature points of periodic motion correspond among the images included in the animations 71-74.
  • the still images of the animations 71 to 74 are given the same index because they represent images at the same time in the cycle. In other words, when still images are compared between the animations 71 to 74, the same index is given to the still images with corresponding feature points.
  • the display control unit 130 When displaying the animations 71 to 74, the display control unit 130 sends the images to the display 10 at a reproduction rate of 30 still images per second. That is, the time slot (display interval) of one still image is 20 msec. The display control unit 130 starts reproducing the still images in the order in which the indexes were assigned. Thus, in this embodiment, the animations 71 to 74 are displayed by reproducing a plurality of still images, so the processing load on the CPU can be reduced. Further, when executing the screen transition from the first display screen to the second display screen, the display control unit 130 displays up to the pre-transition image, which is a still image immediately before the screen transition, and displays the pre-transition image. Select a second index corresponding to the assigned first index. Then, the display control unit 130 starts playing the second still image from the selected second index.
  • the pre-transition image which is a still image immediately before the screen transition
  • the display control unit 130 starts image reproduction from the still image assigned the index "#0" in order to display the "initial state" of the first interactive system 112 in the animation 71. Assume that the operating state of the first dialogue system 112 transitions from the "initial state” to the "speech acceptable state” while the still image assigned the index "#9” is being displayed.
  • the display control unit 130 selects, from among the 60 still images for displaying the animation 72, the still images assigned the same index of “#9” as the index assigned to the still images before the transition.
  • Display control unit 130 displays the pre-transition still image on display 10 until the time slot for the pre-transition still image has passed, and when the time slot for the pre-transition still image has passed, the selected index "# 9”, the image reproduction for displaying the animation 72 is started. That is, when the screen transition is executed, the display control unit 130 maintains continuity between the characteristic points of the periodic movement displayed in the animation 71 and the characteristic points of the periodic movement displayed in the animation 72. to select an index and start playing still images from the selected index.
  • indexes are assigned so that the feature points of periodic motion correspond to the images included in each of the animations 71 to 74, and at the time of screen transitions, the indexes are taken over and still images are reproduced. .
  • the continuity of the animation display can be maintained even if the animation production method changes, so that the animation can be produced naturally before and after the screen transition.
  • the state in which the still image before transition is displayed on the display 10 is maintained, and when the time slot of the still image before transition has passed, the screen is switched to the second display screen. This makes it possible to maintain the continuity of the animation.
  • FIG. 12 represents the display screen of the display 10 after the first dialogue system 112 has recognized the speech regarding the POI.
  • the speech recognition unit 111 detects the speech "Coffee Shop” and executes speech recognition processing.
  • the dialogue system control unit 110 determines from the voice-inputted words related to POI (Point of Interest) that the passenger is looking for a "Coffee Shop" near the current location, and provides guidance related to the POI.
  • POI Point of Interest
  • a control command is output to the display control unit 130 to display the image 80 and the map 90 around the current location on the display 10 .
  • the guide image 80 includes a character string 81 representing the voice recognition result and POI information 82 .
  • a character string 81 is a speech character string recognized by the first dialogue system 112 . In the example of FIG. 12, "Coffee Shop" is displayed. This allows passengers to see what sounds are recognized by the system.
  • the POI information 82 indicates information about POIs such as the names of POIs near the current location and the distance from the current location.
  • the display screen of the display 10 changes to the screen of FIG. After finishing the speech recognition processing based on the speech "Coffee Shop", the first dialogue system 112 becomes ready to receive speech again.
  • an animation 72 representing "speech acceptable state” is displayed at the position where the character string 81 was displayed.
  • the passenger can confirm that the voice following "Coffee Shop” should be uttered by seeing the display of the animation 72 at the position where the character string 81 of "Coffee Shop” is displayed.
  • the first dialogue system 112 detects the voice of the passenger.
  • the display of the animation 70 becomes an animation 73 representing "sound detection state" along with the sound detection.
  • the display of the animation 70 becomes an animation 74 representing "voice processing state" as shown in FIG.
  • the passenger can confirm the processing status of the first interactive system on the display 10 while looking at the POI guidance information and map information.
  • the first dialogue system 112 outputs a control signal indicating that the speech recognition has failed to the display control section 130 .
  • the display control unit 130 displays a gray waveform animation 71 at the position of the animation 73 on the display screen in FIG. Thereby, the passenger can confirm that the voice recognition has failed by the display on the display 10 . If the voice recognition fails, a voice such as "Please input voice again" may be output to prompt the passenger to speak.
  • the display control unit 130 switches the display screen according to the status of the first dialogue system 112. That is, the display control unit 130 controls the screen including the image (corresponding to the "first information display image” of the present invention) displaying the main message 50 and the sample message as shown in FIGS.
  • the display 10 is caused to switch between a screen including an image (corresponding to the "second information display image” of the present invention) displaying information about POIs and map information as shown in FIG.
  • the main message 50 and the sample message correspond to the "first information" of the present invention, and the POI-related information and the map information correspond to the "second information" of the present invention.
  • Each display screen contains an image of the animation 70 . Comparing the pre-transition screen (corresponding to the display screens shown in FIGS.
  • the post-transition screen (corresponding to the display screens shown in FIGS. 12 to 15) reveals that the animation 70 included in the pre-transition image is
  • the display of the animation 70 included in the post-transition image differs in terms of position and size. Accordingly, information presentation using the first dialogue system 112 and display of the operating state of the first dialogue system 112 can be made compatible.
  • the case where the voice processing is not normally completed is the case where a predetermined time has passed in the "speech acceptable state" state. For example, if the occupant does not speak after the display screen of the display 10 changes to the screen shown in FIG. 6, the voice recognition unit 111 does not detect voice. If such a state continues for a predetermined period of time, the speech recognition unit 111 determines that the speech processing is not normally completed. In addition, the case where the audio processing is not completed normally includes the case where an abnormality occurs in the audio processing. The speech recognition unit 111 performs speech recognition processing on the detected speech.
  • the speech recognition unit 111 determines that the speech processing is abnormal when the input speech cannot be analyzed or when there is no command 61 corresponding to the analyzed speech. Such an abnormality determination of audio processing also corresponds to the case where the audio processing is not completed normally.
  • the display 10 displays the main screen as shown in FIG. Then, when it is determined that the audio processing is not completed normally, the display screen of the display 10 becomes the screen shown in FIG. FIG. 16 shows the display screen of the display 10 when the voice processing is not normally completed from the "speech acceptable state".
  • the display control unit 130 displays a category screen as shown in FIG. 16 when the audio processing is not completed normally.
  • the category screen includes category images in which commands 61 are grouped.
  • a category image is obtained by grouping a plurality of commands and displaying the group name with text or an icon.
  • a category image includes a category name 201 and a category icon 202 .
  • a plurality of commands related to telephones such as "call and/or dial", “redial”, and "telephone” are grouped under one category "Phone”.
  • a category icon 202 is displayed next to the category name 201 .
  • Grouping of commands 61 is not limited to telephone, but includes navigation, audio, and the like, and the names and icons of the grouped categories are included in the category screen.
  • an image including the category name 201 and an icon 202 represent areas in which touch operations are enabled. That is, the image including the category name 201 and the icon 202 also serve as a button function for allowing the passenger to select the category. For example, when the passenger touches the image including the name 201 of the category "Phone" or the icon of the category "Phone", the display control unit 130 changes the display screen from the category screen to the list screen described later.
  • the category image displayed on the main screen may include at least one image of the name 201 and the icon 202 .
  • the category screen includes a main message 50 and an animation 70 in addition to the category name 201 and icon 202.
  • An animation 70 is an animation 72 representing a "voice detection state”. The speech recognition processing by the speech recognition unit 111 is effectively functioning even when the category screen is displayed, and the passenger can see from the display of the animation 72 that the operating state of the first dialogue system 112 is the "speech reception enabled state”. It can be confirmed that
  • the dialogue system can be advanced from the display of the category name 201 and the icon 202. Even if the voice cannot be processed normally due to noise or the like, the occupant can operate the dialogue system by displaying a category image (icon) that also functions as a button. In addition, category images that also serve as button functions display frequently used images, thus improving convenience. Furthermore, since the category screen includes the main message 50 and the animation 72, the crew can confirm from the screen display that the system can be advanced by interacting with the controller 100. FIG.
  • the selection of the category and/or list by the crew may be voice input or switch operation. That is, with the category screen displayed on the display 10, the controller 100 determines whether or not to select a category based on any one of the voice of the passenger, the touch operation of the display 10 by the passenger, and the switch operation by the passenger. Judge.
  • the display control unit 130 causes the display 10 to display a list screen as shown in FIG. FIG. 17 shows a list screen displayed on the display 10. As shown in FIG.
  • the list screen is an image containing grouped commands and contains a list 300 having a plurality of commands.
  • List 300 has a plurality of syntaxes 310 , each syntax 310 has at least commands 61 , and in addition to commands, sample text 311 .
  • the list screen does not contain animation 70 .
  • a syntax 310 is a text representation of a combination of a command 61 and a setting value 62 .
  • a sample text 311 represents arbitrary text of the setting value 62 following the command, and indicates either one of the character string and number string information assigned to the syntax 310 .
  • the syntax at the top of the list consists of the command "Call" and the sample text " ⁇ Name>".
  • ⁇ Name> indicates that a setting value 62 such as a person's name or store name is entered after the command. If the occupant touches the display of syntax 310 in the list, to the right of list 300 is displayed an example of setpoint 62 that goes into sample text 311 . Further, when the crew touches the display of the syntax 310 in the list by double tapping or the like, a setting list to enter the sample text 311 as shown in FIG. 18 is displayed. For example, in the display screen of FIG. 17, when the top syntax "Call ⁇ Name>" in the list is selected, either a character string or a number string assigned to the syntax 310 as shown in FIG. One information is displayed.
  • the crew can select a command 61 and a sample assigned to the syntax 310 by touching the number display on the list screen of FIG.
  • the controller 100 may or may call the touched number in the list.
  • a notification to that effect may be displayed on the display 10 .
  • FIG. 19 is a model flow of dialogue regarding the temperature setting of the air conditioner 41.
  • the controller 100 outputs a Japanese message "Please select whether to input the command by voice" with a mechanical sound.
  • the model flow of FIG. 19 for example, while the display screen of FIG. You can change the temperature.
  • Fig. 20 is a model flow of dialogue regarding POI search. For example, if the occupant issues the wakeup word and the voice is successfully processed, the controller 100 responds with the message "Speak command or select.” Next, the passenger issues a message "I want to go to a cafe near my destination" containing words that can identify POIs such as the genre of the shop. After recognizing this message, the controller 100 performs a search using the navigation system and responds with the message "Voice the item number from the displayed list or select another tab.” At this time, the display screen shown in FIG. 21 is displayed on the display 10 . The controller 100 causes the display 10 to display a guidance image 80 regarding the POI. Controller 100 may also display a map.
  • the guidance image 80 includes an animation 72 representing "speech acceptable state" and POI information 82 obtained from the search results.
  • An item number is assigned to the POI information.
  • a crew member can select a POI by issuing the item number.
  • the POI "AAA” is selected by the passenger saying “ichi”.
  • the controller 100 repeats back the voice “Ichi” uttered by the passenger, and then responds with the next message for selection such as "Would you like to change the destination?"
  • FIG. 22 is a model flow of dialogue regarding destination setting.
  • FIG. 23 is a model flow of dialogue regarding audio reproduction.
  • FIG. 24 is a model flow of interaction regarding SMS (Short Message) transmission.
  • FIG. 25 is a model flow of interaction regarding SMS (short message) transmission/reception.
  • the controller 100 causes the display 10 to display the first display screen including the first animation and the second display screen including the second animation. It is an animation that expresses the status of a function by periodic motion, and is displayed by continuously reproducing a plurality of first still images and a plurality of second still images.
  • the first index and the second index correspond to the plurality of first still images and the plurality of second still images so that the periodic motion feature points correspond between the plurality of first still images and the plurality of second still images. It is assigned to each still image.
  • the controller 100 displays the pre-transition image, which is the first still image immediately before the screen transition.
  • a second index corresponding to the assigned first index is selected, and a second still image is reproduced from the selected second index.
  • the first animation corresponds to the animation 70 included in the display screen (first display screen) before screen transition
  • the second animation corresponds to the animation 70 included in the display screen (second display screen) after screen transition.
  • the first still image corresponds to the image for reproducing the first animation
  • the second still image corresponds to the image for reproducing the second animation.
  • the display control method executed by the controller 100 reproduces the plurality of first still images in the order of the first index and displays the second still image in order to display the first animation on the first display screen.
  • the plurality of second still images are continuously reproduced in the order of the second index.
  • the controller displays the pre-transition image, which is the first still image immediately before the screen transition, and gives the pre-transition image.
  • a second index corresponding to the first index is selected, and a second still image is reproduced from the selected second index.
  • an image is displayed on the display 10 to suggest an operation to the passenger. That is, when the voice input is not detected and the voice processing is not normally completed, the category screen as shown in FIG. and icon 202 are displayed. As a result, even if the passenger does not understand the voice for operating the first dialogue system, the crew can proceed with the dialogue system.
  • the controller 100 may select the second index such that the difference in the position of the animation 70 or the difference in the shape of the animation 70 before and after the screen transition is less than a predetermined value. . As a result, it is possible to suppress animation deviation before and after the screen transition.
  • the controller 100 when the first display screen represents the state during voice detection and the second display screen represents the state during voice recognition, when executing the screen transition, A second index that does not correspond to the first index assigned to the pre-transition image may be selected.
  • the state of the audio processing changes, if the state change is represented by animation discontinuity, a second index that does not correspond to the first index is selected before and after the screen transition, and the selected second index is selected. Reproduction of still images may be started from the 2nd index.
  • the controller 100 may display the message images such as the main message 50 and the sample message 60 and the animation 70 in an overlapping manner.
  • the controller 100 when the sound volume is equal to or higher than a predetermined value, the controller 100 does not detect the input sound and changes the amplitude of the waveform animation 70 to the waveform when the sound is detected. May be larger than the animation amplitude. This allows us to confirm from the animation that the sound cannot be detected due to the volume of the sound.
  • the controller 100 may control the display screen of the display 10 so that the positions of the nodes of the waveform move on the display screen as time passes.
  • Vehicle 2 Server 20 Communication Device 30 Handle Switch 32 Second Dialogue System 70 Animation 100 Controller 110 Dialogue System Control Unit 111 Voice Recognition Unit 112 First Dialogue System 120 Startup Unit 121 First Interface 122 Second Interface 130 Display Control Unit

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Mechanical Engineering (AREA)
  • Computer Hardware Design (AREA)
  • General Health & Medical Sciences (AREA)
  • Chemical & Material Sciences (AREA)
  • Combustion & Propulsion (AREA)
  • Transportation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Acoustics & Sound (AREA)
  • Computational Linguistics (AREA)
  • Thermal Sciences (AREA)
  • User Interface Of Digital Computer (AREA)
  • Processing Or Creating Images (AREA)
  • Controls And Circuits For Display Device (AREA)
PCT/JP2021/021229 2021-06-03 2021-06-03 表示制御装置及び表示制御方法 Ceased WO2022254667A1 (ja)

Priority Applications (5)

Application Number Priority Date Filing Date Title
PCT/JP2021/021229 WO2022254667A1 (ja) 2021-06-03 2021-06-03 表示制御装置及び表示制御方法
CN202180004774.XA CN115943358A (zh) 2021-06-03 2021-06-03 显示控制装置和显示控制方法
EP21848123.2A EP4354266A4 (en) 2021-06-03 2021-06-03 DISPLAY CONTROL DEVICE AND DISPLAY CONTROL METHOD
US17/629,830 US11893297B2 (en) 2021-06-03 2021-06-03 Display control device and display control method
JP2022504682A JP7323050B2 (ja) 2021-06-03 2021-06-03 表示制御装置及び表示制御方法

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2021/021229 WO2022254667A1 (ja) 2021-06-03 2021-06-03 表示制御装置及び表示制御方法

Publications (1)

Publication Number Publication Date
WO2022254667A1 true WO2022254667A1 (ja) 2022-12-08

Family

ID=84324038

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/021229 Ceased WO2022254667A1 (ja) 2021-06-03 2021-06-03 表示制御装置及び表示制御方法

Country Status (5)

Country Link
US (1) US11893297B2 (https=)
EP (1) EP4354266A4 (https=)
JP (1) JP7323050B2 (https=)
CN (1) CN115943358A (https=)
WO (1) WO2022254667A1 (https=)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20240371372A1 (en) * 2021-06-03 2024-11-07 Nissan Motor Co., Ltd. Dialogue service device and dialogue system control method

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
USD1092553S1 (en) * 2023-06-16 2025-09-09 Thales Avs France Sas Portion of a display panel with an animated graphical user interface
USD1089252S1 (en) * 2023-11-30 2025-08-19 AIP Technology Corporation Display screen or portion thereof with a graphical user interface

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014178905A (ja) * 2013-03-14 2014-09-25 Ricoh Co Ltd 画像表示装置、画像表示方法およびプログラム
WO2017212758A1 (ja) 2016-06-08 2017-12-14 三菱電機株式会社 ユーザインタフェース装置
JP2018047076A (ja) * 2016-09-22 2018-03-29 株式会社三洋物産 遊技機
JP2020098216A (ja) * 2020-02-04 2020-06-25 株式会社ユピテル システム及びプログラム

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4253875B2 (ja) * 1998-09-30 2009-04-15 ソニー株式会社 送信方法および送信装置、受信方法および受信装置、並びに伝送方法および伝送システム
JP4213835B2 (ja) * 1999-12-28 2009-01-21 ローランド株式会社 波形再生装置
JP2005176925A (ja) * 2003-12-16 2005-07-07 Sankyo Kk 遊技機
JP2007172716A (ja) 2005-12-20 2007-07-05 Sony Corp 再生装置、再生方法および再生プログラム、記録媒体およびデータ構造、ならびに、オーサリング装置、オーサリング方法およびオーサリングプログラム
JP2008109601A (ja) * 2006-10-26 2008-05-08 Masashi Hayashi 動画映像音声情報処理システム
JP2009076982A (ja) * 2007-09-18 2009-04-09 Toshiba Corp 電子機器および顔画像表示方法
WO2014076791A1 (ja) * 2012-11-15 2014-05-22 三菱電機株式会社 ユーザインタフェース装置
US9721587B2 (en) * 2013-01-24 2017-08-01 Microsoft Technology Licensing, Llc Visual feedback for speech recognition system
CN106873863B (zh) * 2015-12-09 2021-02-19 斑马智行网络(香港)有限公司 一种界面切换动画的设置方法及装置
US10409552B1 (en) * 2016-09-19 2019-09-10 Amazon Technologies, Inc. Speech-based audio indicators
CN106873937A (zh) * 2017-02-16 2017-06-20 北京百度网讯科技有限公司 语音输入方法和装置
US11461649B2 (en) * 2020-03-19 2022-10-04 Adobe Inc. Searching for music

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014178905A (ja) * 2013-03-14 2014-09-25 Ricoh Co Ltd 画像表示装置、画像表示方法およびプログラム
WO2017212758A1 (ja) 2016-06-08 2017-12-14 三菱電機株式会社 ユーザインタフェース装置
JP2018047076A (ja) * 2016-09-22 2018-03-29 株式会社三洋物産 遊技機
JP2020098216A (ja) * 2020-02-04 2020-06-25 株式会社ユピテル システム及びプログラム

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4354266A4

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20240371372A1 (en) * 2021-06-03 2024-11-07 Nissan Motor Co., Ltd. Dialogue service device and dialogue system control method

Also Published As

Publication number Publication date
JP7323050B2 (ja) 2023-08-08
US20230359418A1 (en) 2023-11-09
EP4354266A4 (en) 2024-07-31
EP4354266A1 (en) 2024-04-17
JPWO2022254667A1 (https=) 2022-12-08
CN115943358A (zh) 2023-04-07
US11893297B2 (en) 2024-02-06

Similar Documents

Publication Publication Date Title
KR101647848B1 (ko) 정보 입력 및 표시를 위한 운전자 보조 시스템의 멀티모드 사용자 인터페이스
WO2022254669A1 (ja) 対話サービス装置及び対話システム制御方法
KR101613407B1 (ko) 지원 기능을 갖춘 차량 시스템 및 차량 시스템 작동 방법
JP7323050B2 (ja) 表示制御装置及び表示制御方法
CN106030697B (zh) 车载控制装置及车载控制方法
KR101570096B1 (ko) 핸들에 부착되는 입력 장치 및 이를 구비한 차량
CN103718153B (zh) 用户接口装置以及信息处理方法
US20140267035A1 (en) Multimodal User Interface Design
KR101549559B1 (ko) 핸들에 부착되는 입력 장치 및 이를 구비한 차량
CN106796786A (zh) 语音识别系统
WO2014196208A1 (ja) カーナビ用ジェスチャ入力装置
JP2025107443A (ja) 制御装置及び制御方法
US20220258606A1 (en) Method and operating system for detecting a user input for a device of a vehicle
JP7662847B2 (ja) インターフェース制御方法および装置、ならびにシステム
CN114040857B (zh) 用于运行车辆中的操作系统的方法和车辆中的操作系统
WO2018098049A1 (en) Controls for dictated text navigation
JP5795068B2 (ja) ユーザインタフェース装置、情報処理方法および情報処理プログラム
JP2005208798A (ja) 情報提供端末、および情報提供方法

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 2022504682

Country of ref document: JP

Kind code of ref document: A

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21848123

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2021848123

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2021848123

Country of ref document: EP

Effective date: 20240103