JP7233918B2 - In-vehicle equipment, communication system - Google Patents

Description

The present invention relates to an in-vehicle device and a communication system.

2. Description of the Related Art Conventionally, an in-vehicle device has a function of accepting a voice operation so that the user can operate the in-vehicle device while driving without having to operate an operation button or the like. For example, in response to the statement "convenience store nearby", the vehicle-mounted device retrieves and displays a convenience store near the current location or a convenience store on the route to the destination.

FIG. 1 is an example of a diagram for explaining facility search by voice operation. When a vehicle occupant says, for example, "search around", genres of facilities that can be searched by voice are displayed as shown in FIG. 1(a). For example, "Nearby family restaurant", "Nearby fast food", "Nearby convenience store", etc. are displayed. When the passenger of the vehicle says, for example, "convenience store nearby", a list of nearby convenience stores is retrieved and displayed as shown in FIG. 1(b). When the vehicle occupant says "Go here", the facility at the top of the list (convenience store in this case) is set as the destination, and when the driver says "Go to the middle", the facility in the middle of the list is set as the destination. Go to the bottom" is said, the facility at the bottom of the list is set as the destination, and a route to this facility is searched.

FIG. 2 is an example of a diagram illustrating various situations that may arise for a vehicle occupant with respect to voice-activated searches. The situation of the occupants of the running vehicle varies. Therefore, it is difficult to enter the store on the right side.” For these situations, the vehicle occupant may wish to add search elements such as "convenience store on the right", "shop on the left", or "only on the left".

As for such a technique, a technique has been devised for searching for a facility using words related to directions spoken by voice (see, for example, Patent Document 1). Patent Literature 1 discloses a navigation device that determines a search direction such as left, right, front, or back from a statement "What is the building on the left side?" and searches for facilities in the specified direction. there is

Japanese Patent Application Laid-Open No. 2001-133283

However, the conventional technology has the problem that commands for voice operation become long and complicated. For example, in the technique of Patent Literature 1, it is necessary to utter a voice specifying a search element such as a direction in addition to the voice for searching for a facility. Even when ``I want to search only the convenience store on the right side'', it is necessary to say ``nearby convenience store'' and then ``right side'', which may make the operation cumbersome.

In addition, when the user utters long or complicated words, the voice recognition rate drops, so there is also the problem that the in-vehicle device may not be able to recognize the voice or perform an incorrect search.

In this way, the conventional technology can only use relatively simple words, so even if it is possible to perform a search according to distance, such as "nearby convenience stores", it is not possible to add search elements such as directions. was difficult.

In addition, when controlling an in-vehicle device by voice operation, it is necessary to designate the in-vehicle device with a longer name as the types and number of the in-vehicle devices increase. If you want to switch to the image of the right or left side camera, you need to say "right side camera" and "left side camera", which may make you feel that the operation is troublesome.

SUMMARY OF THE INVENTION In view of the above problems, an object of the present invention is to provide an in-vehicle device capable of performing voice operation while suppressing lengthening and complicating commands for voice operation.

In view of the above problems, the present invention provides an in-vehicle device that accepts a voice operation of a vehicle occupant, comprising voice recognition means for recognizing the voice of the vehicle occupant, and line-of-sight direction recognition for recognizing the line-of-sight direction of the vehicle occupant. command detection means for detecting a command from the voice recognized by the voice recognition means; command processing means for processing the command according to the line-of-sight direction recognized by the line-of-sight direction recognition means; and displaying an electronic map. a display, wherein the command processing means searches for POIs existing within the display range of the electronic map displayed on the display when the line-of-sight direction is within the range of the display, and the command detection means detects the command related to the search from the voice recognized by the voice recognition means, and the command processing means adds the line-of-sight direction recognized by the line-of-sight direction recognition means to a search element and processes the command related to the search. It is characterized by

It is possible to provide an in-vehicle device capable of voice operation while suppressing commands for voice operation from becoming long and complicated.

It is an example of the figure explaining the search of the facility by voice operation. 1 is an example of a diagram illustrating various situations that may arise for a vehicle occupant with respect to voice-activated searches; FIG. It is a figure explaining an example of the search result of the facility displayed on the electronic map which an occupant's line-of-sight direction of a vehicle and an in-vehicle device displays. It is a figure explaining the structural example of an in-vehicle device. It is an example of the functional block diagram which shows the function which an in-vehicle apparatus has in block form. FIG. 10 is a diagram for explaining the line-of-sight direction determined to be the right side, the front side, the left side, or the display; FIG. 10 is an example of a diagram for explaining the timing of determination of the line-of-sight direction with respect to the utterance of a command; FIG. 2 is a diagram showing an example of an electronic map displayed on a display; FIG. It is an example of a diagram for explaining a method of determining whether a facility is on the right side or the left side with respect to a route or a road on which one is traveling. It is a figure which shows the example of a display of HUD. FIG. 10 is an example of a flowchart showing a procedure for an in-vehicle device to search for a facility using a line-of-sight direction as a search element; It is a figure explaining an example of the line-of-sight direction of the passenger|crew of a vehicle, and the detour route searched. FIG. 10 is an example of a flowchart showing a procedure for an in-vehicle device to search for a detour route using a line-of-sight direction as a search element; It is a figure which shows the example of arrangement|positioning of the camera mounted in a vehicle. It is an example of the functional block diagram which shows the function which an in-vehicle device has in block form (Example 3). FIG. 10 is an example of a flow chart diagram showing a procedure for switching an image displayed on a display by an in-vehicle device according to a line-of-sight direction.

Hereinafter, an in-vehicle device and a voice operation method performed by the in-vehicle device will be described as an example of embodiments of the present invention with reference to the drawings.

<Overview of search method of this embodiment>
First, with reference to FIG. 3, an outline of the search method of this embodiment will be described. FIG. 3 is a diagram illustrating an example of search results of facilities displayed on the electronic map 301 displayed by the vehicle occupant and the vehicle occupant's line of sight.

One of the features of the in-vehicle device of the present embodiment is that a line-of-sight direction is combined with a command input by voice. When searching for a facility by a command, a search factor related to a direction such as right, left, or front can be added to the facility search according to the line-of-sight direction.

For example, when a vehicle occupant says "a nearby convenience store" while looking at the right side 11, the in-vehicle device searches only the convenience store located on the right side of the road on which the vehicle is traveling or the route to the destination. By combining voice operation and line-of-sight direction, it is possible to perform a search for a facility or the like by combining a command-based search with a direction search element without saying a keyword such as "right".
- FIG. 3(a) is a diagram showing a search result when a vehicle occupant says "a nearby convenience store" while looking at the left side 13. FIG. On the electronic map 301, only convenience stores located on the left along the route are retrieved and displayed.
- FIG. 3(d) is a diagram showing a search result when a vehicle occupant says "a nearby convenience store" while looking at the right side 11. FIG. On the electronic map 301, only convenience stores on the right side along the route are retrieved and displayed.
- FIG. 3(c) is a diagram showing a search result when a vehicle occupant says "a nearby convenience store" while looking at the display 14 of the in-vehicle device. A convenience store in the electronic map being displayed on the display 14 is searched and displayed on the electronic map 301 .
- FIG. 3(b) is a diagram showing a search result when a vehicle occupant says "a nearby convenience store" while looking at the front 12. FIG. If the user speaks while looking at the front 12, it is determined that the occupant of the vehicle has no intention of adding a search element, and convenience stores on both sides of the route are searched and displayed on the electronic map 301.例文帳に追加In other words, a search similar to the conventional one is also possible.

As described above, the in-vehicle device of the present embodiment can recognize the line-of-sight direction, and perform a search by combining the direction in which the vehicle occupant is looking and the voice operation.

<Terms>
A vehicle occupant is a person who searches for a facility. For example, the driver may be a mere passenger. In addition, the case where the passenger does not drive at all is included in the mobile body that moves by automatic driving.

A vehicle is a moving object that is mainly driven by power. For example, in addition to automobiles, motorized two-wheeled vehicles (referred to as motorcycles) may be used. In addition, light vehicles such as bicycles that are moved by human power may be included. Wheelchairs may be included.

A command is an instruction to be input to the in-vehicle device. For example, a predetermined character string becomes a command. Therefore, it may also be referred to as an instruction, operation voice, text, or the like.

Retrieval refers to searching for necessary items from data. It goes without saying that searching for POIs (Points Of Interest) is included, but searching for routes is also included. The search range refers to the range of data to be searched. Changing the search range means changing the range of data to be searched. For example, change the range of POIs to be searched.

Processing a command means operating internally or externally so that the in-vehicle device provides a function according to the content of the command. Command processing may be called command execution.

A facility is a facility prepared for a certain purpose, but it is not necessary to have some kind of physical facility. A specific point (spot) on the map may also be called a POI.

<Configuration example>
FIG. 4 is a diagram illustrating a configuration example of the in-vehicle device 10. As shown in FIG. First, FIG. 4A shows an in-vehicle device 10 mounted on a vehicle. The in-vehicle device 10 is a device mounted in a vehicle, and has at least a facility search function in this embodiment. Also, it is preferable to have a function of searching for a route to the facility and displaying it on the display 14 . Such an in-vehicle device 10 is sometimes called a car navigation device.

The in-vehicle device 10 searches for a route from the departure point to the destination and sets it on the road map, displays the route and the current location on the electronic map 301 displayed on the display 14, or makes a voice call before changing the route based on the route. It outputs guidance and guides the appropriate course with animation etc. In addition, it may have an AV (Audio Visual) playback function, a communication function with the Internet, and the like. The in-vehicle device 10 is sometimes called a tuner, PND (Portable Navigation Device), or car audio.

A display audio device mainly has an AV function and a communication function, but does not have a navigation function. Display audio provides navigation functions by communicating with terminal devices such as smartphones. In this case, an application installed in the smartphone creates a navigation screen, and Display Audio acquires this navigation screen through communication and displays it on the display 14 of the in-vehicle device. CarPlay (registered trademark), Android Auto (registered trademark), and the like are known as such applications that operate on smartphones. The in-vehicle device 10 may be a display audio.

Further, the in-vehicle device 10 may be a general-purpose information processing terminal or may be a dedicated terminal dedicated to the vehicle. General-purpose information processing terminals include, for example, smartphones, tablet terminals, mobile phones, PDAs (Personal Digital Assistants), notebook PCs, and wearable PCs (for example, wristwatch type, sunglasses type, etc.). These devices are usually used as information processing terminals, but when application software for navigation is executed, they perform route searches and route guidance, etc., like dedicated terminals. The dedicated terminal refers to a PND (Portable Navigation Device) or the like in addition to the navigation device, tuner, or car audio described above.

The in-vehicle device 10 may be capable of switching between an in-vehicle state and a portable state, regardless of whether it is a general-purpose information processing terminal or a dedicated navigation terminal. That is, the in-vehicle device 10 may be detachable from the vehicle.

FIG. 4B is a schematic configuration diagram of the communication system 100 having the in-vehicle device 10 and the server 50. As shown in FIG. The in-vehicle device 10 communicates with the server 50 via a network. When searching for a facility or the like, there is a case where the in-vehicle device 10 communicates with the server 50 and performs processing instead of completing the processing in the in-vehicle device 10 . For example, the following forms can be adopted.
(i) The server 50 performs voice recognition, searches for facilities, and transmits the search results to the vehicle.
(ii) The server 50 recognizes the voice and transmits a command to the in-vehicle device 10, and the in-vehicle device 10 searches for facilities.
(iii) The in-vehicle device 10 recognizes the voice and transmits the command to the server 50, the server 50 searches for facilities, and transmits the search result to the vehicle.

Whether the process is completed by the in-vehicle device 10 as shown in FIG. 4(a) or is processed in cooperation with the server 50 as shown in (i) to (iii) of FIG. 4(b) depends on various situations. determined by

First, voice recognition methods include a trigger method and a triggerless method. In the trigger method, a command is detected from the voice when the in-vehicle device 10 detects the pressing of a steering switch or a certain utterance (calling). This method starts speech recognition for The triggerless method is a method in which the in-vehicle device 10 always continues to recognize the voice in order to detect the command while the voice recognition function is activated.

When the server 50 performs voice recognition, it is advantageous in terms of performance of voice recognition, but the triggerless method constantly sends voice data to the server 50, so there is a risk that the amount of communication will increase and the response will be delayed. . Although speech recognition by the in-vehicle device 10 is disadvantageous in terms of speech recognition performance, the amount of communication can be suppressed, and the response to obtain the result of speech recognition becomes faster.

Therefore, the form (iii) of FIGS. 4(a) and 4(b) may be preferable for the triggerless system, and the form (i) and (ii) of FIG. 4(b) may be preferable for the trigger system. have a nature. This embodiment will be described based on the form of FIG. 4(a) in which the process is completed by the in-vehicle device 10, but the forms (i) to (iii) of FIG. 4(b) can also be employed.

In addition, in the form of FIG. 4B, the server 50 can also provide functions related to navigation. For example, when the vehicle 8 transmits the current location, the server 50 creates a navigation screen and transmits it to the vehicle 8 , and when the vehicle 8 transmits two POIs, it searches for a route and transmits it to the vehicle 8 . Further, the server 50 may deliver to the vehicle 8 at least a part of the map DB for drawing the electronic map and the road structure DB representing the structure of the road.

<Functions possessed by the in-vehicle device>
FIG. 5 is an example of a functional block diagram showing functions of the in-vehicle device 10 in block form. The in-vehicle device 10 includes an image data acquisition unit 21, a line-of-sight direction recognition unit 22, a voice data acquisition unit 23, a voice recognition unit 24, a command detection unit 25, a facility search unit 26, a position detection unit 27, a route search unit 28, and route guidance. It has a section 29 , an operation reception section 30 , and a navigation screen creation section 31 .

These functions of the in-vehicle device 10 are general functions of the information processing device of the in-vehicle device 10 (CPU, RAM, auxiliary storage device, input/output I/F, display 14, keyboard, touch panel, communication device, etc.). It is realized by For example, it is a function or means implemented by a CPU executing a program expanded from an auxiliary storage device to a RAM.

The in-vehicle device 10 also has a facility information DB 35, a road structure DB 36, and a map DB 37, which are formed by an auxiliary storage device, RAM, or the like. First, these databases will be explained.

表１（ａ）は、施設情報ＤＢ３５に記憶されているジャンル情報を模式的に示す。ジャンル情報は各施設のジャンルを索引的に検索できるデータベースであり、表１（ｂ）の施設詳細情報の施設と対応付けられている。また、ジャンル情報には一般的な施設のジャンルが格納されている。このジャンルで施設詳細情報が検索される。したがって、音声認識でジャンルを特定できれば、このジャンルの施設を検索できる。

Table 1(a) schematically shows genre information stored in the facility information DB 35 . The genre information is a database in which the genre of each facility can be searched as an index, and is associated with the facility of the detailed facility information in Table 1(b). The genre information stores the genre of general facilities. Facility detailed information is retrieved in this genre. Therefore, if a genre can be specified by voice recognition, facilities of this genre can be searched.

Table 1(b) schematically shows detailed facility information stored in the facility information DB 35 . The detailed facility information has facility information about each facility. For each genre, for example, facility names, store names, addresses, telephone numbers, and genre names are registered. The facility name is, so to speak, a brand name, such as the name of a chain store or a store name. The store name is a name for distinguishing the same facility name, and may not exist. The address indicates the location. A phone number is a contact. The genre is a group name for grouping the facilities according to some attribute, and is associated with the genre in Table 1(a). Note that one facility may belong to multiple genres.

In addition to this, the facility detailed information may be registered with a link to which the facility is in contact, and is not limited to the information shown in Table 1(b).

表２は道路構造ＤＢ３６の構成を模式的に示す。道路構造ＤＢ３６は、車両が通行可能な道路の構造を表すデータであって、表２（ａ）のノードテーブルと表２（ｂ）のリンクテーブルとが相互に連携したデータ構造を有する。

Table 2 schematically shows the configuration of the road structure DB36. The road structure DB 36 is data representing the structure of roads on which vehicles can pass, and has a data structure in which the node table of Table 2(a) and the link table of Table 2(b) are mutually linked.

In the node table, node IDs for identifying nodes, node coordinates (latitude and longitude), presence/absence of intersections, presence/absence of temporary stops, and the like are registered. A node is a node on the road network representation. Nodes are, for example, intersections, branch points, confluence points, bend points, and the like. If the node ID is known, the link connected to the node can be found from the link table. Therefore, the in-vehicle device 10 can trace the structure of the road by referring to the node table and the link table. In addition, the presence or absence of crosswalks, the number of connected links, and the like may be registered in the node table, and the information is not limited to that shown in Table 2.

In the link table, link IDs for identifying links, link start nodes, link end nodes, link lengths, road types, traffic regulations, and the like are registered. A link represents a road that connects nodes, and a link is a line segment that connects nodes. Road types include general roads, highways, exclusive roads, and private roads, for example. Traffic regulations are traffic regulations or traffic rules regarding the driving of vehicles (prohibited or restricted) stipulated in the link. In addition, the number of lanes, width of lanes, information on surrounding facilities, and the like may be registered in the link table, and the information is not limited to the information shown in Table 2.

The map DB 37 stores map data for drawing the electronic map 301 . Information displayed on the electronic map 301 includes data for drawing a map using vector data and raster data. The map data is partitioned into meshes with known longitude, latitude, etc., and an electronic map 301 is created by combining one or more meshes. Since the information displayed on the electronic map 301 includes many display objects such as divisions such as prefectures, green areas and rivers, roads and railroads, symbols and notes, they are classified into similar properties and drawn for each classification. It is possible. The display objects classified respectively or the states in which the display objects are drawn are called layers, and the electronic map 301 is drawn by overlapping several layers.

Returning to FIG. 5, description will be made. The image data acquisition unit 21 acquires image data captured by the camera. The camera is mounted in a position where it can image the face (more specifically, the eyes) of the vehicle occupant. For example, in the center console, steering column, instrument panel, or on the side of the roof near the driver's seat near the front window shield. The camera should irradiate infrared light so that the driver's face can be imaged. The camera repeatedly images the face of the vehicle occupant at a predetermined frame rate.

The image data acquisition unit 21 acquires image data from a camera that repeatedly captures images at a predetermined frame rate. It is not necessary to acquire all the image data captured by the camera. The image data is used for detection of line-of-sight direction, identification of passengers by face recognition, detection of drowsiness, and the like. The system that monitors the occupants is called a driver monitoring system.

A line-of-sight direction recognition unit 22 detects a reference point and a moving point from an eye imaging range in image data showing the face of a vehicle occupant, and recognizes the line-of-sight direction based on the relative positions of the reference point and the moving point. . For example, there is a method of selecting the inner corner of the eye as the reference point and the iris as the moving point, and a method of selecting the corneal reflection as the reference point and the pupil as the moving point. When corneal reflection is used as the reference point, a point light source such as infrared light is irradiated.

The line-of-sight direction recognizing unit 22 identifies the facial part from the image data, identifies characteristic parts such as eyebrows, nostrils, eyes, and lips from the facial part, and identifies the positions of the eyes from the arrangement relationship of these parts. Once the eye is located, the inner corner of the eye, iris, corneal reflection and pupil can be detected. A table in which the line-of-sight direction is associated with the relative position of the reference point and the motion point is prepared in advance, and the line-of-sight direction recognition unit 22 refers to this table to recognize the line-of-sight direction. The line-of-sight direction is represented, for example, by a vector of a three-dimensional coordinate system with the center of the imaging device of the camera as the origin.

Since the line-of-sight direction is sent for each image data, the line-of-sight direction recognition unit 22 recognizes the line-of-sight direction according to the frame rate of the camera. Since the detected line-of-sight direction has a certain amount of error, the line-of-sight direction recognition unit 22 may calculate and use the average of a plurality of line-of-sight directions.

The line-of-sight direction recognition unit 22 determines which of the right side 11, the front side 12, the left side 13, and the display 14 the line-of-sight direction is directed. The line-of-sight direction recognition unit 22 has in advance the range of the line-of-sight direction facing the front 12, the left side 13, the right side 11, or the display 14 as a parameter. For example, when the line-of-sight direction is represented by a three-dimensional vector, it is converted into a horizontal angle and a vertical angle in which the front direction is zero (converted into polar coordinates). When the horizontal angle and vertical angle of the line-of-sight direction are within the ranges associated with the right side 11, the front side 12, the left side 13, or the display 14, the line-of-sight direction is the right side 11, the front side 12, the left side 13, or the display It is judged that 14 is facing. The line-of-sight direction recognition unit 22 sends the line-of-sight direction (right side 11, front side 12, left side 13, or display 14) to the facility search unit 26. FIG.

The audio data acquisition unit 23 converts the audio collected by the microphone 41 into an electric signal by A/D conversion and acquires it as audio data at all times in a triggerless manner. Since the audio data is continuous data, the audio data acquisition unit 23 repeatedly acquires audio data for a predetermined time period (for example, 20 milliseconds). The predetermined time is set as a time advantageous for voice recognition.

The speech recognition unit 24 performs speech recognition by converting speech data into text data. There are known methods and various methods for conversion into text data. An example is described here. First, the speech recognition unit 24 performs processing such as converting the speech data into a frequency spectrum and removing the frequency portion corresponding to noise. Next, formants or MFCCs (Mel Frequency Cepstrum Coefficients) are extracted from the voice data as information useful for pattern recognition. As a result, several dozen-dimensional feature vectors are obtained from the speech data. Then, the speech recognition unit 24 retrieves an acoustic model that associates a waveform of speech (a set of phonemes in which three phonemes are combined called a triphone) with the text data of the speech at that time using the feature vector, and extracts the text data. obtain. Since there are many cases where the text data is not uniquely determined, the speech recognition unit 24 employs the most probabilistic text data using a language model that expresses how words are connected in terms of probability. In this method, morphological analysis is unnecessary because words are recognized word by word. When speech recognition is performed on a character-by-character basis, the speech recognition unit 24 performs morphological analysis.

In recent years, speech recognition using deep learning has become common. Since voice data is time-series data of variable length, a method called RNN (Recurrent Neural Network), which can handle variable length data with a neural network, is used.

Text data recognized by the speech recognition unit 24 is sent to the command detection unit 25 . A command detection unit 25 detects a command included in the text data and calls a function (block) corresponding to the command. For example, in the triggerless method, some commands are defined in advance, and whether or not these commands are included in the text data is monitored. Commands are, for example, "convenience store nearby", "search surrounding area", "gas station nearby", "return home", "route guidance", "delete route", "map display", "audio display", "double zone", "wide area display". They are "detailed display", "front camera", "side camera", "corner camera", and the like.

The command detection unit 25 executes processing according to the command. For example, “nearby convenience store”, “surrounding search”, and “nearby gas station” are commands related to search, so a search request is generated and sent to the facility search unit 26 . Since "return to home", "route guidance", and "delete route" are commands related to route search, a route search request designating the starting point and destination is generated and sent to the route searching section 28. FIG. Since "map display", "audio display", and "double zone" are commands related to the display of the display 14, the navigation screen creation unit 31 is requested to switch the screen display. Since "wide display" and "detailed display" are commands related to the scale of the electronic map 301, the navigation screen creation unit 31 is requested to change the scale of the electronic map 301. FIG. "Front camera", "side camera", and "corner camera" are commands for switching camera images.

Therefore, for example, in response to the command "nearby convenience store", the command detection unit 25 sends to the facility search unit 26 a search request including the distance information "nearby" and the genre "convenience store".

The facility search unit 26 receives the search request and the line-of-sight direction, adds the direction based on the line-of-sight direction as a search element to the search request, and searches the facility information DB 35 .

The operation reception unit 30 inputs two POIs (for example, a departure point and a destination) for route search, starts route guidance, instructs search again, instructs enlargement/reduction of the navigation screen, changes the display range, and the like. instruction from the vehicle occupant.

The position detection unit 27 detects the current position of the vehicle as latitude, longitude, and altitude coordinates by radio positioning performed by the GPS receiver periodically and in response to operation by the vehicle occupant. The position detection unit 27 estimates a detailed position using an autonomous navigation sensor, and further corrects the position of the vehicle on the road by map matching.

The route search unit 28 searches for a route using the road structure DB 36 and creates route information when a route search is requested by a vehicle occupant or when the vehicle deviates from the route and the route is searched again. For route search, convert the link length, width, and traffic congestion into costs, and select the route that minimizes the total cost from the departure point to the destination. * (Aster) method is known. If the angle formed by adjacent links on the determined route is greater than or equal to a predetermined angle, it is determined that the direction of travel needs to be changed, and a guidance point is set before the course change in the route information. The route search unit 28 sends the route information generated by the route search to the route guide unit 29 and the navigation screen creation unit 31 .

The route guidance unit 29 provides route guidance based on the current location (positional information) and route information. That is, when the current position of the vehicle reaches the guidance point included in the route information, the speaker 43 outputs audio data indicating a turning corner or the like. The voice data may be converted into data in a format such as MP3, or may be created by synthesizing voice based on text data for guidance.

The navigation screen creation unit 31 creates, for example, an electronic map 301 including the surroundings of the vehicle's current location, with the route highlighted, using the map data in the map DB 37 . The navigation screen creation unit 31 further displays the current location of the vehicle using a current location mark or the like. A navigation screen creation unit 31 creates a navigation screen by adding menu buttons and the like to the electronic map. Further, when the vehicle approaches a place such as a large intersection where an enlarged map is prepared, the navigation screen creation unit 31 acquires map data from the map DB 37 and displays an enlarged map of the intersection and the like. In addition, we will also create screens for operating AV.

<Examples of line-of-sight directions determined to be right, front, left, or display>
Next, the line-of-sight direction determined to be the right side 11, the front side 12, the left side 13, or the display 14 will be described with reference to FIG. FIG. 6 is a diagram for explaining the viewing direction determined to be the right side 11, the front side 12, the left side 13, or the display 14. FIG. As shown, when the horizontal angle of the viewing direction is between φ1 and φ2 and the vertical angle is between θ1 and θ2, the viewing direction is determined to be left 13 . Similarly, for the front 12, right 11, and display 14, the line-of-sight direction is determined based on whether the line-of-sight direction falls within a predetermined range of horizontal and vertical angles.

It should be noted that such a determination is an example, and when the horizontal angle is left of threshold 1 (horizontal angle indicating the direction slightly left from the front), it is determined to be left 13, and the horizontal angle is determined to be left 13, and the horizontal angle is threshold 2 (slightly left from the front). If it is to the right of the horizontal angle indicating the direction of the right side), it may be determined to be the right side 11, and other than that, it may be determined to be the front side 12. That is, vertical angles are not taken into account in this case. However, if the left side 13 is determined, the vertical angle is compared with a threshold value 3 (the vertical angle indicating the upper side of the display 14) to distinguish whether the display 14 is facing or the left side 13 is facing.

<Timing for judging line-of-sight direction>
It takes a certain amount of time for the vehicle occupant to speak a command, but the line-of-sight direction is determined repeatedly according to the frame rate as an instantaneous value. On the other hand, the vehicle occupant must have finished speaking the command in order for the command to be recognized.

FIG. 7 is an example of a diagram for explaining the timing of determination of the line-of-sight direction with respect to command utterances. From the request that the vehicle occupant must have finished speaking the command, as shown in FIG. be done. It is easy for the vehicle occupant to turn in the desired direction at the end of command.

Considering that the vehicle occupants looking for facilities naturally look in that direction, as shown in FIG. , is also effective for use by the facility search unit 26 . In this case, the facility search unit 26 retains the line-of-sight direction for a certain period of time in the past, and the facility search unit 26 can use the line-of-sight direction of the previous time determined by the length of the command (the time required for the statement). Conceivable. Vehicle passengers can search for facilities in a natural way by uttering commands while looking for facilities in real space.

Considering that the vehicle occupant continuously searches for a facility in a desired direction while uttering a command, as shown in FIG. It is also useful for the part 26 to use. The time during which the command is spoken is predetermined by the length of the command. Since the occupant of the vehicle can look in the desired direction for a certain amount of time while giving the command, erroneous operations can be reduced.

It should be noted that the line-of-sight direction may be blurred (changed significantly) when the occupant of the vehicle brakes the vehicle. In this case, since the reliability of the line-of-sight direction is reduced, when braking is detected, the line-of-sight direction recognition unit 22 stops recognizing the line-of-sight direction, or the facility search unit 26 uses the line-of-sight direction for searching. It is effective not to It should be noted that the braking of the vehicle by the occupant of the vehicle can be detected from the longitudinal acceleration. Also, if the direction of travel is changed (turning left or right or proceeding to a Y-junction), the line of sight direction may be blurred (changed significantly), so the same process may be performed in this case as well. It should be noted that the fact that the vehicle is changing its traveling direction can be detected from the yaw rate.

<Regarding searching for facilities on the right or left>
Next, searching for facilities on the right or left will be described with reference to FIGS. 8 and 9. FIG. First, referring to FIG. 8, the search for facilities "near" the vehicle will be described. FIG. 8 shows an example of an electronic map 301 displayed on the display 14 regarding the description of the search range. In FIG. 8, the current location mark 302 and the route 303 are displayed, but for convenience of explanation, the features are omitted.

Since the electronic map 301 is drawn for each mesh as described above, a single electronic map 301 often has a plurality of meshes 305 . The coordinates of the four vertices of mesh 305 are known. In order to realize smooth scrolling, a buffer area 304 larger than the electronic map 301 is drawn on the memory for at least one mesh.

When a route 303 to the destination is set as shown in FIG. 8, facilities along the route (facing the route) are searched because many passengers do not want to deviate from the route 303 . Also, since it is unlikely that the vehicle will return in the direction opposite to the destination, facilities ahead in the traveling direction are searched. The vicinity of the vehicle may be a range that can be reached in several minutes, or a range displayed on the electronic map 301 . Therefore, the facility search unit 26 searches the facility information DB 35 for facilities existing within the mesh 305 including the route 303 ahead of the current location and within a predetermined distance from the current location. In order to limit the facilities along the route (to limit the facilities facing the route 303), the links to which the facilities registered in the facility information DB 35 are in contact are used for narrowing down.

If the route 303 to the destination is not set, it is considered that the facilities to be provided differ depending on whether the vehicle is running or not. When the vehicle is running, the road on which it is currently running is regarded as the route 303, and facilities are searched in the same way as when the route 303 to the destination is set. If the vehicle is not running, the direction of travel is unknown, so the facility information DB 35 is searched for facilities within a predetermined distance from the current location. For example, the facility is retrieved from the mesh 305 where the current location is located and the eight meshes 305 adjacent thereto. Whether or not the vehicle is running can be determined from at least one of the vehicle speed and current position information.

The in-vehicle device 10 of the present embodiment narrows down only facilities on the right side or the left side of the route 303 or the road on which the vehicle is traveling when the line-of-sight direction is the right side 11 or the left side 13 .

FIG. 9 is an example of a diagram explaining a method of determining whether the facility is on the right side or the left side with respect to the route 303 or the road on which the vehicle is traveling. As shown in FIG. 9A, it is assumed that there are facilities 311 and 312 on the right and left sides of the route 303 or the road on which the vehicle is traveling. First, as shown in FIG. 9(b), the facility search unit 26 retrieves the link 313 closest to the facility from among the links forming the route 303 or the road on which the vehicle is traveling.

Consider a straight line v1 connecting the facility 311 and the starting point of the link 313 seen from the traveling direction, and a straight line v2 connecting the facility 311 and the end point of the link 313 seen from the traveling direction. For facility 312, straight lines v1 and v2 are similarly considered. In the case of the facility 311 on the right side of the route 303 or the road on which one is traveling, the direction of rotation for changing the direction of the straight line v1 to the direction of the straight line v2 is clockwise. , the direction of rotation that changes the direction of the straight line v1 to the direction of the straight line v2 is counterclockwise.

If there are two vectors, there is a method of using the outer product to determine whether the direction is clockwise or counterclockwise. Therefore, for example, it is possible to determine whether the facilities 311 and 312 are on the right side or the left side of the route 303 or the road on which they are traveling, as follows.
Straight line 1 x straight line 2 is positive → left straight line 1 x straight line 2 is negative → right side In this way, the facility search unit 26 searches for nearby facilities, and only the facilities on the right or left side of the route or the road on which you are traveling. can be searched (can be narrowed down).

When the line of sight is directed toward the display 14 , it is sufficient to search for facilities having coordinates within the display range of the electronic map 301 .

<Other viewing directions>
Besides the right side 11, front side 12, left side 13, or display 14, it may be determined whether a vehicle occupant is looking at a HUD (Head Up Display). FIG. 10 shows a display example of the HUD. In the HUD shown in FIG. 10, the icon indicates "turn right 300 m ahead", and the HUD may display the traveling direction in this way.

A vehicle occupant looks at the HUD to predict the timing of changing the direction of travel, but there are cases where the passenger wants to know what facilities are available after the change of the direction of travel, or if there are desired facilities. Therefore, when a vehicle occupant looks at the HUD and issues a command, it is effective for the facility search unit 26 to search the vicinity after the vehicle changes its traveling direction at the nearest guide point.

In this case, the facility search unit 26 searches for facilities from the mesh including the route or the road on which the vehicle is traveling ahead of the node whose direction of travel is to be changed. In this case, facilities on both sides of the route or the road on which the vehicle is traveling are searched, but if the vehicle occupant wishes to narrow down to the right or left side, the search can be performed again according to the line-of-sight direction.

<Operation procedure>
FIG. 11 is an example of a flowchart showing a procedure for searching for facilities by the in-vehicle device 10 using the line-of-sight direction as a search element.

Steps S1-S4 are executed in parallel. In step S1, the image data acquisition unit 21 acquires image data from the camera (S1). The line-of-sight direction recognition unit 22 recognizes the line-of-sight direction from the face image shown in the image data (S2).

The voice data acquisition unit 23 always acquires voice data (S3), and the voice recognition unit 24 converts the voice data into text data (S4).

The command detection unit 25 determines whether or not the text data includes a predetermined command (S5). In this embodiment, a search-related command is detected. If no command is detected (No in S5), the process returns to step S1.

If the command is detected (Yes in S5), the facility search unit 26 identifies the line-of-sight direction to use as a search element (S6). That is, the line-of-sight direction is specified from the start of speech, the end of speech, or an average. Also, when there is braking or a change in the direction of travel, the line-of-sight direction after braking or after the change in the direction of travel is ignored.

Next, the facility search unit 26 searches for, for example, nearby facilities according to the search request (S7). The genre of the facility is designated by the vehicle occupants. It is not necessary for the crew member to specify it explicitly, and for example, they may search for facilities in the genre they frequently visit. If the viewing direction is the display 14, the search can be performed within the range of the displayed electronic map 301. FIG.

Processing is executed according to the line-of-sight direction specified by the facility search unit 26 (S8).
- If the line of sight is to the left 13, the facilities are narrowed down to the facilities on the left (S9).
・If the viewing direction is the front 12, the facilities are not narrowed down.
If the line of sight is to the right 11, narrow down to facilities on the right (S10).
If the viewing direction is the display 14, search is performed within the display range of the electronic map 301 (S11). In this case, step S11 becomes unnecessary if the search is performed in step S7.

The navigation screen creation unit 31 displays the icon or pin mark of the found facility on the electronic map 301 (S12).

<Summary>
As described above, the in-vehicle device 10 of the present embodiment can recognize the line-of-sight direction of the vehicle occupant and search for facilities by combining the direction the vehicle occupant is looking and voice operation. In other words, the facility search range can be changed according to the direction the vehicle occupants are looking. Therefore, there is no need to utter a voice that specifies a different search element such as direction in addition to the voice for searching for facilities, and there is no need to utter long words or complicated words, so the voice recognition rate may decrease. do not have.

In this embodiment, an in-vehicle device 10 that changes the route search method according to the line-of-sight direction in route search will be described. That is, the line-of-sight direction is added to the search element for route search.

<Overview of route search in this embodiment>
FIG. 12 is a diagram illustrating an example of the line-of-sight direction of a vehicle occupant and a searched detour route. First, as shown in FIG. 12(c), the vehicle occupant is looking ahead from the driver's seat. For example, when a vehicle occupant says "detour route" while looking at the right side 11 , the in-vehicle device 10 searches only the detour route 321 on the right side (right turn) of the route 303 . Also, if the user speaks while looking at the left side 13, only the detour route 320 on the left side (left turn) of the route is retrieved.
- Fig. 12(a) is a diagram showing a detour route 320 searched for when the vehicle occupant says "detour route" while looking at the left side 13. Figs. A detour route 320 is displayed to the left of the route 303 .
- FIG. 12(b) is a diagram showing a detour route 321 searched for when the vehicle occupant says "a detour route" while looking at the right side 11. FIG. A detour route 321 is displayed on the right side of the route 303 .

For example, when the right-turn lane is congested, even if a detour route on the right side of the route 303 is proposed, it is difficult for the occupants of the vehicle to adopt it. Also, when the right lane is used and the left lane is congested, even if a detour route on the left side of the route 303 is proposed, it is difficult to adopt it. In this embodiment, in these cases, the search direction of the detour routes 320 and 321 can be controlled simply by turning the line of sight of the vehicle occupant to the right or left.

<Functions possessed by the in-vehicle device>
This embodiment will be described with reference to the functional block diagram of FIG. 5 of the first embodiment. However, in this embodiment, the line-of-sight direction recognized by the line-of-sight direction recognition unit 22 is input to the route search unit 28 . When the line-of-sight direction is input, the route search unit 28 searches for the route 303 or a detour route on the right or left side of the road on which the vehicle is traveling, depending on the line-of-sight direction. The flow of processing is as follows.
- For example, a route search is performed with a node having an intersection ahead of the current location as a starting point and a node where the congestion of the current route 303 is alleviated as an ending point. Alternatively, a node a predetermined distance ahead may be set as the end point.
• Set the cost of the link from the origin to the destination that is included in the current route 303 to the maximum value.
If the line of sight is 11 on the right side, the cost of the link that changes course to the left of the route 303 is set to the maximum value, and if the line of sight is 13 on the left side, the link that changes course to the right of the route 303 is set. set the cost of to the maximum value. By doing so, a detour route on the right or left side of the route 303 can be searched. Note that a link with a course change to the left or right with respect to the route 303 can be determined by the angle formed with the link included in the route 303 .

<Operation procedure>
FIG. 13 is an example of a flowchart showing a procedure for searching for a detour route by the in-vehicle device 10 using the line-of-sight direction as a search element. Note that in the explanation of FIG. 13, differences from FIG. 11 are mainly explained. First, steps S11 to S16 may be the same as steps S1 to S6.

When a command related to route search is detected (S15) and the line-of-sight direction can be specified (S16), processing is executed according to the line-of-sight direction specified by the facility search unit 26 (S17).
When the line-of-sight direction is the left side 13, the route search unit 28 searches for the route 303 or a detour route on the left side of the road on which the vehicle is traveling (S18).
If the line-of-sight direction is the right side 11, the route searching unit 28 searches for the route 303 or a detour route on the right side of the road on which the vehicle is traveling (S19).

The navigation screen creation unit 31 creates an electronic map 301 including a detour route with the detour route emphasized, and displays it on the display 14 (S20).

Therefore, according to the in-vehicle device 10 of the present embodiment, the detour route search direction can be controlled simply by turning the line of sight of the vehicle occupant to the right or left. The detour route can be searched not only when the route 303 to the destination is set, but also when the route 303 is not set. If no route is set, a detour route is searched for the road on which the vehicle is running.

In the present embodiment, the in-vehicle device 10 that switches the image displayed on the display 14 according to the line-of-sight direction will be described.

<Overview of route search in this embodiment>
First, as shown in FIG. 14, one vehicle may be equipped with a plurality of cameras. FIG. 14 shows an example of arrangement of cameras mounted on a vehicle. The front camera 51 is mainly a wide-angle camera and is used to capture images of obstacles on the left and right at T-junctions and crossroads. The right side camera 54 and the left side camera 53 are used to capture images of left or right side blind spots. The back camera 52 is mainly used for taking pictures of the rear when backing up.

For example, when a vehicle occupant says “side camera” while looking at right side 11 , in-vehicle device 10 displays an image captured by right side camera 54 on display 14 . Also, when the user says “side camera” while looking at the left side 13 , the in-vehicle device 10 displays an image captured by the left side camera 53 on the display 14 .
- When a vehicle occupant says "side camera" while looking at the right side 11, an image 55 is displayed on the display 14. - 特許庁The image 55 shows the right rear side of the vehicle 8 .
- When the vehicle occupant says "side camera" while looking at the left side 13, an image 56 is displayed on the display 14. - 特許庁The image 56 shows the rear left side of the vehicle 8 .

When you want to pull the vehicle to the left or right side of the road, or when you want to check the presence of a gutter or an obstacle while driving on a narrow road, the vehicle occupant turns the line of sight to the right or left and says "side camera". The video displayed on the display 14 can be switched to the video captured by the right side camera 54 and the left side camera 53 by simply pressing the button.

<Functions possessed by the in-vehicle device>
FIG. 15 is an example of a functional block diagram showing functions of the in-vehicle device 10 in block form. With regard to the description of FIG. 15, since the components denoted by the same reference numerals in FIG. 5 perform similar functions, only the main components of this embodiment may be mainly described.

The in-vehicle device 10 of this embodiment has a camera switching unit 32 . The line-of-sight direction recognized by the line-of-sight direction recognition section 22 is input to the camera switching section 32 . In addition to the navigation screen, images from the front camera 51 , the back camera 52 , the left side camera 53 , and the right side camera 54 are input to the camera switching unit 32 . The camera switching unit 32 displays the image of the right side camera 54 on the display 14 when the line of sight direction of the right side is input, and displays the image of the left side camera 53 on the display 14 when the line of sight direction of the left side is input. do.

<Operation procedure>
FIG. 16 is an example of a flowchart showing a procedure for switching the image displayed on the display 14 by the in-vehicle device 10 according to the line-of-sight direction. Note that the explanation of FIG. 16 mainly explains the difference from FIG. First, steps S21 to S26 may be the same as steps S1 to S6.

When a route search command is detected (S25) and the line-of-sight direction can be specified (S26), processing is executed according to the line-of-sight direction specified by the line-of-sight direction recognition unit 22 (S27).
- When the line of sight direction is the left side 13, the camera switching unit 32 switches the camera for displaying the image on the display 14 to the left side camera 53 (S28).
If the viewing direction is the right side 11, the camera switching unit 32 switches the camera that displays the image on the display 14 to the right side camera 54 (S29).

As a result, the image captured by the left side camera 53 or the right side camera 54 is displayed on the display 14 (S30).

<Summary>
Therefore, according to the in-vehicle device 10 of the present embodiment, the occupant of the vehicle turns his or her line of sight to the right or left and issues a command for switching the camera image, thereby switching the camera that displays the image on the display 14. be able to.

In this embodiment, the images of the left and right side cameras are switched according to the line-of-sight direction, but if there are two front cameras 51 or two back cameras 52, these images can be switched.

<Other application examples>
Although the best mode for carrying out the present invention has been described above using examples, the present invention is by no means limited to such examples, and various modifications can be made without departing from the scope of the present invention. and substitutions can be added.

For example, although commands are detected by a triggerless method in the present embodiment, commands may be detected by a trigger method.

Further, the facility to be searched is not limited to be along the route or along the road on which the vehicle is traveling, and may be on the right or left side of the route or the road on which the vehicle is traveling.

Further, in triggerless speech recognition, there is a risk that the speech output by the audio-visual device of the in-vehicle device 10 will be collected together with the driver's utterances. In order to solve such a problem, it is effective to branch the voice of the audiovisual device to the voice recognition unit separately from the speaker output for voice recognition. Text data uttered by the driver can be obtained by excluding the voice of the audio-visual device input by branching from the voice data input from the microphone and recognizing the voice.

Further, the configuration example of FIG. 5 and the like is divided according to main functions in order to facilitate understanding of the processing of the in-vehicle device 10 . The present invention is not limited by the division method or name of the unit of processing. Further, the processing of the in-vehicle device 10 can be divided into more processing units according to the content of the processing. Also, one processing unit can be divided to include more processing.

The voice recognition unit 24 is an example of voice recognition means, the line-of-sight direction recognition unit 22 is an example of line-of-sight direction recognition means, the command detection unit 25 is an example of command detection means, the facility search unit 26, the route search unit The unit 28 and the camera switching unit 32 are examples of command processing means.

10 In-Vehicle Device 14 Display 41 Camera 42 Microphone 43 Speaker

Claims (11)

An in-vehicle device that receives a voice operation of a vehicle occupant,
voice recognition means for recognizing the voice of an occupant of the vehicle;
a line-of-sight direction recognition means for recognizing a line-of-sight direction of an occupant of the vehicle;
command detection means for detecting a command from the voice recognized by the voice recognition means;
command processing means for processing the command according to the line-of-sight direction recognized by the line-of-sight direction recognition means;
a display that displays an electronic map;
The command processing means searches for POIs existing within the display range of the electronic map displayed on the display when the line-of-sight direction is within the range of the display;
The command detection means detects the command related to the search from the voice recognized by the voice recognition means,
The in-vehicle apparatus, wherein the command processing means adds the line-of-sight direction recognized by the line-of-sight direction recognition means to a search element and processes the command related to the search. The command detection means detects the command related to the search from the voice recognized by the voice recognition means,
2. The in-vehicle apparatus according to claim 1, wherein said command processing means processes said command relating to said search by changing said search range according to said line-of-sight direction recognized by said line-of-sight direction recognition means. The command relates to searching for POIs (Point Of Interest),
3. The in-vehicle device according to claim 1 , wherein said command processing means changes a range of POIs to be searched according to said line-of-sight direction. 4. The vehicle-mounted vehicle according to claim 3 , wherein said command processing means searches for a POI existing on the right side of the traveling direction or searches for a POI existing on the left side in accordance with said viewing direction. Device. 5. The command processing means according to any one of claims 1 to 4 , wherein when the line-of-sight direction is the front, the command processing means searches for POIs existing on the right side and POIs existing on the left side of the traveling direction. in-vehicle equipment. Said command relates to finding a route,
2. The in-vehicle device according to claim 1, wherein said command processing means changes a route search method according to said line-of-sight direction. An in-vehicle device that receives a voice operation of a vehicle occupant,
voice recognition means for recognizing the voice of an occupant of the vehicle;
a line-of-sight direction recognition means for recognizing a line-of-sight direction of an occupant of the vehicle;
command detection means for detecting a command from the voice recognized by the voice recognition means;
command processing means for processing the command according to the line-of-sight direction recognized by the line-of-sight direction recognition means;
The in-vehicle device, wherein the command processing means searches for a detour route on the right side of the traveling direction or searches for a detour route on the left side in accordance with the line-of-sight direction. The command detection means detects a command related to an image to be displayed on a display from the voice recognized by the voice recognition means,
2. The in-vehicle apparatus according to claim 1, wherein said command processing means processes a command for switching the image displayed on said display in accordance with the line-of-sight direction recognized by said line-of-sight direction recognition means. 9. The method according to claim 8 , wherein the command processing means switches between displaying an image of the right camera on the display and displaying an image of the left camera on the display according to the line-of-sight direction. in-vehicle equipment. A communication system having an in-vehicle device that receives voice operations of a vehicle occupant and a server that communicates with the in-vehicle device via a network,
voice recognition means for recognizing the voice of an occupant of the vehicle;
a line-of-sight direction recognition means for recognizing a line-of-sight direction of an occupant of the vehicle;
command detection means for detecting a command from the voice recognized by the voice recognition means;
command processing means for processing the command according to the line-of-sight direction recognized by the line-of-sight direction recognition means;
a display that displays an electronic map;
The command processing means searches for POIs existing within the display range of the electronic map displayed on the display when the line-of-sight direction is within the range of the display;
The command detection means detects the command related to the search from the voice recognized by the voice recognition means,
The communication system, wherein the command processing means adds the line-of-sight direction recognized by the line-of-sight direction recognition means to a search element and processes the command related to the search. A communication system having an in-vehicle device that receives voice operations of a vehicle occupant and a server that communicates with the in-vehicle device via a network,
voice recognition means for recognizing the voice of an occupant of the vehicle;
a line-of-sight direction recognition means for recognizing a line-of-sight direction of an occupant of the vehicle;
command detection means for detecting a command from the voice recognized by the voice recognition means;
command processing means for processing the command according to the line-of-sight direction recognized by the line-of-sight direction recognition means;
The communication system, wherein the command processing means searches for a detour route on the right side of the direction of travel or searches for a detour route on the left side of the direction of travel, depending on the line-of-sight direction.

Images