WO2014196038A1

WO2014196038A1 - Device for processing information through line-of-sight detection and information processing method

Info

Publication number: WO2014196038A1
Application number: PCT/JP2013/065610
Authority: WO
Inventors: 成晃竹原
Original assignee: 三菱電機株式会社
Priority date: 2013-06-05
Filing date: 2013-06-05
Publication date: 2014-12-11

Abstract

This device for processing information through line-of-sight detection is provided with a search request determination unit (11) for determining whether a search request has been made by a user; a coordinate calculation unit (12) for calculating the map coordinates of an object in the user's line of sight, which is made to be a search subject, on the basis of at least the current position of a moving body and recognition of the line of sight of the user; a superimposed display control unit (16) for superimposing and displaying a search mark for highlighting the search subject on an HUD (18) on the basis of the coordinates of the search subject and map data; a search processing unit (13) for searching for detailed information about the search subject on the basis of the coordinates of the search subject and the map data; and an output information generation unit (15) for generating output information for presenting detailed information about the search subject to the user. As a result, the search subject can be simply identified and displayed on the HUD through the detection of the line of sight of the user in a vehicle or other moving body, and it is therefore possible to effectively present detailed information about the search subject to the user.

Description

Information processing apparatus and information processing method based on gaze detection

The present invention relates to an information processing device that specifies a search object by visual line detection and presents detailed information in a moving body such as a vehicle.

Currently, a method for a driver driving a vehicle or the like to specify an object outside the vehicle as a search object has not been sufficiently considered. For example, in order to find out what a building is 500 m away in a search of a car navigation device, a method of narrowing down by pressing a search button or the like with a touch panel or the like is used.

However, even if there is an object that the driver wants to search outside the vehicle, it is difficult for the driver to narrow down by operating the touch panel etc. There is a problem that it is difficult for a user at to specify a search target by pointing to a specific point.

In order to deal with such a problem, for example, in Patent Document 1, when a question such as “What is that?” Is given to an object such as a building or a sightseeing spot that the driver is looking at, voice recognition and There has been disclosed a navigation device that displays detailed information on a building or the like viewed by a driver on a display by gaze detection.

JP 2001-330450 A

However, in the conventional apparatus as shown in Patent Document 1, for example, it is difficult to narrow down the search range without specifying the search word in detail. Even if the search result is shown from a sufficiently wide search area, it is difficult for the driver to make a decision while driving it. Furthermore, even with a recent CPU, if the search area is too large, the search takes time, and the real-time performance is lowered.

The present invention has been made to solve the above-described problems, and information processing based on line-of-sight detection, in which a user in a moving body such as a vehicle can easily specify a search target and obtain detailed information. An object is to provide an apparatus.

In order to achieve the above object, the present invention provides a display for displaying various types of information superimposed on the user's field of view and a search for determining a search request by the user in an information processing apparatus based on a user's line of sight detection using a moving object. When the request determination unit and the search request determination unit determine that there is a search request, search for an object in the user's line-of-sight direction based on at least the current position of the moving object and the user's line-of-sight recognition A coordinate calculation unit that calculates a coordinate position on a map as an object, and a search that highlights the search object on the display based on the coordinate position and map data of the search object calculated by the coordinate calculation unit A superimposition display control unit that superimposes and displays a mark; and a coordinate position and map data of the search object calculated by the coordinate calculation unit. A search processing unit that searches for detailed information on the search target; an output information creation unit that generates output information for presenting detailed information on the search target searched by the search processing unit to the user; It is provided with.

According to the present invention, by detecting the user's line of sight in a moving body such as a vehicle, a search target is easily specified and displayed on a display that displays various information in a user's field of view such as a HUD (head-up display). Therefore, it is possible to efficiently present detailed information of the search object to the user.

2 is a block diagram illustrating an example of an information processing device in Embodiment 1. FIG. It is a table | surface which shows an example of the content of the search trigger which a speech recognition apparatus recognizes. It is a table | surface which shows an example of the content of the search trigger which a gesture recognition apparatus recognizes. It is a table | surface which shows an example of the information which a coordinate calculation part acquires from GPS. It is a table | surface which shows an example of the information which a coordinate calculation part acquires from a gaze recognition apparatus. It is a schematic explanatory drawing which shows the positional relationship of a user's eyeball position, a gaze recognition apparatus, and a search target object. It is a table | surface which shows an example of the information which a coordinate calculation part acquires from vehicle equipment. It is a table | surface which shows an example of the information which a coordinate calculation part acquires from a sensor. It is a flowchart which shows the input process in which a speech recognition apparatus, a switch, and a gesture recognition apparatus receive a search request. It is a flowchart which shows the coordinate calculation process in a coordinate calculation part. It is a flowchart which shows the superimposition display process in a superimposition display control part. It is a flowchart which shows the search process in a search process part. 6 is a diagram illustrating an example of a transition image of processing in the information processing apparatus according to Embodiment 1. FIG. 10 is a block diagram illustrating an example of an information processing device in Embodiment 2. FIG. 10 is a block diagram illustrating an example of an information processing device in Embodiment 3. FIG. FIG. 10 is a diagram illustrating an example of a transition image of processing in the information processing apparatus according to Embodiment 3. 14 is a flowchart illustrating a process of presenting detailed information using line-of-sight recognition in the information processing apparatus according to Embodiment 3. FIG. 10 is a diagram showing an outline of an information processing system (navigation system) in a fourth embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
According to the present invention, in a moving body such as a vehicle, a search target is specified by detecting a line of sight of a user who uses the moving body, and superimposed on a user's field of view such as a head-up display (HUD) or a head-mounted display (HMD). It is an information processing apparatus that presents detailed information on a display that displays various types of information. In the following embodiments, a case where the information processing apparatus of the present invention is applied to a navigation device mounted on a moving body such as a vehicle will be described as an example. Navigation for a moving body other than a vehicle will be described. It may be a device or may be applied to a server of a navigation system. Moreover, you may apply to the application etc. of the navigation system installed in portable information terminals, such as a smart phone, tablet PC, and a mobile telephone.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing an example of an information processing apparatus according to Embodiment 1 of the present invention. The information processing apparatus includes a search request determination unit 11, a coordinate calculation unit 12, a search processing unit 13, a storage unit 14, an output information creation unit 15, and a superimposed display control unit 16, and at least a head-up display ( HUD) 18.

The search request determination unit 11 includes a voice recognition device 1 for recognizing a voice spoken by the user, a physical switch 2 provided on a handle or the like, a camera or optical sensor in the user (driver) direction, etc. From the information obtained from the gesture recognition device 3 capable of detecting movement, the content of the search trigger is recognized to determine whether a search request has been made, and the determination result is notified to the coordinate calculation unit 12.

Since the speech recognition device 1 recognizes the words uttered by the user (driver) because it is a well-known technology, a description thereof will be omitted here. For example, the content of the search trigger as shown in FIG. Recognize whether there was a request. As the search trigger, “what”, “what would be”, “search”, “tell me” and the like are stored as search trigger words, and it is determined whether or not these search trigger words are included. In addition, the position specifying word includes a word specifying a direction such as right or left, or a word specifying a facility name such as a hospital or a temple, whether that, this, that (that, that, this) Whether or not a word suggesting a distance is included.

In order to make it easy to specify the search coordinates, for example, it may be possible to recognize a position specifying word accompanying such a search trigger word together. Moreover, when there are a plurality of position specifying words, they may be used in combination. For example, when there are a plurality of words whose positions can be specified, such as “What is that temple on the right hand”, all of them may be notified to the coordinate calculation unit 12.

Also, there may be one or more switches 2, for example, three switches are arranged side by side, and if you want to search for the one in the left front direction, use the left switch, and if you want to search for the one in front, If it is desired to search for a switch in the right front direction, the right switch may be pressed.

The gesture recognition device 3 is a device that can detect the movement of the user (driver) with a camera or optical sensor in the direction of the user (driver). For example, the gesture recognition device 3 recognizes the content of the search trigger as shown in FIG. Recognizes whether there is a search request. As a search trigger, it is determined whether blinking has been performed a specified number of times, and as a search trigger + position specification, whether a finger has been pointed or pointed with a chin.

As described above, the search request determination unit 11 determines the information obtained from the speech recognition device 1, the switch 2, and the gesture recognition device 3, and notifies the coordinate calculation unit 12 of the determination results. 1, information of the switch 2, and the gesture recognition device 3 may be used in combination. For example, when there are a plurality of position specification determinations, the coordinate calculation unit 12 may be notified after scoring each position specification result and specifying the approximate search target direction.

The coordinate calculation unit 12 acquires information such as current location information, traveling direction, and traveling speed as shown in FIG. 4 from a GPS (Global Positioning System) 4, and calculates and saves the vehicle position. At this time, the vehicle position is basically calculated from the current location information.However, as the traveling speed increases, the human field of view narrows, so the traveling speed is set to prevent searching for a very close place during high-speed traveling. In addition to use, it is calculated which direction is likely to be searched from the vehicle position depending on the traveling direction.

The coordinate calculation unit 12 receives, from the line-of-sight recognition device 5, for example, a line-of-sight direction as shown in FIG. Gaze information such as a relative relationship with the display device) is acquired. Since the technique for recognizing the user's line of sight by the line-of-sight recognition device 5 is well known, detailed description is omitted here, but the line-of-sight vector can be calculated from the position of the eyeball and the direction in which the left and right eyeballs face.

And since the object that the user (driver) is looking at exists at the intersection of the left and right line of sight, the distance to the user (driver) can be measured by calculating the intersection of the vectors. Thereby, the visual line recognition device 5 can measure the positional relationship between the object and the user (driver). Furthermore, it is possible to expect an improvement in accuracy by using the line-of-sight information in time series.

Here, a specific method for calculating the coordinates of the search object using the vehicle position (position of the moving body) and the user's line-of-sight recognition will be described with an example.
FIG. 6 is a schematic explanatory diagram illustrating the positional relationship between the user's eyeball position, the line-of-sight recognition device, and the search target. In this figure, two reference numerals 21 indicate the left and right eyeballs of the user 20, reference numeral 22 indicates an eyeball position measurement area, and reference numeral 23 indicates a gaze-measurable direction.

The line-of-sight recognition device 5 can measure the distance L1 to the eyeball position A by a stereo camera or pattern light and image recognition. Further, the calibration plane B is set at a certain distance L2 from the line-of-sight recognition device 5, and calibration is performed.
Then, the direction in which the left and right eyeballs 21 of the user 20 are facing is measured using infrared rays and image recognition (for example, corneal reflection method). As a result, since the position and direction of the left and right eyeballs 21 are known, the line-of-sight direction vector can be calculated for the left and right eyeballs.

Here, when the distance between the left and right eyeballs 21 of the user 20 is d1 and the parallax on the calibration plane B is d2, the distance X between the eyeball position A and the search object 80 (object position C) is the user's 20 eyeball. Using the distance (L1 + L2) between the position A and the calibration plane B, it can be calculated by the following equation.
X = (d1 / (d1-d2)) × (L1 + L2)

Further, since the eyeball position and the line-of-sight position are known, a method of calculating the intersection coordinates from the vector using the outer product may be used.
Although the distance can be calculated from the above, the accuracy is not sufficient. However, since the accuracy of the direction is high, it is only necessary to increase the accuracy of the distance to the search object by using the direction mainly and using a distance measuring sensor such as a front camera or millimeter wave in combination.

On the other hand, the line-of-sight recognition device 5 is assembled and fixed to a vehicle (moving body). Moreover, map information, geomagnetic sensor information, highly accurate GPS information, etc. can be acquired. The coordinates of the line-of-sight recognition device 5 are regarded as the same as the GPS information, and the direction of the vehicle is measured by a geomagnetic sensor.
Since the distance and direction from the search object 80 can be measured as described above, the coordinates of the search object 80 are calculated by adding the distance and direction to the GPS information.

The coordinate calculation unit 12 acquires information such as the vehicle speed, the steering angle, whether the vehicle is traveling, whether the vehicle is traveling, or the like as shown in FIG. Here, the vehicle-mounted device 7 includes at least a device that detects the speed of the moving body, and the acquired vehicle speed and steering angle are used for specifying the search range. The determination of whether or not the vehicle is traveling or whether or not the vehicle is traveling is used to limit travel such as making it impossible to search during traveling.

Further, the coordinate calculation unit 12 detects the distance from the sensor 6 such as a camera or radar that detects the peripheral information of the moving object to the target as shown in FIG. 8, character recognition of the target, and the target that is not a facility. Get information such as specific (such as the car you are driving). Here, for example, when a radar is used, it is possible to measure the distance to the search object, and when a camera is used, it is possible to recognize characters and logos drawn on the search object. It is also used to determine whether a facility other than those described on the map, such as cars and motorcycles, is designated.

In this way, the coordinate calculation unit 12 receives the information from the GPS 4, the line-of-sight recognition device 5, the sensor 6 such as the camera, the in-vehicle device 7 and the notification from the search request determination unit 11, and calculates the position coordinates of the search target. To do. For example, when a search request is received from the search request determination unit 11, the coordinates of the vehicle position are calculated by the GPS 4, and the direction and distance of the object whose line of sight is directed by the line-of-sight recognition device 5 are measured. Thereby, the coordinates of the search object are calculated.

In addition, the information of the sensor 6 such as the camera and the in-vehicle device 7 is used for improving accuracy. In this case, it is described that a plurality of information is used in combination. Also good. In addition, when there is overlapping information or conflicting information, a mechanism may be provided in which each is tabulated and scored to determine whether it is useful or not useful.

That is, when the search request determination unit 11 determines that there is a search request, the coordinate calculation unit 12 is based on at least the current position of the moving object acquired from the GPS 4 and the user's line-of-sight recognition by the line-of-sight recognition device 5. The coordinate position on the map is calculated using the object in the line-of-sight direction as the search object.

The superimposition display control unit 16 is notified of the search target coordinates (coordinates of the search target object) by the coordinate calculation unit 12, and the search target coordinates (search target object) on the map data displayed on the head-up display (HUD) 18. The HUD 18 is instructed to superimpose a search mark for emphasizing the search object at the position of the coordinate.

The search mark for emphasizing the search object includes a mark displayed with a frame such as a rectangle or a circle, a mark with an arrow, etc., a building on the map data displayed on the HUD 18, and a search object based on line-of-sight recognition. The mark may have any shape as long as the user can confirm both the position and the position, and the user can recognize which building or the like is the search target.

The HUD 18 is a display device that is displayed so as to overlap the user's (driver's) field of view, and receives an instruction from the superimposed display control unit 16 on a building, a road, or the like in the user's field of view, A search mark that highlights the target object is displayed in a superimposed manner. In addition, although this Embodiment 1 demonstrates as a head-up display (HUD), if it can display various information on a user's visual field, it will be a head mounted display (HMD) etc., for example. Also good.

In addition, the search processing unit 13 receives a notification of the search target coordinates (coordinates of the search target object) from the coordinate calculation unit 12, and based on the map data stored in the storage unit 14, the facility near the search target coordinates Information, that is, detailed information of the search object is searched. When a facility is specified by voice or the like when searching, the search condition may be narrowed down. If a search result exists, the search processing unit 13 notifies the output information creating unit 15 of facility information for user (driver) notification. If the facility search result does not exist, the output information creation unit 15 is notified that the facility search result does not exist.

Here, when there are a plurality of facility search results, the facility closest to the search target coordinates is notified as a search result. In this case, for example, when a facility next to the search target is notified as a search result, the user (driver) uses the voice recognition device 1 to specify “facility in front”, “inner facility”, etc. Then, an additional search may be performed.

The output information creation unit 15 creates voice information or display information as output information of the facility information (detailed information of the search target) searched by the search processing unit 13. In addition, when there is no search result, a voice or display character string of a phrase such as “no search result” is created.
The output device 8 is a device that outputs or displays the information created by the output information creation unit 15 by voice. In the case of display output, the HUD 18 may be used as the output device 8.

The flow of search processing in the first embodiment will be described with reference to the flowcharts shown in FIGS. These processes are started when the key is turned on, and are ended when the key is turned off.
FIG. 9 is a flowchart showing input recognition processing performed by each of the voice recognition device 1, the switch 2, and the gesture recognition device 3.

First, the speech recognition process shown in FIG. 9A will be described. If some kind of search trigger has already been applied and a search is being performed, or if a voice is being output (YES in step ST1), the voice recognition process is not performed. If the search is not being performed and the voice is not being output (NO in step ST1), the voice information is acquired as a waveform (step ST2), and the voice information is recognized from the acquired waveform (step ST3). Since the voice recognition method is a known technique, a description thereof is omitted here, but any of various methods may be used.

Then, it is determined whether or not a search trigger word is included in the recognized voice information (step ST4). If a search trigger word is included (YES in step ST4), a search request notification is executed (step ST5). On the other hand, if the search trigger word is not included (NO in step ST4), the process returns to the beginning without any processing.

Next, the switch recognition process shown in FIG. 9B will be described. In this case as well, when some sort of search trigger has already been applied and the search is being performed, or during voice output (in the case of YES in step ST11), the switch recognition process is not performed (switch input is not accepted). If the search is not being performed and the voice is not being output (NO in step ST11), the switch information is recognized (step ST12).

Then, it is determined whether or not the search request switch has been pressed (step ST13). When the search request switch is pressed (YES in step ST13), a search request notification is executed (step ST14). At this time, if the switch has a role, the role of the pressed switch is also added and notified. On the other hand, if the search request switch is not pressed (NO in step ST13), the process returns to the beginning without performing any processing.

Finally, the gesture recognition process shown in FIG. 9C will be described. Also here, if some kind of search trigger has already been applied and a search is being performed, or if a voice is being output (YES in step ST21), the gesture recognition process is not performed. If the search is not being performed and the voice is not being output (NO in step ST21), gesture information such as camera information and an optical sensor is acquired (step ST22), and what gesture is performed is recognized (step ST22). Step ST23). Since the gesture recognition method is a known technique, description thereof is omitted here, but any of various methods may be used.

Then, it is determined whether or not a search trigger gesture is included in the recognized gesture information (step ST24). When a search trigger gesture is included (YES in step ST24), a search request notification is executed (step ST25). On the other hand, if the search trigger gesture is not included (NO in step ST24), the process returns to the beginning without any processing.

As described above, when the search request notification is executed by any of the voice recognition process, the switch process, and the gesture recognition process, the process proceeds to the coordinate calculation process in the coordinate calculation unit 12.
FIG. 10 is a flowchart showing coordinate calculation processing in the coordinate calculation unit 12.
First, current location information is acquired from the GPS 4 (step ST31). Thereby, the own vehicle position can be grasped.

Next, line-of-sight information is acquired from the line-of-sight recognition device 5 (step ST32). Also, sensor information is acquired from the sensor 6 such as the front camera or radar (step ST33). The direction and distance that the user (driver) is looking at is calculated from the line-of-sight information, and the accuracy of the distance is improved by the sensor information.

Then, the coordinates of the search object are calculated based on the vehicle position information, direction, and distance obtained from the current location information, line-of-sight information, and sensor information (step ST34).
When the coordinate calculation of the search object is completed, the search coordinate is notified to the superimposition display control unit 16 and the search processing unit 13 (step ST35).

Here, in addition to the current position of the moving body and the user's line-of-sight recognition, based on sensor information acquired from a sensor that detects peripheral information of the moving body and at least information acquired from an in-vehicle device that detects the speed of the moving body In the above description, the coordinate position of the search object is calculated with high accuracy. However, the coordinate position of the search object can be calculated if there is at least the current position of the moving object and the user's line-of-sight recognition. That is, based on at least the current position of the moving object and the user's line-of-sight recognition, the coordinate position on the map may be calculated using an object in the user's line-of-sight direction as a search target.

When the notification of the search coordinates is performed, the process proceeds to the superimposed display processing in the superimposed display control unit 16 and the search processing in the search processing unit 13.
FIG. 11 is a flowchart showing a superimposed display process in the superimposed display control unit 16.
First, based on the coordinate data of the search object included in the notified search coordinates, map information (facility information), object position (search coordinates), and the like are extracted from the map data stored in the storage unit 14. Are matched (step ST41).

If there is a facility or the like that is the object at the object position (search coordinate) (YES in step ST42), the HUD 18 is instructed to display the search mark superimposed, that is, the search object is a rectangular object. It is superimposed and displayed so as to be surrounded by search marks (step ST43).

As a result, there is an advantage that the user does not have to add a word for specifying a facility or the like to search for and limit the conditions, for example, “What is the white building on the right side?” is there. Further, the user can obtain detailed information while visually confirming that the object to be searched is surrounded by a rectangle or the like (a search mark is superimposed on the object).

FIG. 12 is a flowchart showing search processing in the search processing unit 13.
First, the facility information at the search coordinates is read out from the map data stored in the storage unit 14 based on the coordinate data of the search object included in the notified search coordinates (step ST46).

At this time, as facility information in the search coordinates, here, information on peripheral facilities within a predetermined range from the search coordinates is read out. In addition to this, for example, when a facility exists at a position corresponding to the search coordinates Alternatively, the facility information may be read, and if there is no facility at the position corresponding to the search coordinates, the facility information existing at the position closest to the search coordinates may be read. Further, when the facility information in the search coordinates is read, if there is a limitation of the search facility by voice or the like, the search may be performed based on that.

When the facility information at the search coordinates exists as a search result, the search result is added to the output information creation unit 15 to notify the output of the search result. When the facility information at the search coordinates does not exist Adds information that the search result does not exist, and notifies the output information creation unit 15 of the search result output (step ST47).

The output information creation unit 15 generates, from the search result information notified from the search processing unit 13, information indicating that the searched facility information or the search result does not exist as a voice or a display character string, and outputs it to the output device 8. Output. That is, when the output device 8 is a sound output device, a sound is generated and output as a sound, and when the output device 8 is a display device, a display character string is generated and displayed.

Here, a specific processing flow will be described with reference to an image diagram of FIG.
FIG. 13 is a diagram illustrating a transition image example of processing of the information processing apparatus according to the first embodiment.
For example, as shown in FIG. 13A, it is assumed that a user (driver) sees a building indicated by a line of sight 81 and utters "What is that building?"

At this time, if neither search nor voice output is being performed (NO in step 1 in FIG. 9A), the voice recognition apparatus 1 acquires voice information as the utterance content and performs voice recognition (steps ST2 to ST3). ). As a result of referring to the table shown in FIG. 2, it is determined that the search trigger word is included (in the case of YES in step ST4), the search request is notified (step ST5).

And the coordinate calculation part 12 is the present location information from GPS4, the gaze information from the gaze recognition apparatus 5 (information regarding the gaze position 81 in FIG. 13A), the information from the sensor 6 such as the camera, and the in-vehicle device. Based on the information from 7, the position coordinates of the search object 80 are calculated, and the search coordinates are notified to the superimposed display control unit 16 and the search processing unit 13 (steps ST31 to ST35 in FIG. 10).

When the superimposition display control unit 16 receives the search coordinates of the search object from the coordinate calculation unit 12, it refers to the map data stored in the storage unit 14 and maps information (facility information) and the object position (search coordinates). ) And instructing the HUD 18 to superimpose and display the search mark (steps ST41 to ST43 in FIG. 11).

As a result, for example, as shown in FIG. 13B, an object in the line of sight of the user (driver) is received on the building or road displayed on the HUD 18 in response to an instruction from the superimposed display control unit 16. A search mark 85 highlighted is displayed in a superimposed manner. Here, a rectangular frame-shaped mark is adopted as the search mark 85.

Further, when the search processing unit 13 receives the search coordinates of the search object from the coordinate calculation unit 12, the search processing unit 13 refers to the map data stored in the storage unit 14, reads the facility information in the search coordinates, and outputs the output information generation unit 15 is notified of search result output (steps ST46 to ST47 in FIG. 12).

As a result, the output information creation unit 15 generates a voice or a display character string for outputting the facility information at the search coordinates. For example, as shown in the voice output or display output 83 of FIG. The voice or display character string “is a bank” is output to the output device 8 (voice output device or display device) to present it to the user (driver).

Here, for example, as shown in the utterance 82 of FIG. 13B, if the user (driver) further utters “What is business hours?”, The speech recognition device 1 recognizes the speech, and the search processing unit 13 further reads out the information of the previous search object (facility information at the search coordinates) and, as shown in the audio output or display output 83 ′ of FIG. 13C, “business hours are from 9:00 AM to PM 5:00 ”is output to the output device 8 (speech output device or display device) and presented to the user (driver).

In this way, when a user (driver) wants to search for detailed information on a facility such as a building in the vicinity of his / her vehicle position, he / she stops and opens the search screen and selects a genre. There is no need to use a complicated procedure for the search procedure, just by pointing the line of sight to the search object and making a search request such as utterance, you can easily identify the search object and search for detailed information, Present it to the user.

Further, when the line-of-sight position 81 is slightly deviated from the search object 80, for example, facility information at a position close to the search coordinates of the line-of-sight position 81 is presented, or a user (driver) from among a plurality of presentations. The facility information of the target object is presented by allowing the user (driver) to speak “before” or “behind” when information on different facilities is presented. Can be.

As described above, according to the first embodiment, a user can easily specify a search target by detecting a user's line of sight in a moving body such as a vehicle, and superimpose it on the user's field of view such as a HUD (head-up display). Since it can be displayed on the display on which information is displayed, the detailed information of the search object can be efficiently presented to the user.
In addition, since a facility in the line-of-sight direction can be pinpointed with a simple trigger such as a voice, a switch, or a gesture, a conventionally troublesome search can be facilitated.

Embodiment 2. FIG.
FIG. 14 is a block diagram showing an example of an information processing apparatus according to Embodiment 2 of the present invention. In addition, the same code | symbol is attached | subjected to the structure similar to what was demonstrated in Embodiment 1, and the overlapping description is abbreviate | omitted. The second embodiment described below further includes a search start button (search start instruction input unit) 31 and a search determination button (search determination instruction input unit) 32 as compared with the first embodiment.

In the second embodiment, as a search start button (search start instruction input unit) 31 and a search determination button (search determination instruction input unit) 32, for example, a physical button is provided on a handle or the like. As will be described, a software menu button or the like displayed on the touch panel may be used. Further, although it is physically one button, it may have a structure in which a process of starting a search when the button is half-pressed and determining a search when the button is deeply pressed is performed.

The search request determination unit 11 can determine a search request by the user from information obtained from any of the speech recognition device 1, the switch 2, and the gesture recognition device 3 as in the first embodiment. It may be determined that there is a search request by pressing the start button 31.
When the search request determination unit 11 determines that there is a search request, the coordinate calculation unit 12 recognizes the line of sight of the user (driver) only while the search start button 31 is being pressed by the user. The coordinate position on the map is calculated using an object in the line-of-sight direction as a search target.

That is, when the user wants to start a search while traveling, when the search start button 31 is pressed, the search request determination unit 11 determines that there is a search request, and the coordinate calculation unit 12 determines a search target from various information. The coordinates of the object are calculated, and the search mark is superimposed.
This is dynamically executed in real time while the search start button 31 is pressed. Therefore, if the line of sight moves to another object while pressing the search start button 31, the search mark is also displayed on the other object.

In addition, when the user presses the search determination button 32, the search processing unit 13 searches for detailed information on the search target object with the search mark superimposed on the head-up display (HUD) 18.
That is, the user presses the search determination button 32 while confirming that the search mark matches the object to be searched, and the search processing unit 13 determines that the search determination button 32 is pressed. To output the search results.

Here, a specific processing flow will be described with reference to the image diagram of FIG. 13 in the first embodiment. Note that the flowchart is only added with a process for determining the pressed state of the search start button 31 and the search determination button 32, and is basically the same as in FIGS. 9 to 12 in the first embodiment. Illustration is omitted.

For example, as shown in FIG. 13A, the user (driver) keeps pressing the search start button 31 while looking at the building indicated by the line of sight 81 while the vehicle is traveling.
As described above, when a search request is received by pressing the search start button 31, utterance or voice recognition is not necessary, but using the specific example (FIG. 13) for performing voice recognition in the first embodiment. Similarly to the explanation, when the user looks at the building indicated by the line of sight 81 and utters “What is that building?” As shown in the utterance 82 in FIG. If the search start button 31 is desired, the search start button 31 is continuously pressed.

And the coordinate calculation part 12 will be the present location information from GPS4, the gaze information from the gaze recognition apparatus 5 (information regarding the gaze position 81 of Fig.13 (a)), if the search start button 31 is pressed. Based on the information from the sensor 6 such as the camera and the information from the in-vehicle device 7, the position coordinates of the search object 80 are calculated, and the search coordinates are notified to the superimposition display control unit 16 and the search processing unit 13 ( Steps ST31 to ST35 in FIG.

As a result, for example, as shown in FIG. 13B, an object in the line of sight of the user (driver) is received on the building or road displayed on the HUD 18 in response to an instruction from the superimposed display control unit 16. The search mark 85 shown is superimposed and displayed. At this time, if the search mark 85 is superimposed on the object that the user wants to search, the user presses the search determination button 32.

When the search processing unit 13 receives the search coordinates of the search object from the coordinate calculation unit 12, when the search determination button 32 is pressed, the search processing unit 13 refers to the map data stored in the storage unit 14. Then, the facility information at the search coordinates is read, and the search result output is notified to the output information creation unit 15 (steps ST46 to ST47 in FIG. 12).

Here, if the search determination button 32 is further pressed, the search processing unit 13 further reads the information of the previous search object (facility information at the search coordinates), and outputs the voice or display in FIG. 13C. As shown in 83 ', a voice or a display character string "business hours are from 9:00 am to 5:00 pm" is output to the output device 8 (voice output device or display device) to the user (driver). Present.

If voice recognition is also involved, for example, as shown by an utterance 82 in FIG. 13B, if the user (driver) further utters “What is business hours?”, The voice recognition device 1 transmits the voice. Recognizing, the search processing unit 13 further reads and processes the information of the previous search object (facility information at the search coordinates). Even in this case, it is assumed that the search determination button 32 is pressed. It is.

In addition, since the search object can be confirmed on the HUD 18 before the search result (detailed information of the search object) is presented, the user can determine which object has been searched since the search result was presented. The problem that (driver) has no choice but to recognize is solved.

Further, when the line-of-sight position 81 is slightly deviated from the search object 80, the user (driver) can perform a search by using the search determination button 32 while visually confirming, so the user (driver) ) Can also be resolved by re-searching by saying "before" or "behind".

As described above, according to the second embodiment, in addition to the effects in the first embodiment, the user can perform a search while visually confirming the search target object, and thus an unintended target object is searched. Thus, a desired object can be searched in real time and reliably, and detailed information of the search object can be efficiently presented to the user.

Embodiment 3 FIG.
FIG. 15 is a block diagram showing an example of an information processing apparatus according to Embodiment 3 of the present invention. In addition, the same code | symbol is attached | subjected to the structure similar to what was demonstrated in Embodiment 1, 2, and the overlapping description is abbreviate | omitted. In the third embodiment described below, as compared with the first embodiment, the search processing unit 13 acquires information from the GPS 4, the sensor 6, and the in-vehicle device 7, and also performs a peripheral search process for searching for a peripheral facility in the current location. In addition, a display device is provided as the output device 8.

The search processing unit 13 searches for a specific facility (for example, a bank, a convenience store, a gas station, etc.) around the vehicle position by using voice, a switch, a GUI (Graphic User Interface), or the like. The peripheral facility as a result of the search is stored in the storage unit 14, and information is sent to the output information creation unit 15 to be superimposed on the output device (display device) 8.

Here, the output device (display device) 8 will be described assuming a head-up display. That is, the head-up display (HUD) 18 and the output device (display device) 8 in FIG. 15 are the same. A specific peripheral facility that is a result of the search by the search processing unit 13 can be displayed according to the viewpoint. The user (driver) viewpoint information can be acquired from the line-of-sight recognition device 5.

That is, the search processing unit 13 searches for a peripheral facility of a mobile body specified in advance as a search point, and the output information creation unit 15 generates output information so that the search point searched by the search processing unit 13 is displayed. To do.

This allows the user (driver) to check the location of the surrounding search results while traveling. However, if a large amount of complicated information is presented on the head-up display (HUD) 18, there is a possibility that the driving may be hindered. Therefore, when a large number of information is displayed, it is realistic to use a mark or a simple word.

The user (driver) can request detailed information by directing his / her line of sight to the mark displayed by the peripheral search result. The detailed information may be displayed on a head-up display (HUD) 18 or output by voice. Here, the output device 8 is assumed to be a head-up display (HUD) 18 which is a display device. explain. In addition, when performing display on the head-up display (HUD) 18, it shall be held at a level that does not hinder driving.

FIG. 16 is a diagram illustrating a transition image example of the processing of the information processing apparatus according to the third embodiment.
For example, when the user (driver) speaks or inputs “bank” as a specific facility around the vehicle position, as shown in FIG. The search point 84 is displayed on the output device (display device) 8 while being displayed. The vicinity search may be input by voice or may be input by a touch panel or the like.

Since the subsequent search processing flow is the same as the flowchart shown in FIGS. 9 to 12 in the first embodiment, illustration and detailed description thereof will be omitted. In the third embodiment, the search request may be notified by any of the voice recognition device 1, the switch 2, or the gesture recognition device 3, but here the search request is pressed by pressing the switch 2. It explains as what notifies.

First, if neither searching nor voice output (NO in step 11 of FIG. 9B), the switch 2 acquires switch pressing information (step ST12). When it is determined that the search switch serving as a search trigger is pressed (YES in step ST13), a search request is notified (step ST14).

And the coordinate calculation part 12 is the present location information from GPS4, the gaze information from the gaze recognition apparatus 5 (information regarding the gaze position 81 of FIG.16 (b)), the information from sensors 6 such as a camera, and vehicle equipment 7 is used to calculate the position coordinates of the search object 80 and notify the search processing unit 13 of the search coordinates (steps ST31 to ST35 in FIG. 10).

When the search processing unit 13 receives the search coordinates of the search object from the coordinate calculation unit 12, the search processing unit 13 refers to the map data stored in the storage unit 14, reads the facility information at the search coordinates, and outputs it to the output information creation unit 15. The search result output is notified (steps ST46 to ST47 in FIG. 12).

As a result, the output information creation unit 15 generates a display character string for outputting the facility information at the search coordinates. For example, as shown in the display output 83 in FIG. Alternatively, the display character string is output to the output device 8 (speech output device or display device) and presented to the user (driver).

Here, when it is desired to search for further detailed information, for example, as shown in the display output 83 of FIG. 16C, the user (driver) sees the line-of-sight detection position 81 for searching for further detailed information. The search processing unit 13 further reads the information of the previous search object 80 (facility information at the search coordinates), and as shown in the display output 83 of FIG. 16D, “business hours are from 9:00 am to pm 5:00 ”is output to the output device (display device) 8 to present it to the user (driver).

As described above, when a user (driver) wants to search for detailed information on a facility such as a building around the vehicle position, a specific search point designated by the user (driver) is displayed in advance. Then, simply point the line of sight to the search object with the search point mark displayed and make a search request such as utterance, then you can easily identify the search object and search for detailed information and present it to the user. Give me. Thereby, as compared with the first embodiment, it is possible to more reliably specify the search object and obtain detailed information.

Note that the output information creation unit 15 determines that the search point that matches when the line-of-sight position used when the coordinate calculation unit 12 calculates the coordinates of the search target that the user wants to search matches with any of the search points 84. The output information may be generated so that 84 is highlighted. That is, for example, as shown in FIG. 16B, when the line of sight 81 overlaps the search point 84 displayed on the search object 80, the search point 84 is displayed in a large size or displayed in a color-changed manner. You may make it display.

FIG. 17 is a flowchart illustrating a process of presenting detailed information using line-of-sight recognition in the information processing apparatus according to the third embodiment.
First, it is determined whether or not the search point 84 is marked in a range visible to the user (driver) by the peripheral search (step ST51). If not marked (NO in step ST51), the process continues to wait until the mark of the search point 84 appears.

On the other hand, when the facility mark (search point 84) is displayed (YES in step ST51), line-of-sight information is acquired (step ST52).
Subsequently, it is determined whether or not the facility detailed information 83 is displayed (step ST53). If the facility detailed information 83 is not displayed (YES in step ST53), it is determined from the line-of-sight information whether or not the facility mark (search point 84) has been viewed for a certain period of time (step ST54).

If the user has been viewing for a certain period of time (in the case of YES in step ST54), the detailed information 83 of the search object having the facility mark (search point 84) pointing at the line of sight is displayed (step ST55). On the other hand, if it is not determined that the facility mark (search point 84) has been viewed for a certain period of time (NO in step ST54), the process returns to the first step ST51.

If the facility detail information 83 is displayed in step ST53 (NO in step ST53), it is determined whether or not the displayed facility detail information 83 has been viewed for a certain period of time (step ST56). ). If the user is viewing for a certain period of time (YES in step ST56), further facility detailed information 83 'is displayed (step ST57). On the other hand, if it is not determined that the facility detailed information 83 has been viewed for a certain period of time (NO in step ST56), the displayed facility detailed information 83 is hidden (step ST58).

As described above, according to the third embodiment, in addition to the effects of the first and second embodiments, the position of a specific peripheral facility to be searched for in a moving body such as a vehicle is displayed in an easy-to-understand manner, and When the user wants to know the detailed information, the search object can be easily specified by detecting the user's line of sight, so that the detailed information of the search object can be efficiently presented to the user.
Moreover, since the user can easily recognize the display by superimposing the display on the display device in the line-of-sight direction, it is possible to search for the surrounding facilities more comfortably.

In the first to third embodiments described above, the description has been made on the premise that the line of sight is directed to the facility outside the vehicle as the search object, but the vehicle-mounted equipment is directed to the vehicle-mounted equipment inside the vehicle. It is also possible to search how to use the equipment as detailed information.
In this case, in-vehicle equipment information such as an information panel, a navigation system, a front camera, an air conditioner, a winker, and a shift lever inside the vehicle is also stored in the storage unit 14.

In recent years, automobiles have been equipped with various functions, but it is doubtful that the user (driver) understands and uses them all. However, by using this invention, complicated usage is assumed. You can easily find out how to use and display the information panel, navigation system, front camera, etc. without opening a thick manual.

At this time, by using the line-of-sight recognition device 5 to visually check the on-vehicle equipment that you want to know the nearer and triggering a search trigger by voice recognition, switch, gesture recognition, etc. Can be output. However, the facilities that can be searched are limited to those in which the user (driver) cannot change the position freely, but the position coordinates of the in-vehicle facilities are basically determined, so that they can be used without much trouble.

Specifically, the user (driver) makes a search request using voice, switches, gestures, etc. while looking at the in-vehicle equipment to be searched. Subsequently, when the search trigger is recognized and a search request is notified, the coordinates of the search target are specified by line-of-sight recognition. And it searches based on a search coordinate, reads the detailed information of the vehicle equipment corresponding to a search coordinate, and outputs information. If there is no corresponding equipment, a message indicating that there is no information is output.

As a result, users (drivers) have no idea how to find out how to use the functions of a car in the past. Can provide a way to easily find out how to use the equipment. As a result, even if the user (driver) does not know all the functions in advance, it is possible to confirm detailed information such as how to use the facility he / she is looking at by simply performing a search with a gaze. Further, when it is explained that the setting can be changed by a button operation, the button for changing the setting may be easily performed by looking at the button for a predetermined time or more.

In the third embodiment, similarly to the second embodiment, a search start button 31 and a search determination button 32 are further provided, and the coordinate position is changed only while the search start button 31 is pressed by the user (driver). Needless to say, the calculation may be performed and the detailed information of the search object may be searched when the search determination button 32 is pressed by the user (driver).

Embodiment 4 FIG.
In the above first to third embodiments, the case where the information processing apparatus according to the present invention is applied to a navigation apparatus mounted on a moving body such as a vehicle has been described as an example. However, the present invention is applied to an in-vehicle navigation apparatus. The navigation device for a moving body including, but not limited to, a person, a vehicle, a railroad, a ship, an airplane, or the like may be used, or the information processing system or the navigation system server may be applied. Further, the present invention can be applied to any form such as an information processing system installed in a portable information terminal such as a smartphone, a tablet PC, or a mobile phone, or an application of a navigation system.

FIG. 18 is a diagram showing an outline of the navigation system according to the fourth embodiment of the present invention. In this navigation system, the in-vehicle device 100 performs search processing and navigation processing in cooperation with at least one of the portable information terminal 101 such as a smartphone and the server 102, or at least one of the portable information terminal 101 such as a smartphone and the server 102. However, the search processing and the navigation processing are performed, and the recognition result and the map information are displayed on the in-vehicle device 100. Hereinafter, a configuration aspect of the navigation system will be described.

In the first to third embodiments, the search processing function is described as being provided in the in-vehicle device 100 shown in FIG. 18, but in the navigation system in the fourth embodiment, the server 102 performs the search processing, and the search result When the mobile information terminal 101 performs search processing in cooperation with the server 102 and displays the search result on the in-car device 100, the user is provided with the user. Will be described.

In this case, the in-vehicle device 100 includes a communication unit capable of communicating with the server 102 using, for example, mobile communication or a wireless LAN. The server 102 includes the search processing unit 13, issues a search request to the external server 102 based on the search target coordinates recognized by the in-vehicle device 100, and the server 102 performs search processing and generates search result output information. Thus, the search result can be presented to the user (driver) by sending it to the in-vehicle device 100. In this case, the storage unit 14 does not need to have map data, and the server 102 only needs to have map data.

First, a case where the server 102 performs a search process and displays the search result on the in-vehicle device 100, that is, a case where the in-vehicle device 100 functions as a display device in cooperation with the server 102 having a search processing function will be described.
In this configuration, it is conceivable that the in-vehicle device 100 communicates directly with the server 102 or the in-vehicle device 100 communicates with the server 102 via the portable information terminal 101. The server 102 has the function of the search processing unit 13 described in the first to third embodiments. The in-vehicle device 100 functions as a display device including at least a display unit for providing a search result by the server 102 to the user.

Further, a case will be described in which the portable information terminal 101 performs a search process in cooperation with the server 102 and the in-vehicle device 100 provides the search result to the user.
In this configuration, the case where the in-vehicle device 100 communicates with the server 102 via the portable information terminal 101 can be considered, and the application of the portable information terminal 101 performs search processing in cooperation with the server 102. The in-vehicle device 100 functions as a display device including at least a display unit for providing a search result by the portable information terminal 101 and the server 102 to the user.

In any case, the in-vehicle device 100 basically has only a communication function and a display function, and receives a search result obtained by cooperation between the portable information terminal 101 and the server 102 and provides it to the user.
That is, the search result of the search object requested by the user is displayed on the in-vehicle device 100 which is a display device by the application of the portable information terminal 101.
Even with this configuration, it is possible to obtain the same effects as in the first to third embodiments.

It is also possible to send the sensor information acquired from the sensor 6 such as a camera to the server and perform a search. For example, it is possible to notify the server of an image obtained from a camera and select the most suitable facility as a search target facility from the result.
Further, by requesting not only the map facility information but also word-of-mouth and comment tags, it is possible to increase the amount of information in the search results.
Furthermore, if there is camera information, it is also possible to search for detailed information on the car or motorcycle when a search is performed with a line of sight directed toward a moving body such as a car or motorcycle.

In the present invention, within the scope of the invention, any combination of the embodiments, or any modification of any component in each embodiment, or omission of any component in each embodiment is possible. .

The information processing device of the present invention is not limited to a vehicle-mounted navigation device, but includes a navigation device for a moving body including a person, a vehicle, a railway, a ship, an aircraft, etc., a portable navigation device, a portable information processing device, etc. In addition, the present invention can be applied to navigation system applications installed in mobile information terminals such as smartphones, tablet PCs, and mobile phones.

1 voice recognition device, 2 switch, 3 gesture recognition device, 4 GPS, 5 line-of-sight recognition device, 6 sensors, 7 in-vehicle equipment, 8 output device (voice output device or display device), 10 navigation device, 11 search request determination unit, 12 coordinate calculation unit, 13 search processing unit, 14 storage unit, 15 output information creation unit, 16 superimposed display control unit, 18 head-up display (HUD), 20 user, 21 eyeball of

user

20, 22 eyeball position measurement area, 23 Gaze measurable direction, 31 Search start button (search start instruction input part), 32 Search decision button (search decision instruction input part), 80 Search object, 81 Gaze position, 82 Utterance (utterance content), 83, 83 'voice Output or display output (detailed facility information), 84 search points, 85 search marks , 100-vehicle device, 101 a portable information terminal, 102 server.

Claims

In an information processing apparatus based on gaze detection of a user who uses a moving object,
A display for displaying various types of information superimposed on the user's field of view;
A search request determination unit for determining a search request by the user;
When the search request determination unit determines that there is a search request, on the map, an object in the user's line-of-sight direction is selected as a search target based on at least the current position of the mobile object and the user's line-of-sight recognition. A coordinate calculation unit for calculating a coordinate position;
Based on the coordinate position of the search object calculated by the coordinate calculation unit and map data, a superimposition display control unit that superimposes a search mark that highlights the search object on the display;
A search processing unit that searches for detailed information of the search object based on the coordinate position and map data of the search object calculated by the coordinate calculation unit;
An information processing apparatus comprising: an output information generating unit that generates output information for presenting detailed information of the search object searched by the search processing unit to the user.
A search start instruction input unit for instructing the coordinate calculation unit to start a search;
A search determination instruction input unit that instructs the search processing unit to determine a search;
The coordinate calculation unit calculates the coordinate position only while the search start instruction input unit is pressed,
2. The information according to claim 1, wherein the search processing unit searches for detailed information of a search object in which the search mark is superimposed on the display when the search determination instruction input unit is pressed. Processing equipment.
The information processing apparatus according to claim 1, wherein the coordinate calculation unit further calculates a coordinate position of the search object based on information acquired from a sensor that detects peripheral information of the moving object.
The information processing apparatus according to claim 1, wherein the coordinate calculation unit further calculates a coordinate position of the search object based on at least information acquired from a device that detects the speed of the moving object.
The search request determination unit determines whether or not there is a search request based on recognition of a voice spoken by the user, a switch pressed by the user, or recognition of a gesture operated by the user. The information processing apparatus according to claim 1, wherein the determination is performed.
The search processing unit searches for a peripheral facility of the mobile body specified in advance as a search point,
The information processing apparatus according to claim 1, wherein the output information creation unit generates output information so as to display the search points searched by the search processing unit.
The output information creation unit, when the user's line-of-sight position used when the coordinate calculation unit calculates the coordinate position of the search target that the user wants to search matches with any of the search points, The information processing apparatus according to claim 6, wherein the output information is generated so that the retrieved search points are highlighted.
An information processing apparatus that causes a display device to display a search result based on gaze detection of a user who uses a mobile object,
A display for displaying various types of information superimposed on the user's field of view;
A search request determination unit for determining a search request by the user;
When the search request determination unit determines that there is a search request, on the map, an object in the user's line-of-sight direction is selected as a search target based on at least the current position of the mobile object and the user's line-of-sight recognition. A coordinate calculation unit for calculating a coordinate position;
Based on the coordinate position of the search object calculated by the coordinate calculation unit and map data, a superimposition display control unit that superimposes a search mark that highlights the search object on the display;
A search processing unit that searches for detailed information of the search object based on the coordinate position and map data of the search object calculated by the coordinate calculation unit;
An output information generating unit that generates output information for presenting detailed information of the search object searched by the search processing unit to the user, and causes the display device to display the generated output information. An information processing apparatus characterized by that.
An information processing method is an information processing method for causing a display device to display a search result based on gaze detection of a user who uses a moving body,
A search request determining unit determining a search request by the user;
When the coordinate calculation unit determines that there is a search request by the search request determination unit, it searches for an object in the user's line-of-sight direction based on at least the current position of the mobile object and the user's line-of-sight recognition Calculating a coordinate position on the map as a target;
The superimposed display control unit emphasizes the search target on a display that displays various information superimposed on the user's field of view based on the coordinate position of the search target calculated by the coordinate calculation unit and map data. A step of superimposing a search mark;
A search processing unit searching for detailed information of the search target based on the coordinate position and map data of the search target calculated by the coordinate calculation unit;
An output information creating unit generating output information for presenting detailed information of the search object searched by the search processing unit to the user, and displaying the generated output information on the display device; An information processing method characterized by comprising: