JP4616786B2 - Information provision device - Google Patents

Description

  The present invention relates to an information providing apparatus that provides information for guiding a vehicle to a driver who drives the vehicle.

Non-Patent Document 1 below describes a route guidance device that retrieves an appropriate route from a current location to a destination by setting a destination and guides a vehicle along the retrieved route. Basically, it is possible to reach the destination by driving along a route guided (guided) by such a route guidance device.
http://www.pioneer.co.jp/carrozzeria/index_navi.html Cognitive behavioral state estimation based on psychophysiological data (http://criepi.denken.or.jp/jp/hfc/DB/Bunken/1992-01556.html) Introduction to Physiological Experiments 2nd Measurement of Visual System Indicators, Hiroshi Shimoda (Kyoto University Graduate School), Journal of Human Interfaces Vol.7, No.2 2005 (http://hydro.energy.kyoto-u.ac.jp/Lab /staff/shimoda/lecture/hipp/ocular2.pdf) (http://hydro.energy.kyoto-u.ac.jp/Lab/staff/shimoda/lecture/hipp/p-psychology1.pdf) Psychology of Pupil Movement Katsuya Matsunaga 1990.9 Nakanishiya Publishing

  However, beginners who are new to driving or those who are not familiar with driving, even if the route is guided by a conventional route guidance device, may pass by losing sight of intersections and landmarks that are provided in the route guidance process. Or, it is inconvenient for the driver (hereinafter also referred to as a driver) because he / she feels uneasy about not being able to find the landmark even though the location is in front of the intersection of the landmark. In addition, when a vehicle is lost in a certain place, the traveling history of the other vehicle after getting lost is not utilized even though there is another vehicle lost in the same place in the past. In addition, if the driver is driving based on the map or his / her memory and the destination is not set using a route guidance device, the route will not be guided. It was troublesome because the destination had to be newly set to the device and the route guided.

  The present invention is intended to solve the above-described problem, and when the driver loses track of the route or gets lost, the driver has a purpose suitable for the driver without performing the route setting operation again. It is an object of the present invention to provide an information providing apparatus capable of providing information on a ground candidate.

  In order to achieve the above object, according to the present invention, line-of-sight detection means for detecting the line-of-sight direction of the driver for each face image based on a plurality of face images of the driver of the vehicle, and the position of the vehicle Based on the gaze direction of the driver for each face image detected by the gaze detection unit, the position information acquisition unit that acquires information, and calculates the amount of movement of the line of sight due to the driver's eye movement. It is acquired by the position information acquisition means when it is determined whether or not the driver is confused on the travel route based on the movement amount of the line of sight, and when it is determined that the driver is confused. Determination / extraction for extracting a series of the driver's line-of-sight direction based on the position information of the vehicle and the driver's line-of-sight direction in the plurality of face images used in the determination. Means, A series of line-of-sight directions when each of a number of drivers has lost his / her route in the past, information on a destination that has arrived after the occurrence of the lost, and from the point where the lost has occurred to the destination The line-of-sight direction sequence extracted by the determination / extraction means in a database that previously creates and stores history information consisting of route information to the destination including target information that serves as landmarks in route guidance, Processing means for generating search request information for a search request for whether there is the history information including the line-of-sight direction sequence having a predetermined correlation, and transmitting the search request information to an external means for performing the search; and information The history information searched by the display means for displaying and the external means for performing the search is received, and the destination information and the hesitation occur based on the received history information Information providing apparatus and a display control means for displaying on said display means the routing information to the arrival place as recommendation information including target information as a landmark in the route guidance to the arrival place from the point is provided. With this configuration, when the driver loses track or gets lost, the driver can be provided with information on suitable destination candidates without causing the driver to perform the route setting operation again. . The predetermined storage area here is a concept including a storage medium such as a DVD (Digital Versatile Disk) in addition to the semiconductor memory.

  Further, in the information providing apparatus of the present invention, the determination / extraction means further adds information on a gaze stop time in which the driver's gaze is stopped to the calculated information on the movement amount of the gaze, and the driving It is a preferable aspect of the present invention to determine whether or not the person is confused about the travel route. With this configuration, it is possible to determine whether or not the driver is at a loss in the travel route.

  Further, in the information providing apparatus of the present invention, from the series of the line-of-sight direction when the driver has lost his / her travel route in the past, the information on the destination reached after the occurrence of the lost, the point where the lost has occurred It further comprises storage means for preliminarily creating and storing history information consisting of route information to the destination including target object information that becomes a landmark in route guidance to the destination, and the processing means includes the determination / extraction means The history information including the line-of-sight direction sequence having a predetermined correlation with the line-of-sight direction sequence extracted by the above is searched for in the storage means, and if the history information is not found, the search is performed. It is a preferred aspect of the present invention to generate search request information for a request and send it to the external means for performing the search. With this configuration, history information can be obtained earlier.

  In the information providing apparatus of the present invention, an operation amount acquisition unit that acquires an operation amount of each operation performed by the driver to drive the vehicle, and the operation amount acquired by the operation amount acquisition unit. Based on the driving operation determination means for determining whether or not the driver is at a loss based on the driver, the result determined by the driving operation determination means and the result determined by the determination / extraction means, It is a preferable aspect of the present invention to further include an integrated determination unit that finally determines whether or not the driver is at a loss in the travel route. With this configuration, it is possible to determine whether or not the driver is at a loss in the travel route. The operation amount of each operation is an operation amount necessary for driving the vehicle, for example, a steering amount indicating how much the steering wheel is rotated, an accelerator amount indicating how much the accelerator is depressed, This refers to the amount of brake that indicates how much the brake has been depressed.

  In the information providing apparatus of the present invention, it is preferable that the amount of movement of the line of sight is calculated for each of the jumping eye movements from the line of sight at the start and end of the jumping eye movements. It is. With this configuration, it is possible to determine whether or not the driver is at a loss in the travel route.

  In the information providing apparatus of the present invention, the line-of-sight direction series includes the position information of the vehicle acquired by the position information acquisition unit, and the plurality of face images used in the determination. Information on objects existing around the travel route viewed by the driver, extracted based on the driver's line-of-sight direction and map information stored in a predetermined storage area or obtained from the outside. It is a preferable aspect of the present invention that the object information sequence is a sequence. With this configuration, more appropriate history information can be searched.

  The information providing apparatus of the present invention has the above-described configuration, and provides information on suitable destination candidates without having to set the route again when the driver loses track of the route or gets lost. be able to. Similarly, by accumulating the line of sight and eye movement of the occupant who has lost sight of the route, as well as the history of driving behavior, information relating to what is difficult for the occupant to understand in route guidance using roadside guide signs and maps is provided. Road managers and map producers can find improvements in guide signs and maps by analyzing the information provided, which can improve the convenience of passengers and the ability to attract customers in stores.

  Hereinafter, an information providing apparatus according to an embodiment of the present invention will be described with reference to FIGS. FIG. 1 is a configuration diagram showing an example of an information providing system including an information providing apparatus according to an embodiment of the present invention. FIG. 2 is a flowchart showing an example of a detection flow of the line-of-sight direction in the information providing apparatus according to the embodiment of the present invention. FIG. 3a is a jumping eye movement in the information providing apparatus according to the embodiment of the present invention (hereinafter also referred to as saccade, rapid eye movement that occurs when the gaze position is changed in the horizontal, vertical, or diagonal directions). It is a figure for demonstrating an example of the amplitude of. FIG. 3b is another diagram for explaining an example of the amplitude of the saccade in the information providing apparatus according to the embodiment of the present invention. FIG. 4 is a flowchart showing an example of a flow for determining whether or not a target is searched for in the line-of-sight determination unit of the information providing apparatus according to the embodiment of the present invention. FIG. 5 is a diagram showing an example of the configuration of the history information database in the roadside information processing apparatus of the information providing system including the information providing apparatus according to the embodiment of the present invention. FIG. 6 is a flowchart showing an example of a series of recording flows from the recording of history data in the driving history recording unit of the information providing apparatus according to the embodiment of the present invention to the recording of history data in the recording unit of the roadside information processing apparatus. .

  FIG. 7 is a flowchart showing an example of a history data search flow from the history information database of the information providing apparatus according to the embodiment of the present invention. FIG. 8 is a flowchart showing an example of a history data search flow from the history information database of the roadside information processing apparatus of the information providing system including the information providing apparatus according to the embodiment of the present invention. FIG. 9 is a flowchart showing an example of an operation flow when the destination is not set in the information providing system including the information providing apparatus according to the embodiment of the present invention. FIG. 10 is a flowchart showing an example of an operation flow when a destination is set in the information providing system including the information providing apparatus according to the embodiment of the present invention. FIG. 11 is a flowchart illustrating an example of a processing flow for guiding a route and a target in the information providing apparatus according to the embodiment of the present invention.

  First, an information providing system including an information providing apparatus according to an embodiment of the present invention will be described with reference to FIG. As shown in FIG. 1, the information providing system includes an information providing device 1, a communication network 2, and a roadside information processing device 3. The information providing apparatus 1 can be connected to the roadside information processing apparatus 3 via the communication network 2 and can inquire various information.

  First, an example of an information providing apparatus according to an embodiment of the present invention will be described with reference to FIG. As shown in FIG. 1, the information providing apparatus 1 includes an image capturing unit 11, a line-of-sight detection unit 12, an operation amount input unit 13, a driving operation detection unit 14, a position detection unit 15, a driving history recording unit 16, a display device 17, The display processing unit 18, the determination unit 19, the information processing unit 20, the communication unit 24, the history information database 25, the GPS 26, and the car navigation device 27 are configured. The determination unit 19 further includes a line-of-sight determination unit 21 and a driving operation determination unit 22. The integrated determination unit 23 is configured. The information providing apparatus 1 is not limited to these components.

  The line-of-sight detection unit 12 detects the driver's line-of-sight direction for each face image based on a plurality of face images of the driver of the vehicle. Specifically, the line-of-sight detection unit 12 determines the face direction Df (vehicle traveling direction from the face image). And the driver's line-of-sight direction De in consideration of the detected driver's face direction Df. Here, the plurality of face images of the driver are captured by the image capturing unit 11, for example. The image capturing unit 11 captures the driver's face a plurality of times as time elapses. For example, the image capturing unit 11 is a CCD (Charge Coupled Device) camera or the like installed in a rear-view mirror (back mirror) in the vehicle. For example, the image capturing unit 11 captures an image (moving image) of 30 frames in one second, but is not limited thereto, and may capture an image (still image) at a larger interval. Here, the image pickup unit 11 and the line-of-sight detection unit 12 are separate components, but the line-of-sight detection (not shown) having the respective functions with the image pickup unit 11 and the line-of-sight detection unit 12 as one unit. It can also be a device.

  As a method for detecting the driver's face direction Df described above, for example, a predetermined inverted triangular shape connecting both eyes (both eyes) and nose or both eyes (both eyes) and mouth with respect to a captured image. There is a method of applying the mask (region) of the above and detecting the face direction Df from the deviation. Further, as a method for detecting the driver's line-of-sight direction De described above, for example, there is a known method for detecting the line-of-sight direction from a combination of a pupil and a cornea reflection image. Specifically, an infrared irradiation apparatus for line-of-sight detection (not shown) is used. By irradiating the driver's eyes with infrared rays, the reflected light is captured by an infrared camera (not shown) to project the driver's eyes, and the pupil center and corneal sphere center are calculated. Here, since the three-dimensional positional relationship between the infrared irradiation device (not shown) and the infrared camera (not shown) is known, a line connecting the calculated corneal sphere center and pupil center three-dimensionally is calculated by a computer. The calculated line is defined as the driver's line-of-sight direction De (see Japanese Patent Laid-Open No. 7-61257). The method for detecting the driver's line-of-sight direction is not limited to this, and other methods may be used.

  Here, FIG. 2 shows a flow for detecting the line-of-sight direction De of the driver. As shown in FIG. 2, first, a driver's face image captured by the image capturing unit 11 is captured (step S201). A plurality of facial feature points (eg, eyes, nose, mouth, etc.) are extracted from the captured face image (step S202). The face orientation Df is detected based on the respective positions of the extracted feature points of the face and the predetermined mask as described above (step S203). The driver's line-of-sight direction De in consideration of the detected face direction Df is detected by a known method as described above (step S204).

  Here, the detection of the gaze direction may be performed for either one of the left and right eyes, or the gaze direction is detected for both eyes, and the average of the detected results is taken. Then, the line-of-sight direction may be detected. It is also possible to set the face direction Df as the line-of-sight direction without performing the line-of-sight direction detection process. The line-of-sight detection unit 12 can also detect the contraction rate and change amount of the radius lr of the pupil. The line-of-sight detection unit 12 may detect blink frequency (blink) frequency and blink duration. Note that Non-Patent Documents 2 to 4 described above can be referred to for pupil movement and blinking such as pupil contraction rate and change amount.

  The operation amount input unit 13 transmits information such as a steering amount, an accelerator amount, a brake amount, and the like obtained from a control unit (not shown) that controls traveling of the vehicle to the driving operation detection unit 14. The driving operation detection unit 14 acquires an operation amount of each operation performed by the driver to run the vehicle. Specifically, the information collected from the operation amount input unit 13 is numerically based on a predetermined criterion. The information is transmitted to the determination unit 19. Note that the driving operation detection unit 14 may measure the speed and acceleration in the traveling direction, the yaw / pitch / roll amount, and quantify the measured amount and transmit it to the determination unit 19. The GPS 26 acquires vehicle position information, and the position detection unit 15 acquires vehicle position information obtained by the GPS 26 via the car navigation device 27 and transmits the vehicle position information to the line-of-sight determination unit 21. .

  The line-of-sight determination unit 21 calculates a movement amount of the line of sight due to the eye movement of the driver based on the driver's line-of-sight direction for each face image detected by the line-of-sight detection unit 12, and the driver based on the calculated movement amount of the line of sight Determines whether or not the driving route is confused, and if the driver determines that the driver is confused, the position information of the vehicle acquired by the GPS 26 and a plurality of face images used for the determination The line-of-sight direction of the driver is extracted based on the line-of-sight direction of the driver.

  More specifically, the gaze determination unit 21 determines the gaze from the gaze direction at the start and end of the jumping eye movement of the driver based on the gaze direction of the driver for each face image detected by the gaze detection unit 12. When the amount of movement is calculated for each jumping eye movement, it is determined whether the driver is confused on the travel route based on the calculated amount of movement of the line of sight, and when it is determined that the driver is confused, The vehicle position information acquired by the GPS 26, the driver's line-of-sight direction in the plurality of face images used for the determination, and a predetermined storage area (such as a semiconductor memory (not shown) in the car navigation device 27) are stored. For example, based on map information obtained from the outside (a storage medium such as a DVD that can be used in the car navigation device 27). Therefore, an object that is a series of information on an object existing around a travel route visually recognized by the driver within a predetermined time (hereinafter, also simply referred to as a predetermined time) among times when a plurality of face images of the driver are captured. An object information series is extracted. Below, although the case of saccade which is one of eye movements is demonstrated, it is not restricted to this. The predetermined time is a part of the time when the plurality of face images of the driver used for the determination are captured, but is not a part of the time but the time during which the image is captured. May be. Specific processing will be described below.

  The line-of-sight determination unit 21 obtains the saccade amplitude Sθ as an amount of change of the line-of-sight direction based on the driver's line-of-sight direction obtained by the line-of-sight detection unit 12, and further, the time Gt when the line of sight stops before the line of sight moves Ask for. Here, the amplitude Sθ of the saccade will be described with reference to FIGS. 3a and 3b. As shown in FIG. 3a, it is assumed that the driver is initially looking at or around the first point 300 in the traveling direction of the vehicle through the windshield of the vehicle. It is assumed that the line of sight moves to or around the second point 301 by moving the line of sight from the state. In this case, as shown in FIG. 3B, an angle θ1 formed by the line-of-sight directions projected on the horizontal plane parallel to the ground is set as the saccade amplitude Sθ. Considering this, the amplitude of the saccade when the driver's line of sight moves from the second point 301 to the third point 302 or the vicinity thereof is θ2, as shown in FIG. 3b. Note that the line-of-sight stop time Gt described above refers to, for example, the time from when the line of sight moves to the second point 301 shown in FIG.

  The line-of-sight determination unit 21 obtains the saccade frequency Sr at which the saccade amplitude Sθ is equal to or greater than a predetermined value, and obtains the line-of-sight stop count Gtc at which the line-of-sight retention time Gt is equal to or greater than a predetermined threshold Gtth. Then, the line-of-sight determination unit 21 determines whether or not the driver is searching for the target object (whether or not hesitated) from the obtained information. That is, when the following formula (1) is satisfied, it is determined that the target is searched. Here, the left side of Expression (1) indicates the search degree S, α is a coefficient, Count (Gt> Gtth) indicates the number of eye-gaze stop times Gtc, and Gtth and Th are predetermined threshold values. Here, it is determined whether or not the driver is looking for a target based on the saccade amplitude Sθ and the line-of-sight stop time Gt, but the determination is made based only on the information on the saccade amplitude Sθ. May be.

  Here, FIG. 4 shows a flow for determining whether or not a target is searched. As shown in FIG. 4, first, the saccade amplitude Sθ and the gaze stop time Gt are calculated as the amount of change in the gaze direction (step S401). The number of saccades (frequency) Sr and the line-of-sight stop count Gtc at which the line-of-sight stop time Gt is equal to or greater than the threshold Gtth are calculated (step S402). The search degree S is calculated from the weighted sum of the number of saccades (frequency) Sr and the number of times of gaze stoppage Gtc (step S403). Then, it is determined whether or not the search degree S is larger than the threshold value Th (step S404), and if it is larger, it is determined that the user is lost (searching for a target) (step S405).

  In this way, when it is determined that the driver is lost (looking for something), the signs and landmarks (objects) seen by the gaze behavior that the driver looks around within the predetermined time The line-of-sight determination unit 21 extracts information based on the vehicle position information acquired by the GPS 26, the line-of-sight direction detected by the line-of-sight detection unit 12, and map information included in the car navigation device 27, for example. If the hardware resources of the information providing device 1 are sufficient, the signs and landmarks seen by the driver before it is determined that the driver is lost (looking for something) (object) It is also possible to extract this information. As a result, faster processing can be performed.

  As another method for determining whether or not he / she is lost (searching for a target), the line-of-sight determination unit 21 contracts the pupil radius lr obtained by the line-of-sight detection unit 12 as described above. An attention amount indicating a degree of attention to the surroundings of the driver may be obtained from the rate and the change amount, and it may be determined whether the driver is looking for or missing the target (mark) in the route guidance process. Further, as described above, the line-of-sight determination unit 21 obtains an attention amount indicating the degree of attention to the driver's surroundings from the blinking frequency (blink) frequency obtained by the line-of-sight detection unit 12 and the duration of the blink. It may be determined whether or not the user is looking for a target (marker) in the route guidance process.

  Further, when the destination is set in advance by the driver and route guidance is provided, the line-of-sight determination unit 21 determines the driver's line-of-sight in the route guidance process based on the position information of the host vehicle detected by the position detection unit 15. It is determined whether or not the target has been found based on whether or not the object (marker) has stayed for a predetermined time or longer. Based on this, the line-of-sight determination unit 21 determines that the vehicle has been lost if it is determined that the target has not been found before the vehicle passes through a position where the target can be properly visually recognized in the route guidance process. You may make it do. On the other hand, a line-of-sight action that looks for a target at a position far before the position where the vehicle's position reaches the position where the target can be properly viewed in the route guidance process, such as an action of looking around the surroundings, is detected. In such a case, it may be determined that the driver is lost. In this way, the line-of-sight determination unit 21 can also determine whether or not the driver is lost, whether or not he is looking for a target.

  The driving operation determination unit 22 determines whether or not the driver is lost in the travel route (looking for a target) based on the operation amount acquired by the driving operation detection unit 14. For example, when the frequency (number of times) Acr of changing the accelerator amount within a predetermined period and the frequency (number of times) Bcr of changing the brake amount exceed a predetermined threshold, or when the steering amount of the steering changes within the predetermined period (number of times) ) When Scr exceeds a predetermined threshold value, it is determined that the driver is searching for a target or is lost.

  The integrated determination unit 23 finally determines whether or not the driver is at a loss in the travel route based on the result determined by the driving operation determination unit 22 and the result determined by the line-of-sight determination unit 21. Yes, specifically, the frequency (number of times) of change in the accelerator amount or the like when determined by the driving operation determination unit 22 and the weighted sum of the numerical values determined by the line-of-sight determination unit 21 are calculated, and a predetermined threshold value is calculated. In the case of the above, the driver finally determines that he is looking for a target or is lost. If it is determined that the driver is lost (searching for a target), the integrated determination unit 23 provides information indicating that the driver is lost, current vehicle position information, predetermined information, etc. Information extracted by the line-of-sight determining unit 21 such as signs and landmarks (objects) viewed by the line-of-sight behavior of the driver looking around in time is transmitted to the information processing unit 20.

  Note that when it is determined that the driver is lost, etc., the determination is made using both the determination results of the line-of-sight determination unit 21 and the driving operation determination unit 22, but the driver is determined only by the determination result of the line-of-sight determination unit 21. You may determine whether you are lost or not. In this case, the integrated determination unit 23 does not perform any specific determination processing, information indicating that the driver is lost on the basis of the determination result transmitted from the line-of-sight determination unit 21, current vehicle position information, and a predetermined time Information extracted by the line-of-sight determination unit 21 such as a sign or a landmark viewed by the line-of-sight behavior of the driver looking around is transmitted to the information processing unit 20.

  The information processing unit 20 includes a series of line-of-sight directions when each of the plurality of drivers has lost his / her travel route, information on an arrival place which has arrived after the occurrence of the lost place, and an arrival place from the point where the lost place has occurred. In the history information database 34 that stores in advance and stores history information that includes route information to the destination including target information that serves as a landmark in the route guidance up to, the line-of-sight direction extracted by the line-of-sight determination unit 21 is stored. Search request information for a search request as to whether or not there is history information including a line-of-sight direction sequence having a predetermined correlation with the sequence is generated and transmitted to an external means (search / recommendation unit 32) for performing the search. Is.

  More specifically, the information processing unit 20 includes an object information series that is a series of information on objects existing around the travel route visually recognized when each of the plurality of drivers has lost his / her travel route. History information consisting of destination information that has arrived after the occurrence of ambiguity, and route information to the destination including target information that is used as a guide for route guidance from the point of ambiguity to the destination In order to request a search for whether there is history information including the object information series having a predetermined correlation with the object information series extracted by the line-of-sight determination unit 21 in the history information database 34 stored respectively. Search request information is generated and transmitted to an external means (search / recommendation unit 32) for performing a search. Specifically, search request information including information received from the integrated determination unit 23 (hereinafter also referred to as communication request information) is generated, and the generated communication request information is transmitted to the roadside information processing device 3 via the communication unit 24. Send to.

  Next, the roadside information processing apparatus 3 will be described. The road-side information processing device 3 includes a communication unit 31, a search / recommendation unit 32, a recording unit 33, and a history information database 34, but is not limited to these components. The communication unit 31 receives communication request information from the information providing device 1 and transmits information searched by the search / recommendation unit 32 to the information providing device 1. After receiving the communication request information from the communication unit 31, the search / recommendation unit 32 includes the vehicle position information included in the communication request information and the signs and landmarks seen by the gaze behavior where the driver looks around within a predetermined time. Based on such information, the history information database 34 is searched for a vehicle to be driven by a driver who has taken a similar gaze action in the past. A specific search method will be described later.

  The history information database 34 includes information such as signs and landmarks seen by gaze behavior within a predetermined time of a driver of a vehicle who has lost his way in the past, destination information indicating where the vehicle has reached, The route information to the destination is stored in association with each other. These information stored in association with each other is registered as history information when each driver gets lost, but as a default information, a plurality of drivers who are not familiar with driving the vehicle in advance actually run. In addition, information when lost can be stored as history information. An example of the history information database 34 is shown in FIG. As shown in FIG. 5, for example, the search position is “Bunkyo-ku, Tokyo xx”, and this search position is extracted based on the position information of the vehicle to be searched.

  In the extracted search position, for example, the driver A who has reached the Tokyo Dome in the past has been looking at the “signpost a” and the “building a” within a predetermined time at the corresponding search position. In addition, driver B who has reached Suidobashi station in the past has seen “signpost a”, “signature b”, “signature c”, and “building c” within a predetermined time at the corresponding search position. Note that the numerical value “1” shown in FIG. 5 indicates that the corresponding sign or building has been seen, and the numerical value “0” indicates that it has not been seen. The route information column stores route data to the destination and information including information on the target that is the target in the route guidance process.

  Here, a specific search method by the search / recommendation unit 32 will be described. The search / recommendation unit 32 receives communication request information via the communication unit 31. The position information of the vehicle included in the communication request information indicates, for example, “Bunkyo-ku, Tokyo xx”, and signs and landmarks seen by the driver's gaze behavior within a predetermined time at the corresponding search position For example, the search / recommendation unit 32 searches the history information database 34 shown in FIG. 5 for the history data of the driver who has a similar gaze action. In this case, the search / recommendation unit 32, for example, a known correlation filtering correlation coefficient method (http://www.ipsj.or.jp/katsudou/sig/sighp/fi/sympo/2002/proceedings/invited/yamanishi (see .pdf), it is determined that the driver's line-of-sight behavior of the driver A who has reached Tokyo Dome in the past is the most similar, and the destination (Tokyo Dome in this example) The route data up to that point and the information on the target (mark) in the route guidance process are transmitted to the communication unit 31 as recommendation information.

  Note that the recommendation information is not limited to one set of information, and may be a plurality of sets of information. Here, the correlation coefficient method of collaborative filtering is based on a series of objects (signs, landmarks, etc.) visually recognized by the driver in a certain time (a certain section) to the current location of the driver, and the driver stored in the history information database 34. Compares a series of objects visually recognized in the certain time (constant section), and selects past drivers who have taken similar line-of-sight behavior according to the magnitude of the correlation (similarity) and a predetermined threshold. Is.

  The communication unit 31 uses the communication network 2 as the recommended information based on the destination received from the search / recommendation unit 32 (in this example, Tokyo Dome), the route data up to that point, and the target (marker) information in the route guidance process. To the information providing apparatus 1. Then, the information processing unit 20 transmits the destination received from the communication unit 31 (in this example, the Tokyo Dome), route data to that point, and information on the target (marker) in the route guidance process to the display processing unit 18. To do.

  The display processing unit 18 receives the history information searched by the search means (search / recommendation unit 32), and based on the received history information, the destination information and the route guidance from the point where the hesitation occurred to the destination The display unit displays the route information to the destination including the target object information that serves as a landmark on the display unit. Specifically, control for displaying the obtained recommendation information on the display device 17 is performed. If there are a plurality of destination candidates, they are displayed on the display device 17 in order from the current location, in order of popularity (in order of the number of times recommended in the past as recommended information), or in the newest order of data update. Provide information to recommend to drivers. When the driver selects a destination from the recommended information, the display processing unit 18 causes the display device 17 to display the route to the selected destination and information on the target in the route guidance process.

  The driving history recording unit 16 associates the destination, the information on the route traveled up to that point, and the history of the gaze behavior on the route after the vehicle has arrived at the destination, as one history data. The history data recorded in the information database 25 is recorded in the history information database 34 by the recording unit 33 of the roadside information processing apparatus 3 via the information processing unit 20 and the communication unit 24.

  A series of recording flows is shown in FIG. As shown in FIG. 6, the information providing apparatus 1 determines whether or not the vehicle has arrived at the destination (step S601). If it is determined that the vehicle has arrived at the destination, the history information database 25 records one piece of history data by associating the destination with the information of the route traveled so far and the history of the gaze behavior on the route ( Step S602). On the other hand, in order to record the same history data in the history information database 34 of the roadside information processing device 3, it is transmitted to the roadside information processing device 3 via the information processing unit 20 and the communication unit 24 and recorded (step S603). In this way, when another vehicle is lost in the future, history data can be provided in response to an inquiry from a vehicle lost.

  As described above, the information providing apparatus 1 requests the roadside information processing apparatus 3 to search for history data of a driver who has performed a line-of-sight behavior similar to that of his own vehicle, and acquires the history data if there is any. It is configured. However, the information providing device 1 does not immediately request a search to the roadside information processing device 3 due to a loss of the road or the like, but the history of the line-of-sight behavior when the driver of the vehicle in which the information is provided has lost in the past. The history information database 25 storing the data may be searched by itself for the corresponding history data, and if there is no corresponding history data, a request for search may be made to the roadside information processing device 3. In this case, similar to the history information database 34, the history information database 25 is object information that is a series of information on objects existing around the travel route visually recognized when the driver has lost his / her travel route in the past. Stores information on the destination, the destination reached after the loss, and the history information consisting of the route information to the destination including the target information used as a guide for route guidance from the point where the loss occurred to the destination. . Such a search flow is shown in FIG.

  As shown in FIG. 7, it is first determined whether or not a destination is set by the car navigation device 27 or the like (step S701). When the destination is set, the information providing apparatus 1 uses the history information database 25 of its own history information database 25 to match the destination, or from the history data of the vicinity, the gaze behavior similar to the driver in the past near the current position. Whether there is history data that has been taken is searched (step S702). On the other hand, when the destination is not set, the information providing apparatus 1 records history data in which the driver has taken similar line-of-sight behavior in the past near the current position from the history data in the history information database 25 of the information providing apparatus 1. It is searched whether there is any (step S703).

  Further, as described above, when the search / recommendation unit 32 of the roadside information processing device 3 searches the history data from the history information database 34, the history data near the position indicated by the position information of the vehicle is searched. When the destination information of the vehicle is also transmitted from the information providing apparatus 1 to the roadside information processing apparatus 3, the search may be performed in consideration of the destination information. The search flow in that case is shown in FIG. As shown in FIG. 8, the roadside information processing apparatus 3 determines whether or not the destination information set by the car navigation apparatus 27 or the like has been received (step S801).

  When the destination information is received, the search / recommendation unit 32 has a line of sight similar to another person or the driver himself / herself in the past near the current position from among the history data whose destinations match or are close in the history information database 34. It is searched whether there is history data for taking an action (step S802). On the other hand, when the destination information is not received, as described above, the search / recommendation unit 32 searches the history data in the history information database 34 so that another person or the driver himself / herself is similar in the past near the current position. It is searched whether there is history data that has been taken (step S803).

  Next, an example of an operation flow in the information providing system including the information providing apparatus according to the embodiment of the present invention will be described with reference to FIGS. In the flow shown in FIG. 9 and FIG. 10, first, history data in the history information database 25 of the information providing apparatus 1 is searched, and if appropriate history data is not detected, the history information database 34 of the roadside information processing apparatus 3 is detected. The history data in is searched. Here, the configuration of the history information database 25 of the information providing device 1 has the same configuration as the history information database 34 of the roadside information processing device 3 shown in FIG. 5 described above.

  Further, this operation flow is an example, and as described above, the history data in the history information database 34 of the roadside information processing device 3 is immediately retrieved without searching the history data in the history information database 25 of the information providing device 1. You may make it search. FIG. 9 is a flow when the destination is not set in the car navigation device 27 or the like, and FIG. 10 is a flow when the destination is set. First, the operation flow when the destination is not set will be described.

  First, the position detection unit 15 performs a process of acquiring vehicle position information (vehicle position / behavior detection process) (step S901). As described with reference to FIG. 2, the line-of-sight detection unit 12 performs a process (line-of-sight behavior detection process) of detecting the line-of-sight direction of the driver based on the plurality of face images captured by the image capturing unit 11 (step S902). As described above, when the determination by the driving operation determination unit 22 is also taken into consideration, the frequency (number of times) Acr at which the accelerator amount changes, the frequency (number of times) Bcr at which the brake amount changes, and the steering amount of the steering change. Processing for obtaining frequency (number of times) Scr and the like (exercise behavior extraction processing) is performed (step S903). Then, the integrated determination unit 23 performs a process (understanding level determination process) for determining whether or not the target is being searched (step S904), and at the same time, it is determined that the user is lost (looking for the target). In this case, a series of information such as signs and landmarks viewed by the driver's visual behavior as described above is extracted.

  Then, as described with reference to FIG. 7, the information processing unit 20 has taken similar line-of-sight behavior in the past from the history data in the history information database 25 based on the extracted information series. A process of searching for history data (similar action area policy process (own vehicle)) is performed (step S905). In the case of the flow shown in FIG. 9, since the destination is not set, the process proceeds to step S703 in FIG. The object (signs, landmarks, etc.) visually recognized by the driver within a predetermined time of the driver by the correlation coefficient method of collaborative filtering similar to the determination made by the information processing unit 20 performed by the search / recommendation unit 32 of the roadside information processing device 3 ) And the series of objects visually recognized by the driver stored in the history information database 25, and determine whether or not the magnitude of the correlation (similarity) is equal to or greater than a predetermined threshold (step S906). .

  If it is equal to or greater than the predetermined threshold, the information processing unit 20 extracts the destination headed by the similar line-of-sight behavior, the route to the destination, and the target in the route guidance process as recommended information (step S907). On the other hand, if it is not equal to or greater than the predetermined threshold value in step S906, the search / recommendation unit 32 determines that the history information database 34 is based on the search request transmitted from the information processing unit 20 via the communication unit 24 as described in FIG. A process (similar action area measure process (other vehicle)) for searching for history data of a driver who has taken a similar line-of-sight behavior in the past is performed (step S908). In the case of the flow shown in FIG. 9, since the destination is not set, the process proceeds to step S803 in FIG.

  The search / recommendation unit 32 uses the correlation coefficient method of collaborative filtering to visually recognize a series of objects (signs, landmarks, etc.) visually recognized by the driver within a predetermined time of the driver and the driver stored in the history information database 34. The series of the target objects is compared, and it is determined whether or not the magnitude (similarity) of the correlation is equal to or greater than a predetermined threshold (step S909). When the value is equal to or greater than the predetermined threshold, the search / recommendation unit 32 extracts the destination to which the driver who has taken similar line-of-sight behavior headed, the route to the destination, and the target in the route guidance process as recommended information, and performs communication. The information is transmitted to the unit 24 and the information processing unit 20 (step S910). On the other hand, if it is not equal to or greater than the predetermined threshold, the process ends.

  The display processing unit 18 displays the extracted or transmitted destination candidates on the display device 17 in accordance with criteria (for example, the order closer to the current location, the newest information, etc.) selected by the driver of the host vehicle (step S911). . Then, the integrated determination unit 23 performs a route / target guidance process described later (step S912). Here, the route / target guidance process will be described with reference to FIG. As shown in FIG. 11, the integrated determination unit 23 determines whether the current position is in front of the guidance start point, in front of the point where the target in the route guidance process can be visually recognized, or the point in which the target in the route guidance process can be visually recognized. Or whether the object has been passed (step S1101).

  When the current position is in front of the guidance start point, the user is prompted to proceed as it is, for example, “It is a road for a while” (step S1102). If the target in the route guidance process is in front of the point where the target can be visually recognized, another target in front of the target is added and notified (step S1103). In the case where the target can be visually recognized in the route guidance process, an easily visible target different from the target is added and notified (step S1104). If the target is passed, the route is searched again and notified together with the new target (step S1105).

  Next, an operation flow when the destination is set will be described with reference to FIG. First, as in FIG. 9, the position detection unit 15 performs a process of acquiring vehicle position information (vehicle position / behavior detection process) (step S1001). Then, the integrated determination unit 23 determines whether or not the current position and the set destination are close enough to be included in a predetermined range (for example, a radius of 1 km) (step S1002). If it is not close enough to be included in the predetermined range, processing similar to that in steps S902 to S910 in FIG. 9 is performed in steps S1003 to S1011. Note that in the similar behavior search processing in steps S1006 and S1009, since the destination is set, the process proceeds to steps S702 and S802. Since other processes to be performed are basically the same, description thereof will be omitted. Then, based on the information extracted or transmitted in step S1008 or step S1011, the integrated determination unit 23 performs a route / target guidance process described later (step S1012).

  The information providing apparatus according to the present invention provides information on candidate destinations suitable for the driver without causing the driver to perform the route setting operation again when the driver loses track or gets lost. Therefore, the present invention is useful for an information providing apparatus that provides information for guiding a vehicle to a driver who drives the vehicle.

It is a block diagram which shows an example of the information provision system containing the information provision apparatus which concerns on embodiment of this invention. It is a flowchart which shows an example of the detection flow of a gaze direction in the information provision apparatus which concerns on embodiment of this invention. It is a figure for demonstrating an example of the amplitude of a saccade in the information provision apparatus which concerns on embodiment of this invention. It is another figure for demonstrating an example of the amplitude of the saccade in the information provision apparatus which concerns on embodiment of this invention. It is a flowchart which shows an example of the flow which determines whether it is looking for the target in the gaze determination part of the information provision apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the structure of the log | history information database in the roadside information processing apparatus of the information provision system containing the information provision apparatus which concerns on embodiment of this invention. It is a flowchart which shows an example of a series of recording flows from the recording of the history data in the driving history recording part of the information provision apparatus which concerns on embodiment of this invention to the recording of the history data in the recording part of a roadside information processing apparatus. It is a flowchart which shows an example of the search flow of the historical data from the historical information database of the information provision apparatus which concerns on embodiment of this invention. It is a flowchart which shows an example of the search flow of the historical data from the historical information database of the roadside information processing apparatus of the information provision system containing the information provision apparatus which concerns on embodiment of this invention. It is a flowchart which shows an example of the operation | movement flow when the destination in the information provision system containing the information provision apparatus which concerns on embodiment of this invention is not set. It is a flowchart which shows an example of the operation | movement flow when the destination in the information provision system containing the information provision apparatus which concerns on embodiment of this invention is set. It is a flowchart which shows an example of the processing flow of the guidance of the path | route and the target in the information provision apparatus which concerns on embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Information provision apparatus 2 Communication network 3 Roadside information processing apparatus 11 Image pick-up part 12 Gaze detection part (gaze detection means)
13 Operation amount input unit 14 Driving operation detection unit (operation amount acquisition means)
15 Position Detection Unit 16 Operation History Recording Unit 17 Display Device (Display Unit)
18 Display processing unit (display control means)
19 determination unit 20 information processing unit (processing means)
21 Gaze determination unit (determination / extraction means)
22 Driving operation determination unit (driving operation determination means)
23. Integrated determination unit (integrated determination means)
24, 31 Communication unit 25 History information database (storage means)
26 GPS (location information acquisition means)
27 Car Navigation Device 32 Search / Recommendation Unit 33 Recording Unit 34 History Information Database 300 First Point 301 Second Point 302 Third Point

Claims (6)

Line-of-sight detection means for detecting the direction of the line of sight of the driver for each face image based on a plurality of face images of the driver of the vehicle;
Position information acquisition means for acquiring position information of the vehicle;
Based on the gaze direction of the driver for each face image detected by the gaze detection means, a gaze movement amount due to the driver's eye movement is calculated, and based on the calculated gaze movement amount When the driver determines whether the driver is confused or not, and determines that the driver is confused, the position information of the vehicle acquired by the position information acquiring means, Determination / extraction means for extracting a series of the driver's line-of-sight direction based on the driver's line-of-sight direction in the plurality of face images used when making the determination;
A series of line-of-sight directions when each of a plurality of drivers has lost his / her travel route, information on a destination reached after the occurrence of the lost, and from the point where the lost occurs to the destination The line-of-sight direction sequence extracted by the determination / extraction means in a database that previously creates and stores history information consisting of route information to the destination including target information that serves as landmarks in route guidance, Processing means for generating search request information for a search request whether there is the history information including the line-of-sight direction sequence having a predetermined correlation, and transmitting the search request information to an external means for performing the search;
Display means for displaying information;
The history information searched by the external means for performing the search is received, and the destination information and the route guidance from the point where the hesitation occurred to the destination are based on the received history information. Display control means for displaying route information to the destination including target information on the display means as recommended information;
An information providing apparatus provided.   Whether the determination / extraction means further adds information on the gaze stop time at which the driver's gaze stops to the calculated information on the movement of the gaze, so that the driver is at a loss in the travel route. The information providing apparatus according to claim 1, which determines whether or not. A series of line-of-sight directions when the driver has lost his / her driving route in the past, information on a destination that has arrived after the driver has lost his head, and a route guide from the point at which he / she has lost to the destination Storage means for creating and storing in advance history information including route information to the destination including target object information,
The processing means searches whether the history information including the line-of-sight direction series having a predetermined correlation with the line-of-sight direction sequence extracted by the determination / extraction means is in the storage means. 3. The information providing apparatus according to claim 1, wherein when there is no history information, search request information for the search request is generated and transmitted to the external means for performing the search. An operation amount acquisition means for acquiring an operation amount of each operation performed by the driver to run the vehicle;
Driving operation determination means for determining whether or not the driver is at a loss in the travel route based on the operation amount acquired by the operation amount acquisition means;
Integrated determination means for finally determining whether or not the driver is at a loss on the travel route based on the result determined by the driving operation determination means and the result determined by the determination / extraction means; The information providing apparatus according to any one of claims 1 to 3, further comprising:   5. The information providing apparatus according to claim 1, wherein the movement amount of the line of sight is calculated for each of the jumping eye movements from the line-of-sight direction at the start and end of the jumping eye movement. The line-of-sight direction series includes the position information of the vehicle acquired by the position information acquisition unit, the line-of-sight direction of the driver in the plurality of face images used for the determination, and a predetermined range An object information series which is a series of information on objects existing around the travel route visually recognized by the driver, which is stored in a storage area or extracted based on map information obtained from the outside. Item 6. The information providing apparatus according to any one of Items 1 to 5.

Images