WO2019221070A1

WO2019221070A1 - Information processing device, information processing method, and information processing program

Info

Publication number: WO2019221070A1
Application number: PCT/JP2019/018971
Authority: WO
Inventors: 知柔今林; 洋治森; 秀人大前; 悠介鵜飼
Original assignee: オムロン株式会社
Priority date: 2018-05-16
Filing date: 2019-05-13
Publication date: 2019-11-21
Also published as: JP6449504B1; JP2019200614A

Abstract

Implemented is an information processing device capable of sensing the interest level of a driver. The present invention involves: acquiring face information about a driver; detecting the visual line from the acquired face information; detecting objects existing in the visual field of the driver; determining an object that is being visually recognized by the driver on the basis of the detected visual line and the detected objects; and generating interest level information indicating the interest level of the driver in the determined object.

Description

Information processing apparatus, information processing method, and information processing program

The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

Conventionally, there is a method of tracking a spot where a consumer is paying attention by performing a gaze information analysis of a specific person using a fixed point camera and a brain information analysis by a wearable electroencephalograph. For example, Patent Literature 1 discloses a portable terminal device that grasps a trend on the side of viewing, such as a portion of interest and content in a display image.

Japanese Patent Publication “JP 2009-5094”

On the other hand, if the inventor can evaluate how much the driver is interested in an object such as a building visually recognized by a driver of a moving body such as an automobile, an effective advertisement And gained knowledge that it is effective from the viewpoint of urban development.

Such a technique cannot be realized with a conventional configuration using a fixed camera.

One embodiment of the present invention has been made in view of the above problems, and an object thereof is to realize an information processing apparatus capable of sensing the degree of interest of a driver.

In order to solve the above problems, an information processing apparatus according to an aspect of the present invention includes a face information acquisition unit that acquires face information of a driver, and a line of sight that detects a line of sight from the face information acquired from the face information acquisition unit. An object that the driver is viewing from: a detection unit; an object detection unit that detects each object existing in the field of view of the driver; and a line of sight detected by the line of sight detection unit and an object detected by the object detection unit. The visual recognition object determination part to determine, and the interest level information generation part which produces | generates the interest level information which shows the interest degree of the said driver with respect to the object which the said visual recognition object determination part determined.

According to one aspect of the present invention, an information processing apparatus that can sense the interest level of a driver can be realized.

It is a block diagram which shows the component of the information processing apparatus which concerns on Embodiment 1 of this invention. It is a figure which shows an example of the picked-up image of the scenery outside a vehicle which an imaging part image | photographs in the information processing apparatus which concerns on Embodiment 1 of this invention. It is a flowchart which shows an example of the flow of a process of the information processing apparatus which concerns on Embodiment 1 of this invention. It is a block diagram which shows the component of the information processing apparatus which concerns on Embodiment 2 of this invention. It is a flowchart which shows the flow of a process of the information processing apparatus which concerns on Embodiment 2 of this invention. It is a block diagram which shows the component of the information processing system which concerns on Embodiment 3 of this invention.

<Embodiment 1>
Hereinafter, an embodiment of the present invention will be described in detail. The configuration of the following specific items (embodiments) may be omitted if it is the same as the configuration described in other items. For convenience of explanation, members having the same functions as those shown in each item are given the same reference numerals, and the explanation thereof is omitted as appropriate.

[Information processing equipment]
The information processing apparatus 1 according to the present embodiment is an apparatus that performs face information acquisition processing, line-of-sight detection processing, object detection processing, visual object determination processing, interest level information generation processing, and the like. The information processing apparatus 1 is mounted on a vehicle or the like, for example, but this does not limit the present embodiment.

The mobile terminal device 2 may be a smartphone, a tablet terminal, or the like.

FIG. 1 is a block diagram showing components of the information processing apparatus 1 according to the present embodiment.

The information processing apparatus 1 includes a face information acquisition unit 12, a line-of-sight detection unit 13, an object detection unit 11, a visual object determination unit 14, an interest level information generation unit 15, a recommendation engine (suggested information generation unit) 16, And a destination presentation unit (navigation system) 17.

(Face information acquisition unit)
The face information acquisition unit 12 acquires driver face information. The driver's face information acquired by the face information acquisition unit 12 is, for example, a captured image of the driver. In order to acquire the captured image of the driver, an imaging device that captures the driver's face is arranged in a vehicle or the like on which the information processing apparatus 1 is mounted.

However, if the line-of-sight detection unit 13 includes sufficient information for detecting the line of sight, the face information of the driver acquired by the face information acquisition unit 12 is the positional information of each part of the face extracted from the captured image of the driver. Etc. The method by which the face information acquisition unit 12 acquires the driver's face information is not particularly limited. For example, the driver's face information is acquired by using a face information acquisition device equipped with a known technique for acquiring a person's face. be able to. The face information acquisition unit 12 outputs the acquired face information to the line-of-sight detection unit 13.

(Gaze detection unit)
The line-of-sight detection unit 13 detects the line of sight from the face information acquired from the face information acquisition unit 12. That is, the gaze detection unit 13 detects the gaze direction of the driver included in the acquired face image. Specifically, the line-of-sight detection unit 13 specifies the line-of-sight angle of the driver imaged in the face image that is the face information, and refers to the line-of-sight direction specifying information indicating the correspondence between the line-of-sight angle and the line-of-sight direction. A line-of-sight direction corresponding to the specified line-of-sight angle is specified.

In the gaze direction specifying information, the correspondence relationship between the gaze angle and the gaze direction may be set as appropriate, a predetermined number of gaze directions may be set, and the range of the corresponding gaze angle may be arbitrarily set.

The line-of-sight direction specifying information is set in advance. However, since it is desirable to set the correspondence between the line-of-sight angle and the line-of-sight direction based on the position of the camera and the face of the driver, the line-of-sight specific information is corrected for each driver or whenever the line-of-sight direction is set. It is desirable to do.

The line-of-sight detection unit 13 identifies a plurality of line-of-sight angles from a plurality of images taken during a predetermined period, and a line-of-sight direction corresponding to an average value, a most frequent angle, an intermediate value, or the like of the plurality of line-of-sight angles. May be specified. Accordingly, even when the driver's line-of-sight angle is not stable, the driver's line-of-sight direction with high reliability can be specified.

Also, the line-of-sight detection unit 13 may specify the line-of-sight angle of the driver imaged in the image, and output the specified line-of-sight angle itself to the visual object determination unit 14.

Note that the method for detecting the user's line-of-sight direction from the image is not limited to the above-described method, and a known technique can be used. For example, the line-of-sight direction may be detected based on the position of the iris.

For example, the line-of-sight detection method is not particularly limited, but the information processing apparatus 1 is provided with a point light source (not shown), and the user captures a corneal reflection image of light from the point light source with the imaging unit for a predetermined time. There is a method for detecting the movement destination of the line of sight. The type of the point light source is not particularly limited, and examples thereof include visible light and infrared light. For example, by using an infrared LED, it is possible to detect the line of sight without causing discomfort to the user. When detecting the line of sight, if the line of sight does not move for a predetermined time or more, it can be said that the same place is being watched.

The line-of-sight detection unit 13 may detect not only the driver's line of sight but also the state of the driver by referring to the state of the pupil of the driver and the number of blinks.

The method for detecting the state of the pupil is not particularly limited, and examples thereof include a method for detecting a circular pupil from an eye image using Hough transform. In general, since humans tend to open their eyes when they are concentrated, the degree of concentration of the driver can be evaluated by detecting the size of the pupil. For example, when the pupil size is detected for a predetermined time and the pupil is enlarged within the predetermined time, it can be said that there is a high possibility that the driver is gazing at an object. A threshold value may be set for the pupil size, and may be evaluated as “open” when the pupil size is equal to or larger than the threshold value, and “closed” when the pupil size is smaller than the threshold value.

The method for detecting the number of blinks is not particularly limited. For example, the driver's eyes are irradiated with infrared light, and the difference in the amount of reflected infrared light between when the eye is opened and when the eye is closed is detected. Methods and the like. In general, humans tend to blink at a stable interval at a low frequency when they are concentrated. Therefore, the concentration level of a driver can be evaluated by detecting the number of blinks. For example, if the number of blinks is detected for a predetermined time and blinking is performed at a stable interval within the predetermined time, it can be said that there is a high possibility that the driver is gazing at an object.

The line-of-sight detection unit 13 may detect at least the line of sight of the driver, but preferably combines the line of sight of the driver and the state of the pupil, or the line of sight of the driver and the number of blinks. By combining the detection methods in this way, the line-of-sight detection unit 13 can suitably evaluate the driver's concentration when viewing a certain object.

Note that the driver concentration described above can be used as an index for evaluating the driver's interest level described later.

The line-of-sight detection unit 13 measures the gaze time for the object that the driver is viewing.

The line-of-sight detection unit 13 outputs the specified line-of-sight direction to the visual object determination unit 14.

(Object detection unit)
The object detection unit 11 detects each object present in the driver's field of view.

The object detection unit 11 includes an imaging unit and an object extraction unit. The imaging unit captures a scene outside the vehicle and generates a captured image.

FIG. 2 is a diagram illustrating an example of a captured image 100 showing a scenery outside the vehicle captured by the imaging unit of the object detection unit 11. Note that the captured image 100 captured by the imaging unit of the object detection unit 11 only needs to include an image of the scenery outside the vehicle, and may include an image of the scenery inside the vehicle. Examples of the object included in the image of the scenery inside the vehicle include a fuel meter, a clock, a display screen of a navigation system, and a passenger.

The object extraction unit extracts an object from the captured image 100 acquired from the imaging unit.

The object extraction unit detects an object from the captured image 100 acquired from the imaging unit, and extracts the detected object. The object extraction unit generates object information indicating the extracted object, and outputs the generated object information to the visual object determination unit.

Further, when generating the object information, the object extracting unit may add position information and size information indicating the position and size of each object in the captured image 100 as additional information of the object to the object information.

In addition, in this specification, although an object refers to the general target object contained in a captured image, as a more specific example, as an object outside a vehicle, buildings and signs, such as a store and a building, a road, for example Traffic signs, traffic lights, people, animals and the like. Further, examples of the object in the vehicle include a fuel meter, a clock, a display screen of a navigation system, and a passenger.

The object information may be information indicating the pixel value of the pixel group in the object region in the captured image 100, or may be the object feature amount such as edge information indicating the edge (contour) of the object. It may be the information shown. Further, the additional information of the object does not need to include both the position information and the size information, and may include at least one.

As an example, the object extraction unit includes an object detection unit and a region extraction unit, and more specifically, the object detection unit and the region extraction unit generate object information.

The object detection unit reads an image template that is a standard image of the object from the storage unit further included in the information processing apparatus 1. Then, the captured image and the image template are matched, and it is determined whether or not the captured image includes the same object as the matched image template. When the object detection unit determines that the same object as the matched image template is included, the object detection unit extracts the object from the captured image, and generates object information indicating the extracted object.

The object detection unit reads a feature amount template indicating the feature amount of the standard image of the object from the storage unit, calculates a feature amount of the captured image, and matches the feature amount of the captured image with the feature amount template. Do. Then, it is determined whether or not the captured image includes the same object as the object having the feature amount indicated by the matched feature amount template. When the object detection unit determines that the same object as the object having the feature amount indicated by the matched feature amount template is included, the object detection unit extracts the object from the captured image, and generates object information indicating the extracted object.

The object detection unit detects an object outside the vehicle or an object inside the vehicle.

In addition, when the name of the object indicated by the template is associated with the image template or the feature amount template, the object detection unit adds object name information indicating the name of the object to the object information indicating the extracted object. It may be added as

The area extraction unit extracts a characteristic area (pixel group) from the query image using algorithms such as Saliency Map and segmentation (segmentation), identifies the extracted area as an object area, Information is generated.

For example, when using the Saliency Map, the area extraction unit generates a feature map indicating the contrast of the feature amount such as color, luminance, and edge from the captured image, and adds and averages each pixel of each feature map to the saliency map (SM) is generated, and a region having a high contrast in SM (for example, a pixel group having a pixel value equal to or larger than a predetermined value) is extracted. Saliency Map is a model of human visual processing, and by extracting a region using Saliency Map, it is possible to automatically specify a region that is likely to be noticed by humans.

In addition, as the area dividing process, specifically, an area dividing process by integrating adjacent pixels, an area dividing process by classifying pixel features, or an area dividing process by a technique called snakes using edges, etc. May be applied.

Also, the object detection unit and the region extraction unit may be configured to be realized using machine learning, similarly to the visual object determination unit 14 described later.

(Visual object determination unit)
The visual recognition object determination unit 14 determines the object that the driver is viewing from the line of sight detected by the line-of-sight detection unit 13 and the object detected by the object detection unit 11. A specific process of the visual object determination unit 14 will be described with reference to FIG.

For example, as shown in FIG. 2, it is assumed that an image of each object is arranged in the captured image 100. The visual recognition object determination unit 14 can acquire the coordinates in the image area of each object with reference to the position information of the vehicle on which the driver is boarded and the map information close to the position information of the vehicle. The visual recognition object determination unit 14 collates the acquired image area coordinates of each object with the visual line information acquired from the visual line detection unit 13, and the position coordinates of the driver's visual line are within the coordinates of the image area of the object. Determine if there is. For example, as illustrated in FIG. 2, when it is determined that the position coordinates of the driver's line of sight A are within the coordinates of the image area of the object OB, the visual object determination unit 14 determines the position coordinates of the driver's line of sight. Starts counting the time remaining within the coordinates of the image area of the object OB. In this way, by comparing the position coordinates of the driver's line of sight with the coordinates of the image area of the object, it is possible to suitably determine which object the driver is viewing. Further, in addition to the information on the line of sight, as described in (Gaze detection unit), by referring to the detection result of the state of the pupil and the number of blinks, it is further determined which object the driver is concentrating on. It can be suitably determined.

The visual recognition object determination unit 14 may detect the line-of-sight movement time. The method for detecting the line-of-sight movement time is not particularly limited. For example, after a part of a specific object is displayed in the captured image, the position coordinates of the line of sight of the driver move within the coordinates of the image area of the object. There is a method of measuring the time to do. Moreover, the method of measuring time until the position coordinate of a driver | operator's eyes | visual_axis moves from a specific object to another object may be sufficient.

The visual object determination unit 14 can preferably determine an object with a high interest level of the driver by referring to the measured time, a change in the facial expression of the driver, and the like.

Note that the visual object determination unit 14 can also calculate the object information that the driver is visually recognizing by machine learning. The specific configuration of the learning process for acquiring the object information visually recognized by the driver is not limited to this embodiment. For example, any one of the following machine learning methods or a combination thereof is used. Can be used.

・ Support Vector Machine (SVM)
・ Clustering
・ Inductive Logic Programming (ILP)
・ Genetic Algorithm (GP)
・ Baysian Network (BN)
・ Neural network (NN)
When using a neural network, the data may be processed and used in advance for input to the neural network. For such processing, a method such as data argumentation (Deta Argumentation) can be used in addition to the one-dimensional arrangement or multidimensional arrangement of data.

Also, when using a neural network, a convolutional neural network (CNN: Convolutional Neural Network) including convolution processing may be used. More specifically, a convolution layer that performs a convolution operation is provided as one or a plurality of layers (layers) included in the neural network, and a filter operation (product-sum operation) is performed on input data input to the layer. It is good also as a structure. Further, when performing the filter operation, a process such as padding may be used together, or an appropriately set stride width may be adopted.

Further, as the neural network, a multi-layer or super multi-layer neural network having several tens to several thousand layers may be used.

(Interest level information generator)
The interest level information generation unit 15 generates interest level information indicating the interest level of the driver with respect to the object determined by the visual object determination unit 14. For example, the interest level information generation unit 15 displays at least one of the number of objects detected by the object detection unit 11, the gaze time detected by the line-of-sight detection unit 13, the line-of-sight movement time, the pupil state, the number of blinks, and the degree of concentration. Refer to and create interest level information.

The relationship between the various information referred to by the interest level information generation unit 15 and the generated interest level information includes, for example, the following.

(Number of objects)
The degree-of-interest information generation unit 15 determines that the degree of interest in the object viewed by the driver is higher as the ratio of the number of objects viewed by the driver to the number of objects in the captured image is smaller, and the degree-of-interest information including the determination result Create

For example, if there are 20 objects in the captured image and there is only one object visually recognized by the driver, when the gaze time is the same, there is only one object in the captured image, and the object visually recognized by the driver is It can be said that the driver is paying more attention to the visually recognized object than the case of one. Therefore, in the former case, it is determined that the degree of interest in the object visually recognized by the driver is higher than in the latter case.

In addition, when there are 10 objects in the captured image and one object is visually recognized by the driver, the driver is more than when there are 10 objects in the captured image and 5 objects visually recognized by the driver. Can be said to be focused by the visually recognized object. Therefore, in the former case, it is determined that the degree of interest in the object visually recognized by the driver is higher than in the latter case.

(Gaze time)
The interest level information generation unit 15 determines that the interest level for an object is higher as the gaze time for a certain object is longer, and generates interest level information including the determination result.

(Gaze movement time)
The degree of interest information generation unit 15 indicates that the shorter the time from when a part of a specific object is displayed in the captured image until the position coordinate of the line of sight of the driver moves within the coordinates of the image area of the object, It is determined that the degree of interest in the object is high, and interest level information including the determination result is created.

In addition, the interest level information generation unit 15 determines that the interest level of each object viewed by the driver is higher as the time interval until the position coordinate of the line of sight of the driver moves from a specific object to another object is longer. The interest level information including the determination result is created.

(Pupil condition)
The degree-of-interest information generation unit 15 determines that the degree of interest in the object is higher as the size of the driver's pupil when viewing a certain object is larger, and creates the degree-of-interest information including the determination result. .

(Number of blinks)
The degree-of-interest information generation unit 15 determines that the degree of interest in the object is higher as the driver blinks when the object is visually recognized at a lower frequency and at a stable interval, and includes the determination result. Create degree information.

(Degree of concentration)
The degree-of-interest information generation unit 15 determines that the degree of interest in the object is higher as the driver's degree of concentration when viewing a certain object is higher, and creates interest level information including the determination result.

The interest level information generation unit 15 may determine whether or not the interest level is high by providing predetermined thresholds for various types of information to be referred to.

Note that the interest level information generation unit 15 may calculate the interest level information by machine learning, similarly to the visual object determination unit 14.

The interest level information generation unit 15 outputs the created interest level information to the recommendation engine 16.

(Recommendation engine)
The recommendation engine 16 refers to the interest level information generated by the interest level information generation unit 15 and generates proposal information for the driver.

As an example, the recommendation engine 16 refers to the information on the object in which the potential interest of the driver is included, which is included in the interest level information generated by the interest level information generation unit 15, and the potential interest is indicated. Proposal information including proposals related to the object is generated.

The proposal information generated by the recommendation engine 16 is notified to the driver by display or voice by the destination presentation unit 17. Moreover, the proposal information produced | generated by the recommendation engine 16 may be transmitted to the portable terminal device 2 directly or via another server or a base station, and the said portable terminal device 2 may display proposal information.

Hereinafter, specific interest level information created by the recommendation engine 16 will be exemplified with reference to Table 1.

(CASE1)
When the interest level information generated by the interest level information generation unit 15 includes information indicating that the driver is paying attention to the store or the signboard belonging to the first category, the recommendation engine 16 determines whether the rest area and the toilet A proposal to stop at least one of them is included in the proposal information.

CASE 1 in Table 1 shows an example of an object noticed in a common sight of the town and proposal information corresponding to the object. Examples of the store belonging to the first category include, as shown in Table 1, a combustion supply station such as a convenience store and a gas station, a power supply station, a restaurant such as a family restaurant, and the like.

(CASE2)
When the interest level information generated by the interest level information generation unit 15 includes information indicating that the driver is paying attention to a store or a signboard belonging to the second category, the recommendation engine 16 is used for restaurants and toilets. A proposal to stop at least one of them is included in the proposal information.

CASE 2 in Table 1 shows an example of the store or signboard belonging to the second category noted in the common sight of the town, and proposal information corresponding to the store or signboard belonging to the second category. As a store which belongs to the 2nd category, as shown in Table 1, restaurants, such as a family restaurant and fast food, a shopping center, etc. are mentioned, for example.

(CASE3)
When the interest level information generated by the interest level information generation unit 15 includes information indicating that attention is paid to a predetermined object in the vehicle, the recommendation engine 16 includes a fuel supply station, a power station, and an automobile supply store. A proposal to stop at at least one of the above is included in the proposal information.

CASE 3 in Table 1 shows an example of an object in the vehicle that has been transferred in the vehicle, and proposal information corresponding to the object in the vehicle. Examples of the objects in the vehicle include a fuel meter, a clock, a display screen of a navigation system, and a passenger. Fuel stations include gas stations and hydrogen stations.

(CASE4)
When the interest level information generated by the interest level information generation unit 15 includes information indicating that the driver is paying attention to the specific logo or mark, the recommendation engine 16 determines that the specific logo or mark, Or the information regarding the goods relevant to the said specific logo or mark is included in proposal information.

CASE 4 in Table 1 shows an example of a display that is noticed in the town and proposal information corresponding to the display. The information regarding the specific logo or mark or the product related to the specific logo or mark may be output to a mobile terminal device such as a smartphone after returning home after driving.

(CASE5)
When the interest level information generated by the interest level information generation unit 15 includes information indicating that the driver is paying attention to a specific vehicle type, the recommendation engine 16 determines that the specific vehicle type The information related to the car belonging to the car model is included in the proposal information.

CASE 5 in Table 1 shows an example of a car noticed on the road and proposal information corresponding to the car. The vehicle type may be a manufacturer name or a model name of each manufacturer's automobile.

(Destination presentation part)
The destination presentation unit 17 refers to the proposal information to the driver generated by the interest level information generation unit 15 and presents the destination to the driver. The destination presentation unit 17 may present the destination by voice, may present the destination as an image on the display unit, or may present the destination by combining them. In addition, the destination presentation unit 17 may present a route to the destination.

For example, as shown in Table 1, in the case of CASE 1 and CASE 2, the destination presentation unit 17 provides voice guidance of the number of nearby spots, the distance to the spot, and the time. In the case of CASE 3, the destination presentation unit 17 sets the destination and the shortest route.

[Processing Example 1 of Information Processing Device]
Next, processing of the information processing apparatus 1 will be described with reference to the flowchart of FIG.

First, use of the information processing apparatus 1 is started, and the process proceeds to steps (hereinafter, “step” is omitted) S11 and S14.

In S11, the face information acquisition unit 12 acquires the driver's face information. Details of the processing are as described in (Face Information Acquisition Unit). The face information acquisition unit 12 transmits the acquired face information to the line-of-sight detection unit 13, and proceeds to S12.

In S12, the line-of-sight detection unit 13 detects and extracts the line of sight from the acquired face image. Details of the processing are as described in (Gaze detection unit).

In S13, the line-of-sight detection unit 13 determines whether or not the driver's line of sight has been focused from the detected line of sight, and if it has been focused (Yes in S13), the identified line-of-sight direction is displayed in the visual object determination unit 14. Send to S15. If the driver's line of sight is not focused (No in S13), the process in S11 is performed again.

In S14, the object detection unit 11 detects and extracts each object existing in the driver's field of view. Specifically, after the imaging unit captures a scene outside the vehicle, the object extraction unit extracts an object from the acquired captured image 100. Details of the processing are as described in (Object Detection Unit). The object extraction unit of the object detection unit 11 transmits the extracted object to the visual object determination unit, and proceeds to S15.

In S15, the visual recognition object determination unit 14 extracts an object that the driver is viewing from the visual line detected by the visual line detection unit 13 and the object detected by the object detection unit 11, and the process proceeds to S16.

In S16, the visual object determination unit 14 determines whether or not the driver has focused on the object that the extracted driver is viewing for a certain period of time, and if the driver has been focused on for a certain period of time (Yes in S16), Proceed to S17. Details of the processing are as described in (visual recognition object determination unit). If the driver has not paid attention for a certain time (No in S16), the process of S15 is performed again.

In S17, the interest level information generation unit 15 generates interest level information related to the interest level of the driver. Details of the processing are as described in (Interesting degree information generation unit). The interest level information generation unit 15 transmits the created interest level information to the recommendation engine 16 and proceeds to S18.

In S18, the recommendation engine 16 refers to the interest level information generated by the interest level information generation unit, and generates proposal information to the driver. Details of the processing are as described in (Recommendation Engine).

<Embodiment 2>
Another embodiment of the present invention will be described below. For convenience of explanation, members having the same functions as those described in the above embodiment are given the same reference numerals, and the description thereof will not be repeated.

FIG. 4 is a block diagram showing components of the information processing apparatus 1 according to the present embodiment.

The information processing apparatus 1 further includes a position information acquisition unit 21 and a facial expression estimation unit 22.

(Location information acquisition unit)
The position information acquisition unit 21 acquires position information of a vehicle on which the driver is boarded. The position information acquisition unit 21 includes at least one of a GPS antenna, a Wi-Fi (registered trademark) antenna, a direction magnet, an acceleration sensor, and the like. The position information acquisition unit 21 can acquire position information from a position detection unit (not shown) configured to be able to detect a direction in which the vehicle is facing, position information such as a current position, and the like. Alternatively, the position information acquisition unit 21 may acquire the position of the vehicle from other than the position detection unit. For example, when the vehicle moves while performing wireless communication, the position information of the vehicle may be acquired from a wireless communication base station. In the description of the present embodiment, the term “location information” refers to vehicle position information. The position information acquisition unit 21 outputs the acquired position information of the vehicle on which the driver is boarded to the object detection unit 11.

(Facial expression estimation unit)
The facial expression estimation unit 22 estimates the driver's facial expression from the driver's face information. The expression estimation unit 22 estimates expression of the driver by acquiring expression information indicating the expression of the driver from the driver face information. The facial expression estimation unit 22 estimates the driver's facial expression by using the feature amount of each part of the face as face information in addition to using the eye feature amount detected by the line-of-sight detection unit 13.

The driver's face information only needs to include expression information sufficient to estimate the driver's expression. For example, smile information indicating the degree of smile of the driver, sadness information indicating the degree of sadness, and tension information indicating the degree of tension. Including emotion information. The method by which the facial expression estimation unit 22 estimates the facial expression of the driver is not particularly limited. For example, the facial expression estimation apparatus 22 uses a facial expression estimation apparatus equipped with a known technique for estimating the facial expression of a person such as OKAO (registered trademark) Vision. Can be estimated.

The facial expression estimation unit 22 refers to the facial information, estimates the facial expression of the driver, and calculates facial expression information. The facial expression estimation unit 22 outputs facial expression information to the interest level information generation unit 15. The facial expression estimation unit 22 may be provided in the vehicle or may be provided outside the vehicle such as a server, as will be described later.

It should be noted that the facial expression estimation unit 22 may be configured to calculate information related to the facial expression of the driver by machine learning, similar to the visual object determination unit 14.

The facial expression estimation unit 22 outputs the estimated emotion information to the interest level information generation unit 15.

(Interest level information generator)
The interest level information generation unit 15 generates the interest level information by further referring to the vehicle position information and the driver's facial expression.

(location information)
The degree-of-interest information generation unit 15 determines that the degree of interest in the object is higher as the position of the vehicle when the driver visually recognizes the object is higher in the traffic volume. In addition, the interest level information generation unit 15 may determine that the degree of interest in the object is high when the position of the vehicle when the object is visually recognized is a predetermined place. Then, the interest level information generation unit 15 creates interest level information including the determination result.

(Driver's expression)
The degree-of-interest information generation unit 15 determines that the degree of interest in the object is higher as the driver looks more happy or surprised when the driver visually recognizes the object, and includes the determination result. Create information.

The interest level information generation unit 15 may generate the interest level information by further referring to at least one of the driver's biological information, date / time information, and weather information.

The biometric information includes, for example, driver's brain information, vital information, and the like. The brain information includes information such as a driver's brain wave. The method for measuring the electroencephalogram is not particularly limited, and examples thereof include a method for measuring with a known electroencephalograph. The vital information includes, for example, information such as the driver's pulse, blood pressure, body temperature, and sweating amount. The method of measuring vitals is not particularly limited, and examples thereof include a method of measuring with a known pulse meter, blood pressure meter, thermometer, sweat amount detector, and the like. The date / time information is date / time information when the driver is driving, and the weather information is weather information on the position of the vehicle acquired from the position information acquisition unit 21.

(Electroencephalogram)
The interest level information generation unit 15 determines that the interest level of the object is high when the driver's brain wave when the driver visually recognizes the object is within a predetermined frequency range, and includes the interest level information including the determination result. create.

(pulse)
The interest level information generation unit 15 determines that the greater the pulse rate of the driver when the driver visually recognizes the object, the higher the interest level with respect to the object, and creates interest level information including the determination result.

(blood pressure)
The interest level information generation unit 15 determines that the higher the blood pressure of the driver when the driver visually recognizes the object, the higher the interest level for the object, and creates the interest level information including the determination result.

(Body temperature)
The interest level information generation unit 15 determines that the higher the body temperature of the driver when the driver visually recognizes the object, the higher the interest level for the object, and creates the interest level information including the determination result.

(Sweating amount)
The degree-of-interest information generation unit 15 determines that the degree of interest in the object is higher as the amount of sweating of the driver when the driver visually recognizes the object is larger, and creates the degree-of-interest information including the determination result.

(Recommendation engine)
As an example, the recommendation engine 16 uses the interest level information generation unit 15 to generate object information that suggests potential interest of the driver included in the interest level information generated by further referring to the vehicle position information and the driver's facial expression. The proposal information including the proposal related to the object in which the potential interest is suggested is generated with reference to the proposal information.

Specific examples of proposal information include the example described in the first embodiment.

Also, the recommendation engine 16 acquires accident information indicating that an accident has occurred, and generates accident information including object information indicating an object visually recognized by the driver in a predetermined period before the accident occurrence time indicated by the accident information. May be.
The accident information may be generated by an accident information generation unit mounted on the vehicle when an impact sensor (acceleration sensor) mounted on the vehicle detects an impact of a predetermined value or more. And it is good also as a structure which transmits such impact information from each vehicle to a server, and the said server produces | generates accident information with reference to the said impact information.

The recommendation engine 16 may include date and time information when the accident occurs, vehicle position information, and weather information in the accident information. The generated accident information can be used as information for skill evaluation of safe driving.

[Processing Example 2 of Information Processing Device]
Next, processing of the information processing apparatus 1 will be described with reference to the flowchart of FIG. S11 to S17 are as described in [Processing example 1 of information processing apparatus], and only differences will be described.

First, use of the information processing apparatus 1 is started, and the process proceeds to S11 and S21.

In S21, the position information acquisition unit 21 acquires the position information of the vehicle on which the driver is boarded. Details of the processing are as described in (Position information acquisition unit). The position information acquisition unit 21 transmits the acquired position information of the vehicle on which the driver is boarded to the object detection unit 11, and proceeds to S14.

In S16, when the driver has focused on the object for a certain period of time (Yes in S16), the process proceeds to S22.

In S22, the facial expression estimation unit 22 estimates the driver's facial expression from the driver's face information, and proceeds to S23. Details of the processing are as described in (Expression estimation unit).

In S23, the facial expression estimation unit 22 measures the emotional change to the visual recognition object, transmits the measured emotion information to the interest level information generation unit 15, and proceeds to S17.

[Embodiment 3]
Another embodiment of the present invention will be described below. For convenience of explanation, members having the same functions as those described in the above embodiment are given the same reference numerals, and the description thereof will not be repeated.

(Configuration of information processing system)
FIG. 6 is a block diagram illustrating a configuration of an information processing system according to the third embodiment of the present invention. As shown in FIG. 6, the information processing system includes a vehicle 3, a server 4, and a mobile terminal device 2.

The vehicle 3 includes an in-vehicle camera 31 and a destination presentation unit 17. In the first embodiment, the destination presentation unit 17 is configured to be included in the information processing apparatus 1, but in the present embodiment, the destination presentation unit 17 is provided in the vehicle 3.

The server 4 includes a face information acquisition unit 12, a line-of-sight detection unit 13, an object detection unit 11, a visual object determination unit 14, an interest level information generation unit 15, and a recommendation engine 16. As described above, the server 4 according to the present embodiment includes a configuration other than the destination presentation unit 17 among the configurations included in the information processing apparatus 1 according to the first embodiment.

The server 4 may be configured to further include at least one of the position information acquisition unit 21 and the facial expression estimation unit 22 described in the second embodiment.

Further, the server and the information processing apparatus include a communication unit (not shown), and information exchange between the server and the information processing apparatus is performed via the communication unit.

The in-vehicle camera 31 has the same configuration as that of the imaging unit included in the object detection unit 11 described in the first embodiment. The in-vehicle camera 31 captures the scenery outside the vehicle to acquire the surrounding environment information, and uses the acquired surrounding environment information as the object of the server 4. Output to the detector 11. Further, the in-vehicle camera 31 captures the face of the driver to acquire face information, and outputs the acquired face information to the face information acquisition unit 12 of the server 4.

The recommendation engine 16 outputs the generated proposal information to the driver to the destination presentation unit 17 as in the first and second embodiments.

As described above, even if the vehicle 3 does not include the visual object determination unit 14, the interest level information generation unit 15, the recommendation engine, and the like, the processing can be performed via the server 4.

Even with such a configuration, the driver's interest level can be sensed.

[Example of software implementation]
The control block of the information processing apparatus 1 may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software.

In the latter case, each unit included in the information processing apparatus 1 includes a computer that executes instructions of a program that is software that realizes each function. The computer includes, for example, at least one processor (control device) and at least one computer-readable recording medium storing the program. In the computer, the processor reads the program from the recording medium and executes the program, thereby achieving the object of the present invention. As the processor, for example, a CPU (Central Processing Unit) can be used. As the recording medium, a “non-temporary tangible medium” such as a ROM (Read Only Memory), a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. Further, a RAM (Random Access Memory) for expanding the program may be further provided. The program may be supplied to the computer via an arbitrary transmission medium (such as a communication network or a broadcast wave) that can transmit the program. Note that one embodiment of the present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

The information processing apparatus according to each aspect of the present invention may be realized by a computer. In this case, the information processing apparatus is operated on each computer by causing the computer to operate as each unit (software element) included in the information processing apparatus. The control program for the information processing apparatus to be realized in this way and a computer-readable recording medium on which the control program is recorded also fall within the scope of the present invention.

[Summary]
An information processing apparatus according to an aspect of the present invention includes a face information acquisition unit that acquires face information of a driver, a line-of-sight detection unit that detects a line of sight from face information acquired from the face information acquisition unit, and the driver's field of view. An object detection unit that detects each existing object; a visual object determination unit that determines an object that the driver is viewing from the line of sight detected by the line-of-sight detection unit and the object detected by the object detection unit; An interest level information generation unit that generates interest level information indicating the interest level of the driver with respect to the object determined by the visual object determination unit.

An information processing apparatus according to an aspect of the present invention further includes a position information acquisition unit that acquires position information of a vehicle on which the driver is boarded, and the interest level information generation unit further refers to the position information of the vehicle. Interest level information may be generated.

The information processing apparatus according to an aspect of the present invention further includes a facial expression estimation unit that estimates the facial expression of the driver from the facial information of the driver, and the interest level information generation unit further refers to the facial expression of the driver, Interest level information may be generated.

In the information processing apparatus according to an aspect of the present invention, the interest level information generation unit may generate the interest level information by further referring to the biological information of the driver.

In the information processing apparatus according to an aspect of the present invention, the interest level information generation unit may generate the interest level information by further referring to the date / time information.

In the information processing apparatus according to one aspect of the present invention, the interest level information generation unit may generate the interest level information by further referring to the weather information.

The information processing apparatus according to an aspect of the present invention may further include a proposal information generation unit that generates the proposal information for the driver with reference to the interest level information generated by the interest level information generation unit.

In the information processing apparatus according to an aspect of the present invention, the proposal information generation unit acquires accident information indicating that an accident has occurred, and the driver performs a predetermined period before the accident occurrence time indicated by the accident information. Accident information including object information indicating the visually recognized object may be generated.

In the information processing apparatus according to an aspect of the present invention, the interest level information generated by the interest level information generation unit includes information indicating that the driver is paying attention to a store or a signboard belonging to the first category. In this case, the proposal information generation unit may include a proposal to stop at at least one of a rest place and a toilet in the proposal information.

In the information processing apparatus according to an aspect of the present invention, the interest level information generated by the interest level information generation unit includes information indicating that the driver is paying attention to a store or a signboard belonging to the second category. In this case, the proposal information generation unit may include a proposal to stop at at least one of a restaurant and a toilet in the proposal information.

The information processing apparatus according to an aspect of the present invention is configured to generate the proposal information when the interest level information generated by the interest level information generation unit includes information indicating that attention is paid to a predetermined object in the vehicle. The section may include a proposal to stop at at least one of a fuel supply station, a power supply station, and an automobile supply store in the proposal information.

The information processing apparatus according to an aspect of the present invention may further include a destination presentation unit that refers to the proposal information and presents the destination to the driver.

In the information processing apparatus according to an aspect of the present invention, when the interest level information generated by the interest level information generation unit includes information indicating that the driver is paying attention to a specific logo or mark. The proposal information generation unit may include information related to the specific logo or mark or a product related to the specific logo or mark in the proposal information.

In the information processing apparatus according to an aspect of the present invention, when the interest level information generated by the interest level information generation unit includes information indicating that the driver is paying attention to a specific vehicle type, The proposal information generation unit may include information regarding the specific vehicle type or an automobile belonging to the specific vehicle type in the proposal information.

An information processing method according to an aspect of the present invention is an information processing method executed by an information processing apparatus, and includes a face information acquisition step for acquiring driver face information, and the face information acquired in the face information acquisition step. The driver visually recognizes a line-of-sight detection step for detecting a line of sight, an object detection step for detecting each object existing in the field of view of the driver, and the line of sight detected in the line-of-sight detection step and the object detected in the object detection step. A visual recognition object determination step for determining the object that is being performed, and an interest level information generation step for generating interest level information indicating the interest level of the driver with respect to the object determined in the visual recognition object determination step.

The program according to an aspect of the present invention is an information processing program for causing a computer to function as an information processing apparatus, and may cause the computer to function as the interest level information generation unit.

The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention. Furthermore, a new technical feature can be formed by combining the technical means disclosed in each embodiment.

DESCRIPTION OF SYMBOLS 1 Information processing apparatus 2 Portable terminal device 3 Vehicle 4 Server 11 Object detection part 12 Face information acquisition part 13 Gaze detection part 14 Visual object determination part 15 Interest degree information generation part 16 Recommendation engine (proposition information generation part)
17 Destination presentation unit (navigation system)
21 Position information acquisition unit 22 Facial expression estimation unit 31 Car-mounted camera

Claims

A face information acquisition unit for acquiring driver face information;
A gaze detection unit that detects a gaze from the face information acquired from the face information acquisition unit;
An object detection unit for detecting each object existing in the field of view of the driver;
A visual object determination unit that determines an object that the driver is viewing from the line of sight detected by the line of sight detection unit and the object detected by the object detection unit;
An information processing apparatus comprising: an interest level information generation unit configured to generate interest level information indicating an interest level of the driver for the object determined by the visual recognition object determination unit.
A position information acquisition unit for acquiring position information of a vehicle on which the driver is boarded;
The information processing apparatus according to claim 1, wherein the interest level information generation unit generates the interest level information by further referring to the position information of the vehicle.
A facial expression estimation unit for estimating the facial expression of the driver from the facial information of the driver;
The information processing apparatus according to claim 1, wherein the interest level information generation unit generates the interest level information by further referring to the facial expression of the driver.
The information processing apparatus according to any one of claims 1 to 3, wherein the interest level information generation unit generates the interest level information by further referring to the biological information of the driver.
5. The information processing apparatus according to claim 1, wherein the interest level information generation unit generates the interest level information by further referring to the date and time information.
6. The information processing apparatus according to claim 1, wherein the interest level information generation unit further generates reference level information by further referring to weather information.
7. The information processing apparatus according to claim 1, further comprising a proposal information generation unit that generates proposal information for the driver with reference to the interest level information generated by the interest level information generation unit. The information processing apparatus described in 1.
The proposal information generation unit
Acquire accident information indicating that an accident occurred,
8. The information processing apparatus according to claim 7, wherein accident information including object information indicating an object viewed by the driver in a predetermined period before an accident occurrence time indicated by the accident information is generated.
When the interest level information generated by the interest level information generation unit includes information indicating that the driver is paying attention to a store or a signboard belonging to the first category,
The information processing apparatus according to claim 7, wherein the proposal information generation unit includes a proposal to stop at at least one of a rest place and a toilet in the proposal information.
When the interest level information generated by the interest level information generation unit includes information indicating that the driver is paying attention to a store or a signboard belonging to the second category,
The information processing apparatus according to any one of claims 7 to 9, wherein the proposal information generation unit includes a proposal to stop at at least one of a restaurant and a toilet in the proposal information.
When the interest level information generated by the interest level information generation unit includes information indicating that attention is paid to a predetermined object in the vehicle,
11. The proposal information generation unit according to claim 7, wherein the proposal information includes a proposal to stop at at least one of a fuel supply station, a power supply station, and an automobile supply store. The information processing apparatus described.
12. The information processing apparatus according to claim 7, further comprising a destination presentation unit that refers to the proposal information and presents the destination to the driver.
When the interest level information generated by the interest level information generation unit includes information indicating that the driver is paying attention to a specific logo or mark,
The proposal information generation unit
The specific logo or mark, or
The information processing apparatus according to any one of claims 7 to 12, wherein information relating to a product related to the specific logo or mark is included in the proposal information.
When the interest level information generated by the interest level information generation unit includes information indicating that the driver is paying attention to a specific vehicle type,
The proposal information generation unit
The above specific vehicle type, or
The information processing apparatus according to any one of claims 7 to 13, wherein information relating to an automobile belonging to the specific vehicle type is included in the proposal information.
An information processing method executed by an information processing apparatus,
A face information acquisition step for acquiring driver face information;
A line of sight detection step of detecting a line of sight from the face information acquired in the face information acquisition step;
An object detection step of detecting each object present in the field of view of the driver;
A visual object determination step for determining an object that the driver is viewing from the visual line detected in the visual line detection step and the object detected in the object detection step;
An interest level information generating step of generating interest level information indicating the level of interest of the driver for the object determined in the visual object determination step.
An information processing program for causing a computer to function as the information processing apparatus according to claim 1, wherein the information processing program causes the computer to function as the interest level information generation unit.