JP2024120092A - In-vehicle device and recording method - Google Patents

Abstract

[Problem] When a driver remembers something while driving, information related to the remembered fact is recorded. [Solution] An in-vehicle device 100 includes a gaze detection unit 120 that detects the gaze direction of the driver of the vehicle, an estimation unit 130 that determines whether the driver is looking at a specific object in the gaze direction based at least on the speed of the driver's gaze movement and estimates the object, an information processing unit 160 that records information related to the object as memo information, and an output unit 190 that outputs the memo information recorded during driving when driving of the vehicle is completed. [Selected Figure] Figure 1

Description

The present invention relates to an in-vehicle device and a recording method.

A technology has been proposed that extracts matters of interest from the behavior of vehicle occupants and presents information related to the matters of interest. Patent Document 1 discloses a technology that detects a driving pattern that is different from usual, determines whether the driver is showing interest based on changes in behavior such as eye movements, and presents information related to facilities that the driver is showing interest in.

Patent Literature 2 presents a technology for extracting the interests of bus passengers and presenting related information, where the interests of passengers are extracted from dynamic information such as gaze and conversation, and customer information registered in advance in the passenger's account.

The techniques disclosed in Patent Documents 1 and 2 make it possible to present information desired by the driver, etc., using a change in behavior as a trigger. However, it is difficult to use these techniques to record something that the driver suddenly remembers while driving the vehicle, or to extract an interest that the driver is not aware of.

JP 2009-168473 A JP 2020-57066 A

As described above, when a driver remembers something while driving, there is a problem in that the driver cannot take his/her hands off the wheel while driving, making it difficult to record information related to the fact that he/she remembered.
In addition, there are many things to think about while driving, such as operating the vehicle at traffic lights and curves, avoiding approaching other vehicles, selecting a driving route, etc., so there is a problem that what you remember is forgotten when you finish driving. Therefore, we provide an in-vehicle device and a recording method that can record information related to a fact that the driver remembers when he or she remembers something while driving.

The present invention provides an in-vehicle device that includes a gaze detection unit that detects the gaze direction of a vehicle driver, an estimation unit that determines whether the driver is looking at a specific target in the gaze direction based at least on the speed of the driver's gaze movement and estimates the target, an information processing unit that records information related to the target as memo information, and an output unit that outputs the memo information recorded during driving when driving of the vehicle is completed.

The present invention provides a recording method including a gaze detection step for detecting the gaze direction of a vehicle driver, an estimation step for determining whether the driver is looking at a specific target in the gaze direction based at least on the speed of the driver's gaze movement and estimating the target, an information processing step for recording information related to the target as memo information, and an output step for outputting the memo information recorded during driving when driving of the vehicle is completed.

According to the present invention, it is possible to provide an in-vehicle device and a recording method that, when a driver remembers something while driving, can record information related to the fact that the driver remembers.

1 is a block diagram showing a configuration of an in-vehicle device according to a first embodiment; 4 is a flowchart showing an operation of the in-vehicle device according to the first embodiment. FIG. 11 is a block diagram showing a configuration of an in-vehicle device according to a second embodiment.

(Embodiment 1)
First, the configuration of an in-vehicle device 100 according to the first embodiment will be described with reference to FIG. 1. The in-vehicle device 100 is, for example, a drive recorder. The in-vehicle device 100 includes an image acquisition unit 1
The image forming apparatus includes a gaze detection unit 10 , a gaze detection unit 120 , an estimation unit 130 , a storage unit 140 , a determination unit 150 , an information processing unit 160 , and an information presentation unit 170 .

The image acquisition unit 110 acquires images of the driver of the vehicle and the outside of the vehicle from the camera 10. The image acquisition unit 110 may acquire images of the driver and the outside of the vehicle simultaneously using an ultra-wide-angle lens (fisheye, 360 lens) or the like, or may acquire multiple images of the driver and the outside of the vehicle captured by separate cameras. The camera 10 may be an infrared camera. Acquisition of images of the outside of the vehicle in the image acquisition unit 110 is not necessarily a required configuration, and may be omitted.

The gaze detection unit 120 detects the gaze direction from the face image of the driver. The gaze direction is detected as one of nine quadrants, for example, upper, middle, lower, left, middle, and right. It is preferable that the gaze direction is detected in detail. Furthermore, the gaze detection unit 120 may detect a period during which the driver is gazing in a certain direction, or may detect gaze movement accompanying the traveling of the vehicle. In addition, the gaze detection unit 120
The gaze detection unit 120 may detect a gaze movement caused by watching a moving object. The gaze detection unit 120 outputs the detected gaze direction to the estimation unit 130.

The gaze detection unit 120 can detect the gaze based on the positional relationship between the inner corner of the eye and the iris, or the positional relationship between the corneal reflex and the pupil. The gaze detection unit 120 may detect the direction in which the driver is looking based on the direction of the face, and use this as the gaze direction. A known technique may be used as the gaze detection technique.

The estimation unit 130 estimates the target that the driver is looking at based on at least the line of sight direction.
For example, when the estimation unit 130 detects that the driver's line of sight is slightly below the center in the left-right direction, it estimates that the object the driver is looking at is the road or another vehicle traveling in front of the vehicle. The object the driver is looking at may be estimated based on the line of sight direction and the vehicle surroundings. The vehicle surroundings are determined from an image captured by the image acquisition unit 110 of the outside of the vehicle, or map information acquired from a car navigation device (not shown) or the Internet via a communication unit (not shown). The estimation unit 130 outputs the estimation result to the determination unit 150.

Here, the estimation unit 130 first determines whether the driver is looking in the entire line of sight or looking (gazing) at a specific object in the line of sight based on the movement of the driver's pupils, changes in the direction of the face, the period during which the driver gazes in a certain direction, etc. For example, if the driver often looks at the center in the left-right direction and the movement of the driver's gaze is small, the estimation unit 130 may determine that the driver is looking in the center of the left-right direction and looking at (gazing at) a specific object in the line of sight.
It is assumed that the driver is looking in the entire direction of the line of sight. The speed and direction of the driver's line of sight movement,
Alternatively, the estimation unit 130 may determine whether or not there is a correlation with the vehicle's moving speed to determine whether the driver is looking at a specific target in the line of sight.
The object that the driver is looking at is estimated. The object may be estimated as a single facility such as "XX Post Office" or as a collective name for multiple stores such as "XX Burger."
The object may be estimated as one genre, such as "ramen shop." The object is not limited to a facility, and may be a rice field, a plantation, or the like.

The estimation unit 130 may estimate the object based on the line of sight and the vehicle exterior image using an image recognition technique. For example, when the driver is looking to the upper left and the vehicle exterior image shows a post office sign, the estimation unit 130 can estimate that the object the driver is looking at is the post office. The estimation unit 130 may estimate that the object the driver is looking at is the post office by performing image recognition of roads, other vehicles, traffic lights, signs, and the like.

The estimation unit 130 may also estimate the object that the driver is looking at based on the line of sight and map information. For example, when the driver is looking slightly to the left ahead, the estimation unit 130 may acquire from the map information that the facility located to the left ahead of the vehicle's current position in the traveling direction is a post office, and may estimate that the object that the driver is looking at is the post office. The estimation unit 130 may also acquire from the map information the presence of a traffic light, a sign, or the like, and estimate that the object that the driver is looking at is the post office.

Furthermore, the estimation unit 130 may estimate an object that the driver is looking at based on the line of sight direction and the vehicle driving information. For example, when the driver looks directly to the left and then the vehicle turns left, the estimation unit 130 may estimate that the object that the driver is looking at is the left side mirror.

The estimation unit 130 may estimate that the object the driver is looking at is the accident site based on the line of sight direction and the information indicating the location of the accident site. The information indicating the location of the accident site may be obtained from the Internet or the like via a communication unit (not shown).

The storage unit 140 is a storage device or storage medium such as a hard disk, a flash memory, or an SD card. The storage unit 140 may also include a volatile storage device such as a RAM (Random Access Memory) that is a storage area for temporarily storing information.
It stores history 141 and memo information 142. The storage unit 140 may further store images captured by the camera 10.

History 141 is a database that manages the history of the estimation results of the estimation unit 130.
The history 141 may store only objects that are not related to the driving of the vehicle. The history 141 may store the estimated object in association with the time when the driver saw the object.
The history 141 may store the length of time that the driver looked at the target. The memo information 142 is information recorded by the information processing unit 160, which will be described later.
may be stored in a server external to the in-vehicle device 100.

When the object estimated by the estimation unit 130 is not related to the driving of the vehicle, the determination unit 150 determines the frequency with which the driver looks at the object, that is, whether the driver usually looks at the object, based on the history 141. The determination result by the determination unit 150 is sent to the information processing unit 160 or the information presentation unit 1
70 is output.

Here, the determination unit 150 first determines whether the object the driver is looking at is related to the driving of the vehicle. The object related to the driving of the vehicle is a road, another vehicle, a traffic light, a sign, a mirror, etc. The object related to the driving of the vehicle may include an accident scene.

Next, the determination unit 150 determines whether the driver usually looks at the target.
may determine whether the frequency of visually observing the target is equal to or greater than a predetermined value 1 depending on whether the target is included in the history 141, or may determine whether the frequency of visually observing the target is equal to or greater than a predetermined number depending on whether the target is included a predetermined number or more in the history 141. The determination unit 150 may also determine the frequency of visually observing the target by referring to the history 141 of the most recent period, such as whether the target is viewed three or more times in one week, or may determine the frequency of visually observing the target by comparing with the driving history, such as whether the target is viewed once or more when passing a road adjacent to the target twice.

Here, the determination unit 150 may make the determination in consideration of the time during which the driver looks at the target (gazing time). For example, the determination unit 150 may determine that the driver usually looks at the target when the number of times the driver looks at the target for one second or more is equal to or greater than a predetermined number.

Here, the determination unit 150 may refer to the history 141 while taking into consideration conditions such as a time period, a day of the week, etc. For example, the determination unit 150 may determine whether the driver usually sees the target, taking into consideration that restaurants tend to be seen at night or on holidays.

When the driver does not normally see the target, the information processing unit 160 records information related to the target as the memo information 142. Therefore, when the target is not related to driving and is not normally seen by the driver, the information processing unit 160 records the memo information 142. The information processing unit 160 may record the name of the target estimated by the estimation unit 130 as the memo information 142, or may acquire an image in which the target is shown from the image acquisition unit 110 and record it as the memo information 142. The information processing unit 160 may capture map information of the target estimated by the estimation unit 130, for example, a screen of a car navigation device (not shown), and record it as the memo information 142.

In addition, the information processing unit 160 creates a message including the name of the target as a part of the memo information 1
For example, if the object the driver is looking at is a post office, the message may be, "Do you have business at the post office?" The information processing unit may create the message as a voice memo or an image memo and record it as memo information 142.

The memo information 142 may also be a voice uttered by the driver. The information processing unit 160 may notify the driver of a message including the name of the target (for example, "Do you have business at the post office?"), and then collect the driver's voice with the microphone 20 and record it as the memo information 142. The information processing unit 160 may output the message from the speaker 30 or to a monitor. This allows the driver to remember that he or she has business at the post office, and to record information by voice or the like ("I have business at the post office").

Here, the information processing unit 160 may record a video of a period including both the time when the target was photographed and the period during which the driver spoke as the memo information 142. The information processing unit 160 may also record a voice uttered by the driver and an external image of the vehicle during the period during which the voice was uttered as the memo information 142. That is, the information processing unit 160 may detect the driver's speech and record an external image of the period until the end of the speech.

It is also preferable to link the estimated name of the object (e.g., "XX Post Office") to the created file. Therefore, the information processing unit 160 may record the memo information 142 and the name of the object in association with each other. For example, the information processing unit 160 may set the name "XX Post Office" as the file name of the memo information 142, and may create a voice memo by voice-synthesizing the name,
The name may be combined with the memo information 142, the name may be superimposed on the image memo as subtitle data, or the name may be embedded in the memo information 142 as metadata.

When the driver normally looks at an object, the information presenting unit 170 presents information related to the object to the driver. Therefore, when the object is not related to driving and is something the driver normally looks at, the information presenting unit 170 presents information. The information presenting unit 170 may output the presented information to the speaker 30 or to a monitor.

The information presentation unit 170 may output information related to the name of the target, such as "You often look at XX ramen." This allows the driver to recognize that he or she is potentially interested in the target.

The information presentation unit 170 may also obtain information related to the target name from the Internet or the like via a communication unit (not shown) and output the obtained information. For example, the information presentation unit 170 may present to the driver whether or not "XX ramen" is running a campaign, or whether the ramen has a good or bad reputation.

Furthermore, the information presenting unit 170 may transmit information related to the target name to a car navigation device (not shown) so that the information can be searched and registered in the car navigation device.

The information presentation unit 170 may present information when the target is related to driving and is not something the driver would normally see. For example, when the driver sees a no parking sign that the driver does not normally see, it can be determined that the driver is looking for a parking space, so the information presentation unit 170 may present the driver with parking space information searched for on the Internet or the like.
For example, if the driver is looking at the rearview mirror more than usual, it can be determined that the driver is concerned about the following vehicle, and the information presentation unit 170 can present information to encourage the driver to pay attention to the road ahead.

Next, the flow of operations of the in-vehicle device 100 will be described with reference to Fig. 2. First, the gaze detection unit 120 detects the gaze direction of the driver from the face image of the driver in the image acquired by the image acquisition unit 110 (step S101). Next, the estimation unit 130 judges whether the driver is looking at the entire gaze direction or is looking at (gazing at) a specific target in the gaze direction (step S102). If the driver is not looking at a specific target (step S
If the driver is looking at a specific target (Yes in step S102), the estimation unit 130 estimates an target in the driver's line of sight (step S103).

Next, the determination unit 150 determines whether the object estimated in step S103 is directly related to driving (step S104). If the object is related to driving of the vehicle (Yes in step S104), the in-vehicle device 100 returns to the process of step S101.

If the target is not related to driving the vehicle (No in step S104), the determination unit 15
The information processing unit 160 determines whether the driver usually looks at the target (step S105). If the driver does not usually look at the target (No in step S105), the information processing unit 160 determines that the driver has remembered the target (step S106), and records an image memo or a voice memo as the memo information 142 (step S107). Here, the information processing unit 160 may notify a message such as "Do you have business at the post office?"

If the driver normally looks at the target (Yes in step S105), the information presenter 170 determines that the driver is interested in the target (step S108), and presents information about the target to the driver (step S109). Finally, the estimation result history 141 is updated (step S110).

The in-vehicle device 100 according to the first embodiment distinguishes between an object that the driver is looking at and an object that the driver is not looking at.
The 00 can present information that the driver is interested in. It can also automatically record information that the driver remembers as memo information while driving, when the driver cannot take his/her hands off the wheel.

(Embodiment 2)
In the in-vehicle device 100a according to the second embodiment, when driving of the vehicle is ended, the information processing unit 16
3 is a block diagram showing an example of the configuration of the in-car device 100a. The following describes the differences from the first embodiment.

The in-vehicle device 100a is connected to a communication terminal 40 carried by a driver via a network 200. The network 200 may be the Internet, an intranet, a mobile phone network,
A communication line network such as a LAN (Local Area Network).
The device is used on mobile phones, smartphones, tablet devices, and PCs (Personal Computers).
The in-car device 100a may output the memo information 142 to the speaker 30, a monitor screen (not shown), or the like, instead of the communication terminal 40.

The in-vehicle device 100a includes a communication unit 180 and an output unit 190 in addition to the components shown in FIG. 1. The communication unit 180 is a communication interface with the network 200.
In the case where the in-car device 100a outputs the memo information 142 to the speaker 30 or the like, the in-car device 100a does not need to include the communication unit 180.

The output unit 190 determines whether or not driving of the vehicle in which the in-vehicle device 100a is installed has ended. The output unit 190 is linked to a car navigation device (not shown) installed in the vehicle,
The driving may be determined to have ended when the vehicle arrives near the destination or when the destination is a predetermined distance away (e.g., 100 m). The destination may be determined not only by the destination set in the car navigation device, but also by the registered home or the current location being a parking lot. The output unit 190 may also acquire gear information of the vehicle to detect when the vehicle has entered a parking operation,
The output unit 190 may determine that driving has ended when it is determined from the driving information that the parking operation has ended. The output unit 190 may determine that driving has ended by detecting that the engine has been turned off.

The output unit 190 outputs the memo information 142 when driving of the vehicle is completed.
The output unit 190 may output a voice memo from the speaker 30, or may display an image memo on a monitor (not shown). The output unit 190 may output the memo information 142 via the communication unit 180 to a communication terminal 40 that is registered in advance as a destination. The output unit 190 may output the memo information 142 via the communication unit 180 to an email address, a telephone number, a social networking service (SNS), or the like that is registered in advance as a destination.

The in-vehicle device 100a according to the second embodiment outputs the recorded memo information when driving is completed. Therefore, the driver can check the memo information after driving is completed and perform the work that he remembers. In addition, the driver can check the memo information output from the speaker, monitor, etc., and take notes in a notebook, etc.

The above-mentioned program can be stored on various types of non-transitory computer-readable media (e.g.,
Non-transitory computer readable media can be used to store and provide information to a computer. Non-transitory computer readable media includes various types of tangible storage media.
Examples of non-transitory computer-readable media include magnetic recording media, magneto-optical recording media, CD-ROMs (Read Only Memories), CD-Rs, CD-R/Ws,
The magnetic recording medium includes, for example, a flexible disk, a magnetic tape, a hard disk drive, etc. The magneto-optical recording medium includes, for example, a magneto-optical disk, etc. The semiconductor memory includes, for example, a mask ROM, a PROM (Programmable ROM), etc.
M), EPROM (Erasable PROM), Flash ROM, RAM (Ran
The program may also be stored on various types of transient computer readable media, such as a temporary access memory (TEMM).
The program may be supplied to the computer via a temporary computer-readable medium. Examples of the temporary computer-readable medium include an electric signal, an optical signal, and an electromagnetic wave. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

The in-vehicle device according to the above-described embodiment can be applied to the field of driver monitoring systems. It should be noted that a device other than an in-vehicle device can detect the gaze of a person and estimate the object that the person is looking at. For example, a similar system can be realized by attaching a gaze detection device and a camera that captures the surrounding scenery to a glasses-type device. This allows:
This makes it possible to investigate a person's potential interests not only while driving but also during their everyday life.

REFERENCE SIGNS LIST 10 Camera 20 Microphone 30 Speaker 40 Communication terminal 100, 100a Vehicle-mounted device 110 Image acquisition unit 120 Line-of-sight detection unit 130 Estimation unit 140 Storage unit 141 History 142 Memo information 150 Determination unit 160 Information processing unit 170 Information presentation unit 180 Communication unit 190 Output unit 200 Network

Claims (3)

A gaze detection unit that detects the gaze direction of a driver of the vehicle;
An estimation unit that determines whether the driver is looking at a specific target in the line of sight direction based at least on a speed of the driver's line of sight movement and estimates the target;
an information processing unit that records information related to the target as memo information;
an output unit that outputs the memo information recorded during driving when driving of the vehicle is completed;
An in-vehicle device comprising: The estimation unit determines whether the driver is looking at a specific target in the line of sight direction based on whether there is a correlation between a speed of the driver's line of sight movement and a moving speed of the vehicle,
The information processing unit records a message including a name of the target as the memo information,
The output unit outputs the message when at least one of the following times is determined as the end of driving of the vehicle: when the vehicle has arrived near a destination; when the destination is a predetermined distance away; and when a parking operation of the vehicle has been completed.
The in-vehicle device according to claim 1 . a gaze detection step of detecting a gaze direction of a driver of the vehicle;
an estimation step of determining that the driver is looking at a specific target in the line of sight direction based at least on a speed of the driver's line of sight movement, and estimating the target;
An information processing step of recording information related to the subject as memo information;
an output step of outputting the memo information recorded during driving when driving of the vehicle is completed;
A recording method comprising:

Images