JP2023127179A - In-vehicle device, information processing method, and program - Google Patents

Abstract

To appropriately determine conditions of users.SOLUTION: An in-vehicle device 1 includes: a camera 5 that is installed at a position near a ceiling in a cabin of a vehicle and generates an image of a user in the cabin of the vehicle; a line-of-sight detection unit 11 that detects the line of sight of the user reflected in the image generated by the camera 5; a determination unit 12 that determines whether or not the user's line of sight detected by the line of sight detection unit 11 is directed toward a predetermined area including a room mirror provided in the vehicle; a state estimation unit 13 that estimates the state of the user from the image generated by the camera 5 and generates state information indicating the estimated state; a correction unit 14 that corrects the state information generated by the state estimation unit 13 using the result of the determination by the determination unit 12; and a presentation unit 20 that presents the state information after correction by the correction unit 14.SELECTED DRAWING: Figure 2

Description

The present invention relates to an in-vehicle device, an information processing method, and a program.

There is a technology that detects the driver's posture in a vehicle and determines the driver's condition (for example, drowsiness) (see Patent Document 1).

Patent No. 6689470

However, with the above technology, it may not be possible to appropriately determine the condition of the driver (also referred to as the user).

Therefore, the present invention provides an in-vehicle device and the like that can appropriately determine a user's condition.

An in-vehicle device according to one aspect of the present invention is a camera installed near a ceiling in a vehicle interior of a vehicle, and includes a camera that generates an image of a user in the vehicle interior, and a camera that generates an image of a user in the vehicle interior. a detection unit that detects the user's line of sight reflected in the generated image; and a detection unit that detects whether the user's line of sight detected by the detection unit is directed toward a predetermined area including a rearview mirror provided in the vehicle. a determination unit that makes a determination; an estimation unit that estimates the state of the user from an image generated by the camera and generates state information indicating the estimated state; The present invention is an in-vehicle device that includes a correction section that performs correction using a determination result made by the correction section, and a presentation section that presents the state information after the correction section has corrected it.

Note that these comprehensive or specific aspects may be realized by a system, a device, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, and the system, device, integrated circuit, computer program and a recording medium may be used in any combination.

The photographing device of the present invention can appropriately determine the user's condition.

FIG. 1 is an explanatory diagram schematically showing the configuration of a vehicle and an in-vehicle device in an embodiment. FIG. 2 is a block diagram showing the functional configuration of the in-vehicle device in the embodiment. FIG. 3 is a first explanatory diagram showing an example of the position where the camera is installed and the photographing direction in the embodiment. FIG. 4 is a second explanatory diagram showing an example of the position where the camera is installed and the photographing direction in the embodiment. FIG. 5 is an explanatory diagram showing an example of an image showing a user in the embodiment. FIG. 6 is an explanatory diagram illustrating an example of a temporal change in the user's line of sight direction in the embodiment. FIG. 7 is a flow diagram illustrating an example of processing of the in-vehicle device according to the embodiment. FIG. 8 is a block diagram showing the functional configuration of an on-vehicle device in a modification of the embodiment.

(Findings that formed the basis of the present invention)
The present inventor has discovered that the following problem occurs with the technology for estimating the driver's condition described in the "Background Art" section.

A camera that photographs the driver inside the vehicle is sometimes installed at a position in front of the driver's seat (see, for example, Patent Document 1). When photographing the driver from a camera installed in front of the driver's seat, the driver's body may be in the shadow of the steering wheel or the driver's hands gripping the steering wheel, making it difficult for the driver to properly capture the image. Sometimes the picture is not taken. If the driver is not properly photographed, the driver's condition cannot be properly estimated.

On the other hand, a camera that photographs the driver is sometimes installed on the ceiling of the vehicle interior or near the rearview mirror. If the camera is installed on the ceiling of the vehicle interior or near the rearview mirror, the driver's body can be appropriately photographed without being in the shadow of the steering wheel or the driver's hands.

By the way, when the driver is driving the vehicle in appropriate conditions, the driver turns his eyes to the rearview mirror at moderate intervals (for example, every few seconds to several tens of seconds) to check the scene behind the vehicle. It is known that The appropriate state of the driver is a state suitable for driving, in other words, a state where the driver does not feel sleepy or a state where the driver is not distracted.

The driver's line of sight can be analyzed by line of sight detection processing on an image in which the driver is shown. Line-of-sight detection technology detects line-of-sight by detecting the position of the driver's eyeball (iris or pupil, etc.) in the image. By using the image taken by the user, it is possible to more clearly determine whether the user's line of sight is directed toward the target.

Therefore, a line of sight detection process using images taken by a camera installed near the ceiling of the vehicle interior (more specifically, near the rearview mirror) is used to detect whether the user is directing his/her line of sight toward the rearview mirror. The inventors of the present invention have found that the determination can be made more clearly, and the driver's condition can be determined more appropriately based on this determination.

An in-vehicle device according to one aspect of the present invention is a camera installed near a ceiling in a vehicle interior of a vehicle, and includes a camera that generates an image of a user in the vehicle interior, and a camera that generates an image of a user in the vehicle interior. a detection unit that detects the user's line of sight reflected in the generated image; and a detection unit that detects whether the user's line of sight detected by the detection unit is directed toward a predetermined area including a rearview mirror provided in the vehicle. a determination unit that makes a determination; an estimation unit that estimates the state of the user from an image generated by the camera and generates state information indicating the estimated state; The present invention is an in-vehicle device that includes a correction section that performs correction using a determination result made by the correction section, and a presentation section that presents the state information after the correction section has corrected it.

According to the above aspect, the in-vehicle device appropriately determines whether or not the user is directing his or her gaze toward the rearview mirror, and uses the determination result to correct the estimated state of the user. can be determined more appropriately. In particular, since a camera installed near the ceiling inside the vehicle is used, it is possible to more appropriately determine whether the user is looking at the rearview mirror, and the final determination result of the user's condition is improved. It has the effect of making it appropriate. In this way, the in-vehicle device can appropriately determine the user's condition.

For example, the camera generates a plurality of images by photographing the user a plurality of times, and in the determination, the determination unit further determines that the detection unit detects the user using the plurality of images. From the temporal change in the line of sight, it may be determined whether the user's line of sight is directed toward the predetermined area at appropriate time intervals.

According to the above aspect, the in-vehicle device corrects the estimated state of the user by further determining whether the time interval between when the user looks at the rearview mirror is appropriate, and the in-vehicle device corrects the estimated state of the user. More appropriate judgment can be made. Therefore, the in-vehicle device can more appropriately determine the user's condition.

For example, the presentation unit includes the room mirror that is an electronic mirror having a display screen, and the in-vehicle device further includes information generated by photographing with the camera at the timing when the presentation unit presents the information on the display screen. The determination unit includes a first adjustment unit that adjusts the predetermined area from the image to include the area where the user's line of sight is directed, which is obtained by the detection unit, and the determination unit The determination may be made using the predetermined area after the determination.

According to the above aspect, the in-vehicle device adjusts the predetermined area and then makes a determination using the adjusted predetermined area. It is assumed that the user turns his/her line of sight to the electronic mirror at the timing when the electronic mirror presents information, so by using the user's line of sight at that time, a predetermined area is set according to the user's physique or posture. This allows the estimated state of the user to be corrected more appropriately. Therefore, the in-vehicle device can more appropriately determine the user's condition.

For example, the vehicle includes a sensor that detects that another vehicle is close to the rear of the vehicle, and the in-vehicle device further detects that another vehicle is close to the rear of the vehicle. a second adjustment of adjusting the predetermined area so as to include the area to which the user's line of sight obtained by the detection unit is directed from the image generated by photography by the camera at the timing when the sensor is sensing; The determination unit may make the determination using the predetermined area adjusted by the second adjustment unit.

According to the above aspect, the in-vehicle device adjusts the predetermined area and then makes a determination using the adjusted predetermined area. It is assumed that the user turns his/her line of sight to the electronic mirror at the timing when the sensor detects that another vehicle is approaching behind the vehicle, so by using the user's line of sight at that time, the user's A predetermined area can be set according to the physique or posture, and thereby the estimated state of the user can be corrected more appropriately. Therefore, the in-vehicle device can more appropriately determine the user's condition.

For example, the second adjustment unit may specify a period during which the sensor is sensing that another vehicle is approaching behind the vehicle, and may be configured to take pictures by the camera multiple times within the specified period. The predetermined area may be adjusted using the plurality of generated images.

According to the above aspect, the in-vehicle device uses a plurality of images taken during a period when another vehicle is actually close to the rear of the vehicle, so that a user who recognizes that a vehicle is close to the rear of the vehicle can Adjustment of a predetermined area can be made more appropriately by using the action of checking the rear of the vehicle by looking at the rearview mirror. Therefore, the in-vehicle device can more appropriately determine the user's condition.

Further, in the information processing method according to one aspect of the present invention, an image of a user in the vehicle interior is generated by a camera installed near a ceiling in the vehicle interior, and an image is generated by the camera. detecting the line of sight of the user reflected in the image, determining whether the detected line of sight of the user is directed toward a predetermined area including a rearview mirror provided in the vehicle; estimating the state of the user from the image, generating state information indicating the estimated state, correcting the generated state information using the result of the determination, and presenting the corrected state information. It is an information processing method.

According to the above aspect, the same effects as the above vehicle-mounted device can be achieved.

Further, a program according to one aspect of the present invention is a program that causes a computer to execute the above information processing method.

According to the above aspect, the same effects as the above photographing device can be achieved.

Note that these comprehensive or specific aspects may be realized by a system, a device, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, and the system, device, integrated circuit, computer program Alternatively, it may be realized using any combination of recording media.

Hereinafter, embodiments will be specifically described with reference to the drawings.

Note that the embodiments described below are all inclusive or specific examples. The numerical values, shapes, materials, components, arrangement positions and connection forms of the components, steps, order of steps, etc. shown in the following embodiments are merely examples, and do not limit the present invention. Further, among the constituent elements in the following embodiments, constituent elements that are not described in the independent claims indicating the most significant concept will be described as arbitrary constituent elements.

(Embodiment)
In this embodiment, an in-vehicle device, an information processing method, and the like that can appropriately determine a user's condition will be described.

FIG. 1 is an explanatory diagram schematically showing the configuration of a vehicle V and an on-vehicle device 1 in this embodiment. FIG. 2 is a block diagram showing the functional configuration of the in-vehicle device 1 in this embodiment.

The vehicle V is, for example, a car, and a person can ride therein. A person riding in vehicle V is called a user U. The user U is, for example, a driver seated in the driver's seat of the vehicle V, and this case will be described as an example, but the user U is not limited to this, and may be a passenger sitting in the front passenger seat or the rear seat.

Vehicle V is driven by user U and travels on a road surface. The vehicle V has a cabin S that is a space for passengers to ride. A rearview mirror, a seat for a passenger, and the like are arranged in the vehicle interior S.

The in-vehicle device 1 is a device that determines the state of the user U. For example, the in-vehicle device 1 determines whether the user U is in a state suitable for driving the vehicle V or not. A state suitable for driving includes, for example, not feeling sleepy, not distracted, or not emotional.

As shown in FIGS. 1 and 2, the in-vehicle device 1 includes a camera 5, a processing section 10, and a presentation section 20.

The camera 5 is a photographing device installed in the cabin S of the vehicle V. The camera 5 generates an image of the user U inside the vehicle interior S. The position and posture where the camera 5 is installed will be explained in detail later.

The processing unit 10 is a functional unit that performs processing to determine the state of the user U. The processing unit 10 determines the state of the user U using information acquired from the user U shown in the image by processing the image generated by the camera 5, and outputs the determination result. Details of the processing executed by the processing unit 10 will be explained in detail later. The processing unit 10 can be realized by a processor (for example, a CPU (Central Processing Unit)) (not shown) included in the in-vehicle device 1 executing a predetermined program using a memory (not shown). The processing unit 10 is arranged, for example, in the dashboard of the vehicle V, but the arrangement position is not limited thereto.

The processing unit 10 includes a line of sight detection unit 11, a determination unit 12, a state estimation unit 13, and a correction unit 14 as functional units.

The line of sight detection unit 11 detects the line of sight of the user U shown in the image generated by the camera 5. Detection of the user's U's line of sight may be performed using a well-known line of sight detection technique for the image. The line of sight detection unit 11 also acquires the position and posture of the user's U head. The position and posture of the head of the user U are obtained from the position and posture of the camera 5 in the vehicle V and the position and posture of the head of the user U shown in the image, and the position of the user U relative to the camera 5. and posture. The line of sight detection unit 11 uses the acquired position and posture of the head of the user U and the detected line of sight of the user U to generate information indicating the position where the user U is directing his/her line of sight (also referred to as line of sight information). and provides it to the determination unit 12. The line of sight detection section 11 is also referred to as a detection section.

The line of sight detection unit 11 may further detect the line of sight of the user U in each of the plurality of images generated by the camera 5, and obtain temporal changes in the line of sight of the user U during the period in which the plurality of images are generated. .

The determining unit 12 determines whether the user's U's line of sight is directed toward a predetermined area. The predetermined area is an area including at least a rearview mirror M included in the vehicle V. Specifically, the determination unit 12 acquires line-of-sight information from the line-of-sight detection unit 11 and determines whether the position indicated by the acquired line-of-sight information is included in a predetermined area. The determination unit 12 provides the determination result to the correction unit 14.

Note that the predetermined area may be a predetermined area where the room mirror M exists, or may be adjusted based on the area where the user U's line of sight is actually directed. An example of the method of adjustment will be explained in the modified example described below.

Note that the determination unit 12 further determines that the line of sight of the user U is detected in a predetermined area at appropriate time intervals based on the temporal change in the line of sight of the user U detected by the line of sight detection unit 11 using a plurality of images generated by the camera 5. It may also be determined whether or not the camera is directed toward the camera.

The state estimation unit 13 estimates the state of the user U from the image generated by the camera 5. The state estimation unit 13 generates state information indicating the estimated state of the user U and provides it to the correction unit 14. The state of user U is, for example, the degree of sleepiness, and this case will be explained as an example, but it may also be the degree of fatigue, emotion, presence or absence of sudden illness, degree of absentmindedness, or degree of inattentiveness. The state estimation section 13 is also referred to as an estimation section.

The correction unit 14 corrects the state information generated by the state estimation unit 13 using the result of the determination by the determination unit 12. The correction unit 14 provides the presentation unit 20 with the corrected state information. Specifically, when the determination unit 12 determines that the position indicated by the line-of-sight information (that is, the position where the user U is directing his or her line of sight) is included in the predetermined area, the state estimation unit 13 The generated state information is corrected to indicate that the state of the user U is suitable for driving the vehicle V. In addition, when the determination unit 12 determines that the position indicated by the line-of-sight information is not included in the predetermined area, the correction unit 14 adjusts the state information generated by the state estimation unit 13 so that the state of the user U is not included in the driving of the vehicle V. Correct to indicate that the condition is unsuitable for

For example, when the state of user U is the degree of sleepiness of user U and is expressed as a numerical value from 1 (not sleepy at all, awake) to 6 (sleep), the state estimation unit 13 determines the degree of sleepiness as 3. It may be estimated. The determination unit 12 may determine that the position indicated by the line-of-sight information is not included in the predetermined area. In that case, the correction unit 14 corrects the degree of sleepiness estimated by the state estimation unit 13 to a value larger than 3 (for example, 4).

The presentation unit 20 presents status information provided from the processing unit 10 (more specifically, the correction unit 14). It is assumed that the presented result information is visually recognized by the user U. The presentation unit 20 presents the result information, for example, by displaying an image on a display screen of an infotainment system (including a car navigation system) installed in the vehicle V. Note that the presentation unit 20 may present the result information by outputting sound from an on-vehicle speaker of the vehicle V via the infotainment system.

In addition, when the room mirror M is an electronic mirror provided with a display screen, the presentation part 20 may be the display screen of the room mirror M.

Hereinafter, the position and posture where the camera 5 is installed will be explained. The camera 5 is installed, for example, on the ceiling, and this case will mainly be described, but it may also be installed on a room mirror.

3 and 4 are explanatory diagrams showing examples of the position where the camera 5 is installed and the photographing direction in the embodiment. 3 shows an example of the position where the camera 5 is installed and the photographing direction when the vehicle V is viewed from the side, and FIG. 4 is an example of the position where the camera 5 is installed and the photographing direction when the vehicle V is viewed from the top. Give an example.

As shown in FIG. 3, the camera 5 is installed, for example, on the ceiling in the vehicle interior S, and is installed in such a manner that the photographing direction 6 of the camera 5 faces downward from the horizontal. Note that the vicinity of the ceiling is an area defined as an area relatively close to the ceiling in the vehicle interior S, for example, from the ceiling to the height where the user U's head is located in the vertical direction (height direction). It can be an area. More specifically, the vicinity of the ceiling is, for example, an area within the vehicle interior S where the distance from the ceiling is less than or equal to the distance H2. The distance H2 can be approximately 1/5 to 1/10 of the distance H1 in the vertical direction (that is, the height direction) of the vehicle compartment S.

Further, as shown in FIG. 3, the camera 5 is installed at a position closer to the traveling direction of the vehicle V than the position where the user U gets in, and the photographing direction 6 of the camera 5 is the same as the traveling direction of the vehicle V. They are installed in almost opposite positions. The position where the camera 5 is installed is, for example, an area within the vehicle interior S where the distance from the front end of the vehicle interior S is less than or equal to the distance L2. The distance L2 can be approximately 1/2 to 1/3 of the horizontal distance L1 of the vehicle interior S.

Further, as shown in FIGS. 3 and 4, the camera 5 is installed in a position and posture that allows the user U to fall within the shooting angle of view 7. The camera is installed in a position and posture that allows the user's U's eyes (eyes) to fall within the photographing angle of view 7.

Note that the camera 5 may be installed by being embedded in the ceiling, or may be installed by being attached to the ceiling.

The position of the camera 5 can also be said to be near the rearview mirror M. The vicinity of the rearview mirror M is an area defined as a relatively close area to the rearview mirror M in the vehicle interior S, for example, an area within about 30 cm from the rearview mirror M.

Note that the camera 5 may be installed on the rearview mirror M.

Generally, the rearview mirror M is a target that a user U driving a vehicle V looks at in order to check the rear of the vehicle V. When the camera 5 is installed on the rear-view mirror M, when determining the state of the user U based on the image captured by the camera 5 (more specifically, detecting the line of sight of the user U), the user U is placed on the rear-view mirror M. This has the effect of making it possible to more accurately determine whether or not the user is directing his/her line of sight. This has the effect that the determination unit 12 can more accurately determine whether the user U is looking at the rearview mirror M while driving.

Furthermore, when the camera 5 is installed on the room mirror M, it is installed closer to the user's U's head than when it is installed on the ceiling, so the user's U's face or expression can be seen more clearly. It contributes to the generation of photographed images and has the effect of improving the accuracy of processing based on the images.

Next, the image taken by the camera 5 will be explained.

FIG. 5 is an explanatory diagram showing an example of an image in which user U is shown in this embodiment.

The image shown in FIG. 5 shows the user U photographed by the camera 5. The user U shown in the image is looking at the room mirror M with his/her line of sight slightly upward and slightly to the left from the front of the user U.

The image shown in FIG. 5 is a target for line-of-sight detection by the line-of-sight detection unit 11 and a target for state estimation by the state estimation unit 13.

Hereinafter, the line of sight of the user U will be described with reference to FIGS. 3 and 4 again.

FIG. 3 shows the line of sight of the user U when the vehicle V is viewed from the side. The line of sight shown in FIG. 3 is indicated by the line of sight information generated by the line of sight detection unit 11 based on the image generated by the camera 5.

Specifically, FIG. 3 shows the line-of-sight direction 8 of the user U when viewing the rearview mirror M. The angle (that is, the elevation angle) between the line-of-sight direction 8 and the horizontal direction is assumed to be θ. The area where the room mirror M is located corresponds to the predetermined area.

The determination unit 12 determines that the user's U's line of sight is directed toward a predetermined area when the rearview mirror M is located ahead in the line-of-sight direction 8 in a side view. In other words, the determining unit 12 determines that the user's U's line of sight is directed toward the predetermined area when the line-of-sight direction 8 is included in the angle range 9A from which the user U looks into the room mirror M.

FIG. 4 shows the line of sight of the user U when the vehicle V is viewed from above. The line of sight shown in FIG. 4 is indicated by the line of sight information generated by the line of sight detection unit 11 based on the image generated by the camera 5.

Specifically, FIG. 4 shows the line-of-sight direction 8 of the user U when viewing the rearview mirror M. The angle (that is, the azimuth angle) formed by the line-of-sight direction 8 and the direction of travel is assumed to be φ. The area where the room mirror M is located corresponds to the predetermined area.

The determining unit 12 determines that the user's U's line of sight is directed toward a predetermined area when the room mirror M is located ahead in the line-of-sight direction 8 when viewed from above. In other words, the determining unit 12 determines that the user's U's line of sight is directed toward the predetermined area when the line-of-sight direction 8 is included in the angle range 9B from which the user U looks into the room mirror M.

FIG. 6 is an explanatory diagram showing an example of a temporal change in the line-of-sight direction of the user U in this embodiment.

Specifically, (a) in FIG. 6 shows the temporal change in the elevation angle θ of the user U's line of sight, and (b) in FIG. 6 shows the temporal change in the azimuth angle φ in the user U's line of sight. It shows. In FIG. 6(a), the angle range 9A is shown as a range from angle A1 to angle A2. Further, in FIG. 6(b), an angular range 9B is shown as a range from angle A3 to angle A4.

As shown in FIG. 6(a), the elevation angle θ belongs to the angle range 9A at times T1, T2, and T3. Furthermore, as shown in FIG. 6(b), the azimuth angle φ belongs to the angular range 9B at times T1, T2, and T3.

The determination unit 12 determines that the elevation angle θ belongs to the angle range 9A and that the azimuth angle φ belongs to the angle range 9B at times T1, T2, and T3. It can be determined that the user's U's line of sight is directed toward the rearview mirror M at .

Furthermore, the determination unit 12 can obtain the time interval at which the user's U's line of sight is directed toward the rearview mirror M from the temporal changes in the elevation angle θ and the azimuth angle φ. Specifically, the determination unit 12 obtains the time difference between times T1 and T2 and the time difference between times T2 and T3 as the time interval at which the user's U's line of sight is directed toward the rearview mirror M. Then, the determining unit 12 can determine whether the acquired time interval is included in an appropriate time interval range (for example, from several seconds to several tens of seconds). When the determination unit 12 determines that the time difference is within a proper time interval range, the correction unit 14 corrects the state information in a direction that indicates that the user U is in a state suitable for driving. good. Further, when the determination unit 12 determines that the time difference is not included in the appropriate time interval range, the correction unit 14 corrects the state information in a direction indicating that the user U is not in a state suitable for driving. It's fine.

FIG. 7 is a flow diagram illustrating an example of processing of the in-vehicle device 1 according to the present embodiment.

In step S101, the camera 5 generates an image of the user U inside the cabin S of the vehicle V.

In step S102, the line-of-sight detection unit 11 detects the line-of-sight of the user U through a line-of-sight detection process based on the image generated in step S101.

In step S103, the determination unit 12 determines whether the line of sight detected in step S102 is directed toward a predetermined area.

In step S104, the state estimation unit 13 estimates the state of the user U based on the image generated in step S101, and generates state information.

In step S105, the correction unit 14 corrects the state information generated in step S104 using the result of the determination in step S103.

In step S106, the presentation unit 20 presents the state information corrected in step S105.

Through the series of processes shown in FIG. 7, the in-vehicle device 1 can appropriately determine the state of the user U.

(Modified example of embodiment)
FIG. 8 is a block diagram showing the functional configuration of the in-vehicle device 1A in a modification of the present embodiment.

As shown in FIG. 8, the in-vehicle device 1A includes a camera 5, a processing section 10A, and a presentation section 20. The camera 5 and presentation unit 20 are the same as those in the above embodiment. Hereinafter, the processing unit 10A will be explained in detail.

As shown in FIG. 8, the processing section 10A includes a line of sight detection section 11, a determination section 12, a state estimation section 13, a correction section 14, and an adjustment section 15. The line of sight detection section 11, determination section 12, state estimation section 13, and correction section 14 are the same as those in the above embodiment.

The adjustment unit 15 is a functional unit that adjusts a predetermined area that the determination unit 12 uses for determination.

For example, when the presentation unit 20 is a rearview mirror M that is an electronic mirror including a display screen, the adjustment unit 15 can adjust the predetermined area using the timing of information display by the rearview mirror M. . Specifically, the adjustment unit 15 determines the direction of the user U's line of sight obtained by the line of sight detection unit 11 from the image generated by the camera 5 when the presentation unit 20 presents the information on the display screen. Adjust the predetermined area to include the area. When the adjustment unit 15 adjusts the predetermined area, the determination unit 12 makes a determination using the adjusted predetermined area. Note that the functional unit that executes the above function is also referred to as a first adjustment unit.

Further, as an example, when the vehicle V is equipped with a sensor that detects that another vehicle is approaching behind the vehicle V, the adjustment unit 15 adjusts the timing when another vehicle is approaching. can be used to adjust the predetermined area. Specifically, the adjustment unit 15 uses the line-of-sight detection unit 11 to calculate the amount of data obtained by the line-of-sight detection unit 11 from the image generated by the camera 5 at the timing when the sensor detects that another vehicle is approaching behind the vehicle V. The predetermined area is adjusted so as to include the area where the user U's line of sight is directed. Note that the functional unit that executes the above function is also referred to as a second adjustment unit. The sensor is, for example, an ultrasonic sensor, a LiDAR (light detection and ranging), or a rear camera. Note that the sensor may receive information regarding a vehicle approaching vehicle V via a network connecting the vehicles.

Note that the adjustment unit 15 specifies a period during which the sensor is sensing that another vehicle is approaching behind the vehicle V, and adjusts the plurality of images generated by the plurality of images taken by the camera 5 within the specified period. The predetermined area may be adjusted using the image.

As described above, the in-vehicle device of this embodiment appropriately determines whether or not the user is directing his or her gaze toward the rearview mirror, and uses the determination result to correct the estimated state of the user. , the user's condition can be determined more appropriately. In particular, since a camera installed near the ceiling inside the vehicle is used, it is possible to more appropriately determine whether the user is looking at the rearview mirror, and the final determination result of the user's condition is improved. It has the effect of making it appropriate. In this way, the in-vehicle device can appropriately determine the user's condition.

In addition, the in-vehicle device further determines whether the time interval between when the user looks at the rearview mirror is appropriate, corrects the user's estimated state, and determines the user's state more appropriately. can do. Therefore, the in-vehicle device can more appropriately determine the user's condition.

Further, the in-vehicle device adjusts the predetermined area and then makes a determination using the adjusted predetermined area. It is assumed that the user turns his/her line of sight to the electronic mirror at the timing when the electronic mirror presents information, so by using the user's line of sight at that time, a predetermined area is set according to the user's physique or posture. This allows the estimated state of the user to be corrected more appropriately. Therefore, the in-vehicle device can more appropriately determine the user's condition.

Further, the in-vehicle device adjusts the predetermined area and then makes a determination using the adjusted predetermined area. It is assumed that the user turns his/her line of sight to the electronic mirror at the timing when the sensor detects that another vehicle is approaching behind the vehicle, so by using the user's line of sight at that time, the user's A predetermined area can be set according to the physique or posture, and thereby the estimated state of the user can be corrected more appropriately. Therefore, the in-vehicle device can more appropriately determine the user's condition.

In addition, since the in-vehicle device uses multiple images taken during the period when another vehicle is actually close to the rear of the vehicle, the user who recognizes that the vehicle is close to the rear looks in the rearview mirror. Adjustment of a predetermined area can be made more appropriately by using the action of checking the rear of the vehicle by pointing at the vehicle. Therefore, the in-vehicle device can more appropriately determine the user's condition.

Note that in the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. Here, the software that implements the server etc. of the above embodiment is the following program.

That is, this program causes a computer to generate an image of a user in the vehicle interior taken by a camera installed near the ceiling of the vehicle interior, and to display the image captured by the camera. detects the user's line of sight, determines whether the detected user's line of sight is directed toward a predetermined area including a rearview mirror provided in the vehicle, and detects the user's line of sight based on the image generated by the camera. executing an information processing method of estimating a state of, generating state information indicating the estimated state, correcting the generated state information using the result of the determination, and presenting the corrected state information. This is a program that allows you to

Although the in-vehicle devices and the like according to one or more aspects have been described above based on the embodiments, the present invention is not limited to these embodiments. Unless departing from the spirit of the present invention, various modifications that can be thought of by those skilled in the art to this embodiment, and embodiments constructed by combining components of different embodiments are within the scope of one or more embodiments. may be included within.

INDUSTRIAL APPLICATION This invention can be utilized for the vehicle-mounted apparatus which determines the state of the user in a vehicle, such as a driver of a car.

1, 1A Vehicle-mounted device 5 Camera 6 Photographing direction 7 Photographing angle of view 8 Direction of line of sight 9A, 9B Angle range 10, 10A Processing section 11 Line of sight detection section 12 Judgment section 13 State estimation section 14 Correction section 15 Adjustment section 20 Presentation section M Rearview mirror S Vehicle interior U User V Vehicle

Claims (7)

A camera installed near the ceiling of a vehicle interior, the camera generating an image of a user in the vehicle interior;
a detection unit that detects the user's line of sight reflected in the image generated by the camera;
a determination unit that determines whether the line of sight of the user detected by the detection unit is directed toward a predetermined area including a rearview mirror provided in the vehicle;
an estimation unit that estimates a state of the user from an image generated by the camera and generates state information indicating the estimated state;
a correction unit that corrects the state information generated by the estimation unit using the determination result by the determination unit;
An in-vehicle device comprising: a presentation section that presents the state information corrected by the correction section. The camera generates a plurality of images by photographing the user a plurality of times,
The determination unit includes:
In the determination, further, based on the temporal change in the user's line of sight detected by the detection unit using a plurality of the images, whether or not the user's line of sight is directed to the predetermined area at appropriate time intervals. The in-vehicle device according to claim 1. The presentation unit includes the room mirror that is an electronic mirror including a display screen,
The in-vehicle device further includes:
The predetermined area is configured such that the predetermined area includes an area where the user's line of sight obtained by the detection unit is directed from the image generated by photography by the camera at the timing when the presentation unit presents information on the display screen. Equipped with a first adjustment section that adjusts the
The determination unit includes:
The in-vehicle device according to claim 1 or 2, wherein the determination is made using the predetermined area adjusted by the first adjustment section. The vehicle includes a sensor that detects that another vehicle is close to the rear of the vehicle,
The in-vehicle device further includes:
The user's line of sight obtained by the detection unit is determined from the image generated by the camera at a timing when the sensor senses that another vehicle is approaching behind the vehicle. a second adjustment unit that adjusts the predetermined area to include the area where the predetermined area is located;
The determination unit includes:
The vehicle-mounted device according to any one of claims 1 to 3, wherein the determination is made using the predetermined area adjusted by the second adjustment section. The second adjustment section is
A period during which the sensor detects that another vehicle is approaching behind the vehicle is specified, and a plurality of images generated by the camera multiple times during the specified period are used. The in-vehicle device according to claim 4, wherein the predetermined area is adjusted by adjusting the predetermined area. Generating an image of a user in the vehicle interior by a camera installed near the ceiling of the vehicle interior;
detecting the user's line of sight in the image generated by the camera;
determining whether the detected line of sight of the user is directed toward a predetermined area including a rearview mirror provided in the vehicle;
estimating the state of the user from an image generated by the camera and generating state information indicating the estimated state;
correcting the generated state information using the result of the determination,
An information processing method that presents the state information after being corrected. A program that causes a computer to execute the information processing method according to claim 6.

Images