JP2020113066A - Video processing device, video processing method, and program - Google Patents

Abstract

To determine what a driver was looking at in the event of a vehicle accident.SOLUTION: An image processing device 100 includes: a video data acquisition unit 111 that acquires video data from an imaging unit that images the surroundings of a vehicle; a gaze estimation unit 112 that estimates a gaze direction of a driver driving the vehicle on the basis of a position and orientation of a head of the driver; a gazing point deriving unit 113 that derives a gazing point of the driver based on the driver's gaze direction; a video processing unit 114 that cuts out a gaze video data based on the gazing point from the video data; and an output control unit 115 that outputs gaze video data.SELECTED DRAWING: Figure 1

Description

The present invention relates to a video processing device, a video processing method, and a program.

There is known a technique for grasping what kind of driving was performed by the driver who caused the accident when a vehicle accident or the like occurred.

For example, in Patent Document 1, there is a technique that can grasp the driver's line-of-sight direction by superimposing and displaying a marker indicating the line-of-sight direction of the driver who is driving the vehicle on a peripheral image of the vehicle. It is disclosed.

JP, 2007-72567, A

In Patent Document 1, the driver's line-of-sight direction in the peripheral image can be specified, but it may not be possible to specify what the driver is actually looking at. Therefore, there is a demand for a technique that enables a driver to determine what he/she was looking at when a vehicle accident or the like occurs.

It is an object of the present invention to provide a video processing device, a video processing method, and a program capable of determining what a driver is looking at when a vehicle accident or the like occurs.

The video processing device of the first aspect of the present invention is based on a video data acquisition unit that acquires video data from an imaging unit that images the surroundings of the vehicle, and the position and orientation of the head of the driver who drives the vehicle, A gaze estimation unit that estimates the driver's gaze direction, a gaze point deriving unit that derives a gaze point of the driver based on the gaze direction of the driver, and gaze image data based on the gaze point Is provided from the video data, and an output control unit that outputs the gaze video data.

A video processing method according to a second aspect of the present invention is based on the step of acquiring video data from an imaging unit for imaging the surroundings of a vehicle and the position and orientation of the head of a driver who drives the vehicle, based on the driver. Estimating a line-of-sight direction of the driver, deriving a gaze point of the driver based on the line-of-sight direction of the driver, and cutting out gaze image data based on the gaze point from the image data And a step of outputting the gaze image data.

A program according to a third aspect of the present invention is based on a step of acquiring video data from an image capturing unit that captures an image of the surroundings of a vehicle, and the line of sight of the driver based on the position and orientation of the head of the driver who drives the vehicle. A step of estimating a direction, a step of deriving a gazing point of the driver based on the line-of-sight direction of the driver, a step of cutting out gaze video data based on the gazing point from the video data, The step of outputting the gaze image data is executed by a computer operating as an image processing device.

According to the present invention, it is possible to determine what the driver was looking at when a vehicle accident or the like occurred.

FIG. 1 is a block diagram showing the configuration of a video processing system according to the first embodiment of the present invention. FIG. 2 is a schematic diagram showing the configuration of the video processing system according to the first embodiment of the present invention. FIG. 3 is a schematic diagram for explaining a range in which the periphery of the vehicle is imaged. FIG. 4 is a diagram for explaining a method of deriving position data of a gazing point in peripheral video data and cutting out the gazing video data with the derived gazing point as a reference. FIG. 5 is a diagram for explaining an example of a method of enlarging the gaze image data. FIG. 6 is a diagram showing a state in which the cut out gaze image data is enlarged. FIG. 7 is a diagram for explaining an example of a method of displaying the trajectory of the line of sight of the driver. FIG. 8 is a diagram for explaining an example of a method of displaying the range of the trajectory of the line of sight of the driver. FIG. 9 is a flowchart showing an example of the flow of processing of the video processing system according to the first embodiment of the present invention. FIG. 10 is a flowchart showing an example of the processing flow of the video processing system according to the first embodiment of the present invention. FIG. 11 is a block diagram showing the configuration of the video processing system according to the first modified example of the first embodiment of the present invention. FIG. 12 is a block diagram showing the configuration of the video processing system according to the second modification of the first embodiment of the present invention. FIG. 13 is a diagram for explaining a method of changing the cut-out range of the gaze video data. FIG. 14 is a flowchart showing an example of the processing flow of the video processing system according to the second modification of the first embodiment of the present invention. FIG. 15 is a block diagram showing the configuration of the video processing system according to the second embodiment of the present invention. FIG. 16 is a flowchart showing an example of the processing flow of the video processing system according to the second embodiment of the present invention. FIG. 17 is a block diagram showing the configuration of the video processing system according to the third embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the drawings, the same or corresponding parts are designated by the same reference numerals, and description thereof will be omitted as appropriate. In addition, constituent elements in the following embodiments or examples include elements that can be easily replaced by those skilled in the art, or substantially the same elements. Furthermore, the constituent elements described below can be combined as appropriate, and when there are a plurality of embodiments or examples, it is also possible to combine the respective embodiments or examples.

FIG. 1 is a block diagram showing the configuration of a video processing system according to the first embodiment of the present invention. As shown in FIG. 1, the video processing system 1 includes an imaging unit 10, a display unit 20, and a video processing device 100. As shown in FIG. 2, the video processing device 100 is mounted on the vehicle C. FIG. 2 shows the left side mirror 2, the right side mirror 3, the room mirror 4, and the steering wheel 5. The image processing system 1 is, for example, a drive recorder mounted on the vehicle C, and displays both an image around the vehicle C and an image obtained by cutting out what the driver sees from the image around the vehicle C. Displayed on or recorded in a storage medium.

The imaging unit 10 includes a peripheral imaging unit 11 and a driver imaging unit 12. In FIG. 2, the imaging unit 10 is arranged above the room mirror 4, but this is a schematic example and the position where the imaging unit 10 is arranged is not limited.

The peripheral imaging unit 11 captures an image around the vehicle C. The peripheral imaging unit 11 is, for example, a wide-angle camera such as a spherical camera. The peripheral imaging unit 11 includes, for example, a plurality of cameras including a front camera that captures the front of the vehicle C, a rear camera that captures the rear, a left camera that captures the left side, and a right camera that captures the right side. It may be composed of. The peripheral imaging unit 11 may be a camera that drives according to the direction of the line of sight of the driver. In this case, the peripheral image capturing unit 11 normally captures the traveling direction of the vehicle C, and drives by following the driver's head imaged by the driver image capturing unit 12 described later and the direction of the line of sight to drive the vehicle. The configuration may be such that the direction of the line of sight of the person is imaged.

FIG. 3 is a diagram for explaining the peripheral video data 300 imaged by the peripheral imaging unit 11. As shown in FIG. 3, the peripheral imaging unit 11 images the surroundings of the vehicle C including the front, the rear, the left side, and the right side. The peripheral imaging unit 11 outputs the imaged front as the front video data 210 to the video data acquisition unit 111. The peripheral imaging unit 11 outputs the captured rearward as rear image data 220 to the image data acquisition unit 111. The peripheral imaging unit 11 outputs the captured left side as the left side video data 230 to the video data acquisition unit 111. The peripheral imaging unit 11 outputs the captured right side as the right side video data 240 to the video data acquisition unit 111. When the omnidirectional camera captures a omnidirectional image, the peripheral imaging unit 11 outputs the image to the image data acquisition unit 111 as omnidirectional image data.

Referring back to FIG. The driver imaging unit 12 images the driver's head so that at least the driver's face orientation can be determined. The driver image capturing unit 12 preferably captures the driver's eyeball when the driver's eyeball can be captured. The driver imaging unit 12 is provided, for example, near the rearview mirror 4 or in front of the steering wheel 5. Note that the driver imaging unit 12 is, for example, a omnidirectional camera in which the peripheral imaging unit 11 is provided inside the vehicle C, and may not be provided if the peripheral imaging unit 11 can image the driver. Good.

The display unit 20 is a display including, for example, a liquid crystal display (LCD: Liquid Crystal Display) or an organic EL (Organic Electro-Luminescence) display. The display unit 20 displays the image processed by the image processing device 100. Specifically, the display unit 20 may display the peripheral image data 300, or at least the front image data 210.

The video processing device 100 includes a control unit 110 and a storage unit 130.

The control unit 110 controls each unit included in the video processing system 1. Specifically, the control unit 110 controls each unit configuring the video processing system 1 by expanding and executing the program stored in the storage unit 130. The control unit 110 can be realized by, for example, an electronic circuit including a CPU (Central Processing Unit). The control unit 110 includes a video data acquisition unit 111, a line-of-sight estimation unit 112, a gazing point derivation unit 113, a video processing unit 114, and an output control unit 115.

The video data acquisition unit 111 acquires the peripheral video data 300 of the vehicle C captured by the peripheral imaging unit 11. The video data acquisition unit 111 acquires the head video data of the driver captured by the driver imaging unit 12. The video data acquisition unit 111 outputs the peripheral video data 300 to the gazing point derivation unit 113 and outputs the head video data to the gaze estimation unit 112. The video data acquisition unit 111 outputs the peripheral video data 300 to, for example, the storage unit 130 or an external storage device. That is, in the present embodiment, both the peripheral image data 300 and the image in the direction of the line of sight of the driver are stored or output.

The line-of-sight estimating unit 112 estimates the direction of the line of sight of the driver based on the head image data of the driver. Specifically, the line-of-sight estimation unit 112 estimates the direction of the driver's line of sight based on the orientation of the driver's face. The gaze estimation unit 112 outputs the estimation result of the driver's gaze direction to the gazing point derivation unit 113. The gaze estimation unit 112 stores the estimation result in the storage unit 130 as a history.

The gazing point deriving unit 113 derives position data of the gazing point in the peripheral image data 300 based on the peripheral image data 300 and the estimation result of the direction of the line of sight of the driver. Specifically, the position data of the gazing point means an intersection between the direction vector of the line of sight of the driver and the peripheral image data 300.

FIG. 4 is a diagram for explaining a method of deriving position data of a gazing point in the peripheral video data 300 and cutting out the gazing video data based on the derived gazing point. In FIG. 4, the front image data 210 is shown as an example of the peripheral image data 300. First, the gazing point deriving unit 113 derives the line-of-sight vector V based on the estimation result of the line-of-sight of the driver. Then, the gazing point deriving unit 113 derives the position data of the gazing point P at the intersection of the line-of-sight vector V and the front video data 210, for example. The gazing point deriving unit 113 outputs the derived position data of the gazing point P to the video processing unit 114. Hereinafter, the front video data 210 will be described as an example, but the same applies to the rear video data 220, the left video data 230, and the right video data 240.

The video processing unit 114 cuts out the gazing video data in a predetermined range with the position of the gazing point P as a reference (center) from the front video data 210. As shown in FIG. 4, the video processing unit 114 cuts out the gazing video data in the range X from the front video data 210 with the gazing point P as a reference. Here, the image processing unit 114 cuts out, for example, a range in which a person's attention is exerted, that is, a range in which a person can recognize an object to be photographed from the image with the gazing point P as a reference. The image processing unit 114 may change the cut-out range X according to the direction of the line of sight of the driver. In this case, the image processing unit 114 may be preset with a range to be cut out when the left and right directions are visually observed. For example, the image processing unit 114 may narrow the range of the gaze image data cut out from the left side image data 230 and the right side image data 240 to be narrower than the range of the gaze image data cut out from the front image data 210.

The image processing unit 114 may change the cut-out range for each driver, for example. In this case, for example, the cut-out range for each driver may be stored in the storage unit 130 in advance. The image processing unit 114, for example, enlarges and displays the periphery of the derived gazing point P or changes the cut-out range when the movement of the line of sight of the driver is different from the normal movement. May be. In this case, for example, the movement of the driver's normal line of sight may be stored in the storage unit 130. The movement that differs from the movement of the line of sight during normal operation means movement that the driver's line of sight moves outside the average range of movement, movement outside the range of average speed at which the line of sight moves, and average time during which the line of sight remains For example, the movement is longer than the time.

The image processing unit 114 may change the cut-out position each time the position of the gazing point P changes, or after the gazing point P is detected at the first position, a predetermined distance from the first position. The position to be cut out may be changed only when the gazing point P is detected at a position distant from the above. In this case, the image processing unit 114 changes the position to be cut out when the gazing point P is detected at a position closer than 50% of the distance from the first position to the outer edge of the range X, for example. Further, the image processing unit 114 may change the position to be cut out when the gazing point P is detected at a position closer to the outer edge of the range X by 20% or more of the distance from the first position, or 80% or more. The position to be cut out when the gazing point P is detected at the approaching position may be changed. That is, the predetermined distance from the first position may be set to a value desired by the user.

A method of enlarging the gaze image data will be described with reference to FIG. FIG. 5 is a diagram for explaining an example of a method of enlarging the gaze image data.

For example, when the driver is gazing at a specific place for a predetermined time (for example, 5 seconds), the image processing unit 114 may enlarge the place. In the example shown in FIG. 5, it is assumed that the driver is gazing at the vehicle C2 for a predetermined time. In this case, the video processing unit 114 enlarges and displays the predetermined range based on the gazing point P. Here, it is preferable that the range Xa expanded by the image processing unit 114 be smaller than the normal cutout range X shown in FIG.

FIG. 6 is a diagram showing a state in which the cut out gaze image data is enlarged. As illustrated in FIG. 6, the video processing unit 114 may enlarge the cut gaze video data. In this case, the image processing unit 114 may extract the objects included in the range Xa by subjecting the enlarged gaze image data to object recognition processing, for example.

A method of displaying the trajectory of the line of sight of the driver will be described with reference to FIG. 7. FIG. 7 is a diagram for explaining an example of a method of displaying the trajectory of the line of sight of the driver.

As shown in FIG. 7, for example, the image processing unit 114 may superimpose the trajectory of the line of sight of the driver in the cut out range X on the front image data 210 and display it. In the example shown in FIG. 7, the trajectory T of the line of sight of the driver is shown in the front video data 210. On the trajectory T of the driver's line of sight, the points that have been gazing for a certain time, that is, the first gazing point P1, the second gazing point P2, the third gazing point P3, the fourth gazing point P4, and the fifth gazing point P4. The gazing point P5, the sixth gazing point P6, the seventh gazing point P7, the eighth gazing point P8, and the ninth gazing point P9 are indicated by white circles. In this case, it means that the driver sequentially moved the line of sight from the first gazing point P1 to the ninth gazing point P9. In this case, the center of the range X is the ninth gazing point P9. Here, for example, the sizes of the first gazing point P1 to the ninth gazing point P9 may be changed according to the time the driver is gazing. In the example shown in FIG. 7, the first gazing point P1 is the largest, the sixth gazing point P6 and the ninth gazing point P9 are next, the second gazing point P2, the third gazing point P3, and the fourth gazing point P3. The gazing point P4, the fifth gazing point P5, the seventh gazing point P7, and the eighth gazing point P8 are the smallest. In this case, the driver has watched the first gazing point P1 for the longest time, then has the sixth gazing point P6 and the ninth gazing point P9 for a long time, and the second gazing point. P2, the third gazing point P3, the fourth gazing point P4, the fifth gazing point P5, the seventh gazing point P7, and the eighth gazing point P8 mean that the time is the shortest. ing. The image processing unit 114 may always display the trajectory of the line of sight of the driver as described above, or may display it only when a specific motion of the driver is detected. The specific operation of the driver includes, but is not limited to, gazing at a specific point for a predetermined time (for example, 5 seconds), the movement of the line of sight changes more drastically than in normal times, and the like. In addition, the image processing unit 114 may change the time for displaying the trajectory of the line of sight of the driver, depending on the driving conditions of the vehicle. For example, when the image processing unit 114 is traveling in an urban area or a downtown area, the time for displaying the locus may be shortened or lengthened. Further, the image processing unit 114 may lengthen the time for displaying the locus in a situation in which it is necessary to check the surrounding situation more carefully, such as when turning right or left at an intersection.

FIG. 8: is a figure for demonstrating an example different from FIG. 7 of the method of displaying the locus|trajectory of a driver|operator's gaze. As shown in FIG. 8, the image processing unit 114 may display the range in which the driver is viewing in the peripheral image data 300. In FIG. 8, as the driver's range, the first range A1 in the left image data 230, the second range A2 in the front image data 210, and the third range A3 in the right image data 240 are shown. It is shown. In this case, for example, the first range A1 is the range seen by the driver at the time t1, and the second range A2 is the range seen by the driver at the time t2 after the time t1. The third range A3 is the range that the driver is watching at the time t3 after the time t2. That is, the video processing unit 114 visualizes the temporal change in the range that is being viewed in the peripheral video data 300.

The video processing unit 114 may switch the display system of FIG. 6, FIG. 7, and FIG. 8 depending on the situation. Further, the display system of FIGS. 6, 7, and 8 may be freely switched by the user.

The image processing unit 114 may display the trajectory of the line of sight of the driver in the peripheral image data 300 when the vehicle C is traveling in a specific place. Examples of the specific place include, but are not limited to, a place having many blind spots such as an urban area.

The image processor 114 outputs the gaze image data to the output controller 115.

The output control unit 115 outputs the gaze image data. The output control unit 115 outputs the gaze image data to the display unit 20, for example. Thereby, the driver or the like can confirm the gaze image data by visually recognizing the display unit 20. The output control unit 115 may output the gaze image data to the storage unit 130, or may output it to a portable recording medium such as an SD card (not shown), for example. As a result, the driver or the like can confirm the gaze image data at a place other than the vehicle.

The storage unit 130 stores a program for the control unit 110 to control each unit of the video processing system 1. The storage unit 130 may store, for example, a range in which the gaze image data is cut out from the peripheral image data 300 for each driver. The storage unit 130 stores, for example, the tendency of the movement of the line of sight during normal driving for each driver. The storage unit 130 stores, for example, the estimated direction of the line of sight of the driver as a history. The storage unit 130 is, for example, a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory (Flash Memory), or a storage device such as a hard disk, a solid state drive, or an optical disk.

The processing of the control unit 110 of the video processing device 100 will be described with reference to FIG. 9. FIG. 9 is a flowchart showing an example of the processing flow of the control unit 110.

First, the control unit 110 acquires peripheral image data and head image data (step S101). Specifically, the control unit 110 causes the video data acquisition unit 111 to acquire peripheral image data and head video data. Then, the control unit 110 proceeds to step S102.

The control unit 110 estimates the direction of the driver's line of sight based on the head image data (step S102). Specifically, the control unit 110 causes the line-of-sight estimation unit 112 to estimate the direction of the driver's line of sight. Then, the control unit 110 proceeds to step S103.

The control unit 110 derives the position data of the gazing point based on the peripheral image data 300 and the estimation result of the direction of the line of sight of the driver (step S103). Specifically, the control unit 110 causes the gazing point deriving unit 113 to derive the position data of the gazing point. Then, the control unit 110 proceeds to step S104.

The control unit 110 cuts out the gazing video data in a predetermined range based on the gazing point from the peripheral video data 300 (step S104). Specifically, the control unit 110 causes the video processing unit 114 to cut out the gaze video data from the peripheral video data 300. Then, the control unit 110 proceeds to step S105.

The control unit 110 outputs the gaze image data (step S105). Specifically, the control unit 110 causes the output control unit 115 to output the gaze image data. Then, the control unit 110 ends the processing of FIG.

Processing different from that of FIG. 9 of the control unit 110 of the video processing device 100 will be described with reference to FIG. 10. FIG. 10 is a flowchart showing an example of the processing flow of the control unit 110.

Since steps S201 to S204 are the same as steps S101 to S104 shown in FIG. 9, description thereof will be omitted.

The control unit 110 detects a specific motion of the driver (step S205). Specifically, the control unit 110 causes the video processing unit 114 to detect a specific motion of the driver. In step S205, it may be detected whether or not the vehicle is traveling in a specific place. Then, the control unit 110 proceeds to step S206.

When the specific motion of the driver is detected (“Yes” in step S206), the control unit 110 proceeds to step S207 and expands the gaze image data and the gaze image data that displays the trajectory of the line of sight of the driver. Is output (step S207). Specifically, the control unit 110 causes the image processing unit 114 to output at least one of the enlarged gaze image data and the gaze image data displaying the trajectory of the line of sight of the driver. Then, the control unit 110 ends the processing of FIG.

On the other hand, when the specific motion of the driver is not detected (“No” in step S206), the control unit 110 proceeds to step S208 and outputs the gaze image data cut out in step S204 (step S208). Specifically, the control unit 110 causes the video processing unit 114 to output the gaze video data cut out in step S204. Then, the control unit 110 ends the processing of FIG.

As described above, the first embodiment can output, as the gaze image data, the portion of the peripheral image data that is estimated to be gazed by the driver while driving. Thereby, the first embodiment can provide, for example, a drive recorder that records an image in the line-of-sight direction of the driver.

Further, in the first embodiment, since the image of the driver's line-of-sight direction can be recorded, the oversight of the red traffic light, the delay in noticing the pedestrian's jumping out, the side-view driving, and the like can be determined from the image. As a result, the first embodiment can accurately analyze the driving situation of the driver when a car accident occurs, for example.

Furthermore, since the first embodiment can record an image in the direction of the driver's line of sight, it is possible to record, for example, an unusual building, a beautiful landscape, a sign or a vehicle traveling around from the driver's perspective. .. As a result, the first embodiment can record pleasant memories such as a trip from the driver's point of view.

[First Modification of First Embodiment]
FIG. 11 is a block diagram showing a configuration of a video processing system 1A according to a first modified example of the first embodiment of the present invention. As shown in FIG. 11, the video processing system 1A differs from the video processing system 1 in that the control unit 110A of the video processing device 100A includes a line-of-sight detection unit 116.

In the image processing system 1A, the driver image capturing unit 12 captures an image of the eyes of the driver. In this case, the driver imaging unit 12 outputs the state of the eyes of the driver to the video data acquisition unit 111 as eyeball video data.

The video data acquisition unit 111 acquires eyeball video data from the driver imaging unit 12. The video data acquisition unit 111 outputs the eyeball video data to the visual line detection unit 116.

The line-of-sight detection unit 116 detects the direction of the driver's line of sight from the eyeball image data based on, for example, the contour of the eye and the position of the black eye. There is no particular limitation on the method by which the line-of-sight detection unit 116 detects the direction of the line of sight of the driver, and any known technique may be used. For example, the line-of-sight detection unit 116 detects the line of sight of the driver by eye tracking, gaze plot, or the like. The line-of-sight detection unit 116 outputs the detection result to the gazing point derivation unit 113. The line-of-sight detection unit 116 stores the line-of-sight detection result in the storage unit 130 as a history.

The gazing point deriving unit 113 derives position data of the gazing point in the peripheral video data 300 based on the peripheral video data 300 and the detection result of the line of sight of the driver. The gazing point deriving unit 113 outputs the position data of the gazing point to the video processing unit 114. The processes of the video processing unit 114 and the output control unit 115 are the same as those in the first embodiment, and thus the description thereof will be omitted.

As described above, in the first modified example of the first embodiment, the part of the peripheral image data 300 that the driver gazes while driving can be cut out as the gaze image data. Thereby, the first modification of the first embodiment can further improve the accuracy.

[Second Modification of First Embodiment]
FIG. 12 is a block diagram showing the configuration of the video processing system 1B according to the second modified example of the first embodiment. The video processing system 1B includes an illuminance sensor 30 and a rainfall sensor 40, and the control unit 110B of the video processing device 100B includes a vehicle information acquisition unit 117 and a peripheral information acquisition unit 118. It is different from the video processing system 1. The image processing system 1B changes the cut-out range according to the detection results of the illuminance sensor 30 and the rainfall sensor 40 and the vehicle information acquired by the vehicle information acquisition unit 117. Specifically, the image processing system 1B cuts out a range visually recognized by the driver, which changes according to vehicle information and surrounding information.

The illuminance sensor 30 detects the illuminance around a vehicle equipped with the video processing system 1B. The illuminance sensor 30 outputs the detected illuminance to the video processing unit 114.

The rainfall sensor 40 detects the amount of rainfall around the vehicle equipped with the video processing system 1B. The rainfall sensor 40 outputs the detected rainfall amount to the video processing unit 114.

The illuminance sensor 30 and the rainfall sensor 40 are examples of sensors that acquire peripheral information about the vehicle C included in the image processing system 1B, and the types of sensors included in the present embodiment are not limited. The image processing system 1B is not limited to the illuminance sensor 30 and the rainfall sensor 40, and may include a sensor that detects other peripheral information around the vehicle C. In this case, the video processing unit 114 changes the cutout range according to the detected peripheral information.

The vehicle information acquisition unit 117 acquires vehicle information including the speed of the vehicle C from a CAN (Controller Area Network) or various sensors that sense the state of the vehicle C, for example. The vehicle information acquisition unit 117 acquires, as the vehicle information, the current position information of the vehicle C acquired using a GPS (Global Positioning System) receiver, for example. The vehicle information acquisition unit 117 outputs the acquired vehicle information to the video processing unit 114.

The peripheral information acquisition unit 118 acquires the detection result of the illuminance around the vehicle C from the illuminance sensor 30. The peripheral information acquisition unit 118 acquires the detection result of the amount of rainfall around the vehicle C from the rainfall sensor 40. When the image processing system 1B includes a sensor that detects other peripheral information, the peripheral information acquisition unit 118 acquires the peripheral information acquired by the sensor.

The image processing unit 114 changes the cutout range of the gaze image data according to the illuminance around the vehicle. For example, the image processing unit 114 widens the cutout range of the gaze image data as the brightness around the vehicle becomes darker. In this case, the video processing unit 114 may linearly change the cutout range of the gaze video data, or may change it stepwise.

The image processing unit 114 changes the cutout range of the gaze image data according to the amount of rainfall around the vehicle. For example, the image processing unit 114 widens the cutout range of the gaze image data as the amount of rainfall around the vehicle increases. In this case, the video processing unit 114 may linearly change the cutout range of the gaze video data, or may change it stepwise.

The video processing unit 114 changes the cutout range of the gaze video data according to the vehicle information. The image processing unit 114 changes the cut-out range of the gaze image data based on the vehicle speed information included in the vehicle information, for example. The image processing unit 114 narrows the cutout range as the vehicle speed increases, for example. In this case, the video processing unit 114 may linearly change the cutout range of the gaze video data, or may change it stepwise.

The image processing unit 114 may change the cutout range according to the position information included in the vehicle information, for example. For example, when it is determined that the current position of the vehicle C is the highway, the image processing unit 114 narrows the cutout range of the gaze image data. For example, when the current position of the vehicle C is a place where there are many things such as a shopping street or the width of the road is narrow, the image processing unit 114 narrows the cutout range. For example, when the road is wide, the image processing unit 114 widens the cutout range.

The image processing unit 114 may change the cutout range based on both the peripheral information and the vehicle information, or may change the cutout range based on at least one of the peripheral information and the vehicle information. Good.

FIG. 13 is a diagram for explaining a method of changing the cut-out range of the gaze video data. As shown in FIG. 13, for example, the video processing unit 114 normally cuts out the gazing video data of the range X1 from the front video data 210 with the gazing point P as a reference. When the image processing unit 114 determines that the cut-out range is narrowed based on the peripheral information and the vehicle information, the image processing unit 114 cuts out the gaze image data in the range X2 from the front image data 210 with the gaze point P as a reference. When the video processing unit 114 determines to widen the cutout range based on the peripheral information and the vehicle information, the video processing unit 114 cuts out the watched video data in the range X3 from the front video data 210 with the gazing point P as a reference. The image processing unit 114 may change the range of the gaze image data and display the trajectory of the line of sight of the driver within the changed range of the gaze image data.

The processing of the control unit 110B of the video processing device 100B will be described with reference to FIG. FIG. 14 is a flowchart showing an example of the processing flow of the control unit 110B.

Since steps S301 to S303 are the same as steps S101 to S103 shown in FIG. 9, description thereof will be omitted.

Control unit 110B acquires at least one of vehicle information and peripheral information (step S304). Specifically, the control unit 110B acquires the vehicle information by the vehicle information acquisition unit 117 and acquires the peripheral information by the peripheral information acquisition unit 118. Then, the control unit 110B proceeds to step S305.

The control unit 110B cuts out the gaze image data in a predetermined range based on the gaze point from the peripheral image data 300 based on at least one of the vehicle information and the peripheral information (step S305). Specifically, the control unit 110B causes the video processing unit 114 to cut out a predetermined range of gaze video data from the peripheral video data 300. Then, the control unit 110B proceeds to step S306.

Since step S306 is similar to step S105 shown in FIG. 9, description thereof will be omitted. Then, the control unit 110B ends the process of FIG.

As described above, in the second modification of the first embodiment, the range in which the gaze video data is cut out from the peripheral video data 300 can be changed based on at least one of the vehicle information and the peripheral information. Thereby, the second modification of the first embodiment can further improve the accuracy. Further, the second modified example of the first embodiment determines the movement of the driver's line of sight in more detail by displaying the trajectory of the driver's line of sight in the gaze image data whose range is changed according to the situation. be able to.

[Second embodiment]
A video processing system 1C according to the second embodiment of the present invention will be described with reference to FIG. FIG. 15 is a block diagram showing the configuration of the video processing system 1C.

As shown in FIG. 15, the video processing system 1C is a video in that the control unit 110C of the video processing device 100C includes a gazing point history management unit 119, a mirror position acquisition unit 120, and a relative position calculation unit 121. It is different from the processing system 1.

The gazing point history management unit 119 acquires, for example, the most frequent gazing point that the driver sees most frequently in the left-right direction, based on the history of the direction of the driver's line of sight stored in the storage unit 130. The gazing point history management unit 119 acquires, for example, a portion frequently viewed by the driver in the vertical direction. The gazing point history management unit 119 outputs the acquisition result to the video processing unit 114.

The mirror position acquisition unit 120 acquires, for example, the positions of the left side mirror 2, the right side mirror 3, and the room mirror 4 in the vehicle C as shown in FIG. 2 from CAN, for example. The mirror position acquisition unit 120 outputs the position information of the left side mirror 2, the right side mirror 3, and the room mirror 4 to the relative position calculation unit 121.

The relative position calculation unit 121 calculates the position of each mirror as a relative position from the driver based on the positional information of the left side mirror 2, the right side mirror 3, and the room mirror 4. Here, for example, the image data acquisition unit 111 outputs the driver's head image data to the relative position calculation unit 121. In this case, the relative position calculation unit 121 determines each position based on the position information of the left side mirror 2, the right side mirror 3, and the room mirror 4, and the position of the driver's head included in the head image data. The position of the mirror is calculated as a relative position from the driver. The relative position calculation unit 121 outputs the relative position of each mirror to the video processing unit 114.

The image processing unit 114 matches the most frequent gaze point in the left direction with the relative position of the left side mirror 2. The image processing unit 114 matches the most frequent gaze point in the right direction with the relative position of the right side mirror 3. The image processing unit 114 matches the most frequent gaze point in the vertical direction with the relative position of the room mirror 4. This is because the most frequent gaze point in the left direction matches the relative position of the left side mirror 2, the most gaze point in the right direction coincides with the relative position of the right side mirror 3, and the most frequent gaze point in the vertical direction is the room. This is because it is considered that the position matches the relative position of the mirror 4. Then, the image processing unit 114 adjusts to cut out the gaze image data so that the angle of the line of sight and the relative position of each mirror match. As a result, in the present embodiment, the reference of the gazing point is corrected, so that the accuracy of deriving the gazing point of the driver can be improved.

Processing of the control unit 110C of the video processing device 100C will be described with reference to FIG. FIG. 16 is a flowchart showing an example of the processing flow of the control unit 110C.

The control unit 110C acquires at least one most frequent gaze point in the horizontal direction and the vertical direction (step S401). Specifically, the control unit 110C acquires the most frequent gaze point by the gaze point history management unit 119. Then, the control unit 110C proceeds to step S402.

The control unit 110C acquires at least one position of the left side mirror 2, the right side mirror 3, and the room mirror 4 (step S402). Specifically, the control unit 110C causes the mirror position acquisition unit 120 to acquire the position of the mirror corresponding to the most frequently watched point acquired in step S401. Then, the control unit 110C proceeds to step S403.

The control unit 110C acquires the driver's head image data (step S403). Specifically, the control unit 110C causes the video data acquisition unit 111 to acquire the head video data of the driver. Then, the control unit 110C proceeds to step S404.

The control unit 110C calculates the relative position of the mirror based on the position of the mirror acquired in step S402 and the head image data of the driver acquired in step S403 (step S404). Specifically, the control unit 110C causes the relative position calculation unit 121 to calculate the relative position of the mirror. Then, the control unit 110C proceeds to step S405.

The control unit 110C adjusts the cutout position of the gaze image data based on the most frequent gaze point acquired in step S401 and the relative position of the mirror calculated in step S404 (step S405). Specifically, the control unit 110C causes the video processing unit 114 to adjust the cutout position of the gaze video data. Then, the control unit 110C ends the process of FIG.

As described above, in the second embodiment, the cutout position of the gaze image data can be adjusted based on the driver's most frequent gaze point and the relative position of the mirror. Thereby, the second embodiment can further improve the accuracy.

[Third embodiment]
The configuration of the video processing system 1D according to the third embodiment of the present invention will be described with reference to FIG. FIG. 17 is a block diagram showing the configuration of a video processing system 1D according to the third embodiment of the present invention.

As shown in FIG. 17, the video processing system 1D is different from the video processing system 1 in that the control unit 110D of the video processing device 100D includes a driving technique determination unit 122.

The driving skill determination unit 122, for example, based on the driver's head image data acquired by the image data acquisition unit 111, the gaze image data processed by the image processing unit 114, and the like, the driver's skill level of the driving technique. To judge.

The driving skill determination unit 122 determines the frequency with which the driver confirms the left side mirror 2, the right side mirror 3, and the room mirror 4 shown in FIG. 2, for example, based on the head image data. The driving skill determination unit 122 determines where the driver's attention is directed, for example, by determining whether a sign, another vehicle, or a pedestrian is being confirmed based on the gaze image data.

The driving skill determination unit 122 may score the driving skill of the driver, for example, according to the determination result. Specifically, the driving skill determination unit 122 gives a low score when the driver does not properly move the line of sight or does not properly visually recognize the sign. For example, when the driver's signal is sparsely confirmed and the driver does not start properly when the red signal changes to the green signal, the driving skill determination unit 122 gives a warning to the driver. Good. In this case, the driving skill determination unit 122 may output a voice signal to a speaker included in the vehicle C and issue a warning by voice, for example.

As described above, the third embodiment can determine the driving skill of the driver based on the head image data and the image data. With this, the driver can grasp the delay in recognizing a near-miss event, for example, based on the determination result, and thus the determination result can be used for improving the driving technique. That is, the third embodiment can also be used as a device for driving training that improves the driving skill of the driver.

1, 1A, 1B, 1C, 1D Image processing system 2 Left side mirror 3 Right side mirror 4 Rearview mirror 5 Steering wheel 10 Imaging unit 11 Peripheral imaging unit 12 Driver imaging unit 20 Display unit 30 Illuminance sensor 40 Rainfall sensor 100, 100A , 100B, 100C, 100D Video processing device 110, 110A, 110B, 110C, 110D Control unit 111 Video data acquisition unit 112 Eye gaze estimation unit 113 Gaze point deriving unit 114 Video processing unit 115 Output control unit 116 Eye gaze detection unit 117 Vehicle information acquisition Part 118 Surrounding information acquisition part 119 Gaze point history management part 120 Mirror position acquisition part 121 Relative position calculation part 122 Driving skill determination part 130 Storage part

Claims (9)

A video data acquisition unit that acquires video data from an imaging unit that images the surroundings of the vehicle;
A line-of-sight estimation unit that estimates the line-of-sight direction of the driver based on the position and orientation of the head of the driver who drives the vehicle,
A gazing point deriving unit for deriving a gazing point of the driver based on the line-of-sight direction of the driver,
An image processing unit that cuts out gaze image data based on the gaze point from the image data,
An output control unit for outputting the gaze image data,
Video processing device. Further comprising a line-of-sight direction detection unit that detects the line-of-sight direction of the driver based on the image of the eyeball of the driver,
The video processing device according to claim 1. Further comprising a vehicle information acquisition unit for acquiring vehicle information including the vehicle speed of the vehicle,
The image processing unit determines a cutout range of the gaze image data based on the vehicle information and the line-of-sight direction,
The video processing device according to claim 1. The video processing unit determines a cutout range of the gaze video data based on peripheral information of the periphery of the vehicle,
The video processing device according to claim 1. The image processing unit visualizes the movement of the driver's line of sight in the image data or the gaze image data,
The video processing device according to claim 1. Based on the movement of the driver's line of sight, further comprising a driving skill determination unit that determines the skill level of the driving skill of the driver,
The video processing device according to claim 5. A gazing point history management unit that manages the history of the gazing point and derives the most frequent gazing point in at least one of the left direction, the right direction, and the vertical direction of the gazing point,
Further comprising a mirror position acquisition unit for acquiring the position of the side mirror or the rearview mirror of the vehicle as a relative position from the driver,
The image processing unit corrects the reference of the gazing point by matching the most frequent gazing point and the relative position of the side mirror or the room mirror.
The video processing device according to claim 1. A step of acquiring video data from an image capturing unit that captures an image of the surroundings of the vehicle;
Estimating the line-of-sight direction of the driver based on the position and orientation of the head of the driver driving the vehicle,
Deriving a gazing point of the driver based on the line-of-sight direction of the driver,
Cutting out gaze image data based on the gaze point from the image data,
Outputting the gaze image data,
Video processing method. A step of acquiring video data from an image capturing unit that captures an image of the surroundings of the vehicle;
Estimating the line-of-sight direction of the driver based on the position and orientation of the head of the driver driving the vehicle,
Deriving a gazing point of the driver based on the line-of-sight direction of the driver,
Cutting out gaze image data based on the gaze point from the image data,
Outputting the gaze image data,
A program for causing a computer that operates as a video processing device to execute.

Images