DE102018005909A1 - Driver status recognizer, driver condition recognition system and driver condition detection method - Google Patents

Abstract

There is provided a driver condition recognizing apparatus which can accurately determine whether a driver is in a condition capable of immediately assuming a driving operation, particularly an operation of a steering wheel during autonomous driving based on a holding state of an object through the hands of the driver. A driver condition recognizing apparatus 20 which recognizes a condition of a driver 4 of a vehicle 2 provided with an autonomous driving system 1 includes an image capturing unit 22a which captures an image captured by a camera 31 which captures the driver Hand detection unit which detects the hands of the driver 4 from the image detected by the image capturing unit 22a, an object holding state detection unit 22c which detects the holding state of an object by the hands of the driver 4 detected by the hand detecting unit 22b, and a Standby determination unit 22d which determines whether the driver 4 is in a state capable of directly operating a steering wheel 5 of the vehicle 4 during autonomous driving, based on the holding state of the object detected by the object holding state detection unit 22c becomes.

Description

REFERENCES TO RELATED APPLICATIONS

This application claims priority Japanese Patent Application No. 2017-169962 , filed on Sep. 5, 2017, the entire contents of which are incorporated herein by reference.

TERRITORY

The present invention relates to a driver condition recognizing apparatus, a driver condition recognizing system, and a driver condition recognizing method, and more particularly to a driver condition recognizing apparatus, a driver condition recognizing system, and a driver condition recognizing method which determines a condition of a driver of a vehicle recognize which can drive autonomously.

BACKGROUND

In recent years, development of technologies for autonomous driving for vehicles has been actively pursued. International investigations concerning technical standards relating to vehicles for autonomous driving were also pursued. For example, with regard to international standards relating to autonomous steering wheel operation, discussions are currently underway for revision of international standards relating to autonomous steering wheel technologies (such as autonomous lane keeping technologies and autonomous change technologies) a lane) for a state in which the steering wheel is held when the vehicle is running at or more than 10 km / h.

In addition, there are plans also to pursue investigations concerning the international standards relating to technologies for autonomous steering wheel operation (such as autonomous lane keeping technologies and continuous autonomous driving technologies) for a state in which the steering wheel is not is held when a vehicle is traveling at or exceeding 10 km / h. For setting standards relating to the above technologies for autonomous steering wheel operation for a state in which the steering wheel is not held, it is considered that various requirements are required. As one of these requirements, for example, monitoring the condition of the driver in an autonomous driving system to allow the driver to smoothly undertake manual driving of autonomous driving when a request is obtained from the system is examined.

As a technology for monitoring the driver's condition, the following discloses JP 2014-106573 a technology for determining while the driver is driving a vehicle, whether the driver operates a portable terminal based on steering wheel holding information and information regarding the line of sight of the driver, and when determining that the driver is operating a portable terminal , limiting the functions of the portable terminal. According to the technology, which in JP 2014-106573 It is possible to avoid obstructing safe driving by restricting operation functions of a portable terminal while the driver is driving a vehicle.

Incidentally, in a situation where the driver operates a portable terminal during autonomous driving, an abnormality or failure occurs in the autonomous driving system, and a takeover request for taking manual driving is suddenly output is not easy for the driver to immediately take a driving or driving.

Accordingly, it is also conceivable that technology, which in JP 2014-106573 is disclosed as adopting a technology for recognizing a condition of a driver of a vehicle capable of autonomous driving. However, in technology, which in JP 2014-106573 discloses that a driver operates a portable terminal in a case where the driver does not hold the steering wheel with both hands and the driver's line of sight is not directed forward. That is, in technology, which in JP 2014-106573 is disclosed, a current state of the driver's hands can not be recognized at all in the case where the driver does not hold the steering wheel.

During autonomous driving, which does not require holding the steering wheel, it is envisaged that the driver, while holding various objects in his or her hand, performs actions separately from holding and operating a portable terminal. These acts also include actions that make it difficult to immediately take a ride, such as eating or drinking with both hands. In technology, which in JP 2014 - 106573 can be various actions by the / the driver who uses his or her hands, which is thought of or imagined that they performed during an autonomous driving Therefore, there is a problem that it can not be accurately recognized whether the hands of the driver are in a condition capable of directly assuming an operation of the steering wheel.

JP 2014-106573 is an example of the prior art.

SUMMARY

The present invention has been made in consideration of the above circumstances, and it is an object thereof to provide a driver condition recognizing apparatus, a driver condition recognizing system, and a driver condition recognizing method which can accurately recognize whether a condition of the hands of one Driver is a state during autonomous driving, to be able to immediately take over an operation of a steering wheel.

• eine Bilderfassungseinheit, welche konfiguriert ist, um ein Bild zu erfassen bzw. zu erhalten, welches durch eine Kamera aufgenommen wird, welche für ein Aufnehmen des Fahrers konfiguriert bzw. aufgebaut ist;
• eine Handdetektionseinheit, welche konfiguriert ist, um eine Hand des Fahrers aus dem Bild zu detektieren, welches durch die Bilderfassungseinheit erfasst bzw. erhalten wurde;
• eine Gegenstandshaltezustands-Detektionseinheit, welche konfiguriert ist, um einen haltenden bzw. Haltezustand eines Gegenstands bzw. Objekts durch die Hand des Fahrers zu detektieren, welche durch die Handdetektionseinheit detektiert wird;
• eine Bereitschafts-Bestimmungseinheit, welche konfiguriert ist, um zu bestimmen, ob sich die Hand des Fahrers in einem Zustand befindet fähig zu sein, unmittelbar ein Lenkrad des Fahrzeugs während eines autonomen Fahrens zu betätigen, basierend auf dem Haltezustand des Gegenstands, welcher durch die Gegenstandshaltezustands-Detektionseinheit bzw. Detektionseinheit eines den Gegenstand haltenden Zustands detektiert wird, und wobei der Gegenstand ein Gegenstand ist, welcher verschieden von einem Zubehör bzw. Zubehörteil des Fahrzeugs ist, welches der Fahrer mit der Hand betätigt.

In order to achieve the above object, a driver condition recognizing apparatus according to the present invention is a driver condition recognizing apparatus for detecting a condition of a driver of a vehicle provided with an autonomous driving system, the apparatus including:

• an image capture unit configured to acquire an image captured by a camera configured to receive the driver;
• a hand detection unit configured to detect a driver's hand from the image acquired by the image capture unit;
• an article holding state detection unit configured to detect a holding state of an object by the driver's hand, which is detected by the hand detecting unit;
• a standby determining unit configured to determine whether the driver's hand is in a state capable of directly operating a steering wheel of the vehicle during autonomous driving, based on the holding state of the object caused by the item holding state Detection unit or detection unit of the object holding state is detected, and wherein the object is an object which is different from an accessory or accessory of the vehicle, which the driver operates by hand.

According to the above driver state recognition apparatus, hands of the driver are detected from the image, the holding state of the object is detected by the hands of the driver, and it is determined whether the hands of the driver are in a state capable of be to directly actuate the steering wheel of the vehicle during autonomous driving, based on the holding state of the object. In this way, it is possible to accurately recognize whether the hands of the driver are in a condition capable of directly taking over the operation of the steering wheel during autonomous driving, and it is possible to appropriately provide assistance so that assumption of manual driving from autonomous driving, in particular, takeover of the operation of the steering wheel can promptly and smoothly be performed even in cases where failure occurs in the autonomous driving system during autonomous driving.

Also, in the driver state recognizing apparatus according to the present invention, the item holding state detecting unit may detect a feature of a region portion moving with the hand of a driver from the image, and may determine whether the driver is in a state of Holding the object is located, based on the feature of the region portion, which moves with the driver's hand and is detected from the image.

According to the above driver state recognition apparatus, the feature of the region portion moving with the hands of the driver is detected from the image, and it is detected whether the driver is in a state of holding the object based on the feature of the region portion or area detected from the image which moves with the hands of the driver. When the object is held by the hands of the driver, the object moves with the hands of the driver, and so it is by analyzing the feature of the region portion which moves with the hands of the driver in the image, i. by analyzing the feature of a region portion adjacent to the hands of the driver, it is possible to accurately detect whether the driver is in a state of holding the object. The feature of the region portion that moves with the hands of the driver may be the size or shape of the region portion, may be color information or luminance information, or may be a combination thereof.

Also, in the driver state recognizing apparatus according to the present invention, when the object holding state detection unit detects a predetermined state in which the driver does not hold the object in one hand, the standby determination unit may determine that the driver is in the state capable of be directly to operate the steering wheel during autonomous driving.

According to the above driver state recognition apparatus, when the predetermined state in which the driver does not hold the object in any one hand is detected, it is determined that the driver is in a state capable of directly applying the steering wheel during autonomous driving actuate. The predetermined state in which the driver does not hold the object in any one hand may be a state in which, for example, a state where the driver does not hold the object in any hand has continued for a predetermined period of time, or may be a state in which a state where the driver does not hold the object in any hand has been detected a predetermined number of times within a certain period of time. In this way, it becomes possible to accurately determine that the driver is in a condition capable of directly operating the steering wheel with both hands.

Also, in the driver state recognizing apparatus according to the present invention, when the object holding state detection unit detects a predetermined state in which the driver holds the article with at least one hand, the standby determination unit may determine that the driver is not in the state capable to be able to directly operate the steering wheel during autonomous driving.

According to the above driver state recognition apparatus, when the predetermined state in which the driver holds the article with at least one hand is detected, it is determined that the driver is not in a state capable of directly steering the steering wheel during autonomous driving to press. The predetermined state in which the driver holds the article with at least one hand may be a state in which, for example, a state where the driver holds the article with at least one hand has continued for a predetermined period of time, or may be a state in which a state where the driver holds the article with at least one hand was detected a predetermined number of times within a certain period of time. In this way, it is possible to accurately determine that the driver is not in a condition capable of directly operating the steering wheel with both hands.

Also, in the driver state recognizing apparatus according to the present invention, when the object holding state detection unit detects the holding state of the object, the standby determination unit determines whether the driver's hand is in the state capable of directly turning the steering wheel of the vehicle during a autonomous driving, based on determining a size of the object.

According to the above driver state recognition apparatus, whether the driver's hands are in a state capable of directly operating the steering wheel of the vehicle during autonomous driving is determined based on the size of the object, and thus it is when the size of the article is a size which does not hinder steering wheel hold, for example, it is possible to determine that there is a readiness.

Also, the driver state recognizing apparatus according to the present invention may include a steering wheel holding state detection unit of a steering wheel holding state which is configured to detect a holding state of the steering wheel by the driver, and the standby determination unit may be whether the driver is in the state of being able to directly operate the steering wheel during autonomous driving, based on the holding state of the object detected by the article holding state detection unit and the holding state of the steering wheel determining which one is detected by the steering wheel holding state detection unit.

According to the above driver state recognition apparatus, whether the driver is in a state capable of directly operating the steering wheel during autonomous driving is determined based on the holding state of the object and the holding state of the steering wheel. In this way, it is possible to prevent erroneous detection that the state of holding the steering wheel is a state of holding an object different from the steering wheel, and it is also possible to accurately detect whether the driver is different from an object from the steering wheel stops. As a result, it is possible to improve a determination accuracy for determining whether the hands of the driver are in a state capable of directly operating the steering wheel of the vehicle during autonomous driving.

Also, the driver state recognizing apparatus according to the present invention may include a sensor signal detecting unit configured to acquire a sensor signal based on contact of the driver's hand with a sensor provided in the steering wheel and the steering wheel holding state detection unit may detect the holding state of the steering wheel by the driver based on the sensor signal detected by the sensor signal detecting unit.

According to the above driver state recognition apparatus, a holding state of the steering wheel is detected by the driver based on the sensor signal detected by the sensor provided in the steering wheel. Accordingly, by detecting the sensor signal from the sensor provided in the steering wheel, it is possible to directly detect a holding state of the steering wheel by the driver.

Also, in the driver state recognizing apparatus according to the present invention, the steering wheel holding state detecting unit may detect the holding state of the steering wheel by the driver from the image.

According to the above driver state recognition apparatus, a holding state of the steering wheel by the driver can be detected from the image without providing a sensor in the steering wheel or the like. For example, in a case where the movement of the driver's hands detected from the image is the same as the direction in which the steering wheel is turned, it can be determined that the driver holds the steering wheel. Also, in a case where the movement of the driver's hands detected from the image is different from the direction in which the steering wheel is rotated, it may be determined that the driver is not holding the steering wheel. Also, by detecting from the image whether the hands of the driver are on the steering wheel, a holding state of the steering wheel by the driver can be detected. In addition, a holding state of the steering wheel may be detected from a method in which the driver's arms are extended or the like.

Also, the driver state recognition apparatus according to the present invention may include a support execution unit configured to perform assistance to the driver when the standby determination unit determines that the driver's hand is not in the state of being able to directly to operate the steering wheel.

According to the above driver state recognition apparatus, if it is determined that the driver's hands are not in the condition of being able to directly operate the steering wheel, assistance to the driver is performed. In this way, it is possible to perform assistance to prompt the driver to correct the condition of his or her hands in order to maintain a state of the hands that allows the driver to directly control the steering wheel himself to operate an autonomous driving.

Also, the driver state recognizing apparatus according to the present invention may include an information acquiring unit configured to acquire information that arises during autonomous driving from the autonomous driving system, and may include the assist execution unit Carry out a communication processing to the driver based on a determination result from the standby determination unit and the information that occurs during autonomous driving, which is detected by the information detection unit.

According to the above driver state recognition apparatus, notification processing according to the determination result is performed by the standby determination unit and the information that arises during autonomous driving, which is acquired by the information acquisition unit. In this way, it is not necessary to unnecessarily make various communications to the driver according to the situation of the autonomous driving system, thereby making it possible to reduce performance and processing required for communication.

Also, in the driver state recognizing apparatus according to the present invention, the information that arises during autonomous driving may include information for determining whether surroundings of the vehicle are in a safe state and if it is determined by the standby determination unit that the driver's hand is not in the state of being able to directly operate the steering wheel, the assist execution unit may execute a negotiation processing after changing a communication level to prompt the driver to change a state of the hand based thereon whether the surroundings of the vehicle are in a safe state.

According to the above driver state recognition apparatus, when it is determined that the driver's hands are not in the condition capable of directly operating the steering wheel, it is possible to perform a negotiation processing after changing the communication level to the driver (s) / to prompt or to correct the condition of his or her hands based on whether the vehicle's surroundings are in a safe state. As the information for determining whether the surroundings of the vehicle are in a safe state, it is preferable to include, for example, monitoring information of the surroundings of the vehicle. The monitoring information of the surroundings of the vehicle may be, for example, information indicating that the vehicle is rapidly approaching another vehicle, or may be information that the vehicle will travel on a road where the functional limit of the system is taken into consideration is conceivable, such as a narrow road with sharp turns.

For example, in a case where the surroundings of the vehicle are not in a safe state, it is preferable to raise the notification level and alert the driver more, such as by combining a display, an audio output , a vibration, etc., so that, for example, the driver assumes a state of the hands in which the driver can directly take over the operation of the steering wheel. On the other hand, when the surroundings of the vehicle are in a safe state, it is preferable to lower the level of attention and to make communication at a low level, for example, only by display.

Also, in the driver state recognition apparatus according to the present invention, the information that arises during autonomous driving may include takeover request information for taking manual driving from autonomous driving, and when it is determined by the standby determination unit that the If the driver is not in the state of being able to directly operate the steering wheel, and the information acquisition unit receives the transfer request information, the assistance execution unit may execute a negotiation processing to make the driver take over the operation of the steering wheel ,

According to the above driver state recognition apparatus, when it is determined that the driver's hands are not in a state capable of directly operating the steering wheel and the takeover request information is obtained, it is possible to perform the negotiation processing to perform the operation To cause driver to take over the operation of the steering wheel. The takeover request information may be, for example, information indicating that the vehicle has entered a takeover zone for taking over manual driving from autonomous driving, or information that notifies that an abnormality or a failure in one Part of the autonomous driving system has occurred. When the take-over request information is obtained, an actual operation of the steering wheel is required, and thus it is preferable, for example, that an agreement be made such that the driver quickly holds the steering wheel.

Also, a driver state recognition system of the present invention may include any of the above driver state recognition apparatuses and a camera configured to accommodate the driver.

According to the above driver state recognition system, the system is constructed to include the driver state recognizing apparatus and a camera that receives the driver. In this configuration, it is possible to construct a system which can accurately recognize whether the hands of the driver are in a condition capable of directly assuming the steering wheel even during autonomous driving, and which is also capable of suitably Accordingly, assistance can be provided so that assumption of manual driving from autonomous driving can be performed quickly and smoothly even in cases where failure occurs in the autonomous driving system during autonomous driving.

• einen Schritt eines Erfassens bzw. Erhaltens eines Bilds, welches durch eine Kamera aufgenommen wird, welche konfiguriert ist, um den Fahrer aufzunehmen;
• einen Schritt eines Speicherns des erfassten bzw. erhaltenen Bilds in einer Bildspeichereinheit;
• einen Schritt eines Auslesens des Bilds aus der Bildspeichereinheit;
• einen Schritt eines Detektierens einer Hand des Fahrers aus dem gelesenen Bild;
• einen Schritt eines Detektierens eines haltenden bzw. Haltezustands eines Gegenstands bzw. Objekts durch die detektierte Hand des Fahrers; und
• einen Schritt eines Bestimmens, ob sich die Hand des Fahrers in einem Zustand befindet fähig zu sein, unmittelbar das Lenkrad des Fahrzeugs während eines autonomen Fahrens zu betätigen, basierend auf dem detektierten haltenden Zustand des Gegenstands, und wobei der Gegenstand ein Gegenstand ist, welcher verschieden von einem Zubehör bzw. Zubehörteil des Fahrzeugs ist, welches der Fahrer mit der Hand betätigt.

Also, a driver state recognition method according to the present invention is a driver state recognition method for recognizing a driver state of a vehicle provided with an autonomous driving system, the method including:

• a step of acquiring an image picked up by a camera configured to pick up the driver;
• a step of storing the acquired image in an image storage unit;
• a step of reading out the image from the image storage unit;
• a step of detecting a hand of the driver from the read image;
• a step of detecting a holding state of an object by the detected hand of the driver; and
• a step of determining whether the driver's hand is in a state capable of directly operating the steering wheel of the vehicle during autonomous driving based on the detected holding state of the object, and wherein the object is an object different of an accessory of the vehicle which the driver operates by hand.

According to the above driver state recognition method, the hands of the driver are detected from the image, the holding state of the object is detected by the hands of the driver, and it is determined based on the holding state of the object Hands of the driver in a condition are able to directly operate the steering wheel of the vehicle during autonomous driving. In this way, it is possible to accurately recognize whether the hands of the driver are in a state capable of directly taking over the operation of the steering wheel during autonomous driving, and it is also possible to appropriately provide assistance so that takeover of manual driving from autonomous driving, in particular, takeover of the operation of the steering wheel can promptly and smoothly be performed even in cases where, for example, a failure occurs in the autonomous driving system during autonomous driving.

list of figures

• 1 FIG. 10 is a block diagram showing a configuration of a relevant portion of an autonomous driving system including a driver state recognizing apparatus according to an embodiment. FIG 1 of the present invention.
• 2 FIG. 13 is a side view showing a vicinity of a driver's seat of a vehicle in which a driver state recognition system according to the embodiment. FIG 1 is installed.
• 3 FIG. 10 is a block diagram showing a hardware configuration of the driver state recognition apparatus according to the embodiment. FIG 1 shows.
• 4 FIG. 10 is a flowchart showing a processing operation performed by a control unit in the driver state recognition apparatus according to the embodiment. FIG 1 is carried out.
• 5 FIG. 10 is a block diagram showing a configuration of a relevant portion of an autonomous driving system including a driver state recognizing apparatus according to an embodiment. FIG 2 contains.
• 6 FIG. 10 is a side view showing an environment of a driver's seat of a vehicle in which a driver state recognition system according to the embodiment. FIG 2 is installed.
• 7 FIG. 10 is a block diagram showing a hardware configuration of the driver state recognition apparatus according to the embodiment. FIG 2 shows.
• 8th FIG. 10 is a flowchart showing a processing operation performed by a control unit in the driver state recognition apparatus according to the embodiment. FIG 2 is carried out.

DETAILED DESCRIPTION

Hereinafter, embodiments of a driver state recognition apparatus, a driver state recognition system, and a driver state recognition method according to the present invention will be described based on the drawings. It should be noted that the following embodiments are preferred specific examples of the present invention and various technical limitations are applied, but the scope of the present invention is not limited to these embodiments unless it is particularly so given in the following description.

1 FIG. 10 is a block diagram showing a configuration of a relevant section of an autonomous driving system. FIG 1 showing which is a driver state recognition apparatus 20 according to one embodiment 1 contains.

The autonomous driving system or system 1 for autonomous driving is configured to the driver state recognition apparatus 20 , a camera 31 , a navigation device 40 , a regulation or control apparatus 50 for autonomous driving, an environmental monitoring sensor 60 , an audio output unit 61 and a display unit 62 and these units are via a bus line 70 connected.

The autonomous driving system 1 includes an autonomous driving mode in which the system, as the agent, autonomously at least part of a Driving control including performing acceleration, steering and braking of a vehicle, and a manual driving mode in which a driver performs the driving, and the system is constructed such that between these modes or Types can be switched.

For the autonomous driving mode is contemplated in the present invention that it is a mode in which the autonomous driving system 1 autonomously performs all of acceleration, steering and braking, and the driver handles or copes with requirements when using the autonomous driving system 1 be obtained (level of automation equal to a so-called level or level 3 or greater), but the application of the present invention is not limited to this level of automation. Also include times to which the autonomous driving system 1 requesting manual driving during autonomous driving, for example, a time of occurrence of a system abnormality or a system failure, a functional limit time of the system, and an autonomous driving end time.

The driver state recognizer 20 is an apparatus for detecting a condition of a driver of a vehicle, which is the autonomous driving system 1 includes, and is an apparatus for providing assistance by recognizing the driver's hands state and determining whether the driver is in a state capable of directly operating a steering wheel even during autonomous driving to be the driver To enable drivers to immediately undertake manual driving, in particular take over, the operation of the steering wheel when there is a takeover request for manual driving from the autonomous driving system 1 gives.

The driver state recognizer 20 is configured to be an external interface (external I / F) 21, a control unit 22 and a storage unit 26 to include. The control unit 22 is configured to be a central processing unit (CPU) 23 , a random access memory (RAM) 24 and a read-only memory (ROM) 25 to include.

The storage unit 26 is configured to include a storage device that stores data with a semiconductor device, such as a flash memory, a hard disk, a solid state drive, or other nonvolatile memory or volatile memory. The storage unit 26 saves a program 27 which is detected by the driver state recognition apparatus 20 and the like. Run is. It should be noted or noted that part or all of the program 27 in the ROM 25 the control unit 22 can be stored.

The camera 31 is an apparatus for picking up a driver sitting in a driver's seat. The camera 31 is configured to include a lens unit, an image sensor unit, a light irradiation unit, an input / output unit and a control unit for controlling these units, which are not shown. The image sensor unit is configured to include an image sensor such as a CCD or CMOS sensor, filters, and microlenses. The light irradiation unit includes a light emitting device such as an LED, and may be an infrared LED or the like to be able to capture images of the driver's state both day and night. The control unit is configured to include, for example, a CPU, a RAM, and a ROM, and may be configured to include an image processing circuit. The control unit controls the image sensor unit and the light irradiation unit to emit light (eg, near-infrared light, etc.) from the light irradiation unit, and performs control for capturing an image of reflected light of the irradiated light by using the image sensor unit.

2 FIG. 13 is a side view showing a vicinity of the driver's seat of a vehicle. FIG 2 shows in which a driver state recognition system 10 according to the embodiment 1 is installed.

A fixing position of the camera 31 in the vehicle is not particularly limited as long as it is a position at which a driver's upper body 4 which is in a driver's seat 3 sits, can be taken, and it is for the camera 31 preferred to be set at a position of which a spatial portion of a steering wheel 5 up to the driver's seat 3 can be included. For example, it's like this in 2 is shown, preferably, the camera 31 set at a position near a rearview mirror. Also, as another fixing position, the camera can 31 in the navigation device 40 or the like, in addition to an inner portion A of the steering wheel, in an inner section B of the instrument panel, on a dashboard C in a column section of the steering wheel 5 or installed on a section E of an A pillar.

The number of cameras 31 can be one, or can be two or more. The camera 31 may be separate from the driver state recognizer 20 configured (ie, configured as a separate body), or may be integral with the driver state recognizer 20 configured (ie, configured as an integrated body). Also is the type of camera 31 not particularly limited, and the camera 31 may be a visible light camera, a near infrared camera, a far infrared camera, or a combination of these cameras. In addition, the camera can 31 a monocular camera, a monocular 3D camera, a stereo camera, or the like. The monocular 3D camera includes a distance measurement image chip as an image sensor and is constructed to be able to detect presence / absence, size, position, posture and the like of an object to be monitored. Image data taken by the camera 31 are taken to the driver state recognition apparatus 20 sent. The driver state recognition system 10 is configured to the driver state recognizer 20 and the camera 31 to include. The driver state recognizer 20 and the camera 31 may be connected by a cable, or may be wirelessly connected.

The navigation device or the navigation device 40 which (s) in 1 is shown, is a device for providing the driver with information such as a current position of the vehicle and a route from the present position to a destination, and is configured to be a control unit, a display unit to include an audio output unit, an operation unit, a map data storage unit, and the like (not shown). Also is the navigation device 40 configured to be able to obtain a signal transmitted from a GPS receiver, a gyro sensor, a vehicle speed sensor, and the like (not shown).

The navigation device 40 derives information such as the road or lane on which the vehicle is traveling, and displays the current position on the display unit based on the vehicle position information measured by the GPS receiver and the map information of the map data storage unit. In addition, the navigation device calculates 40 a route from the present position of the vehicle to the destination and the like, displays the route information and the like on the display unit, and performs audio output of a route guide and the like from the audio output unit.

Also, some kinds of information such as vehicle position information, information concerning the road on which the vehicle is traveling, and information regarding a planned travel route generated by the navigation device 40 be calculated to the control apparatus 50 issued for autonomous driving. The information regarding the planned travel route may also include information regarding switching of control between autonomous driving and manual driving, such as information regarding start and end points of autonomous driving zones and information regarding zones for taking manual driving include autonomous driving.

The regulation or control apparatus 50 for autonomous driving is an apparatus for carrying out various kinds of control related to autonomous driving of the vehicle, and is configured by an electronic control unit having a control unit. Control unit, a storage unit, an input / output unit and the like (not shown) includes. The regulation or control apparatus 50 for autonomous driving is also associated with a steering control apparatus, a supply source control apparatus, a brake control apparatus, a steering sensor, an accelerator pedal sensor, a brake pedal sensor, and the like (not shown). These control devices and sensors may be in the configuration of the autonomous driving system 1 be included.

The regulation or control apparatus 50 for autonomous driving outputs a control signal for performing autonomous driving to each of the control apparatuses, performs control of autonomous driving of the vehicle (autonomous steering control, control) autonomous setting of speed, autonomous braking control and the like), and also performs switching of control for switching between the autonomous traveling mode and the manual driving mode based on information obtained from each unit which is in the autonomous driving system 1 is included.

"Autonomous driving" refers to allowing the vehicle to autonomously travel along a road under control, which is controlled by the control apparatus 50 for autonomous driving is performed without the driver in the driver's seat performing the driving operation. For example, a driving state in which the Vehicle is allowed to drive autonomously in accordance with a predetermined route to the destination or a driving route, which was automatically generated based on circumstances outside of the vehicle and a map information, as an autonomous driving included. Then, when a predetermined cancellation condition of autonomous driving is satisfied, the control apparatus can 50 for an autonomous driving an autonomous driving finish (cancel). For example, in the case where the control apparatus 50 for autonomous driving, determines that the autonomous vehicle has arrived at a predetermined end point of autonomous driving, the control apparatus 50 for autonomous driving, perform control for stopping autonomous driving. Also, when the driver performs an autonomous driving cancellation operation (for example, an operation of an autonomous driving cancellation button or an operation of the steering wheel, acceleration or braking, or the like performed by the driver), the rule or control apparatus 50 for autonomous driving, perform control for stopping autonomous driving. "Manual driving" refers to driving in which the driver drives the vehicle as the operator who performs the driving operation.

The environment monitoring sensor or environmental monitoring sensor 60 is a sensor that detects targets or objects existing around the vehicle. The target objects may include road markings (such as a white line), a safety fence, a highway center divider, or other structures that affect driving of the vehicle and the like, in addition to moving objects such as vehicles, bicycles, and people , The sensor monitoring the environment 60 includes at least one of a front area monitoring camera, a rear area monitoring camera, a radar, LIDAR (ie, Light Detection and Ranging or Laser Imaging Detection and Ranging) and an ultrasonic sensor. Detection data of the target object detected by the environment monitoring sensor 60 be detected are to the control or control apparatus 50 for autonomous driving and the like issued. As the front-range monitoring camera and the rear-range monitoring camera, a stereo camera, a monocular camera or the like may be used. The radar transmits radio waves, such as millimeter waves, to the surroundings of the vehicle and detects, for example, positions, directions and distances of the target objects by receiving radio waves reflected by the target objects existing in the surroundings of the vehicle. LIDER involves transmitting laser light to the surroundings of the vehicle and detecting, for example, positions, directions and distances of the target objects by receiving light reflected by the target objects existing in the surroundings of the vehicle.

The audio output unit 61 is an apparatus which provides various types of notifications based on instructions issued by the driver state recognizer 20 is provided with sound and voice, and is configured to include a speaker and the like.

The display unit 62 is an apparatus which provides various types of communications and guidance based on instructions provided by the driver state recognizer 20 is displayed with characters and graphics or by lighting and flashing a lamp or the like, and is configured to include various types of displays and indicator lamps.

3 FIG. 10 is a block diagram showing a hardware configuration of the driver state recognition apparatus. FIG 20 according to the embodiment 1 shows.

The driver state recognizer 20 is configured to the external interface (external I / F) 21, the control unit 22 and the storage unit 26 to include. The external I / F 21 is configured to be capable in addition to the camera 31 , with each unit of the autonomous driving system 1 such as the control apparatus 50 for an autonomous driving, the environments monitoring or environment monitoring sensor 60 , the audio output unit 61 , the display unit 62 and is constructed by an interface circuit and connecting connector for transmitting and receiving a signal to and from each of these units.

The control unit 22 is configured to be an image capture unit 22a , a hand detection unit 22b , an item holding state detection unit 22c and a standby determination unit 22d and may be configured to further include a message processing unit 22e to include. The communication processing unit 22e is an example of a support or support execution unit. The storage unit 26 is configured to be an image storage unit 26a , a hand detection method storage unit 26b , an item holding state detection method storage unit 26c and a determination method storage unit 26d to include.

The image storage unit 26a stores image data of the camera 31 passing through the image capture unit 22a be recorded or obtained. The hand detection method storage unit 26b stores, for example, a hand-detection program, which by the hand-detection unit 22b the control unit 22 and data necessary for executing the program.

The object holding state detection process storage unit 26c stores an item holding state detection program which is detected by the item holding state detection unit 22c the control unit 22 for detecting a holding state of an object by the driver's hand, and data necessary for executing the program, etc.

The determination method storage unit 26d stores a determination program which is determined by the standby determining unit 22d the control unit 22 to determine whether the hands of the driver are in a state capable of directly operating the steering wheel and data required for executing the program, etc.

The control unit 22 is an apparatus which performs the functions of the image capture unit 22a , the hand detection unit 22b , the article holding state detecting unit 22c , the standby determination unit 22d , the communication processing unit 22e etc. realized by interacting with the storage unit 26 processing for storing various data in the storage unit 26 is done and by adding different types of data and programs stored in the storage unit 26 are stored, read out and causes the CPU 23 executes these programs.

The image capture unit 22a which the control unit 22 Performing an editing for capturing the driver's image by the camera 31 and performs editing for storing the obtained image data in the image memory unit 26a by. The image of the driver may be a still image or may be a moving image. For example, the image of the driver may be at a predetermined interval (frame rate) after activation of the driver state recognition apparatus 20 to be obtained.

The hand detection unit 22b reads image data from the image storage unit 26a from, performs a predetermined image processing for the image data based on the program, which is from the manual detection process storage unit 26b has been read out, and performs editing for detecting the driver's hand from the image. As the driver's hand, the portion may be included from the wrist to the front end, or may also include the arm.

As the working, which in the hand detection unit 22b can be performed, various types of image processing for detecting coordinates or a region in which (r) the hands of the driver are shown by editing the image, which by the camera 31 is recorded, used or used. For example, if a visible light camera for the camera 31 and the captured image is a color image, an image processing is performed in which a flesh-colored region is extracted from image information such as RGB values of each pixel and the driver's hand from a feature such as size or shape or shape of the extracted flesh-colored region is detected.

Also, if a near infrared camera for the camera 31 and the captured image is a near infrared image, an image processing is performed in which a luminance region corresponding to a flesh-colored region is extracted from information such as a luminance value of each pixel and the hands of the Driver from a feature such as the size or shape of the extracted luminance region.

Also, if a far-infrared camera for the camera 31 and the captured image is a far-infrared image, image processing is performed in which a skin temperature region is extracted based on skin temperature information that can be obtained from the image and the driver's hands from a characteristic such as size or form of the extracted skin temperature region. Also, if a monocular 3D camera for the camera 31 is used, the position of the hands of the driver are detected using the obtained information, such as the detected position or posture of each part of the driver, which is detected by the monocular 3D camera.

The item holding state detection unit 22c performs processing for detecting a holding state of an object by the hands of the driver, which is performed by the hand detection unit 22b are detected, and specifically performs an editing such as detecting whether the driver is in a state of holding an object different from an accessory of the vehicle 2 which the driver operates or operates with his or her hands. Accessories of the vehicle 2 which the driver operates with his or her hands includes the steering wheel 5 and a gear lever (not shown).

As an editing performed by the item holding state detection unit 22c can be performed, various types of image processing for detecting a holding state of an object by the hands of the driver by editing the image, which by the camera 31 is used. For example, by detecting a feature of a region region that moves with the driver's hands, ie, a region portion adjacent to the driver's hands, from the image captured by the camera 31 is received, the processing for determining whether the driver is in a state of holding an item, be performed based on a feature of the region portion, which moves with the driver's hand.

The feature of the region portion that moves with the driver's hand may be the size or shape of the region portion, color information, or luminance information, or may be a combination thereof.

According to this processing, movement of the driver's hand is detected, and when color information of a region adjacent to the driver's hand, which is obtained before and after the movement of the driver's hand, is substantially the same, the region in which the color information is substantially the same as determining the region showing an object which the driver is holding, and thus it is possible to detect a state in which the driver holds the article.

It may also be detected whether the driver is in a state of holding an object based on the size or shape of the region in which the color information is substantially the same. According to this configuration, when the size or shape of the region in which the color information is substantially the same is a size or shape that does not hinder steering wheel hold, in other words, when an object which the driver holds is small is, for example, a small object, such as a cigarette or a vehicle key, determined that the driver does not hold an object.

The standby determination unit 22d performs an edit for determining whether the driver's hands are in a state capable of directly operating the steering wheel during autonomous driving, that is, determining whether the readiness for operating the steering wheel of the vehicle is high based on a holding state of an object, which is detected by the object holding state detection unit 22c is detected.

For example, when a predetermined state in which the driver does not hold an object in any hand may be detected by the article holding state detection unit 22c is detected, it is determined that the readiness for operating the steering wheel is high. The predetermined state in which the driver does not hold an object in any hand may be, for example, a state in which a state where the driver does not hold an object in any hand has continued for a predetermined period of time, or may be a state in which a state where the driver does not hold the object in any hand has been detected a predetermined number of times within a certain period of time.

The readiness for the operation of the steering wheel refers to the speed with which the driver turns the steering wheel 5 Can operate when the autonomous driving system 1 an operation for any reason stops or interrupts and an autonomous driving can not be continued. Accordingly, the state of high readiness for the operation of the steering wheel means the state in which the driver can operate the steering wheel as soon as the driver has to operate the steering wheel, and it is therefore not necessary for the driver to hold the steering wheel.

Also, when a predetermined state in which the driver holds an object with at least one hand, by the item holding state detection unit 22c is detected, it is determined that the readiness for operating the steering wheel is low. The predetermined state in which the driver holds an object with at least one hand may be, for example, a state in which a state where the driver holds an object with at least one hand has continued for a predetermined period of time, or may be a state in which a condition where the driver with the item holds at least one hand detected a predetermined number of times within a certain period of time.

When the standby determination unit 22e determines that the driver is not in a state capable of directly operating the steering wheel, that is, when it is determined that the readiness for operating the steering wheel is low, the communication processing unit leads 22d the negotiation processing to cause the audio output unit 61 and the display unit 62 perform an audio output and a display output to cause the driver to assume a state to be able to directly operate the steering wheel. Also, the communication processing unit 22e a signal informing to continue autonomous driving instead of canceling autonomous driving to the control apparatus 50 spend on autonomous driving.

4 FIG. 10 is a flowchart showing a processing operation performed by the control unit. FIG 22 in the driver state recognition apparatus 20 according to the embodiment 1 is carried out. Here, a description will be given assuming a state in which the autonomous driving system 1 is set to the autonomous driving mode, that is, the vehicle is in a state of driving under control of autonomous driving. This processing operation is repeatedly performed during the period in which the autonomous driving mode is set.

First, in one step S1 processing for obtaining image data of the driver by the camera 31 be taken. Image data can be taken from the camera 31 each as a frame at a time, or a plurality of frames of image data or frames of image data may be obtained together over a certain period of time. In the following step S2 becomes an edit for storing the obtained image data in the image memory unit 26a and then processing moves to a step S3 ,

In the step S3 become the image data stored in the image memory unit 26a are stored, read, and then the editing moves to a step S4 , The image data may be taken from the image storage unit 26a may be read out as one frame at a time, respectively, or the plural frames of picture data or frames of picture data may be jointly read out over a certain period of time. In the step S4 For example, editing for detecting the driver's hands from the read image data, for example, editing for detecting coordinates or a region of the driver's hands in the image is performed, and thereafter the processing moves to a step S5 ,

For the editing, which in the step S4 is performed to determine the hands of the driver from the image data, various types of image processing can be performed according to the characteristics of the camera 31 be applied.

For example, if a visible light camera for the camera 31 and the captured image is a color image, an editing is performed in which a flesh-colored region is extracted from image information such as RGB values of each pixel, the driver's hand from a feature such as size or shape or shape of the extracted flesh-colored region is detected, and coordinate data or region data of the hands of the driver is output in the image.

Also, if a near infrared camera for the camera 31 and the captured image is a near-infrared image, an editing in which a luminance region corresponding to a flesh-colored region is extracted from information such as a luminance value of each pixel is performed by the hands of the Driver are detected from a feature such as the size or shape of the extracted luminance region, and coordinate data or region data of the driver's hands are output in the image.

Also, if a far-infrared camera for the camera 31 and the captured image is a far-infrared image, an editing is performed in which a skin temperature region is extracted based on skin temperature information that can be obtained from the image, the driver's hand from a characteristic such as size or Form of the extracted skin temperature region is detected, and coordinate data or region data of the hands of the driver are output in the image.

Also, if a monocular 3D camera for the camera 31 is used, a position of the driver's hands are detected using obtained information, such as the detected position or posture of each part of the driver detected by the monocular 3D camera, and the detected data can be output.

In the step S5 determines whether the hands of the driver were detected. In the step S5 whether or not the hands of the driver have been continuously detected for a predetermined period of time may be determined, or it may be determined whether the hands of the driver have not been detected a predetermined number of times within a certain period of time. A case in which the hands of the driver have not been detected is a case in which the hands of the driver are not shown in the image, such as a state in which the hands of the driver are not in the captured area of the camera 31 or a condition in which the driver's hands are behind some object.

If in the step S5 is determined that the hands of the driver were not detected, then moving to a step moves S9 , On the other hand moves when in the step S5 it is determined that the hands of the driver have been detected, then editing to a step S6 , An edit for detecting a holding state of an object by the hands of the driver becomes in the step S6 and then processing moves to a step S7 , Various types of image processing can be applied to the editing which is in the step S6 is performed to detect a holding state of an object by the hands of the driver. An article described herein is an article which is different from accessories of the vehicle 2 (For example, the steering wheel, a gear lever and the like.) Which operates the / the driver with his or her hands.

For example, a feature of a region portion that moves with the hands of the driver (a region portion adjacent to the hands) is detected from the image, and editing for detecting whether the driver holds an object based on a feature of FIG Region section are performed, which moves with the driver's hand.

The feature of the region portion that moves with the driver's hand may be the size or shape of the region portion, color information, or luminance information, or may be a combination thereof.

For example, the movement of the driver is detected when color information or luminance information of a region adjacent to the driver's hand, which is obtained before and after the movement of the driver's hand, is substantially the same, it becomes the region in which the color information or the luminance information is substantially the same as determining the region showing an object which the driver holds, and it can be detected such that the state is a state in which the driver holds the object. It may also be detected whether the driver is in a state of holding an object based on the size or shape of the region in which the color information or the luminance information obtained before and after the movement of the driver's hand, in FIG Essentially the same.

In the step S7 It is determined whether a predetermined state in which the driver holds an object with at least one hand from the detection processing step S6 is detected, and when it is determined that the predetermined state in which the driver holds an object with at least one hand is detected, the processing thereafter moves to a step S8 , The predetermined state in which the driver holds an object with at least one hand may be a state in which a state where the driver holds an object with at least one hand has continued for a predetermined period of time or may a state in which a state where the driver holds the article with at least one hand was detected a predetermined number of times within a certain period of time.

It should be noted that instead of editing the step S7 an edit for determining whether a predetermined state in which the driver does not hold an object in any one hand is detected can be used. In this case, when it is determined that the predetermined state in which the driver does not hold an object in any one hand is not detected, the processing to the step. S8, while if it is determined that the predetermined state in which the driver does not hold an object in any hand is detected, the processing comes to a step S10 can move.

In the step S8 It is determined whether the size of an object which the driver holds is a size which does not hinder a holding of the steering wheel, that is, it is determined whether an object which the driver holds is small, and if the object which the Rider holds, is not small, that is, if it is determined that the size of the object is a size that obstructs a holding of the steering wheel, then the machining moves to a step S9 , It should be noted that the step S8 is not always necessary, and it may, if determined that the predetermined state in which the driver an object with at least one hand holds in the step S7 is detected, the editing to the step S9 move.

In the step S9 A stand-by flag f for determining whether the driver's hand is in a state capable of directly manipulating the steering wheel of the vehicle during autonomous driving is set to 0, and thereafter the processing moves to a step S11 , A state in which the stand-by flag is zero shows that the driver is not in a state capable of directly operating the steering wheel, that is, the readiness for operating the steering wheel is in a low state.

On the other hand moves in the step S7 when it is determined that the predetermined state in which the driver holds an object with at least one hand is not detected, in other words, when it is determined that a state in which the driver does not hold an object in any one hand, is detected, then editing to the step S10 ,

Also moves in the step S8 when it is determined that the size of an object which the driver holds is a size that does not hinder holding of the steering wheel, that is, when it is determined that the object which the driver holds is small, then the machining to the step S10 , For example, determining whether the variable is interfering with holding the steering wheel can be defined based on whether the thumb and fingers other than the thumb can contact each other when the driver holds an object. An article that does not interfere with holding the steering wheel includes a small object, such as a cigarette or the vehicle key, which can be held with two fingers. On the other hand, an article whose size hinders holding of the steering wheel includes a food whose size is approximately the same as the palm, such as a hamburger, a beverage container such as a PET bottle, a portable terminal such as a smartphone, or a An article which is large enough to be difficult to hold without full use of both hands, such as a magazine or a book.

In the step S10 For example, the stand-by flag f for determining whether the hands of the driver are in a state capable of directly operating the steering wheel of the vehicle during autonomous driving becomes 1 is set, and then the processing moves to a step S11 , A state in which the ready flag 1 shows that the driver is in a state capable of directly operating the steering wheel, that is, the readiness for operating the steering wheel is in a high state.

It gets in the step S11 determines if the ready flag 1 and if it is determined that the ready flag 1 That is, when it is determined that the stand-by for operating the steering wheel is in the high state, the machining ends thereafter. It should be noted that, in another embodiment, a negotiation processing for informing the driver that the hands are in the high state may be performed by, for example, communicating an appropriate lamp in an appropriate posture display unit 62 is provided and a communication lamp of a suitable attitude is turned on. Alternatively, a signal informing the driver that the driver is in a state capable of directly operating the steering wheel, that is, the driver is taking an appropriate posture for continuing autonomous driving may be attached to the rule - or control apparatus 50 for autonomous driving.

On the other hand, if the ready flag does not move 1 but is 0, that is, when it is determined that the standby for operating the steering wheel is in the low state, the machining to a step S12 , In the step S12 An agreement processing is carried out for or to the driver.

As the negotiation processing, editing may be for outputting a predetermined audio output from the audio output unit 61 or editing may be performed for displaying a predetermined display on the display unit 62 be performed. The negotiation processing is an editing for alerting the driver so that the driver assumes a state of the hands that allows the driver to directly operate the steering wheel in the case where failure or the like occurs in the autonomous driving system 1 occurs. For example, an audio output such as "Please discard what you hold in your hand" may be issued, or an indicator may be made for prompting the driver, not something in his or her hands to keep. Also, in the step S12 a signal communicating to continue an autonomous driving rather than an autonomous driving, to the control apparatus 50 for autonomous driving.

In the above driver state recognition apparatus 20 according to the embodiment 1 detects the hand detection unit 22b the driver's hands from a picture taken by the camera 31 is picked up, the object holding state detection unit detects 22c a holding state of an object by the hands of the driver and determines the standby determining unit 22d whether the hands of the driver are in a condition capable of directly actuating the steering wheel of the vehicle during autonomous driving, based on the holding state of the object. In this way, it is possible to accurately recognize whether the state of the hands is a state capable of directly taking over the operation of the steering wheel during autonomous driving, and it is also possible to provide such assistance as is appropriate to provide that a takeover of manual driving autonomous driving, in particular a takeover of the operation of the steering wheel can be performed quickly and smoothly even in cases where, for example, a failure in the autonomous driving system 1 occurs during autonomous driving.

Also, it is according to the driver state recognition apparatus 20 because a feature of a region portion moving with the hands of the driver in the image is the item holding state detection unit 22c is analyzed, it is possible to accurately detect whether the driver is in a state of holding an object which is different from an accessory of the vehicle which the driver operates with his or her hands. Also, when the size or shape of the region portion that moves with the driver's hand is a size or shape that does not hinder holding of the steering wheel, it is possible to detect the condition as a condition in which the driver does not hold an object, and it is possible to perform detection in accordance with the actual operation of the steering wheel, such as whether the driver is in a state of holding an object, which prevents a holding of the steering wheel.

Also, in the driver state recognition apparatus 20 That is, when the standby determination unit detects a predetermined state in which the driver holds an object with at least one hand, it determines that the standby is in the low state. In this way, it is possible to accurately determine that the driver is not in a condition capable of directly operating the steering wheel with both hands.

Also, according to the driver state recognizing apparatus 20 That is, when the standby determination unit detects a predetermined state in which the driver does not hold an object in any hand, it determines that the standby is in the high state. In this way, it is possible to accurately determine that the driver is in a condition capable of directly operating the steering wheel with both hands.

Also performs according to the driver state recognition apparatus 20 when the standby determination unit 22d determines that the driver is not in a state capable of directly operating the steering wheel of the vehicle, the communication processing unit 22e an edit for transmitting a message to the driver. In this way, it is possible to prompt the driver to correct the condition of his or her hands in order to maintain a state of the hands which enables the driver to directly operate the steering wheel himself during autonomous driving ,

5 FIG. 10 is a block diagram showing a configuration of a relevant section of an autonomous driving system. FIG 1 showing which is a driver state recognition apparatus 20A according to one embodiment 2 includes. It should be noted that constituent components have the same functions as those of the autonomous driving system 1 which is in 1 10, the same reference numerals are assigned and a description thereof will be omitted here.

The embodiments 1 and 2 differed greatly in that the camera 31 in the driver state recognition system 10 according to the embodiment 1 is used while a steering wheel sensor 32 in addition to the camera 31 in a driver state recognition system 10A according to the embodiment 2 is provided.

6 FIG. 13 is a side view showing a vicinity of a driver's seat of the vehicle. FIG 2 shows in which the driver state recognition system 10A according to the embodiment 2 is installed.

The steering wheel sensor 32 is in the steering wheel 5 of the vehicle 2 is provided and is not particularly limited as long as the sensor is able to detect a hand holding the steering wheel 5 touched. The steering wheel sensor 32 For example, it may be a sensor of an electrostatic capacitance or a pressure sensitive sensor, or it may be a biosensor that detects biological information, such as a heartbeat or a pulse from a hand, which is the steering wheel 5 touched.

The electrostatic capacity sensor is a sensor that makes contact with the steering wheel 5 detected by hand by detecting a change in an electrostatic capacitance which is detected between an electrode unit included in the steering wheel 5 is provided, and the hand occurs. The pressure-sensitive sensor is a sensor which makes contact with the steering wheel 5 detected by the hand by a pressure generated at the time when the driver turns the steering wheel 5 is detected from a change in a contact area (resistance value) between the electrode unit and a detection unit which is in the steering wheel 5 is provided. Several steering wheel sensors 32 may be in a peripheral portion or a spoke portion of the steering wheel 5 be provided to a contact position of the hand on the steering wheel 5 to detect. A signal generated by the steering wheel sensor 32 is detected is sent to the driver state recognition apparatus 20A output.

7 FIG. 10 is a block diagram showing an example of a hardware configuration of the driver state recognition apparatus. FIG 20A according to the embodiment 2 shows. It should be noted that constituent components having the same functions as those of the driver condition detecting apparatus 20 which is in 3 10, the same reference numerals are assigned and a description thereof is omitted here.

The driver state recognizer 20A according to the embodiment 2 is different from the driver state recognition apparatus 20 according to the embodiment 1 in that a control unit 22A further, a sensor signal detection unit 22f , a steering wheel holding state detection unit 22g and an information acquiring unit 22i and that the processing performed by the standby determination unit 22h and a communication processing unit 22j subsequently differentiated to the addition of these units.

The control unit 22A is configured to the image capture unit 22a , the hand detection unit 22b , the object holding state detection unit 22c , the sensor signal detection unit 22f , the steering wheel holding state detection unit 22g , the standby determination unit 22h , the information acquisition unit 22i and the communication processing unit 22j to include. The communication processing unit 22j is an example of a support execution unit. A storage unit 26A is configured to the image storage unit 26a , the hand detection method storage unit 26b , the item holding state detection method storage unit 26c and a determination method storage unit 26e to include.

The image storage unit 26a stores image data of the camera 31 passing through the image capture unit 22a be recorded or obtained. The hand detection method storage unit 26b stores a hand detection program, which is passed through the hand detection unit 22b the control unit 22 to perform, data necessary for executing the program, and the like. The object holding state detection process storage unit 26c stores an item holding state detection program which is detected by the item holding state detection unit 22c the control unit 22A is to perform, data necessary for executing the program, and the like.

The determination method storage unit 26e stores a determination program which is determined by the standby determining unit 22h the control unit 22A to detect whether the hands of the driver are in a state capable of directly operating the steering wheel, data required for executing the program, and the like.

The image capture unit 22a which the control unit 22A Performing processing for acquiring the image of the driver, which is performed by the camera 31 and performs editing for storing the obtained image data in the image memory unit 26a by.

The hand detection unit 22b reads image data from the image storage unit 26a from, performs predetermined image processing for the image data based on the program, which is from the hand-detection method storage unit 26b is read out, and performs editing for detecting the hands of the driver in the image.

The item holding state detection unit 22c performs an editing for detecting a holding state of an object by the hands of the driver, which by the hand detection unit 22b be detected, for example, it performs a detection whether the driver is in a state of holding an object or object.

The sensor signal detection unit 22f performs an edit for detecting a sensor signal received by the steering wheel sensor 32 is detected. The steering wheel holding state detection unit 22g performs an editing for detecting a holding state of the steering wheel by the driver based on the sensor signal, which by the sensor signal detection unit 22f was recorded or received. It should be noted that as a holding state of the steering wheel, not only a state in which the steering wheel is held by the hands, but also a state in which the hands contact the steering wheel (a state in which the Hands are arranged on the steering wheel), may be included.

The standby determination unit 22h performs an edit for determining whether the driver's hands are in a state capable of directly operating the steering wheel during autonomous driving, based on a holding state of an object caused by the article holding state detection unit 22c is detected, and a holding state of the steering wheel, which by the Lenkradhaltezustands detection unit 22g is detected.

For example, when a state in which the steering wheel is not held by the steering wheel holding state detection unit 22g is detected, and a state in which the driver does not hold an object in any hand by the object holding state detection unit 22c is detected, it is determined that the driver is in a state capable of directly operating the steering wheel during autonomous driving.

Also, when a state in which the steering wheel is not held by the Lenkradhaltezustands detection unit 22g is detected, and a state in which the driver holds an object with at least one hand by the object holding state detection unit 22c is detected, it is determined that the driver is not in a state capable of directly operating the steering wheel during autonomous driving.

Also, when a state in which the steering wheel is held with both hands, by the Lenkradhaltezustands detection unit 22g is detected regardless of a detection state by the object holding state detection unit 22c determining that the driver is in a condition capable of directly actuating the steering wheel during autonomous driving.

The information acquisition unit 22i detects or obtains information that occurs during autonomous driving from the units of the autonomous driving system 1 , The information that arises during autonomous driving includes at least one of monitoring information from environments of the vehicle detected by the environment monitoring sensor 60 and takeover request information for taking over manual driving from autonomous driving which is from the control apparatus 50 for autonomous driving.

When the standby determination unit 22g determines that the driver is not in a state to be able to directly operate the steering wheel leads the communication processing unit 22j edit by to the audio output unit 61 and the display unit 62 to cause an output processing for an audio output and a display for prompting the driver to assume a state of the hands, which allows the driver to operate the steering wheel immediately, according to the information that arises during autonomous driving or occurs, which by the information acquisition unit 22i is detected. Also, the communication processing unit 22j output a signal representing the control apparatus 50 for an autonomous driving, to continue an autonomous driving rather than to pick up an autonomous driving.

8th FIG. 10 is a flowchart showing a processing operation performed by the control unit. FIG 22A in the driver state recognition apparatus 20A according to the embodiment 2 is carried out. The processing operation of the flowchart which is in 8th is different from that of the flowchart shown in FIG 4 is shown, to the effect that steps S21 and 22 between the steps S6 and S7 are used, and that steps S31 to S36 instead of the step S12 are used. Processing operations having the same contents as those of the flowchart shown in FIG 4 are shown, the same reference numerals are assigned, and a description thereof will be omitted.

First, in the step S1 an edit for obtaining image data of the driver by the camera 31 be recorded, performed, and then in the step S2 an edit for storing the obtained image in the image memory unit 26a carried out. The image is taken from the image storage unit 26a in the step S3 read out and in the following step S4 an edit for detecting the driver's hand is performed from the read image data, and thereafter, the processing moves to the step S5 ,

In the step S5 determines whether the hands of the driver are detected. If it is determined that the driver's hands are not in the step S5 are detected, the processing moves to the step S9 while if it is determined that the driver's hands in the step S5 be detected, the editing to the step S6 emotional. The processing for detecting a holding state of an object by the hands of the driver becomes in the step S6 and then processing moves to the step S21 ,

A processing for obtaining a sensor signal from the steering wheel sensor 32 is in the step S21 carried out, and in the subsequent step S22 It is determined whether the driver holds the steering wheel with both hands based on the obtained sensor signal. If the sensor signal is a signal showing that there are two contact points, it is determined that the steering wheel is held with both hands.

If in the step S22 it is determined that the steering wheel is not held with both hands, that is, if it is determined that the steering wheel is held with one hand or not held by any hand, then the machining moves to the step S7 ,

In the step S7 it is determined whether a predetermined state in which the driver holds an object with at least one hand, from a result of the detection processing step S6 is detected, and when it is determined that the predetermined state in which the driver holds an object with at least one hand is detected, the processing thereafter moves to the step S8 ,

In the step S8 It is determined whether the size of an object which the driver holds is a size which does not hinder a holding of the steering wheel, that is, it is determined whether an object which the driver holds is small, and if the object which the Rider holds, is not small, that is, if it is determined that the size of the object is a size that obstructs a holding of the steering wheel, then the machining moves to the step S9 , It should be noted that the step S8 is not always necessary, and when it is determined that the predetermined state in which the driver holds an object with at least one hand in the step S7 is detected, the processing may become the step S9 move.

In the step S9 the stand-by flag f is set to 0, and then the processing moves to the step S11 , A state in which the ready flag 0 shows that the driver is not in a state capable of directly operating the steering wheel, that is, that the readiness for operating the steering wheel is in a low state.

On the other hand moves when in the step S22 it is determined that the steering wheel is held with both hands, the processing to the step S10 ,

Also moves in the step S7 when it is determined that the predetermined state in which the driver holds an object with at least one hand is not detected, in other words, when it is determined that a state in which the driver does not hold an object in any one hand, is detected, then editing to the step S10 ,

Further moves in the step S8 when it is determined that the size of an object which the driver holds is a size which does not hinder a holding of the steering wheel, that is, when it is determined that the object which the driver holds is small, then the machining is done the step S10 ,

In the step S10 the standby flag f is set to 1, and then the processing moves to the step S11 , A state in which the ready flag 1 shows that the driver is in a state capable of directly operating the steering wheel, that is, the readiness for operating the steering wheel is in a high state.

It is in the step S11 determines if the ready flag 1 and if it is determined that the readiness flag 1 That is, when it is determined that the stand-by for operating the steering wheel is in the high state, the machining ends thereafter. On the other hand, when the stand-by flag is not 1, but 0, that is, when it is determined that the standby for operating the steering wheel is in the low state, the machining moves to a step S31 ,

Information is provided by the autonomous driving system 3 in the step S31 and then editing moves to a step S32 , The information includes the monitoring information of environments of the vehicle detected by the environment monitoring sensor 60 and the takeover request information for taking over manual driving which is from the control apparatus 50 is spent for autonomous driving. The takeover request information includes, for example, a system occurrence abnormality (failure) occurrence signal, a system functional limit signal, or a Entrance signal indicating entry into a takeover zone.

In the step S32 It is determined whether the surroundings of the vehicle are in a safe state based on the monitoring information of environments of the vehicle detected by the environment monitoring sensor 60 is obtained. In the step S32 moves when it is determined that the surroundings of the vehicle are not in a safe state, such as when it is determined, for example, that information indicating that another vehicle, person, or other obstacle is in a certain area of the environment Vehicle (any of forward, lateral or backward) is detected, information that notifies that another vehicle is approaching rapidly, or information indicating that the vehicle will drive on a road where the functional Limit of the system is to be expected, such as a narrow road with sharp curves, was obtained, the editing of a communication processing at a high level of a step S34 , In the step 34 A high-level negotiation processing to cause the driver to rapidly assume a state of the hands which enables the driver to directly operate the steering wheel is performed, and thereafter the processing ends. In the high-level negotiation processing, it is preferable that a communication be performed in which a display and an audio output are combined. An understanding other than a display or an audio output, such as applying vibrations to the driver's seat or the like, may be added. Also, in the step S34 a signal that informs to continue autonomous driving instead of canceling autonomous driving, to the control apparatus 50 for autonomous driving.

On the other hand moves in the step S32 if it is determined that the surroundings of the vehicle are in a safe state, then processing to a step S33 , In the step S33 It is determined whether the takeover request information for taking over manual driving from the control apparatus 50 for autonomous driving, ie it is determined if there is a takeover request.

In the step S33 If it is determined that there is no takeover request, processing moves to low level negotiation processing S35 , In the step S35 For example, the negotiation processing is performed at a low level to cause the driver to assume a state of the hands, which allows the driver to hold the steering wheel, and then the processing ends. In the low-level negotiation processing, it is preferable to gently communicate the driver, such as an announcement by a display only, to achieve harmony between the autonomous driving and the driver.

On the other hand moves in the step S33 if it is determined that there is the takeover request, editing to a step S36 , In the step S36 For example, an acquisition understanding is performed by an audio output or display such that the driver immediately holds the steering wheel and takes the driving, and then the editing ends.

With the driver state recognizer 20A according to the embodiment 2 It is determined whether the driver is in a state capable of directly operating the steering wheel during autonomous driving, based on a holding state of an object which is detected by the object holding state detection unit 22c is detected, and a holding state of the steering wheel, which by the steering wheel holding state detection unit 22g is detected. In this way, it is possible to prevent erroneous detection that the state of holding the steering wheel is a state of holding an object different from the steering wheel, and thus it is possible to accurately detect whether the driver is an object different from the Steering wheel stops. As a result, it is possible to improve a determination accuracy for determining whether the driver's hand is in a state capable of directly operating the steering wheel during autonomous driving.

Also performs according to the driver state recognition apparatus 20A the communication processing unit 22j a communication processing to the driver according to the information which occurs during autonomous driving, which by the information acquiring unit 22i is received or recorded. When environment monitoring information indicating that the vehicle environments are not in a safe state is obtained, it is possible to raise the level of communication that the communication processing unit has 22j is performed to more alert or alert the driver, so that the driver assumes a posture, which allows the driver to take over immediately the operation of the steering wheel. Also, if the surroundings of the vehicle are safe, and therefore there is no takeover request, it is possible to gently cause the driver to to correct the condition of his or her hands by communication at a low level. On the other hand, if there is the take-over request information, it is possible to inform the driver to take a posture for immediately taking over manual driving.

In this way, it is not necessary, not necessarily different understandings to the driver according to the state of the autonomous driving system 1 Therefore, performance and processing required for communication can be reduced. Also, at the time of occurrence of a failure of an operating limit of the system, or at the time of requesting to take manual driving, the time period until the driver operates the steering wheel and the assumption of the driving operation is completed , are shortened, thereby allowing a quick and smooth or smooth takeover is performed. It is also possible to have a corresponding communication, which is gentle to the driver, according to the state of the autonomous driving system 1 perform.

It should be noted that the information acquisition unit 22i in the driver state recognition apparatus 20 according to the embodiment 1 can be provided, and that instead of the step S12 which is in 4 shown is a similar editing of the steps S31 to s36, which in 8th that is, assistance to the driver such as various kinds of communication processing and the like can be performed for the driver according to information that occurs during autonomous driving by the information acquiring unit 22i is obtained.

Also, the above driver condition detecting apparatus is 20A according to the embodiment 2 configured such that the sensor signal from the steering wheel sensor 32 and a holding state of the steering wheel is detected based on the obtained sensor signal, however, in another embodiment, whether the driver holds the steering wheel can be detected from the image transmitted through the camera 31 is recorded without the steering wheel sensor 32 provide or make available.

For example, when the steering wheel is captured in an image, which may be transmitted through the camera 31 is picked up, the hand, which is arranged on the steering wheel, are detected based on skin color information or luminance information. Also, if the steering wheel is not included in an image, which is through the camera 31 a state in which the driver holds the steering wheel is detected from information on an image showing whether the driver's arms are extended in the direction of the steering wheel, and the like.

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• JP 2017169962 [0001]
• JP 2014106573 [0005, 0007, 0009]
• JP 2014 [0008]
• JP 106573 [0008]

Claims (14)

A driver condition recognizing apparatus for recognizing a condition of a driver of a vehicle provided with an autonomous driving system, comprising: an image capture unit configured to capture an image captured by a camera configured to receive the driver; a hand detection unit configured to detect a driver's hand from the image acquired by the image capture unit; an item holding state detecting unit configured to detect a holding state of an item by the driver's hand detected by the hand detecting unit; and a standby determining unit configured to determine whether the driver's hand is in a state capable of directly operating a steering wheel of the vehicle during autonomous driving, based on the holding state of the object caused by the item holding state Detection unit is detected, wherein the article is an article which is different from an accessory of the vehicle which the driver operates by hand. Driver state recognizer behind Claim 1 wherein the item holding state detection unit detects a feature of a region portion moving with the hand of a driver from the image, and determines whether the driver is in a state of holding the item based on the feature of the region portion which is moved with the driver's hand and detected from the image. A driver condition detecting apparatus according to any one of the preceding claims, wherein when the item holding state detection unit detects a predetermined condition in which the driver does not hold the item in one hand, the standby determination unit determines that the driver is in the condition capable of being to directly operate the steering wheel during autonomous driving. Driver state recognizer behind Claim 1 or 2 wherein, when the item holding state detection unit detects a predetermined condition in which the driver holds the item with at least one hand, the standby determination unit determines that the driver is not in the condition to be able to directly apply the steering wheel during an autonomous Driving. The driver status recognizing apparatus according to claim 1, wherein when the item holding state detecting unit detects the holding condition of the item, the standby determination unit determines whether the driver's hand is in the condition capable of directly turning the steering wheel of the vehicle during an autonomous Driving, determined based on a size of the object. A driver condition detection apparatus according to any one of the preceding claims, further comprising: a steering wheel holding state detection unit configured to detect a holding state of the steering wheel by the driver; wherein the standby determination unit determines whether the driver is in the state capable of directly operating the steering wheel during autonomous driving, based on the holding state of the object detected by the object holding state detection unit and the holding state of the steering wheel, which is detected by the steering wheel holding state detection unit. Driver state recognizer behind Claim 6 further comprising: a sensor signal detection unit configured to acquire a sensor signal based on contact of the driver's hand from a sensor provided in the steering wheel, wherein the steering wheel holding state detection unit detects Hold state of the steering wheel detected by the driver based on the sensor signal, which is detected by the sensor signal detection unit. Driver state recognizer behind Claim 6 or 7 wherein the steering wheel holding state detection unit detects the holding state of the steering wheel by the driver from the image. A driver condition detection apparatus according to any one of the preceding claims, further comprising: a support execution unit configured to perform assistance to the driver when the standby determination unit determines that the driver's hand is not in the state capable of directly operating the steering wheel. Driver state recognizer behind Claim 9 further comprising: an information acquiring unit configured to acquire information that arises during autonomous driving from the autonomous driving system, wherein the assist execution unit performs negotiation processing on the driver based on a determination result from the standby determination unit and the information generated during autonomous driving detected by the information acquisition unit. Driver state recognizer behind Claim 10 wherein the information occurring during autonomous driving includes information for determining whether surroundings of the vehicle are in a safe state and when it is determined by the standby determination unit that the driver's hand is not in the state being able to directly operate the steering wheel, the assistance execution unit performs a negotiation processing after changing a communication level to prompt the driver to change a state of the hand based on whether the surroundings of the vehicle are in a safe State are. Driver state recognizer behind Claim 10 or 11 wherein the information that arises during autonomous driving includes takeover request information for taking manual driving from autonomous driving, and when determined by the standby determination unit that the driver's hand is not in the state capable to directly operate the steering wheel, and the information acquiring unit obtains the takeover request information, the assist execution unit performs a negotiation processing to cause the driver to take over the operation of the steering wheel. Driver status recognition system comprising: the driver condition recognizing apparatus according to any one of the preceding claims; and a camera configured to accommodate the driver. A driver state recognition method for recognizing a state of a driver of a vehicle provided with an autonomous driving system, comprising: a step of acquiring an image picked up by a camera configured to pick up the driver; a step of storing the acquired image in an image storage unit; a step of reading out the image from the image storage unit; a step of detecting a hand of the driver from the read image; a step of detecting a holding state of an object by the detected hand of the driver; and a step of determining whether the driver's hand is in a state capable of directly actuating the steering wheel of the vehicle during autonomous driving, based on the detected holding state of the object, wherein the article is an article which is different from an accessory of the vehicle which the driver operates by hand.

Images