JP2017049636A - Driver state detection device - Google Patents

Driver state detection device Download PDF

Info

Publication number
JP2017049636A
JP2017049636A JP2015170118A JP2015170118A JP2017049636A JP 2017049636 A JP2017049636 A JP 2017049636A JP 2015170118 A JP2015170118 A JP 2015170118A JP 2015170118 A JP2015170118 A JP 2015170118A JP 2017049636 A JP2017049636 A JP 2017049636A
Authority
JP
Japan
Prior art keywords
driver
state
consciousness
image
vehicle
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
JP2015170118A
Other languages
Japanese (ja)
Other versions
JP6256433B2 (en
Inventor
洋一朗 中畑
Yoichiro Nakahata
洋一朗 中畑
Original Assignee
マツダ株式会社
Mazda Motor Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by マツダ株式会社, Mazda Motor Corp filed Critical マツダ株式会社
Priority to JP2015170118A priority Critical patent/JP6256433B2/en
Publication of JP2017049636A publication Critical patent/JP2017049636A/en
Application granted granted Critical
Publication of JP6256433B2 publication Critical patent/JP6256433B2/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Abstract

PROBLEM TO BE SOLVED: To provide a driver state detection device capable of rapidly determining a consciousness loss state of a driver with a simple configuration by determining the consciousness loss state of the driver with a posture of the driver as a parameter to determine a consciousness state.SOLUTION: A driver state detection device 1 for detecting a consciousness loss state of a driver of a vehicle includes: a first camera 3 capable of imaging a head of a driver from the front of the driver; a second camera 4 capable of imaging the head of the driver from above the driver; and a state determination section 8c capable of determining a driver state on the basis of first and second captured images of the first and second cameras 3, 4. When both of the first and second captured images of the first and second cameras 3, 4 include the image of a face of the driver or when both of the first and second captured images of the first and second cameras 3, 4 do not include the image of the face of the driver, the state determination section 8c determines that the driver is in a consciousness loss state.SELECTED DRAWING: Figure 1

Description

本発明は、運転者状態検出装置に関し、特に車両の運転者の意識消失状態を検出する運転者状態検出装置に関する。   The present invention relates to a driver state detection device, and more particularly to a driver state detection device that detects a loss of consciousness of a driver of a vehicle.
従来より、運転者の状態、特に運転者の意識がどのような状態にあるのかを的確に把握し、運転者の意識状態に応じて運転操作を支援することが行われている。
特許文献1の車両の緊急退避装置は、運転者の生理状態を推定する生理状態推定部と、自動停車制御の実行開始指令を出力する運転者指示入力部と、運転者に対して自動停車制御の実行を確認する意識確認部と、自動停車を行う自動停車制御部とを備え、生理状態推定部による運転者の意識状態レベルに応じて運転者指示入力部の表示位置を変更し、運転者が意識消失状態のときには意識確認処理を省略して自動停車制御を実行する。
この生理状態推定部では、検出された運転者の脳波、心拍数、発汗量、血圧、呼気中の成分等の生体情報に基づいて運転者の意識状態を推定している。
Conventionally, a driver's state, in particular, a state in which the driver is conscious, is accurately grasped, and driving operation is supported according to the driver's conscious state.
The emergency evacuation device for a vehicle disclosed in Patent Document 1 includes a physiological state estimation unit that estimates a driver's physiological state, a driver instruction input unit that outputs an execution start command for automatic stop control, and automatic stop control for the driver. A consciousness confirmation unit that confirms the execution of the vehicle and an automatic stop control unit that automatically stops the vehicle. The display position of the driver instruction input unit is changed according to the driver's consciousness level by the physiological state estimation unit. When the vehicle is in the state of loss of consciousness, the automatic stop control is executed with the consciousness confirmation process omitted.
The physiological state estimation unit estimates the driver's consciousness state based on the detected biological information such as the brain wave, heart rate, sweating volume, blood pressure, and components in exhaled breath.
居眠り状態のような運転者の低覚醒状態を判定する場合、カメラ等の撮像手段によって運転者の正面から運転者の顔面を撮像し、この撮像画像に基づいて運転者の開眼度を検出することが行われている(特許文献2)。
また、車両走行中、低覚醒状態では修正舵角操作が多くなることに着目して、ステアリングホイールの修正操舵量や修正操舵回数を所定の判定値と比較し、実行された修正操舵量や修正操舵回数が判定値よりも大きい場合、運転者の覚醒度の低下を推定することが行われている(特許文献3)。
When determining a driver's low arousal state such as a dozing state, the driver's face is imaged from the front of the driver by an imaging means such as a camera, and the degree of eye opening of the driver is detected based on the captured image (Patent Document 2).
Also, paying attention to the fact that the corrected steering angle operation increases in the low awake state while the vehicle is running, the corrected steering amount of the steering wheel and the number of corrected steering operations are compared with a predetermined judgment value, and the corrected steering amount and correction executed. When the number of times of steering is larger than a determination value, a decrease in the driver's arousal level is estimated (Patent Document 3).
特許第5527411号公報Japanese Patent No. 5527411 特開2008−99884号公報JP 2008-99884 A 特開2011−39735号公報JP 2011-39735 A
特許文献1の車両の緊急退避装置は、運転者の脳波、心拍数、発汗量、血圧、呼気中の成分等の生体情報に基づいて運転者の意識状態を推定することができる。
しかし、特許文献1の車両の緊急退避装置では、複数の生体情報センサ、これらセンサの保持部やハーネス等が必要になることから、運転者による車両操作性が低下する虞がある。しかも、部品点数の増加等の面からも、一般車両に適用するには実用的でない。
The emergency evacuation device for a vehicle disclosed in Patent Literature 1 can estimate the driver's consciousness state based on biological information such as the driver's brain wave, heart rate, sweat rate, blood pressure, and components in exhalation.
However, since the vehicle emergency evacuation device disclosed in Patent Document 1 requires a plurality of biological information sensors, a holding unit for these sensors, a harness, and the like, the vehicle operability by the driver may be reduced. Moreover, it is not practical to apply to general vehicles from the viewpoint of increasing the number of parts.
ところで、運転者の意識状態レベルが低い低意識状態には、意識確認処理である警報(刺激)等によって覚醒状態に復帰可能な低覚醒状態と、運転者の意識の覚醒を促しても意識の覚醒が難しく運転者による運転操作が困難な意識消失状態とが存在している。
特許文献2の状態推定装置では、運転者の姿勢(座席に対する着座状態)によっては開眼度を検出できない虞がある。
つまり、運転操作が困難な意識消失状態のときには、運転者の顔面が必ずしもカメラに向かって正面を向いているとは限らず、両眼が撮像できない姿勢の場合には運転者の開眼度を検出することができない。
しかも、運転者が意識消失状態の場合、意識を覚醒させるために所定時間警報を行っても意識が覚醒されないため、自動停車等の運転支援の実行開始が遅くなる虞がある。
特許文献3の状態推定装置でも、運転者によって操作された修正舵角の確認によって覚醒度の低下を推定しているため、特許文献2と同様に、迅速な運転支援の実行を期待することができない。
By the way, in the low consciousness state where the driver's consciousness level is low, the low wakefulness state that can be returned to the awakening state by an alarm (stimulation) or the like that is a consciousness confirmation process, There is a state of loss of consciousness that is difficult to wake up and difficult for the driver to drive.
In the state estimation device of Patent Literature 2, there is a possibility that the degree of eye opening cannot be detected depending on the posture of the driver (sitting state with respect to the seat).
In other words, when the driver is in a state of loss of consciousness that is difficult to operate, the driver's face is not always facing the front toward the camera. Can not do it.
In addition, when the driver is in a state of loss of consciousness, even if an alarm is given for a predetermined time in order to wake up the consciousness, the consciousness is not awakened, so that the start of execution of driving assistance such as automatic stopping may be delayed.
Even in the state estimation device of Patent Document 3, since the decrease in the arousal level is estimated by confirming the corrected steering angle operated by the driver, similar to Patent Document 2, it can be expected to promptly perform driving assistance. Can not.
そこで、本発明者が運転席に着座した運転者の姿勢と運転者の意識状態レベルの関係を検証した結果、運転者による運転操作が困難な意識消失状態のとき、正常状態に対して極端な前傾姿勢又は後傾姿勢になることが知見された。
これは、シートベルト着用時には運転者が運転席に対して拘束されるため、シートベルトによる運転席への押付け力によって意識消失状態の運転者の姿勢が主に前述の2形態に特定されるものと推測される。
Therefore, as a result of verifying the relationship between the posture of the driver seated in the driver's seat and the driver's consciousness state level, the present inventor is extremely dissatisfied with the normal state when the driver's driving operation is difficult. It has been found that the posture is a forward leaning posture or a backward leaning posture.
This is because the driver is restrained with respect to the driver's seat when the seat belt is worn, and thus the posture of the driver who has lost his or her consciousness is mainly specified in the two forms described above by the pressing force of the seat belt on the driver's seat. It is guessed.
本発明の目的は、簡単な構成で運転者の意識消失状態を迅速に判定することができる運転者状態検出装置等を提供することである。   An object of the present invention is to provide a driver state detection device or the like that can quickly determine a driver's loss of consciousness state with a simple configuration.
請求項1の運転者状態検出装置は、車両の運転者の意識消失状態を検出する運転者状態検出装置において、運転者の正面から運転者頭部を撮像可能な第1撮像手段と、運転者の上方から運転者頭部を撮像可能な第2撮像手段と、前記第1,第2撮像手段の撮像画像に基づいて運転者状態を判定可能な状態判定手段とを備え、前記状態判定手段は、前記第1,第2撮像手段の撮像画像が双方とも運転者の顔面の画像を含むとき、又は第1,第2撮像手段の撮像画像が双方とも運転者の顔面の画像を含まないとき、運転者が意識消失状態であると判定することを特徴としている。   The driver state detection device according to claim 1 is a driver state detection device that detects a loss of consciousness of the driver of the vehicle. The first imaging means capable of imaging the driver's head from the front of the driver, and the driver A second imaging unit capable of imaging the driver's head from above, and a state determination unit capable of determining the driver state based on the captured images of the first and second imaging units, the state determination unit comprising: When the captured images of the first and second imaging means both include an image of the driver's face, or when the captured images of the first and second imaging means do not include the image of the driver's face, It is characterized by determining that the driver is in an unconscious state.
この運転者状態検出装置では、運転者の正面から運転者頭部を撮像可能な第1撮像手段と、運転者の上方から運転者頭部を撮像可能な第2撮像手段と、第1,第2撮像手段の撮像画像に基づいて運転者状態を判定可能な状態判定手段とを備えるため、運転者の顔面が指向している顔面方向を最少限の撮像手段で判定することができ、この顔面方向に基いて運転者の前傾姿勢又は後傾姿勢を判定することができる。
状態判定手段が、第1,第2撮像手段の撮像画像が双方とも運転者の顔面の画像を含むとき、又は第1,第2撮像手段の撮像画像が双方とも運転者の顔面の画像を含まないとき、運転者が意識消失状態であると判定するため、運転者の実際の姿勢に基いて運転者の運転操作が困難な意識消失状態を判定することができる。
In this driver state detection device, first imaging means capable of imaging the driver's head from the front of the driver, second imaging means capable of imaging the driver's head from above the driver, first and first 2 is provided with a state determination unit capable of determining the driver state based on the captured image of the image pickup unit, so that the face direction in which the driver's face is directed can be determined by the minimum number of image pickup units. Based on the direction, the forward leaning posture or the backward leaning posture of the driver can be determined.
When the state determination means includes both the captured images of the first and second imaging means including the driver's face image, or both the captured images of the first and second imaging means include the driver's face image. When it is not, since it is determined that the driver is in an unconscious state, it is possible to determine an unconscious state in which it is difficult for the driver to drive based on the actual posture of the driver.
請求項2の発明は、請求項1の発明において、前記状態判定手段は、前記第1撮像手段の撮像画像が運転者の顔面の画像を含み且つ前記第2撮像手段の撮像画像が運転者の顔面の画像を含まないとき、運転者が意識覚醒状態であると判定することを特徴としている。
この構成によれば、運転者の意識覚醒状態を簡単な構成で精度良く判定することができる。
According to a second aspect of the present invention, in the first aspect of the invention, the state determination unit includes a captured image of the first imaging unit including an image of a driver's face, and a captured image of the second imaging unit is a driver's face. When the image of the face is not included, the driver is determined to be in a conscious awakening state.
According to this configuration, the driver's consciousness awakening state can be accurately determined with a simple configuration.
請求項3の発明は、請求項1又は2の発明において、前記状態判定手段は、運転者が意識消失状態であると判定したとき、車両を緊急停車させることを特徴としている。
この構成によれば、運転者が意識消失状態であっても、車両を安全に停車させることができる。
A third aspect of the invention is characterized in that, in the first or second aspect of the invention, the state determining means causes the vehicle to stop urgently when it is determined that the driver is in an unconscious state.
According to this configuration, the vehicle can be safely stopped even when the driver is in an unconscious state.
請求項4の発明は、請求項1〜3の何れか1項の発明において、前記状態判定手段は、運転者が意識消失状態であると判定したとき、緊急異常状態を外部に通報することを特徴としている。
この構成によれば、運転者が意識消失状態であっても、緊急異常状態を外部に通報することによって運転者を救助することができる。
According to a fourth aspect of the present invention, in the invention according to any one of the first to third aspects, when the state determination means determines that the driver is in an unconscious state, the emergency determination state is notified to the outside. It is a feature.
According to this configuration, even when the driver is in an unconscious state, the driver can be rescued by reporting the emergency abnormal state to the outside.
本発明の運転者状態検出装置によれば、運転者の姿勢を意識状態判定のパラメータとして運転者の意識消失状態を判定することにより、簡単な構成で運転者の意識消失状態を迅速に判定することができる。   According to the driver state detection device of the present invention, by determining the driver's loss of consciousness state using the driver's posture as a consciousness state determination parameter, the driver's loss of consciousness state can be quickly determined with a simple configuration. be able to.
実施例1に係る運転者状態検出装置のブロック図である。1 is a block diagram of a driver state detection device according to a first embodiment. 意識覚醒状態の運転者の着座モデルを示し、(a)は正面図、(b)は側面図、(c)は平面図である。The driver's seating model in a conscious awakening state is shown, (a) is a front view, (b) is a side view, and (c) is a plan view. 第1意識消失状態の運転者の着座モデルを示し、(a)は正面図、(b)は側面図、(c)は平面図である。The driver's seating model of a 1st consciousness loss state is shown, (a) is a front view, (b) is a side view, (c) is a top view. 第2意識消失状態の運転者の着座モデルを示し、(a)は正面図、(b)は側面図、(c)は平面図である。The seating model of the driver | operator in the 2nd consciousness loss state is shown, (a) is a front view, (b) is a side view, (c) is a top view. 第1,第2カメラによって撮像された意識覚醒状態の運転者の撮像画像を示し、(a)は第1撮像画像、(b)は第2撮像画像の例である。The captured image of the driver | operator of the conscious awakening imaged with the 1st, 2nd camera is shown, (a) is a 1st captured image, (b) is an example of a 2nd captured image. 第1,第2カメラによって撮像された第1意識消失状態の運転者の撮像画像を示し、(a)は第1撮像画像、(b)は第2撮像画像の例である。The captured image of the driver | operator of the 1st loss of consciousness imaged with the 1st, 2nd camera is shown, (a) is a 1st captured image, (b) is an example of a 2nd captured image. 第1,第2カメラによって撮像された第2意識消失状態の運転者の撮像画像を示し、(a)は第1撮像画像、(b)は第2撮像画像の例である。The captured image of the driver | operator of the 2nd loss of consciousness imaged with the 1st, 2nd camera is shown, (a) is a 1st captured image, (b) is an example of a 2nd captured image. 運転者状態検出処理を示すフローチャートである。It is a flowchart which shows a driver | operator state detection process. 運転者状態判定処理を示すフローチャートである。It is a flowchart which shows a driver | operator state determination process.
以下、本発明の実施形態を図面に基づいて詳細に説明する。
以下の説明は、本発明を車両に適用したものを例示したものであり、本発明、その適用物、或いは、その用途を制限するものではない。
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
The following description exemplifies a case where the present invention is applied to a vehicle, and does not limit the present invention, its application, or its use.
以下、本発明の実施例1について図1〜図9に基づいて説明する。
運転者状態検出装置1は、運転者の意識状態、特に運転者による車両(自車両ともいう)の運転操作が困難な意識消失状態を判定可能な運転支援装置である。
図1に示すように、運転者状態検出装置1は、車両前方を含む周辺風景を撮像する車外カメラ2と、シートベルトを装着して運転席に着座した運転者の頭部を夫々撮像する車内カメラとしての第1,第2カメラ3,4と、ナビゲーションシステム5と、運転制御部6と、通信部7と、ECU(Electronic Control Unit)8等を備えている。
Embodiment 1 of the present invention will be described below with reference to FIGS.
The driver state detection device 1 is a driving support device that can determine a driver's consciousness state, particularly a loss of consciousness state in which it is difficult for the driver to drive the vehicle (also referred to as the host vehicle).
As shown in FIG. 1, the driver state detection device 1 includes a vehicle exterior camera 2 that captures a surrounding landscape including the front of the vehicle, and a vehicle interior that captures the head of the driver seated in the driver seat while wearing a seat belt. The camera includes first and second cameras 3 and 4, a navigation system 5, an operation control unit 6, a communication unit 7, an ECU (Electronic Control Unit) 8, and the like.
車外カメラ2は、例えばルーフパネル前端側下面のバックミラー(図示略)近傍位置に装着され、フロントウインドガラス(図示略)を介して車体前方を静止画または動画で撮像可能なCCD(Charge Coupled Device)カメラによって構成されている。   The outside camera 2 is mounted in the vicinity of a rearview mirror (not shown) on the lower surface of the front end side of the roof panel, for example, and can be a CCD (Charge Coupled Device) that can capture the front of the vehicle body as a still image or a moving image through a front window glass (not shown). ) Consists of cameras.
第1カメラ3は、CCDカメラによって構成され、例えばインスツルメントパネル(図示略)上部の運転者対向位置に装着されている。図2〜図4に示すように、第1カメラ3は、運転席に着座してシートベルトを着用した運転者の顔面を含む頭部を前方正面から撮像可能に形成され、撮像された頭部画像は、運転者の顔面部分の画像を切り出して、顔面の指向方向、運転者の表情の特定、目のアイリス(虹彩)を拡大検出して瞳孔寸法や視線方向の特定等を実行可能に構成されている。   The first camera 3 is constituted by a CCD camera, and is mounted, for example, at a driver-facing position above an instrument panel (not shown). As shown in FIGS. 2 to 4, the first camera 3 is formed such that the head including the face of the driver who is seated in the driver's seat and wears the seat belt can be imaged from the front in front. The image is configured so that the image of the driver's face can be cut out and the orientation of the face, identification of the driver's facial expression, the iris of the eye (iris) can be enlarged and the pupil size and gaze direction can be specified, etc. Has been.
第2カメラ4は、CCDカメラによって構成され、例えば運転席の真上のルーフパネル(図示略)に装着されている。図2〜図4に示すように、第2カメラ4は、運転席に着座してシートベルトを着用した運転者の顔面を含む頭部上面を真上から撮像可能に形成され、撮像された頭部画像は、運転者の顔面部分と顔面以外の部分とを識別可能に構成されている。   The second camera 4 is constituted by a CCD camera, and is mounted on, for example, a roof panel (not shown) directly above the driver's seat. As shown in FIGS. 2 to 4, the second camera 4 is formed so that the upper surface of the head including the face of the driver sitting on the driver's seat and wearing the seat belt can be imaged from directly above, and the imaged head The partial image is configured to be able to identify the face portion of the driver and the portion other than the face.
ナビゲーションシステム5は、車両の経路案内を行うシステムである。
ナビゲーションシステム5は、外部の情報センタ9を介して複数のGPS衛星からの位置信号を受信することで自車両の現在位置情報を検出可能に構成されている。
このナビゲーションシステム5は、道路地図データを記憶した地図データベースと、交通規則データを記憶した交通規則データベース(何れも図示略)等を備えている。
これにより、ナビゲーションシステム5は、自車両の現在位置データ、地図データベースの道路地図データ及び交通規則データベースの交通規則データを利用して運転者に目的地までの経路案内を行う。
The navigation system 5 is a system that provides vehicle route guidance.
The navigation system 5 is configured to be able to detect current position information of the host vehicle by receiving position signals from a plurality of GPS satellites via an external information center 9.
The navigation system 5 includes a map database that stores road map data, a traffic rule database that stores traffic rule data (both not shown), and the like.
Thereby, the navigation system 5 performs route guidance to the driver to the destination using the current position data of the own vehicle, the road map data of the map database, and the traffic rule data of the traffic rule database.
運転制御部6は、自車両の走行制御を実行するための駆動機構や操舵機構である。
この運転制御部6は、エンジン制御部、ステアリングアクチュエータ、ブレーキアクチュエータ、及びシフトアクチュエータ等によって構成されている。
運転制御部6は、ECU8から緊急指令が入力されたとき、自車両の走行制御を実行する。この走行制御は、例えば、緊急停車や自動運転による道路端縁側等の安全地帯への緊急避難等である。自動運転は、先行車両、対向車両、走行車線等の検出情報に基づいて実行される公知の技術であるため、詳細な説明を省略する。
The driving control unit 6 is a drive mechanism or a steering mechanism for executing traveling control of the host vehicle.
The operation control unit 6 includes an engine control unit, a steering actuator, a brake actuator, a shift actuator, and the like.
When an emergency command is input from the ECU 8, the operation control unit 6 executes traveling control of the host vehicle. This traveling control is, for example, emergency evacuation to a safety zone such as a road edge side by an emergency stop or automatic driving. Since the automatic driving is a known technique that is executed based on detection information of the preceding vehicle, the oncoming vehicle, the traveling lane, and the like, detailed description thereof is omitted.
通信部7は、アンテナ(図示略)等を備え、情報センタ9と送受信可能に構成されている。この通信部7は、ECU8から緊急指令が入力されたとき、緊急異常状態を通知するための緊急救助信号と、自車両の車両IDと、自車両位置情報と、運転者の姿勢状態(前傾姿勢のうつ伏せ状態又は後傾姿勢の仰向け状態)情報等を情報センタ9に緊急通報する。   The communication unit 7 includes an antenna (not shown) and the like, and is configured to be able to transmit and receive with the information center 9. When an emergency command is input from the ECU 8, the communication unit 7 transmits an emergency rescue signal for notifying an emergency abnormal state, the vehicle ID of the host vehicle, the host vehicle position information, and the driver's posture state (forward leaning). The information center 9 is urgently notified of information such as the prone state of the posture or the back-facing state of the tilted posture.
次に、ECU8について説明する。
図1に示すように、ECU8は、画像処理部8aと、運転者の感情を判定する感情推定部8bと、運転者の意識消失状態を判定する状態判定部8c等を備えている。
ECU8は、CPU、ROM、RAM等からなる電子制御ユニットであり、ROMに記憶されているアプリケーションプログラムをRAMにロードし、CPUで実行することにより各種演算処理を行っている。
Next, the ECU 8 will be described.
As shown in FIG. 1, the ECU 8 includes an image processing unit 8a, an emotion estimation unit 8b that determines the driver's emotion, a state determination unit 8c that determines the driver's loss of consciousness, and the like.
The ECU 8 is an electronic control unit including a CPU, a ROM, a RAM, and the like, and performs various arithmetic processes by loading an application program stored in the ROM into the RAM and executing it by the CPU.
画像処理部8aは、車外カメラ2及び第1,第2カメラ3,4によって撮像された画像データの画像処理、画像データの一時記憶、画像データの圧縮・伸張の処理、複数の撮像画像を合成可能に構成されている。
車外カメラ21の撮像画像によって、自車両に先行する先行車両、反対車線を走行する対向車両、自車両が走行する走行車線及び路側帯等を検出している。
先行車両及び対向車両は、予め格納された車両判定用テンプレートとパターンマッチングすることにより抽出する。走行車線及び路側帯の白線部分は、前方画像の輝度を水平方向に微分処理することにより、白線の両端部に高周波成分となるエッジが発生することを利用して走行車線の白線部分を推定し、この推定された白線部分について輝度及び路面とのコントラストから定められる閾値や白線幅の閾値等に基づいて白線を抽出している。
The image processing unit 8a performs image processing of image data captured by the outside camera 2 and the first and second cameras 3 and 4, temporary storage of image data, compression / decompression processing of image data, and synthesis of a plurality of captured images. It is configured to be possible.
Based on the captured image of the camera 21 outside the vehicle, a preceding vehicle preceding the own vehicle, an oncoming vehicle traveling in the opposite lane, a traveling lane in which the own vehicle is traveling, a roadside zone, and the like are detected.
The preceding vehicle and the oncoming vehicle are extracted by pattern matching with a vehicle determination template stored in advance. The white line part of the driving lane and the roadside band is estimated by using the fact that edges that are high frequency components are generated at both ends of the white line by differentiating the brightness of the front image in the horizontal direction. The white line is extracted from the estimated white line portion based on the threshold value determined from the brightness and the contrast with the road surface, the threshold value of the white line width, and the like.
画像処理部8aは、第1カメラ3による第1撮像画像に基づき運転席に着座した運転者の頭部及び運転者の顔面を検出し、第1画像データ毎に顔面認識処理を行っている。
顔面認識は、例えば目の位置、鼻の位置、口の位置等の認識及びこれらの位置関係等を認識し、これらの数値データを予め格納された顔判定データと比較して運転者の顔認識を行う。更に、第1カメラ3で顔認識された第1画像データに基いて、視線方向変化数や瞳孔径が検出される。
画像処理部8aは、第2カメラ4による第2撮像画像に基づき運転席に着座した運転者の頭部及び運転者の顔面を検出し、第2画像データ毎に顔面認識処理を行っている。
顔面認識は、例えば目の位置、鼻の位置、口の位置等の認識及びこれらの位置関係等を認識し、これらの数値データを予め格納された顔判定データと比較して運転者の顔認識を行う。
The image processing unit 8a detects the driver's head seated on the driver's seat and the driver's face based on the first captured image from the first camera 3, and performs face recognition processing for each first image data.
For example, the face recognition recognizes the position of the eyes, the position of the nose, the position of the mouth, and the positional relationship thereof, and compares the numerical data with the face determination data stored in advance to recognize the face of the driver. I do. Further, based on the first image data whose face is recognized by the first camera 3, the number of gaze direction changes and the pupil diameter are detected.
The image processing unit 8a detects the driver's head and the driver's face seated in the driver's seat based on the second image captured by the second camera 4, and performs face recognition processing for each second image data.
For example, the face recognition recognizes the position of the eyes, the position of the nose, the position of the mouth, and the positional relationship thereof, and compares the numerical data with the face determination data stored in advance to recognize the face of the driver. I do.
感情推定部8bは、第1カメラ3による顔画像データに基いて、運転者の現在の感情を判定している。運転者の各感情に対応して順序付けされた表情マスタ画像が予め準備され、顔画像データと表情マスタ画像とを比較して特定の表情マスタ画像と適合したとき、運転者の感情が特定の感情であることを推定している。この推定された運転者の感情は、視線方向変化数や瞳孔径によって補正される。
運転者感情の判定結果に基いて、運転者の感情が悪化している場合には、運転者の感情を改善する方向に各種車載機器を制御している(特願2015−15334号参照)。
The emotion estimation unit 8b determines the current emotion of the driver based on the face image data from the first camera 3. A facial expression master image ordered corresponding to each emotion of the driver is prepared in advance, and when the facial image data and the facial expression master image are compared and matched with a specific facial expression master image, the driver's emotion is the specific emotion It is estimated that. The estimated driver's feeling is corrected by the number of gaze direction changes and the pupil diameter.
When the driver's emotion is deteriorated based on the determination result of the driver's emotion, various in-vehicle devices are controlled in a direction to improve the driver's emotion (see Japanese Patent Application No. 2015-15334).
状態判定部8cは、第1,第2カメラ3,4の顔認識結果に基いて運転者による車両の運転操作が困難な意識消失状態を判定している。
この状態判定部8cは、第1撮像画像が運転者の顔面の画像を含み且つ第2撮像画像が運転者の顔面画像を含まないとき、正常に車両の運転操作が可能な意識覚醒状態、第1,第2撮像画像が双方とも運転者の顔面画像を含まないとき、運転者が前傾姿勢である第1意識消失状態、第1,第2撮像画像が双方とも運転者の顔面画像を含むとき、運転者が後傾姿勢である第2意識消失状態を判定し、第1,第2意識消失状態のうち何れかの意識消失状態が判定されたとき、運転制御部6及び通信部7に緊急指令を出力する。
The state determination unit 8c determines a loss of consciousness state in which it is difficult for the driver to drive the vehicle based on the face recognition results of the first and second cameras 3 and 4.
When the first captured image includes the driver's facial image and the second captured image does not include the driver's facial image, the state determination unit 8c When both the first and second captured images do not include the driver's facial image, the driver is in a first consciousness state in which the driver is leaning forward, and both the first and second captured images include the driver's facial image. When the driver determines the second state of loss of consciousness in a backward tilted posture, and when one of the first and second consciousness loss states is determined, the driving control unit 6 and the communication unit 7 An emergency command is output.
ここで、図2〜図7に基いて、乗車中の運転者の意識状態について説明する。
運転者の意識状態レベルには、正常に車両の運転操作が可能な意識覚醒状態と、警報等の刺激によって意識覚醒状態に復帰可能な低覚醒状態と、意識覚醒を促しても意識の覚醒が難しく運転者による運転操作が困難な意識消失状態とが存在している。
図2(a)〜図2(c)に示すように、運転者が意識覚醒状態のとき、運転者は車両進行方向に視線を指向させている。また、眠気による低覚醒状態であっても、極端な前傾姿勢或いは後傾姿勢にはならないため、意識覚醒状態と略同様の運転者姿勢に分類することができる。このとき、図5(a)に示すように、第1撮像画像には運転者の顔面が認識され、図5(b)に示すように、第2撮像画像には運転者の頭部上面が認識されている。
Here, based on FIGS. 2-7, the conscious state of the driver | operator in boarding is demonstrated.
The driver's consciousness level includes the consciousness-awakening state in which the vehicle can be driven normally, the low-awakening state in which the vehicle can return to the consciousness-awakening state by stimulating alarms, etc. There is a state of loss of consciousness that is difficult and difficult for the driver to drive.
As shown in FIGS. 2 (a) to 2 (c), when the driver is in a consciousness awakening state, the driver directs his / her line of sight in the vehicle traveling direction. In addition, even in a low arousal state due to sleepiness, since it does not become an extreme forward leaning posture or a backward leaning posture, it can be classified into a driver posture substantially similar to the conscious awakening state. At this time, as shown in FIG. 5A, the driver's face is recognized in the first captured image, and as shown in FIG. 5B, the driver's head upper surface is displayed in the second captured image. Recognized.
図3(a)〜図3(c)に示すように、運転者が第1意識消失状態のとき、運転者は頭部をステアリングホイールに持たれ掛けたうつ伏せ状の極端な前傾姿勢になる。このとき、図6(a)に示すように、第1撮像画像には運転者の頭部上面が認識され、図6(b)に示すように、第2撮像画像には運転者の頭部後面が認識されている。
また、図4(a)〜図4(c)に示すように、運転者が第2意識消失状態のとき、運転者は腰が前方に移行した仰向け状の極端な後傾姿勢になる。このとき、図7(a)に示すように、第1撮像画像には運転者の顔面が認識され、図7(b)に示すように、第2撮像画像には運転者の顔面が認識されている。
これにより、運転者の姿勢、換言すれば運転者の顔面が指向する方向を指標として運転者の意識消失状態を的確に判定することができる。
As shown in FIGS. 3 (a) to 3 (c), when the driver is in the first loss of consciousness, the driver is in a prone and extremely leaning posture with his head held on the steering wheel. . At this time, as shown in FIG. 6A, the upper surface of the driver's head is recognized in the first captured image, and as shown in FIG. 6B, the driver's head is displayed in the second captured image. The back is recognized.
Further, as shown in FIGS. 4A to 4C, when the driver is in the second consciousness loss state, the driver assumes an extremely backward tilted posture in which the waist is shifted forward. At this time, as shown in FIG. 7A, the driver's face is recognized in the first captured image, and as shown in FIG. 7B, the driver's face is recognized in the second captured image. ing.
Thus, the driver's consciousness loss state can be accurately determined using the driver's posture, in other words, the direction in which the driver's face is directed as an index.
次に、図8,図9のフローチャートに基づいて、運転者状態検出処理について説明する。尚、Si(i=1,2…)は、各処理のためのステップを示す。
まず、図8に示すように運転者状態検出装置1では、運転席着座センサやシートベルト装着センサ等の各種機器からの出力を入力し(S1)、運転者の運転席着座とシートベルト装着を条件としてS2へ移行する。
S2では、運転者状態判定処理を行っている。運転者状態判定処理では、運転者の意識状態が、意識覚醒状態、第1意識消失状態、第2意識消失状態のうちの何れの意識消失状態か判定する。
Next, the driver state detection process will be described based on the flowcharts of FIGS. Si (i = 1, 2,...) Indicates a step for each process.
First, as shown in FIG. 8, in the driver state detection device 1, outputs from various devices such as a driver seat seating sensor and a seat belt mounting sensor are input (S1), and the driver seat seating and seat belt mounting are performed. The process proceeds to S2 as a condition.
In S2, a driver state determination process is performed. In the driver state determination process, it is determined whether the driver's consciousness state is any of the consciousness awakening state, the first consciousness loss state, or the second consciousness loss state.
運転者状態判定処理の後、S3に移行して、運転者の意識状態が第1意識消失状態又は第2意識消失状態か否か判定する。
S3の判定の結果、運転者の意識状態が第1意識消失状態又は第2意識消失状態の場合、S4に移行し、車両を緊急停車させる。S3の判定の結果、運転者の意識状態が第1意識消失状態又は第2意識消失状態ではない場合、リターンする。
車両の緊急停車の後、S5に移行して、情報センタ9に救助要請を通報してリターンする。
After the driver state determination process, the process proceeds to S3 to determine whether the driver's consciousness state is the first consciousness loss state or the second consciousness loss state.
As a result of the determination in S3, when the driver's consciousness state is the first consciousness loss state or the second consciousness loss state, the process proceeds to S4 to stop the vehicle urgently. As a result of the determination in S3, if the driver's consciousness state is not the first consciousness loss state or the second consciousness loss state, the process returns.
After the emergency stop of the vehicle, the process proceeds to S5, a rescue request is notified to the information center 9, and the process returns.
次に、運転者状態判定処理について説明する。
図9のフローチャートに示すように、第1,第2カメラ3,4によって着座状態の運転者の頭部を夫々撮像し(S10)、S11に移行する。
S11では、第1カメラ3で撮像した第1撮像画像に運転者の頭部が認識されるか、換言すれば運転者の頭部が存在しているか否か判定する。
S11の判定の結果、第1撮像画像に運転者の頭部が存在している場合、S12に移行して、第1撮像画像に運転者の顔面が存在しているか否かを判定する。
Next, the driver state determination process will be described.
As shown in the flowchart of FIG. 9, the heads of the seated drivers are respectively imaged by the first and second cameras 3 and 4 (S10), and the process proceeds to S11.
In S11, it is determined whether or not the driver's head is recognized in the first captured image captured by the first camera 3, in other words, whether or not the driver's head is present.
As a result of the determination in S11, when the driver's head is present in the first captured image, the process proceeds to S12 to determine whether the driver's face is present in the first captured image.
S12の判定の結果、第1撮像画像に運転者の顔面が存在している場合、S13に移行して、第2カメラ4で撮像した第2撮像画像に運転者の顔面が存在しているか否かを判定する。
S13の判定の結果、第2撮像画像に運転者の顔面が存在している場合、運転者が図4及び図7に示す極端な後傾姿勢であるため、S14に移行して、第2意識消失状態を判定して終了する。
S13の判定の結果、第2撮像画像に運転者の顔面が存在していない(頭部が存在する)場合、運転者が図2及び図5に示す正常姿勢であるため、S15に移行して、意識覚醒状態を判定して終了する。
As a result of the determination in S12, if the driver's face is present in the first captured image, the process proceeds to S13, and whether the driver's face is present in the second captured image captured by the second camera 4 or not. Determine whether.
As a result of the determination in S13, when the driver's face is present in the second captured image, the driver is in the extremely backward tilted posture shown in FIGS. Determine the disappearance state and end.
As a result of the determination in S13, when the driver's face is not present in the second captured image (the head is present), the driver is in the normal posture shown in FIG. 2 and FIG. Then, the state of conscious awakening is determined and the process ends.
S12の判定の結果、第1撮像画像に運転者の顔面が存在していない(頭部が存在する)場合、S16に移行して、第2カメラ4で撮像した第2撮像画像に運転者の顔面が存在しているか否かを判定する。
S16の判定の結果、第2撮像画像に運転者の顔面が存在している場合、運転者が特異な姿勢であるため、S10にリターンする。
S16の判定の結果、第2撮像画像に運転者の顔面が存在していない(頭部が存在する)場合、運転者が図3及び図6に示す極端な前傾姿勢であるため、S17に移行して、第1意識消失状態を判定して終了する。
As a result of the determination in S12, when the driver's face is not present in the first captured image (the head is present), the process proceeds to S16, and the driver's face is captured in the second captured image captured by the second camera 4. It is determined whether or not a face exists.
As a result of the determination in S16, if the driver's face is present in the second captured image, the driver returns to S10 because the driver is in a unique posture.
If the driver's face is not present in the second captured image (the head is present) as a result of the determination in S16, the driver is in the extreme forward leaning posture shown in FIG. 3 and FIG. It shifts, it determines with a 1st consciousness loss state, and complete | finishes.
次に、上記運転者状態検出装置1の作用、効果について説明する。
本運転者状態検出装置1によれば、運転者の正面から運転者頭部を撮像可能な第1カメラ3と、運転者の上方から運転者頭部を撮像可能な第2カメラ4と、第1,第2カメラ3,4の第1,第2撮像画像に基づいて運転者状態を判定可能な状態判定部8cとを備えるため、運転者の顔面が指向している顔面方向を最少限の第1,第2カメラ3,4で判定することができ、この顔面方向に基いて運転者の前傾姿勢又は後傾姿勢を判定することができる。状態判定部8cが、第1,第2カメラ3,4の第1,第2撮像画像が双方とも運転者の顔面の画像を含むとき、又は第1,第2撮像手段の第1,第2撮像画像が双方とも運転者の顔面の画像を含まないとき、運転者が意識消失状態であると判定するため、運転者の実際の姿勢に基いて運転者の運転操作が困難な意識消失状態を判定することができる。
Next, the operation and effect of the driver state detection device 1 will be described.
According to the driver state detection device 1, the first camera 3 that can image the driver's head from the front of the driver, the second camera 4 that can image the driver's head from above the driver, 1 and a state determination unit 8c capable of determining the driver state based on the first and second captured images of the second cameras 3 and 4, so that the face direction to which the driver's face is directed is minimized. The determination can be made by the first and second cameras 3 and 4, and the forward leaning posture or the backward leaning posture of the driver can be determined based on the face direction. When the state determination unit 8c includes both the first and second captured images of the first and second cameras 3 and 4 including the image of the driver's face, or the first and second of the first and second imaging means. When both the captured images do not include the image of the driver's face, it is determined that the driver is in a state of loss of consciousness. Can be determined.
状態判定部8cは、第1カメラ3の第1撮像画像が運転者の顔面の画像を含み且つ第2カメラ4の第2撮像画像が運転者の顔面の画像を含まないとき、運転者が意識覚醒状態であると判定するため、運転者の意識覚醒状態を簡単な構成で精度良く判定することができる。   When the first captured image of the first camera 3 includes an image of the driver's face and the second captured image of the second camera 4 does not include the image of the driver's face, the state determination unit 8c recognizes the driver. Since it is determined that the state is an awakening state, the driver's awareness state can be accurately determined with a simple configuration.
状態判定部8cは、運転者が意識消失状態であると判定したとき、車両を緊急停車させるため、運転者が意識消失状態であっても、車両を安全に停車させることができる。   When the driver determines that the driver is in an unconscious state, the state determiner 8c stops the vehicle urgently, so that the vehicle can be stopped safely even if the driver is in an unconscious state.
状態判定部8cは、運転者が意識消失状態であると判定したとき、緊急異常状態を外部に通報するため、運転者が意識消失状態であっても、緊急異常状態を外部に通報することによって運転者を救助することができる。   When the driver determines that the driver is in a state of loss of consciousness, the state determination unit 8c reports the emergency abnormal state to the outside. The driver can be rescued.
次に、前記実施形態を部分的に変更した変形例について説明する。
1〕前記実施形態においては、運転者の表情を検出するため、第2カメラよりも高性能な第1カメラを運転者の感情推定に兼用した例を説明したが、運転者の感情推定機能を省略することで、第1,第2カメラを共に顔面認識のみ実行可能な機能にしてコストを抑えても良い。
Next, a modified example in which the embodiment is partially changed will be described.
1) In the above-described embodiment, the example in which the first camera having a higher performance than the second camera is used for estimating the driver's emotion in order to detect the driver's facial expression has been described. By omitting, it is possible to reduce the cost by making both the first and second cameras functions to perform only face recognition.
2〕前記実施形態においては、低覚醒状態を意識覚醒状態に含めた例を説明したが、意識覚醒状態と低覚醒状態とを区分し、低覚醒状態を検出したとき、運転者の意識覚醒を促す意識覚醒機構と組み合わせて用いることも可能である。 2) In the above embodiment, the example in which the low wakefulness state is included in the consciousness wakefulness state has been described. However, when the consciousness wakefulness state and the low wakefulness state are separated and the low wakefulness state is detected, the driver's consciousness wakefulness is detected. It can also be used in combination with a conscious awakening mechanism.
3〕その他、当業者であれば、本発明の趣旨を逸脱することなく、前記実施形態に種々の変更を付加した形態で実施可能であり、本発明はそのような変更形態も包含するものである。 3) In addition, those skilled in the art can implement the present invention with various modifications added without departing from the spirit of the present invention, and the present invention includes such modifications. is there.
1 運転者状態検出装置
3 第1カメラ
4 第2カメラ
8 ECU
8c 状態判定部
DESCRIPTION OF SYMBOLS 1 Driver state detection apparatus 3 1st camera 4 2nd camera 8 ECU
8c State determination unit

Claims (4)

  1. 車両の運転者の意識消失状態を検出する運転者状態検出装置において、
    運転者の正面から運転者頭部を撮像可能な第1撮像手段と、
    運転者の上方から運転者頭部を撮像可能な第2撮像手段と、
    前記第1,第2撮像手段の撮像画像に基づいて運転者状態を判定可能な状態判定手段とを備え、
    前記状態判定手段は、前記第1,第2撮像手段の撮像画像が双方とも運転者の顔面の画像を含むとき、又は第1,第2撮像手段の撮像画像が双方とも運転者の顔面の画像を含まないとき、運転者が意識消失状態であると判定することを特徴とする運転者状態検出装置。
    In the driver state detection device that detects the loss of consciousness of the driver of the vehicle,
    First imaging means capable of imaging the driver's head from the front of the driver;
    A second imaging means capable of imaging the driver's head from above the driver;
    A state determination unit capable of determining a driver state based on the captured images of the first and second imaging units;
    The state determination means is configured such that when both the captured images of the first and second imaging means include an image of the driver's face, or both the captured images of the first and second imaging means are images of the driver's face. When the vehicle is not included, it is determined that the driver is in a state of loss of consciousness.
  2. 前記状態判定手段は、前記第1撮像手段の撮像画像が運転者の顔面の画像を含み且つ前記第2撮像手段の撮像画像が運転者の顔面の画像を含まないとき、運転者が意識覚醒状態であると判定することを特徴とする請求項1に記載の運転者状態検出装置。   When the captured image of the first imaging unit includes an image of the driver's face and the captured image of the second imaging unit does not include the image of the driver's face, the state determination unit The driver state detection device according to claim 1, wherein the driver state detection device is determined to be.
  3. 前記状態判定手段は、運転者が意識消失状態であると判定したとき、車両を緊急停車させることを特徴とする請求項1又は2に記載の運転者状態検出装置。   3. The driver state detection device according to claim 1, wherein when the driver determines that the driver is in a state of loss of consciousness, the driver determines the state of emergency.
  4. 前記状態判定手段は、運転者が意識消失状態であると判定したとき、緊急異常状態を外部に通報することを特徴とする請求項1〜3の何れか1項に記載の運転者状態検出装置。   The driver state detection device according to any one of claims 1 to 3, wherein when the driver determines that the driver is in a state of loss of consciousness, the driver determines the emergency abnormal state to the outside. .
JP2015170118A 2015-08-31 2015-08-31 Driver status detection device Active JP6256433B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2015170118A JP6256433B2 (en) 2015-08-31 2015-08-31 Driver status detection device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
JP2015170118A JP6256433B2 (en) 2015-08-31 2015-08-31 Driver status detection device

Publications (2)

Publication Number Publication Date
JP2017049636A true JP2017049636A (en) 2017-03-09
JP6256433B2 JP6256433B2 (en) 2018-01-10

Family

ID=58278791

Family Applications (1)

Application Number Title Priority Date Filing Date
JP2015170118A Active JP6256433B2 (en) 2015-08-31 2015-08-31 Driver status detection device

Country Status (1)

Country Link
JP (1) JP6256433B2 (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060011399A1 (en) * 2004-07-15 2006-01-19 International Business Machines Corporation System and method for controlling vehicle operation based on a user's facial expressions and physical state
JP2008037218A (en) * 2006-08-03 2008-02-21 Aisin Aw Co Ltd Vehicle control device
JP2008065705A (en) * 2006-09-08 2008-03-21 Toyota Motor Corp Dozing detector and dozing detecting method
JP2010204847A (en) * 2009-03-02 2010-09-16 Toyota Motor Corp Driver condition detection device and collision early detection device using same
JP2012254745A (en) * 2011-06-09 2012-12-27 Toyota Motor Corp Method for determining necessity for abnormality-responsive drive control, and drive control device based on necessity determination for abnormality-responsive drive control
JP2016009257A (en) * 2014-06-23 2016-01-18 株式会社デンソー Driver's undrivable state detector

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060011399A1 (en) * 2004-07-15 2006-01-19 International Business Machines Corporation System and method for controlling vehicle operation based on a user's facial expressions and physical state
JP2008037218A (en) * 2006-08-03 2008-02-21 Aisin Aw Co Ltd Vehicle control device
JP2008065705A (en) * 2006-09-08 2008-03-21 Toyota Motor Corp Dozing detector and dozing detecting method
JP2010204847A (en) * 2009-03-02 2010-09-16 Toyota Motor Corp Driver condition detection device and collision early detection device using same
JP2012254745A (en) * 2011-06-09 2012-12-27 Toyota Motor Corp Method for determining necessity for abnormality-responsive drive control, and drive control device based on necessity determination for abnormality-responsive drive control
JP2016009257A (en) * 2014-06-23 2016-01-18 株式会社デンソー Driver's undrivable state detector

Also Published As

Publication number Publication date
JP6256433B2 (en) 2018-01-10

Similar Documents

Publication Publication Date Title
US10572746B2 (en) Apparatus detecting driving incapability state of driver
US9855956B2 (en) Driving assist device
CN106463065B (en) Driving incapability state detection device for driver
CN106663377B (en) The driving of driver is unable to condition checkout gear
JP5288045B2 (en) Emergency vehicle evacuation device
JP5527411B2 (en) Emergency vehicle evacuation device
JP2016045714A (en) On-vehicle control device
JP2016045713A (en) On-vehicle control device
US9888875B2 (en) Driver monitoring apparatus
JP6361312B2 (en) Driver inoperability detection device
JP2016009256A (en) Driver's undrivable state detector
JP2018008575A (en) Vehicle control device
JP6520737B2 (en) Biological information measuring device, vehicle-mounted device, and biological information measuring system
JP6256433B2 (en) Driver status detection device
WO2018211966A1 (en) Driver monitoring device, driver monitoring method and program for monitoring driver
JP2019133488A (en) Driving support device and driving support method
JP6607287B2 (en) Driver inoperability detection device
JP6614286B2 (en) Driver inoperability detection device
JP2020102252A (en) Driver monitoring system
JP2021089652A (en) Driver monitor, driver state estimation method and program
WO2017150466A1 (en) Driver monitoring system
JP2005004503A (en) Vehicle traveling support device

Legal Events

Date Code Title Description
A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20170323

A977 Report on retrieval

Free format text: JAPANESE INTERMEDIATE CODE: A971007

Effective date: 20171012

TRDD Decision of grant or rejection written
A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

Effective date: 20171107

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20171120

R150 Certificate of patent or registration of utility model

Ref document number: 6256433

Country of ref document: JP

Free format text: JAPANESE INTERMEDIATE CODE: R150