JP2010224827A - Eye opening/closing condition determination device, forward/sideways-facing condition determination device, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accurately determine an eye opening/closing condition of a driver by a simple structure. <P>SOLUTION: Various sensors detect a yaw angular velocity, a lateral acceleration, a vehicle velocity, and a steering angle. A target course estimation part 24 estimates a target course. An eye opening time steering angle estimation part 26 estimates a steering angle for the eye opening condition based on a deviation from the target course at a forward gazing point. An eye closing time steering angle estimation part 28 estimates a steering angle for the eye closing condition by assuming that a deviation from the target course at the forward gazing point is 0. An eye opening/closing condition determination part 30 compares the respective estimated steering angles for the eye opening condition and the eye closing condition with the steering angle detected by a steering angle sensor 18, and based on the comparison result, determines an eye opening/closing condition of the driver. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an open / closed eye state determination device, a front sideways state determination device, and a program. The present invention relates to a program, a front lateral state determination device and a program for determining a forward state and a lateral state of a driver.

  2. Description of the Related Art Conventionally, a technique for detecting an open / closed eye state of a driver using a camera is known (for example, Patent Document 1). In addition, a technique for detecting an open / closed eye state of a driver using an EOG (electro-oculograph) method is known.

  Further, a frequency component (human frequency component) that changes in a predetermined dozing operation is extracted from the steering angle detection signal, the absolute value of the extracted frequency component is added (integrated), and the added value reaches a predetermined value. An apparatus that generates an alarm when the time until the time is within a predetermined range is known (for example, Patent Document 2). In this device, a snooze operation is detected by capturing the wobbling that frequently occurs during a nap.

JP 2008-171107 A JP-A-8-268190

  However, the technique described in the above-mentioned Patent Document 1 has a problem that the cost increases because it is necessary to mount a camera for photographing the driver.

  In addition, in the technology for detecting the open / closed eye state of the driver using the above EOG, it is necessary to attach a sensor to the driver itself.

  Further, in the technique described in Patent Document 2 described above, since the stagger component to be extracted is generated even when the driver is a beginner or due to the influence of disturbance such as wind, wrinkles, and curves, many false detections occur. There is a problem.

  The present invention has been made in order to solve the above-described problems. It is a first object of the present invention to provide an open / closed eye state determination apparatus and program capable of accurately determining the open / closed eye state of a driver with a simple configuration. The purpose.

  It is a second object of the present invention to provide a front lateral state determination device and a program that can accurately determine whether a driver is in a forward state or a lateral state with a simple configuration.

  In order to achieve the above object, an open / closed eye state determination device according to a first aspect of the present invention includes a state detection unit that detects a traveling state of a host vehicle and an operation state when a driver operates the host vehicle, and the state detection unit. The operation state estimation means for estimating the operation state for each of the eye open state and the eye closed state of the driver based on the travel state detected by the driver, or the travel state and the operation state, and the operation state estimation unit Open / closed eye state determination for comparing each of the operation states for the opened and closed eye states with the operation state detected by the state detecting unit and determining the open / closed eye state of the driver based on the comparison result Means.

  According to a second aspect of the present invention, there is provided a program that causes the computer to detect the traveling state detected by the state detecting means that detects the traveling state of the host vehicle and the operation state when the driver operates the host vehicle, or the traveling state and the operation. Each of the operation state estimation means for estimating the operation state for each of the open and closed eye states of the driver based on the state, and each of the operation states for the open and closed eye states estimated by the operation state estimation means; It is a program for comparing with the operation state detected by the state detection unit and functioning as an open / closed eye state determination unit that determines the open / closed eye state of the driver based on the comparison result.

  According to the first and second inventions, the state detection means detects the traveling state of the host vehicle and the operation state when the driver operates the host vehicle. Based on the running state detected by the state detecting unit or the running state and the operating state, the operation state estimating unit estimates an operating state for each of the eye open state and the closed eye state of the driver.

  Then, the open / closed eye state determination unit compares the operation state for the open eye state and the closed eye state estimated by the operation state estimation unit with the operation state detected by the state detection unit, and based on the comparison result, the driver The open / closed eye state is determined.

  In this way, by comparing each of the estimated operation states for the open and closed eye states with the detected operation state and determining the open / close eye state of the driver, the open / close eye of the driver can be obtained with a simple configuration. The state can be accurately determined.

  The open / closed eye state determination device according to the first aspect of the invention further includes a dozing determination unit that determines whether or not the driver is in a dozing state based on time-series data of the open / closed eye state determined by the open / closed eye state determination unit. Can be included. Thereby, it is possible to accurately determine whether or not the driver is dozing with a simple configuration.

  The open / closed eye state determination device according to the first aspect of the present invention can further include a feature amount calculation unit that calculates a blink feature amount based on time-series data of the open / closed eye state determined by the open / closed eye state determination unit. . As a result, the blink feature amount can be accurately calculated with a simple configuration.

  The state detection means according to the first invention detects the yaw angular velocity, the lateral acceleration, and the vehicle speed as the traveling state of the host vehicle, detects the steering angle as the operation state, and the operation state estimation means includes the detected yaw Based on the angular velocity, lateral acceleration, and vehicle speed, the deviation from the target course at the forward gaze point is estimated, and the steering angle for the open eye state is calculated based on the deviation from the target course at the estimated forward gaze point according to the forward gaze model. The steering angle with respect to the closed-eye state can be estimated with the deviation from the target course at the forward gazing point as a predetermined value. Thereby, the steering angle with respect to the open eye state and the closed eye state can be estimated using the forward gaze model.

  The state detection means according to the first invention detects the yaw angular velocity, the lateral acceleration, and the vehicle speed as the traveling state of the host vehicle, detects the steering angle as the operation state, and the operation state estimation means includes the detected yaw The lateral position of the host vehicle with respect to the target course is estimated based on the angular velocity, the lateral acceleration, and the vehicle speed, and the steering angle with respect to the open eye state is estimated based on the lateral position of the host vehicle with respect to the estimated target course. With the lateral position of the vehicle as a predetermined value, the steering angle for the closed eye state can be estimated. Thus, the steering angle for the open eye state and the closed eye state can be estimated using the lateral position of the host vehicle with respect to the target course.

  The open / closed eye state determination device includes: an imaging unit that captures an area including the driver's eyes; and a second open / closed eye state determination unit that determines an open / closed eye state of the driver based on an image captured by the imaging unit. And a determination result integrating means for determining the open / closed eye state of the driver by integrating the determination result by the open / closed eye state determining unit and the determination result by the second open / closed eye state determining unit. Thereby, the open / closed eye state of the driver can be determined more accurately.

  A front lateral state determination device according to a third aspect of the present invention includes a state detection unit that detects a traveling state of the host vehicle and an operation state when the driver operates the host vehicle, and the traveling state detected by the state detection unit, Alternatively, based on the traveling state and the operation state, an operation state estimation unit that estimates the operation state for each of the forward state and the lateral state of the driver, and the forward state and the lateral state estimated by the operation state estimation unit Each of the operation states is compared with the operation state detected by the state detection means, and based on the comparison result, it is determined whether the driver is in a forward state or a lateral state. Means.

  According to a fourth aspect of the present invention, there is provided a program that causes the computer to detect the traveling state detected by the state detecting unit that detects the traveling state of the host vehicle and the operation state when the driver operates the host vehicle, or the traveling state and the operation. Based on the state, the operation state estimation means for estimating the operation state for each of the forward state and the lateral state of the driver, and each of the operation state for the forward state and the lateral state estimated by the operation state estimation unit, A program for comparing the operation state detected by the state detection unit and causing the driver to function as a front side state determination unit that determines whether the driver is in a front side state or a side state based on the comparison result It is.

  According to the third and fourth inventions, the state detection means detects the traveling state of the host vehicle and the operation state when the driver operates the host vehicle. Based on the running state detected by the state detecting unit, or the running state and the operating state, the operating state estimating unit estimates an operating state for each of the forward state and the lateral state of the driver.

  Then, the front side state determination unit compares each of the operation state for the front side state and the side state estimated by the operation state estimation unit with the operation state detected by the state detection unit, and based on the comparison result, the driver Is in a forward-facing state or a lateral-facing state.

  Thus, by comparing each of the estimated operation states for the forward state and the lateral state with the detected operation state and determining the forward state and the lateral state of the driver, the driver can be configured with a simple configuration. Whether it is a forward-facing state or a lateral-facing state can be accurately determined.

  The front sideways state determination device according to the third aspect of the present invention can further include a side look determination unit that determines whether or not the sideways state is based on the time series data of the determination result by the front sideways state determination unit. This makes it possible to accurately determine whether or not the driver is looking aside with a simple configuration.

  As described above, according to the open / closed eye state determination apparatus and program of the present invention, each of the estimated open state and closed eye state is compared with the detected open state, and the driver's open / closed eye state is compared. By determining the state, it is possible to accurately determine the open / closed eye state of the driver with a simple configuration.

  According to the front lateral state determination device and the program of the present invention, the estimated operational state for the forward state and the lateral state is compared with the detected operational state to determine the forward state and the lateral state of the driver. Thus, it is possible to accurately determine whether the driver is in the forward-facing state or the lateral-facing state with a simple configuration.

It is a block diagram which shows the structure of the dozing determination apparatus which concerns on the 1st Embodiment of this invention. It is a figure for demonstrating the shift | offset | difference from the target course in a front gaze point. It is a figure for demonstrating the calculation method of the deviation | shift from the target course in a front gaze point. It is a figure for demonstrating the flow of a process of the steering angle estimation part at the time of an eye opening, the steering angle estimation part at the time of an eye closing, and an eye open state determination part. It is a flowchart which shows the content of the dozing determination processing routine in the dozing determination apparatus which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the content of the opening-and-closing eye determination processing routine in the dozing determination apparatus which concerns on the 1st Embodiment of this invention. It is a block diagram which shows the structure of the dozing determination apparatus which concerns on the 2nd Embodiment of this invention. It is a flowchart which shows the content of the dozing determination processing routine in the dozing determination apparatus which concerns on the 2nd Embodiment of this invention. It is a block diagram which shows the structure of the looking-aside state determination apparatus which concerns on the 4th Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. A case where the present invention is applied to a dozing determination apparatus mounted on a vehicle will be described as an example.

  As shown in FIG. 1, the dozing determination device 10 according to the first embodiment detects a yaw angular velocity sensor 12 that detects a yaw angular velocity as a traveling state of the host vehicle and a lateral acceleration as a traveling state of the host vehicle. A lateral acceleration sensor 14, a vehicle speed sensor 16 that detects a vehicle speed as a traveling state of the host vehicle, a steering angle sensor 18 that detects a steering angle of a steering wheel as an operation state when the driver operates the host vehicle, Based on the outputs from the angular velocity sensor 12, the lateral acceleration sensor 14, the vehicle speed sensor 16, and the steering angle sensor 18, it is determined whether or not the driver is dozing, and a warning message is displayed on the display device 40 according to the determination result. And a computer 20 for displaying.

  The yaw angular velocity sensor 12 outputs a yaw angular velocity signal indicating the detected yaw angular velocity, and the lateral acceleration sensor 14 outputs a lateral acceleration signal indicating the detected lateral acceleration. The vehicle speed sensor 16 outputs a vehicle speed signal indicating the detected vehicle speed of the host vehicle, and the steering angle sensor 18 outputs a steering angle signal indicating the detected steering angle of the steering wheel. The yaw angular velocity sensor 12, the lateral acceleration sensor 14, the vehicle speed sensor 16, and the steering angle sensor 18 input respective signals to the computer 20 at certain intervals (for example, every 100 ms).

  The computer 20 includes a CPU, a RAM, and a ROM that stores a program for executing a dozing determination processing routine, which will be described later, and is functionally configured as follows. The computer 20 acquires a steering angle signal from the steering angle sensor 18 for a predetermined time and stores it in a memory (not shown), a yaw angular velocity signal from the yaw angular velocity sensor 12, and a lateral acceleration sensor 14. Based on the lateral acceleration signal and the vehicle speed signal from the vehicle speed sensor 16, the target course estimation unit 24 for estimating the target course, and the steering angle when the driver is in an open state based on the estimated target course are predetermined. Eye-opening steering angle estimation unit 26 that estimates the time and stores it in the memory, steering angle estimation unit 28 that closes the eye when the driver is in the closed-eye state and stores it in the memory, and sensor values The steering angle indicated by the steering angle signal acquired by the acquisition unit 22 is compared with each of the estimated steering angle in the open eye state and the steering angle in the closed eye state, and the open / closed eye state of the driver Based on the determined eye-opening state determination unit 30 and the time-series data of the determined open / closed eye state, it is determined whether or not the driver is dozing, and a warning message is displayed on the display device 40 according to the determination result. And a dozing state determination unit 32 to be performed. The dozing state determination unit 32 is an example of a dozing determination unit and a feature amount calculation unit.

  The sensor value acquisition unit 22 acquires the steering angle signal from the steering angle sensor 18 for a predetermined time ΔT and stores it in a memory (not shown).

  The target course estimator 24 obtains the yaw angular velocity and the lateral acceleration signal from the yaw angular velocity sensor 12 and the lateral acceleration sensor 14 by applying a filter having a cutoff frequency of 0.5 Hz or less, for example. A side slip angle is calculated based on the lateral acceleration. Based on the skid angle calculated by time t, the yaw angular velocity obtained by time t, and the vehicle speed indicated by the vehicle speed signal obtained from the vehicle speed sensor 16 by time t, the target course estimation unit 24 From the expressions (1) to (3), the position of the target course at time t is estimated.

However, X (t) and Y (t) are the front and rear positions and the horizontal position at time t, and θ (t) is the yaw angle at time t. V is the vehicle speed, β is the skid angle, and r is the yaw angular velocity. X ₀ is an initial value of the front and rear position, Y ₀ is an initial value of the horizontal position, θ ₀ is an initial value of the yaw angle, and predetermined values are used as these initial values. That's fine.

  As described above, the target course estimation unit 24 sequentially estimates the position of the target course based on the output from each sensor, and stores the estimated position of the target course in the memory.

  Further, the computer 20 sequentially calculates the side slip angle based on the yaw angular velocity signal from the yaw angular velocity sensor 12 and the lateral acceleration signal from the lateral acceleration sensor 14, and calculates the side slip angle calculated by time t until time t. Based on the obtained yaw angular velocity and the vehicle speed indicated by the vehicle speed signal obtained from the vehicle speed sensor 16 until time t, the position of the host vehicle at time t is sequentially calculated according to the above equations (1) to (3). Store in memory.

Eye opening when the steering angle estimation unit 26, as described below, each of which estimates the steering angle in the open-eye state from time t ₀ to time t _{0 +} [Delta] T.

First, at time t ₀ , the yaw angular velocity signal from the yaw angular velocity sensor 12, the lateral acceleration signal from the lateral acceleration sensor 14, the vehicle speed signal from the vehicle speed sensor 16, and the steering angle signal from the steering angle sensor 18 are acquired, and the yaw angular velocity signal is obtained. on the basis of the angular velocity and lateral acceleration, it calculates the slip angle at time t _0.

Yaw rate, side slip angle at the vehicle position and the time t ₀ at time t _0, and on the basis of the vehicle speed, and calculates the vehicle position at time t _{0 +} Delta] t in accordance with equation (1) to (3). Note that the vehicle speed, using a vehicle speed indicated by the speed signal obtained from the vehicle speed sensor 16 at time t _0.

Also, obtained from various sensors, yaw rate, vehicle speed, and the steering angle, based on the calculated slip angle, the following (4) in accordance with the vehicle model shown in the expression, the differential value of the yaw angular velocity at time t ₀ And the differential value of the skid angle is calculated.

However, I is a moment of _inertia, l f, _{l r} is the inter-axle distance back and _forth, k f, _{k r} is a longitudinal cornering power, [delta] is a steering angle. The vehicle model represented by the above equation (4) is a two-wheel steering model described in non-patent literature (written by Masato Abe, “Motion and Control of Automobile”).

From the differential value of the differential value and the slip angle of the yaw rate in the calculated time t _0, to calculate the yaw rate and the slip angle at a subsequent time t _{0 +} Δt, is calculated according to the above (1) to (3) and the vehicle position at time t _{0 +} Delta] t, yaw rate, side slip angle at a subsequent time t _{0 +} Delta] t, and based on the vehicle speed, to calculate the position of the vehicle at the next time t ₀ +2 · Δt.

Then, a deviation ε from the target course at the forward gazing point at the next time t ₀ + Δt is calculated. As shown in FIG. 2, the deviation ε from the target course at the front gazing point is the difference between the target course at the front gazing point (position away from the front gazing distance L) and the vehicle position at the front gazing point. The lateral position deviation ε is shown.

Specifically, as shown in FIG. 3, the lateral position of the target course at time t ₀ + Δt estimated by the target course estimation unit 24 and the lateral position of the host vehicle at time t ₀ + Δt calculated as described above. the difference y between the lateral position and velocity v _y of the vehicle obtained from the differentiation of the lateral position of the vehicle, based on the forward fixed point time T ₀ previously obtained, in accordance with the following equation (5), forward Note The deviation ε from the target course at the viewpoint is calculated.

Next, using a forward gaze model described in non-patent literature (Masato Abe, “Motion and Control of Automobile”), the transfer function H of the deviation ε from the target course at the front gaze point and the steering angle δ. (s) using, from the deviation ε from the target course in the forward gaze point at time t _{0 +} Delta] t, to estimate the time t _{0 +} Delta] t steering angle [delta]. The transfer function H (s) is expressed by the following equation (6).

However, h, τ _I , τ _L , and τ _D are transfer function constants.

  Here, the deviation ε from the target course at the front gazing point is set as the deviation from the target course at the front gazing point in the open eye state, and the estimated steering angle is set as an estimated value of the steering angle in the open eye state.

Further, a steering angle in the estimated time t _{0 +} Delta] t, by using the yaw rate and slip angle at time t _{0 +} Delta] t which is calculated in the above (4) in accordance with equation of the yaw angular velocity of the differential value and the slip angle The differential value is calculated, and the yaw angular velocity and the skid angle at the next time t ₀ + 2 · Δt are calculated from the calculated differential value of the yaw angular velocity and the differential value of the skid angle. The position of the vehicle in the calculated time t ₀ +2 · Δt, yaw rate, side slip angle at a subsequent time t ₀ +2 · Δt, and on the basis of the vehicle speed, the position of the vehicle at the next time t ₀ +3 · Δt Is calculated.

As shown in FIG. 4, the eye opening steering angle estimation unit 26 repeats the above process to estimate the steering angle in the eye opening state from the time t ₀ to the predetermined time ΔT and stores it in the memory.

As will be described below, the closed-eye steering angle estimation unit 28 estimates the steering angle in the open eye state at times t _{0 to} t ₀ + ΔT.

First, as in the eye opening when the steering angle estimation unit 26, the vehicle position at the next time t _{0 +} Delta] t, yaw rate, and calculates the sideslip angle, and calculates the position of the vehicle at the next time t ₀ +2 · Δt.

Then, as the deviation from the target course in the closed eye state, the deviation ε from the target course at the forward gazing point is set to 0, and the closed eye state at time t ₀ + Δt using the transfer function H (s) of the above equation (6). Is estimated.

Further, using the estimated steering angle and the yaw angular velocity and side slip angle calculated above, the yaw angular velocity differential value and the side slip angle differential value are calculated according to the above equation (4), and the calculated yaw angle is calculated. The yaw angular velocity and the skid angle at the next time t ₀ + 2 · Δt are calculated from the differential value of the angular velocity and the differential value of the skid angle. The position of the vehicle in the calculated time t ₀ +2 · Δt, yaw rate, side slip angle at a subsequent time t ₀ +2 · Δt, and on the basis of the vehicle speed, the position of the vehicle at the next time t ₀ +3 · Δt Is calculated.

Eyes closed when the steering angle estimation unit 28, as shown in FIG. 4, by repeating the above process, estimating the steering angle at the closed-eye state from time t ₀ until a predetermined time ΔT destination, stored in the memory.

As shown in FIG. 4, the open eye state determination unit 30 determines the steering angle in the open eye state estimated from the time t ₀ to the time t ₀ + ΔT, and the time t ₀ to the time t ₀ acquired by the sensor value acquisition unit 22. The difference is calculated by comparing the steering angle of + ΔT for each time, and the moving average of the difference is calculated. Moreover, the open-eye state determination unit 30, a steering angle in the closed-eye state is estimated for a time t _{0 ~} time t _{0 +} [Delta] T, the steering angle of the time t _{0 ~} time t _{0 +} [Delta] T obtained by the sensor value obtaining section 22 The difference is calculated by comparing each time, and the moving average of the difference is calculated. When the moving average of the difference is compared and the moving average of the difference with respect to the closed eye state is larger, it is determined that the eye is in the open state, and when the moving average of the difference with respect to the opened eye state is larger, It is determined that there is. A square error may be calculated as the difference in steering angle.

  The dozing state determination unit 32 calculates the eye closure time within a predetermined time as the blink feature amount from the time-series data of the determination result of the eye opening state determination unit 30, performs threshold determination for the calculated eye closing time, and It is determined whether or not is in a dozing state. Further, when the dozing state determination unit 32 determines that the driver is in the dozing state, the dozing state determination unit 32 causes the display device 40 to display a warning message indicating the dozing state.

  Next, the operation of the dozing determination device 10 according to the first embodiment will be described. While the vehicle equipped with the dozing determination device 10 is running, the computer 20 executes the dozing determination processing routine shown in FIG.

  First, in step 100, a process for determining an open / closed eye state is performed. Step 100 is realized by the open / closed eye determination processing routine shown in FIG. Hereinafter, the opening / closing eye determination processing routine will be described.

In step 118, the variable t representing the time, to set the current time t ₀ as an initial value. In step 120, signals output from each of the yaw angular velocity sensor 12, the lateral acceleration sensor 14, the vehicle speed sensor 16, and the steering angle sensor 18 are acquired and stored in the memory. Step 120 is repeated until it is determined in step 124 that the predetermined time ΔT has elapsed, and when the predetermined time ΔT has elapsed, the process proceeds to step 128. Thereby, a yaw angular velocity signal, a lateral acceleration signal, a vehicle speed signal, and a steering angle signal for a predetermined time ΔT are stored in the memory.

In the next step 128, based on the value obtained by multiplying the filter to each of the outputs of the steps 120 times t ₀ obtained by _to time t _{0 +} [Delta] T yaw angular velocity sensor 12, lateral acceleration sensor 14, a vehicle speed sensor 16 Thus, the position of the target course of the host vehicle from the step time t ₀ to the time t ₀ + ΔT is estimated. In step 130, time t is set to the next time (t + Δt), and in step 132, the position of the host vehicle at time t (= t ₀ + Δt) is calculated from the position of the host vehicle sequentially calculated by the computer 20. To get.

In step 134, the position of the host vehicle with respect to the open eye state at the next time t + Δt is calculated. If t = t ₀ + Δt, based on the sensor value at time t acquired at step 120 and the vehicle position at time t acquired at step 132, the host vehicle with respect to the eye open state at the next time t + Δt The position of is calculated. In other cases, the next time t + Δt is determined according to the vehicle model based on the steering angle in the open eye state at time t estimated in step 140 described later and the vehicle position at time t calculated in the previous step 134. The position of the host vehicle with respect to the eye open state is calculated.

In step 136, the position of the host vehicle with respect to the closed eye state at the next time t + Δt is calculated. In the case of t = t ₀ + Δt, on the basis of the sensor value at time t acquired at step 120 and the vehicle position at time t acquired at step 132, the vehicle's vehicle with respect to the closed eye state at the next time t + Δt Calculate the position. Otherwise, the next time t + Δt according to the vehicle model based on the steering angle in the closed eye state at time t estimated in step 142 described later and the vehicle position at time t calculated in the previous step 136. The position of the own vehicle with respect to the closed eye state is calculated.

  Then, in step 138, the position of the target course at time t estimated in step 128 and the position of the host vehicle at time t acquired in step 132 or the position of the host vehicle calculated in step 134 are set. Based on this, the deviation from the target course at the forward gazing point is calculated.

  In the next step 140, the steering angle in the open eye state at time t is estimated based on the deviation from the target course at the forward gazing point calculated in step 138 and stored in the memory. In step 142, the deviation from the target course is set to 0, and the steering angle in the closed eye state at time t is estimated and stored in the memory.

Then, in step 144, the time t is determined whether the host vehicle has reached the _{t 0} + [Delta] T, if not reached _{t 0} + [Delta] T, in step 146, the time t, and set the next time t + Delta] t, Return to step 134. On the other hand, when the time t reaches t ₀ + ΔT, the process proceeds to step 148.
In step 148, steering for open-eye state of the steering angle indicated by the steering angle signal until the time _{t 0} + _{Δt~t 0} + _ΔT obtained in step 120, until time _{t 0} + _{Δt~t 0} + _ΔT estimated in step 140 The corners are compared for each time to calculate the difference, and the total value of the differences is calculated.

In the next step 150, the closed-eye state of the steering angle indicated by the steering angle signal until the time _{t 0} + _{Δt~t 0} + _ΔT obtained in step 120, until time _{t 0} + _{Δt~t 0} + _ΔT estimated in step 142 A difference is calculated by comparing the steering angle with respect to each time, and a total value of the differences is calculated.

  In step 152, the total difference value for the open eye state calculated in step 148 is compared with the total difference value for the closed eye state calculated in step 150, and the total difference value for the open eye state is calculated. If it is smaller, it is determined that the driver is in the open eye state, and if the sum of the differences with respect to the closed eye state is smaller, it is determined that the driver is in the closed eye state, and the determination result is stored in the memory. Then, the open / closed eye determination processing routine is terminated.

  Then, in step 102 of the dozing determination processing routine of FIG. 5, it is determined whether or not a predetermined time has elapsed from the start of the processing. If not, processing for determining the open / closed eye state is performed again in step 100. Do. On the other hand, if a predetermined time has elapsed since the start of processing, the routine proceeds to step 104. At this time, the determination result of the open / closed eye state for a predetermined time is stored in the memory.

  In step 104, the eye closure time within a predetermined time is measured based on the determination result of the open / closed eye state for a predetermined time. In step 106, it is determined whether or not the closed eye time measured in step 104 is equal to or greater than a threshold value. If it is determined that the closed eye time is equal to or greater than the threshold value, it is determined that the driver is dozing. Therefore, in step 108, a warning message indicating that the driver is dozing is displayed on the display device 40. Then, the dozing determination processing routine is terminated. On the other hand, if the eye-closing time is less than the threshold value in step 106, it is determined that the driver is not dozing. Therefore, in step 110, a message indicating that the driver state is normal is displayed on the display device 40. Is displayed, and the dozing judgment routine ends.

  As described above, according to the snoozing determination apparatus according to the first embodiment, the steering angle for the estimated open eye state and the closed eye state is compared with the steering angle detected from the sensor, and the driver By determining the open / closed eye state, the open / closed eye state of the driver can be accurately determined with a simple configuration. In addition, with a simple configuration, it can be accurately determined whether or not the driver is dozing.

  In addition, from the driving state and the operation state detected by the sensor, the driver's forward gaze model and vehicle model are used to determine whether the eye is open or closed, and based on that, a doze state is determined, and a camera, an EOG, etc. The dozing state can be determined using the opening / closing eyes of the driver determined without using a device that measures the driver itself.

  Further, it is possible to determine the dozing state with high accuracy by determining the open / closed eye state having a high correlation with the dozing operation.

  Next, a second embodiment will be described. In addition, about the part of the structure similar to 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  In the second embodiment, the point that the driver's open / closed eye state is further determined based on the eye image captured by the imaging device and the determination result are integrated are the first embodiment. Mainly different from the form.

  As shown in FIG. 7, the dozing determination device 210 according to the second embodiment is installed diagonally forward of the yaw angular velocity sensor 12, the lateral acceleration sensor 14, the vehicle speed sensor 16, the steering angle sensor 18, and the driver. An image pickup device 212 that picks up an image of the area including the eyes of the driver obliquely from above, an eye image picked up by the image pickup device 212, a yaw angular velocity sensor 12, a lateral acceleration sensor 14, a vehicle speed sensor 16, and a steering angle sensor 18. And a computer 220 that determines whether or not the driver is dozing, and causes the display device 40 to display a warning message according to the determination result.

  The computer 220 receives a predetermined time from the sensor value acquisition unit 22, the target course estimation unit 24, the eye opening steering angle estimation unit 26, the eye closing steering angle estimation unit 28, the eye opening state determination unit 30, and the imaging device 212. An image acquisition unit 226 that acquires the eye image, a second eye opening state determination unit 228 that determines the eye open / closed state by detecting the eyelid opening from eye images for a predetermined time, and the determination result of the eye opening state determination unit 30 and Based on the result integration unit 230 that finally determines the open / closed eye state of the driver by integrating the determination results of the second open eye state determination unit 228 and the time-series data of the finally determined open / closed eye state And a dozing state determination unit 232 that displays a warning message on the display device 40 according to the determination result.

  The second eye open state determination unit 228 determines the open / closed eye state as follows.

  First, an eye area representing the eyes of the driver is extracted from the eye image, and a eyelid opening degree indicating the degree of eyelid opening is detected from each eye area. Then, when the detected eyelid opening is equal to or less than the eye closing threshold, it is determined that the eye is closed. On the other hand, when the detected eyelid opening is larger than the closed eye threshold, it is determined that the eye is in an open state.

  If the result integrating unit 230 determines that at least one determination result is the closed eye state based on the determination result by the eye open state determination unit 30 and the determination result by the second eye open state determination unit 228, the result integration unit 230 Therefore, it is determined that the eye is closed. Moreover, when it determines with both determination results being an open eye state, it is finally determined that it is an open eye state.

  The doze state determination unit 232 calculates the eye closure time within a predetermined time from the time series data of the final determination result by the result integration unit 230, performs a threshold determination on the calculated eye closure time, and the driver is in the doze state. It is determined whether or not there is. Further, when the dozing state determination unit 232 determines that the driver is dozing, the display device 40 displays a warning message indicating that the driver is dozing.

  Next, a dozing determination processing routine according to the second embodiment will be described with reference to FIG. In addition, about the process similar to 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  First, in step 250, an eye image captured by the imaging device 212 is acquired, and in step 100, the open / closed eye state is determined by executing an open / closed eye determination processing routine in the same manner as in the first embodiment.

  In step 252, the eyelid opening degree is detected from the eye image acquired in step 250. In step 254, the eyelid opening degree detected in step 252 is compared with the eye closing threshold value, so that the open / closed eye state of the driver is determined. Determine.

  In the next step 256, the determination result determined in step 100 and the determination result determined in step 254 are integrated to finally determine the open / closed eye state and store it in the memory.

  In step 102, it is determined whether or not a predetermined time has elapsed from the start of processing. If not, the process returns to step 250. On the other hand, if a predetermined time has elapsed since the start of processing, the routine proceeds to step 104.

  In step 104, the eye closure time within the predetermined time is measured based on the final determination result for the predetermined time obtained in step 256. In step 106, it is determined whether or not the measured eye closure time is equal to or greater than a threshold value. If it is determined that the closed eye time is equal to or greater than the threshold, a warning message indicating that the driver is dozing is displayed on the display device 40 in step 108, and the dozing determination processing routine is terminated. On the other hand, if the closed eye time is less than the threshold value in step 106, a message indicating that the driver is in the normal state is displayed on the display device 40 in step 110, and the dozing judgment routine is terminated. .

  As described above, according to the snoozing determination apparatus according to the second embodiment, the steering angle for each of the estimated open eye state and the closed eye state is compared with the steering angle detected from the sensor, and the driver The open / closed eye state of the driver can be determined with higher accuracy by determining the open / closed eye state of the driver from the eye image captured by the imaging device.

  In the above-described embodiment, the case where the determination results are integrated so as to finally determine the closed eye state when at least one of the determination results is the closed eye state is described as an example. The determination results are not limited, and the determination results may be integrated by other methods. For example, when at least one of the determination results is an open eye state, the determination results may be integrated so that the eye open state is finally determined.

  Next, a third embodiment will be described. Since the configuration of the dozing determination apparatus according to the third embodiment is the same as that of the first embodiment, description thereof is omitted.

  The third embodiment is different from the first embodiment in that the steering angle in the open eye state is estimated using the generalized predictive control theory.

  The principle of estimating the steering angle for the open eye state in the third embodiment will be described. In the present embodiment, using the generalized predictive control theory described in non-patent literature (Shinichiro Horiuchi, Atsushi Sunada, “Evaluation Using Steering Support System and Multi-Input Driver Model for Lane Tracking”), Estimate the steering angle in the open eye state.

  It is assumed that the driver steers the vehicle so as to minimize the evaluation function J expressed by the following equation (7), which is calculated by the sum of squares of the lateral position and the steering angle up to M samples.

  A steering angle δ that minimizes the evaluation function J expressed by the above equation (7) with respect to the lateral position y of the vehicle position at time t calculated from the vehicle model with reference to the target course at time t. Obtained as an estimated value of the steering angle in the eye open state at time t.

  When solving the steering angle that minimizes the evaluation function J of the above expression (7), the relational expression between the lateral position y and the steering angle δ shown in the following expression (8) and the following expression (9): The solution is obtained by solving the Riccati equation using the relationship between the sensor value z and the lateral position y shown in FIG.

However, A and B are polynomials described by a time delay operator q ^{−1 that} delays a signal by one time. C is a constant representing the relationship between the sensor value z and the lateral position y.

  Further, the above expression (8) is a relational expression obtained from the following expressions (10) to (12).

The above equation (10) is a relational expression representing the relationship between the yaw angular velocity r, the lateral position velocity v _y and the steering angle δ based on the vehicle model, and the above equation (11) is a relational expression regarding the forward deviation ε. Further, the above expression (12) is a relational expression regarding the dot of the forward deviation speed ε, and the above expression (13) is a relational expression regarding the yaw angular velocity θ.

Eye opening when the steering angle estimation unit 26, like the first embodiment, calculates the yaw rate and slip angle at a subsequent time t _{0 +} Delta] t, calculates the position of the vehicle at the next time t ₀ +2 · Δt To do.

Then, based on the target course at the next time t ₀ + Δt and the calculated position of the host vehicle, the lateral position of the host vehicle with respect to the target course is calculated as a target error.

Then, on the basis of the lateral position of the vehicle at time t _{0 +} Delta] t, and calculates a steering angle which minimizes the evaluation function J described above is estimated as the steering angle in the open-eye state at the next time t _{0 +} Δt.

Also, using the estimated steering angle and the yaw angular velocity and skid angle calculated above, the yaw angular velocity differential value and the skid angle differential value are calculated according to the above equation (4). The yaw angular velocity and the side slip angle at the next time t ₀ + 2 · Δt are calculated from the calculated differential value of the yaw angular velocity and the differential value of the side slip angle. The position of the vehicle in the calculated time t ₀ +2 · Δt, yaw rate, side slip angle at a subsequent time t ₀ +2 · Δt, and on the basis of the vehicle speed, the position of the vehicle at the next time t ₀ +3 · Δt Is calculated.

The eye opening steering angle estimation unit 26 repeats the above processing to estimate the steering angle in the eye opening state from the time t ₀ to the predetermined time ΔT ahead, and stores it in the memory.

Eyes closed when the steering angle estimation unit 28, like the first embodiment, calculates the yaw rate and slip angle at a subsequent time t _{0 +} Delta] t, calculates the position of the vehicle at the next time t ₀ +2 · Δt To do.

Then, assuming that the target error is constant, the target course at time t _0, the lateral position of the vehicle relative to the target course obtained based on the position of the vehicle calculated at time t ₀ (target error) Is the horizontal position after the next time t ₀ + Δt.

Next, the steering angle that minimizes the evaluation function J is calculated based on the lateral position of the host vehicle, and is estimated as the steering angle in the closed eye state at the next time t ₀ + Δt.

Also, using the estimated steering angle and the yaw angular velocity and skid angle calculated above, the yaw angular velocity differential value and the skid angle differential value are calculated according to the above equation (4). The yaw angular velocity and the side slip angle at the next time t ₀ + 2 · Δt are calculated from the calculated differential value of the yaw angular velocity and the differential value of the side slip angle. The position of the vehicle in the calculated time t ₀ +2 · Δt, yaw rate, side slip angle at a subsequent time t ₀ +2 · Δt, and on the basis of the vehicle speed, the position of the vehicle at the next time t ₀ +3 · Δt Is calculated.

The closed-eye steering angle estimator 28 estimates the steering angle in the closed-eye state from time t ₀ to a predetermined time ΔT ahead by repeating the above processing, and stores it in the memory.

  In addition, since it is the same as that of 1st Embodiment about the other structure and effect | action of the dozing determination apparatus which concerns on 3rd Embodiment, description is abbreviate | omitted.

  In this way, from the running state and the operation state detected by the sensor, the generalized predictive control theory is used as a driver model to determine whether the eye is open or closed, and based on that, the doze state is determined. The dozing state can be determined using the opening / closing eyes of the driver determined without using a device that measures the driver itself such as EOG.

  In the first to third embodiments described above, the case where it is determined that the eye is determined to be dozing when the measured closed eye time is equal to or greater than the threshold value is described as an example. However, the present invention is not limited thereto. Instead, it may be determined that the patient is in a dozing state when the average closed eye time obtained from a plurality of measurement results is equal to or greater than a threshold value.

  Further, the case where the eye closing time is obtained as the blink feature amount has been described as an example. However, the present invention is not limited to this. For example, PERCLOS (ratio of eye closing time with respect to unit time) is calculated, and the calculated PERCLOS is calculated. It may be used to determine whether or not it is a doze state.

  Next, a fourth embodiment will be described. In addition, about the part which becomes the structure similar to 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  In the fourth embodiment, a case will be described in which the present invention is applied to a side-by-side state determination device that determines a side-by-side state of a driver.

  As shown in FIG. 9, the look-ahead state determination device 410 according to the fourth embodiment includes a yaw angular velocity sensor 12, a lateral acceleration sensor 14, a vehicle speed sensor 16, a steering angle sensor 18, a yaw angular velocity sensor 12, A computer that determines whether or not the driver is looking aside based on outputs from the lateral acceleration sensor 14, the vehicle speed sensor 16, and the steering angle sensor 18, and displays a warning message on the display device 40 according to the determination result. 420.

  The computer 420 includes a CPU, a RAM, and a ROM that stores a program for executing a look-aside state determination process routine, which will be described later, and is functionally configured as follows. The computer 420 estimates the steering angle when the driver is in the forward state for a predetermined time based on the estimated target course, the target course estimation unit 24, and the estimated target course, and stores the estimated value in the memory. The steering angle estimating unit 426, the steering angle estimating unit 428 for estimating the steering angle when the driver is in the sideways state for a predetermined time and storing it in the memory, and the steering angle signal acquired by the sensor value acquiring unit 22 are A forward-state determination unit 430 that determines whether the driver is in a forward-facing state or a lateral-facing state by comparing the indicated steering angle with the estimated steering angle in the forward-facing state and the steering angle in the lateral-facing state. Based on the time-series data of the determined determination result, it is determined whether or not the driver is in a look-aside state, and a warning message is displayed on the display device 40 according to the determination result. And a state determining unit 432.

The forward steering angle estimation unit 426 estimates the steering angle by using the deviation from the target course at the front gazing point, similarly to the eye opening steering angle estimation unit 26 described in the first embodiment. Te, and the estimated value of the steering angle in the forward facing state, estimates the steering angle in the forward state from time t ₀ to time t _{0 +} [Delta] t.

The lateral steering angle estimation unit 428 estimates the steering angle by setting the deviation from the target course at the front gazing point as 0, similarly to the closed-eye steering angle estimation unit 28 described in the first embodiment. Te, and the estimated value of the steering angle in the horizontal state, and estimates a steering angle in the horizontal state from time t ₀ to time t _{0 +} [Delta] t.

Forward state determination unit 430, the time _{t 0} + _{Δt~t 0} + a steering angle in the forward facing state estimated for [Delta] T, the time obtained by the sensor value obtaining unit 22 _{t 0} + _{Δt~t 0} + ΔT of the steering angle and the time for each The difference is calculated in comparison with each other, and the moving average of the difference is calculated. Also, forward state determination unit 430, a steering angle in the horizontal state estimated for the time _{t 0} + _{Δt~t 0} + _ΔT, and a steering angle of the time _{t 0} + _{Δt~t 0} + _ΔT obtained by the sensor value obtaining section 22 The difference is calculated by comparing each time, and the moving average of the difference is calculated. Comparing the moving average of the difference, if the moving average of the difference with respect to the horizontal state is larger, it is determined to be a forward state, and if the moving average of the difference with respect to the forward state is larger, It is determined that there is.

  The looking-aside state determination unit 432 calculates a lateral time within a predetermined time from the time-series data of the determination result of the forward state determination unit 430, performs a threshold determination on the calculated lateral direction time, and determines whether the driver is in the aside state Determine whether or not. Further, when the driver is determined to be in the side-by-side state, the side-by-side state determination unit 432 causes the display device 40 to display a warning message indicating the side-by-side state.

  Next, the operation of the looking-aside state determination device 410 according to the fourth embodiment will be described. While the vehicle equipped with the looking-aside state determination device 410 is traveling, a side-by-side state determination processing routine is executed in the computer 420 as described below.

  First, by performing processing similar to the open / closed eye determination processing routine in the first embodiment, processing for determining whether the driver is in a forward-facing state or a lateral-facing state is performed.

  Then, the above determination process is repeated until a predetermined time has elapsed from the start of the process. Thereby, the determination result of the forward state or the lateral state for a predetermined time is stored in the memory.

  Next, based on the determination result for a predetermined time stored in the memory, the horizontal time within the predetermined time is measured, and it is determined whether or not the measured horizontal time is equal to or greater than a threshold value. When it is determined that the sideways time is greater than or equal to the threshold, it is determined that the driver is in the side-by-side state. Therefore, a warning message indicating that the driver is in the side-by-side state is displayed on the display device 40, and the side-by-side determination process. End the routine. On the other hand, when the sideways time is less than the threshold value, it is determined that the driver is not in the look-aside state, so a message indicating that the driver is in the normal state is displayed on the display device 40, and the look-ahead state determination process End the routine.

  As described above, according to the aside look state determination device according to the fourth embodiment, each of the estimated steering angles for the forward state and the lateral state is compared with the steering angle detected from the sensor, By determining the forward state and the lateral state of the driver, the forward state and the lateral state of the driver can be accurately determined with a simple configuration. In addition, it is possible to accurately determine whether or not the driver is looking aside with a simple configuration.

  Also, from the driving state and operation state detected by the sensor, the driver's forward gaze model and vehicle model are used to determine whether the vehicle is in a forward-facing state or a lateral-facing state, and based on that, a side-by-side state is determined. It is possible to determine the look-ahead state using the forward-facing state and the lateral-facing state of the driver determined without using a device that measures the driver itself.

  In the first to fourth embodiments described above, the case where the position of the target course is estimated using the sensor value has been described as an example. However, the present invention is not limited to this. You may make it estimate the position of a target course based on the front image from the imaging device installed so that the front of a vehicle may be imaged. In this case, based on the position of the target course estimated from the front image, it is possible to estimate the positional deviation of the front gazing point from the target course and the lateral position of the host vehicle with respect to the target course.

  Further, as an example of a closed eye driver model, the case where the deviation from the target course at the forward gazing point is set to 0, or the case where the lateral position of the host vehicle with respect to the target course is fixed has been described as an example, but the present invention is limited to this. It is not a thing. For example, the steering angle in the closed eye state may be estimated on the assumption that the deviation from the target course at the forward gazing point is constant (not updated). Further, the steering angle in the closed eye state may be estimated on the assumption that the steering angle is constant. Further, the steering angle in the closed eye state may be estimated on the assumption that the steering torque is constant.

  Moreover, although the case where the yaw angular velocity, the lateral acceleration, the vehicle speed, and the steering angle are adopted as the operation state or the traveling state to be detected has been described as an example, the present invention is not limited to this. For example, at least one or more of the vehicle speed, the yaw angular velocity, the skid angle, the lateral acceleration, and the travel state amount that can be easily converted into these values may be detected as the travel state. Further, as the operation state, at least one of a steering angle, an actual steering angle, a steering torque, an accelerator pedal operation amount, and a brake pedal operation amount may be detected. In this case, the operation state in each of the open eye state and the closed eye state, or the operation state in each of the forward and lateral states is estimated using the detected travel state or the detected travel state and operation state. You can do it.

  The program of the present invention can be provided by being stored in a storage medium.

10, 210 Dozing state determination device 12 Yaw angular velocity sensor 14 Lateral acceleration sensor 16 Vehicle speed sensor 18 Steering angle sensors 20, 220, 420 Computer 22 Sensor value acquisition unit 24 Target course estimation unit 26 Eye opening steering angle estimation unit 28 Eye closing steering angle Estimating unit 30 Eye open state determining unit 32, 232 Dozing state determining unit 212 Imaging device 228 Second eye open state determining unit 230 Result integrating unit 410 Side look state determining device 426 Front steering angle estimating unit 428 Side steering angle estimating unit 430 Front state Judgment unit 432 Aside look state judgment unit

Claims (10)

A state detecting means for detecting a traveling state of the host vehicle and an operation state when the driver operates the host vehicle;
An operation state estimation unit that estimates the operation state for each of the eye open state and the eye closed state of the driver based on the travel state detected by the state detection unit, or the travel state and the operation state;
Each of the operation states with respect to the open state and the closed state estimated by the operation state estimation unit is compared with the operation state detected by the state detection unit, and based on the comparison result, the opening / closing eye of the driver Open / closed eye state determining means for determining a state;
Opening and closing eye state determination device.   The open / closed eye state determination device according to claim 1, further comprising a doze determination unit that determines whether or not the driver is a doze based on time-series data of the open / closed eye state determined by the open / closed eye state determination unit.   The open / closed eye state determination device according to claim 1, further comprising a feature amount calculating unit that calculates a blink feature amount based on time-series data of the open / closed eye state determined by the open / closed eye state determining unit. The state detection means detects a yaw angular velocity, a lateral acceleration, and a vehicle speed as a running state of the host vehicle, and detects a steering angle as the operation state,
The operation state estimating means estimates a deviation from the target course at the front gaze point based on the detected yaw angular velocity, lateral acceleration, and vehicle speed, and according to the front gaze model, the target course at the estimated front gaze point 4. The steering angle for the closed eye state is estimated based on a deviation from the target course at the forward gazing point based on a deviation from the target eye, and a deviation from the target course at the front gazing point is set as a predetermined value. 5. Open / closed eye state determination device. The state detection means detects a yaw angular velocity, a lateral acceleration, and a vehicle speed as a running state of the host vehicle, and detects a steering angle as the operation state,
The operation state estimating means estimates the lateral position of the host vehicle with respect to the target course based on the detected yaw angular velocity, lateral acceleration, and vehicle speed, and opens the eye based on the lateral position of the host vehicle with respect to the estimated target course. The open / closed eye state determination device according to any one of claims 1 to 3, wherein a steering angle with respect to a state is estimated, and a steering angle with respect to a closed eye state is estimated with a lateral position of the host vehicle with respect to a target course as a predetermined value. An imaging means for imaging an area including the eyes of the driver;
Second open / close eye state determination means for determining the open / close eye state of the driver based on an image captured by the image capturing means;
6. The determination result integrating unit for determining the open / closed eye state of the driver by integrating the determination result by the open / closed eye state determining unit and the determination result by the second open / closed eye state determining unit. The open / closed eye state determination device according to claim 1. A state detecting means for detecting a traveling state of the host vehicle and an operation state when the driver operates the host vehicle;
Operation state estimation means for estimating the operation state for each of the forward-facing state and the lateral state of the driver based on the running state detected by the state detecting means, or the running state and the operation state;
Each of the operation states for the forward state and the lateral state estimated by the operation state estimation unit is compared with the operation state detected by the state detection unit, and based on the comparison result, the driver is in a forward state. A front side state determination means for determining whether the state is a sideways state or
Front sideways state determination device including   The front sideways state determination device according to claim 7, further comprising a sideward determination unit that determines whether or not the sideways state is based on time-series data of a determination result by the front sideways state determination unit. Computer
Based on the traveling state detected by the state detecting means for detecting the traveling state of the host vehicle and the operating state when the driver operates the host vehicle, or the open state and the closed state of the driver based on the traveling state and the operating state Operating state estimating means for estimating the operating state for each of the above, each of the operating states for the open and closed states estimated by the operating state estimating means, and the operating state detected by the state detecting means A program for functioning as an open / closed eye state determination unit that determines the open / closed eye state of the driver based on the comparison result. Computer
Based on the traveling state detected by the state detecting means for detecting the traveling state of the own vehicle and the operation state when the driver operates the own vehicle, or the forward state and the lateral state of the driver based on the traveling state and the operating state Operation state estimating means for estimating the operation state for each of the operation state, each of the operation states for the forward and lateral states estimated by the operation state estimation means, and the operation state detected by the state detection means And a function for determining whether the driver is in a forward-facing state or a lateral-facing state based on the comparison result.

Images