KR101821496B1 - safe driving support system operating method - Google Patents

Abstract

The present invention relates to a safe driving support system and an operating method thereof. The safe driving support system comprises: a camera (100) which is installed inside a vehicle to recognize a driver (10) and a pedestrian (20) at the same time and photographs image data; a single board computer (200) which has a server for analyzing the image data of the camera (100) through an image processing algorithm; and a smart device (300) which receives a signal from the single board computer (200) and informs the driver (10). The image data acquired through the camera (100) installed inside the vehicle to recognize the driver (10) and the pedestrian (20) at the same time is transmitted to the server of the single board computer (200). The server transmits information to the smart device (300) and informs the driver (10) through a UI, sound, or vibration in the smart device (300), if the server determines that there is a dangerous situation based on the analyzed data through the image processing algorithm. The safe driving support system and the operating method thereof according to the present invention recognize the driver and the pedestrian at the same time through two or more camera sensors. One or more camera sensors recognize the face and eyes of the driver. The other one or more camera sensors recognize the forward pedestrian. Therefore, it is possible to prevent a pedestrian accident and a drowsy driving accident in advance by supporting the driver in a way of informing the driver to avoid a dangerous situation when the driver cannot concentrate on driving through the gaze of the driver, or the pedestrian is in front.

Description

[0001] The present invention relates to a safe driving support system operating method,

More particularly, the present invention relates to a method of operating a safe driving assist system, and more particularly, to a method of operating a safe driving assist system by recognizing a driver and a pedestrian through two or more camera sensors, The sensor recognizes the pedestrian in front and informs the driver to avoid the dangerous situation when the driver can not concentrate on driving through the driver's gaze or if the pedestrian is in front of the driver so that the pedestrian accident and the sleepy driving accident And more particularly, to a method of operating a safe driving assist system which can prevent a driving assist system from occurring.

Generally, an accident occurs when a sleeping driving accident occurs when a driver drives a drowsy driver, or when a driver can not concentrate on driving, for example, a pedestrian accident occurs when the driver does not watch ahead.

In the pedestrian accidents, the place where accidents occur frequently is the school zone.

In the school zone, the accident rate accounted for 58% of the total pedestrian accident rate, and the incidence rate of children aged 7 years old, who entered elementary school, was the highest at 13.2%.

The pedestrian accident is an accident because the driver can not recognize when the pedestrian appears because the driver does not pay attention to the front.

Also, the drowsiness driving accident often occurs. However, the driver is a short time but the speed of the vehicle not controlled by the driver is maintained or the driver unconsciously presses the accelerator more frequently, thereby increasing the speed of the vehicle.

At this time, although the vehicle is running for a short time, it can become a serious threat to a car or pedestrian around the vehicle.

In fact, if a driver loses about 5 seconds, a vehicle with a speed of 60 km per hour is about 80 m, and a speed of 100 km per hour is about 140 m without a driver.

Therefore, it is necessary to be able to avoid dangerous situations that occur when the driver can not concentrate on driving.

As a conventional art, the drowsiness operation alarm device of Registration Practical Utility Model Registration No. 20-0418306 includes a housing having fixing means capable of being attached to a cap; A PCB secured within the housing; A motion sensor mounted on the PCB; A microprocessor mounted on the PCB; An alarm device mounted on the housing; A control program for generating an alarm when a wearer's movement is continuously sensed from a motion sensor and the movement of the wearer is determined to be a characteristic movement of a drowsy operation other than a normal movement is incorporated in a microprocessor Prevention alarm device.

As another prior art, a pedestrian recognition-based accident prevention system and control method thereof of Patent Registration No. 10-1344034 includes a camera for photographing at least one first image; Detecting at least one first pedestrian included in the first image and tracking the detected first pedestrian when the first image includes the first zone, A control unit for determining whether or not the at least one portion of the first zone is located; An output unit for outputting first information informing the first pedestrian of the danger under the control of the control unit when the tracked first pedestrian is located at least a part of the first zone; Location information module; Wherein the controller estimates a feature vector for the first image and detects the first pedestrian using an adaboost algorithm and an SVM algorithm to which the estimated feature vector is applied, Wherein the control unit tracks the first pedestrian by applying a mean shift algorithm to the detected first pedestrian, the control unit detects at least one first vehicle included in the first image, The first pedestrian and the first vehicle are detected together and the collision between the first pedestrian and the first vehicle is judged and when it is judged that the first pedestrian and the first vehicle have collided with each other, Information on the collision position information and the collision information through the wireless communication unit, Characterized in that the control to be transmitted, it is described that the security system via the pedestrian recognition.

However, in order to prevent the driver from drowsy, the driver of the related art as described above must be worn by a driver or placed in contact with a user so as to be manufactured in a proper size, and not only inconvenient at the time of use but also be determined by direction, speed and time The camera installed so that it can not accurately determine the drowsy state of the driver and the camera which is installed so as not to reach the driver detects obstacles and pedestrians in front of the vehicle and focuses on the driving because the driver is sleeping or talking with another person, It can not be confirmed that there is a high risk of accidents.

Accordingly, the driver and the pedestrian are simultaneously recognized through two or more camera sensors, one or more camera sensors recognize the face and eyes of the driver through the safety driving assist system and the operation method of the present invention, The driver is notified that the driver can not concentrate on the driving through the driver's gaze or if the driver is in front of the driver, the driver is informed to avoid the dangerous situation so that the driver can be assisted to prevent the pedestrian accident and the sleepy driving accident And a method of operating the same.

The safe driving assist system of the present invention includes a camera 100 for recognizing a driver 10 and a pedestrian 20 at the same time and photographing image data; A single board computer 200 provided with a server 210 for analyzing image data of the camera 100 through an image processing algorithm; A smart device 300 that receives a signal from the single board computer 200 and informs the driver 10; .

The method of operating the safe driving assist system according to the present invention is a method of operating the safe driving assist system in which the image data acquired through the camera 100 installed in the vehicle is recognized by the server (not shown) of the single board computer 200 so as to simultaneously recognize the driver 10 and the pedestrian 20 210, and the server 210 transmits information to the smart device 300 when the risk is determined based on the analyzed data through the image processing algorithm, and transmits the UI, sound, Or to notify the driver 10 through vibration.

The safety driving assist system and the operation method of the present invention simultaneously recognize a driver and a pedestrian through two or more camera sensors, one or more camera sensors recognize a driver's face and eyes, and the other one or more camera sensors detect a frontal pedestrian By recognizing, it is possible to prevent a pedestrian accident and a sleepy driving accident by informing the driver to avoid a dangerous situation when the driver can not concentrate on driving through the driver's gaze or if the pedestrian is ahead There is a remarkable effect.

1 is a block diagram of the present invention
2 is a perspective view of the raspberry pie of the present invention
FIG. 3 is a conceptual diagram of face recognition using the Haar Cascade of the present invention
4 is a conceptual diagram illustrating UI output when it is determined that the drowsy driving of the present invention
FIG. 5 is a conceptual diagram of facial recognition when turning the head using the Haar Cascade of the present invention
6 is a conceptual diagram for recognizing a pedestrian using OpenCV of the present invention

The safe driving assist system of the present invention includes a camera 100 for recognizing a driver 10 and a pedestrian 20 at the same time and photographing image data; A single board computer 200 provided with a server 210 for analyzing image data of the camera 100 through an image processing algorithm; A smart device 300 that receives a signal from the single board computer 200 and informs the driver 10; .

Also, the camera 100 may use a multi-directional sensing camera or a plurality of cameras so that the driver 10 and the pedestrian 20 can be simultaneously recognized.

The method of operating the safe driving assist system of the present invention is a method of operating the safe driving assist system in which the image data acquired through the camera 100 installed in the vehicle to recognize the driver 10 and the pedestrian 20 simultaneously in the vehicle is transmitted to the server 210 of the single board computer 200 The server 210 transmits information to the smart device 300 and transmits the UI, sound, or sound to the smart device 300 when it is determined that the risk is based on the data analyzed through the image processing algorithm And informs the driver (10) through the vibration.

In addition, the server 210 is installed in the single board computer 200 and analyzes the image data through an image processing algorithm.

Also, the single board computer 200 uses JAVA language.

In addition, the server 210 may determine that the mobile terminal 100 is in a dangerous state when one or more of the situations in which the eyes are not recognized or the face region is not detected through the image processing algorithm is determined to be a dangerous situation, Value is transmitted.

The present invention will be described in detail with reference to the accompanying drawings.

2 is a perspective view of the raspberry pie of the present invention, Fig. 3 is a conceptual view of face recognition using the Haar Cascade of the present invention, Fig. 4 is a view showing a UI FIG. 5 is a conceptual diagram of facial recognition when turning the head using the Haar Cascade of the present invention, and FIG. 6 is a conceptual diagram of pedestrian recognition using OpenCV of the present invention.

The present invention will now be described more fully with reference to the accompanying drawings, in which: FIG. 1 is a block diagram of a camera system according to a first embodiment of the present invention; A single board computer 200 provided with a server 210 for analyzing image data of the camera 100 through an image processing algorithm; A smart device 300 that receives a signal from the single board computer 200 and informs the driver 10; .

In order for the server 210 to transmit information to the smart device, a wireless communication method such as Bluetooth or a wired communication method using a connection through a connection may be used.

The camera 100 uses a multi-directional sensing camera or a plurality of cameras so that the driver 10 and the pedestrian 20 can be simultaneously recognized.

The multi-directional sensing camera has a plurality of camera sensors built into one camera to photograph the surroundings.

In particular, the multi-directional sensing camera may use a 360 degree camera.

The plurality of cameras are installed so that at least one of the cameras recognizes the pedestrian 20 in front of the vehicle, and one or more cameras are installed to recognize the face and the eyes of the driver 10. [

Therefore, the plurality of cameras use two or more cameras, and as the number of installed cameras increases, it becomes possible to determine more accurately.

The image data acquired through the camera 100 is transmitted to the server 210 of the single board computer 200 and the server 210 is installed in the single board computer 200 itself to perform an image processing algorithm Or transmitted to an external device installed in the server 210 through the single board computer 200 and analyzed by the server 210 installed in an external device through the image processing algorithm.

The single board computer 200 may use raspberry pie.

The raspberry pie is a single board computer used for educational purposes. It is inexpensive, small in size, and different in weight and price depending on the model, but generally has a length or width of less than 10 cm and a weight of less than 100 g.

The raspberry pie is suitable for use in a space where a communication unit, a control unit, a power supply unit, a connector, a pin, and the like are provided on a single board, a size of a credit card is small, and a space is limited.

The single board computer 200 can connect a separate input / output device to the connector, and a plurality of pins can be connected to each of the pins to couple a sensor or module to the input / output device through the wireless communication module. You can also connect wirelessly.

The single board computer 200 uses software that can be easily used with hardware and a simple interface, and is widely used in video communication.

Therefore, the single board computer 200 is easy to carry because it is light in weight, has a small volume, is easy to install even in a narrow space, and is suitable for installation in a vehicle having limited space because it does not obstruct the view.

The server 210 transmits information to the smart device 300 and transmits the information to the driver 300 through the UI, sound, or vibration in the smart device 300 10).

When the smart device 300 receives the information, the smart device 300 displays it on the screen through an app or informs the driver 10 of a sound, vibration, etc. including a warning sound.

In order to transfer image data from the camera 100 to the server 210, a JAVA-based single board computer 200 is used.

The server 210 determines the direction and the face of the driver 10 based on the image data photographed by the camera 100 and performs an event process.

More specifically, the server 210 first recognizes the face portion of the driver 10 from the image data, and intensively detects the angle and pupils of the head.

In particular, the server 210 determines Haar features when detecting eyes, and then recognizes eyes, nose, mouth, face area, and the like using a cascade classifier.

The harsh feature is to obtain a pattern using the fact that the eyes are dark and the nose is bright.

Therefore, the cascade classifier can grasp the eyes, nose, mouth, face region, etc. through the pattern obtained through the Haar feature.

When the server 210 receives the image of the eyes of the driver 10 as the image data, the image processing algorithm does not detect the eye through the Haar feature from the image data, so that the eye 210 can not be recognized.

Therefore, the server 210 generates an algorithm for determining that the eyes of the driver 10 are not covered by the eyes of the driver 10.

As shown in FIG. 3, when the angle of the head of the driver 10 is lowered, the server 210 does not recognize the face area due to the hallmark, The algorithm is written to judge.

The server 210 judges that it is a dangerous situation when one or more of the situations where the eyes are not recognized or the face area is not detected through the image processing algorithm is detected as a dangerous state and the value is sent to the smart device 300 through the socket communication It will deliver.

The smart device receives the value and receives the current position of the driver 10 through the Google Map API in the event processing, searches for a resting place such as a drowsiness shelter or a resting place closest to the current position, And outputs the calculated distance to the smart device 300 as a UI (User Interface) as shown in FIG.

As an example of the UI output to the smart device 300, if the resting place is a drowsy shelter, the screen of the smart device 300 outputs a drowsiness shelter, and below it is displayed how many meters the distance to the sleeping shelter has remained And on the right side, the distance is changed in real time according to the change of the position of the vehicle, and the distance remained is indicated.

As shown in FIG. 4, the server 210 receives the image data of the driver 10 photographed by the camera 100, and then recognizes the face region and the eye region according to the Hall characteristic .

The server 210 recognizes the hairstyle according to a pattern such as eyes, nose, mouth, face, and the like, and then determines the eyes, nose, mouth, and face area in the categorizer through the recognized hairstyle.

The server 210 is slightly smaller than when the driver 10 turns his / her head when the face area is front, and the driver 10 turns his / her head to the right through the eyes of the opposite side of the head turning direction , Or to the left.

The recognition of the driver 10 turning his / her head is used as a result of recognizing the pedestrian 20 and used to sound a warning to the driver 10 to warn the pedestrian 20.

The camera 100 recognizes the pedestrian 20 using image data installed to detect the pedestrian 20 in front of the vehicle, and processes the event.

For example, the driver 10 may not be able to watch ahead while paying attention to some sight on the left or right side of the driver 10.

In particular, since the driver 10 is running slowly in an area where the speed of the vehicle such as a school zone or an alleyway is not high, he / she is often distracted and can not look forward and look around.

However, since the area where the speed of the vehicle such as the school zone or the alleyway is not high is an area where the pedestrian 20 is concentrated, an accident occurs due to the driver's intention.

Therefore, if the server 210 determines that the driver 10 is not looking forward but the pedestrian 20 appears, the driver 10 is warned by preventing the accident.

The server 210 uses a Histogram of oriented gradients (HOG) feature for an object ahead of the moving image of the image captured through the camera 100 and then performs a pedestrian recognition algorithm to determine that the moving object is the pedestrian 20 do.

The HOG feature generates a local histogram by determining a degree of change with respect to an object in a continuous image, and generates a one-dimensional vector by continuously connecting the local histogram.

Therefore, the server 210 uses the pedestrian recognition algorithm through the one-dimensional vector generated using the HOG feature, determines an object moving through the pedestrian recognition algorithm, and determines that the object is the pedestrian 20.

6 shows a result of recognition of the pedestrian 20 by performing the HOG feature on the pedestrian test data.

The server 210 notifies the driver 10 that the driver 10 can not recognize the pedestrian 20 through the sight line direction of the driver 10 and the appearance position of the pedestrian 20, The smart device 300 transmits a signal to the smart device 300 to warn the driver 10 through an alarm, a warning sound, or a vibration.

The server 210 recognizes that the pedestrian 20 has appeared on the right side as a result of checking through the image data. When the driver 10 turns to the left, 20) and notifies the smart device 300 of the signal.

A signal is received by the smart device 300 and an alarm is sounded to alert the driver 10.

The direction of the head of the driver 10 can be determined by the size of the face area and the recognition of the eyes or the difference of the distance between the center point of the face area and the center of the eyes.

For example, if there is no difference in the distance between the center point of the face area and the center of the eyes, it is determined that the driver 10 is facing the front, and the difference between the center point of the face area and the centers of the eyes If it is larger than the previous one, it can be recognized that the face is turned in one direction.

The smart device 300 may be used as a stand-up display, or may be used as a head-up display to emit an image on the windshield of a vehicle.

The smart device 300 is mounted on the smart device 300 so that the smart device 300 can be mounted on the smart device 300 separately from the smart device 300, 300) is visible to the driver.

The smart device 300 is used as a head-up display. The front glass of the vehicle uses a windshield constructed so that an image output from the smart device 300 can be seen by a driver. The dashboard is provided with a cradle or a mounting groove for mounting the smart device 300 for outputting an image so that the display unit faces the front glass of the vehicle.

The image output from the display unit of the smart device 300 is displayed on the display unit of the smart device 300 after the smart device 300 is mounted in the cradle or the mounting groove so that the display unit of the smart device 300 faces the vehicle front- It is reflected on the windshield of the vehicle and is visible to the driver.

In particular, when the image output from the display unit of the smart device 300 is reflected in the direction of the glass, the left and right inverted images are used so that the user can not see the inverted images.

Accordingly, the safe driving assist system and the operation method of the present invention simultaneously recognize the driver and the pedestrian through two or more camera sensors, one or more camera sensors recognize the driver's face and eyes, By informing the driver that the driver can not concentrate on the driving through the driver's eyes or avoid the dangerous situation when the pedestrian is in front of the driver, the driver is assisted to prevent the pedestrian accident and the sleepy driving accident There is a remarkable effect that can be.

10: Driver
20: Pedestrians
100: camera
200: Single board computer
210: Server
300: Smart device

Claims (5)

delete delete Video data acquired through the camera 100 installed inside the vehicle so as to simultaneously recognize the driver 10 and the pedestrian 20 inside the vehicle is transmitted to the server of the single board computer 200, A safety driving assistant system 300 for transmitting information to the smart device 300 and informing the driver 10 through UI, sound, or vibration in the smart device 300, In an operating method,
The camera 100 is installed to recognize the driver 10 and the pedestrian 20 at the same time in the vehicle and photographs the image data. The image data of the camera 100 is transmitted to the single board computer 200 through an image processing algorithm The smart device 300 receives a signal from the single board computer 200 and notifies the driver 10 of the received signal,
The single board computer 200 uses the JAVA language based image data to transmit the image data from the camera 100 to the server 210, and uses Raspberry pie. The Raspberry pie includes a communication unit, a control unit, A connector, and a pin, and the input / output device may be connected to the connector, and a plurality of pins may be provided to couple a sensor or a module to each pin, and among the modules, It can connect wirelessly to the input / output device,
In order to transmit information to the smart device from the server 210, a wireless communication method using a Bluetooth method or a wired communication method using a connection through a connection can be used.
The camera 100 uses a multi-directional sensing camera or a plurality of cameras so that the driver 10 and the pedestrian 20 can be simultaneously recognized. The multi-directional sensing camera includes a plurality of camera sensors The camera is installed so that at least one of the cameras recognizes the pedestrian 20 in front of the vehicle, and one or more cameras are installed to recognize the face and eyes of the driver 10 However,
The image data acquired through the camera 100 is transmitted to the server 210 of the single board computer 200 and the server 210 is installed in the single board computer 200 itself to perform an image processing algorithm Or transmitted to an external device equipped with the server 210 through the single board computer 200 and analyzed by the server 210 installed in the external device through the image processing algorithm,
The server 210 recognizes the facial portion of the driver 10 from the image data and intensively detects the angle and pupil of the head. The server 210 determines the Haar feature And then recognizing the eyes, nose, mouth, and face area using a cascade classifier, wherein the harsh feature is obtained by using the fact that the eyes are dark and the nose is bright and the cascade classifier The eye, nose, mouth, and face area can be grasped through the pattern obtained through the Haar feature,
The server 210 uses a pedestrian recognition algorithm using a one-dimensional vector generated using HOG (Histogram of oriented gradients) for an object in front of a moving image of an image photographed through the camera 100, Judges the moving object to be the pedestrian 20,
The HOG feature generates a local histogram by determining a degree of change with respect to an object in a continuous image, and generates a one-dimensional vector by continuously connecting the local histogram,
The server 210 processes the event by determining the direction and the face of the face of the driver 10 through the image data photographed by the camera 100. The server 210 recognizes the eyes through the image processing algorithm The face area is not detected, it is determined that it is a dangerous situation, and the value is transmitted to the smart device 300 through the socket communication,
The smart device 300 receives the value, receives the current position of the driver 10 through the Google Map API through event processing, searches for the closest place from the current position, calculates the distance from the current position to the resting place , And then outputting to the UI (User Interface)
The name of the resting place is displayed on the screen of the smart device 300 as a UI output to the smart device 300. The name of the resting place is displayed on the screen of the smart device 300, It changes the distance in real time according to the change of the position of the vehicle and informs the remaining distance,
The server 210 notifies the driver 10 that the driver 10 can not recognize the pedestrian 20 through the sight line direction of the driver 10 and the appearance position of the pedestrian 20, The smart device 300 transmits a signal to the smart device 300 to warn the driver 10 through sound or vibration,
The server 210 recognizes that the pedestrian 20 has appeared on the right side as a result of checking through the image data. When the driver 10 recognizes the pedestrian 20 when the head of the driver 10 is reversely turned to the left And transmits a signal to the smart device 300. The smart device 300 receives a signal and sounds an alarm to alert the driver 10,
The direction of the head of the driver 10 can be determined by the size of the face area and the recognition of the eyes or by the difference in the distance between the center point of the face area and the center of the eyes. If the distance between the center of the face area and the centers of the eyes is greater than the previous one, the face is turned in one direction And,
The smart device 300 can be used as a head-up display to mount or use the image on a front glass of a vehicle.
When the smart device 300 is used, the smart device 300 may be mounted on the smart device 300 separately installed in the vehicle so that the smart device 300 ) Is displayed to the driver,
When the smart device 300 is used as a head-up display, the front glass of the vehicle uses a windshield constructed so that an image output from the smart device 300 can be seen by the driver, The display unit of the smart device 300 may be mounted on the dashboard in a direction of the vehicle front glass so that the smart device 300 for outputting an image may be mounted on the display unit in the direction of the front glass of the vehicle, The image output from the display unit of the smart device 300 is reflected on the windshield of the vehicle on which the windshield is installed and is visible to the driver,
Wherein the image output from the display unit of the smart device (300) uses left and right inverted images so that the left and right inverted images are not visible to the user when reflected by the glass
delete delete

Images