WO2012160251A2

WO2012160251A2 - A system, a detection unit, a method and a computer program for detecting passengers' seat belt use in a vehicle

Info

Publication number: WO2012160251A2
Application number: PCT/FI2012/050477
Authority: WO
Inventors: Maria Jokela; Matti Kutila; Jouko Viitanen; Pasi PYYKÖNEN
Original assignee: VTT Technical Research Centre of Finland Ltd
Current assignee: VTT Technical Research Centre of Finland Ltd
Priority date: 2011-05-23
Filing date: 2012-05-21
Publication date: 2012-11-29
Anticipated expiration: 2013-11-23
Also published as: FI20115502A0; WO2012160251A3

Abstract

The invention relates to a roadside system for detecting passengers' seat belt use from a grayscale image of a vehicle. The invention also relates to a detection unit, to a computer program and to a method for detecting passengers' seat belt use. The detection is based on algorithm utilizing image processing techniques and thus provides more reliable detection compared to the related technology.

Description

A SYSTEM, A DETECTION UNIT, A METHOD AND A COMPUTER PROGRAM FOR DETECTING PASSENGERS' SEAT BELT USE IN A VEHICLE Field of the Invention

This application relates to a solution for monitoring seat belt usage. In particular, the invention concerns a system, a detection unit, a method and a computer program for detecting passengers' seat belt use in a vehicle.

Background of the Invention

It has been studied that 17% of all traffic deaths in Europe are related to non- use of seat belts. In some countries the number is even higher. Because of this, effort has been put to improve traffic enforcement and consequently traffic safety. One of the priority policies is to raise the seat belt usage in European Union member states. By monitoring the seat belt usage automatically, the threat of being punished for not wearing them is evoked. This way, even the most stubborn drivers are being driven to wear seat belts that might save their lives.

Today's vehicles often have a built-in alert system which notifies the passengers if the seat belt is not locked. Because these systems provide information on seat belt usage to the passengers only, the systems may be easily ignored or even cheated. Moreover, the information is not available for the authorities in charge of supervising seat belt usage and therefore, cannot be utilised for enforcement purposes.

An example of an automatic passenger counting system is disclosed on "High Occupancy Vehicle Detection" by Perez-Jimenez, Guardiola and Perez-Cortes (Lecture Notes in Computer Science, 2008, Volume 5342/2008). The system is a real-time passenger detection system, where the passenger detection is based on analysis of a visual image. In the system, each person is detected by mixing information from different types of classifiers. One of the classifiers is a seat belt detector that is used for detecting a person if the person's face is hidden, e.g. behind a car sunshade. The seat belt detection is carried out by comparing the captured image to a set of images (e.g. a set of 200 images) which have been trained to the system. The images in the set of images have been labelled manually to indicate the seat belt positions, which information is utilized when seat belts in the monitored vehicle is aimed to be detected. The system outputs information on whether a seat belt is used, which further indicates the presence of a person.

Development of the automatic roadside monitoring systems for seat belts have been considered difficult and even impossible, because these systems are required to have a reliability of 99,9%. Especially the related technology lacks an algorithm-based monitoring system for seat belt usage that is also capable of giving information to the enforcement authorities.

Summary of the Invention

According to a first aspect of the invention, a system is provided for detecting passengers' seat belt use from a grayscale image of a vehicle. The system comprises detecting means configured to define one or more regions of interest in the image of the vehicle, a region of interest corresponding a passenger inside the vehicle; processing means configured to form an edge image of each of the region of interest, and to segment said edges in the edge image into line segments, and to select line segments having an angle within a predefined limits, said predefined limits corresponding angles of the seat belts in use.

According to a second aspect of the invention, a detection unit is provided for detecting passengers' seat belt use from a grayscale image of a vehicle. The detection unit is configured to define one or more regions of interest in the image of the vehicle, a region of interest corresponding a passenger inside the vehicle, to form an edge image of each of the region of interest, and to segment said edges in the edge image into line segments, and to select line segments having an angle within a predefined limits, said predefined limits corresponding angles of the seat belts in use. According to a third aspect of the invention, a computer program is provided for detecting passengers' seat belt use from a grayscale image of a vehicle. The program is adapted to perform, when run on a device, the steps of defining one or more regions of interest in the image of the vehicle, a region of interest corresponding a passenger inside the vehicle, forming an edge image of each of the region of interest, and segmenting said edges in the edge image into line segments, and selecting line segments having an angle within a predefined limits, said predefined limits corresponding angles of the seat belts in use.

According to a fourth aspect of the invention, a method is provided for detecting passengers' seat belt use from a grayscale image of a vehicle. The method comprises steps for defining one or more regions of interest in the image of the vehicle, a region of interest corresponding a passenger inside the vehicle, forming an edge image of each of the region of interest, segmenting said edges in the edge image into line segments, and selecting line segments having an angle within a predefined limits, said predefined limits corresponding angles of the seat belts in use.

According to different embodiments of the invention, the system, the detection unit, the method and the computer program can be used to select - from a group of the selected line segments - line segment pairs which are located within a predefined distance from each others, said predefined distance corresponding a width of the seat belt. The system, the detection unit, the method and the computer program may also be used to determine difference in mean gray values from a grayscale image between a background and selected line segment pair, whereby when the difference is greater than a pre-defined limit, the processing means is configured to deduce the seat belt usage. The system, the detection unit, the method and the computer program may also be used to process the image to provide a histogram equalized image before detecting the region of interest. The system, the detection unit, the method and the computer program may also be used to locate the vehicle on the image. The system, the detection unit, the method and the computer program may also be used to detect a windshield of the vehicle by thresholding high intensity areas from a grayscale image of the vehicle to create a mask image and by removing small objects form the mask image; and segmenting means configured to segment the windshield from the grayscale image by using the mask image. The system, the detection unit, the method and the computer program may also be used to detect passengers inside the vehicle by forming a histogram equalized image for the segmented windshield; dividing windshield in two or more sections; and for each section: thresholding high contrast areas inside the windshield to find high contrast areas; removing small objects; finding largest area from the mask image to locate passenger's face; using the mask to define region of interest in the original image. The system, the detection unit, the method and the computer program may also be used to remove small objects by morphological erode operation. The system, the detection unit, the method and the computer program may also be used to detect windshield from the mask image by finding the largest area above a license plate of the vehicle. The system, the detection unit, the method and the computer program may also be used to receive the vehicle image from a road side camera. The system, the detection unit, the method and the computer program may also be used to capture an image of the vehicle. The system, the detection unit, the method and the computer program may also be used to determine owner-related information based on the vehicle's license plate. The system, the detection unit, the method and the computer program may also be used to transmit owner-related information to an authority. To achieve the aim of the invention, the system according to the invention is primarily characterized in what is presented in the appended claim 1 . The detection unit according to the invention is primarily characterized in what is presented in the attached claim 14. The computer program according to the invention is primarily characterized in what is presented in the attached claim 23. The method according to the invention is primarily characterized in what is presented in the attached claim 24. The dependent claims will describe other embodiments of the invention.

The present solution is an algorithm-based monitoring system that may also provide information on the seat belt usage to the authorities. The system does not need to be trained in advance, which makes it more reliable compared to the solutions of related art. In addition, the present solution provides means to retrieve owner-related data based on vehicle's identity information (i.e. license plate number) and - by this - to make police officers' work easier by pre-sorting the vehicles which driver's ignore seat belt usage. These and other advantages and features of the invention, together with the organization and manner of operation thereof, will become apparent from the following detailed description when taken in conjunction with the accompanying drawing.

Description of the Drawings

Figure 1 illustrates an example of equipment arranged for the present solution,

Figures 2a— 2b illustrate an example of an original image received from the road side camera and the original image after preprocessing,

Figures 3a— 3e illustrate example of images where a windshield is being detected,

Figures 4a— 4b illustrate example of images where a passenger is being detected,

Figures 5a— 5c illustrate example of images where a seat belt is being detected, Figure 6 is a general example of the algorithm for detecting seat belt usage,

Figure 7 is an example of a function for detecting windshield, Figure 8 is an example of a function for detecting passenger(s), and

Figure 9 is an example of a function for detecting a seat belt. Detailed Description of the Invention In the following an example of an automatic detection of seat belt (i.e. safety belt) compliance is described. Figure 1 illustrates an example of the equipment that is arranged for the present solution. The system 120 for monitoring seat belt usage may operate with an roadside camera equipment 1 10 including a high resolution camera for monitoring vehicles 100 in front and sufficient illumination for obtaining images with feasible quality. Roadside camera systems 1 10 are known to monitor the passing vehicles 100 and to provide information of every one of them, e.g. a license plate number, a location and dimensions of the vehicle. One example of the roadside camera system is T-EXSPEED -system of Kria S.r.l. that provides a 2-dimensional image, which shows the location of the license plate and the dimensions of the vehicle. Instead of operating with an external roadside camera, the system 120, itself, may comprise a camera equipment to capture an image of a passing car. The system 120 can be placed on roadside with the camera 1 10 faced towards the vehicles 100 so that windscreen and interior of the vehicle 100 is in the camera view. The seat belt detection unit 122 (later called as "detection unit") is configured to use such an image processing technique (to be discussed later), by which an image of a windscreen area of the vehicle can be isolated. From this image, the detection unit 122 is capable of detecting passengers, and - by means of a seat belt detection algorithm - a seat belt when worn by a driver and/or one or more passengers. The seat belt detection is based on gradient search from a segmented area in the passenger's body. If a line having an angle corresponding the range of seat belt angles is detected, the seat belt detection algorithm may provide a positive detection. However, it is also possible to continue the detection procedure (and to improve the reliability of the detection) by looking for two lines matching one of the known seat belt widths, and when found, the seat belt detection algorithm may provide a positive detection. It is to be noted that in the present description, term "passenger" is used as a general term covering both the driver and the passengers. In case of offences, i.e. the detection unit has not found any probable seat belt locations, the system is able to retrieve 124 owner-related data of the vehicle based on the license plate number shown in the image. This information could be turned into a notification 126 and sent to the authorities 130 for verification and punishment.

The example of the solution is now described in more detailed manner by means of figures 2— 5. The detection unit of the monitoring system may receive an image (Figure 2a) from the roadside camera. The size of the image can be 1360 x 1024 and it has been taken of a passing car. The image received from the roadside camera may be a colour image, which is transformed into a grayscale image in the detection unit. The image may go through a pre-processing, where contrast-limited adaptive histogram equalization is applied to the grayscale image. The image after the histogram equalization is shown in Figure 2b.

The pre-processing is followed by a windshield detection. One way for carrying it out, is to segment (Figure 3a) a vehicle from the equalized image based on location of the license plate and dimensions of the vehicle. The image on the segmented vehicle can be used for thresholding high intensity areas from the equalized image in vertical dimension to create an image mask. The resulted image is shown in Figure 3b. The mask image (shown in Fig. 3b) can then be affected with morphological erode operation to remove small objects, which results in as an image shown in Figure 3c. Now, it is possible to detect windshield. This can be done by finding the largest light area above the license plate from the image (Figure 3d). In order to segment the windshield from the grayscale image (Figure 3e), the previously created (Fig. 3b) image mask can be used. The image on the segmented windshield can be used for locating front-seat passengers inside the vehicle. This may be started with contrast-limited adaptive histogram equalization for the windshield area in the grayscale image. The image on the windshield area after histogram equalization is shown in Figure 4a. The windshield may then be divided in as many sections are there are passengers (including the driver) in the front seat. Typically the windshield is divided in two sections - left and right - but in some situations the windshield may be divided into three sections. A van or a pick-up is an example of a car enabling three passengers in a front seat. It is appreciated that the present solution is capable of detecting three-point safety belts, and therefore the passenger sitting in the middle should have one instead of a pelvic seat belt.

The driver section of the divided windshield is shown in Figure 4b. Each of the sections then goes through the passenger detection steps to determine on which seats the seat belt should be used. The passenger detection may start with thresholding high contrast areas inside the windshield to find contrast areas (Fig. 4c). Then morphological erode operation may be used to remove small objects. Next passenger's face can be located by finding a largest area from the mask image. The mask image can also be used for defining a region of interest in the grayscale image. In this context the region of interest contains a face and an upper body of the passenger.

The seat belt is then looked for from each region of interest. The seat belt detection start with edge detection that is applied to each region of interest. The edge detection can be carried out by any known algorithm, e.g. Canny edge detection algorithm. The resulted edge image is shown in Figure 5a. The found edges are linked and segmented to produce a set of line segments as shown in Figure 5b. The purpose of the edge linking and segmentation is to find line junctions and endings in the edge image and to form straight line segments into the image. One example of edge linking and segmentation are Peter Kovesi's methods.

Next, angle for each line segment is determined, and such line segments are accepted and selected whose angle is within predefined limits. The predefined limit relates to a range of probable seat belt angles. The predefined limit can be between -65° and -35° for the driver. The predefined limit can, however, vary from [-70°, -30°]. In order to improve the reliability of the detection, the selected line segments can then be used for comparing them with the other line segments so that line segment pairs within a predefined limits can be accepted and selected. The pre-defined limit relates to the distance between the line segment pair and that can be between 4.5 pixels and 10 pixels. Differences in mean gray values can then be determined between the background and the probable seat belt (being defined by line pair). Only those images are determined to show a seat belt usage, where the difference is greater than a predefined limit. An example of the predefined limit is 20 (±5). The result of the seat belt detection is shown in Figure 5c.

Figure 6 illustrates an example of the algorithm for the present solution in a general level. In this example, the algorithm comprises steps for receiving an image (600), pre-processing the image (610), locating a vehicle and a windshield from the image (620), locating passengers in the vehicle (630) and detecting seat belt of the passengers (640). In the pre-processing (610) a contrast-limited adaptive histogram equalization is applied to the image. Function for locating a vehicle and a windshield is illustrated by Figure 7. The function starts with segmenting the vehicle from the image (710). The segmentation can be performed by locating the license plate and the dimensions of the vehicle. The resulted image (containing the segmented vehicle) may go through a thresholding (720), where high intensity areas from grayscale image are thresholded in vertical dimension to create a mask image. Then small objects can be removed (730) from the mask image by means of morphological erode operation. As a result of this, the windshield can be detected (740) and segmented from the original image (750).

Function for locating passengers in the vehicle is illustrated in Figure 8. The function may start with contrast-limited adaptive histogram equalization for the windshield area (810). Then the windshield can be divided in two sections (820) to separate possible locations for the passengers. Then a section may go through a thresholding (830) of high intensity areas inside the windshield to find high intensity areas. This can be continued by removing small objects (840) by morphological erode operation. Passenger's face can be located (850) next, by finding a largest area from the mask image. A region of interest (face and upper body) can then be defined (860). This can be done by using the mask image. The procedure 830— 860 may be repeated for all sections (870, 880), so that each passenger is detected in the vehicle. In a situation, where passenger's face is hidden or is not easily recognizable from the interior of the car, the region of interest cannot be defined as accurately as in a situation, where face is detected. In that case the region of interest is the other half (or one of the parts) of the windshield. Figure 9 illustrates a function for detecting seat belt of the passengers. This is done by processing the region of interest(s) being defined in the function of Figure 8. At first the image of region of interest is processed to produce an edge image (910). The edge image then goes through an edge linking and segmenting (920) to provide an image with line segments. Then such line segments are selected that has angle within predefined limits (930). By this, line segments resembling the angle of the seat belt can be identified. This can be continued with selecting line segment pairs that are close enough to each other (940). As a result of this, it is possible to identify any line segment pair that resembles width of the seat belt. The last step (950) in this example is to detect, which of the selected line segment pairs actually correspond the seat belts. The detection can be performed by calculating differences in mean gray values between the background and the probable seat belt in the original image and accept only those as an indication of the seat belt use, where difference is greater than the predefined limit. It is appreciated that where the difference is smaller and the same, the indication relates to non- use of the seat belts (offense). The steps 940 and 950 for selecting line segment pairs and calculating mean gray values are aimed for improving the classification results. In some cases, the seat belt detection can only be based on the determined angle of a line segment (as in 930).

In case of offence, i.e. the detection algorithm has not found any probable seat belt locations, the system may transmit this information to an authority or retrieve information on the owner of the vehicle via road infrastructure to vehicle communication channel or road side variable message sign.

In the above, the present solution for detecting seat belt usage has been described by means of an example of a system. It is appreciated that the system configuration is not limited to what has been presented, but may incorporate other devices like different type of cameras as well to enhance the system's performance.

The present invention is also described in the general context of method steps, which may be implemented in one embodiment by a program product including computer-executable instructions, such as program code. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of program code for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps. Software implementations of the present invention could be accomplished with standard programming techniques with rule based logic and other logic to accomplish the various steps Therefore it is realized, that the foregoing description of embodiments of the present invention have been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the present invention to the precise form disclosed, and modifications and variations are possible in light of the above teachings or may be acquired from practice of the present invention. The embodiments were chosen and described in order to explain the principles of the present invention and its practical application to enable one skilled in the art to utilize the present invention in various embodiments and with various modifications as are suited to the particular use contemplated.

Claims

:

A roadside system for detecting passengers' seat belt use from a grayscale image of a vehicle, the system comprising

- detecting means configured to define one or more regions of interest in the image of the vehicle, a region of interest corresponding a passenger inside the vehicle

characterized in that the system comprises processing means configured

- to form an edge image of each of the region of interest, and to segment said edges in the edge image into line segments, and

- to select line segments having an angle within a predefined limits, said predefined limits corresponding angles of the seat belts in use.

A system according to claim 1 , wherein the processing means is further configured to select - from a group of the selected line segments - line segment pairs which are located within a predefined distance from each others, said predefined distance corresponding a width of the seat belt.

A system according to claim 2, wherein the processing means is further configured to determine difference in mean gray values between a background and selected line segment pair, whereby when the difference is greater than a pre-defined limit, the processing means is configured to deduce the seat belt usage.

A system according to any claim 1 to 3, wherein the system further comprises pre-processing means configured to process the image to provide a histogram equalized image before detecting the region of interest.

A system according to claim 4, wherein the system further comprises segmenting means configured to locate the vehicle on the image.

A system according to claim 5, wherein the system further comprises detecting means configured to detect a windshield of the vehicle by thresholding high intensity areas from a grayscale image of the vehicle to create a mask image and by removing small objects form the mask image; and segmenting means configured to segment the windshield from the grayscale image by using the mask image.

7. A system according to claim 6, wherein the system comprises detecting means configured to detect passengers inside the vehicle by forming a histogram equalized image for the segmented windshield; dividing windshield in two sections; and for each section: thresholding high contrast areas inside the windshield to find high contrast areas; removing small objects; finding largest area from the mask image to locate passenger's face; using the mask to define region of interest in the original image 8. A system of claim 6 or 7, where processing means is configured to remove small objects by morphological erode operation.

9. A system of claim 6 or 7, wherein the detecting means configured to detect windshield from the mask image is configured to find the largest area above a license plate of the vehicle.

10. A system according to any claim 1 to 9, where the system is configured to receive the grayscale image from a road side camera. 1 1 . A system according to any claim 1 to 9, where the system comprises means for capturing an image of the vehicle.

12. A system according to any claim 1 to 1 1 , where the system is configured to determine owner-related information based on vehicle's license plate.

13. A system according to claim 12, where the system is configured to transmit owner-related information to an authority. 14. A detection unit for detecting passengers' seat belt use from a grayscale image of a vehicle, the detection unit being configured - to define one or more regions of interest in the image of the vehicle, a region of interest corresponding a passenger inside the vehicle

characterized in that the detection unit is further configured

15. A detection unit according to claim 14, being further configured to select - from a group of the selected line segments - line segment pairs which are located within a predefined distance from each others, said predefined distance corresponding a width of the seat belt.

16. A detection unit according to claim 15, being further configured to determine difference in mean gray values between a background and selected line segment pair, whereby when the difference is greater than a pre-defined limit, the processing means is configured to deduce the seat belt usage.

17. A detection unit according to any claim 14 to 16, being further configured to process the image to provide a histogram equalized image before detecting the region of interest.

18. A detection unit according to claim 17, being further configured to locate the vehicle on the image.

19. A detection unit according to claim 18, being further configured to detect a windshield of the vehicle by thresholding high intensity areas from a grayscale image of the vehicle to create a mask image and by removing small objects form the mask image; and segmenting means configured to segment the windshield from the grayscale image by using the mask image.

20. A detection unit according to claim 19, being further configured to detect passengers inside the vehicle by forming a histogram equalized image for the segmented windshield; dividing windshield in two sections; and for each section: thresholding high contrast areas inside the windshield to find high contrast areas; removing small objects; finding largest area from the mask image to locate passenger's face; using the mask to define region of interest in the original image

21 . A detection unit according to claim 19 and 20, being further configured to remove small objects by morphological erode operation.

22. A detection unit according to claim 19 and 20, being further configured to detect windshield from the mask image is configured to find the largest area above a license plate of the vehicle.

23. A computer program for detecting passengers' seat belt use from a grayscale image of a vehicle, said program being adapted to perform, when run on a device, the steps of

- defining one or more regions of interest in the image of the vehicle, a region of interest corresponding a passenger inside the vehicle characterized in that said program is further being adapted to perform,

- forming an edge image of each of the region of interest, and segmenting said edges in the edge image into line segments, and

- selecting line segments having an angle within a predefined limits, said predefined limits corresponding angles of the seat belts in use.

24. A method for detecting passengers' seat belt use from a grayscale image of a vehicle, the method comprising steps for

- defining one or more regions of interest in the image of the vehicle, a region of interest corresponding a passenger inside the vehicle

characterized in that the method further comprises

- forming an edge image of each of the region of interest, and segmenting said edges in the edge image into line segments, and - selecting line segments having an angle within a predefined limits, said predefined limits corresponding angles of the seat belts in use. 25. A method according to claim 24, further comprising selecting - from a group of the selected line segments - line segment pairs which are located within a predefined distance from each others, said predefined distance corresponding a width of the seat belt. 26. A method according to claim 25, further comprising determining difference in mean gray values between a background and selected line segment pair, whereby when the difference is greater than a predefined limit, the seat belt usage is deduced.