US11538257B2

US11538257B2 - Detection, counting and identification of occupants in vehicles

Info

Publication number: US11538257B2
Application number: US16/154,130
Authority: US
Inventors: Robert Rozploch; Simon Shi; Marc Hansen; Jonathan Nazemi
Original assignee: Gatekeeper Inc
Current assignee: Gatekeeper Inc
Priority date: 2017-12-08
Filing date: 2018-10-08
Publication date: 2022-12-27
Anticipated expiration: 2038-10-08
Also published as: EP3721422A4; EP3721422A1; US20190180125A1; WO2019113427A1

Abstract

A method of detecting occupants in a vehicle includes detecting an oncoming vehicle and acquiring a plurality of images of occupants in the vehicle in response to detection of the vehicle. The method includes performing automated facial detection on the plurality of images and adding a facial image for each face detected to a gallery of facial images for the occupants of the vehicle. The method includes performing automated facial recognition on the gallery of facial images to group the facial images into groups based on which occupant is in the respective facial images, and counting the final group of unique facial images to determine how many occupants are in the vehicle.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority to U.S. Provisional Patent Application No. 62/596,497 filed Dec. 8, 2017, which is incorporated by reference herein in its entirety.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present disclosure relates to facial detection and recognition, and more particularly to facial detection and recognition for occupants in vehicles.

2. Description of Related Art

At security check points, border crossings, high occupancy vehicle (HOV) lanes, and the like it, is desirable to know how many occupants are in each vehicle that passes. At a traditional checkpoint an officer can count occupants that are visible in a vehicle. In security applications, it can be desirable to know who the occupants of a vehicle are. An officer can verify this by inspection of identification documents such as a photo ID for each occupant of the vehicle. However, these techniques require each vehicle to stop for inspection before passing through.

The conventional techniques have been considered satisfactory for their intended purpose. However, there is an ever present need for improved systems and methods for detecting, counting, and identifying occupants in vehicles. This disclosure provides a solution for this need.

SUMMARY OF THE INVENTION

A system for detecting occupants in a vehicle includes a controller and a plurality of camera systems external to the vehicle in a vehicle approach area, wherein each camera system is operatively connected to the controller. A trigger in the vehicle approach area is operatively connected to the controller to detect an approaching vehicle and control the camera systems to acquire images of the approaching vehicle. The controller includes machine readable instructions configured to cause the controller to perform any method as disclosed herein.

Each camera system can include an imaging sensor, a pulsed illumination device, and a processor operatively connecting the imaging sensor to the pulsed illumination source for synchronizing illumination a pulse from the pulsed illumination device with exposure of the imaging sensor. Each camera system can include a lens optically coupled to the imaging sensor, an optical bandpass filter operatively connected to filter light passing through the lens, and a linear polarization filter operatively connected to filter light passing through the lens.

A method of detecting occupants in a vehicle includes detecting an oncoming vehicle and acquiring a plurality of images of occupants in the vehicle in response to detection of the vehicle. The method includes performing automated facial detection on the plurality of images and adding a facial image for each face detected to a gallery of facial images for the occupants of the vehicle. The method includes performing automated facial recognition on the gallery of facial images to group the facial images into groups based on which occupant is in the respective facial images, and counting the groups to determine how many occupants are in the vehicle.

The method can include selecting a representative image from each group, and outputting a set of cropped selected images, one uniquely cropped selected image for each of the occupants. It is contemplated that no duplicate images of a given occupant need be stored or displayed. Selecting the representative image from each group can include selecting images based on corresponding confidence scores from the automated facial detection. Selecting the representative image from each group can include selecting images based on which image in the group has least facial offset angle from line of sight of an imaging sensor which acquired the respective image. The method can include running the selective images through a database to check for matches between the occupants and known individuals in the database. The method can include initiating a response upon finding a match in the database, wherein the response include at least one of outputting an alert on a visual display, sounding an audible alarm, closing a physical barrier, transmitting a citation, mailing a citation, and/or dispatching an officer. It is also contemplated that the method can include initiating a response upon determining an improper number of occupants in the vehicle, wherein the response includes at least one of outputting an alert on a visual display, sounding an audible alarm, closing a physical barrier, transmitting a citation, mailing a citation, and/or dispatching an officer.

Each image can be acquired from a different sensor viewing the vehicle from a different respective angle. The method can include illuminating the vehicle with a respective pulse of illumination for each image acquired, wherein each pulse of illumination is performed at a different time to reduce shadows cast onto the occupants while acquiring the plurality of images.

One of the sensors can be a primary sensor that acquires a primary image of occupants in the vehicle, wherein faces detected in primary image serve as references in the gallery for facial recognition for subsequent ones of the images of occupants in the vehicle. The method can include adding a new face to the gallery each time a detected face in a subsequent one of the images of occupants in the vehicle does not match with a face already in the gallery. The method can include iteratively comparing faces detected in subsequent ones of the images of occupants in the vehicle and adding each face detected to the gallery that is not already in the gallery until there is an image in the gallery of each face detected by performing automated facial detection.

These and other features of the systems and methods of the subject disclosure will become more readily apparent to those skilled in the art from the following detailed description of the preferred embodiments taken in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

So that those skilled in the art to which the subject disclosure appertains will readily understand how to make and use the devices and methods of the subject disclosure without undue experimentation, preferred embodiments thereof will be described in detail herein below with reference to certain figures, wherein:

FIG. 1 is a schematic side elevation view of an exemplary embodiment of a system constructed in accordance with the present disclosure, showing multiple camera systems with an approaching vehicle;

FIG. 2 is a schematic plan view of the system of FIG. 1 , showing the positions of the camera systems;

FIG. 3 is a schematic view of one of the camera systems of FIGS. 1 and 2 , showing the imaging sensor; and

FIG. 4 is a schematic view of a method in accordance with the subject disclosure, showing a process using images acquired by the system of FIG. 1 .

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Reference will now be made to the drawings wherein like reference numerals identify similar structural features or aspects of the subject disclosure. For purposes of explanation and illustration, and not limitation, a partial view of an exemplary embodiment of a system in accordance with the disclosure is shown in FIG. 1 and is designated generally by reference character 100. Other embodiments of systems in accordance with the disclosure, or aspects thereof, are provided in FIGS. 2-4 , as will be described. The systems and methods described herein can be used for automated counting and identification of occupants in vehicles.

The system 100 for detecting occupants in a vehicle 102 includes a controller 104 and a plurality of

camera systems

106, 108, and 110 that are external to the vehicle 105 in the vehicle approach area 112. Each

camera system

106, 108, and 110 is operatively connected to the controller 104. A trigger 114 in the vehicle approach area 112 is operatively connected to the controller 104 to detect an approaching vehicle 105 and to control the

camera systems

106, 108, and 110 to acquire images of the approaching vehicle 105. The controller 104 includes machine readable instructions configured to cause the controller 104 to perform any method as disclosed herein. As shown in FIGS. 1 and 2 , each

camera system

106, 108, and 110 is in a different location for acquiring images with sensors viewing the vehicle from different respective angles.

With reference now to FIG. 3 , camera system 106 includes an imaging sensor 116, a pulsed illumination device 118, and a processor 120 operatively connecting the imaging sensor 116 to the pulsed illumination source 118 for synchronizing an illumination pulse from the pulsed illumination device 118 with exposure of the imaging sensor 116. The illumination device 118 can be located on camera as in camera system 106 shown in FIG. 3 , or can be located off-camera as in

camera systems

108 and 110 shown in FIGS. 1-2 . The camera system 106 include a lens 122 optically coupled to the imaging sensor 116, an optical bandpass filter 124 operatively connected to filter light passing through the lens 122 to the imaging sensor 116. The camera system 106 also includes a linear polarization filter 126 operatively connected to filter light passing through the lens 122 to the sensor 116, e.g., to reduce glare from glass windshields and windows of the vehicle 105.

Imaging sensors

108 and 110 can include the same components as camera system 106.

With reference now to FIG. 4 , a method of detecting occupants in a vehicle includes detecting an oncoming vehicle, e.g., detecting oncoming vehicle 105 using trigger 114 as shown in FIG. 1 . When trigger 114 detects an oncoming vehicle 105, it signals the controller 104. Controller 104 then commands the

camera systems

106, 108, and 110 to acquire a plurality of images of occupants in the vehicle 105. Each

camera system

106, 108, and 110 can acquire a

respective image

128, 130, and 132, forming a set 200 of acquired images as shown in FIG. 4 . Controller 104 can illuminate the vehicle 105 with a respective pulse of illumination from each respective illumination device 118 for each image acquired, wherein each pulse of illumination is performed at a different time to reduce shadows cast onto the occupants while acquiring the plurality of images.

The method includes having controller 104 perform automated facial detection on the plurality of

images

128, 130, and 132, and to add a facial image for each face detected to a gallery 202 of facial images for the occupants of the vehicle 105. For the image 128, three faces are detected and four faces are detected from each of

images

130 and 132. Controller 104 performs automated facial recognition on the facial images of gallery 202 to group the facial images into

groups

134, 136, 138, and 140 based on which occupant is in the respective facial images, as indicated by facial recognition groupings 204 in FIG. 4 . While multiple images are shown in FIG. 4 in each of the

groups

134, 136, 138, and 140, it should be understood that the

groups

134, 136, 138, and 140 need not ever actually contain multiple images in each group. For example during facial recognition, each time a new image of a given individual is identified, controller 104 can decide whether the new image is better than the previous best image of the individual (based on facial detection confidence scores, facial offset angle, or the like, as described below), and if so the new image replaces the previous image in the respective group. In this way each

group

134, 136, 138, and 140 only ever includes one image.

Facial detection and facial recognition need not necessarily be performed one after another, but instead can be performed together on the fly. One of the sensors 120 can be a primary sensor, e.g., the sensor 120 of camera system 106, that acquires a primary image, e.g., image 128, of occupants in the vehicle 105. The faces detected in primary image 128 can serve as references in the gallery 202 for facial recognition for subsequent ones of the

images

130 and 132 of occupants in the vehicle. The controller 104 can add a new face to the gallery 202 each time a detected face in a subsequent one of the

images

130 and 132 does not match with a face already in the gallery 202. The controller 104 can iteratively compare faces detected in subsequent ones of the

images

128, 130, and 132 and add each face detected to the gallery 202 that is not already in the gallery 202 until there is an image in the gallery 202 of each face detected by performing automated facial detection.

Whenever a face is detected for which there is already an image in the gallery 202, the best image of the face can be retained in the image gallery 202. Controller 104 selects a

representative image

142, 144, 146, and 148 from each

group

134, 136, 138, and 145 and can output a set 206 of cropped selected images, one uniquely cropped selected image for each of the occupants. Set 206 includes no duplicate images, i.e. no more than one image is in set 206 for a given occupant, so duplicate images of a given occupant need be stored or displayed. The controller 104 can select the

representative image

142, 144, 146, and 148 from each

group

134, 136, 138, and 140 by selecting images based on corresponding confidence scores from the automated facial detection. It is also contemplated that controller 104 can selecting the

representative image

142, 144, 146, and 148 from each

group

134, 136, 138, and 140 by selecting images based on which image in the group has least facial offset angle from line of sight of the imaging sensor 120 which acquired the respective image. This selection process can be run on the fly with facial detection and facial recognition to winnow the gallery 202 down to the set 206.

The controller 104 can determine how many occupants are in the vehicle 105 by counting the

groups

134, 136, 138, and 140. In this example, there are four

groups

134, 136, 138, and 140 indicating there are four occupants in the vehicle 105. If

groups

134, 136, 138, and 140 are conflated down to the set 206 on the fly as described above, then the

groups

134, 136, 138, and 140 can be counted indirectly by simply counting the final cropped images in set 206 to determine how many occupants are in the vehicle 105.

The controller 104 can output the number of occupants in the vehicle 105, and can provide other output actions as needed. For example, controller 104 can initiate a response, e.g., via the output device 150, upon determining an improper number of occupants in the vehicle. For example, if controller 104 determines there are not enough occupants in a vehicle in an HOV lane, controller 150 can use the output device 150 to output an alert on a visual display, sound an audible alarm, close a physical barrier, transmit a citation, mail a citation, update a database, and/or dispatch an officer.

It is also contemplated that with the set of images 206, controller 104 can run the final cropped facial images through a facial recognition database, either locally or remotely, to check for matches between the occupants and known individuals in the database. If a match is found, e.g., one of the occupants in the vehicle 105 is on a watch list, the controller 104 can initiate an output response, e.g., using output device 150, such as outputting an alert on a visual display, sounding an audible alarm, closing a physical barrier, transmitting a citation, mailing a citation, updating a database, and/or dispatching an officer.

While shown and described herein in an exemplary context where there are n=3 camera systems and m=4 occupants in the vehicle 105, those skilled in the art will readily appreciate that any suitable number n of camera systems can be used, and any suitable number m of occupants in a vehicle can be counted/identified without departing from the scope of this disclosure.

The methods and systems of the present disclosure, as described above and shown in the drawings, provide for counting and identifying occupants in vehicles with superior properties including reliable, automated detection and identification of all occupants in a moving vehicle. While the apparatus and methods of the subject disclosure have been shown and described with reference to preferred embodiments, those skilled in the art will readily appreciate that changes and/or modifications may be made thereto without departing from the scope of the subject disclosure.

Claims

What is claimed is:

1. A method of detecting occupants in a vehicle using a system comprising a first camera positioned on a driver's side of the vehicle and a second camera positioned on a passenger's side of the vehicle, the method comprising:

detecting an oncoming vehicle;

using the first camera and the second camera, acquiring a plurality of images of occupants in the vehicle in response to detection of the vehicle, wherein the first camera acquires a first portion of the plurality of images from a first angle, wherein the second camera acquires a second portion of the plurality of images from a second angle, and wherein the first angle is different from the second angle;

performing automated facial detection on the plurality of images;

based on the automated facial detection, generating a gallery of facial images from the plurality of images, wherein the gallery of facial images comprises a first facial image of at least one of the occupants taken from the first angle and a second facial image of said at least one of the occupants taken from the second angle, thereby resulting in said gallery of facial images comprising multiple images of said at least one of the occupants;

performing, after forming the gallery of facial images, automated facial recognition on the gallery of facial images to form facial groupings, wherein each of the facial groupings comprise facial images of only one of the occupants and wherein at least one of said facial groupings comprises said multiple images of the at least one of the occupants and no facial images of other ones of said occupants;

generating a confidence score for each of the images in each of the facial groupings based on the automated facial detection performed on each of the facial groupings;

selecting a representative image from each of the facial groupings based on the corresponding confidence score;

outputting the representative image from each of the facial groupings to create a set of unique images, each of which is representative of only one of the occupants; and

counting the groups to determine how many occupants are in the vehicle.

2. The method as recited in claim 1, wherein no multiple images of a given occupant are stored or displayed.

3. The method as recited in claim 1, wherein selecting the representative image from each of the facial groupings comprises:

for each image in each of the facial groupings, determining a facial offset angle from a line of sight of an imaging sensor that acquired the image; and

selecting the representative image based on which image in the facial grouping has a smallest facial offset angle.

4. The method as recited in claim 1, further comprising:

comparing each of the unique images to images of known individuals in a database and determining if any of the unique images match at least one of said images of known individuals.

5. The method as recited in claim 4, further comprising initiating a response upon finding a match in the database, wherein the response includes at least one of outputting an alert on a visual display, sounding an audible alarm, closing a physical barrier, transmitting a citation, mailing a citation, or dispatching an officer.

6. The method as recited in claim 1, further comprising initiating a response upon determining an improper number of occupants in the vehicle, wherein the response includes at least one of outputting an alert on a visual display, sounding an audible alarm, closing a physical barrier, transmitting a citation, mailing a citation, or dispatching an officer.

7. The method as recited in claim 1, further comprising illuminating the vehicle with a respective pulse of illumination for each image acquired, wherein each pulse of illumination is performed at a different time to reduce shadows cast onto the occupants while acquiring the plurality of images.

8. The method as recited in claim 1, further comprising capturing a primary image of the occupants in the vehicle, applying facial detection to the primary image to generate reference images and using said reference images in the automated facial recognition to form the facial groupings.

9. The method as recited in claim 8, further comprising adding a new face to the gallery of facial images each time a detected face does not match with a face already in the gallery of facial images.

10. The method as recited in claim 8, further comprising adding each face detected to the gallery of facial images that is not already in the gallery until there is an image in the gallery of facial images of each face detected by said automated facial detection process.

11. A system for detecting occupants in a vehicle using a plurality of images, comprising:

a controller;

a first camera positioned on a driver's side of the vehicle and a second camera positioned on a passenger's side of the vehicle, wherein the first camera and the second camera are operatively coupled to the controller, and wherein each of the first camera and the second camera comprises an imaging sensor, a pulsed illumination device, and a processor operatively coupling the imaging sensor to the pulsed illumination source for synchronizing pulses of illumination with exposure of the imaging sensor; and

a trigger operatively connected to the controller to detect the vehicle and control the first camera and the second camera to acquire images of the vehicle, wherein the first camera acquires a first portion of the plurality of images from a first angle, wherein the second camera acquires a second portion of the plurality of images from a second angle, and wherein the first angle is different from the second angle; and

machine readable instructions stored in a non-transient storage medium and configured to be executed by the controller, wherein executing the machine readable instructions causes the controller to:

detect an oncoming vehicle based upon the trigger;

acquire the first portion of the plurality of images and the second portion of the plurality of images;

perform automated facial detection on the plurality of images;

based on the automated facial detection, generate a gallery of facial images from the plurality images, wherein the gallery of facial images comprises a first facial image of at least one of the occupants taken from the first angle and a second facial image of said at least one of the occupants taken from the second angle, thereby resulting in said gallery of facial images comprising multiple images of said at least one of the occupants;

perform, after formation of the gallery of facial images, automated facial recognition on the gallery of facial images to form facial groupings, wherein each of the facial groupings comprise facial images of only one of the occupants and wherein at least one of said facial groupings comprises said multiple images of the at least one of the occupants and no facial images of other ones of said occupants;

generate a confidence score for each of the images in the facial groupings based on the automated facial detection performed on each of the facial groupings;

select a representative image from each of the facial groupings based on the corresponding confidence scores;

output the representative image from each of the facial groupings to create a set of unique images, each of which is representative of only one of the occupants; and

count the groups to determine how many occupants are in the vehicle.

12. The system as recited in claim 11, wherein the machine readable instructions comprise instructions that, when executed, cause the controller to illuminate the vehicle with a pulse of illumination from the pulsed illumination device for each of the plurality of images, wherein each pulse of illumination is performed at a different time to reduce shadows cast onto the occupants while acquiring the plurality of images.

13. The system as recited in claim 11, wherein each of the first camera and the second camera comprises a lens optically coupled to the imaging sensor, and at least one of an optical bandpass filter operatively configured to filter light passing through the lens or a linear polarization filter operatively configured to filter light passing through the lens.

14. The system as recited in claim 1, wherein the machine readable instructions comprise instructions that, when executed, cause the controller to determine a facial offset angle from a line of sight of an imaging sensor that acquired the image for each image in each of the facial groupings and select the representative image based on which image in the facial grouping has a smallest facial offset angle.

15. A method of detecting occupants in a vehicle using a system comprising a first camera positioned on a driver's side of the vehicle and a second camera positioned on a passenger's side of the vehicle, The method comprising:

detecting an oncoming vehicle;

performing automated facial detection on the plurality of images;

determining a facial offset angle from a line of sight of an imaging sensor that acquired the image for each image in each of the facial groupings;

selecting a representative image from each of the facial groupings based on which image in the facial grouping has a smallest offset angle;

counting the groupings to determine how many occupants are in the vehicle.

16. The method as recited in claim 15, wherein no multiple images of a given occupant are stored or displayed.

17. The method as recited in claim 15, further comprising: comparing each of the unique images to images of known individuals in a database and determining if any of the unique images match at least one of said images of known individuals.

18. The method as recited in claim 15, further comprising initiating a response upon finding a match in the database, wherein the response includes at least one of outputting an alert on a visual display, sounding an audible alarm, closing a physical barrier, transmitting a citation, mailing a citation, or dispatching an officer.

19. The method as recited in claim 15, further comprising initiating a response upon determining an improper number of occupants in the vehicle, wherein the response includes at least one of outputting an alert on a visual display, sounding an audible alarm, closing a physical barrier, transmitting a citation, mailing a citation, or dispatching an officer.

20. The method as recited in claim 15, further comprising illuminating the vehicle with a respective pulse of illumination for each image acquired, wherein each pulse of illumination is performed at a different time to reduce shadows cast onto the occupants while acquiring the plurality of images.

21. The method as recited in claim 15, further comprising capturing a primary image of the occupants in the vehicle, applying facial detection to the primary image to generate reference images and using said reference images in the automated facial recognition to form the facial groupings.

22. The method as recited in claim 21, further comprising adding a new face to the gallery of facial images each time a detected face does not match with a face already in the gallery of facial images.

23. The method as recited in claim 21, further comprising adding each face detected to the gallery of facial images that is not already in the gallery until there is an image in the gallery of facial images of each face detected by said automated facial detection process.

24. A system for detecting occupants in a vehicle using a plurality of images, comprising:

a controller;

a first camera positioned on a driver's side of the vehicle and a second camera positioned on a passenger's side of the vehicle, wherein the first camera and second camera are operatively coupled to the controller and wherein each of the first camera and the second camera comprises an imaging sensor, a pulsed illumination device, and a processor operatively coupling the imaging sensor to the pulsed illumination source for synchronizing pulses of illumination with exposure of the imaging sensor; and

detect an oncoming vehicle based upon the trigger;

perform automated facial detection on the plurality of images;

determine a facial offset angle from a line of sight of an imaging sensor that acquired the image for each image in each of the facial groupings;

select a representative image from each of the facial groupings based on which image in the facial grouping has a smallest facial offset angle;

count the groupings to determine how many occupants are in the vehicle.

25. The system as recited in claim 24, wherein the machine readable instructions comprise instructions that, when executed, cause the controller to illuminate the vehicle with a pulse of illumination from the pulsed illumination device for each of the plurality of images, wherein each pulse of illumination is performed at a different time to reduce shadows cast onto the occupants while acquiring the plurality of images.

26. The system as recited in claim 25, wherein each of the first camera and the second camera comprises a lens optically coupled to the imaging sensor, and at least one of an optical bandpass filter operatively configured to filter light passing through the lens or a linear polarization filter operatively configured to filter light passing through the lens.