WO2022245770A1

WO2022245770A1 - Personalized hmi using occupant biometrics

Info

Publication number: WO2022245770A1
Application number: PCT/US2022/029555
Authority: WO
Inventors: Esaias Aguilar PECH; Mahadevan VANCHISWARAN; Daniel Cashen; Piyush ANAM; Shashank DHARIWAL; Rudra Narayan HOTA; Arthur S. BROWN
Original assignee: Continental Automotive Systems, Inc.
Priority date: 2021-05-17
Filing date: 2022-05-17
Publication date: 2022-11-24

Abstract

A method and system for identifying and locating a vehicle occupant within a vehicle cabin is disclosed. The method and system automatically obtain information of an occupant within a vehicle cabin with at least one sensing device disposed within the vehicle cabin. The information includes video and audio information obtained from at least one camera and at least one microphone located in the vehicle cabin. The video and audio information are used to determine a characteristic of the occupant and adjust a setting of a vehicle parameter based on a profile associated with the occupant. A computer accessible medium on which instructions are stored for identifying a vehicle occupant is also disclosed.

Description

PERSONALIZED HMI USING OCCUPANT BIOMETRICS

CROSS REFERENCE TO RELATED APPLICATION

[0001] The present application daims the benefit of U.S. Provisional Application 63/189,305, filed May 17, 2021, which is hereby incorporated by reference.

TECHNICAL FIELD

[0002] The present disdosure relates to a system and method for identifying an occupant using biometric devices and sensors and applying occupant spedfic preferences based on the identified occupant.

BACKGROUND

[0003] Vehides may have machine human interfaces that can be personalized to suit a driver or other vehide occupant. The personalized preferences can include mirror position, seat position, temperature setting along with numerous other possible vehide settings. The individual personal preference is typically activated utilizing a mechanical button and/or activation of a specific keyfob. Current personalization features in a vehicle are biased toward a vehide operator’s preferences. Vehicles are increasingly providing options for other occupants to personalize an individual space. Current methods for recalling personalized preferences require a positive actuation of a mechanical button or keyfob.

[0004] The background description provided herein is for the purpose of generally presenting a context of this disdosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disdosure.

SUMMARY

[0005] A method of identifying a vehide occupant according to a disclosed embodiment indudes, among other possible things, the steps of automatically obtaining information indicative of an occupant within a vehicle cabin with at least one sensing device disposed within the vehicle cabin, determining a characteristic of the occupant based on the information obtained, and adjusting a setting of a vehicle parameter based on the determined occupant characteristic.

[0006] In an embodiment of the foregoing method, the at least one sensing device comprises at least one camera and further comprises automatically obtaining images of a vehicle occupant with the at least one camera disposed within the vehicle cabin and determining at least one occupant characteristic based on the obtained image.

[0007] Another embodiment of any of the foregoing methods, further comprises determining a location within the vehicle cabin of the vehicle occupant based on the obtained image.

[0008] Another embodiment of any of the foregoing methods, further comprises identifying a specific individual occupant based on the obtained image.

[0009] Another embodiment of any of the foregoing methods, further comprises comparing the specific individual occupant to a predefined occupant profile and identifying the individual based on a positive match to the predefined occupant profile.

[0010] Another embodiment of any of the foregoing methods, further comprises identifying a physical characteristic of the vehicle occupant based on the obtained image.

[0011] Another embodiment of any of the foregoing methods, further comprises identifying an emotional state of the vehicle occupant based on the obtained image.

[0012] Another embodiment of any of the foregoing methods, wherein the at least one sensing device comprises at least one microphone and the method further comprises obtaining sound information from the at least one microphone disposed within the vehicle cabin and identifying the vehicle occupant based on the obtained sound information. [0013] Another embodiment of any of the foregoing methods, further comprises determining a specific individual vehicle occupant based on a comparison of the obtained sound information and stored sound information.

[0014] Another embodiment of any of the foregoing methods, wherein the setting of a vehicle parameter is set based on a saved profile associated with the identified occupant characteristic.

[0015] Another embodiment of any of the foregoing methods, wherein the saved profile includes information relating to a specific vehicle occupant.

[0016] Another embodiment of any of the foregoing methods, wherein the saved profile includes information relating to physical characteristics of the vehicle occupant

[0017] A system for adjusting individualized vehicle parameters according to an embodiment of this disclosure includes, among other possible things, a controller configured for automatically obtaining information indicative of an occupant within a vehicle cabin from at least one sensing device disposed within the vehicle cabin, determining a characteristic of the occupant based on the information obtained, and adjusting a setting of a vehicle parameter based on the determined occupant characteristic.

[0018] In another embodiment of the foregoing system, the at least one sensing device comprises at least one camera and the controller is further configured for automatically obtaining images of a vehicle occupant with the at least one camera disposed within the vehicle cabin and determining at least one occupant characteristic based on the obtained image.

[0019] In another embodiment of any of the foregoing systems, the at least one sensing device comprises at least one microphone and the controller is further configured for automatically obtaining sound information from the at least one microphone disposed within the vehicle cabin and identifying the vehicle occupant based on the obtained sound information.

[0020] In another embodiment of any of the foregoing systems, the setting of a vehicle parameter is set based on a saved profile associated with the identified occupant characteristic. [0021] A computer accessible medium on which instructions are stored for identifying a vehicle occupant according to another embodiment of the foregoing includes, among other possible things, a first code segment used for automatically obtaining information indicative of an occupant within a vehicle cabin with at least one sensing device disposed within the vehicle cabin, a second code segment used for determining a characteristic of the occupant based on the information obtained and a third code segment used for adjusting a setting of a vehicle parameter based on the determined occupant characteristic.

[0022] In another embodiment of the foregoing computer accessible medium, the at least one sensing device comprises at least one camera and the first code segment is used for automatically obtaining images of a vehicle occupant with the at least one camera disposed within the vehicle cabin and determining at least one occupant characteristic based on the obtained image.

[0023] In another embodiment of the foregoing computer accessible medium, the at least one sensing device comprises at least one microphone and the first code segment is used for automatically obtaining sound information from the at least one microphone disposed within the vehicle cabin and identifying the vehicle occupant based on the obtained sound information.

[0024] In another embodiment of the foregoing computer accessible medium, the third code segment is used for setting of a vehicle parameter based on a saved profile associated with the identified occupant characteristic.

[0025] Although the different examples have the specific components shown in the illustrations, embodiments of this disclosure are not limited to those particular combinations. It is possible to use some of the components or features from one of the examples in combination with features or components from another one of the examples.

[0026] These and other features disclosed herein can be best understood from the following specification and drawings, the following of which is a brief description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a schematic view of a vehicle cabin; and

[0028] Figure 2 is a flow chart illustrating steps of a method of automatically identifying a vehicle occupant and adjusting a vehicle setting based on the identified vehicle occupant.

DETAILED DESCRIPTION

[0029] Referring to Figure 1, a cabin 22 of a vehicle 20 is shown schematically and includes a system 25 for automatically detecting and identifying characteristics of occupants. The example system includes cameras 32, 34 and microphones 36, 38 that provide information to a controller 40. The vehicle cabin 22 includes a driver’s zone 24, a front passenger zone 26 and rear passenger zones 28, 29 and 30. In this example, at least one of a plurality of displays 42, 44, 46, 48, 50 and 52 are associated with each of the zones.

[0030] Some vehicles currently allow for interfaces to be personalized, especially those with infotainment displays as part of instrument cluster and central infotainment display as well as passenger and rear seat displays. Applying personalized settings is typically an activity initiated by the driver. Personalization can include general appearance of the user interface or cabin lighting (colors, patterns, etc.), the format of presented instrumentation and infotainment content, personal device access, preferred vehicle settings, climate settings, radio settings, seat settings, etc. A vehicle may allow one or more personalized settings to be saved, commonly referred to as a ‘profile’. Restoring or changing a profile can involve pressing a mechanical button, or interacting with steering wheel controls and an instrument cluster interface, or via touch screen interaction a graphic user interface (Ul). Restoring a profile may even be triggered with detection of associated personal device or a key fob. Personalization can also serve to restrict access or features for new or non-regular users, such as for example with the use of a valet mode. The convenance provided by such personalized settings may not be completely realized because they require some active action by a vehicle occupant and therefore may not always be implemented. Moreover, mechanical access to a profile provides limited, if any privacy and/or security protections.

[0031] The disclosed system 25 integrates the cameras 32, 34 and microphones 36, 38 along with video and audio processing algorithms to automatically obtain information indicative of an occupant within the vehicle cabin 22 that is matched with a saved profile and utilized to automatically adjust settings of at least one vehicle parameter based on the profile corresponding with the occupant.

[0032] The disclosed system 25 uses images captured by the cameras 32, 34 and audio detected by the microphones 36, 38 to detect occupant biometrics unique to each occupant. In this disclosed example, the video images may include face and voice detection along with other image detection methods. Face detection may include use of any subsets of facial metrics such as periocular, gait, and ear. The physical attributes of an occupant's face are used to automatically identify the occupant and match that occupant with saved profiles. Image detection may also be implemented to utilize other detectable biometrics attributes such as emotions, age, fingerprints, heartrates, temperature, etc. The image and voice data obtained from the cameras 32, 34 and the microphones 36, 38 regarding a specific occupant is associated to a specific profile, and then later used as a trigger to authenticate and/or restore that profile.

[0033] The different zones 24, 26, 28, 29 and 30 within the vehicle cabin 22 include occupant specific settings that maybe saved as an occupant profile. The settings could include a specific screen setting for a display associated with the zone, a temperature setting, a volume setting, a seat position and/or any other setting of a vehicle system that may be associated with that zone. The settings may also be tailored to the specific zone in the vehicle. For example, the driver’s zone 24 will have settings associated with systems for operating the vehicle. Such settings may include for example, settings for the drivers display 42, the center display 46, seat position, temperature settings as well as specific communication pairing with the driver’s mobile phone or communication device. The front and rear passenger zones 26, 28, 29 and 30 may only include settings associated with infotainment features of the vehicle and available on each individual display 44, 48, 50 and 52. [0034] Displays maybe adjusted to show a predefined default screen unique to each occupant. The predefined screen may also include modifications depending on the occupant, such as use of larger fonts for older or younger occupants. Moreover, the screens maybe prompted to limit access to certain functions based on characteristics of the occupant. For example, younger occupants maybe prevented for adjusting temperature controls or accessing outside communications.

[0035] The use of image and audio data may also be utilized to provide an additional level of security. The image and audio data associated with an individual occupant may be matched to other security authorization means, such as a password or pin. If the image and audio data does not match the password or pin, access to can be limited or denied.

[0036] Moreover, the image and audio data may be utilized to ascertain an emotional state of an occupant. The emotional state of the occupant could be used to adjust a color or audio cues to the occupant. Moreover, the emotional state could be used to initiate additional safety protocols.

[0037] Additionally, the image and audio data could be utilized to detect the presence and location of an animal or pet within the vehicle cabin and initiate settings that have been predefined for that animal. Settings may include certain temperature settings that correspond to the family pet to provide a level of comfort.

[0038] Each of the example vehicle parameter adjustments are only limited by the capability provided within the vehicle. The example system provides for the recall of saved profiles based on the automatic identification performed utilizing included sensors. In this example, the cameras 32, 34 and microphones 36, 38 are utilized, however, other sensors as maybe present within the vehicle cabin 22 can be utilized by the controller 40 to identify the occupant.

[0039] Moreover, a profile unique to a specific occupant can move with that occupant within the different zones of the vehicle. For example, preferences input while an occupant was sitting in the front passenger zone 26 could move with that occupant to one of the rear zones 28, 29 and 30. Upon recognition and location of an occupant within the vehicle, the preferences saved as part of the unique user profile can be provided in the different location without any action required by the occupant

[0040] Identification of the occupant may be to identify a specific individual vehicle occupant with an associated profile saved in the vehicle or a more general identification utilized to adjust vehicle parameters. For example, certain vehicle operations may be limited based on identification of a new driver. Accordingly, the vehicle may be switched to a valet mode upon detection of a new driver that is not associated with a saved profile.

[0041] Moreover, identification of a new passenger could provide for adjustment of certain vehicle features based on the detected physical characteristics as well as a position within the vehicle. For example, if the occupant is identified as an adult, one set of vehicle parameters will be made accessible, whereas identification of the occupant as a child would make another different set of parameters and functions accessible.

[0042] Additionally, automatic identification of physical attributes could be utilized by the system 25 to adjust operation of vehicle safety devices such as seatbelts and airbag system. Furthermore, any system with the vehicle that can be adjusted based on an occupant profile could be adjusted based on the determinations and identification provided by the example system 25.

[0043] In operation, the controller 40 gathers and utilizes information from sensors within the vehicle cabin 22 to identify a position of an occupant along with an identity of the occupant. The controller 40 determines a characteristic of the occupant based on the information obtained and then automatically adjusts a setting of a vehicle parameter based on the determined occupant characteristic.

[0044] The disclosed example controller 40 may be a hardware device for executing software, particularly software stored in memory. The controller 40 can include a custom made or commercially available processor, a central processing unit (CPU), an auxiliary processor among several processors associated with the computing device, a semiconductor-based microprocessor (in the form of a microchip or chip set) or generally any device for executing software instructions. [0045] The memory can include any one or combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, VRAM, etc.)) and/or nonvolatile memory elements (e.g., ROM, hard drive, tape, CD- ROM, etc.). Moreover, the memory may incorporate electronic, magnetic, optical, and/or other types of storage media. Note that the memory can also have a distributed architecture, where various components are situated remotely from one another, but can be accessed by the processor.

[0046] The software in the memory may include one or more separate programs, each of which includes an ordered listing of executable instructions for implementing logical functions. A system component embodied as software may also be construed as a source program, executable program (object code), script, or any other entity comprising a set of instructions to be performed. When constructed as a source program, the program is translated via a compiler, assembler, interpreter, or the like, which may or may not be included within the memory.

[0047] The Input/Output devices that may be coupled to system I/O Interface(s) may include input devices, for example but not limited to, a keyboard, mouse, scanner, microphone, camera, proximity device, etc. Finally, the Input/Output devices may further include devices that communicate both as inputs and outputs, for instance but not limited to, a modulator/demodulator (modem; for accessing another device, system, or network), a radio frequency (RF) or other transceiver, a telephonic interface, a bridge, a router, etc.

[0048] When the controller 40 is in operation, the processor can be configured to execute software stored within the memory, to communicate data to and from the memory, and to generally control operations of the computing device pursuant to the software. Software in memory, in whole or in part, is read by the processor, perhaps buffered within the processor, and then executed.

[0049] The disclosed system may be embodied as software stored on a computer accessible non-transitory medium 54 on which instructions 62 are stored for instructing operation of the controller 40 to identify a vehicle occupant. In one disclosed example, the instructions 62 include a first code segment 56 used for automatically obtaining information indicative of an occupant within a vehicle cabin with at least one sensing device disposed within the vehicle cabin, a second code segment 58 used for determining a characteristic of the occupant based on the information obtained and a third code segment 60 used for adjusting a setting of a vehicle parameter based on the determined occupant characteristic. It should be understood, that this is only one disclosed example and that further code segments maybe included as part of the instructions 62 and are within the scope and contemplation of this disclosure.

[0050] In the disclosed example, the first code segment 56 is used for automatically obtaining images of a vehicle occupant with at least one camera disposed within the vehicle cabin and determining at least one occupant characteristic based on the obtained image. In the disclosed example, the first code segment 56 is further used for automatically obtaining sound information from the at least one microphone disposed within the vehicle cabin and identifying the vehicle occupant based on the obtained sound information. Additionally, in the disclosed example, the third code segment 60 is used for setting of a vehicle parameter based on a saved profile associated with the identified occupant characteristic.

[0051] Referring to Figure 2 with continued reference to Figure 1, a flow chart 64 is shown to illustrate example operation of the disclosed system 25. This example operation is separated into an identity recognition phase 66 and a cabin location identification phase 68. Beginning with the recognition phase 66, video images are captured as indicated at 80 and audio is captured as indicated at 82. The captured video images and audio are processed along parallel analysis paths as indicated at 70. The voice and audio are processed through detection algorithms as indicated at 72. As appreciated, the corresponding video and audio detection algorithms will be medium specific and involve analysis unique to video and audio. Accordingly, the output from the recognition algorithms for the video and audio will be different and processed according to the different data types.

[0052] The output from the detection algorithms 72 are provided to a prediction profile database as indicated at 74. The database 74 may include unique profiles that have been previously saved as well as generic profiles corresponding to detected and recognized physical attributes of the occupant. The prediction profile 74 provides an output to a prediction confidence algorithm indicated at 76. The prediction confidence algorithm 76 generates an output based on the match with a stored profile. If the match is within a predefined criterion a match will be determined and that match forward as an identified profile as indicated at 78. The confidence level of the match can differ due to changes in appearance of an occupant or based on different factors. Accordingly, the prediction confidence algorithm 76 accounts for differences between current image and voice data as compared to a saved profile.

[0053] Once the profile is confirmed that profile is correlated and matched with the position results as indicated at 84.

[0054] The position phase 68 is a determination that is performed concurrently with the identification phase 66 using the same video and audio data 80, 82 captured as indicated again at 70. In the position location phase 68, the captured video and audio data is provided to at least one detection and location algorithm as indicated at 88. The image and audio information could be processed separately or combined to ascertain what zone within the vehicle cabin an occupant is located. The video images can associate a position of an occupant that is confirmed with any audio data. As appreciated, the audio data may not be immediately available and therefore location determination may be determined based only on the image data. A prediction is made of the location of the occupant as indicated at 90. The prediction is based on the image and audio data gathered in this example. Moreover, other data available indicative a location of an occupant may also be utilized to further improve the prediction of location. Such additional data may include information from weight sensors, seat belt use or any other data available within a specific vehicle.

[0055] The prediction is confirmed to a specific predefined confidence level as indicated at 92. As appreciated, if the predefined confidence level is not achieved, the system may implement a default set of setting based on a predefined setting.

[0056] Upon confirmation of the predicted position within a predefined confidence level, the position of the occupant is verified as indicated at 94 and forwarded to be correlated with the identity information as indicated at 84. Once the identity and position of an occupant is identified within the predefined confidence level, the system 25 will adjust parameters associated with the occupant and the location to a saved profile as indicated at 86. The profile may include any number of individualized settings ranging from temperature settings, seat position to settings applicable to vehicle safety device operation. Any vehicle setting that can be set based on a personal preference can be applied to the seat zone upon the automatic recognition of the occupant

[0057] Accordingly, the disclosed occupant identification system identifies an occupant and adjusts vehicle settings based on a profile associated with the occupant and location within the vehicle cabin.

[0058] Although the different non-limiting embodiments are illustrated as having specific components or steps, the embodiments of this disclosure are not limited to those particular combinations. It is possible to use some of the components or features from any of the non-limiting embodiments in combination with features or components from any of the other non-limiting embodiments.

[0059] It should be understood that like reference numerals identify corresponding or similar elements throughout the several drawings. It should be understood that although a particular component arrangement is disclosed and illustrated in these exemplary embodiments, other arrangements could also benefit from the teachings of this disclosure.

[0060] The foregoing description shall be interpreted as illustrative and not in any limiting sense. A worker of ordinary skill in the art would understand that certain modifications could come within the scope of this disclosure. For these reasons, the following claims should be studied to determine the true scope and content of this disclosure.

Claims

CLAIMS What is claimed is:

1. A method of identifying a vehicle occupant comprising: automatically obtaining information indicative of an occupant within a vehicle cabin with at least one sensing device disposed within the vehicle cabin; determining a characteristic of the occupant based on the information obtained; and adjusting a setting of a vehicle parameter based on the determined occupant characteristic.

2. The method as recited in claim 1 , wherein the at least one sensing device comprises at least one camera and further comprising automatically obtaining images of a vehicle occupant with the at least one camera disposed within the vehicle cabin and determining at least one occupant characteristic based on the obtained image.

3. The method as recited in claim 2, further comprising determining a location within the vehicle cabin of the vehicle occupant based on the obtained image.

4. The method as recited in claim 2, further comprising identifying a specific individual occupant based on the obtained image.

5. The method as recited in claim 4, further comprising comparing the specific individual occupant to a predefined occupant profile and identifying the individual based on a positive match to the predefined occupant profile.

6. The method as recited in claim 2, further comprising identifying a physical characteristic of the vehicle occupant based on the obtained image.

7. The method as recited in claim 2, further comprising identifying an emotional state of the vehicle occupant based on the obtained image.

8. The method as recited in claim 1 , wherein the at least one sensing device comprises at least one microphone, the method further comprising obtaining sound information from the at least one microphone disposed within the vehicle cabin and identifying the vehicle occupant based on the obtained sound information.

9. The method as recited in claim 8, further comprising determining a specific individual vehicle occupant based on a comparison of the obtained sound information and stored sound information.

10. The method as recited in claim 1, wherein the setting of a vehicle parameter is set based on a saved profile associated with the identified occupant characteristic.

11. The method as recited in claim 10, wherein the saved profile includes information relating to a specific vehicle occupant.

12. The method as recited in claim 11, wherein the saved profile includes information relating to physical characteristics of the vehicle occupant.

13. A system for adjusting individualized vehicle parameters comprising: a controller configured for automatically obtaining information indicative of an occupant within a vehicle cabin from at least one sensing device disposed within the vehicle cabin, determining a characteristic of the occupant based on the information obtained, and adjusting a setting of a vehicle parameter based on the determined occupant characteristic.

14. The system as recited in claim 13, wherein the at least one sensing device comprises at least one camera and the controller is further configured for automatically obtaining images of a vehicle occupant with the at least one camera disposed within the vehicle cabin and determining at least one occupant characteristic based on the obtained image.

15. The method as recited in claim 13, wherein the at least sensing device comprises at least one microphone and the controller is further configured for automatically obtaining sound information from the at least one microphone disposed within the vehicle cabin and identifying the vehicle occupant based on the obtained sound information.

16. The system as recited in claim 13, wherein the setting of a vehicle parameter is set based on a saved profile associated with the identified occupant characteristic.

17. A computer accessible medium on which instructions are stored for identifying a vehicle occupant, the instructions comprising: a first code segment used for automatically obtaining information indicative of an occupant within a vehicle cabin with at least one sensing device disposed within the vehicle cabin; a second code segment used for determining a characteristic of the occupant based on the information obtained; and a third code segment used for adjusting a setting of a vehicle parameter based on the determined occupant characteristic.

18. The computer accessible medium as recited in claim 17, wherein the at least one sensing device comprises at least one camera and the first code segment is used for automatically obtaining images of a vehicle occupant with the at least one camera disposed within the vehicle cabin and determining at least one occupant characteristic based on the obtained image.

19. The computer accessible medium as recited in claim 17, wherein the at least one sensing device comprises at least one microphone and the first code segment is used for automatically obtaining sound information from the at least one microphone disposed within the vehicle cabin and identifying the vehicle occupant based on the obtained sound information.

20. The computer accessible medium as recited in claim 17, wherein the third code segment is used for setting of a vehicle parameter based on a saved profile associated with the identified occupant characteristic.