WO2023113717A1 - Smart vehicle assistant with artificial intelligence - Google Patents

Smart vehicle assistant with artificial intelligence Download PDF

Info

Publication number
WO2023113717A1
WO2023113717A1 PCT/TR2022/050946 TR2022050946W WO2023113717A1 WO 2023113717 A1 WO2023113717 A1 WO 2023113717A1 TR 2022050946 W TR2022050946 W TR 2022050946W WO 2023113717 A1 WO2023113717 A1 WO 2023113717A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
vehicle
face
speaking
assistant
Prior art date
Application number
PCT/TR2022/050946
Other languages
French (fr)
Inventor
Serdar SULTANOĞLU
Original Assignee
Di̇zaynvi̇p Teknoloji̇ Bi̇li̇şi̇m Ve Otomoti̇v Sanayi̇ Anoni̇m Şi̇rketi̇
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Di̇zaynvi̇p Teknoloji̇ Bi̇li̇şi̇m Ve Otomoti̇v Sanayi̇ Anoni̇m Şi̇rketi̇ filed Critical Di̇zaynvi̇p Teknoloji̇ Bi̇li̇şi̇m Ve Otomoti̇v Sanayi̇ Anoni̇m Şi̇rketi̇
Publication of WO2023113717A1 publication Critical patent/WO2023113717A1/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/172—Classification, e.g. identification
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/08—Interaction between the driver and the control system
    • B60W50/14—Means for informing the driver, warning the driver or prompting a driver intervention
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16—Sound input; Sound output
    • G06F3/167—Audio in a user interface, e.g. using voice commands for navigating, audio feedback
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
    • G06V20/597—Recognising the driver's state or behaviour, e.g. attention or drowsiness
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174—Facial expression recognition
    • B60K2360/29—
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/08—Interaction between the driver and the control system
    • B60W50/14—Means for informing the driver, warning the driver or prompting a driver intervention
    • B60W2050/146—Display means
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2540/00—Input parameters relating to occupants
    • B60W2540/21—Voice
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2540/00—Input parameters relating to occupants
    • B60W2540/22—Psychological state; Stress level or workload

Definitions

  • the invention relates to a smart car assistant with artificial intelligence designed for use in automobiles, VIP vehicles, commercial vehicles and smart home systems, which recognizes the user's face, detects the user's emotional state, offers suggestions according to the detected emotion, and has a three- dimensional holographic face to create an emotional bond between the user and the vehicle, has the ability to read news, give weather information, send e-mails, create notes and record alarms, and allows voice control of the equipment in the vehicle, can translate in different languages, can tell the vehicle malfunctions, can transmit range information, connects to the phone and allows you to use the features of the phone, makes it possible to perceive the commands given by speaking in a daily and natural speaking language, in Turkish or in the desired language, and to answer the questions asked.
  • the present disclosure relates to a smart car assistant with artificial intelligence developed for eliminating the aforementioned disadvantages and providing new advantages to the respective technical field.
  • the aim of the invention is to provide an smart vehicle assistant with a face.
  • Another aim of the invention is to provide the creation of an smart vehicle assistant with features such as reading news, giving weather information, sending e-mails, creating notes and recording alarms.
  • Another aim of the invention is to provide a smart vehicle assistant, which allows voice control of the equipment in the vehicle by detecting what is said in daily and natural speaking language, in Vietnamese or in the desired language, and also has the ability to answer the questions asked.
  • Another aim of the invention is to provide the translation of what is spoken in the vehicle to the desired language, thanks to its ability to translate in different languages.
  • Another aim of the invention is to enable the user to be told audibly how far can be traveled with the remaining fuel or battery amount, as well as the malfunctions that may occur in the vehicle.
  • Another aim of the invention is to provide the use of the features of the phone by establishing a connection with the phone.
  • the desired video will be played, navigation information will be transferred and other features on the phone will be used.
  • the invention relates to a smart car assistant with artificial intelligence designed for use in automobiles, VIP vehicles, commercial vehicles and smart home systems, which recognizes the user's face, detects the user's emotional state, offers suggestions according to the detected emotion, and has a three- dimensional holographic face to create an emotional bond between the user and the vehicle, has the ability to read news, give weather information, send e-mails, create notes and record alarms, and allows voice control of the equipment in the vehicle, makes it possible to perceive the commands given by speaking in a daily and natural speaking language and to answer the questions asked.
  • the system of the invention generally includes at least one camera to recognize the user's face and detect the happiness, sadness, anger and similar feelings of the user, at least one microphone to detect user-directed voice commands and questions, at least one electronic motherboard to run the artificial intelligence software, where all the visual and audio data received are processed and the necessary actions are created, at least one electronic card containing motor drivers to control the equipment inside the vehicle, at least one three-dimensional hologram display or a two-dimensional display inside the vehicle to give the assistant a three-dimensional face.
  • the electronic card which receives data from the electronic motherboard with the CANBUS protocol, uses this data and the CANBUS protocol to allow the operation of the seats, coffee machine, curtains, lights and similar equipment in the vehicle.
  • the protocols used vary according to the vehicle, the vehicles usually have CANBUS protocol.
  • the system of the invention can be arranged according to the existing protocol in the vehicle.
  • the system make suggestions to the person. For example, when the owner of the vehicle gets into the vehicle, the person is recognized and suggestions such as “Welcome... Sir/Madam, you look unhappy today, would you like me to make you coffee?” can be offered. In addition, daily news and weather information can be transmitted to the person by the system. The user uses “Hey Dizzy” or a similar wake-up word before speaking with the system. Thus, other conversations in the vehicle and assistant conversations do not interfere with each other. In addition, the commands given to the system by the user are carried out provided that the system sees the user's face. If the user's face is not visible to the camera, the system gives a warning as "I need to see you in order to fulfill your request.”
  • the assistant system of the invention can answer the questions asked, with artificial intelligence. This is done with the “question and answer” model.
  • this question is searched by the software in the search engine and Wikipedia, the information in the form of a paragraph is simplified using the “levenshtein” model, then sent to the “Google BERT” model with new layers added, so that a logical answer is found within the paragraph, and the answer is read by the assistant, thereby the answer is given to the question.
  • the BERT model used in the invention is being updated and it is planned to use the “GPT-3” or “Megatron” model as well.
  • the assistant of the invention has a three-dimensional holographic face image.
  • the desired face type can be selected, as well as the real three-dimensional face of a person known as the assistant's face or a character can be determined.
  • the feeling of speaking with that person can be created.
  • the assistant speaks his/her lips can move according to the word he/she speaks, thus making it feel like a real person is speaking.
  • two-dimensional screens available in vehicles can also be used, and thus a more cost-effective structure can be obtained.

Abstract

The invention relates to a smart car assistant with artificial intelligence designed for use in automobiles, VIP vehicles, commercial vehicles and smart home systems, which recognizes the user's face, detects the user's emotional state, offers suggestions according to the detected emotion, and has a three-dimensional holographic face to create an emotional bond between the user and the vehicle, has the ability to read news, give weather information, send e-mails, create notes and record alarms, and allows voice control of the equipment in the vehicle by speaking in Turkish or in any desired language, makes it possible to perceive the commands given by speaking in a daily and natural speaking language, and to answer the questions asked, can translate in different languages, can tell the malfunctions that may occur in the vehicle and transmit range information audibly, connects to the phone and allows you to use the features of the phone.

Description

SMART VEHICLE ASSISTANT WITH ARTIFICIAL INTELLIGENCE
TECHNICAL FIELD
The invention relates to a smart car assistant with artificial intelligence designed for use in automobiles, VIP vehicles, commercial vehicles and smart home systems, which recognizes the user's face, detects the user's emotional state, offers suggestions according to the detected emotion, and has a three- dimensional holographic face to create an emotional bond between the user and the vehicle, has the ability to read news, give weather information, send e-mails, create notes and record alarms, and allows voice control of the equipment in the vehicle, can translate in different languages, can tell the vehicle malfunctions, can transmit range information, connects to the phone and allows you to use the features of the phone, makes it possible to perceive the commands given by speaking in a daily and natural speaking language, in Turkish or in the desired language, and to answer the questions asked.
PRIOR ART
People establish emotional bonds with other people or living creatures and thus continue their lives. It's nice to bond with someone, to feel that someone understands, cares, and protects you. However, these feelings only occur between humans and other living creatures.
Automobiles, VIP vehicles and similar vehicles purchased with large budgets do not have the quality to allow the user to create emotional bonds and cannot go beyond being a pile of metal. Emotional bonds cannot be established with the vehicles used today, communication cannot be ensured with natural speaking language, these vehicles do not recognize their owners, cannot detect and learn the behaviors and emotions of their owners, cannot answer the questions asked, and most importantly, these vehicles do not have a face. There are voice assistant systems in the vehicles produced by some vehicle manufacturers, and these systems can only perform a limited number of operations such as turning on and off and controlling equipment such as multimedia and air conditioning. In addition, these systems only work with the specified commands, and when something different is said other than these commands, these words are not perceived by the system.
Apart from the voice assistant systems mentioned above, there are also applications provided by some mobile devices and these applications only allow the use of features of the mobile device such as navigation and multimedia on the vehicle screen.
There is a need for an smart vehicle assistant that will provide establishing an emotional bond with the vehicle used, recognize and detect the face and emotions of the user, make suggestions, speak in a natural speaking language, in Turkish or in the desired language and answer questions.
Thus, the need to eliminate such shortcomings and disadvantages of the embodiments and practices employed in the prior art entails an improvement in the respective tehnical field.
AIM OF THE INVENTION
The present disclosure relates to a smart car assistant with artificial intelligence developed for eliminating the aforementioned disadvantages and providing new advantages to the respective technical field.
The aim of the invention is to provide an smart vehicle assistant with a face.
Another aim of the invention is to provide creating an smart vehicle assistant that recognizes the user's face, detects the emotional state and offers suggestions according to the detected emotion. Another aim of the invention is to provide an smart vehicle assistant that enables the user to create an emotional bond with his/her vehicle.
Another aim of the invention is to provide the creation of an smart vehicle assistant with features such as reading news, giving weather information, sending e-mails, creating notes and recording alarms.
Another aim of the invention is to provide a smart vehicle assistant, which allows voice control of the equipment in the vehicle by detecting what is said in daily and natural speaking language, in Turkish or in the desired language, and also has the ability to answer the questions asked.
Another aim of the invention is to provide the translation of what is spoken in the vehicle to the desired language, thanks to its ability to translate in different languages.
Another aim of the invention is to enable the user to be told audibly how far can be traveled with the remaining fuel or battery amount, as well as the malfunctions that may occur in the vehicle.
Another aim of the invention is to provide the use of the features of the phone by establishing a connection with the phone. Thus, the desired video will be played, navigation information will be transferred and other features on the phone will be used.
The structural and characteristic features of the present disclosure, including all of its advantages, will be more clearly understood when the detailed description given below is read, and thus, the present disclosure should be evaluated by taking this detailed description into consideration.
DETAILED DESCRIPTION OF THE INVENTION The preferred embodiments of the smart car assistant with artificial intelligence according to the present disclosure, which are mentioned in this detailed description, are only intended for providing a better understanding of the subject- matter, and should not be construed in any restrictive sense.
The invention relates to a smart car assistant with artificial intelligence designed for use in automobiles, VIP vehicles, commercial vehicles and smart home systems, which recognizes the user's face, detects the user's emotional state, offers suggestions according to the detected emotion, and has a three- dimensional holographic face to create an emotional bond between the user and the vehicle, has the ability to read news, give weather information, send e-mails, create notes and record alarms, and allows voice control of the equipment in the vehicle, makes it possible to perceive the commands given by speaking in a daily and natural speaking language and to answer the questions asked.
The system of the invention generally includes at least one camera to recognize the user's face and detect the happiness, sadness, anger and similar feelings of the user, at least one microphone to detect user-directed voice commands and questions, at least one electronic motherboard to run the artificial intelligence software, where all the visual and audio data received are processed and the necessary actions are created, at least one electronic card containing motor drivers to control the equipment inside the vehicle, at least one three-dimensional hologram display or a two-dimensional display inside the vehicle to give the assistant a three-dimensional face.
Electronic motherboard and electronic board containing motor drivers communicate via Ethernet connection and CANBUS protocol. The electronic card, which receives data from the electronic motherboard with the CANBUS protocol, uses this data and the CANBUS protocol to allow the operation of the seats, coffee machine, curtains, lights and similar equipment in the vehicle. Although the protocols used vary according to the vehicle, the vehicles usually have CANBUS protocol. The system of the invention can be arranged according to the existing protocol in the vehicle.
When the owner of the vehicle gets into the vehicle, the image is processed by the camera inside, and the face of the person is recognized by the system, and the emotional state of the person is also detected. According to the detected emotion, the system make suggestions to the person. For example, when the owner of the vehicle gets into the vehicle, the person is recognized and suggestions such as “Welcome... Sir/Madam, you look unhappy today, would you like me to make you coffee?” can be offered. In addition, daily news and weather information can be transmitted to the person by the system. The user uses “Hey Dizzy” or a similar wake-up word before speaking with the system. Thus, other conversations in the vehicle and assistant conversations do not interfere with each other. In addition, the commands given to the system by the user are carried out provided that the system sees the user's face. If the user's face is not visible to the camera, the system gives a warning as "I need to see you in order to fulfill your request."
The assistant system of the invention can answer the questions asked, with artificial intelligence. This is done with the “question and answer” model. When a question is asked, this question is searched by the software in the search engine and Wikipedia, the information in the form of a paragraph is simplified using the “levenshtein” model, then sent to the “Google BERT” model with new layers added, so that a logical answer is found within the paragraph, and the answer is read by the assistant, thereby the answer is given to the question. The BERT model used in the invention is being updated and it is planned to use the “GPT-3” or “Megatron” model as well.
The assistant of the invention has a three-dimensional holographic face image. Here, the desired face type can be selected, as well as the real three-dimensional face of a person known as the assistant's face or a character can be determined. By choosing the face of a loved one or the face of a deceased relative, the feeling of speaking with that person can be created. When the assistant speaks, his/her lips can move according to the word he/she speaks, thus making it feel like a real person is speaking. Instead of the three-dimensional hologram screen in the system of the invention, two-dimensional screens available in vehicles can also be used, and thus a more cost-effective structure can be obtained.

Claims

CLAIMS A smart car assistant with artificial intelligence designed for use in automobiles, VIP vehicles, commercial vehicles and smart home systems, which recognizes the user's face, detects the user's emotional state, offers suggestions according to the detected emotion, and has a three- dimensional holographic face to create an emotional bond between the user and the vehicle, has the ability to read news, give weather information, send e-mails, create notes and record alarms, and allows voice control of the equipment in the vehicle, makes it possible to perceive the commands given by speaking in a daily and natural speaking language, in Turkish or in the desired language, and to answer the questions asked, can translate in different languages, can tell the malfunctions that may occur in the vehicle, can transmit range information, connects to the phone and allows you to use the features of the phone, characterized in that it includes:
- at least one camera to recognize the user's face and detect the happiness, sadness, anger and similar feelings of the user,
- at least one microphone to detect user-directed voice commands and questions,
- at least one electronic motherboard to run the artificial intelligence software, where all the visual and audio data received are processed and the necessary actions are created,
- at least one electronic card containing motor drivers to control the equipment inside the vehicle,
- at least one three-dimensional hologram screen or at least one two- dimensional screen to give the assistant a face.
7
PCT/TR2022/050946 2021-12-13 2022-09-05 Smart vehicle assistant with artificial intelligence WO2023113717A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
TR2021/019768A TR2021019768A2 (en) 2021-12-13 2021-12-13 SMART VEHICLE ASSISTANT WITH ARTIFICIAL INTELLIGENCE
TR2021/019768 2021-12-13

Publications (1)

Publication Number Publication Date
WO2023113717A1 true WO2023113717A1 (en) 2023-06-22

Family

ID=85114232

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/TR2022/050946 WO2023113717A1 (en) 2021-12-13 2022-09-05 Smart vehicle assistant with artificial intelligence

Country Status (2)

Country Link
TR (1) TR2021019768A2 (en)
WO (1) WO2023113717A1 (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019213177A1 (en) * 2018-04-30 2019-11-07 Ramaci Jonathan E Vehicle telematic assistive apparatus and system
US20200211553A1 (en) * 2018-12-28 2020-07-02 Harman International Industries, Incorporated Two-way in-vehicle virtual personal assistant
US20200223352A1 (en) * 2019-01-14 2020-07-16 Samsung Eletrônica da Amazônia Ltda. System and method for providing automated digital assistant in self-driving vehicles
US20210078539A1 (en) * 2019-07-29 2021-03-18 Airwire Technologies Vehicle intelligent assistant using contextual data

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2019213177A1 (en) * 2018-04-30 2019-11-07 Ramaci Jonathan E Vehicle telematic assistive apparatus and system
US20200211553A1 (en) * 2018-12-28 2020-07-02 Harman International Industries, Incorporated Two-way in-vehicle virtual personal assistant
US20200223352A1 (en) * 2019-01-14 2020-07-16 Samsung Eletrônica da Amazônia Ltda. System and method for providing automated digital assistant in self-driving vehicles
US20210078539A1 (en) * 2019-07-29 2021-03-18 Airwire Technologies Vehicle intelligent assistant using contextual data

Also Published As

Publication number Publication date
TR2021019768A2 (en) 2021-12-21

Similar Documents

Publication Publication Date Title
US10893236B2 (en) System and method for providing virtual interpersonal communication
Hill Jr et al. Virtual Humans in the Mission Rehearsal Exercise System.
Weng et al. Conversational in-vehicle dialog systems: The past, present, and future
CN112204564A (en) System and method for speech understanding via integrated audio and visual based speech recognition
Bermúdez Understanding" I": language and thought
US11302325B2 (en) Automatic dialogue design
US9796095B1 (en) System and method for controlling intelligent animated characters
US20190251701A1 (en) System and method for identifying a point of interest based on intersecting visual trajectories
CN102039898A (en) Emotive advisory system
Yacobi Interart narrative:(Un) reliability and ekphrasis
US11308312B2 (en) System and method for reconstructing unoccupied 3D space
US10785489B2 (en) System and method for visual rendering based on sparse samples with predicted motion
EP3620319B1 (en) Method for operating a virtual assistant for a motor vehicle and corresponding backend system
US20190251350A1 (en) System and method for inferring scenes based on visual context-free grammar model
CN1646969A (en) Device for projecting an object in a space inside a vehicle
JP7266432B2 (en) AGENT DEVICE, CONTROL METHOD OF AGENT DEVICE, AND PROGRAM
JP2019158975A (en) Utterance system
WO2023113717A1 (en) Smart vehicle assistant with artificial intelligence
US20200317215A1 (en) Agent system, agent server, method of controlling agent server, and storage medium
WO2020087534A1 (en) Generating response in conversation
KR102460553B1 (en) Method, device and computer program for providing sign language response using neural network in avehicle
JP2002207413A (en) Action recognizing speech type language learning device
JP7469467B2 (en) Digital human-based vehicle interior interaction method, device, and vehicle
JP7425681B2 (en) Social ability generation device, social ability generation method, and communication robot
JP2022138930A (en) Information processing apparatus, information processing method, and trained model

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22908099

Country of ref document: EP

Kind code of ref document: A1