KR20220013581A - In-vehicle acoustic monitoring system for driver and passenger - Google Patents

Abstract

An inaudible signal is generated from one or a plurality of speakers installed in a vehicle. The inaudible signal reflected on a driver or a passenger of the vehicle is received by one or a plurality of microphones installed in the vehicle. A behavior induction acoustic pattern is detected based on the reflected ultrasonic waves. A behavior of the driver or the passenger of the vehicle is recognized by analyzing the behavior induction acoustic pattern. A warning is generated by the recognized behavior of the driver or the passenger of the vehicle.

Description

IN-VEHICLE ACOUSTIC MONITORING SYSTEM FOR DRIVER AND PASSENGER

BACKGROUND OF THE INVENTION Embodiments of the present invention relate generally to the operation of autonomous vehicles (ADVs), and more particularly, embodiments of the present invention relate to sensing systems for ADVs.

In the operation of an autonomous vehicle, real-time behavioral detection and monitoring of a driver and/or passengers regardless of the presence of an active driver is very important. Inappropriate behaviors such as cell phone use, eating, drowsy driving and aggressive behavior have the potential to significantly impair driving safety, especially when autonomous driving mode is turned off (or partially turned off). Other gestures, such as hand gestures, are available for vehicle control and interaction with the vehicle. However, in-vehicle cameras and visual solutions can pose privacy concerns. In particular, autonomous vehicles (ADVs) have always lacked effective methods for monitoring the vehicle's drivers or passengers.

One aspect of the present application provides a computer-implemented method for operating a vehicle, the method comprising: generating a voice signal, which is an inaudible voice signal, using one or a plurality of speakers disposed in the vehicle; receiving a voice signal reflected from a driver or passenger of the vehicle by one or a plurality of microphones disposed in the vehicle; detecting an action-inducing sound pattern based on the reflected voice signal received by the one or more microphones; determining a behavior of a driver or a passenger of the vehicle by analyzing the action-inducing sound pattern; and generating an alert according to an action of a driver or passenger of the vehicle, based on one or a set of a plurality of rules.

Another aspect of the present application provides a non-transitory machine-readable medium having instructions stored thereon, when the instructions are executed by a processor, to perform an operation on the processor, wherein the operation includes: generating an audio signal that is an inaudible audio signal using a plurality of speakers; receiving a voice signal reflected from a driver or passenger of the vehicle by one or a plurality of microphones disposed in the vehicle; detecting an action-inducing sound pattern based on the reflected voice signal received by the one or more microphones; determining a behavior of a driver or a passenger of the vehicle by analyzing the action-inducing sound pattern; and generating an alert according to an action of a driver or passenger of the vehicle, based on one or a set of a plurality of rules.

Another aspect of the present application provides a data processing system having a processor and a memory coupled to the processor to store instructions, and when the instructions are executed by the processor, cause the processor to execute operations and , The operation may include generating an inaudible voice signal using one or a plurality of speakers disposed in the vehicle; generating a voice signal that is an inaudible voice signal using one or a plurality of speakers disposed in the vehicle; receiving a voice signal reflected from a driver or passenger of the vehicle by one or a plurality of microphones disposed in the vehicle; detecting an action-inducing sound pattern based on the reflected voice signal received by the one or more microphones; determining a behavior of a driver or a passenger of the vehicle by analyzing the action-inducing sound pattern; and generating an alert according to an action of a driver or passenger of the vehicle, based on one or a set of a plurality of rules.

Embodiments of the present invention are provided in the form of examples and not limitations in each accompanying drawing, in which like reference numerals denote like elements.
1 is a block diagram illustrating a networked system according to an embodiment.
2 is a block diagram illustrating an example of an autonomous driving vehicle according to an embodiment.
3A-3B are block diagrams illustrating an example of an autonomous driving system used with an autonomous driving vehicle according to an embodiment.
4 is a block diagram illustrating an example of a monitoring system according to an embodiment.
5A is a block diagram illustrating a top view of an ADV with one or more speakers, according to one embodiment.
5B is a block diagram illustrating a side view of an ADV with one or more speakers according to an embodiment.
5C and 5D are block diagrams illustrating one or a plurality of microphones disposed in an ADV according to an embodiment.
6 is a flowchart illustrating an example of acoustically monitoring the behavior of a driver or passenger of an autonomous vehicle according to an embodiment.
7A shows an example of data received in acoustic monitoring according to an embodiment.
7B shows an example of an event detected in acoustic monitoring according to one embodiment.
7C shows an example of a power spectrum from acoustic monitoring according to one embodiment.
7D shows an example of an enhanced power spectrum from acoustic monitoring in accordance with one embodiment.
7E shows an example of an action-inducing acoustic pattern detected in the enhanced power spectrum.
7F shows an example of a behavior corresponding to a recognized, action-inducing sound pattern detected in FIG. 7E .
8 is a block diagram illustrating an example of ADV behavior and response detected in acoustic monitoring according to one embodiment.
9 is a flowchart illustrating a process of acoustically monitoring the behavior of a driver or passenger of an ADV.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Various embodiments and aspects of the present invention are described below with reference to the detailed description, in which the accompanying drawings illustrate the various embodiments. The following description and drawings are for explaining the present invention, and should not be construed as limiting the present invention. Numerous specific details are set forth in order to provide a thorough understanding of various embodiments of the invention. In only some instances, descriptions of well-known or conventional details are omitted in order to provide brevity of the description of the embodiments of the present invention.

In this specification, reference to "one embodiment" or "an embodiment" means that a specified feature, structure, or characteristic coupled to that embodiment may be included in at least one embodiment of the present invention. The phrases "in one embodiment" in various places in this specification are not all referring to the same embodiment.

According to some embodiments, acoustic sensing and the Doppler effect are used to sense the behavior of drivers and passengers of autonomous vehicles (ADVs). For example, in-vehicle audio systems are used to implement driver/passenger behavior detection. A voice generator or voice capture device (eg, a speaker and a microphone in a vehicle) is used as a transmitter and receiver of a voice signal (eg, an ultrasonic signal). The action-inducing sound pattern is extracted and analyzed using, for example, machine learning and deep learning methods, or by matching the action-inducing sound pattern with a predetermined sound pattern list representing a pre-classified action list. Based on the behavior and identity of the performer, ADV can infer the risk level of the behavior. ADV can select the corresponding driving strategy change and/or response, which improves driving safety and realizes intelligent and streamlined passenger-vehicle interaction.

According to some embodiments, the inaudible voice signal is generated by one or more voice generating devices (eg, a speaker disposed in a vehicle). The reflected voice signal reflected or returned by the driver or passenger of the vehicle is captured by one or more voice capture devices (eg, microphones) disposed in the vehicle. An action-inducing acoustic pattern is detected based on the reflected ultrasound signal. For example, by using an artificial intelligence (AI) model, or by comparing and matching the action-inducing sound pattern with a predetermined sound pattern list corresponding to the predetermined action list, the action-inducing sound pattern is analyzed by the driver or passenger of the vehicle. Recognize action. A response or alert is generated by a perceived action of a driver or passenger of the vehicle.

1 is a block diagram showing the configuration of an autonomous driving network according to an embodiment of the present invention. Referring to FIG. 1 , a network configuration 100 includes an autonomous vehicle (ADV) 101 communicatively coupled to one or a plurality of servers 103 , 104 via a network 102 . While one ADV is illustrated, several ADVs may be coupled to each other via network 102 and/or to servers 103 , 104 . Network 102 may be any type of network, such as a wired or wireless local area network (LAN), a wide area network (WAN) such as the Internet, a cellular network, a satellite network, or a combination thereof. Servers 103 and 104 may be any type of server or server cluster, for example, a network or cloud server, an application server, a backend server, or a combination thereof. The servers 103 and 104 may be a data analysis server, a content server, a traffic information server, a map and point of interest (MPOI) server, or a location server.

ADV refers to a vehicle that can be configured for autonomous driving mode, in which the vehicle navigates through an environment with little or no input from the driver. Such an ADV may include a sensor system having one or a plurality of sensors configured to detect information associated with the vehicle driving environment. The corresponding vehicle and the controller associated with the vehicle pass through the corresponding environment through navigation using the detected information. ADV 101 can drive in manual mode, fully autonomous driving mode or partially autonomous driving mode.

In one embodiment, the ADV 101 includes an autonomous driving system (ADS) 110 , a vehicle control system 111 , a wireless communication system 112 , a user interface system 113 , and a sensor system 115 . However, the present invention is not limited thereto. The ADV 101 may further include some common components included in a general vehicle, such as an engine, wheels, steering wheel, and transmission, and the components are the vehicle control system 111 and/or the ADS 110 for various communication Control may be performed using signals and/or commands, for example, an acceleration signal or command, a deceleration signal or command, a steering signal or command, a brake signal or command, and the like. In one embodiment, ADV 101 may include a monitoring system 116 for monitoring driver or passenger behavior based on voice signals. This will be described in detail below.

The components 110 to 116 may be communicatively connected to each other through an interconnector, a bus, a network, or a combination thereof. For example, the components 110 to 116 may be communicatively connected to each other through a controller local area network (CAN) bus. The CAN bus is a vehicle bus specification designed to allow microcontrollers and devices to communicate with each other in applications without a host computer. It is a message-based protocol initially designed for multiple electrical wiring in automobiles, but is being used in many other environments as well.

2 , in one embodiment, the sensor system 115 includes one or more cameras 211 , a global positioning system (GPS) unit 212 , an inertial measurement unit (IMU) 213 , a radar unit 214 and a light detection and range (LIDAR) unit 215 . The GPS unit 212 may include a transceiver operable to provide information regarding the location of the ADV. The IMU unit 213 may detect a change in the position and orientation of the ADV based on the inertial acceleration. The radar unit 214 may represent a system for detecting an object in the local environment of the ADV using a radio signal. In some embodiments, in addition to detecting the object, the radar unit 214 may also sense the speed and/or forward direction of the object. The LIDAR unit 215 may detect an object in an environment in which the ADV is located with a laser. Besides other system components, the LIDAR unit 215 may also include one or more laser light sources, a laser scanner, and one or more detectors. The camera 211 may include one or more devices for collecting images of the ADV's surrounding environment. The camera 211 may be a still camera and/or a video camera. The camera may be moved mechanically, for example by installing the camera on a rotating and/or inclined platform.

The sensor system 115 may further include other sensors such as, for example, a sonar sensor, an infrared sensor, a steering sensor, a throttle sensor, a brake sensor, and an audio sensor (eg, a microphone). The audio sensor may be configured to collect voice from the environment surrounding the ADV. The steering sensor may be configured to sense a steering angle of a steering wheel, a wheel of a vehicle, or a combination thereof. The throttle sensor and the brake sensor sense a throttle position and a brake position of the vehicle, respectively. In some situations, the throttle sensor and brake sensor may be integrated into an integrated throttle/brake sensor.

In one embodiment, the vehicle control system 111 includes, but is not limited to, a steering unit 201 , a throttle unit 202 (also referred to as an acceleration unit) and a brake unit 203 . The steering unit 201 is used to adjust the direction or forward direction of the vehicle. The throttle unit 202 is used to control the speed of the motor or engine, and the speed of the motor or engine further controls the speed and acceleration of the vehicle. The brake unit 203 reduces the speed of the vehicle by reducing the speed of the wheels or tires of the vehicle by providing friction. It should be noted that the components shown in FIG. 2 may be implemented by hardware, software, or a combination thereof.

Referring back to FIG. 1 , the wireless communication system 112 enables communication between the ADV 101 and external systems such as devices, sensors, and other vehicles. For example, the wireless communication system 112 may communicate wirelessly over a communication network, such as directly wirelessly communicating with one or a plurality of devices, or communicating with the servers 103 , 104 over the network 102 . The wireless communication system 112 may use any cellular communication network or wireless local area network (WLAN), for example, may communicate with other components or systems using WiFi. The wireless communication system 112 may communicate directly with a device (eg, a passenger's mobile device, a display device, a speaker in the vehicle 101 ) using, for example, an infrared link, Bluetooth, or the like. The user interface system 113 is a part of a peripheral device implemented in the vehicle 101 , and may include, for example, a keyboard, a touch screen display device, a microphone, and a speaker.

Some or all of the functions of the ADV 101 may be controlled or managed by the ADS 110 , particularly when operated in an autonomous driving mode. ADS 110 receives information from sensor system 115 , control system 111 , wireless communication system 112 , and/or user interface system 113 , processes the received information, and routes or After planning the route, hardware (eg, processor, memory, storage) and software (eg, operation system, planning and routing program) necessary to drive the vehicle 101 based on the planning and control information includes In some cases, the ADS 110 may be integrated with the vehicle control system 111 .

For example, a user as a passenger may specify the origin and destination of a route, for example via a user interface. The ADS 110 acquires route-related data. For example, the ADS 110 may obtain location and route data from an MPOI server, which may be part of the servers 103 and 104 . The location server provides a location service, and the MPOI server provides a map service and a POI of a specific location. In some cases, such location and MPOI information may be cached locally in persistent storage of the ADS 110 .

When the ADV 101 moves along a route, the ADS 110 may also obtain real-time traffic information from a traffic information system or server (TIS). It should be noted that the servers 103 and 104 are operable by a third party entity. In some cases, the functions of the servers 103 and 104 may be integrated with the ADS 110 . Based on real-time traffic information, MPOI information and location information, and real-time local environment data detected or detected by the sensor system 115 (eg, obstacles, objects, nearby vehicles), the ADS 110 selects an optimal route It is possible to safely and efficiently arrive at a designated destination by driving the vehicle 101 through the control system 111 according to the planned and planned route, for example.

The server 103 may be a data analysis system for executing data analysis services for various clients. In one embodiment, the data analysis system 103 includes a data collector 121 and a machine learning engine 122 . The data collector 121 collects driving statistics data 123 from various vehicles (ADV or a vehicle driven by a human driver). Driving statistics data 123 includes transmitted driving commands (eg, throttle, brake, steering commands) and responses of the vehicle (eg, speed, acceleration, deceleration, direction) captured at different points in time by sensors of the vehicle. It contains information indicating The driving statistics data 123 may also include information describing the driving environment at different points in time, for example, a route (including a departure location and a destination location), an MPOI, a road condition, a weather condition, and the like.

Based on the driving statistics data 123 , the machine learning engine 122 generates or trains a rule set, algorithm and/or predictive model 124 according to various purposes. In one embodiment, the algorithm 124 is an algorithm or model for generating an inaudible speech signal by one or a plurality of speakers, an algorithm or model for receiving an inaudible reflected speech signal by one or a plurality of microphones. , an algorithm or model for detecting behavior-inducing acoustic patterns based on reflected ultrasound signals, an algorithm or model for recognizing driver or passenger behavior by analyzing behavior-inducing acoustic patterns, and determining the risk level of driver or passenger behavior and/or algorithms or models for generating a response according to the determined, risk level of the driver's or passenger's behavior. The algorithm 124 can then be uploaded to the ADV and used in real-time for the autonomous driving process.

3A and 3B are block diagrams illustrating an example of an autonomous driving system used with an ADV according to an embodiment. System 300 may be implemented as part of ADV 101 of FIG. 1 , including but not limited to ADS 110 , control system 111 , and sensor system 115 . 3A to 3B , the ADS 110 includes a positioning module 301 , a detection module 302 , a prediction module 303 , a determination module 304 , a planning module 305 , a control module 306 and including but not limited to routing module 307 .

The modules 301 to 307 may be implemented in software, hardware, or a combination thereof. For example, such a module may be loaded into the memory 351 installed in the persistent storage device 352 and executed by one or a plurality of processors (not shown). It should be noted that some or all of these modules may be communicatively coupled to, or integrated with, some or all of the modules of the vehicle control system 111 of FIG. 2 . Some of the modules 301 to 307 may be integrated into an integrated module.

The positioning module 301 (eg, using the GPS unit 212) checks the current location of the ADV 300, and manages all data related to the user's route or route. The positioning module 301 (also referred to as a map and route module) manages all data related to a user's route or route. The user can log in via the user interface, for example, and specify the origin and destination of the route. The positioning module 301 acquires route-related data by communicating with other components such as maps and route data 311 of the ADV 300 . For example, the positioning module 301 may obtain location and route data from a location server, a map, and a POI (MPOI) server. The location server provides a location service, and the MPOI server provides a map service and a POI of a specific location, and may be cached as part of the map and route data 311 . When the ADV 300 moves along a route, the positioning module 301 may obtain real-time traffic information from a traffic information system or server.

The sensing module 302 determines sensing of the surrounding environment based on the sensor data provided by the sensor system 115 and the positioning information acquired by the positioning module 301 . The sensed information may indicate what the general driver senses in the vicinity of the vehicle the driver is currently driving. Sensing may include, for example, lane configuration employing object types, signals from traffic lights, relative positions of other vehicles, pedestrians, buildings, crosswalks, or other traffic-related signs (e.g., stop signs, yield signs), etc. can Lane configuration describes one or more lanes, such as, for example, the shape of the lanes (e.g., straight or curved), lane width, number of lanes on the road, one-way or two-way lanes, merging or classified lanes, exit lanes, etc. information is included.

The sensing module 302 may include a computer vision system or the functionality of a computer vision system to process and analyze images collected from one or more cameras to recognize objects and/or features in the ADV environment. The object may include traffic signals, road boundaries, other vehicles, pedestrians and/or obstacles, and the like. Computer vision systems may use object recognition algorithms, video tracking, and other computer vision technologies. In some embodiments, the computer vision system may create a map of the environment to track the object, estimate the object's velocity, and the like. The detection module 302 may detect an object based on data from other sensors provided by other sensors such as radar and/or LIDAR.

For each object, the prediction module 303 predicts how the object will represent in this case. Prediction is performed based on the sensed data that detects the driving environment at a point in time in consideration of a series of map/route data 311 and traffic rules 312 . For example, when the object is in a vehicle in the opposite direction and an intersection is included in the current driving environment, the prediction module 303 predicts whether the vehicle will go straight or turn. When the sensing data indicates that there is no traffic light at the intersection, the prediction module 303 may predict that the vehicle must come to a complete stop before entering the intersection. When the sensed data indicates that the vehicle is currently in the left-turn only lane or the right-turn only lane, the prediction module 303 may predict that the vehicle is likely to turn left or right, respectively.

For each object, the decision module 304 determines how to deal with the object. For example, for a particular object (eg, another vehicle on an intersecting route) and metadata describing the object (eg, speed, direction, steering angle), the determination module 304 may determine how to encounter that object. (eg, overtaking, yielding, stopping, exceeding). The decision module 304 may make the above decision according to a rule set such as a traffic rule or a driving rule 312 , and the rule set may be stored in the permanent storage device 352 .

The routing module 307 is configured to provide one or more routes or routes from a starting point to a destination point. For example, for a given route from the origin to the destination, received from the user, the routing module 307 obtains the route and map data 311 and lists all possible routes or routes from the origin to the destination. decide The routing module 307 may generate a reference line in the form of a topographic map in which each route for arriving at a destination from a source is determined. Reference lines indicate ideal routes or routes that are not subject to any interference from other objects such as other vehicles, obstacles or traffic conditions. In other words, if there are no other vehicles, pedestrians or obstacles on the road, the ADV follows the reference line precisely or strictly. The topographic map is then provided to the determining module 304 and/or the planning module 305 . The decision module 304 and/or the planning module 305 examines all possible routes and other data provided by other modules, eg, traffic conditions by the positioning module 301 , by the sensing module 302 . One of the optimal routes is selected and corrected according to the sensed driving environment and the traffic conditions predicted by the prediction module 303 . The actual route or route for controlling the ADV may be similar to and different from the reference line provided by the routing module 307 according to a specific driving environment in time.

The planning module 305 may configure a route or route and travel parameters (eg, distance, speed and/or distance or steering angle). That is, for a given object, the decision module 304 determines what to do with that object, while the planning module 305 determines how to execute it. For example, for a given object, the determining module 304 may determine to overtake the object, and the planning module 305 may determine to overtake the object to the left or right. The planning and control data is generated by the planning module 305 and includes information describing how the vehicle 300 will travel in the next travel cycle (eg, the next route/route segment). For example, planning and control data may instruct vehicle 300 to travel 10 meters at a speed of 30 miles/hour (mph) and then change to the right lane at a speed of 25 mph.

The control module 306 controls and drives the ADV by sending an appropriate command or signal to the vehicle control system 111 according to the route or route defined by the planning and control data, based on the planning and control data. The planning and control data transfers the vehicle from a first point on a route or route to a second point along the route or route, using appropriate vehicle settings or driving parameters (eg, throttle, brake, steering commands) at different points in time, along the route or route. Include enough information to drive to

In one embodiment, the planning phase is implemented in several planning cycles (also referred to as travel cycles). For example, it is executed at a time interval of 100 milliseconds (ms). One or a plurality of control commands are transmitted based on the planning and control data for each planning cycle or travel cycle. That is, every 100 ms, the planning module 305 plans the next route segment or route segment including, for example, the target position and the time required for the ADV to reach the target position. Alternatively, the planning module 305 may additionally designate a specific speed, direction and/or steering angle. In one embodiment, the planning module 305 plans a route segment or route segment for the next predetermined period of time (eg, for 5 seconds). For each planning cycle, the planning module 305 plans the target position of the current cycle (eg, next 5 seconds) based on the target position planned in the previous cycle. The control module 306 then generates one or more control commands (eg, throttle, brake, steering control commands) based on the planning and control data of the current cycle.

It should be noted that the decision module 304 and the planning module 305 may be integrated into an integrated module. The determining module 304 / planning module 305 may include a navigation system or a function of a navigation system for determining a driving route of the ADV. For example, a navigation system can determine a set of speeds and forward directions that affect the movement of the ADV along a path such that the path advances the ADV based on a road-based path to its final destination; At the same time, it is possible to substantially avoid the detected obstacle. The destination may be set by a user input through the user interface system 113 . The navigation system may dynamically update the driving route while the ADV is driving. The navigation system may merge data from the GPS system and one or more maps to determine the ADV's driving route.

4 is a block diagram illustrating an example of a monitoring system 116 in accordance with one embodiment. Monitoring system 116 may be implemented to monitor driver or passenger behavior and generate a response. Referring to FIG. 4 , the monitoring system 116 includes a speaker unit 401 , a microphone unit 402 , a detector unit 403 , a recognition unit 404 , a risk level unit 405 and a response unit 406 . However, it is not limited thereto. The monitoring system 116 is configured to recognize and monitor the behavior of the moving vehicle in order to increase the safety of driving and realize the intelligent and smooth interaction between the passenger and the vehicle. Monitoring system 116 bridges the gap between task recognition and vehicle operation mode adjustment to form a closed-loop system.

In one embodiment, the speaker unit 401 includes one or more voice generators, such as speakers, and is configured to generate an inaudible voice signal using one or more voice generators disposed at respective locations within the vehicle. do. The voice signal may be reflected or returned from the driver or passenger of the vehicle. The microphone unit 402 includes one or a plurality of voice capture devices, such as a microphone, and uses one or a plurality of microphones disposed at different locations within the vehicle to provide inaudible reflected voices reflected or returned from the driver or passenger of the vehicle. configured to acquire a signal. The detector unit 403 is configured to detect the action-inducing acoustic pattern based on the reflected ultrasound signal. The recognition unit 404 is configured to analyze the action-inducing acoustic pattern to determine and recognize the action of a driver or passenger of the vehicle.

The voice signal is propagated through the air in one or more speakers, and is received by one or more microphones. For example, based on the Doppler effect, the sound wave is likely to be deflected or interrupted by the user's movement along the way, but still reach the microphone. For example, the voice signal may hit the driver or passenger of the vehicle, and may be reflected back from the driver or passenger. The voice signal reflected from the driver or passenger may cause a sudden change in the voice signal due to the Doppler effect.

In one embodiment, the behavior of the driver or passenger may be determined by applying an AI model or a machine learning model to a feature set extracted from the reflected speech signal. In one embodiment, the behavior may be determined by comparing or matching an acoustic pattern to a predefined list of acoustic patterns representing a predefined list of behaviors. By analyzing acoustic patterns, human behavior can be determined. For example, when a driver or passenger of ADV moves, based on the Doppler effect, a reflected signal (inaudible audio signal reflected from the driver or passenger) may cause a sudden change in the audio signal. The Doppler effect (or Doppler shift) is the change in wave frequency for an observer moving with respect to a wave source. A sudden change in the voice signal may indicate a behavioral event detected from the driver or passenger. In one embodiment, after applying a band-pass filter to the reflected speech signal, a band-stop filter may be applied to remove an audio component having an exact frequency of the probe signal, wherein the probe signal is generated by one or a plurality of speakers It may be an inaudible voice signal. Accordingly, based on the Doppler effect, it is possible to maintain only the reflected voice signal generated or caused by the movement of the human body.

The risk level unit 405 is configured to determine the risk level of the driver's or passenger's action based on the driver's or passenger's action, by the set of risk level thresholds. For example, a certain behavior of a passenger may not be dangerous, but the same behavior of a driver may be considered dangerous. The response unit 406 is configured to generate a response or an alert according to the determined, risk level of the driver's or passenger's action.

5A is a block diagram 500a illustrating a plan view of one or a plurality of speakers 510 disposed in the ADV 501 . 5B is a block diagram 500b illustrating a side view of one or a plurality of speakers 510 disposed in the ADV 501 . 5A and 5B , the monitoring system 116 includes a speaker unit 401 including one or a plurality of speakers 510 . One or more speakers 510 are configured to generate a voice signal inaudible to humans. The voice signal can be generated with a fixed frequency or a modulated variable frequency. For example, one or more speakers 510 are configured to generate an ultrasonic signal. The frequency range that the average human can hear is about 20Hz to 20kHz. Therefore, in general, audio signals outside this range (>20 kHz or <20 Hz) are inaudible to humans. Generally, sounds with a frequency of 20 kHz or higher are considered ultrasound. Low-frequency sounds that cannot be detected by the human ear are called infrasound. In one embodiment, the inaudible voice signal may include an ultrasonic signal. In one embodiment, the inaudible voice signal may include an infrasound voice signal.

One or a plurality of speakers 510 may be disposed at the four corners of the ADV. One or a plurality of speakers 510 may be disposed on the left front side, left rear side, right front side, and right rear side of the ADV. In one embodiment, one or more speakers 510 may utilize the ADV's in-vehicle audio system. In one embodiment, one or more speakers 510 may be configured to generate ultrasound signals. One or a plurality of speakers 510 may be any type or all types of speakers as long as they can generate a voice signal inaudible to humans.

5C and 5D are block diagrams 500c and 500d of one or more microphones (eg, 521 - 527 ) disposed in the ADV 501 . One or more microphones may be disposed in the upper left front (eg 521), upper right front (eg 522), or between the driver's seat and the passenger seat (eg 523). One or more microphones may be located on the rear side of the driver's seat (eg 524) or on the back side of the passenger seat (eg 525) or between the rear seats of two passengers (eg 526) or centrally located on the front (eg, 527). One or a plurality of microphones may be disposed at different positions within the ADV 501 , and the voice signal reflected from the driver or passenger is transmitted to an obstacle in the ADV (eg, a vehicle seat, door, console box, rear corner, etc.) not disturbed One or a plurality of microphones (eg, 521 to 527) is configured to receive a voice signal reflected from the driver or passenger. An inaudible voice signal (eg, an ultrasonic signal) generated by one or a plurality of speakers (eg, 510) may be incident on the driver or passenger, and the voice signal may be reflected from the driver or passenger . The voice signal reflected from the driver or passenger may be received by one or a plurality of microphones (eg, 521 to 527 ).

6 is a flowchart illustrating an example of acoustically monitoring the behavior of a driver or passenger of an autonomous vehicle according to an embodiment. The monitoring system 116 is configured to monitor the behavior of the driver or passenger of the ADV based on the acoustic monitoring. In this way, there is no need to worry about excessive leakage of personal information. It is possible to intervene in dangerous behavior by the driver, thereby increasing driving safety. In addition, it can improve passenger comfort and realize smooth interaction between passengers and vehicles.

At block 601a, one or more speakers or other voice generators of the ADV may generate an inaudible voice signal (eg, an ultrasonic signal).

As shown in block 601, one or more microphones or other audio receivers may receive an inaudible audio signal (eg, an ultrasonic signal). Data obtained from one or more microphones or other voice receivers (as described in conjunction with FIG. 4 ) may be input to the detector unit 403 of the monitoring system 118 . 700A of FIG. 7A shows an example of data received from one or more microphones or other voice receivers.

At block 602, a driver or passenger behavioral event may be detected. A behavioral event is an event of an action or movement caused by a driver or passenger of an ADV. For example, when a driver or passenger of ADV moves, based on the Doppler effect, a reflected signal (inaudible audio signal reflected from the driver or passenger) may cause a sudden change in the audio signal. The Doppler effect (or Doppler shift) is the change in wave frequency for an observer moving relative to the wave source. A sudden change in the voice signal may indicate a behavioral event detected in the driver or passenger. 700B of FIG. 7B shows an example of a behavioral event detected in a voice signal.

In block 603, a frame of the speech signal is segmented for feature extraction and behavior recognition.

In block 604, a behavioral characteristic including an action-inducing sound pattern is extracted from the voice signal. In block 605, the audio signal may be filtered to denoise. The audio signal may be filtered to reduce noise outside the frequency range of interest. For example, the filter may be a bandpass filter or a bandstop filter. For example, the inaudible audio signal is an ultrasonic signal, and the filter may be configured to filter the audio signal outside a predetermined range. For example, the predetermined range may be 19 kHz to 21 kHz. The audio signal outside the predetermined range may be noise or an outlier.

In block 606, a power spectrum of the filtered speech signal may be obtained. The spectrogram intuitively represents the frequency spectrum of a signal that changes with time. The power spectrum represents the power level of a signal in each frequency band over time. In one embodiment, after applying the band-pass filter to the audio signal, the band-stop filter may be applied to remove an audio component having the correct frequency of the probe signal, and the probe signal is generated by one or a plurality of speakers. It may be an inaudible voice signal. Accordingly, based on the Doppler effect, only the voice signal generated or caused by the movement of the human body may be withheld. For example, the band-stop filter may block a voice signal having an undesired frequency range greater than a first predetermined threshold and less than a second predetermined threshold. A voice signal greater than the first predetermined threshold and less than the second predetermined threshold is dominated by the probe signal. In one embodiment, the probe signal may have a frequency of 20 kHz. As an example, the first predetermined threshold value may be 20.2 kHz, and the second predetermined threshold value may be 19.8 kHz.

Then, a short-time Fourier transform (STFT) is applied to obtain the power spectrum of the speech signal. 7C shows an example of a power spectrum 700c of a voice signal. As shown in FIG. 7C , the x-axis represents time, the y-axis represents frequency, and the value of each coordinate point represents the amplitude of the power of the audio signal. In the power spectrum, different colors or different shades of color represent different power levels, which can represent different behaviors. In FIG. 7C , a light color (yellow) indicates a high power level, and a dark color (dark blue) indicates a low power level.

In block 607, an advanced processing and enhanced power spectrum may be obtained based on the power spectrum. 7d shows an example of an enhanced power spectrum 700d. A threshold of the power value may be determined based on the power spectrum 700c. The threshold of the power value may be determined by a voice signal and related motion. The enhanced power spectrum 700d may be obtained for each point of the power spectrum 700c by setting the amplitude of the power value of the corresponding point to 0 when the power value of the corresponding point is less than the threshold value of the power value. An enhanced power spectrum 700d may be obtained by applying an additional image denoising or smoothing filter to obtain a better effect.

At block 608 , the driver's or passenger's behavior is detected based on the enhanced power spectrum, for example, by comparing or matching the power spectrum with a predetermined power spectrum list corresponding to the predetermined behavioral list. It can be recognized based on the acoustic pattern. 7E shows an example of an action-inducing acoustic pattern 710 detected in the enhanced power spectrum 700e. 7F shows an example of a recognized action 720 corresponding to a detected action-inducing sound pattern 710 . Different actions or movements of the driver or passengers are corresponded to different action-inducing acoustic patterns by the Doppler effect. As an example, the applause action shown in FIG. 7F corresponds to the action-inducing sound pattern 710 shown in FIG. 7E . When there is movement or behavior of the driver or passenger (hand gestures, phone use, eating, nodding, etc.), action-inducing acoustic patterns can be detected. Machine learning and deep learning methods may be used to analyze a behavioral sound pattern (eg, 710 ) to recognize a corresponding behavior (eg, 720 ).

In one embodiment, it further determines whether the action is that of a vehicle driver or that of a passenger. If the behavior is that of the driver, there is a possibility that a potential risk of safety problem exists. Therefore, it is important to determine whether the behavior is that of the vehicle driver or that of a passenger. Based on the source of the detected voice signal (eg, inaudible voice signal), it is possible to determine whether the behavior is the behavior of the vehicle driver or the behavior of the passenger.

8 is a block diagram illustrating an example of detected, behavior and response of an ADV. The detected behavior 801 may include a driver's behavior 802 or a passenger's behavior 803 . For example, the driver's behavior 802 may include hand gestures, phone use, eating, drowsy driving or careless driving, and the like. The passenger's behavior 803 may include hand gestures, phone use, eating, drowsiness/sleep, and the like. There is also the possibility that other actions of the driver or passengers may be detected. As shown in FIG. 8 , it is possible to determine the perceived risk level of the driver's or passenger's behavior.

As an example, the perceived risk level of the behavior can be divided into no risk, potential risk, or high risk. For example, the driver's hand gestures may be determined to be non-hazardous, the driver's phone use and eating may be determined to be potentially hazardous, and drowsy driving or careless driving may be determined to be high risk. The behavior of the passenger may be determined to be risk-free. In addition, different types of behavior can be classified in different ways.

The ADV's response can be generated based on a determined, risk level of the driver's or passenger's behavior. In one embodiment, if the determined risk level is a potential risk, the ADV may generate an alert and activate an autonomous driving mode to temporarily take over driving, as shown in block 804 .

In one embodiment, if the determined risk level is high risk, as shown in block 805, the ADV generates an alert and activates an autonomous driving mode to temporarily take over driving to slow down and stop by the side of the road. have. However, if the determined hazard level is a potential hazard, if there is no response from the driver after generating the alert, the ADV can take over the drive and slow down to a stop on the side of the road.

If the determined risk level is no risk and the behavior is the driver's behavior (eg, the driver beckoning behavior), then, as shown in block 806, the ADV turns on the radio/music, one or more You can respond by adjusting the audible volume of your speaker or continuously monitoring it.

If the detected behavior is that of a passenger and the vehicle is in an autonomous driving mode, the ADV may slow the ADV or switch to the comfort mode of the autonomous driving mode, as shown in block 808 , or block As shown at 807 , the ADV may lower the audible volume of one or more speakers.

9 is a flowchart illustrating a process of acoustically monitoring the behavior of a driver or passenger of an ADV. Through this process, the behavior of drivers or passengers in ADV can be monitored, but there is no need to worry about excessive leakage of personal information. It is possible to determine the level of risk of a driver's or passenger's behavior and generate a response to enhance the safety and comfort of driving. Process 900 may be performed by processing logic including software, hardware, or a combination thereof. For example, process 900 may be performed by monitoring system 116 .

Referring to Fig. 9, in operation 901, the processing logic generates an inaudible audio signal through one or a plurality of speakers disposed in the vehicle. In one embodiment, the inaudible audio signal is continuously generated by one or a plurality of speakers disposed in the vehicle, and the inaudible audio signal is an ultrasonic signal.

In operation 902, the processing logic receives the inaudible reflected audio signal via one or more microphones disposed within the vehicle. In one embodiment, the one or the plurality of microphones are disposed above the left front side, above the right front side, between the driver's seat and the front passenger's seat, the rear side of the driver's seat or the front passenger's seat, or between the two rear seats of the vehicle.

In operation 903, the processing logic detects an action-inducing acoustic pattern based on the reflected speech signal. In one embodiment, the processing logic may detect a behavioral event based on the reflected speech signal. In one embodiment, the processing logic may filter the reflected speech signal.

In one embodiment, the processing logic may also extract a power spectrum based on the reflected speech signal. In one embodiment, the processing logic may obtain an enhanced power spectrum based on the power spectrum. In one embodiment, the processing logic may obtain a call-to-action acoustic pattern based on the enhanced power spectrum.

In operation 904, the processing logic analyzes the action-inducing acoustic pattern to recognize the behavior of the driver or passenger of the vehicle. In one embodiment, the processing logic may determine whether the action is that of a driver of the vehicle or that of a passenger of the vehicle.

In operation 905, the processing logic generates a response by the recognized, action of the driver or passenger of the vehicle.

In one embodiment, the processing logic may determine a perceived risk level of the driver or passenger behavior and may also generate a response based on the determined and perceived risk level of the driver or passenger behavior.

In one embodiment, if the action is a driver's action, the processing logic generates an alert in response to the determined risk level being high risk while simultaneously activating the autonomous driving mode and slowing down to a roadside stop, or the determined risk In response to the level being a potential hazard, it generates an alert and activates an autonomous driving mode.

In one embodiment, if the action is that of a passenger and the vehicle is in an autonomous driving mode, the processing logic may slow the vehicle down, or switch to a comfort mode in the autonomous driving mode.

It should be noted that some or all of the components as shown and described above may be implemented in software, hardware, or a combination thereof. For example, these components may be implemented as software installed and stored on a persistent storage device, and the software is loaded into memory and executed by a processor (not shown), whereby the processes or operations described throughout this application are executed. can be performed. Alternatively, such components may execute programmed or embodied in dedicated hardware, such as, for example, an integrated circuit (eg, application specific integrated circuits or ASICs), digital signal processors (DSPs), or field programmable gate arrays (FPGAs)). It is implemented as executable code, which executable code can be accessed by applications through its drivers and/or operating system. These components may also be implemented as specific hardware logic in a processor or processor core as part of an instruction set accessible through one or more specific instructions by a software component.

Portions of the foregoing detailed description appear according to algorithmic and symbolic representations of operations on bits of data within a computer memory. Descriptions and representations of these algorithms are the manner used by those skilled in the data processing arts to most effectively convey their work to others skilled in the art. In the present context, an algorithm is generally considered to be a coherent sequence of operations that results in a desired result. These manipulations refer to manipulations that must be physically manipulated and controlled with respect to a physical quantity.

However, all such similar terms are intended to be associated with the appropriate physical quantities, and are merely convenient notations for application to these quantities. Unless otherwise specified in the above discussion, throughout the specification, discussions using terms (such as those set forth in the claims) should be understood to refer to the operation and processing of a computer system or similar electronic computing device, and A computer system or electronic computing device controls data represented as physical (electronic) quantities in registers and memories of the computer system, and similarly stores the data in computer system memory or registers or other types of information storage, transmission or display devices. It is converted into other data expressed as a physical quantity.

An embodiment of the present invention also relates to an apparatus for performing an operation in the text. Such a computer program is stored in a non-transitory computer-readable medium. Machine-readable media includes any mechanism for storing information in a form readable by a machine (eg, a computer). For example, machine-readable (eg, computer-readable) media may include machine (eg, computer)-readable storage media (eg, read-only memory (“ROM”), random access memory (“RAM”) ”), magnetic disk storage media, optical storage media, flash memory devices).

A process or method illustrated in the foregoing drawings is performed by processing logic including hardware (eg, circuitry, dedicated logic, etc.), software (eg, embodied in a non-transitory computer-readable medium), or a combination thereof. can be performed. Although the above process or method has been described with some sequential operations, it should be understood that some of the above operations may be performed according to a different order. In addition, some operations may be performed in parallel instead of sequentially.

Embodiments of the present invention have not been described with reference to any particular programming language. It should be understood that a variety of programming languages may be used to implement the teachings of embodiments of the present invention as described herein.

In the above specification, embodiments of the present invention are described with reference to specific exemplary embodiments of the present invention. It will be apparent that various modifications may be made to the present invention without departing from the broader spirit and scope of the invention as set forth in the appended claims. Accordingly, the specification and drawings are to be understood in an illustrative rather than a restrictive sense.

Claims (21)

A computer implemented method for operating a vehicle, comprising:
generating a voice signal that is an inaudible voice signal using one or a plurality of speakers disposed in the vehicle;
receiving a voice signal reflected from a driver or passenger of the vehicle by one or a plurality of microphones disposed in the vehicle;
detecting an action-inducing sound pattern based on the reflected voice signal received by the one or more microphones;
determining a behavior of a driver or a passenger of the vehicle by analyzing the action-inducing sound pattern; and
generating an alert according to an action of a driver or passenger of the vehicle, based on one or a set of a plurality of rules;
A computer implemented method for operating a vehicle, comprising: According to claim 1,
A computer implemented method for operating a vehicle, wherein the voice signal includes an ultrasonic signal. According to claim 1,
The computer implemented method for operating a vehicle, wherein the one or more microphones are disposed in the upper left front, upper right front, between a driver's seat and a passenger seat, behind the driver's seat or the front passenger seat, or between two rear seats of the vehicle. According to claim 1,
The step of determining the behavior of a driver or a passenger of the vehicle by analyzing the action-inducing sound pattern includes matching the action-inducing sound pattern with a predetermined sound pattern list displaying a predetermined action list. A computer implemented method for manipulation. According to claim 1,
The computer-implemented method for operating a vehicle, wherein detecting the action-inducing sound pattern based on the reflected voice signal includes extracting a power spectrum based on the reflected voice signal. 6. The method of claim 5,
wherein analyzing the action-inducing sound pattern to determine the behavior of a driver or passenger of the vehicle comprises comparing the power spectrum with a predetermined power spectrum associated with a particular action. Way. According to claim 1,
A computer implemented method for operating a vehicle, further comprising: determining a risk level of the behavior of the driver or the passenger; and generating the alert based on the risk level of the behavior of the driver or the passenger. 8. The method of claim 7,
In the step of generating the alert based on the risk level of the behavior of the driver or the passenger, when the behavior is the behavior of the driver, the determined risk level is higher than a predetermined risk threshold, generating an alert. and simultaneously activating the autonomous driving mode to reduce the speed and stop the vehicle at the side of the road. 8. The method of claim 7,
The generating of the alert based on the risk level of the behavior of the driver or the passenger may include: if the behavior is the behavior of the passenger and the vehicle is in an autonomous driving mode, reduce the speed of the vehicle or A computer-implemented method for operating a vehicle, comprising switching from a driving mode to a comfort mode. A non-transitory machine-readable medium having instructions stored thereon, comprising:
When the instruction is executed by the processor, cause the processor to perform an operation, wherein the operation comprises:
generating a voice signal that is an inaudible voice signal using one or a plurality of speakers disposed in the vehicle;
receiving a voice signal reflected from a driver or passenger of the vehicle by one or a plurality of microphones disposed in the vehicle;
detecting an action-inducing sound pattern based on the reflected voice signal received by the one or more microphones;
determining a behavior of a driver or a passenger of the vehicle by analyzing the action-inducing sound pattern; and
and generating an alert according to an action of a driver or passenger of the vehicle based on one or a set of a plurality of rules. 11. The method of claim 10,
wherein the audio signal comprises an ultrasound signal. 11. The method of claim 10,
wherein the one or more microphones are disposed in the upper left front, upper right front, between a driver's seat and a passenger seat, behind the driver's seat or the front passenger seat, or between two rear seats of the vehicle. 11. The method of claim 10,
The step of analyzing the action-inducing sound pattern to determine the behavior of a driver or a passenger of the vehicle includes matching the action-inducing sound pattern with a predetermined sound pattern list displaying a predetermined action list, Machine-readable media. 11. The method of claim 10,
The detecting of the action-inducing sound pattern based on the reflected speech signal includes extracting a power spectrum based on the speech signal. 15. The method of claim 14,
and analyzing the action-inducing sound pattern to determine the behavior of a driver or passenger of the vehicle comprises comparing the power spectrum to a predetermined power spectrum associated with a particular behavior. 11. The method of claim 10,
wherein the manipulation further comprises determining a risk level of the behavior of the driver or the passenger, and generating the alert based on the risk level of the behavior of the driver or the passenger. . 17. The method of claim 16,
The generating the alert based on the risk level of the driver's or the passenger's behavior may include generating an alert in response to the risk level being higher than a predetermined risk threshold if the behavior is the driver's behavior. and simultaneously activating an autonomous driving mode to reduce speed to stop at the side of the road. 17. The method of claim 16,
The generating of the alert based on the risk level of the behavior of the driver or the passenger may include: if the behavior is the behavior of the passenger and the vehicle is in an autonomous driving mode, reduce the speed of the vehicle or A non-transitory machine readable medium comprising transitioning from a driving mode to a comfort mode. processor and
A data processing system comprising: a memory coupled to the processor for storing instructions, the memory configured to cause the processor to execute an operation when the instructions are executed by the processor, the data processing system comprising:
The operation is
generating a voice signal that is an inaudible voice signal using one or a plurality of speakers disposed in the vehicle;
receiving a voice signal reflected from a driver or passenger of the vehicle by one or a plurality of microphones disposed in the vehicle;
detecting an action-inducing sound pattern based on the reflected voice signal received by the one or more microphones;
determining a behavior of a driver or a passenger of the vehicle by analyzing the action-inducing sound pattern; and
generating an alert according to an action of a driver or passenger of the vehicle, based on one or a set of a plurality of rules. 20. The method of claim 19,
The data processing system, wherein the audio signal includes an ultrasonic signal. In a computer program stored in a computer-readable storage medium,
A computer program stored in a storage medium to implement the method according to any one of claims 1 to 9 when the computer program is executed by a processor.

Images