JP2019505034A - Automatic prediction and altruistic response of vehicle lane interruptions - Google Patents

Abstract

Systems and methods are provided for detecting and responding to an interrupting vehicle and for navigating while taking into account altruistic behavior parameters. In one implementation, a vehicle interrupt detection and response system for a host vehicle system may include a data interface and at least one processing device. At least one processing device receives, via the data interface, a plurality of images from at least one image capture device associated with the host vehicle, and a second lane in which the host vehicle is traveling in the plurality of images. At least that the target vehicle will change from the first lane to the second lane based on identifying the image of the target vehicle traveling in a different first lane and analyzing the plurality of images Identifying one indicator; detecting whether at least one predetermined interrupt sensitivity changing element is present in the host vehicle environment; and if no predetermined interrupt sensitivity changing element is detected, at least one Based on the identification of the indicator and based on the value associated with the first interrupt sensitivity parameter, Generating a first navigation response, and if at least one predetermined interrupt sensitivity changing element is detected, based on the identification of at least one indicator and based on a value associated with the second interrupt sensitivity parameter And generating a second navigation response at the host vehicle, wherein the second interrupt sensitivity parameter is different from the first interrupt sensitivity parameter.

Description

This application is related to US Provisional Patent Application No. 62 / 260,281 filed on November 26, 2015 and US Provisional Patent Application No. 62 / 361,343 filed on July 12, 2016. Claims the interest of the priority of the issue. All of the above applications are incorporated herein by reference in their entirety.

The present disclosure relates generally to autonomous vehicle navigation. The present disclosure further relates to a system and method for detecting and responding to an interrupted vehicle and navigating while taking into account altruistic operating parameters.

Background Information As technology continues to evolve, the goal of fully autonomous vehicles that can be navigated on the road is becoming reality. Autonomous vehicles may need to take into account various factors, and based on those factors, can make appropriate decisions to reach the intended destination safely and accurately. For example, autonomous vehicles may need to process and interpret visual information (eg, information captured from a camera) and from other sources (eg, global positioning system (GPS) devices, speed sensors, accelerations) Information obtained from a meter, suspension sensor, etc.) may be used. At the same time, in order to navigate to the destination, the autonomous vehicle identifies its position in a particular road (eg, a particular lane in a multi-lane road), navigates alongside other vehicles, and It may also be necessary to avoid pedestrians, observe traffic signals and signs, and drive from one road to another at appropriate intersections or interchanges.

  During navigation, the autonomous vehicle may encounter other vehicles attempting lane shifts. For example, a vehicle in the lane on the left or right of the lane in which the autonomous vehicle is traveling may change to the lane in which the autonomous vehicle is traveling, that is, try to interrupt. When such an interruption occurs, the autonomous vehicle may cause a navigation response, for example by changing its speed or acceleration and / or changing to another lane in order to avoid interruption by the other vehicle. There must be.

  In some instances, the other vehicle may appear to be interrupting, but may not eventually be interrupted (for example, the driver of the other vehicle has changed his mind, Because one vehicle is just sliding). Although delaying the execution of the navigation response can prevent unnecessary braking until there is sufficient chance that the other vehicle will be interrupted, such a delay also increases the risk of collision, and This may cause a brake that may cause the ride comfort of the occupant in the autonomous vehicle to deteriorate. Accordingly, there is a need to improve the prediction of vehicle interruption attempts.

  Further, in some cases, interruptions by other vehicles may be required, for example, due to road and / or traffic rules. However, in other cases, the interrupt is optionally obtained, for example, when another vehicle simply wants to overtake a slow vehicle. Because autonomous vehicles can be programmed to travel safely without delay to their destination, autonomous vehicles may not necessarily allow an interrupt by another vehicle if the interrupt is not required. However, in some cases it may be preferable to allow such interruptions for an autonomous vehicle operator and / or for the overall efficiency of traffic. Therefore, an interrupt process that includes altruistic behavior is necessary.

SUMMARY Embodiments according to the present disclosure provide a system and method for autonomous vehicle navigation. The disclosed embodiments may use a camera to provide autonomous vehicle navigation features. For example, according to embodiments of the present disclosure, the disclosed system may include one, two, or three or more cameras that monitor the environment of the vehicle. The disclosed system may provide navigation responses based on, for example, analysis of images captured by one or more of the cameras. The navigation response may also consider other data including, for example, global positioning (GPS) data, sensor data (eg, from accelerometers, speed sensors, suspension sensors, etc.), and / or other map data.

  According to the disclosed embodiments, a vehicle interrupt detection and response system for a host vehicle is provided. The system can include a data interface and at least one processing device. At least one processing device receives, via the data interface, a plurality of images from at least one image capture device associated with the host vehicle, and a second lane in which the host vehicle is traveling in the plurality of images. At least that the target vehicle will change from the first lane to the second lane based on identifying the image of the target vehicle traveling in a different first lane and analyzing the plurality of images Identifying one indicator; detecting whether at least one predetermined interrupt sensitivity changing element is present in the host vehicle environment; and if no predetermined interrupt sensitivity changing element is detected, at least one Based on the identification of the indicator and based on the value associated with the first interrupt sensitivity parameter, Generating a first navigation response, and if at least one predetermined interrupt sensitivity changing element is detected, based on the identification of at least one indicator and based on a value associated with the second interrupt sensitivity parameter And generating a second navigation response at the host vehicle, wherein the second interrupt sensitivity parameter is different from the first interrupt sensitivity parameter.

  According to other disclosed embodiments, the host vehicle may include a vehicle body, at least one image capture device, and at least one processing device. The at least one processing device receives a plurality of images from at least one image capture device associated with the host vehicle and a first lane that is different from the second lane in which the host vehicle is traveling in the plurality of images. Identifying at least one indicator that the target vehicle will change from the first lane to the second lane based on the analysis of the plurality of images and identifying an image of the target vehicle traveling on the road And detecting whether at least one predetermined interrupt sensitivity changing element is present in the environment of the host vehicle and, if no predetermined interrupt sensitivity changing element is detected, based on the identification of at least one indicator And a first navigation response in the host vehicle based on a value associated with the first interrupt sensitivity parameter And when the at least one predetermined interrupt sensitivity change factor is detected, the second in the host vehicle based on the identification of the at least one indicator and based on the value associated with the second interrupt sensitivity parameter. And the second interrupt sensitivity parameter is different from the first interrupt sensitivity parameter.

  According to yet another disclosed embodiment, a method for detecting and responding to an interrupt by a target vehicle is provided. The method receives a plurality of images from at least one image capture device associated with the host vehicle and travels in a first lane different from the second lane in which the host vehicle is traveling in the plurality of images. Identifying at least one indicator that the target vehicle will change from the first lane to the second lane based on the analysis of the plurality of images; Detecting whether at least one predetermined interrupt sensitivity changing element is present in the environment of the host vehicle, and if no predetermined interrupt sensitivity changing element is detected, based on the identification of the at least one indicator and Generating a first navigation response in the host vehicle based on a value associated with one interrupt sensitivity parameter; If at least one predetermined interrupt sensitivity change factor is detected, a second navigation response in the host vehicle is determined based on the identification of the at least one indicator and based on a value associated with the second interrupt sensitivity parameter. And the second interrupt sensitivity parameter is different from the first interrupt sensitivity parameter.

  According to the disclosed embodiments, a navigation system for a host vehicle is provided. The system can include a data interface and at least one processing device. At least one processing device receives at least one image from at least one image capture device associated with the host vehicle via the data interface and based on analysis of the plurality of images, at least one in the environment of the host vehicle. Identifying one target vehicle, identifying one or more situational features associated with the target vehicle based on analysis of multiple images, and identifying a current value associated with the altruistic behavior parameter And, based on one or more situational features associated with the target vehicle, that it is not necessary to change the navigation state of the host vehicle, but based on the current value associated with the altruistic behavior parameter , And one or more situational features associated with the target vehicle, Both may be programmed to perform a to cause one navigation change.

  According to other disclosed embodiments, the host vehicle may include a body, at least one image capture device, and at least one processing device. At least one processing device receives the plurality of images from the at least one image capture device, identifies at least one target vehicle in the environment of the host vehicle based on the analysis of the plurality of images; Identifying one or more contextual characteristics associated with the target vehicle, identifying a current value associated with the altruistic behavior parameter, and one associated with the target vehicle Or based on a plurality of situational features, identifying that no change in the navigation state of the host vehicle is necessary, but based on the current value associated with the altruistic behavior parameter and / or associated with the target vehicle Causing at least one navigational change in the host vehicle based on the plurality of situational features It may be configured sea urchin.

  According to yet another disclosed embodiment, a method for navigating a host vehicle is provided. The method receives a plurality of images from at least one image capture device associated with the vehicle, and identifies at least one target vehicle in the environment of the host vehicle based on the analysis of the plurality of images; Based on the analysis of the plurality of images, identifying one or more situational features associated with the target vehicle, identifying a current value associated with the altruistic behavior parameter, and associated with the target vehicle 1 One that identifies that no change in the navigation state of the host vehicle is necessary based on one or more situational features, but that is based on the current value associated with the altruistic behavior parameter and one associated with the target vehicle Or causing at least one navigational change in the host vehicle based on a plurality of situational features. .

  According to other disclosed embodiments, a non-transitory computer readable storage medium may store program instructions that are executed by at least one processing device and that perform any of the methods described herein.

  The foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the claims.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate various disclosed embodiments.

2 is a diagrammatic representation of an exemplary system according to disclosed embodiments. 1 is a side view representation of an exemplary vehicle including a system according to disclosed embodiments. 2B is a top view representation of the vehicle and system shown in FIG. 2A according to disclosed embodiments. FIG. 5 is a top view representation of another embodiment of a vehicle including a system according to disclosed embodiments. FIG. 6 is a top view representation of yet another embodiment of a vehicle including a system according to disclosed embodiments. FIG. 6 is a top view representation of yet another embodiment of a vehicle including a system according to disclosed embodiments. 2 is a diagrammatic representation of an exemplary vehicle control system according to disclosed embodiments. 1 is a diagrammatic representation of an interior of a vehicle including a rearview mirror and a user interface of a vehicle imaging system according to disclosed embodiments. FIG. 6 is an illustration of an example camera mount configured to be positioned behind a rearview mirror and opposite a vehicle windshield, according to disclosed embodiments. 3C is a view of the camera mount shown in FIG. 3B from a different perspective, according to disclosed embodiments. FIG. FIG. 6 is an illustration of an example camera mount configured to be positioned behind a rearview mirror and opposite a vehicle windshield, according to disclosed embodiments. FIG. 4 is an exemplary block diagram of a memory configured to store instructions that perform one or more operations according to disclosed embodiments. 6 is a flowchart illustrating an exemplary process for generating one or more navigation responses based on monocular image analysis, according to disclosed embodiments. 6 is a flowchart illustrating an exemplary process for detecting one or more vehicles and / or pedestrians in a set of images, according to disclosed embodiments. 6 is a flowchart illustrating an exemplary process for detecting road mark and / or lane geometry information in a set of images according to disclosed embodiments. 6 is a flowchart illustrating an exemplary process for detecting traffic lights in a set of images according to disclosed embodiments. 4 is a flowchart of an exemplary process for generating one or more navigation responses based on a vehicle route according to disclosed embodiments. 6 is a flowchart illustrating an exemplary process for identifying whether a preceding vehicle is changing lanes, according to disclosed embodiments. 6 is a flowchart illustrating an example process for generating one or more navigation responses based on stereoscopic image analysis, according to disclosed embodiments. 6 is a flowchart illustrating an exemplary process for generating one or more navigation responses based on analysis of three sets of images, according to disclosed embodiments. FIG. 4 is an exemplary block diagram of a memory configured to store instructions for performing one or more operations according to disclosed embodiments. FIG. 6 is an example situation in which a vehicle may detect and respond to an interrupt according to disclosed embodiments. Fig. 4 illustrates an exemplary predetermined interrupt sensitivity modification element according to disclosed embodiments. Fig. 4 illustrates an exemplary predetermined interrupt sensitivity modification element according to disclosed embodiments. Fig. 4 illustrates an exemplary predetermined interrupt sensitivity modification element according to disclosed embodiments. Fig. 4 illustrates an exemplary predetermined interrupt sensitivity modification element according to disclosed embodiments. FIG. 6 is an illustration of an exemplary situation in which a vehicle may engage in altruistic behavior, according to disclosed embodiments. 6 is a flowchart illustrating an exemplary process for vehicle interrupt detection and response according to disclosed embodiments. 7 is a flowchart illustrating an exemplary process 1200 for navigating while taking into account altruistic behavior, in accordance with disclosed embodiments.

DETAILED DESCRIPTION The following detailed description refers to the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the following description to refer to the same or like parts. Although some exemplary embodiments are described herein, modifications, adaptations, and other implementations are possible. For example, substitutions, additions, or modifications to the components shown in the drawings may be made, and the exemplary methods described herein may replace, reorder, delete, or replace the steps of the disclosed methods. It can be changed by adding. Accordingly, the following detailed description is not limited to the disclosed embodiments and examples. Instead, the appropriate scope is defined by the appended claims.

Autonomous Vehicle Overview As used throughout this disclosure, the term “autonomous vehicle” refers to a vehicle capable of performing at least one navigational change without driver input. “Navigation change” refers to one or more changes in vehicle steering, braking, or acceleration. To be autonomous, the vehicle does not need to be fully automatic (eg, full operation without a driver or driver input). Rather, autonomous vehicles include vehicles that operate under driver control during certain time periods and can operate without driver control during other time periods. An autonomous vehicle controls only some aspects of vehicle navigation, such as steering (eg, to maintain a vehicle course between vehicle lane constraints), but can leave other aspects (eg, brakes) to the driver. Can also be included. In some cases, an autonomous vehicle may handle some or all aspects of vehicle braking, speed control, and / or steering.

  Since human drivers typically rely on visual cues and observations to control the vehicle, the traffic infrastructure is built accordingly, and lane marks, traffic signs, and traffic lights all provide visual information to the driver. Designed to provide to. In view of these traffic infrastructure design features, an autonomous vehicle may include a camera and a processing unit that analyzes visual information captured from the environment of the vehicle. Visual information can include, for example, traffic infrastructure components (eg, lane marks, traffic signs, traffic lights, etc.) and other obstacles (eg, other vehicles, pedestrians, rubble, etc.) that are observable by the driver. . In addition, the motor vehicle may use stored information such as information that provides a model of the vehicle's environment when navigating. For example, the vehicle may use GPS data, sensor data (eg, from accelerometers, speed sensors, suspension sensors, etc.), and / or other map data to Relevant information can be provided, and the vehicle (and other vehicles) can use the information to determine its own location in the model.

System Overview FIG. 1 is a block diagram representation of a system 100 in accordance with the disclosed exemplary embodiments. System 100 may include various components depending on specific implementation requirements. In some embodiments, the system 100 includes a processing unit 110, an image acquisition unit 120, a position sensor 130, one or more memory units 140, 150, a map database 160, a user interface 170, and a wireless transceiver 172. obtain. The processing unit 110 may include one or more processing devices. In some embodiments, the processing unit 110 may include an application processor 180, an image processor 190, or any other suitable processing device. Similarly, the image acquisition unit 120 may include any number of image acquisition devices and components, depending on the specific application requirements. In some embodiments, the image acquisition unit 120 may include one or more image capture devices (eg, cameras) such as the image capture device 122, the image capture device 124, and the image capture device 126. The system 100 may also include a data interface 128 that communicatively connects the processing unit 110 to the image acquisition unit 120. For example, the data interface 128 may include any one or more wired and / or wireless links that transmit image data acquired by the image acquisition unit 120 to the processing unit 110.

  The wireless transceiver 172 is configured to exchange transmission signals with one or more networks (eg, cellular, internet, etc.) via a wireless interface through the use of radio frequency, infrared frequency, magnetic field, or electric field 1 One or more devices may be included. The wireless transceiver 172 may transmit and / or receive data using any known standard (eg, Wi-Fi®, Bluetooth®, Bluetooth Smart, 802.15.4). , ZigBee (registered trademark), etc.).

  Both application processor 180 and image processor 190 may include various types of processing devices. For example, one or both of the application processor 180 and the image processor 190 are a microprocessor, a preprocessor (such as an image preprocessor), a graphics processor, a central processing unit (CPU), a support circuit, a digital signal processor, an integrated circuit, and a memory. Or any other type of device suitable for running applications and processing and analyzing images. In some embodiments, application processor 180 and / or image processor 190 may include any type of single-core or multi-core processor, mobile device microcontroller, central processing unit, and the like. For example, various processing devices can be used, including processors available from manufacturers such as Intel (registered trademark), AMD (registered trademark), etc., and various architectures (eg, x86 processor, ARM (registered trademark), etc.) ).

  In some embodiments, the application processor 180 and / or the image processor 190 may include any EyeQ series processor available from Mobileye®. These processor designs include multiple processing units each having a local memory and an instruction set. Such a processor may include a video input that receives image data from multiple image sensors and may also include a video output function. In one example, EyeQ2® uses 90 nm-micron technology operating at 332 MHz. The EyeQ2® architecture consists of two floating point hyperthreaded 32-bit RISC CPUs (MIPS32® 34K® core), five vision calculation engines (VCE), three vector microcode processors (VMP ( (Registered trademark)), Denali 64-bit mobile DDR controller, 128-bit internal audio interconnect, dual 16-bit video input and 18-bit video output controller, 16-channel DMA, and several peripherals. The MIPS 34K CPU manages five VCEs, three VMP ™ and DMA, a second MIPS 34K CPU and multi-channel DMA, and other peripheral devices. Five VCEs, three VMPs, and MIPS 34K CPUs can perform the intensive vision calculations required by the multi-function bundle application. In another example, in the disclosed embodiment, a third generation processor, EyeQ3®, which is 6 times more powerful than EyeQ2® may be used.

  Any processing device disclosed herein may be configured to perform a particular function. Configuring a processing device, such as any of the described EyeQ processors or other controllers or microprocessors, to perform a specific function programs computer-executable instructions and executes them during operation of the processing device Providing the instructions to the processing device. In some embodiments, configuring the processing device may include programming architectural instructions directly on the processing device. In other embodiments, configuring the processing device may include storing executable instructions in a memory accessible to the processing device during operation. For example, the processing device may access the memory to obtain and execute stored instructions during operation.

  Although FIG. 1 shows two separate processing devices included in the processing unit 110, more or fewer processing devices may be used. For example, in some embodiments, a single processing device may be used to accomplish the tasks of application processor 180 and image processor 190. In other embodiments, these tasks may be performed by more than two processing devices. Further, in some embodiments, the system 100 may include one or more processing units 110 without including other components such as the image acquisition unit 120.

  The processing unit 110 can include various types of devices. For example, the processing unit 110 may be a variety of such as a controller, image preprocessor, central processing unit (CPU), support circuitry, digital signal processor, integrated circuit, memory, or any other type of device that processes and analyzes images. Devices can be included. The image preprocessor may include a video processor that captures, digitizes, and processes images from the image sensor. The CPU may include any number of microcontrollers or microprocessors. The support circuitry can be any number of circuits generally known in the art, including caches, power supplies, clocks, and input / output circuits. The memory may store software that, when executed by the processor, controls the operation of the system. The memory may include a database and image processing software. The memory may include any number of random access memory, read only memory, flash memory, disk drive, optical storage device, tape storage device, removable storage device, and other types of storage devices. In one example, the memory can be separate from the processing unit 110. In another example, the memory may be integrated into the processing unit 110.

  Each memory 140, 150 may include software instructions that, when executed by a processor (eg, application processor 180 and / or image processor 190), may control the operation of various aspects of system 100. These memory units may include various databases and image processing software. The memory unit may include random access memory, read only memory, flash memory, disk drive, optical storage device, tape storage device, removable storage device, and / or any other type of storage device. In some embodiments, the memory units 140, 150 may be separate from the application processor 180 and / or the image processor 190. In other embodiments, these memory units may be integrated into application processor 180 and / or image processor 190.

  The position sensor 130 may include any type of device suitable for determining a position associated with at least one component of the system 100. In some embodiments, the position sensor 130 may include a GPS receiver. Such a receiver can determine the user's position and velocity by processing signals broadcast by global positioning system satellites. Position information from position sensor 130 may be provided to application processor 180 and / or image processor 190.

  In some embodiments, the system 100 may include components such as a speed sensor (eg, a tachometer) that measures the speed of the vehicle 200 and / or an accelerometer that measures the acceleration of the vehicle 200.

  User interface 170 may include any device suitable for providing information or receiving input from one or more users of system 100. In some embodiments, user interface 170 may include user input devices including, for example, touch screens, microphones, keyboards, pointer devices, track wheels, cameras, knobs, buttons, and the like. With such an input device, the user can type commands or information, provide voice commands, use buttons, pointers, or eye tracking functions, or any other that communicates information to the system 100. It may be possible to provide information input or commands to the system 100 by selecting menu choices on the screen through suitable techniques.

  The user interface 170 comprises one or more processing devices configured to provide information to the user or receive information from the user and process the information for use by, for example, the application processor 180. obtain. In some embodiments, such a processing device may recognize instructions for recognizing and tracking eye movements, instructions for receiving and interpreting voice commands, touches and / or gestures made on a touch screen. Commands to interpret, commands to respond to keyboard input or menu selections, etc. may be executed. In some embodiments, the user interface 170 may include a display, speakers, haptic devices, and / or any other device that provides output information to the user.

  The map database 160 may include any type of database that stores map data useful for the system 100. In some embodiments, the map database 160 stores data related to the location of various items in a reference coordinate system, including roads, water features, geographic features, businesses, points of interest, restaurants, gas stations, etc. May be included. The map database 160 may store not only the location of such items, but also descriptors associated with those items, including, for example, names associated with any of the stored features. In some embodiments, the map database 160 may be physically located with other components of the system 100. Alternatively or additionally, the map database 160 or a portion thereof may be remotely located with respect to other components of the system 100 (eg, the processing unit 110). In such an embodiment, information from the map database 160 may be downloaded to the network via a wired or wireless data connection (eg, via a cellular network and / or the Internet, etc.).

  Image capture devices 122, 124, and 126 may each include any type of device suitable for capturing at least one image from the environment. In addition, any number of image capture devices may be used to acquire an image for input to the image processor. Some embodiments may include only a single image capture device, while other embodiments may include two, three, and even four or more image capture devices. Image capture devices 122, 124, and 126 are described further below with reference to FIGS. 2B-2E.

  The system 100 or various components of the system 100 can be incorporated into a variety of different platforms. In some embodiments, the system 100 may be included in a vehicle 200 as shown in FIG. 2A. For example, the vehicle 200 may include the processing unit 110 and any other components of the system 100 as described above with respect to FIG. In some embodiments, the vehicle 200 may include only a single image capture device (e.g., a camera), while in other embodiments, such as the embodiments discussed in connection with FIGS. Image capture devices can be used. For example, as shown in FIG. 2A, any of the image capture devices 122 and 124 of the vehicle 200 may be part of an ADAS (Advanced Driver Assistance System) imaging set.

  The image capture device included in the vehicle 200 as part of the image acquisition unit 120 may be located in any suitable location. In some embodiments, as shown in FIGS. 2A-2E and 3A-3C, the image capture device 122 may be located in the vicinity of the rearview mirror. This position may provide a line of sight similar to the driver of the vehicle 200 and may assist in determining what the driver sees and cannot see. Image capture device 122 may be located anywhere near the rearview mirror, but positioning image capture device 122 on the driver side of the mirror further assists in obtaining an image representing the driver's field of view and / or line of sight. Can do.

  Other locations may be used for the image capture device of the image acquisition unit 120. For example, the image capture device 124 may be located on or in the bumper of the vehicle 200. Such a position may be particularly suitable for image capture devices having a wide field of view. The line of sight of the image capture device placed on the bumper can be different from the line of sight of the driver, so the bumper image capture device and the driver are not always looking at the same object. Image capture devices (eg, image capture devices 122, 124, and 126) can also be located at other locations. For example, the image capture device may be located on one or both of the side mirrors of the vehicle 200, the roof of the vehicle 200, the hood of the vehicle 200, the trunk of the vehicle 200, the side of the vehicle 200, and mounted on any window of the vehicle 200. It can be positioned behind, or positioned in front, and mounted on the front and / or rear lights of the vehicle 200 or in the vicinity thereof.

  In addition to the image capture device, the vehicle 200 may include various other components of the system 100. For example, the processing unit 110 may be integrated into the vehicle engine control unit (ECU) or included in the vehicle 200 separately from the ECU. The vehicle 200 may include a position sensor 130 such as a GPS receiver, and the vehicle 200 may include a map database 160 and memory units 140 and 150.

  As described above, the wireless transceiver 172 may receive and / or receive data via one or more networks (eg, a cellular network, the Internet, etc.). For example, the wireless transceiver 172 may upload data collected by the system 100 to one or more servers and download the data from one or more servers. Via the wireless transceiver 172, the system 100 may receive updates to data stored in the map database 160, memory 140, and / or memory 150, for example, periodically or on demand. Similarly, the wireless transceiver 172 may include any data from the system 100 (eg, images captured by the image acquisition unit 120, position sensors 130, other sensors, or data received by the vehicle control system, etc.) and / or Or any data processed by the processing unit 110 may be uploaded to one or more servers.

  The system 100 may upload data to a server (eg, cloud) based on the privacy level setting. For example, the system 100 may implement a privacy level setting that regulates or restricts the types of data (including metadata) that may be uniquely identified to the vehicle and / or driver / owner of the vehicle that is transmitted to the server. Such settings may be set, for example, by the user via the wireless transceiver 172, initialized by factory default settings, or set by data received by the wireless transceiver 172. .

  In some embodiments, the system 100 may upload data according to a “high” privacy level, and under certain settings, the system 100 may provide data without any details about a particular vehicle and / or driver / owner ( For example, position information related to the route, captured images, etc.) may be transmitted. For example, when uploading data according to a “high” privacy level, the system 100 does not include a vehicle identification number (VIN) or the name of the driver or owner of the vehicle, but instead is limited in relation to captured images and / or routes. The received position information and other data can be transmitted.

  Other privacy levels are also contemplated. For example, the system 100 may send data to a server according to a “medium” privacy level, such as “high” privacy, such as vehicle and / or vehicle type manufacturer and / or model (eg, passenger car, sport utility vehicle, truck, etc.). It may contain additional information not included below the level. In some embodiments, the system 100 may upload data according to a “low” privacy level. Under the “low” privacy level setting, the system 100 uploads enough data to uniquely identify a particular vehicle, owner / driver, and / or part or all of the route the vehicle has traveled, Such information may be included. Such “low” privacy level data may include, for example, VIN, driver / owner name, vehicle departure point prior to departure, vehicle intended destination, vehicle manufacturer and / or model, vehicle type, etc. One or more may be included.

  FIG. 2A is a side view representation of an exemplary vehicle imaging system in accordance with the disclosed embodiments. FIG. 2B is a top view representation of the embodiment shown in FIG. 2A. As shown in FIG. 2B, the disclosed embodiments include a first image capture device 122 positioned in the vicinity of the rearview mirror and / or in the vicinity of the driver of the vehicle 200, and a bumper area of the vehicle 200 (eg, a bumper area). 210) may include a system 100 having a second image capture device 124 positioned on or within a bumper region and a processing unit 110 within the body.

  As shown in FIG. 2C, both image capture devices 122 and 124 may be positioned near the rearview mirror of the vehicle 200 and / or near the driver. Further, although two image capture devices 122 and 124 are shown in FIGS. 2B and 2C, it should be understood that other embodiments may include more than two image capture devices. For example, in the embodiment shown in FIGS. 2D and 2E, a first image capture device 122, a second image capture device 124, and a third image capture device 126 are included in the system 100 of the vehicle 200.

  As shown in FIG. 2D, the image capture device 122 may be positioned near the rearview mirror of the vehicle 200 and / or near the driver, and the image capture devices 124 and 126 One of 210). Also, as shown in FIG. 2E, the image capture devices 122, 124, and 126 may be positioned near the rearview mirror of the vehicle 200 and / or near the driver seat. The disclosed embodiments are not limited to any particular number and configuration of image capture devices, which may be positioned in any suitable location within and / or on the vehicle 200.

  It should be understood that the disclosed embodiments are not limited to vehicles and are applicable in other situations. It should also be understood that the disclosed embodiments are not limited to a particular type of vehicle 200 and may be applicable to all types of vehicles, including automobiles, trucks, trailers, and other types of vehicles.

  The first image capture device 122 may include any suitable type of image capture device. Image capture device 122 may include an optical axis. In one example, the image capture device 122 may include an Aptina M9V024 WVGA sensor with a global shutter. In other embodiments, the image capture device 122 may provide a resolution of 1280 × 960 pixels and may include a rolling shutter. Image capture device 122 may include various optical elements. In some embodiments, one or more lenses may be included to provide the desired focal length and field of view of the image capture device, for example. In some embodiments, a 6 mm lens or a 12 mm lens may be associated with the image capture device 122. In some embodiments, the image capture device 122 may be configured to capture an image having a desired field of view (FOV) 202, as shown in FIG. 2D. For example, the image capture device 122 may be configured to have a normal FOV, such as within a range of 40 degrees to 56 degrees, including frequencies greater than 46 degrees FOV, 50 degrees FOV, 52 degrees FOV, or 52 degrees FOV. . Alternatively, the image capture device 122 may be configured to have a narrow FOV in the range of 23-40 degrees, such as a 28 degree FOV or a 36 degree FOV. In addition, the image capture device 122 may be configured to have a wide FOV in the range of 100-180 degrees. In some embodiments, the image capture device 122 may include a wide angle bumper camera or a bumper camera having a FOV up to 180 degrees. In some embodiments, the image capture device 122 may be a 7.2 M pixel image capture device with an aspect ratio of about 2: 1 (eg, H × V = 3800 × 1900 pixels) with a horizontal FOV of about 100 degrees. . Such an image capture device may be used instead of a three-dimensional image capture device configuration. Due to the large lens distortion, the vertical FOV of such an image capture device can be much lower than 50 degrees in implementations where the image capture device uses a radially symmetric lens. For example, such a lens is not radially symmetric, thereby allowing a vertical FOV greater than 50 degrees with a horizontal FOV of 100 degrees.

  The first image capture device 122 may obtain a plurality of first images for a scene associated with the vehicle 200. The plurality of first images can each be acquired as a series of image scan lines, which can be captured using a rolling shutter. Each scan line may include a plurality of pixels.

  The first image capture device 122 may have a scan rate associated with each acquisition of the first series of image scan lines. A scan rate may refer to a rate at which the image sensor can acquire image data associated with each pixel included in a particular scan line.

  Image capture devices 122, 124, and 126 may include any suitable type and number of image sensors, including, for example, CCD sensors or CMOS sensors. In one embodiment, a CMOS image sensor may be utilized with a rolling shutter so that each pixel in the row is read one at a time and the row scan is advanced row by row until the entire image frame is captured. . In some embodiments, rows may be captured sequentially from top to bottom with respect to the frame.

  In some embodiments, one or more of the image capture devices disclosed herein (eg, image capture devices 122, 124, and 126) may constitute a high-resolution imager, greater than 5M pixels, 7M It may have a resolution greater than 10 pixels, or greater.

  By using a rolling shutter, pixels in different rows can be exposed and captured at different times, which can cause skew and other image artifacts in the captured image frame. On the other hand, if the image capture device 122 is configured to operate with a global or synchronous shutter, all pixels may be exposed during a common exposure period for the same amount of time. As a result, the image data in the frame collected from the system using the global shutter represents a snapshot of the entire FOV (such as FOV 202) at a specific time. Conversely, when applying a rolling shutter, each row in the frame is exposed and data is captured at different times. Thus, a moving object may appear distorted on an image capture device having a rolling shutter. This phenomenon will be described in more detail below.

  The second image capture device 124 and the third image capture device 126 may be any type of image capture device. Like the first image capture device 122, each of the image capture devices 124 and 126 may include an optical axis. In one embodiment, each of the image capture devices 124 and 126 may include an Aptina M9V024 WVGA sensor with a global shutter. Alternatively, each of the image capture devices 124 and 126 may include a rolling shutter. Like image capture device 122, image capture devices 124 and 126 may be configured to include various lenses and optical elements. In some embodiments, the lenses associated with the image capture devices 124 and 126 are the same as the FOV (such as FOV 202) associated with the image capture device 122 or provide a narrow FOV (such as FOV 204 and 206). Can do. For example, the image capture devices 124 and 126 may have a FOV of 40 degrees, 30 degrees, 26 degrees, 23 degrees, 20 degrees, or less than 20 degrees.

  Image capture devices 124 and 126 may obtain a plurality of second and third images for a scene associated with vehicle 200. Each of the plurality of second and third images may be acquired as a second and third series of image scan lines, which may be captured using a rolling shutter. Each scan line or row may have multiple pixels. Image capture devices 124 and 126 may have second and third scan rates associated with acquiring a second series and a third series of image scan lines.

  Each image capture device 122, 124, and 126 may be positioned in any suitable position and in any suitable orientation with respect to the vehicle 200. The relative positions of the image capture devices 122, 124, and 126 may be selected to assist in fusing information obtained from the image capture devices together. For example, in some embodiments, the FOV associated with the image capture device 124 (FOV 204) is the FOV associated with the image capture device 122 (such as FOV 202) and the FOV associated with the image capture device 126 (such as FOV 206). And may partially or completely overlap.

Image capture devices 122, 124, and 126 may be placed on vehicle 200 at any suitable relative height. In one example, there may be a height difference between the image capture devices 122, 124, and 126, and the height difference may provide sufficient disparity information to allow stereo analysis. For example, as shown in FIG. 2A, the two image capture devices 122 and 124 are at different heights. There may also be lateral displacement differences between the image capture devices 122, 124, and 126, for example providing additional disparity information for stereo analysis by the processing unit 110. The lateral displacement difference may be indicated by d _x as shown in FIGS. 2C and 2D. In some embodiments, a front displacement or a rear displacement (eg, range displacement) can exist between the image capture devices 122, 124, 126. For example, the image capture device 122 may be positioned 0.5-2 meters or more behind the image capture device 124 and / or the image capture device 126. With this type of displacement, one of the image capture devices may be able to cover the potential blind spots of other image capture devices.

  The image capture device 122 may have any suitable resolution capability (eg, the number of pixels associated with the image sensor) and the resolution of the image sensor associated with the image capture device 122 is associated with the image capture devices 124 and 126. The resolution may be higher, lower or the same as the resolution of the image sensor provided. In some embodiments, the image sensor associated with the image capture device 122 and / or the image capture devices 124 and 126 has a resolution of 640 × 480, 1024 × 768, 1280 × 960, or any other suitable resolution. Can do.

  The frame rate (eg, the rate at which the image capture device acquires a set of pixel data for one image frame before moving on to capture pixel data associated with the next image frame) may be controllable. The frame rate associated with the image capture device 122 may be higher, lower, or the same as the frame rate associated with the image capture devices 124 and 126. The frame rate associated with the image capture devices 122, 124, and 126 may depend on various factors that can affect the timing of the frame rate. For example, one or more of the image capture devices 122, 124, and 126 may be prior to acquisition of image data associated with one or more pixels of the image sensor in the image capture devices 122, 124, and / or 126, or It may include a selectable pixel delay period imposed after acquisition. In general, the image data corresponding to each pixel may be acquired according to the clock rate of the device (eg, one pixel per clock cycle). Further, in embodiments that include a rolling shutter, one or more of the image capture devices 122, 124, and 126 may be associated with image data associated with pixel rows of image sensors within the image capture devices 122, 124, and / or 126. May include a selectable horizontal blanking period imposed before or after acquisition. Further, one or more of the image capture devices 122, 124, and / or 126 are selectable imposed before or after acquisition of image data associated with the image frames of the image capture devices 122, 124, and 126. A vertical blank period may be included.

  These timing controls may allow the frame rates associated with the image capture devices 122, 124, and 126 to be synchronized even when the line scan rate of each image capture device is different. Further, as will be discussed in more detail below, among other factors (eg, image sensor resolution, maximum line scan rate, etc.), these selectable timing controls allow the field of view of image capture device 122 to be captured by image capture device 124. And 126 may be able to synchronize image capture from areas where the FOV of the image capture device 122 overlaps one or more FOVs of the image capture devices 124 and 126.

  Frame rate timing at the image capture devices 122, 124, and 126 may depend on the resolution of the associated image sensor. For example, assuming that both devices have similar line scan rates, if one device contains an image sensor with a resolution of 640 × 480 and the other device contains an image sensor with a resolution of 1280 × 960, the higher resolution The longer the time required to acquire a frame of image data from the sensor, the longer the time is required.

  Another factor that can affect the timing of image data acquisition at the image capture devices 122, 124, and 126 is the maximum line scan rate. For example, obtaining image data rows from image sensors included in image capture devices 122, 124, and 126 requires some minimum amount of time. Assuming no pixel delay period is added, this minimum amount of time to acquire an image data row will be related to the maximum line scan rate of a particular device. Devices that provide a high maximum line scan rate have the potential to provide a higher frame rate than devices that have a lower maximum line scan rate. In some embodiments, one or both of image capture devices 124 and 126 may have a maximum line scan rate that is higher than the maximum line scan rate associated with image capture device 122. In some embodiments, the maximum line scan rate of the image capture device 124 and / or 126 is 1.25 times, 1.5 times, 1.75 times, or 2 times the maximum line scan rate of the image capture device 122. That can be the case.

  In another embodiment, the image capture devices 122, 124, and 126 may have the same maximum line scan rate, but the image capture device 122 may operate at a scan rate below that maximum scan rate. The system may be configured such that one or both of the image capture devices 124 and 126 operate at a line scan rate that is equal to the line scan rate of the image capture device 122. In other examples, the system may have a line scan rate of the image capture device 124 and / or 126 that is 1.25 times, 1.5 times, 1.75 times, or more than twice the line scan rate of the image capture device 122. Can be configured to be.

  In some embodiments, the image capture devices 122, 124, and 126 may be asymmetric. That is, these image capture devices may include cameras with different fields of view (FOV) and focal lengths. The field of view of the image capture devices 122, 124, and 126 may include any desired area for the environment of the vehicle 200, for example. In some embodiments, one or more of the image capture devices 122, 124, and 126 may capture images from the environment in front of the vehicle 200, the environment behind the vehicle 200, the environment on both sides of the vehicle 200, or combinations thereof. It can be configured to obtain data.

  Further, the focal length associated with each image capture device 122, 124, and / or 126 may be selectable such that each device acquires an image of an object in a desired distance range from the vehicle 200 (eg, By including appropriate lenses, etc.). For example, in some embodiments, the image capture devices 122, 124, and 126 may acquire images of nearby objects within a few meters of the vehicle. The image capture devices 122, 124, 126 can also be configured to acquire images of objects at ranges further away from the vehicle (eg, 25m, 50m, 100m, 150m, or more). Further, the focal length of the image capture devices 122, 124, and 126 is such that an image capture device (eg, the image capture device 122) acquires an image of an object that is relatively close to the vehicle (eg, within 10m or within 20m). While other image capture devices (e.g., image capture devices 124 and 126) acquire images of objects further away from the vehicle 200 (e.g., greater than 20m, greater than 50m, greater than 100m, greater than 150m, etc.). You can choose to be able to.

  According to some embodiments, the FOV of one or more image capture devices 122, 124, and 126 may have a wide angle. For example, it may be advantageous to have a 140 degree FOV for the image capture devices 122, 124, and 126 that may be used, for example, to acquire images in the vicinity of the vehicle 200 in particular. For example, the image capture device 122 may be used to capture images in the right or left area of the vehicle 200, and in such embodiments, the image capture device 122 may have a wide FOV (eg, at least 140 degrees). Sometimes desirable.

  The field of view associated with each of the image capture devices 122, 124, and 126 may depend on each focal length. For example, as the focal length increases, the corresponding field of view decreases.

  Image capture devices 122, 124, and 126 may be configured to have any suitable field of view. In one particular example, image capture device 122 may have a horizontal FOV of 46 degrees, image capture device 124 may have a horizontal FOV of 23 degrees, and image capture device 126 may have a horizontal FOV of 23-46 degrees. In another example, image capture device 122 may have a horizontal FOV of 52 degrees, image capture device 124 may have a horizontal FOV of 26 degrees, and image capture device 126 may have a horizontal FOV of 26-52 degrees. In some embodiments, the ratio of the FOV of the image capture device 122 to the FOV of the image capture device 124 and / or the image capture device 126 can vary from 1.5 to 2.0. In other embodiments, this ratio can vary from 1.25 to 2.25.

  System 100 may be configured such that the field of view of image capture device 122 at least partially or completely overlaps the field of view of image capture device 124 and / or image capture device 126. In some embodiments, the system 100 includes the field of view of the image capture devices 124 and 126 within, for example, the field of view of the image capture device 122 (eg, smaller than the field of view of the image capture device 122). Can be configured to share a common center. In other embodiments, the image capture devices 122, 124, and 126 may capture adjacent FOVs or have partially overlapping FOVs. In some embodiments, the field of view of the image capture devices 122, 124, and 126 is positioned in the lower half of the field of view of the device 122 with the wider FOV, with the center of the image capture device 124 and / or 126 with the narrower FOV being located. Align as you get.

  FIG. 2F is a diagrammatic representation of an exemplary vehicle control system in accordance with the disclosed embodiments. As shown in FIG. 2F, the vehicle 200 may include a throttle system 220, a brake system 230, and a steering system 240. The system 100 may include one or more of a throttle system 220, a brake system 230, and a steering system 240 via one or more data links (eg, any wired and / or wireless link or link that transmits data). May provide an input (eg, a control signal). For example, based on an analysis of images acquired by the image capture devices 122, 124, and / or 126, the system 100 can provide control signals for navigating the vehicle 200 to the throttle system 220, the brake system 230, and the steering system 240. One or more may be provided (eg, by causing acceleration, turns, lane shifts, etc.). In addition, the system 100 provides input (eg, speed, whether the vehicle 200 is braking and / or turning, etc.) indicating the operating status of the vehicle 200 to the throttle system 220, the brake system 230, and the steering system 24. One or more may be received. In the following, further details will be provided in connection with FIGS.

  As shown in FIG. 3A, the vehicle 200 may also include a user interface 170 that interacts with a driver or occupant of the vehicle 200. For example, the user interface 170 in the vehicle application may include a touch screen 320, a knob 330, a button 340, and a microphone 350. The driver or occupant of the vehicle 200 uses a handle (for example, disposed on or near the steering column of the vehicle 200 including a winker handle) and a button (for example, disposed on the handle of the vehicle 200). It can also interact with the system 100. In some embodiments, the microphone 350 may be positioned adjacent to the rearview mirror 310. Similarly, in some embodiments, the image capture device 122 may be located near the rearview mirror 310. In some embodiments, the user interface 170 may also include one or more speakers 360 (eg, speakers of a vehicle audio system). For example, the system 100 may provide various notifications (eg, alerts) via the speaker 360.

  3B-3D are views of an exemplary camera mount 370 configured to be positioned behind a rearview mirror (eg, rearview mirror 310) and opposite a vehicle windscreen, according to disclosed embodiments. It is. As shown in FIG. 3B, the camera mount 370 may include image capture devices 122, 124, and 126. Image capture devices 124 and 126 may be positioned behind glare shield 380, which may be in direct contact with the windshield and may include a composition of film and / or antireflective material. For example, the glare shield 380 may be positioned to be aligned opposite a windshield having a matching slope. In some embodiments, each of the image capture devices 122, 124, and 126 may be positioned behind the glare shield 380, for example, as shown in FIG. 3D. The disclosed embodiments are not limited to any particular configuration of image capture devices 122, 124 and 126, camera mount 370, and glare shield 380. FIG. 3C is a view of the camera mount 370 shown in FIG. 3B as viewed from the front.

  Many variations and / or modifications to the above disclosed embodiments may be made, as will be appreciated by those skilled in the art having the benefit of this disclosure. For example, not all components are essential to the operation of system 100. Further, any component may be located in any suitable portion of system 100, and the components may be rearranged in various configurations while providing the functionality of the disclosed embodiments. Thus, the configuration described above is an example, and regardless of the configuration described above, the system 100 can provide a wide range of functions for analyzing the surroundings of the vehicle 200 and navigating the vehicle 200 in response to the analysis.

  As discussed in further detail below, according to various disclosed embodiments, system 100 may provide various features related to autonomous driving and / or driver assistance techniques. For example, the system 100 may analyze image data, location data (eg, GPS location information), map data, speed data, and / or data from sensors included in the vehicle 200. System 100 may collect data for analysis from, for example, image acquisition unit 120, position sensor 130, and other sensors. Further, the system 100 may analyze the collected data to determine whether the vehicle 200 should take a specific action and then automatically take the determined action without human intervention. For example, if the vehicle 200 navigates without human participation, the system 100 may automatically control the braking, acceleration, and / or steering of the vehicle 200 (eg, control signals to the throttle system 220, brake system 230, etc. , And by sending to one or more of the steering systems 240). Further, the system 100 may analyze the collected data and issue warnings and / or alerts to vehicle occupants based on the analysis of the collected data. Further details regarding various embodiments provided by the system 100 are provided below.

Forward-facing Multi-Imaging System As described above, the system 100 can provide a driving assistance function that uses a multi-camera system. A multi-camera system may use one or more cameras facing the front of the vehicle. In other embodiments, the multi-camera system may include one or more cameras facing the side of the vehicle or the rear of the vehicle. In one embodiment, for example, system 100 may use a two-camera imaging system, in which case the first camera and the second camera (eg, image capture devices 122 and 124) are vehicles (eg, vehicle 200). Can be positioned on the front and / or side of the. The first camera may have a field of view that is larger, smaller, or partially overlapping than the field of view of the second camera. Further, the first camera can be connected to a first image processor to perform monocular image analysis of the image provided by the first camera, and the second camera is connected to the second image processor. A monocular image analysis of the image provided by the second camera may be performed. The outputs (eg, processed information) of the first and second image processors may be combined. In some embodiments, the second image processor may receive images from both the first camera and the second camera to perform stereo analysis. In another embodiment, the system 100 may use a three camera imaging system, where each camera has a different field of view. Thus, such a system can make decisions based on information derived from objects at various distances both on the front and side of the vehicle. Reference to monocular image analysis may refer to the case where image analysis is performed based on images captured from a single viewpoint (eg, a single camera). Stereoscopic image analysis may refer to the case where image analysis is performed based on two or more images captured with one or more of the image capture parameters changed. For example, captured images suitable for performing stereoscopic image analysis include images captured from two or more different positions, images captured from different fields of view, images captured using different focal lengths, with disparity information It can include captured images and the like.

  For example, in one embodiment, the system 100 may implement a three camera configuration using image capture devices 122-126. In such a configuration, the image capture device 122 may provide a narrow field of view (eg, 34 degrees or other values selected from a range of about 20-45 degrees, etc.) and the image capture device 124 may have a wide field of view ( For example, 150 degrees or other values selected from a range of about 100 to about 180 degrees may be provided, and the image capture device 126 may be selected from a medium field of view (eg, 46 degrees or a range of about 35 to about 60 degrees) Other values to be provided). In some embodiments, the image capture device 126 may operate as a primary or primary camera. The image capture devices 122-126 can be positioned substantially side-by-side (eg, 6 cm apart) behind the rearview mirror 310. Further, in some embodiments, as described above, one or more of the image capture devices 122-126 may be mounted behind a glare shield 380 that is flush with the windshield of the vehicle 200. Such a shield may operate to minimize the effect of any reflections from the interior of the vehicle on the image capture devices 122-126.

  In another embodiment, as described above in connection with FIGS. 3B and 3C, a wide field camera (eg, image capture device 124 in the above example) is a narrow main field camera (eg, image capture device 122 in the above example). And 126). This configuration may provide free line of sight from the wide field camera. To reduce reflection, the camera may be mounted near the windshield of the vehicle 200 and the camera may include a polarizer that attenuates the reflected light.

  A three-camera system may provide specific performance characteristics. For example, some embodiments may include the ability to verify object detection by one camera based on detection results from another camera. In the three-camera configuration described above, the processing unit 110 may include, for example, three processing devices (eg, three EyeQ series processor chips as described above), each processing device being one of the image capture devices 122-126. Directed to processing images captured by one or more.

  In a three-camera system, the first processing device may receive images from both the main camera and the narrow field camera and perform the vision processing of the narrow FOV camera, eg, other vehicles, pedestrians, lane marks, Traffic signs, traffic lights, and other road objects may be detected. In addition, the first processing device may calculate a pixel mismatch between the image from the primary camera and the image from the narrow camera to create a 3D reconstruction of the environment of the vehicle 200. The first processing device may then combine the 3D reconstruction with 3D information calculated based on 3D map data or information from another camera.

  The second processing device may receive images from the primary camera and perform vision processing to detect other vehicles, pedestrians, lane marks, traffic signs, traffic lights, and other road objects. Further, the second processing device may calculate camera displacement, calculate pixel mismatches between successive images based on the displacement, and create a 3D reconstruction of the scene (eg, structure from motion). The second processing device may send a structure from motion based on the 3D reconstruction to the first processing device and combine the structure from motion with the stereoscopic 3D image.

  A third processing device may receive images from the wide FOV camera and process the images to detect vehicles, pedestrians, lane marks, traffic signs, traffic lights, and other road objects. The third processing device may further execute additional processing instructions to analyze the image and identify moving objects in the image, such as a lane changing vehicle, pedestrian, etc.

  In some embodiments, independently capturing and processing an image-based information stream may provide an opportunity to provide redundancy in the system. Such redundancy verifies information obtained by capturing and processing image information from at least a second image capture device using, for example, a first image capture device and an image processed from that device. And / or may be captured.

  In some embodiments, the system 100 may use two image capture devices (eg, image capture devices 122 and 124) in providing navigation assistance to the vehicle 200, and a third image capture device (eg, image). The capture device 126) may be used to provide redundancy and verify analysis of data received from the other two image capture devices. For example, in such a configuration, the image capture devices 122 and 124 may provide a stereoscopic analysis image by the system 100 for navigating the vehicle 200, while the image capture device 126 is for monocular analysis by the system 100. An image may be provided to provide redundancy and verification of information obtained based on images captured from the image capture device 123 and / or the image capture device 124. That is, the image capture device 126 (and corresponding processing device) provides a redundant subsystem that provides a check to the analysis derived from the image capture devices 122 and 124 (eg, provides an automatic emergency braking (AEB) system). Can be considered).

  Those skilled in the art will recognize that the camera configuration, camera placement, number of cameras, camera position, etc. are merely examples. These components and the like described for the overall system may be assembled and used in a variety of different configurations without departing from the scope of the disclosed embodiments. Further details regarding the use of the multi-camera system to provide driver assistance and / or autonomous vehicle functions follow.

  FIG. 4 is an exemplary functional block diagram of memories 140 and / or 150 that may store / program instructions for performing one or more operations in accordance with the disclosed embodiments. In the following, reference will be made to memory 140, but those skilled in the art will recognize that instructions can be stored in memory 140 and / or 150.

  As shown in FIG. 4, the memory 140 may store a monocular image analysis module 402, a stereoscopic image analysis module 404, a speed and acceleration module 406, and a navigation response module 408. The disclosed embodiments are not limited to any particular configuration of memory 140. Further, application processor 180 and / or image processor 190 may execute instructions stored in any module 402-408 included in memory 140. Those skilled in the art will appreciate that references to processing unit 110 in the following discussion may refer to application processor 180 and image processor 190 individually or collectively. Thus, any of the following process steps may be performed by one or more processing devices.

  In one embodiment, the monocular image analysis module 402 may store instructions (such as computer vision software) that are acquired by one of the image capture devices 122, 124, and 126 when executed by the processing unit 110. Perform a monocular image analysis of the set of images. In some embodiments, the processing unit 110 may combine information from the set of images with additional sensor information (eg, information from a radar) to perform monocular image analysis. As described in connection with FIGS. 5A-5D below, the monocular image analysis module 402 associates with lane marks, vehicles, pedestrians, road signs, highway exit lamps, traffic lights, dangerous goods, and vehicle environments. Instructions for detecting a set of features within the set of images, such as any other feature that is provided. Based on the analysis, the system 100 may generate one or more navigation responses in the vehicle 200, such as turns, lane shifts, and acceleration changes, as discussed below in connection with the navigation response module 408 (eg, , Via processing unit 110).

  In one embodiment, the stereoscopic image analysis module 404 may store instructions (such as computer vision software) that, when executed by the processing unit 110, select selected image captures from the image capture devices 122, 124, and 126. Perform stereo image analysis of the first and second sets of images acquired by the combination of devices. In some embodiments, the processing unit 110 may combine information from the first and second sets of images with additional sensor information (eg, information from a radar) to perform stereoscopic image analysis. For example, the stereoscopic image analysis module 404 may issue instructions to perform a stereoscopic image analysis based on a first set of images acquired by the image capture device 124 and a second set of images acquired by the image capture device 126. May be included. As will be described below in connection with FIG. 6, the stereoscopic image analysis module 404 includes first and second sets of lane marks, vehicles, pedestrians, road signs, highway exit lamps, traffic lights, and dangerous goods. Instructions for detecting a set of features in the image. Based on the analysis, the processing unit 110 may generate one or more navigation responses in the vehicle 200, such as turns, lane shifts, and acceleration changes, as described below in connection with the navigation response module 408.

  In one embodiment, the speed and acceleration module 406 analyzes data received from one or more computational and electromechanical devices in the vehicle 200 configured to change the speed and / or acceleration of the vehicle 200. Software configured as described above may be stored. For example, the processing unit 110 executes instructions associated with the velocity and acceleration module 406 and based on data derived from the execution of the monocular image analysis module 402 and / or the stereoscopic image analysis module 404, the target of the vehicle 200 The speed can be calculated. Such data includes, for example, target position, speed, and / or acceleration, position and / or speed of vehicle 200 with respect to nearby vehicles, pedestrians, or road objects, and position information of vehicle 200 with respect to road lane marks. And the like. In addition, the processing unit 110 receives sensor inputs (eg, information from a radar) and inputs from other systems of the vehicle 200 such as the throttle system 220, brake system 230, and / or steering system 240 of the vehicle 200. Based on this, the target speed of the vehicle 200 can be calculated. Based on the calculated target speed, the processing unit 110 sends an electronic signal to the throttle system 220, brake system 230, and / or steering system 240 of the vehicle 200, for example, to physically weaken the brake of the vehicle 200. Or a change in speed and / or acceleration may be triggered by weakening the accelerator.

  In one embodiment, the navigation response module 408 is executable by the processing unit 110 and determines a desired navigation response based on data derived from the execution of the monocular image analysis module 402 and / or the stereoscopic image analysis module 404. Software to be stored. Such data may include location and speed information associated with nearby vehicles, pedestrians, and road objects, target location information for the vehicle 200, and the like. Further, in some embodiments, the navigation response is detected from the map data, the predetermined location of the vehicle 200, and / or the vehicle 200 and the execution of the monocular image analysis module 402 and / or the stereoscopic image analysis module 4041. It may be based (partially or completely) on the relative velocity or relative acceleration between one or more objects. The navigation response module 408 is based on sensor inputs (eg, information from a radar) and inputs from other systems of the vehicle 200 such as the throttle system 220, brake system 230, and steering system 240 of the vehicle 200. The navigation response can also be determined. Based on the desired navigation response, the processing unit 110 sends an electronic signal to the throttle system 220, the brake system 230, and the steering system 240 of the vehicle 200, for example, to turn the steering wheel of the vehicle 200, at a predetermined angle. By achieving rotation, a desired navigation response can be triggered. In some embodiments, the processing unit 110 uses the output of the navigation response module 408 (eg, the desired navigation response) as an input to the execution of the speed and acceleration module 406 to calculate the speed change of the vehicle 200. Can do.

  FIG. 5A is a flowchart illustrating an example process 500A for generating one or more navigation responses based on monocular image analysis, according to disclosed embodiments. In step 510, the processing unit 110 may receive a plurality of images via the data interface 128 between the processing unit 110 and the image acquisition unit 120. For example, a camera (such as an image capture device 122 having a field of view 202) included in the image acquisition unit 120 captures multiple images of an area in front of the vehicle 200 (or side or rear of the vehicle, for example) and data connection It can be transmitted to the processing unit 110 via (for example, digital, wired, USB, wireless, Bluetooth, etc.). The processing unit 110 may execute the monocular image analysis module 402 to analyze the plurality of images at step 520, as described in more detail below in connection with FIGS. 5B-5D. By performing the analysis, the processing unit 110 may detect a set of features in the set of images, such as lane marks, vehicles, pedestrians, road signs, highway exit lamps, and traffic lights.

  The processing unit 110 may also execute a monocular image analysis module 402 in step 520 to detect various road hazards such as, for example, truck tire parts, dropped road signs, loose cargo, and small animals. . The structure, shape, size, and color of road hazards can vary, making detection of such hazards more difficult. In some embodiments, the processing unit 110 may execute the monocular image analysis module 402 to perform multi-frame analysis on multiple images to detect road hazards. For example, the processing unit 110 may estimate camera movement between successive image frames and calculate pixel mismatches between frames to build a 3D map of the road. Next, the processing unit 110 may use the 3D map to detect the road surface and dangerous objects present on the road surface.

  In step 530, processing unit 110 executes navigation response module 408 to provide one or more navigation responses to vehicle 200 based on the analysis performed in step 520 and the techniques described above in connection with FIG. Can be generated. The navigation response can include, for example, turns, lane shifts, acceleration changes, and the like. In some embodiments, the processing unit 110 may use data derived from the execution of the velocity and acceleration module 406 to generate one or more navigation responses. Furthermore, multiple navigation responses may occur simultaneously, may occur sequentially, or may occur in any combination thereof. For example, the processing unit 110 may cause the vehicle 200 to exceed one lane by, for example, sequentially transmitting control signals to the steering system 240 and the throttle system 220 of the vehicle 200 and then, for example, accelerate. Alternatively, the processing unit 110 may cause the vehicle 200 to brake and simultaneously shift the lane, for example, by sending control signals to the brake system 230 and steering system 240 of the vehicle 200 simultaneously.

  FIG. 5B is a flowchart illustrating an exemplary process 500B for detecting one or more vehicles and / or pedestrians in a set of images according to disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to perform process 500B. In step 540, the processing unit 110 may identify a set of candidate objects representing vehicles and / or pedestrians that may be present. For example, the processing unit 110 scans one or more images, compares the images to one or more predetermined patterns, and within each image, the target object (eg, vehicle, pedestrian, or portion thereof) ) May be identified. The predetermined pattern may be specified to achieve a high rate of “false hits” and a low rate of “missing”. For example, the processing unit 110 may use a low similarity threshold to a predetermined pattern to identify candidate objects as potential vehicles or pedestrians. By doing so, the processing unit 110 may be able to reduce the probability of missing (eg, not identifying) candidate objects representing the vehicle or pedestrian.

  In step 542, the processing unit 110 may filter the set of candidate objects to exclude certain candidates (eg, irrelevant or less relevant objects) based on the classification criteria. Such criteria may be derived from various characteristics associated with object types stored in a database (eg, a database stored in memory 140). The characteristics may include the shape, dimensions, texture, position, and the like (eg, relative to the vehicle 200) of the object. Accordingly, processing unit 110 may reject false candidates from a set of candidate objects using one or more sets of criteria.

  In step 544, the processing unit 110 may analyze the plurality of image frames to determine whether an object in the set of candidate images represents a vehicle and / or a pedestrian. For example, the processing unit 110 may track candidate objects detected over successive frames and accumulate per-frame data (eg, size, position relative to the vehicle 200, etc.) associated with the detected objects. Furthermore, the processing unit 110 may estimate the parameters of the detected object and compare the frame-by-frame position data of the object with the predicted position.

  In step 546, processing unit 110 may construct a set of detected object measurements. Such measurements may include, for example, position, velocity, and acceleration values (relative to vehicle 200) associated with the detected object. In some embodiments, the processing unit 110 may use an estimation technique that uses a series of time-based observations such as a Kalman filter or linear quadratic estimation (LQE) and / or different object types (eg, car, truck, pedestrian, Measurements can be constructed based on modeling data available on bicycles, road signs, etc.). The Kalman filter may be based on a measurement of the scale of the object, where the scale measurement is proportional to the time to collision (eg, the amount of time until the vehicle 200 reaches the object). Accordingly, by executing steps 540-546, the processing unit 110 identifies vehicles and pedestrians that appear in the set of captured images and information associated with the vehicles and pedestrians (eg, position, speed, size). Can be derived. Based on the identified and derived information, the processing unit 110 may generate one or more navigation responses in the vehicle 200 as described above in connection with FIG. 5A.

  In step 548, processing unit 110 may perform an optical flow analysis of one or more images to reduce the probability of detecting a “false hit” and the missed candidate object representing the vehicle or pedestrian. Optical flow analysis may refer to, for example, analyzing a movement pattern that is distinct from road surface movement for the vehicle 200 in one or more images associated with other vehicles and pedestrians. The processing unit 110 may calculate the movement of the candidate object by observing different positions of the object across multiple image frames captured at different times. The processing unit 110 may calculate the movement of the candidate object using the position and time values as inputs to the mathematical model. Thus, optical flow analysis may provide another way to detect vehicles and pedestrians in the vicinity of the vehicle 200. The processing unit 110 may perform optical flow analysis in combination with steps 540-546 to provide redundancy for detecting vehicles and pedestrians and increase the reliability of the system 100.

  FIG. 5C is a flowchart illustrating an example process 500C for detecting road marks and / or lane geometry information in a set of images according to disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to perform process 500C. In step 550, the processing unit 110 may detect the set of objects by scanning one or more images. In order to detect lane mark segmentation, lane geometry information, and other related road marks, the processing unit 110 filters the set of objects and determines that they are irrelevant (eg, small holes, small rocks, etc.). Can be excluded. In step 552, the processing unit 110 may group together the segments detected in step 550 that belong to the same road mark or lane mark. Based on the grouping, processing unit 110 may develop a model, such as a mathematical model, to represent the detected segments.

  At step 554, processing unit 110 may construct a set of measurements associated with the detected segment. In some embodiments, the processing unit 110 may create a projection of the detected segment from the image plane to the real world plane. The projection may be characterized using a third order polynomial having coefficients corresponding to physical properties such as detected road position, slope, curvature, and curvature derivative. In generating the projection, the processing unit 110 may take into account road changes and the pitch and roll rate associated with the vehicle 200. In addition, the processing unit 110 may model the road height by analyzing the position and motion cues present on the road surface. Further, the processing unit 110 may estimate the pitch rate and roll rate associated with the vehicle 200 by tracking a set of feature points in one or more images.

  In step 556, the processing unit 110 may perform multi-frame analysis, for example, by tracking the detected segment over successive image frames and accumulating per-frame data associated with the detected segment. If processing unit 110 performs multi-frame analysis, the set of measurements constructed in step 554 can be more reliable and can be associated with an increasingly higher confidence level. Thus, by performing steps 550-556, the processing unit 110 may identify road marks that appear in the set of captured images and derive lane geometry information. Based on the identified and derived information, the processing unit 110 may generate one or more navigation responses in the vehicle 200 as described above in connection with FIG. 5A.

  In step 558, the processing unit 110 may further develop a safety model of the vehicle 200 in the surrounding environment of the vehicle, taking into account additional information sources. Processing unit 110 may use a safety model to define situations in which system 100 can safely perform autonomous control of vehicle 200. In order to develop a safety model, in some embodiments, the processing unit 110 may include other vehicle locations and movements, detected road edges and barriers, and / or map data (such as data from the map database 160). ) Can be taken into account. By considering additional information sources, the processing unit 110 may provide redundancy for detecting road marks and lane geometry and increase the reliability of the system 100.

  FIG. 5D is a flowchart illustrating an example process 500D for detecting traffic lights in a set of images according to disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to perform process 500D. In step 560, the processing unit 110 may scan the set of images and identify objects that appear at locations in the image that are likely to contain traffic lights. For example, the processing unit 110 may filter the identified objects to construct a set of candidate objects that excludes objects that are unlikely to correspond to traffic lights. Filtering may be based on various characteristics associated with the traffic light, such as shape, size, texture, and location (eg, relative to vehicle 200). Such characteristics can be based on many examples of traffic lights and traffic control signals and can be stored in a database. In some embodiments, the processing unit 110 may perform multiframe analysis on a set of candidate objects that may be traffic lights. For example, the processing unit 110 may track candidate objects over successive image frames, estimate the real world position of the candidate objects, and filter out moving objects (which are unlikely to be traffic lights). In some embodiments, the processing unit 110 may perform color analysis on the candidate object and identify the relative position of the detected color represented within the candidate object that may be a traffic light.

  In step 562, the processing unit 110 may analyze the intersection geometry. The analysis is extracted from (i) the number of lanes detected on both sides of the vehicle 200, (ii) marks detected on the road (such as arrow marks), and (iii) map data (such as data from the map database 160). May be based on any combination of intersection descriptions. Processing unit 110 may perform analysis using information derived from the execution of monocular analysis module 402. In addition, the processing unit 110 may identify the correspondence between the traffic light detected in step 560 and the lane that appears in the vicinity of the vehicle 200.

  As the vehicle 200 approaches the intersection, at step 564, the processing unit 110 may update the analyzed intersection geometry and the confidence associated with the detected traffic light. For example, the number of traffic lights that are estimated to appear at an intersection compared to the number that actually appears at the intersection can affect the reliability. Thus, based on the reliability, the processing unit 110 may delegate control to the driver of the vehicle 200 to improve the safety situation. By performing steps 560-564, the processing unit 110 may identify traffic lights that appear in the set of captured images and analyze the intersection geometry information. Based on the identification and analysis, the processing unit 110 may generate one or more navigation responses in the vehicle 200 as described above with respect to FIG. 5A.

FIG. 5E is a flowchart of an example process 500E for generating one or more navigation responses in the vehicle 200 based on the vehicle path according to disclosed embodiments. In step 570, the processing unit 110 may construct an initial vehicle path associated with the vehicle 200. Vehicle path, the coordinates (x, y) may represent using a set of points represented by the distance d _i between two points in the set of points may be in the range of 1 to 5 meters. In one embodiment, the processing unit 110 may construct an initial vehicle route using two polynomials, such as left and right road polynomials. The processing unit 110 calculates the geometric midpoint between the two polynomials and if there is a predetermined offset (offset 0 may correspond to traveling in the middle of the lane), the predetermined offset (e.g. smart lane). Each point included in the resulting vehicle path can be offset by (offset). The offset can be in a direction perpendicular to the section between any two points in the vehicle path. In another embodiment, the processing unit 110 uses a single polynomial and estimated lane width to mark each point in the vehicle path by half the estimated lane width plus a predetermined offset (eg, smart lane offset). Can be offset.

In step 572, the processing unit 110 may update the vehicle path constructed in step 570. The processing unit 110 uses the higher resolution so that the two-point distance d _k in the set of points representing the vehicle path is shorter than the distance d _i described above. Can be rebuilt. For example, the distance d _k can range from 0.1 to 0.3 meters. Processing unit 110 may reconstruct the vehicle path using a parabolic spline algorithm, which may result in a cumulative distance vector S corresponding to the total length of the vehicle path (ie, based on a set of points representing the vehicle path). .

In step 574, the processing unit 110 may identify a look-ahead point (represented in coordinates as (x _l , z _l )) based on the updated vehicle path performed in step 572. The processing unit 110 may extract a prefetch point from the cumulative distance vector S, and the prefetch point may be associated with a prefetch distance and a prefetch time. The look-ahead distance may have a lower range of 10-20 meters and may be calculated as the product of the speed of the vehicle 200 and the look-ahead time. For example, as the speed of the vehicle 200 decreases, the look-ahead distance can also decrease (eg, until a lower limit is reached). The look-ahead time, which can range from 0.5 to 1.5 seconds, can be inversely proportional to the gain of one or more control loops associated with generating a navigation response in the vehicle 200, such as a progress error tracking control loop. . For example, the gain of the progress error tracking control loop may depend on bandwidth such as yaw rate loop, steering actuator loop, and vehicle lateral dynamics. Therefore, the higher the gain of the progress error tracking control loop, the shorter the look-ahead time.

In step 576, processing unit 110 may determine a progress error and yaw rate command based on the look-ahead point identified in step 574. The processing unit 110 may identify the progress error by calculating the arctangent of the look-ahead point, eg, arctan (x ₁ / z ₁ ). Processing unit 110 may determine the yaw rate command as the product of the progress error and the high level control gain. The high level control gain may be equal to (2 / prefetch time) if the prefetch distance is not at the lower limit. If the look-ahead distance is a lower limit, the high level control gain may be equal to (2 * speed of vehicle 200 / read-ahead distance).

  FIG. 5F is a flowchart illustrating an exemplary process 500F for identifying whether a preceding vehicle is changing lanes, according to disclosed embodiments. In step 580, the processing unit 110 may identify navigation information associated with a preceding vehicle (eg, a vehicle moving in front of the vehicle 200). For example, the processing unit 110 may determine the position, speed (eg, direction and speed), and / or acceleration of a preceding vehicle using the techniques described above in connection with FIGS. 5A and 5B. Processing unit 110 may use one or more road polynomials, look-ahead points (associated with vehicle 200), and / or snail trails (eg, taken by a preceding vehicle) using the techniques described above in connection with FIG. 5E. It is also possible to specify a set of points describing a route.

  In step 582, the processing unit 110 may analyze the navigation information identified in step 580. In one embodiment, the processing unit 110 may calculate the distance (eg, along the trail) between the snail trail and the road polynomial. This difference in distance along the trail is a predetermined threshold (for example, 0.1 to 0.2 meters on straight roads, 0.3 to 0.4 meters on gently curved roads, 0.5 to 0.4 on sharply curved roads). If it exceeds 0.6 meters), the processing unit 110 may determine that the preceding vehicle is likely to be changing lanes. If multiple vehicles are detected traveling in front of the vehicle 200, the processing unit 110 may compare the snail trails associated with each vehicle. Based on the comparison, the processing unit 110 may determine that a vehicle whose snail trail does not match that of another vehicle is likely to be changing lanes. The processing unit 110 may further compare the curvature of the snail trail (associated with the preceding vehicle) with the expected curvature of the road segment on which the preceding vehicle is moving. Expected curvature may be extracted from map data (eg, data from map database 160), road polynomials, other vehicle snail trails, prior knowledge about roads, and the like. If the difference between the curvature of the snail trail and the expected curvature of the road segment exceeds a predetermined threshold, the processing unit 110 may determine that the preceding vehicle is likely to be changing lanes.

In another embodiment, the processing unit 110 compares the instantaneous position of the preceding vehicle with the look-ahead point (associated with the vehicle 200) over a specific time period (eg, 0.5-1.5 seconds). obtain. The cumulative sum of the difference and difference between the instantaneous position of the preceding vehicle and the look-ahead point during a certain time period is a predetermined threshold (for example, 0.3 to 0.4 meters on a straight road, gently curved If the road exceeds 0.7 to 0.8 meters on a road that has been cut and 1.3 to 1.7 meters on a sharply curved road), the processing unit 110 determines that there is a high possibility that the preceding vehicle is changing lanes. obtain. In another embodiment, the processing unit 110 may analyze the snail trail geometry by comparing the lateral distance traveled along the trail to the expected curvature of the snail trail. The expected radius of curvature is calculated:
(Δ _z ² + δ _x ² ) / 2 / (δ _x )
Where σ _x represents the lateral travel distance and σ _z represents the longitudinal travel distance. If the difference between the lateral travel distance and the expected curvature exceeds a predetermined threshold (eg, 500-700 meters), the processing unit 110 may determine that the preceding vehicle is likely to be changing lanes. In another embodiment, the processing unit 110 may analyze the position of the preceding vehicle. If the position of the preceding vehicle makes the road polynomial invisible (eg, the preceding vehicle overlaps the road polynomial), the processing unit 110 may determine that the preceding vehicle is likely to be changing lanes. If the position of the preceding vehicle is such that another vehicle is detected in front of the preceding vehicle and the snail trails of the two vehicles are not parallel, the processing unit 110 will change the lane to the (closer) preceding vehicle. It can be determined that there is a high possibility of being in the middle.

  In step 584, the processing unit 110 may identify whether the preceding vehicle 200 is changing lanes based on the analysis performed in step 582. For example, processing unit 110 may make that determination based on a weighted average of the individual analyzes performed in step 582. Under such a scheme, for example, the value “1” may be assigned to a determination by the processing unit 110 based on a particular type of analysis that the preceding vehicle is likely to be changing lanes (“0”). Indicates that it is unlikely that the preceding vehicle is changing lanes). Different weights may be assigned to the different analyzes performed in step 582, and the disclosed embodiments are not limited to any particular combination of analysis and weight.

  FIG. 6 is a flowchart illustrating an example process 600 for generating one or more navigation responses based on stereoscopic image analysis, according to disclosed embodiments. In step 610, the processing unit 110 may receive the first and second plurality of images via the data interface 128. For example, a camera included in the image acquisition unit 120 (such as image capture devices 122 and 124 having fields of view 202 and 204) captures a first and second plurality of images of an area in front of the vehicle 200 and is digitally connected ( For example, it can be transmitted to the processing unit 110 via USB, wireless, Bluetooth, or the like. In some embodiments, the processing unit 110 may receive the first and second plurality of images via two or more data interfaces. The disclosed embodiments are not limited to any particular data interface configuration or protocol.

  In step 620, the processing unit 110 executes the stereoscopic image analysis module 404 to perform stereoscopic image analysis of the first and second plurality of images to create a 3D map of the road in front of the vehicle, and the lane Features in the image such as marks, vehicles, pedestrians, road signs, highway exit lamps, traffic lights, and road hazards may be detected. Stereoscopic image analysis may be performed similar to the steps described above in connection with FIGS. 5A-5D. For example, the processing unit 110 executes the stereoscopic image analysis module 404 to detect candidate objects (eg, vehicles, pedestrians, road marks, traffic lights, road hazards, etc.) in the first and second plurality of images. And filtering and excluding a subset of candidate objects based on various criteria, performing multi-frame analysis, constructing measurements, and determining the confidence of the remaining candidate objects. In performing the above steps, processing unit 110 may consider information from both the first and second plurality of images, rather than information from only one set of images. For example, the processing unit 110 may analyze differences in pixel level data (or other data subsets from within the two streams of captured images) of candidate objects that appear in both the first and second plurality of images. As another example, the processing unit 110 exists by observing that an object appears in one of a plurality of images but not in the remaining other images, or for an object appearing in two image streams. For other differences that may be obtained, the position and / or velocity of the candidate object (eg, relative to the vehicle 200) may be estimated. For example, the position, speed, and / or acceleration relative to the vehicle 200 may be determined based on trajectories, positions, movement characteristics, etc. of features associated with objects appearing in one or both of the image streams.

  In step 630, the processing unit 110 executes the navigation response module 408 to perform one or more navigation operations on the vehicle 200 based on the analysis performed in step 620 and the technique described above in connection with FIG. Can be generated. Navigation responses can include, for example, turns, lane shifts, acceleration changes, speed changes, brakes, and the like. In some embodiments, the processing unit 110 may use data derived from the execution of the velocity and acceleration module 406 to generate one or more navigation responses. Furthermore, the multiple navigation responses may be performed simultaneously, sequentially, or in any combination thereof.

  FIG. 7 is a flowchart illustrating an example process 700 for generating one or more navigation responses based on analysis of three sets of images, according to disclosed embodiments. In step 710, the processing unit 110 may receive the first, second, and third plurality of images via the data interface 128. For example, the cameras included in the image acquisition unit 120 (such as image capture devices 122, 124, and 126 having fields of view 202, 204, and 206) may include first and second areas in front and / or side areas of the vehicle 200. , And a third plurality of images may be captured and transmitted to the processing unit 110 via a digital connection (eg, USB, wireless, Bluetooth, etc.). In some embodiments, the processing unit 110 may receive the first, second, and third plurality of images via more than two data interfaces. For example, each of the image capture devices 122, 124, and 126 may have an associated data interface that communicates data to the processing unit 110. The disclosed embodiments are not limited to any particular data interface configuration or protocol.

  In step 720, the processing unit 110 analyzes the first, second, and third plurality of images to identify lane marks, vehicles, pedestrians, road signs, highway exit lamps, traffic lights, road hazards, and the like. The features in the image can be detected. The analysis may be performed similarly to the steps described above in connection with FIGS. 5A-5D and FIG. For example, the processing unit 110 may perform monocular image analysis on each of the first, second, and third plurality of images (eg, related to the implementation of the monocular image analysis module 402 and FIGS. 5A-5D). And based on the steps described above). Alternatively, the processing unit 110 performs stereo image analysis on the first and second plurality of images, the second and third plurality of images, and / or the first and third plurality of images. (E.g., through execution of the stereoscopic image analysis module 404 and based on the steps described above in connection with FIG. 6). Processed information corresponding to the analysis of the first, second, and / or third plurality of images may be combined. In some embodiments, the processing unit 110 may perform a combination of monocular image analysis and stereo image analysis. For example, the processing unit 110 performs monocular image analysis on the first plurality of images (eg, via execution of the monocular image analysis module 402) and performs stereoscopic image analysis on the second and third plurality of images. (Eg, through execution of the stereoscopic image analysis module 404). The configuration of the image capture devices 122, 124, and 126 (including each position and field of view 202, 204, and 206) affects the type of analysis performed on the first, second, and third images. Can affect. The disclosed embodiments are not limited to the specific configuration of the image capture devices 122, 124, and 126 or the type of analysis performed on the first, second, and third plurality of images.

  In some embodiments, the processing unit 110 may perform a test on the system 100 based on the images acquired and analyzed in steps 710 and 720. Such a test may provide an indicator of the overall performance of the system 100 with a particular configuration of the image capture devices 122, 124, and 126. For example, the processing unit 110 may identify a “false hit” (eg, when the system 100 incorrectly determines that a vehicle or pedestrian is present) and a “missing” rate.

  In step 730, the processing unit 110 may generate one or more navigation responses at the vehicle 200 based on information derived from two of the first, second, and third plurality of images. Selecting two of the first, second, and third plurality of images depends on various factors such as the number, type, and size of objects detected in each of the plurality of images, for example. Can do. The processing unit 110 determines the quality and resolution of the image, the effective field of view reflected in the image, the number of captured frames, and the extent to which the object or objects of interest actually appear in the frame (e.g., the percentage of frames in which the object appears) The selection can be made based on, for example, the rate at which the object appears in each such frame.

  In some embodiments, the processing unit 110 identifies first, second, and third by identifying the degree to which information derived from one image source is consistent with information derived from other image sources. Information derived from two of the plurality of images may be selected. For example, the processing unit 110 combines processed information derived from each of the image capture devices 122, 124, and 126 (whether monocular analysis, stereo analysis, or any combination of the two). A visual indicator that is consistent across images captured from each of the image capture devices 122, 124, and 126 (eg, lane marks, detected vehicles and / or their location and / or paths, detected traffic lights, etc.) ) Can be specified. The processing unit 110 may also exclude inconsistent information across captured images (eg, a vehicle that is changing lanes, a lane model that indicates vehicles that are too close to the vehicle 200, etc.). Accordingly, processing unit 110 may select derived information from two of the first, second, and third plurality of images based on the identification of consistent information and inconsistent information.

  The navigation response may include, for example, turns, lane shifts, and acceleration changes. Processing unit 110 may generate one or more navigation responses based on the analysis performed in step 720 and the technique described above in connection with FIG. The processing unit 110 may also generate one or more navigation responses using data derived from the execution of the velocity and acceleration module 406. In some embodiments, the processing unit 110 may have a relative position, relative velocity, and / or relative between the vehicle 200 and an object detected in any of the first, second, and third images. Based on the acceleration, one or more navigation responses may be generated. Multiple navigation responses may be performed simultaneously, sequentially, or any combination thereof.

Interrupted Vehicle Prediction and Altruistic Behavior Response During navigation, an autonomous vehicle, such as vehicle 200, may encounter another vehicle that is attempting a lane shift. For example, a vehicle in a lane on the left or right of the lane in which the vehicle 200 is traveling (for example, a lane designated by a mark on the road or a lane aligned with the route of the vehicle 200 without a mark on the road) May attempt to change or interrupt the lane in which the vehicle 200 is traveling. Such a vehicle may be the target vehicle. When such an interruption occurs, the vehicle 200 may need to make a navigation response. For example, the vehicle 200 can avoid interruption by the target vehicle by changing its speed or acceleration and / or changing to another lane.

  In some examples, the target vehicle may appear to be interrupting, but it may not be interrupted eventually. The driver of the target vehicle (or even a fully or partially autonomous navigation system associated with the target vehicle) may have changed mind, for example, or otherwise changed the navigation plan other than a lane change, or the target vehicle simply slipped It can only be a thing. Accordingly, in order to avoid frequent unnecessary braking and / or acceleration, the vehicle 200 delays execution of the navigation response until it is determined that the possibility of interruption by the target vehicle is sufficient. It may be desirable. On the other hand, in some situations (especially when a course change to the route of the host vehicle by the target vehicle is expected), it may be desirable for the vehicle 200 to perform the navigation response earlier. Such navigation, based at least in part on expected behavior, can help avoid sudden braking and can significantly increase the safety margin. Improving the prediction of an interrupt attempt by the target vehicle can help minimize both unnecessary and sudden braking. Such improved prediction can be referred to as interrupt detection, and the navigation response taken when an interrupt is detected can be referred to as an interrupt response.

  In some embodiments, such improved predictions may include, for example, static road features (eg, lane end, road split), dynamic road features (eg, ahead of a vehicle that may attempt an interrupt). Presence of other vehicles) and / or from monocular and / or stereoscopic image analysis and / or other sources (e.g., GPS devices, speed sensors, etc.) to detect traffic rules and driving habits within a geographical area Can depend on information obtained from accelerometers, suspension sensors, etc. These static road features, dynamic road features, and / or traffic rules and driving habits can be referred to as predetermined interrupt sensitivity changing factors, and the presence of one or more predetermined interrupt sensitivity changing factors can be 200 can change its sensitivity to an interrupt attempt by the target vehicle. For example, if a predetermined interrupt sensitivity changing factor is present in the environment (eg, the target vehicle is following other vehicles that are moving at a lower speed at a shorter distance), the vehicle 200 may use the first interrupt sensitivity parameter. If there is no predetermined interrupt sensitivity changing factor in the environment (eg, there is only the target vehicle in that lane), the vehicle 200 may depend on the second interrupt sensitivity parameter. The second interrupt sensitivity parameter can be different (eg, more sensitive) than the first interrupt sensitivity parameter.

  In some cases, an interruption by the target vehicle may be necessary. For example, the lane in which the target vehicle is traveling may end, the road may be split, or the lane in which the target vehicle is traveling is obstructed (eg, a stopped vehicle, object, Or other types of disorders). However, in other cases, the interruption by the target vehicle is optional. For example, the target vehicle may simply attempt an interrupt to overtake a slower moving vehicle. If the interrupt is optional, whether the target vehicle attempts to interrupt may depend on how the vehicle 200 (eg, the host vehicle) behaves. For example, the vehicle 200 may signal that the target vehicle may be interrupted by decelerating, or may indicate that the target vehicle should not be interrupted by accelerating. Acceleration may in many cases be desirable for the vehicle 200 as long as the vehicle 200 can reach its destination earlier, but the operator of the vehicle 200 (or a navigation system that controls the vehicle 200 fully or partially). May tend to allow interrupts in some or all cases. This can be called altruistic behavior. Thus, it may be desirable to take into account considerations regarding altruistic behavior when determining whether the vehicle 200 permits an interruption by the target vehicle.

  FIG. 8 is an exemplary functional block of memories 140 and / or 150 that may be stored / programmed with instructions to perform one or more operations according to the disclosed embodiments. The following relates to the memory 140, but those skilled in the art will appreciate that the instructions can be stored in the memory 140 and / or 150 and that the instructions can be executed by the processing unit 110 of the system 100.

  As shown in FIG. 8, the memory 140 may include a monocular image analysis module 402, a stereoscopic image analysis module 404, a velocity and acceleration module 406, and a navigation response module 408, which are related to FIG. Thus, any of the forms of modules described above can be taken. The memory 140 may further include an interrupt detection module 802, an interrupt response module 804, and an altruistic behavior module 806. The disclosed embodiments are not limited to any particular configuration of memory 140. Further, application processor 180 and / or image processor 190 may execute instructions stored in any of modules 402-408 and 802-806 included in memory 140. Those skilled in the art will appreciate that references to processing unit 110 in the following discussion may refer to application processor 180 and image processor 190 individually or collectively. Accordingly, any of the following process steps may be performed by one or more processing devices. Further, any of modules 402-408 and 802-806 may be stored remotely from vehicle 200 (eg, one or more connected to a network and accessible via a wireless transceiver 172 of vehicle 200 over the network) Distributed to servers).

  In some embodiments, the interrupt detection module 802, when executed by the processing unit 110, may store instructions that allow detection of the target vehicle. The target vehicle may be a vehicle traveling in a lane adjacent to the lane in which the vehicle 200 is traveling, and in some cases may be a preceding vehicle. In some embodiments, the interrupt detection module 802 is a set of images acquired by one of the image capture devices 122, 124, and 126, as described above in connection with the monocular image analysis module 402. The target vehicle may be detected by performing a monocular image analysis. In some embodiments, the processing unit 110 may combine information from the set of images with additional sensor information (eg, information from a radar or lidar) to perform monocular image analysis. As described above in connection with FIGS. 5A-5D, such monocular image analysis can be performed on a set of features within a set of images, such as vehicle edge features, vehicle lights (or other vehicle-related features). Element), lane marks, vehicles, or road signs. For example, detecting the target vehicle may be performed as described in connection with FIG. 5B, including identifying a set of candidate objects that include the target vehicle and filtering the set of candidate objects. And performing a multi-frame analysis of the set of candidate objects, constructing a set of measurements of detected objects (including the target vehicle), and performing an optical flow analysis. The measurement value may include, for example, a position, speed, and acceleration value (related to the vehicle 200) associated with the target coordinates.

  Alternatively or additionally, in some embodiments, the interrupt detection module 802 selects from any of the image capture devices 122, 124, and 126 as described above in connection with the stereoscopic image analysis module 404. The target vehicle may be detected by performing a stereo image analysis of the first and second sets of images acquired by the combined image capture device. In some embodiments, the processing unit 110 combines information from the first and second sets of images with additional sensor information (eg, information from a radar or lidar) to perform stereoscopic image analysis. Can do. For example, the stereoscopic image analysis module 404 can provide instructions to perform a stereoscopic image analysis based on a first set of images acquired by the image capture device 124 and a second set of images acquired by the image capture device 126. May be included. As described in connection with FIG. 6, stereoscopic image analysis can be performed on feature sets in the first and second sets of images, such as vehicle edge features, vehicle lights (or other vehicle-related features). Element), lane mark, car or road sign.

  In other embodiments, as an alternative to detecting one or more vehicles (eg, target vehicles) by analyzing images acquired by one of the image capture devices 122, 124, and 126. The interrupt detection module 802 may instead detect the vehicle through analysis of sensor information, such as information obtained via a radar or lidar device included in the system 100.

  In some embodiments, the interrupt detection module 802, when executed by the processing unit 110, will also cause the target vehicle to attempt an interrupt (i.e., the target vehicle is directed to the lane in which the vehicle 200 is traveling). Instructions may be stored that allow the identification of an indicator that a lane change or other host vehicle will attempt to move into the travel path. In some embodiments, identifying the indicator may detect the position and / or velocity of the target vehicle using monocular and / or stereoscopic image analysis, as described above in connection with FIG. 5B. Can be included. In some embodiments, identifying the indicator may further include detecting one or more road marks, as described above in connection with FIG. 5C. Also, in some embodiments, identifying the indicator may further include detecting that the target vehicle is changing lanes, as described above in connection with FIG. 5F. In particular, the processing unit 110 uses the techniques described above in connection with FIGS. 5A and 5B to navigate associated with the target vehicle, such as the position, speed (eg, direction and speed), and / or acceleration of the preceding vehicle. Information can be specified. The processing unit 110 may also use one or more road polynomials, look-ahead points (associated with the vehicle 200), and / or snail trails (e.g., leading vehicles) using the techniques described above in connection with FIG. A set of points describing the path followed) can also be specified. Further, the processing unit 110 may, for example, calculate the distance (eg, along the trail) between the snail trail and the road polynomial, as described above in connection with FIG. The navigation information may be analyzed by comparing the instantaneous position of the preceding vehicle with the look-ahead point (associated with the vehicle 200) over (e.g., 0.5-1.5 seconds). Based on the analysis, the processing unit 110 may identify whether there is an indicator that the preceding vehicle is attempting to interrupt.

  The interrupt detection module 802 may further store instructions that, when executed by the processing unit 110, make it possible to detect that a predetermined interrupt sensitivity changing element is present in the environment of the vehicle 200. The predetermined interrupt sensitivity changing element may include any indicator that suggests a tendency for the target vehicle to remain on the current route or to change course to the host vehicle's route. Such sensitivity changing elements include static road features (eg, lane termination, road splits, barriers, objects), dynamic road features (eg, the presence of other vehicles in front of the vehicle that may attempt to interrupt). And / or traffic rules and driving habits within a geographic region. In some embodiments, detecting the predetermined interrupt sensitivity changing element uses monocular and / or stereo image analysis to set a set of features in the set of images, such as a lane mark, a vehicle, a pedestrian, a road Detecting signs, highway exit ramps, traffic lights, dangerous goods, and any other features associated with the vehicle environment, such as described above in connection with FIGS. Alternatively or additionally, in some embodiments, detecting a predetermined interrupt sensitivity modification element may include using other sensor information, eg, GPS data. For example, the end of a lane, highway division, etc. can be specified based on map data. As another example, traffic rules may be identified based on GPS location data.

  Interrupt response module 804 may store instructions that, when executed by processing unit 110, allow an interrupt response to be executed. The interrupt response can take any of the forms described above with respect to the navigation response, such as turns, lane shifts, acceleration changes, etc., which will be discussed later in connection with the navigation response module 408.

  In some embodiments, whether an interrupt response is performed may depend on the presence or absence of a predetermined interrupt sensitivity changing factor in the environment. For example, if a given interrupt sensitivity change factor is not present in the environment (eg, there is only a vehicle attempting to interrupt in that lane, either through image analysis or GPS / map / traffic data review, The vehicle 200 may rely on the first interrupt sensitivity parameter, such as no obstacle is detected. In some embodiments, if the predetermined interrupt sensitivity modification factor is not in the environment, the vehicle 200 may rely on a value associated with the first interrupt sensitivity parameter. The value may include any value directly associated with and / or measured with the first interrupt sensitivity parameter, or may be one or more values indirectly related to the first interrupt sensitivity parameter. .

  On the other hand, if one or more predetermined interrupt sensitivity changing elements are present in the environment (e.g., the lane shift or If the lane end condition is detected visually through image analysis or through GPS / map / traffic information reference, the local area is an area that does not encourage driving the left lane, etc.), the vehicle 200 has a second interrupt sensitivity. Can depend on parameters. The second interrupt sensitivity parameter may be different (eg, more sensitive) than the first interrupt sensitivity parameter. In some embodiments, the vehicle 200 may rely on a value associated with the second interrupt sensitivity parameter if one or more predetermined interrupt sensitivity modification factors are present in the environment. Similar to the discussion above regarding the value associated with the first interrupt sensitivity parameter, the value associated with the second interrupt sensitivity parameter can be any directly associated and / or measured with respect to the second interrupt sensitivity parameter. The value may include a value or may be one or more values indirectly related to the second interrupt sensitivity parameter.

  The first and second interrupt sensitivity parameters may establish two operating states: a first one in which the host vehicle may be less sensitive to movement or course changes by the target vehicle, and a host vehicle in the target vehicle. A second state that can be highly sensitive can be established by moving or changing course. Such a state-based scheme can reduce the number or degree of deceleration, acceleration, or course changes in the first state when course changes by the target vehicle are not expected based on any environmental conditions. However, in the second state, if at least one state is detected or identified in the target vehicle's environment that is expected to cause a course change (or other navigation response) of the target vehicle, the host vehicle May be more sensitive to course changes or navigation changes of the target vehicle, and may make course changes, accelerations, decelerations, etc. based on slight changes recognized in the navigation of the target vehicle. Such a small change in the second state can be interpreted to be consistent with the expected navigation changes by the target vehicle, and thus can ensure a faster response compared to navigation in the first state.

  Furthermore, in the second state, the course change of the host vehicle can be performed without detecting the navigation response or change of the target vehicle based solely on the detection of the sensitivity changing element. For example, if image analysis of the host vehicle system indicates the presence of a sensitivity-changing element (for example, recognizing a road sign indicating the lane end state in the captured image, recognizing the lane end in front of the target vehicle in the image By recognizing obstacles ahead of the target vehicle or vehicles moving at a slower speed, or any other sensitivity changing element), the host vehicle can One or more preemptive actions (deceleration, acceleration, course change, etc.) may be taken. Such changes can be guaranteed based on the expectation that the target vehicle will need to make navigation changes in the future. Such a change may increase or maximize the length of time between a host vehicle navigation change and an expected target vehicle navigation change.

  The interrupt sensitivity parameter may take the form of a threshold (eg, a distance threshold between a snail trail and a road polynomial (eg, along the trail), or during a particular time period, as described above with respect to FIG. 5F. Distance threshold between the instantaneous position of the target vehicle (e.g. 0.5 to 1.5 seconds) and the look-ahead point (associated with vehicle 200)), a combination of such thresholds (e.g. A combination of a lateral movement speed threshold and a lateral position threshold of the vehicle relative to the vehicle 200), and / or a weighted average of such thresholds). As mentioned above, in some embodiments, the interrupt sensitivity parameter can also be any variable that is arbitrarily assigned, any value whose value is associated with the parameter or indirectly related to the value of the parameter. Can be used to set the two (or more) sensitivity states described above. For example, such a variable may be associated with a first value that indicates a low sensitivity state when no sensitivity change element is detected, and the higher sensitivity state when the sensitivity change element is discovered in the environment of the target vehicle. May be associated with a second value indicating.

  In some embodiments, the interrupt response module 804 may evaluate the distance from the vehicle 200 to a vehicle following the vehicle 200 and include that distance in the interrupt sensitivity parameter. For example, the vehicle 200 may include one or more rear sensors (eg, radar sensors) to identify the distance from the vehicle 200 to a vehicle following the vehicle 200. Furthermore, in some embodiments, the vehicle 200 may include one or more rear cameras, which provide the system 100 with an image for analysis to determine the distance from the vehicle 200 to a vehicle following it. Supply. In yet another embodiment, the system 100 may use data from one or more sensors and from one or more rear cameras to determine the distance from the vehicle 200 to the vehicle following it. Based on the distance from the vehicle 200, the interrupt response module 804 may evaluate whether it is safe for the vehicle 200 to decelerate based on the distance between the vehicle 200 and a vehicle following the vehicle 200.

  Alternatively or additionally, the interrupt sensitivity parameter may be derived from the case using machine learning techniques such as neural networks. For example, a neural network may be trained to identify interrupt sensitivity parameters based on scenarios. As an example, in the neural network, the road includes three traveling lanes, the vehicle 200 is traveling in the central lane, the central lane has a preceding vehicle, two vehicles are in the left lane, and one vehicle. Can train on the scenario in the right lane. The scenario can specify the longitudinal position, longitudinal speed, lateral position, and lateral speed of each vehicle for the neural network. Based on the scenario, the neural network may, for example, provide a binary output (eg, whether the target vehicle attempts to interrupt in the next N frames) or a value output (eg, the target vehicle interrupts the central lane). Time to try) can be specified. Binary or value outputs derived from these outputs, or other values, can be used as interrupt sensitivity parameters. The neural network may similarly be trained for other scenarios, such as a scenario where there are only two driving lanes on the road, no vehicles preceding the central lane, or no vehicles in the right lane. In some embodiments, the neural network may be provided via one or more programming modules stored in memory 140 and / or 150. In another embodiment, as an addition or alternative to memory 140 and / or 150, the neural network (or an aspect of the neural network) is located remotely from vehicle 200 and via a wireless transceiver 172 on the network. It can be provided via one or more accessible servers.

  In some embodiments, scenarios can be classified according to the number of road lanes and the location of the vehicle 200 to minimize the number of scenarios. For example, the scenario is a scenario where there are two lanes, the vehicle 200 is in the left lane, a scenario where there are two lanes, the vehicle 200 is in the right lane, a scenario where there are three lanes, and the vehicle 200 is in the left lane, Scenario with 3 lanes, vehicle 200 in the center lane Scenario with 3 lanes, vehicle in the right lane Scenario with 4 lanes, vehicle 200 in the leftmost lane, 4 lanes There may be included a silio or the like in which the vehicle 200 is in the center left lane. In each scenario, the preceding vehicle may be traveling in the same lane as the vehicle 200, and two vehicles may be traveling in each of the other lanes. To account for scenarios where there is no one or more of these other vehicles (for example, there is no preceding vehicle or only one vehicle in the right lane), the longitudinal distance of the non-existing vehicle is set to infinity Can do.

  As described above, the vehicle 200 detects the target vehicle (eg, identifies an indicator that the target vehicle is attempting to interrupt) and detects whether any sensitivity changing elements are present in the environment. The interrupt response module 804 may provide the target vehicle, an indicator, and / or a sensitivity change factor (if any) to the neural network, which may select the scenario closest to the situation of the vehicle 200. Based on the selected scenario, the neural network may output a binary output (eg, whether the target vehicle attempts to interrupt during the next N frames) or a value output (eg, the target vehicle goes to the central lane as described above. Time before attempting to interrupt), from which the interrupt sensitivity parameter can be derived. Based on the interrupt sensitivity parameter, the interrupt response module 804 causes an interrupt response to be executed.

  As described above, the interrupt response can take any of the forms described above in connection with the navigation response, such as a turn, lane shift, and / or acceleration change discussed above with respect to the navigation response module 408. . In some embodiments, the processing unit 110 may generate one or more interrupt responses using data derived from the execution of the velocity and acceleration module 406 described above. Additionally, multiple interrupt responses can occur simultaneously, sequentially, or any combination thereof. For example, the processing unit 110 may cause the vehicle 200 to shift one lane and then accelerate, for example, by sequentially sending control signals to the steering system 240 and the throttle system 220 of the vehicle 200. Alternatively, the processing unit 110 may cause the vehicle 200 to brake simultaneously while shifting lanes, for example, by sending control signals to the braking system 230 and steering system 240 of the vehicle 200 simultaneously. In some embodiments, the interrupt response module 804 analyzes an image acquired by one of the image capture devices 122, 124, and 126 and any interrupt sensitivity changing element is present in the environment of the vehicle 200. It can be detected. In other embodiments, in addition to or as an alternative to analyzing the image, the interrupt response module 804 may indicate to the system 100 whether any interrupt sensitivity modification elements are present in the environment of the vehicle 200. Detection may be through analysis of sensor information, such as information acquired through included radar or lidar devices.

  In some embodiments, the neural network described above may be further configured to identify an interrupt response. For example, the neural network may identify and output indicating which navigation response is to be performed based on the detected target vehicle, indicator, and / or sensitivity change factor (if any). In some embodiments, such output may be identified based on previously detected driver behavior. In some embodiments, previously detected driver behavior may be filtered using a cost function analysis that prioritizes smooth driving behavior (eg, smooth driving behavior may include deceleration or acceleration over a period of time). Can be measured to minimize the square of the integral of).

  The interrupt detection module 802 and the interrupt response module 804 are further described below in connection with FIGS.

  The altruistic behavior module 806, when executed by the processing unit 110, provides considerations regarding altruistic behavior in order to determine whether the vehicle 200 should allow the target vehicle to interrupt the driving lane of the vehicle 200. Instructions that can be taken into account may be stored. To this end, the processing unit 110 may detect the target vehicle by any of the methods described above in connection with the interrupt detection module 802.

  Further, the processing unit 110 may identify one or more situational features associated with the target vehicle. The situational feature can be any feature that indicates, for example, that the target vehicle will benefit from changing lanes to the lane in which the vehicle 200 is traveling. For example, the situational feature is that the target vehicle is traveling in a lane adjacent to the traveling lane of the vehicle 200 and that the target vehicle is behind another vehicle that is traveling at a lower speed than the target vehicle. Can be shown. In general, the contextual feature may indicate that an interrupt or other navigation response by the target vehicle may not be necessary, but an interrupt may be advantageous for the target vehicle. Other such situations may include, for example, traffic jams, congested detours, lane exit situations, or any other situation where the target vehicle may wish to travel to the host vehicle's route.

  In some embodiments, detecting the contextual feature may be a feature within the set of images using monocular and / or stereo image analysis, as described above in connection with FIGS. Detecting a set of lane marks, vehicles, pedestrians, road signs, highway exit lamps, traffic lights, dangerous goods, and any other features associated with the vehicle environment. For example, identifying the contextual feature may be performed using monocular and / or stereo image analysis, as described above in connection with FIG. 5B, of the target vehicle and / or one or more other vehicles. It may include detecting position and / or velocity. In some embodiments, identifying the contextual feature may further include detecting one or more road marks, as described above in connection with FIG. 5C. Alternatively or additionally, in some embodiments, detecting the contextual feature may include using other sensor information such as GPS data. For example, a road entrance ramp may be identified based on map data. In some situations, detecting the situational feature may identify the location of the target vehicle relative to other vehicles and / or the number of vehicles in the vicinity of the target vehicle, etc. from the captured image. Can be included. In yet another embodiment, detecting the contextual feature may include using other sensor information, such as information obtained via a radar or lidar device included in the system 100.

  The altruistic behavior module 806, when executed by the processing unit 110, further performs navigation changes (eg, any of the navigation responses described above) that allow the target vehicle to interrupt the driving lane of the vehicle 200. Instructions can be stored that determine whether to occur. Such a determination may be made based on altruistic behavior parameters.

  The altruistic behavior parameter may be set based on an input from an operator of the vehicle 200, for example. For example, the operator allows altruistic behavior to interrupt all the target vehicles to the lane in which the vehicle 200 is traveling, so that only one in every n target vehicles can be transferred to the lane in which the vehicle 200 is traveling. To allow interruption, the target vehicle may be set to interrupt the driving lane of the vehicle 200 only when other vehicles ahead of the target vehicle are traveling below a certain speed, etc. . Alternatively or additionally, in some embodiments, the altruistic behavior parameter may be selectable by the user before or during navigation (eg, a value or state selectable by the user). For example, when entering the vehicle 200, the user may select whether to be altruistic during navigation (eg, through the user interface 170). As another example, when one or more contextual features are detected, the user may select whether or not to be altruistic in that case (eg, through user interface 170).

  In some embodiments, the altruistic behavior parameter may be set based on at least one information element identified by analyzing calendar registration information for an operator or occupant of the vehicle 200. For example, in some embodiments, the altruistic behavior parameter is such that the target vehicle can interrupt the driving lane of the vehicle 200 as long as the vehicle 200 arrives at the destination within a time frame acceptable to the operator of the vehicle 200. It can indicate that it is allowed. For example, if the operator or occupant is not in a hurry, the criteria may allow interruption by more target vehicles, but if the operator or occupant is in a hurry, the criteria may allow interruption by fewer target vehicles, or Any target vehicle interruption may not be allowed at all. Whether the operator or passenger is in a hurry is determined by the processing unit 110, for example, calendar registration information associated with the operator or passenger (eg, a calendar event indicating that the user wants to arrive at a specific location by a specific time) and vehicle Navigation data for 200 (eg, GPS and / or map data that predicts the arrival time of the location) may be identified.

  In some embodiments, the altruistic behavior parameter may be set based on the output of the random number generator function. For example, a specific output of the random number generation function may allow the vehicle 200 to allow the target vehicle 1002 to interrupt the first lane 1004, while another output of the random number generation function causes the vehicle 200 to One lane 1004 may not be allowed to be interrupted. Further alternatively or additionally, the altruistic behavior parameter may be a predetermined number that is encountered by the target vehicle indicating that one or more situational features will benefit from a course change to the first lane 1004. Can be set based on

  In some embodiments, the altruistic behavior parameter may be constant. Alternatively, the altruistic behavior parameter is at least a predetermined percentage of the vehicle 200's encounter with the target vehicle that the one or more situational characteristics indicate that it will benefit from a course change to the route the vehicle 200 is on. It can be updated to make navigation changes. For example, the predetermined percentage can be at least 10%, at least 20%, at least 30%, and the like.

  In some embodiments, the altruistic behavior parameter may specify a rule such that navigation changes are performed if certain criteria are met. For example, the predetermined altruistic behavior parameter may be that if the vehicle 200 is approaching a target vehicle in an adjacent lane and the target vehicle is traveling behind another vehicle that is moving slower than the target vehicle, The vehicle indicates that it wants to interrupt (eg, through the use of a turn indicator or through a lateral movement toward the driving lane of the vehicle 200), and the deceleration of the vehicle 200 is below a certain threshold (eg, Unless there is a vehicle behind the vehicle 200 that cannot be safely decelerated (to avoid excessive sudden braking), the vehicle 200 may decelerate to indicate that the target vehicle should be allowed to interrupt.

  In some embodiments, altruistic behavior parameters may be designed to be inconsistent, so that different navigation changes may be made, or no navigation changes will be made, even with the same contextual characteristics and criteria. For example, various altruistic actions can be taken randomly and / or periodically as a result of altruistic behavior parameters. As an example, the altruistic behavior parameter may allow only one interrupt for every n target vehicles. In some embodiments, n may be selected randomly, may vary randomly, may increase each time a contextual feature is detected, and / or may be reset to a small value when interrupts are allowed. . As another example, the altruistic behavior parameter may determine whether to allow an interruption by the target vehicle based on a comparison between a randomly generated number and a predetermined threshold.

  In some embodiments, the altruistic behavior module 804 may consider how many vehicles are driving behind the vehicle 200 and add this information to the altruistic behavior parameters. For example, the vehicle 200 provides one or more rear sensors (eg, radar sensors) for detecting the following vehicle and / or images to the system 100 for analysis and / or wireless traffic information. One or more rear cameras provided in the connection may be included. In this case, based on the number of vehicles following the vehicle 200, the altruistic behavior module 804 may evaluate the potential impact of whether the vehicle 200 allows interruption by the target vehicle. For example, if it is determined that a long row of vehicles (e.g., 5, 10 or more vehicles) is traveling behind the vehicle 200, the target is not allowed to be interrupted by the target vehicle. The vehicle is unlikely to have an opportunity to interrupt until the vehicle behind it (which can take a considerable amount of time). However, if only a small number of vehicles (for example, one or two) are traveling behind the vehicle 200, the target vehicle will be short in time (for example, its Have the opportunity to interrupt (after a small number of vehicles have passed).

  In some embodiments, altruistic behavior parameters may be derived from examples using machine learning techniques such as neural networks. For example, the neural network can be trained to identify altruistic behavior parameters based on scenarios as described above. In some embodiments, altruistic behavior parameters may be identified based on previously detected driver behavior. For example, altruistic behavior by drivers can be weighted positively, so that the neural network prioritizes altruistic behavior. As mentioned above, inconsistencies can be added through random or periodic changes as described above.

  The altruistic behavior module 806 is further described below with respect to FIGS.

  FIG. 9A is an illustration of an exemplary situation in which vehicle 200 may detect and respond to an interrupt. As illustrated, the vehicle 200 may run parallel to the target vehicle 902 on the road 900. Target vehicle 902 may be traveling in first lane 904 and vehicle 200 may be traveling in second lane 906. The first lane 904 is shown to be the left lane and the second lane 906 is shown to be the right lane, but the first and second lanes 904 and 906 are arbitrary on the road 900. Can be an adjacent lane. Furthermore, although only two lanes 904, 906 are shown on the road 900, it should be understood that more lanes may exist. Also, the term “lane” can be understood more generally to refer to a route along which the vehicle 200 and the target vehicle 902 are traveling. For example, in some embodiments, reference to a “lane” may refer to a route that is aligned with the direction of travel or route of the vehicle 200.

  The vehicle 200 may be configured to receive images of the environment surrounding the road 900 from one or more image capture devices associated with the vehicle 200, such as, for example, image capture devices 122, 124, and / or 126. . The vehicle 200 may also receive other sensor information such as GPS data, map data, radar data, or lidar data. Based on the image and / or other sensor information, the vehicle 200 may detect the target vehicle 902. For example, the vehicle 200 may identify an image of the target vehicle 902 in the plurality of images. The vehicle 200 may detect the target vehicle 902 in other ways including any of the methods described above in connection with the interrupt detection module 802.

  Further, based on the image and / or other sensor information, the vehicle 200 may identify at least one indicator that the target vehicle 902 will change from the first lane 904 to the second lane 906. For example, the vehicle 200 may be based on monocular and / or stereoscopic image analysis of the image (or based on other sensor data such as radar or lidar data) and the position of the target vehicle 902 as described in connection with FIG. 5B. And / or speed and / or the location of one or more road marks on the road 900 as described in connection with FIG. 5C. As another example, the vehicle 200 may detect that the target vehicle 902 has initiated a lane change, as described above in connection with FIG. 5F. As another example, the vehicle 200 can detect that the first lane 904 is being completed based on the map data. The vehicle 200 may identify at least one indicator in other ways including any of the methods described above in connection with the interrupt detection module 802.

  As described above, when the indicator is detected, the vehicle 200 may determine whether to take a navigation response. In order to minimize both unnecessary braking and sudden braking, the vehicle 200 determines whether there is a predetermined interrupt sensitivity change factor that affects the likelihood that the target vehicle 902 will interrupt the second lane 906 when making this determination. Can be considered. If the predetermined interrupt sensitivity changing element is not detected, a navigation response can be generated in the vehicle 200 based on the identification of the indicator and based on the first interrupt sensitivity parameter. On the other hand, if a predetermined sensitivity factor is detected, a navigation response can be generated in the vehicle 200 based on the identification of the indicator and based on the second interrupt sensitivity parameter. The second interrupt sensitivity parameter can be different (eg, more sensitive) than the first interrupt sensitivity parameter.

  Additional exemplary situations involving predetermined interrupt sensitivity modification elements are shown in FIGS.

  FIG. 9B shows an exemplary predetermined interrupt sensitivity changing element, which takes the form of an obstacle in the first lane 904. As shown, the target vehicle 902 is traveling in the first lane 904, and the vehicle 200 is traveling in the second lane 906. An obstacle may serve as a predetermined interrupt sensitivity changing element if the obstacle makes it more likely that the target vehicle 200 will attempt to interrupt than if there is no obstacle.

  As shown, the obstacle may be detected through detection of another vehicle 908 traveling in front of the target vehicle 902 in the first lane 904. Although the obstacle is shown as another vehicle 908, in some embodiments, the obstacle may take the form of a stopped vehicle, an accident, a dangerous object, a pedestrian, and the like. In one embodiment where the obstacle is another vehicle 908, the vehicle 200 may detect the obstacle by detecting that the other vehicle 908 is traveling at a lower speed than the target vehicle 902. This is because if the other vehicle 908 is traveling at a lower speed than the target vehicle 902, it is more likely that the target vehicle 902 will attempt to interrupt the second lane 906 due to the lower speed of the other vehicle 908. Because it becomes. Such an obstacle may constitute a predetermined interrupt sensitivity changing element.

  The vehicle 200 may detect an obstruction in any of the methods described above for detecting sensitivity changing elements in connection with FIG. For example, the vehicle 200 may detect the position and / or velocity of other vehicles 908 in addition to those of the target vehicle 902 based on monocular and / or stereoscopic image analysis of the images. The position and / or speed of other vehicles 908 can also be detected based on other sensor information. As another example, the vehicle 200 may detect an obstacle (eg, another vehicle 908) based on GPS data, map data, and / or traffic data from a traffic application such as, for example, Waze. As another example, the vehicle 200 may detect an obstacle through analysis of radar or lidar data.

  When a predetermined interrupt sensitivity changing element is detected (ie, when the vehicle 200 detects that there is an obstacle on the first lane 904), the vehicle 200 determines whether the second interrupt sensitivity is based on the identification of the indicator. Based on the parameters, a navigation response can be generated. The navigation response may include, for example, acceleration of the vehicle 200, deceleration of the vehicle 200, or (if possible) a lane change of the vehicle 200. The second interrupt sensitivity parameter may be more sensitive than the first interrupt sensitivity parameter when the predetermined interrupt sensitivity changing element is not detected, which means that if the predetermined interrupt sensitivity changing element is present, the target vehicle 902 This is because the possibility of trying to interrupt the second lane 906 increases.

  In some embodiments, the second interrupt sensitivity parameter may depend on the presence and behavior of vehicles around the vehicle 200. For example, the second interrupt sensitivity parameter may take into account the lateral speed and lateral position of the target vehicle 902, and the sensitivity of the second interrupt sensitivity parameter may be for each of the lateral speed and lateral position of the target vehicle 902. Can be correlated with a threshold. For example, if the lateral speed threshold is high and / or the lateral position threshold is low, the sensitivity may be lower, and the lower lateral speed threshold and / or high lateral direction may result in higher sensitivity and faster navigation response. Compared to the position threshold, the navigation response is slow.

  A higher sensitivity (ie, a higher sensitivity interrupt sensitivity parameter) may be desirable in certain situations. For example, if the target vehicle 902 is moving faster than the vehicle 200 and other vehicles 908 are moving much slower than the target vehicle 902 in the first lane 904, the target vehicle 902 decelerates to the first Stay in lane 904, decelerate and change to second lane 906 behind vehicle 200, change to another lane to the left of first lane 904 (if such a lane exists), Or it appears that either by interrupting the second lane 906, the behavior may have to be corrected. If the target vehicle 902 has to decelerate rapidly to avoid a collision with another vehicle 908, there is a higher probability of interruption to the second lane 906. Similarly, if the target vehicle 902 has to decelerate rapidly to change to the second lane 906 behind the vehicle 200, there is a higher probability of interruption into the second lane 906.

  The sensitivity of the second interrupt sensitivity parameter may further depend on the presence and behavior of other vehicles around the vehicle 200. For example, if there are no other vehicles 908 in the first lane 904 and the distance between the vehicle 200 and the nearest vehicle preceding the vehicle 200 in the second lane 906 is short, the sensitivity may be lower. As another example, other vehicles 908 in the first lane 904 are moving at approximately the same speed and / or faster than the target vehicle 902 and precede the vehicles 200 and 200 in the second lane 906. The sensitivity can be lower if the distance to the nearest vehicle is short. As another example, if the target vehicle 902 is in an overtaking lane (eg, in a country where the vehicle is on the right side of the road 900, the first lane 904 is to the left of the second lane 906), the first Another vehicle 908 in lane 904 is moving at approximately the same speed and / or higher speed as target vehicle 902 and the second vehicle 200 in second lane 906 and the nearest vehicle preceding vehicle 200 If the distance between is long, the sensitivity can be slightly higher (eg, low to moderate sensitivity). In the same situation, if the target vehicle 902 is not in the overtaking lane (eg, in a country where the vehicle is on the right side of the road 900, the first lane 904 is on the left side of the second lane 906), the sensitivity is lower. Can be.

  The sensitivity of the second interrupt sensitivity parameter may further take into account any acceleration or deceleration by the target vehicle 902. For example, if the target vehicle 902 is moving faster than the other vehicles 908 in the first lane 904 and the target vehicle 902 accelerates, the sensitivity can be high. As another example, if the target vehicle 902 is moving at a higher speed in the first lane 904 than the other vehicle 908, the deceleration of the target vehicle 902 required to avoid a collision with the other vehicle 908 is, for example, If it is greater than 0.1 g and the target vehicle 902 is not decelerating, the sensitivity may increase but not at the highest level. As another example, the target vehicle 902 is moving at a higher speed than the other vehicles 908 in the first lane 904, and the deceleration of the target vehicle 902 required to avoid a collision with the other vehicles 908 is, for example, If greater than 0.5 g and the target vehicle 902 is not decelerating, the sensitivity may be at its highest. However, in the same situation, the closest vehicle preceding the vehicle 200 in the second lane 906 is moving at a lower speed than the target vehicle 902, and the closest vehicle preceding the vehicle 200 in the second lane 906 is If the deceleration of the target vehicle 902 required to avoid a collision is greater than necessary to avoid hitting another vehicle 908, the sensitivity can be high but not the highest. However, in the same situation, if the lane on the other side of the second lane 906 is free enough to allow more gradual deceleration by the target vehicle 902, the sensitivity can be its highest, This is because the target vehicle 902 is likely to interrupt the second lane 906 to enter the lane on the other side of the second lane 906. The sensitivity levels described above and throughout the disclosure may exist on a distribution, which includes any number of sensitivity levels arranged, for example, in any desired relative arrangement along the distribution.

  As another example, the target vehicle 902 is moving at a higher speed in the first lane 904 than the other vehicle 908, and the deceleration of the target vehicle 902 necessary to avoid a collision with the other vehicle 908 is 0, for example. If the target vehicle 902 decelerates to the speed of the vehicle 200 in the second lane 906 and / or the nearest vehicle preceding the vehicle 200, greater than. As another example, the target vehicle 902 is moving at a higher speed in the first lane 904 than the other vehicles 908, and the target vehicle 902 precedes and / or precedes the vehicle 200 in the second lane 906. When decelerating to a speed slower than the nearest vehicle speed, the sensitivity can be low. As another example, the target vehicle 902 is moving at a higher speed than the other vehicles 908 in the first lane 904, and the deceleration of the target vehicle 902 necessary to avoid a collision with the other vehicles 908 is, for example, 0. If it is smaller than 2 g and the target vehicle 902 is decelerating, the sensitivity may be low.

  In some embodiments, the vehicle 200 may consider other behaviors of the target vehicle 902 when determining the second interrupt sensitivity parameter. For example, if the target vehicle 902 is in an overtaking lane (eg, in a country where the vehicle is on the right side of the road 900, the first lane 904 is to the left of the second lane 906), If another vehicle 908 is traveling slower than the target vehicle 902 and the target vehicle 902 is passing with its headlights relative to the other vehicle 908, lower sensitivity may be used, which is This is because 902 displays the intention to stay in the first lane 904. In another example, if the target vehicle 902 activates the turn signal and it indicates an intention to try to interrupt the second lane 906, the sensitivity may be high, but not the highest, which is Not only does 902 indicate an intention to interrupt the second lane 906, but it is also carefully driving or driving. For example, after activation of the turn signal, an interrupt may be detected when the target vehicle 902 shows a significant lateral movement (eg, 0.5 m toward the lane marker).

  The navigation response taken by the vehicle 200 may similarly depend on the presence and behavior of vehicles around the vehicle 200. For example, in some cases, the vehicle 200 decelerates to allow the target vehicle 902 to interrupt the second lane 906, but in other situations, the vehicle 200 accelerates and the target vehicle 902 is behind the vehicle 200. Can be changed to the second lane 906. For example, the vehicle 200 is traveling at a speed equal to or lower than the legal speed, and the vehicle 200 and the target vehicle 902 are running in parallel at substantially the same speed, or there is no vehicle preceding the vehicle 200 on the second lane 906, The nearest vehicle that precedes the vehicle 200 in the second lane 906 is at a safe distance, or there is no other vehicle 908 in front of the target vehicle 902 (or the other vehicle 908 moves at a lower speed than the target vehicle 902). Or if there is no other free lane that the target vehicle 902 can change. In some cases, the vehicle 200 is faster if necessary (eg, if the deceleration required for the target vehicle 902 to change to the second lane 906 behind the vehicle 200 is greater than 0.5 g, for example). Can accelerate to.

  FIG. 9C shows an example of a predetermined interrupt sensitivity changing element that takes the form of a geographical region. As shown in the figure, the target vehicle 902 is traveling in the first lane 904, and the vehicle 200 is traveling in the second lane 906. A geographic region may be attempted if the target vehicle 902 may attempt to interrupt due to a geographic region (eg, traffic rules and / or driving habits within that geographic region), but the target vehicle 902 is not in that geographic region. When it becomes higher than the possibility, it can serve as a predetermined interrupt sensitivity changing element. In some embodiments, a geographic region may include a country or other region that has specific legal rules and / or driving habits that apply to driving.

  As shown, the geographic area can be detected through detection of a road sign 910 from which the geographic area can be identified (eg, using monocular and / or stereoscopic image analysis of the road sign 910). Alternatively or additionally, in some embodiments, the geographic area may be GPS or map data (or other associated with the vehicle 200) in other ways, such as through one or more geographic indicators or landmarks. May be ascertained by others using a location system). By detecting the geographic region, the vehicle 200 may attempt to interrupt the target vehicle 902 due to traffic rules and / or driving habits within the geographic region, but may attempt when the target vehicle 902 is not in that geographic region. It can be specified whether it becomes higher than sex. If so, the geographic region may constitute a predetermined interrupt sensitivity changing factor.

  When a predetermined interrupt sensitivity change factor is detected (i.e., when the vehicle 200 detects that the target vehicle 902 is traveling within its geographic region), the vehicle 200 may also be based on the indicator identification and A navigation response may be generated based on the second interrupt sensitivity parameter. The navigation response may include, for example, acceleration of the vehicle 200, deceleration of the vehicle 200, or (if possible) a lane change by the vehicle 200. The second interrupt sensitivity parameter may be more sensitive than the first interrupt sensitivity parameter when the predetermined interrupt sensitivity changing element is not detected, because the presence of the interrupt sensitivity changing element causes the target vehicle 902 to This is because the possibility of trying to interrupt the lane 906 is increased.

  FIG. 9D shows an example of a predetermined interrupt sensitivity changing element that takes the form of a lane end state. As shown in the figure, the target vehicle 902 is traveling in the first lane 904, and the vehicle 200 is traveling in the second lane 906. Lane end is a predetermined interrupt sensitivity change factor when the target vehicle 902 is more likely to attempt an interrupt due to a lane end condition than the target vehicle 902 will attempt if the first lane 904 does not end. Can play a role.

  As shown, the lane end state may be detected from a road sign 912 from which the lane end state may be ascertained (eg, using a monocular and / or stereo augmentation analysis of the road sign 912) and / or a road mark. Detection may be through 914 detection (eg, using monocular and / or stereo image analysis of road sign 912). Alternatively or additionally, in some embodiments, the lane end condition may be confirmed in other ways, such as through GPS or map data (or other location system associated with the vehicle 200) and the like. By detecting the lane end condition, the vehicle 200 may determine that the likelihood that the target vehicle 902 will attempt to interrupt is higher than the likelihood that the target vehicle 902 will attempt if the first lane 904 does not end. Accordingly, the lane end state may constitute a predetermined interrupt sensitivity changing element.

  When a predetermined interrupt sensitivity change factor is detected (i.e., when the vehicle 200 detects that the first lane 904 is finished), the vehicle 200 determines whether the second interrupt sensitivity parameter is based on the identification of the indicator. Based on this, a navigation response can be generated. The navigation response may include, for example, acceleration of the vehicle 200, deceleration of the vehicle 200, or (if possible) a lane change by the vehicle 200. The second interrupt sensitivity parameter may be more sensitive than the first interrupt sensitivity parameter when the predetermined interrupt sensitivity changing element is not detected, because the presence of the interrupt sensitivity changing element causes the target vehicle 902 to This is because the possibility of trying to interrupt the lane 906 is increased.

  FIG. 9E shows an example of a predetermined interrupt sensitivity changing element that takes the form of a road split state. As shown in the figure, the target vehicle 902 is traveling in the first lane 904, and the vehicle 200 is traveling in the second lane 906. Road 900 is being divided. The road split state is a predetermined interrupt sensitivity changing factor when the possibility that the target vehicle 902 will try to interrupt due to the road split state is higher than the possibility that the target vehicle 902 will try if the first lane 904 does not end. Can play a role.

  As shown, the road segmentation state is detected from the road sign 916 from which the road segmentation state can be confirmed (eg, using monocular and / or stereoscopic image analysis of the road sign 912) and / or road marks 918a, 918b. Detection (eg, using monocular and / or stereoscopic image analysis of road sign 912). Alternatively or additionally, in some embodiments, the road split condition may be verified in other ways, such as through GPS or map data (or other location system associated with the vehicle 200) and the like. By detecting the road split state, the vehicle 200 may determine that the likelihood that the target vehicle 902 will attempt to interrupt is higher than the likelihood that the target vehicle 902 will attempt if the road 900 is not split. Therefore, the road division state can constitute a predetermined interrupt sensitivity changing element.

  When a predetermined interrupt sensitivity change factor is detected (ie, when the vehicle 200 detects that the road 900 is being split), the vehicle 200 is based on the identification of the indicator and also in the second interrupt sensitivity parameter. Based on this, a navigation response can be generated. The navigation response may include, for example, acceleration of the vehicle 200, deceleration of the vehicle 200, or (if possible) a lane change by the vehicle 200. The second interrupt sensitivity parameter may be more sensitive than the first interrupt sensitivity parameter when the predetermined interrupt sensitivity changing element is not detected, because the presence of the interrupt sensitivity changing element causes the target vehicle 902 to This is because the possibility of trying to interrupt the lane 906 is increased.

  While certain predetermined interrupt sensitivity changing elements have been described, other factors may be included, including any environmental element that may cause and / or contribute to a situation where the interruption by the target vehicle 902 is higher or lower than otherwise. It will be appreciated that certain interrupt sensitivity modification factors are also possible.

  FIG. 10 is an illustration of an exemplary situation in which the vehicle 200 may engage in altruistic behavior, according to disclosed embodiments. As shown, the vehicle 200 can run in parallel with the target vehicle 1002 on the road 1000. The vehicle 200 may be traveling in the first lane 1004 and the target vehicle 1002 may be traveling in the second lane 1006. The first lane 1004 is shown to be the left lane and the second lane 1006 is shown to be the right lane, but the first and second lanes 1004 and 1006 are arbitrary on the road 1000. It should be understood that it can be an adjacent lane. Furthermore, although only two lanes 1004, 1006 are shown on the road 1000, it can be understood that more lanes are possible. The term “lane” is used for convenience, and in some situations (eg, when the lane is not clearly labeled with a lane mark), the term “lane” is more commonly referred to as the vehicle 200 and the target. It can be understood to refer to a route along which the vehicle 1002 is traveling.

  The vehicle 200 may be configured to receive images of the environment surrounding the road 1000 from one or more image capture devices associated with the vehicle 200, such as, for example, image capture devices 122, 124, and / or 126. . The vehicle 200 may also receive other sensor information such as GPS or map data. Further, in some embodiments, the vehicle 200 may receive other sensor information, such as information obtained via a radar device or lidar device included in the system 100. Based on the image and / or other sensor information, the vehicle 200 may detect the target vehicle 1002. For example, the vehicle 200 may identify an image of the target vehicle 1002 among the plurality of images. The vehicle 200 may detect the target vehicle 1002 in other ways, including any of the methods described above in connection with the interrupt detection module 802.

  Further based on the image and / or other sensor information, the vehicle 200 may identify one or more contextual features associated with the target vehicle 1002. The contextual feature can be any characteristic that indicates, for example, that the target vehicle 1002 benefits from a lane change to a running lane of the vehicle 200. For example, as shown, the contextual feature may indicate that the target vehicle 1002 is traveling behind another vehicle 1008 that travels at a lower speed than the target vehicle 1002. In general, the contextual feature may indicate that an interrupt or other navigation response by the target vehicle 1002 may not be necessary, but an interrupt is advantageous for the target vehicle 1002.

  In some embodiments, the vehicle 200 can determine the location and / or speed of the target vehicle 1002 and / or FIG. 5C as described in connection with FIG. 5B based on monocular and / or stereoscopic image analysis of the image. As described in connection with, the location of one or more road marks on the road 900 may be detected. As another example, the vehicle 200 may detect that the target vehicle 1002 has initiated a lane change, as described above in connection with FIG. 5F. As another example, the vehicle 200 indicates that the target vehicle 1002 has started a lane change based on an analysis of other sensor information such as information acquired via a radar device or a lidar device included in the system 100. Can be detected. The vehicle 200 may identify the one or more situational features in other ways, including any of the methods as described above with respect to the altruistic behavior module 806.

  If one or more situational features are detected, the vehicle 200 may use an altruistic behavior parameter to determine whether the vehicle 200 allows the target vehicle 1002 to interrupt the first lane 1004. The current value associated can be specified. The altruistic behavior parameter may take any of the forms described above in connection with the altruistic behavior module 806. For example, the altruistic behavior parameter may be a random number generation based on at least one information element identified by analyzing calendar registration information for an operator or passenger of the vehicle 200 based on input from an operator of the vehicle 200. Based on the output of the function and / or based on a predetermined number of times the target vehicle is encountered indicating that one or more situational features can benefit from course changes to the first lane 1004 . The altruistic behavior parameter may be constant, or at least a predetermined percentage of encounters with the target vehicle that one or more situational characteristics indicate that it can benefit from a course change to the route that the vehicle 200 is on. It can be updated to make navigation changes within 200. For example, the predetermined percentage can be, for example, at least 10%, at least 20%, at least 30%, and the like.

  The vehicle 200 may determine that a change in the navigation state of the vehicle 200 may not be necessary based on one or more situational features associated with the target vehicle 1002. That is, one or more situational features may indicate that an interrupt to the first lane 1004 is advantageous to the target vehicle 1002, but such an interrupt may not be necessary (eg, by traffic rules or safety Can show that) However, the target vehicle 1002 may, in some examples, allow the target vehicle 1002 to interrupt the first lane 1004 based on altruistic behavior parameters and one or more situational features. At least one navigational change may occur.

  For example, if the altruistic behavior parameter may be set based on input from an operator of the vehicle 200, the operator may provide an input indicating that the vehicle 200 should allow an interruption by the target vehicle 1002. Based on the altruistic behavior parameter and one or more situational features, the target vehicle 1002 may change its speed to allow the target vehicle 1002 to interrupt the first lane 1004. As another example, if the altruistic behavior parameter can be set based on the output of the random number generation function, the random number generation function may not allow the vehicle 200 to allow the target vehicle 1002 to interrupt the first lane 1004. May be provided. For example, the random number generation function may output a binary output (eg, “no” or “0”, or may output a value output that does not satisfy a specific threshold (eg, when the threshold is “> −5”, Based on the altruistic behavior parameter and one or more situational features, the target vehicle 1002 may determine its speed to prevent the target vehicle 1002 from interrupting the first lane 1004. As yet another example, if the altruistic behavior parameter is set based on at least one information element identified by analyzing calendar registration information relating to the operator of the vehicle 200, the calendar registration information of the operator May indicate that the operator wants to arrive at the destination by the desired time, and altruistic behavior parameters may be As long as it arrives by the desired time, it may indicate that the target vehicle 1002 should be interrupted Based on the altruistic behavior parameter and one or more situational features, the target vehicle 1002 An interrupt to the first lane 1004 by 1002 causes the navigation change to be allowed to be performed unless the operator can no longer reach the destination at the desired time, but the first lane by the target vehicle 1002 An interrupt to 1004 is not allowed if doing so prevents the operator from reaching the destination at the desired time.

  In some cases, multiple target vehicles may benefit from an interrupt to the first lane 1004. For example, as shown, in addition to the target vehicle 1002, an additional target vehicle 1010 may be traveling behind another vehicle 1008 in the second lane 1006. One or more situational features, like target vehicle 1002, may indicate that an interrupt is beneficial to additional target vehicle 1010, although an interrupt or other navigation response by additional target vehicle 1010 may not be required. Can show. For example, as shown, the additional target vehicle 1010 may also be traveling behind another vehicle 1008, and the other vehicle 1008 may be traveling at a lower speed than the additional target vehicle 1010. In these cases, the altruistic behavior parameter causes the vehicle 200 to treat each of the target vehicle 1002 and the additional target vehicle 1010 the same (ie, allow both to interrupt the first lane 1004 or both). Not allowed) or altruistic behavior parameters may cause the vehicle 200 to treat the target vehicle 1002 and the additional target vehicle 1010 differently.

  For example, the altruistic behavior parameter may be set based on the number of encounters with a target vehicle such as the vehicle 1002, 1010, which indicates that the contextual feature will benefit from a course change to the first lane 1004. . For example, in the case of two target vehicles 1002, 1010, the altruistic behavior parameter may cause the vehicle 200 to cause only the target vehicle 1002 to interrupt the first lane 1004 and not allow additional target vehicles 1010. (On the other hand, the altruistic behavior parameter may have allowed any interruption between the target vehicle 1002 and the additional target vehicle 1010 when encountered alone).

  As another example, in these cases, the altruistic behavior parameter may be that the encounter with the target vehicle 1002, 1010 indicates that one or more situational features indicate that it can benefit from a course change to the first lane 1004. At least a predetermined percentage may be updated to make navigation changes within the vehicle 200. For example, the altruistic behavior parameter may specify that the vehicle 200 should perform navigation changes in at least 10% of encounters. Based on the altruistic behavior parameter, the vehicle 200 is subject to the target vehicle 1002 and the additional target vehicle 1010 as long as the vehicle 200 makes navigation changes in at least 10% of the encounter, with an overall percentage of encounters determined continuously. One or both may be accepted.

  Further, although the discussion of altruistic behavior parameters was discussed above in connection with the example shown in FIG. 10, the altruistic behavior parameters are considered in any of the examples discussed above in connection with FIGS. obtain.

  FIG. 11 is a flowchart illustrating an example process 1100 for vehicle interrupt detection and response according to disclosed embodiments. In some embodiments, the processing unit 110 of the system 100 may execute one or more modules 402-408 and 802-806. In other embodiments, instructions stored in one or more of modules 402-408 and 802-806 may be executed remotely from system 100 (eg, accessible via a wireless transceiver 172 over a network). Through the server). In yet another embodiment, instructions associated with one or more of modules 402-408 and 802-806 may be executed by processing unit 110 and a remote server.

  As shown, process 1100 includes receiving an image at step 1102. For example, the vehicle 200 may receive a plurality of images from at least one image capture device (eg, image capture devices 122, 124, 126) associated with the vehicle 200 via the data interface. As described above, in other embodiments, the vehicle 200 may be an alternative to analyzing the image, or in addition thereto, information obtained via a radar device or lidar device included in the system 100 instead, etc. Other sensor information may be analyzed.

  In step 1104, process 1100 includes identifying the target vehicle. For example, the vehicle 200 may identify images of a target vehicle that is traveling in a first lane that is different from the second lane in which the vehicle 200 is traveling in the plurality of images. Identifying the image of the target vehicle may include, for example, monocular or stereoscopic image analysis and / or other sensor information, as described above in connection with detection module 802. In other embodiments, as described above, identifying an image of the target vehicle was obtained via a radar or lidar device included in system 100 as an alternative to or in addition to image analysis. Analyzing other sensor information, such as information.

  At step 1106, process 1100 includes identifying an indicator that the target vehicle is changing lanes. For example, the vehicle 200 may identify at least one indicator that the target vehicle changes from a first lane to a second lane based on analysis of the plurality of images. Identifying the indicator may include, for example, monocular or stereoscopic image analysis and / or other sensor information (eg, radar or lidar data), as described above in connection with the interrupt detection module 802.

  In step 1108, the process 1100 includes determining whether a predetermined interrupt sensitivity change factor is present. For example, the vehicle 200 may determine whether there is at least one predetermined interrupt sensitivity changing element in the environment of the vehicle 200. The at least one predetermined interrupt sensitivity changing element may take any of the forms described above in connection with, for example, FIGS. The predetermined interrupt sensitivity changing element may include, for example, a lane end condition, an obstacle in the route of the target vehicle, a road split, or a geographical area. Detecting at least one predetermined interrupt sensitivity changing element may be, for example, monocular or stereoscopic image analysis and / or other sensor information (eg, radar or lidar, as described above in connection with interrupt detection module 802). Data).

  If the predetermined interrupt sensitivity change factor is not detected, the process 1100 may proceed to step 1110 and generate a first navigation response based on the indicator and the first interrupt sensitivity parameter. For example, the vehicle 200 may cause a first navigation response in the vehicle based on the identification of at least one indicator and based on the first interrupt sensitivity parameter if no interrupt sensitivity changing element is detected. The first interrupt sensitivity parameter may take any of the forms described above in connection with the interrupt response module 804. Further, as described above, in some embodiments, step 1110 instead includes generating a first navigation response based on the indicator and a value associated with the first predetermined interrupt sensitivity parameter. Thus, the first navigation response may take any of the forms described for the navigation response with respect to FIG.

  On the other hand, if at least one predetermined interrupt sensitivity change factor is detected, the process 1100 may proceed to step 1112 to generate a second navigation response based on the indicator and the second interrupt sensitivity parameter. For example, the vehicle 200 may cause a second navigation response in the vehicle based on the identification of at least one indicator and based on the second interrupt sensitivity parameter when an interrupt sensitivity change factor is detected. The second interrupt sensitivity parameter may take any of the forms described above in connection with the interrupt response module 804. The second interrupt sensitivity parameter can be different (eg, more sensitive) than the first interrupt sensitivity parameter. Further, as described above, in some embodiments, step 1112 instead includes generating a second navigation response based on the indicator and a value associated with the second predetermined interrupt sensitivity parameter. obtain. The second navigation response can take any of the forms described for the navigation response with respect to FIG.

  FIG. 12 is a flowchart illustrating an example process 1200 for navigating while taking into account altruistic behavior, according to disclosed embodiments. In some embodiments, the processing unit 110 of the system 100 may execute one or more modules 402-408 and 802-806. In other embodiments, instructions stored in one or more of modules 402-408 and 802-806 may be executed remotely from system 100 (eg, accessible via a wireless transceiver 172 over a network). Through the server). In yet another embodiment, instructions associated with one or more of modules 402-408 and 802-806 may be executed by processing unit 110 and a remote server.

  As shown, process 1200 includes receiving an image at step 1202. For example, the vehicle 200 may receive a plurality of images from at least one image capture device (eg, image capture devices 122, 124, 126) associated with the vehicle 200 via the data interface. As described above, in other embodiments, the vehicle 200 may be an alternative to analyzing the image, or in addition thereto, information obtained via a radar device or lidar device included in the system 100 instead, etc. Other sensor information may be analyzed.

  Process 1200 includes identifying a target vehicle at step 1204. For example, the vehicle 200 may identify at least one target vehicle in the environment of the vehicle 200 in the plurality of images. Identifying the target vehicle may include, for example, monocular or stereoscopic image analysis and / or other sensor information, as described above in connection with the altruistic behavior module 806.

  In step 1206, process 1200 includes identifying one or more contextual features associated with the target vehicle. For example, the vehicle 200 may identify one or more contextual features associated with the target vehicle based on the analysis of multiple images. The contextual feature can take any of the forms described above in connection with the altruistic behavior module 806 and / or FIGS. For example, the contextual feature may indicate that the target vehicle benefits from a course change to the route of the vehicle 200. As another example, the contextual feature is that the target vehicle is traveling in a lane adjacent to the lane in which the vehicle 200 is traveling, and the target vehicle is traveling at a lower speed than the target vehicle and at a lower speed than the vehicle 200. It may indicate that the vehicle is behind the vehicle. Identifying the contextual features may include, for example, monocular or stereoscopic image analysis and / or other sensor information (eg, radar or lidar data), as described above with respect to the altruistic behavior module 806.

  At step 1208, process 1200 includes identifying a current value for the altruistic behavior parameter. For example, the vehicle 200 may specify a current value associated with the altruistic behavior parameter. The altruistic behavior parameter may take any of the forms described above in connection with the altruistic behavior module 806 and / or FIG. 10, and identifying the altruistic behavior parameter may include Any of the methods described above in connection with can be performed. The value of the altruistic behavior parameter is output to the random number generation function based on at least one information element specified by analyzing the calendar registration information related to the operator of the vehicle 200 based on the input from the operator of the vehicle 200. Based on and / or based on a predetermined number of encounters with such a target vehicle that indicate that one or more contextual features will benefit from a course change to the route of the vehicle 200. Alternatively or additionally, the value of the altruistic behavior parameter may be at least a predetermined percentage of encounters with the target vehicle that the one or more contextual characteristics indicate that it will benefit from a course change to the route the vehicle 200 is on. (E.g., 10%, 20%, 30%, etc.) may be updated to cause navigation changes within the vehicle 200.

  In step 1210, the process 1200 includes causing navigation changes based on the current value of the one or more contextual features and the altruistic behavior parameter. For example, the vehicle 200 may determine that a change in the navigational state of the vehicle 200 may not be necessary based on one or more situational features associated with the target vehicle, but still relates to the altruistic behavior parameter. At least one navigational change in the vehicle 200 may occur based on the current value being obtained and based on one or more situational features associated with the target vehicle. Producing navigation changes can be done in any of the methods described above in connection with the altruistic behavior module 806 and / or FIG.

  As described above, in addition to performing image analysis of images captured from one or more front and / or rear cameras in any process or process step described in this disclosure, system 100 ( The vehicle 200) may also analyze other sensor information, such as information obtained via a radar device and / or a lidar device.

  The above description is presented for purposes of illustration. The above description is not exhaustive and is not limited to the precise forms or embodiments disclosed. Modifications and adaptations will become apparent to those skilled in the art from consideration of the specification and practice of the disclosed embodiments. Furthermore, although aspects of the disclosed embodiments are described as being stored in a memory, these aspects are not limited to other types of computer readable media, such as auxiliary storage, such as a hard disk or a CD ROM or other Those skilled in the art will appreciate that it can also be stored in the form of RAM or ROM, USB media, DVD, Blu-ray®, 4K Super HD Blu-ray, or other optical drive media.

  Computer programs based on the description and the disclosed method are within the skills of experienced developers. The various programs or program modules can be created using any technique known to those skilled in the art or can be designed in relation to existing software. For example, a program section or a program module may include .Net Framework, .Net Compact Framework (and related languages such as Visual Basic, C), Java (registered trademark), C ++, Objective-C, HTML, HTML / AJAX combination, XML Or can be designed in or with HTML containing Java applets.

Furthermore, although exemplary embodiments have been described herein, the scope of any embodiment is intended to be equivalent elements, modifications, omissions, combinations (e.g., as understood by one of ordinary skill in the art based on this disclosure). Of adaptations and / or alternatives of aspects over various embodiments. Limitations in the claims should be construed broadly based on the language utilized in the claims, and are not limited to the examples described herein or the running examples of the present application. Examples should be interpreted as non-exclusive. Further, the steps of the disclosed method may be altered in any manner, including ordering of steps and / or insertion or deletion of steps. Accordingly, it is intended that the specification and examples be considered as exemplary only, with a true scope and spirit being indicated by the following claims and their full scope of equivalents.

Claims (40)

A vehicle interrupt detection and response system for a host vehicle, comprising:
A data interface;
At least one processing device,
Receiving a plurality of images from at least one image capture device associated with the host vehicle via the data interface;
Identifying an image of a target vehicle traveling in a first lane different from a second lane in which the host vehicle is traveling in the plurality of images;
Identifying at least one indicator that the target vehicle will change from the first lane to the second lane based on the analysis of the plurality of images;
Detecting whether at least one predetermined interrupt sensitivity changing element is present in the environment of the host vehicle;
If a predetermined interrupt sensitivity change factor is not detected, a first navigation response in the host vehicle is generated based on the identification of the at least one indicator and based on a value associated with a first interrupt sensitivity parameter. And
If the at least one predetermined interrupt sensitivity change factor is detected, a second in the host vehicle based on the identification of the at least one indicator and based on a value associated with a second interrupt sensitivity parameter. Generating at least one navigation response; and at least one processing device programmed to perform
Including
The system, wherein the second interrupt sensitivity parameter is different from the first interrupt sensitivity parameter.   The predetermined interrupt sensitivity modification element includes a lane end condition, and the value associated with the second interrupt sensitivity parameter is lower than the value associated with the first interrupt sensitivity parameter. system.   The system according to claim 2, wherein the lane end state is specified based on an analysis of the plurality of images.   The system of claim 2, wherein the lane end state is identified based on available map data and an output of a location system associated with the host vehicle.   The predetermined interrupt sensitivity changing element includes an obstacle in the path of the target vehicle, and the value associated with the second interrupt sensitivity parameter is lower than the value associated with the first interrupt sensitivity parameter; The system of claim 1.   The system according to claim 5, wherein the obstacle includes a vehicle identified as moving at a lower speed than the target vehicle based on the plurality of images.   The predetermined interrupt sensitivity changing element includes a forward lane shift of the target vehicle, and the value associated with the second interrupt sensitivity parameter is lower than the value associated with the first interrupt sensitivity parameter. Item 4. The system according to Item 1.   The system of claim 7, wherein the lane shift is detected based on an analysis of the plurality of images.   8. The system of claim 7, wherein the lane shift is detected based on available map data and a location system output associated with the host vehicle.   The predetermined interrupt sensitivity changing element includes a specific geographical area where the target vehicle is located, and the value associated with the second interrupt sensitivity parameter is greater than the value associated with the first interrupt sensitivity parameter. The system of claim 1, wherein the system is low.   The system of claim 10, wherein the specific geographic region is identified based on an analysis of the plurality of images.   The system of claim 10, wherein the specific geographic region is identified based on a location system output associated with the host vehicle.   The system of claim 10, wherein the specific geographic region includes a country.   The system of claim 1, wherein the second navigation response includes at least one of deceleration of the vehicle, acceleration of the vehicle, or lane change.   The system of claim 1, wherein detecting whether at least one predetermined interrupt sensitivity changing element is present in an environment of the host vehicle is based on an analysis of the plurality of images. A host vehicle,
The car body,
At least one image capture device;
At least one processing device,
Receiving a plurality of images from the at least one image capture device;
Identifying an image of a target vehicle traveling in a first lane different from a second lane in which the host vehicle is traveling in the plurality of images;
Identifying at least one indicator that the target vehicle will change from the first lane to the second lane based on the analysis of the plurality of images;
Detecting whether at least one predetermined interrupt sensitivity changing element is present in the environment of the host vehicle;
If a predetermined interrupt sensitivity change factor is not detected, a first navigation response in the host vehicle is generated based on the identification of the at least one indicator and based on a value associated with a first interrupt sensitivity parameter. And
If the at least one predetermined interrupt sensitivity change factor is detected, a second in the host vehicle based on the identification of the at least one indicator and based on a value associated with a second interrupt sensitivity parameter. Generating at least one navigation response; and at least one processing device programmed to perform
Including
The host vehicle, wherein the second interrupt sensitivity parameter is different from the first interrupt sensitivity parameter.   The predetermined interrupt sensitivity modification element includes a lane end condition, and the value associated with the second interrupt sensitivity parameter is lower than the value associated with the first interrupt sensitivity parameter. Host vehicle.   The predetermined interrupt sensitivity changing element includes an obstacle in the path of the target vehicle, and the value associated with the second interrupt sensitivity parameter is lower than the value associated with the first interrupt sensitivity parameter; The host vehicle according to claim 16.   The predetermined interrupt sensitivity changing element includes a forward lane shift of the target vehicle, and the value associated with the second interrupt sensitivity parameter is lower than the value associated with the first interrupt sensitivity parameter. Item 15. The host vehicle according to Item 15. A method of detecting and responding to an interrupt by a target vehicle,
Receiving a plurality of images from at least one image capture device associated with the host vehicle;
Identifying an image of the target vehicle traveling in a first lane different from a second lane in which the host vehicle is traveling in the plurality of images;
Identifying at least one indicator that the target vehicle will change from the first lane to the second lane based on the analysis of the plurality of images;
Detecting whether at least one predetermined interrupt sensitivity changing element is present in the environment of the host vehicle;
If a predetermined interrupt sensitivity change factor is not detected, a first navigation response in the host vehicle is generated based on the identification of the at least one indicator and based on a value associated with a first interrupt sensitivity parameter. And
If the at least one predetermined interrupt sensitivity change factor is detected, a second in the host vehicle based on the identification of the at least one indicator and based on a value associated with a second interrupt sensitivity parameter. Generating a navigation response,
The method, wherein the second interrupt sensitivity parameter is different from the first interrupt sensitivity parameter. A navigation system for a host vehicle,
A data interface;
At least one processing device,
Receiving a plurality of images from at least one image capture device associated with the host vehicle via the data interface;
Identifying at least one target vehicle in the environment of the host vehicle based on the analysis of the plurality of images;
Identifying one or more contextual features associated with the target vehicle based on the analysis of the plurality of images;
Identifying current values associated with altruistic behavior parameters;
Based on the one or more contextual characteristics associated with the target vehicle, identifying that no change in the navigation state of the host vehicle is necessary, but with the current value associated with the altruistic behavior parameter And at least one processing device programmed to cause at least one navigation change in the host vehicle based on the one or more situational features associated with the target vehicle Including system.   The system of claim 21, wherein the one or more contextual features indicate that the target vehicle will benefit from a course change to the route of the host vehicle.   The one or more situational features indicate that the target vehicle is traveling in a lane adjacent to a lane in which the host vehicle is traveling, and the target vehicle is slower than the target vehicle. 23. The system of claim 21, wherein the system is behind a vehicle that is moving at a lower speed than the host vehicle.   The system of claim 21, wherein the value of the altruistic behavior parameter is set based on input from an operator of the host vehicle.   The system of claim 21, wherein the value of the altruistic behavior parameter is set based on at least one information element identified by analyzing calendar registration information for an operator of the host vehicle.   The system of claim 21, wherein the value of the altruistic behavior parameter is set based on an output of a random number generation function.   The value of the altruistic behavior parameter is based on a predetermined number of encounters with a target vehicle indicating that the one or more situational features will benefit from course changes to the route of the host vehicle. The system of claim 21, wherein the system is configured.   The value of the altruistic behavior parameter is at least a predetermined percentage of encounters with the target vehicle that the one or more situational characteristics indicate will benefit from a course change to the route the host vehicle is on. 23. The system of claim 21, wherein the system is updated to cause navigation changes in the host vehicle.   30. The system of claim 28, wherein the predetermined percentage is at least 10%.   30. The system of claim 28, wherein the predetermined percentage is at least 20%. A host vehicle,
The body,
At least one image capture device;
At least one processing device,
Receiving a plurality of images from the at least one image capture device;
Identifying at least one target vehicle in the environment of the host vehicle based on the analysis of the plurality of images;
Identifying one or more contextual features associated with the target vehicle based on the analysis of the plurality of images;
Identifying current values associated with altruistic behavior parameters;
Based on the one or more contextual characteristics associated with the target vehicle, identifying that no change in the navigation state of the host vehicle is necessary, but with the current value associated with the altruistic behavior parameter And at least one processing device programmed to cause at least one navigation change in the host vehicle based on the one or more situational features associated with the target vehicle Including host vehicle.   32. The host vehicle of claim 31, wherein the one or more contextual features indicate that the target vehicle will benefit from a course change to the route of the host vehicle.   The one or more situational features indicate that the target vehicle is traveling in a lane adjacent to a lane in which the host vehicle is traveling, and the target vehicle is slower than the target vehicle. 32. The host vehicle of claim 31, wherein the host vehicle is behind a vehicle moving at a lower speed than the host vehicle.   32. The host vehicle according to claim 31, wherein the value of the altruistic behavior parameter is set based on an input from an operator of the host vehicle.   32. The host vehicle of claim 31, wherein the value of the altruistic behavior parameter is set based on at least one information element identified by analyzing calendar registration information for an operator of the host vehicle.   The host vehicle according to claim 31, wherein the value of the altruistic behavior parameter is set based on an output of a random number generation function.   The value of the altruistic behavior parameter is based on a predetermined number of encounters with a target vehicle indicating that the one or more situational features will benefit from course changes to the route of the host vehicle. 32. The host vehicle according to claim 31, wherein the host vehicle is set.   The value of the altruistic behavior parameter is at least a predetermined percentage of encounters with the target vehicle that the one or more situational characteristics indicate will benefit from a course change to the route the host vehicle is on. 32. The host vehicle of claim 31, wherein the host vehicle is updated to cause navigation changes in the host vehicle.   40. The host vehicle of claim 38, wherein the predetermined percentage is at least 10%. A method for navigating a host vehicle,
Receiving a plurality of images from at least one image capture device associated with the vehicle;
Identifying at least one target vehicle in the environment of the host vehicle based on the analysis of the plurality of images;
Identifying one or more contextual features associated with the target vehicle based on the analysis of the plurality of images;
Identifying current values associated with altruistic behavior parameters;
Based on the one or more contextual characteristics associated with the target vehicle, identifying that no change in the navigation state of the host vehicle is necessary, but with the current value associated with the altruistic behavior parameter And causing at least one navigational change in the host vehicle based on the one or more situational features associated with the target vehicle.