US11978265B2 - System and method for providing lane identification on an augmented reality display - Google Patents
System and method for providing lane identification on an augmented reality display Download PDFInfo
- Publication number
- US11978265B2 US11978265B2 US17/692,488 US202217692488A US11978265B2 US 11978265 B2 US11978265 B2 US 11978265B2 US 202217692488 A US202217692488 A US 202217692488A US 11978265 B2 US11978265 B2 US 11978265B2
- Authority
- US
- United States
- Prior art keywords
- vehicle
- lane
- roadway
- virtual image
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
- 238000000034 method Methods 0.000 title claims abstract description 84
- 230000003190 augmentative effect Effects 0.000 title claims abstract description 55
- 230000004044 response Effects 0.000 claims abstract description 15
- 238000004891 communication Methods 0.000 claims description 27
- 230000037361 pathway Effects 0.000 claims description 11
- 210000003128 head Anatomy 0.000 description 12
- 230000005540 biological transmission Effects 0.000 description 7
- 238000012545 processing Methods 0.000 description 6
- 238000013500 data storage Methods 0.000 description 5
- 230000007613 environmental effect Effects 0.000 description 4
- 238000013459 approach Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000002485 combustion reaction Methods 0.000 description 1
- 230000001010 compromised effect Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000000446 fuel Substances 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 239000003550 marker Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 230000000149 penetrating effect Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000001172 regenerating effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 230000011664 signaling Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
- G06V20/582—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads of traffic signs
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/183—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a single remote source
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
- B60K2360/149—Instrument input by detecting viewing direction not otherwise provided for
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
- B60K2360/16—Type of output information
- B60K2360/166—Navigation
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
- B60K2360/16—Type of output information
- B60K2360/168—Target or limit values
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
- B60K2360/16—Type of output information
- B60K2360/177—Augmented reality
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
- B60K2360/20—Optical features of instruments
- B60K2360/31—Virtual images
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
- B60K2360/741—Instruments adapted for user detection
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
- B60K35/20—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor
- B60K35/21—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor using visual output, e.g. blinking lights or matrix displays
- B60K35/23—Head-up displays [HUD]
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
- B60K35/20—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor
- B60K35/28—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
- B60K35/65—Instruments specially adapted for specific vehicle types or users, e.g. for left- or right-hand drive
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
- G06T2207/30256—Lane; Road marking
Definitions
- the present disclosure relates to a system and method for displaying lane information on an augmented reality display.
- Some vehicles include displays to provide information to a vehicle user. However, these displays do not necessarily present lane information relating to lane usage. Providing lane usage information may help the vehicle user select a lane while driving. It is therefore desirable to develop a system and a method for displaying lane information on an augmented reality display based on real-time vehicle positioning within the current driving environment.
- the present disclosure describes a system and method that uses an augmented reality display, which may be configured as a HUD or an information cluster display.
- the augmented reality display employs conformal graphics overlaid on the roadway to show identification and/or rule information about the driving lanes.
- the spatially placed labels registered to the roadway lanes describe the purpose of each lane and/or roadway rules associated with the particular lane.
- the system activates on demand, in poor driving conditions, and when the vehicle is approaching a multiple-lane intersection.
- the presently disclosed method and system uses roadway markers, vehicle sensors, and driving head and eye position to calculate and adjust the location of the projected conformal graphics overlaid on the roadway in real-time.
- the presently disclosed method and system provides awareness in automated driving modes to increase user trust and aid in vehicle take-over.
- the method for displaying lane information on an augmented reality display includes receiving roadway data.
- the roadway data includes information about a roadway along a route of a vehicle.
- the roadway includes a plurality of lanes.
- the roadway data includes lane information about at least one of the plurality of lanes along the route of the vehicle.
- the method further includes receiving vehicle-location data.
- the vehicle-location data indicates a location of the vehicle.
- the method further includes determining that that the vehicle is approaching a road junction using the vehicle-location data and the roadway data.
- the method further includes, in response to determining that the vehicle is approaching the road junction, transmitting a command signal to a dual-focal plane augmented reality display to display at least one virtual image that is indicative of the lane information about at least one of the plurality of lanes along the route of the vehicle as the vehicle is approaching the road junction.
- the method described in this paragraph improves vehicle technology by providing a vehicle user with lane information when the vehicle approaches a road junction or when the visibility is poor.
- the method further includes identifying lane markers of the plurality of lanes.
- the method further includes determining a position of the vehicle relative to the plurality of lanes of the roadway along the route.
- the method further includes determining virtual images to be displayed on the dual-focal plane augmented reality display based on the lane information.
- the method further includes determining a location of eyes of a user of the vehicle.
- the method further includes determining a location of at least one virtual image on the dual-focal plane augmented reality display based on the location of the eyes of the user of the vehicle.
- the method further includes commanding the dual-focal plane augmented reality display to display at least one virtual image at the location previously determined.
- the virtual image includes the lane information of at least one of the plurality of lanes along the route of the vehicle as the vehicle is approaching the road junction.
- the method further includes determining whether the vehicle has passed the road junction.
- the method includes ending the method in response to determining that the vehicle has passed the road junction.
- the method further includes receiving image data.
- the image data includes information about a visibility of the roadway along the route of the vehicle.
- the method further includes determining that the visibility of a road sign along the route of the vehicle is less than a predetermined minimum-confidence threshold using the image data.
- the method further includes, in response to determining that the visibility of the road sign along the roadway of the vehicle is less than the predetermined minimum-confidence threshold, transmitting a command signal to a dual-focal plane augmented reality display to display at least one virtual image including the lane information of each of the plurality of lanes along the route of the vehicle as the vehicle is approaching the road junction.
- a method for displaying lane information on an augmented reality display includes receiving roadway data.
- the roadway data includes information about a roadway along a route of a vehicle.
- the roadway includes a plurality of lanes.
- the roadway data includes lane information about each of the plurality of lanes along the route of the vehicle.
- the method further includes receiving image data.
- the image data includes information about a visibility of the road along the route of the vehicle.
- the method further includes determining that a visibility of a road sign along the route of the vehicle is less than a predetermined minimum-confidence threshold using the image data.
- the method further includes, in response to determining that the visibility of the road sign along the road of the vehicle is less than the predetermined minimum-confidence threshold, transmitting a command signal to a dual-focal plane augmented reality display to display at least one virtual image including lane information of at least one of the plurality of lanes along the route of the vehicle as the vehicle is approaching a road junction.
- the method described in this paragraph improves vehicle technology by providing a vehicle user with lane information when the visibility is poor.
- the method further includes identifying lane markers of the plurality of lanes.
- the method further includes determining a position of the vehicle relative to the plurality of lanes of the roadway along the route.
- the method further includes determining at least one virtual image to be displayed on the dual-focal plane augmented reality display based on the lane information.
- the method further includes determining a location of eyes of a user of the vehicle.
- the method further includes determining a location of at least one virtual image on the dual-focal plane augmented reality display based on the location of the eyes of the user of the vehicle.
- a system for displaying lane information on an augmented reality display in a vehicle includes a plurality of sensors. At least one of the plurality of the sensors is configured to collect roadway data.
- the roadway data includes information about a roadway along a route of the vehicle.
- the roadway includes a plurality of lanes.
- the roadway data further includes lane information about each of the plurality of lanes along the route of the vehicle.
- the system further includes a dual-focal plane augmented reality display and a controller in communication with the plurality of sensors and the dual-focal plane augmented reality display.
- the controller is configured to receive the roadway data from at least one of the plurality of sensors and receive vehicle-location data from at least one of the plurality of sensors.
- the vehicle-location data indicates a location of the vehicle.
- the controller is configured to: determine that that the vehicle is approaching a road junction using the roadway data and the vehicle-location data; and in response to determining that the vehicle is approaching the road junction, transmit a command signal to a dual-focal plane augmented reality display to display at least one virtual image including the lane information of each of the plurality of lanes along the route of the vehicle as the vehicle is approaching the road junction.
- the plurality of sensors includes a camera.
- the controller is further configured to identify lane markers of the plurality of lanes based on an input from the camera.
- the controller is configured to determine a position of the vehicle relative to the plurality of lanes of the roadway along the route.
- the controller is further configured to determine at least one virtual image to be displayed on the dual-focal plane augmented reality display based on the lane information.
- the plurality of sensors includes a user tracker.
- the controller is further configured to determine a location of eyes of a user of the vehicle using an input from the user tracker.
- the controller is configured to determine a location of at least one virtual image on the dual-focal plane augmented reality display based on the location of the eyes of the user of the vehicle.
- FIG. 1 is a block diagram depicting an embodiment of a vehicle including a system for displaying lane information on an augmented reality display;
- FIG. 2 is a schematic front view of the dual-focal plane augmented reality display of FIG. 1 , wherein the dual-focal plane augmented reality display is configured as a head-up display (HUD) and shows lane identification information.
- HUD head-up display
- FIG. 3 is a schematic front view of the dual-focal plane augmented reality display of FIG. 1 , wherein the dual-focal plane augmented reality display is configured as a HUD and shows lane rule information.
- FIG. 4 is a schematic front view of the dual-focal plane augmented reality display of FIG. 1 , highlighting a second image plane of the dual-focal plane augmented reality display of FIG. 1 ;
- FIG. 5 is a schematic diagram of the second image plane of the dual-focal plane augmented reality display of FIG. 1 ;
- FIG. 6 is a schematic diagram of part of the system for displaying lane information on the dual-focal plane augmented reality display of FIG. 1 ;
- FIG. 7 is a flowchart of a method for displaying lane information on dual-focal plane augmented reality display of FIG. 1 .
- a vehicle 10 generally includes a chassis 12 , a body 14 , front and rear wheels 17 and may be referred to as a vehicle system.
- the vehicle 10 includes two front wheels 17 a and two rear wheels 17 b .
- the body 14 is arranged on the chassis 12 and substantially encloses components of the vehicle 10 .
- the body 14 and the chassis 12 may jointly form a frame.
- the wheels 17 are each rotationally coupled to the chassis 12 near a respective corner of the body 14 .
- the vehicle 10 includes a front axle 19 coupled to the front wheels 17 a and a rear axle 25 coupled to the rear wheels 17 b.
- the vehicle 10 may be an autonomous vehicle and a control system 98 is incorporated into the vehicle 10 .
- the control system 98 may be referred to as the system or the system for displaying lane information on a display 29 .
- the vehicle 10 is, for example, a vehicle that is automatically controlled to carry passengers from one location to another.
- the vehicle 10 is depicted in the illustrated embodiment as a pickup truck, but it should be appreciated that other vehicles including, trucks, sedans, coupes, sport utility vehicles (SUVs), recreational vehicles (RVs), etc., may also be used.
- the vehicle 10 may be a so-called a Level Two, a Level Three, Level Four, or Level Five automation system.
- a Level Four system indicates “high automation,” referring to the driving mode-specific performance by an automated driving system of aspects of the dynamic driving task, even if a human driver does not respond appropriately to a request to intervene.
- a Level Five system indicates “full automation,” referring to the full-time performance by an automated driving system of aspects of the dynamic driving task under a number of roadway and environmental conditions that can be managed by a human driver.
- the vehicle systems perform the entire dynamic driving task (DDT) within the area that it is designed to do so. The vehicle operator is only expected to be responsible for the DDT-fallback when the vehicle 10 essentially “asks” the driver to take over if something goes wrong or the vehicle is about to leave the zone where it is able to operate.
- systems provide steering, brake/acceleration support, lane centering, and adaptive cruise control. However, even if these systems are activated, the vehicle operator at the wheel must be driving and constantly supervising the automated features.
- the vehicle 10 generally includes a propulsion system 20 , a transmission system 22 , a steering system 24 , a brake system 26 , a sensor system 28 , an actuator system 30 , at least one data storage device 32 , at least one controller 34 , and a communication system 36 .
- the propulsion system 20 may, in various embodiments, include an electric machine such as a traction motor and/or a fuel cell propulsion system.
- the vehicle 10 may further include a battery (or battery pack) 21 electrically connected to the propulsion system 20 . Accordingly, the battery 21 is configured to store electrical energy and to provide electrical energy to the propulsion system 20 .
- the propulsion system 20 may include an internal combustion engine.
- the transmission system 22 is configured to transmit power from the propulsion system 20 to the vehicle wheels 17 according to selectable speed ratios.
- the transmission system 22 may include a step-ratio automatic transmission, a continuously-variable transmission, or other appropriate transmission.
- the brake system 26 is configured to provide braking torque to the vehicle wheels 17 .
- the brake system 26 may, in various embodiments, include friction brakes, brake by wire, a regenerative braking system such as an electric machine, and/or other appropriate braking systems.
- the steering system 24 influences a position of the vehicle wheels 17 and may include a steering wheel 33 . While depicted as including a steering wheel 33 for illustrative purposes, in some embodiments contemplated within the scope of the present disclosure, the steering system 24 may not include a steering wheel 33 .
- the sensor system 28 includes one or more sensors 40 (i.e., sensing devices) that sense observable conditions of the exterior environment and/or the interior environment of the vehicle 10 .
- the sensors 40 are in communication with the controller 34 and may include, but are not limited to, one or more radars, one or more light detection and ranging (lidar) sensors, one or more proximity sensors, one or more odometers, one or more ground penetrating radar (GPR) sensors, one or more steering angle sensors, one or more global positioning systems (GPS) transceivers 45 , one or more tire pressure sensors, one or more cameras 41 (e.g., optical cameras and/or infrared cameras), one or more gyroscopes, one or more accelerometers, one or more inclinometers, one or more speed sensors, one or more ultrasonic sensors, one or more inertial measurement units (IMUs) and/or other sensors.
- sensors 40 i.e., sensing devices
- the sensors 40 are in communication with the controller 34 and may include
- Each sensor 40 is configured to generate a signal that is indicative of the sensed observable conditions of the exterior environment and/or the interior environment of the vehicle 10 . Because the sensor system 28 provides data to the controller 34 , the sensor system 28 and its sensors 40 are considered sources of information (or simply sources).
- the sensor system 28 includes one or more Global Navigation Satellite System (GNSS) transceivers 45 (e.g., Global Positioning System (GPS) transceivers) configured to detect and monitor the route data (i.e., route information).
- GNSS Global Navigation Satellite System
- the GNSS transceiver 45 is considered one of the sensors 40 and is configured to communicate with a GNSS to locate the position of the vehicle 10 in the globe.
- the GNSS transceiver 45 is in electronic communication with the controller 34 .
- the actuator system 30 includes one or more actuator devices 42 that control one or more vehicle features such as, but not limited to, the propulsion system 20 , the transmission system 22 , the steering system 24 , and the brake system 26 .
- vehicle features may further include interior and/or exterior vehicle features such as, but are not limited to, doors, a trunk, and cabin features such as air, music, lighting, etc.
- the data storage device 32 stores data for use in automatically controlling the vehicle 10 .
- the data storage device 32 stores defined maps of the navigable environment.
- the defined maps may be predefined by and obtained from a remote system.
- the defined maps may be assembled by the remote system and communicated to the vehicle 10 (wirelessly and/or in a wired manner) and stored in the data storage device 32 .
- the data storage device 32 may be part of the controller 34 , separate from the controller 34 , or part of the controller 34 and part of a separate system.
- the vehicle 10 may further include one or more airbags 35 in communication with the controller 34 or another controller of the vehicle 10 .
- the airbag 35 includes an inflatable bladder and is configured to transition between a stowed configuration and a deployed configuration to cushion the effects of an external force applied to the vehicle 10 .
- the sensors 40 may include an airbag sensor, such as an IMU, configured to detect an external force and generate a signal indicative of the magnitude of such external force.
- the controller 34 is configured to command the airbag 35 to deploy based on the signal from one or more sensors 40 , such as the airbag sensor. Accordingly, the controller 34 is configured to determine when the airbag 35 has been deployed.
- the controller 34 includes at least one processor 44 and a non-transitory computer readable storage device or media 46 .
- the processor 44 may be a custom made or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor among several processors associated with the controller 34 , a semiconductor-based microprocessor (in the form of a microchip or chip set), a macroprocessor, a combination thereof, or generally a device for executing instructions.
- the computer readable storage device or media 46 may include volatile and nonvolatile storage in read-only memory (ROM), random-access memory (RAM), and keep-alive memory (KAM), for example.
- KAM is a persistent or non-volatile memory that may be used to store various operating variables while the processor 44 is powered down.
- the computer-readable storage device or media 46 may be implemented using a number of memory devices such as PROMs (programmable read-only memory), EPROMs (electrically PROM), EEPROMs (electrically erasable PROM), flash memory, or another electric, magnetic, optical, or combination memory devices capable of storing data, some of which represent executable instructions, used by the controller 34 in controlling the vehicle 10 .
- the controller 34 of the vehicle 10 may be referred to as a vehicle controller and may programmed to execute a method 100 ( FIG. 7 ) for displaying lane information on AR display as described in detail below.
- the instructions may include one or more separate programs, each of which comprises an ordered listing of executable instructions for implementing logical functions.
- the instructions when executed by the processor 44 , receive and process signals from the sensor system 28 , perform logic, calculations, methods and/or algorithms for automatically controlling the components of the vehicle 10 , and generate control signals to the actuator system 30 to automatically control the components of the vehicle 10 based on the logic, calculations, methods, and/or algorithms.
- a single controller 34 is shown in FIG. 1
- embodiments of the vehicle 10 may include a plurality of controllers 34 that communicate over a suitable communication medium or a combination of communication mediums and that cooperate to process the sensor signals, perform logic, calculations, methods, and/or algorithms, and generate control signals to automatically control features of the vehicle 10 .
- the vehicle 10 includes a user interface 23 , which may be a touchscreen in the dashboard.
- the user interface 23 may include, but is not limited to, an alarm, such as one or more speakers 27 to provide an audible sound, haptic feedback in a vehicle seat or other object, one or more displays 29 , one or more microphones 31 and/or other devices suitable to provide a notification to the vehicle user of the vehicle 10 .
- the user interface 23 is in electronic communication with the controller 34 and is configured to receive inputs by a user (e.g., a vehicle operator or a vehicle passenger).
- the user interface 23 may include a touch screen and/or buttons configured to receive inputs from a vehicle user.
- the controller 34 is configured to receive inputs from the user via the user interface 23 .
- the display 29 is configured to display information to the user (e.g., vehicle operator or passenger) and may be an augmented reality (AR) display.
- AR display means a display that presents information to users, while still allowing sight of the outside world.
- the display 29 may be configured as a heads-up display (HUD), and/or an information cluster display. Accordingly, the display 29 may be an AR HUD or an AR information cluster display.
- images are projected on a windshield 39 ( FIG. 6 ) of the vehicle 10 .
- the information cluster panel of the vehicle 10 includes a screen that presents an image of the front of the vehicle 10 captured by one or more cameras 41 (i.e., a forward image) and virtual images presented on that forward image.
- the display 29 may be a dual-focal plane AR display.
- the communication system 36 is in communication with the controller 34 and is configured to wirelessly communicate information to and from other entities 48 , such as but not limited to, other vehicles (“V2V” communication), infrastructure (“V2I” communication), remote systems at a remote call center (e.g., ON-STAR by GENERAL MOTORS) and/or personal devices.
- the communication system 36 is a wireless communication system configured to communicate via a wireless local area network (WLAN) using IEEE 802.11 standards or by using cellular data communication.
- WLAN wireless local area network
- DSRC dedicated short-range communications
- the communication system 36 may include one or more antennas and/or communication transceivers 37 for receiving and/or transmitting signals, such as cooperative sensing messages (CSMs).
- CSMs cooperative sensing messages
- the communication transceivers 37 may be considered sensors 40 .
- the communication system 36 is configured to wirelessly communicate information between the vehicle 10 and another vehicle. Further, the communication system 36 is configured to wirelessly communicate information between the vehicle 10 and infrastructure or other vehicles.
- the system 98 is configured to command the display 29 , such as an AR HUD or an AR information cluster display, to show lane information (e.g., lane identification, lane purpose, and/or lane rules) using data received from the sensors 40 in the vehicle 10 , data received from other entities 48 through the communication system 36 , and/or eye-tracking data.
- the display 29 shows the lane information in a contextual manner by augmenting the road scene with conformal graphics.
- the term “conformal graphic” means syntenic generated content (i.e., a virtual image 50 ) presented as part of the outside world. Accordingly, the display 29 is conformal display.
- the term “conformal display” means a display that has the ability to represent synthetically generated content (i.e., one or more virtual images 50 ) as part of the outside world.
- the display 29 shows virtual images 50 with lane identification information as part of the outside world.
- FIG. 2 therefore shows a view of a roadway 63 through a windshield 39 ( FIG. 6 ) in addition to the virtual images 50 or an image of the roadway 63 with the virtual images 50 .
- the term “roadway” means an open way for vehicles 10 .
- the virtual images 50 identifies the lanes 61 with the number or name of the interstates or street where lane would to. Specifically, a left lane 52 would lead to interstate 94 , whereas a central lane 54 would lead to interstate 75 . A right lane 56 would lead to an exit to main street.
- the virtual image 50 on the right lane 56 also indicates a rule for that lane (i.e., that a vehicle 10 may only exit the expressway using that far-right lane). As discussed below, these virtual images 50 are particularly helpful in situations of poor visibility.
- the term “central lane” means the lane where the vehicle 10 is currently positioned.
- the term “right lane” means the lane directly to the right of the central lane
- the term “left lane” means the lane directly to the left of the central lane. Because the roadway 64 leads to other roads (e.g., streets, interstate, and/or exits), this portion of the road is considered a road junction 65 .
- road junction means a point in the roadway 63 where two or more roads meet.
- the road junction 65 may be an intersection or an interchange. At interchanges, roads pass above or below each other, using grade separation and slip roads. At intersections, roads cross at-grade.
- the display 29 shows virtual images 50 with lane rule information as part of the outside world.
- the virtual images 50 provide the user with information about the rules for each particular lane and therefore are shaped as arrows.
- the vehicle 10 may turn left from the left lane 52 or keep straight in the left lane 52 .
- the vehicle 10 may only continue forward (i.e., it cannot turn) while traveling on the central lane 54 , and the vehicle 10 may turn right from the right lane 54 or keep straight in the right lane 56 .
- the display 29 may be a dual-focal plane AR display as mentioned above.
- the display 29 has a first image plane 58 and a second image plane 60 .
- the first image plane 58 shows the view of the outside world
- the second image plane 60 is reserved for displaying the virtual images 50 ( FIG. 2 ).
- the second image plane 60 spans multiple lanes and the virtual images 50 appear at a location farther on the roadway surface 62 relative to the first image plane 58 .
- the second image plane 60 covers the left lane 52 , the central lane 54 , and the right lane 56 .
- the second image plane 60 starts at a first predetermined distance D 1 (e.g., twenty-five meters) from the vehicle 10 and ends at a second predetermined distance D 2 (e.g., ninety meters) from the vehicle 10 .
- the second predetermined distance D 2 is greater than the first predetermined distance D 1 to help the vehicle user see the virtual images 50 ( FIG. 2 ).
- the second image plane 60 is delimited by a sloped boundary that starts at the first predetermined distance D 1 from the vehicle 10 and ends at a third predetermined distance D 3 (e.g., fifty meters) from the vehicle 10 .
- the third predetermined distance D 3 is greater than the first predetermined distance D 1 and less than the second predetermined distance D 2 to help the vehicle user see the virtual images 50 ( FIG. 2 ).
- the term “dual-focal plane AR display” means an AR display that presents images in a first image plane and a second image plane, wherein the first image plane and the second image plane are at different locations. It is desirable to configure the display 29 as a dual-focal plane AR display to facilitate manipulation of the virtual images 50 on the view of the outside word. For instance, by using a dual-focal plane AR display, the size, location, and characteristics of the virtual images 50 may be changed based on, for example, the location of the eyes of the vehicle user.
- the system 98 includes a user tracker 43 (e.g., eye tracker and/or head tracker) configured to track the position and movement of the eyes 66 or the head 69 of the vehicle user.
- the user tracker may be configured as one or more of cameras 41 of the vehicle 10 .
- the cameras 41 are considered sensors 40 of the vehicle 10 .
- the user tracker 43 is in communication with the controller 34 , which includes a system manager 68 .
- the system manager 68 receives at least a first input 70 and a second input 72 .
- the first input 70 is indicative of the position of the vehicle in space (i.e., the vehicle location in space), and the second input 72 is indicative of the vehicle user position in the vehicle 10 (e.g., the position of the eyes and/or head of the user in the vehicle 10 ).
- the first input 70 may include data such as GNSS data (e.g., GPS data), vehicle speed roadway curvature, and vehicle steering, and this data may be collected from the sensors 40 of the vehicle 10 and/or other entities 48 through the communication system 36 of the vehicle 10 .
- the second input 72 may be received from the user tracker (e.g., eye tracker and/or head tracker).
- the system manager 68 is configured to determine (e.g., compute) the type, size, shape, and color of the conformal graphics (i.e., virtual images 50 ) based on the first input 70 (i.e., the vehicle location in space), the second input 72 (e.g., the position of the eyes and/or head of the user in the vehicle 10 ), and the sensed vehicle driving environment (which may be obtained through the sensors 40 ).
- the type, size, shape, and color of the conformal graphics of the virtual image 50 may be collectively referred to as the virtual image characteristics.
- the system 98 further includes an image engine 74 , which is part of the display 29 , and may be an integrated circuit configured to generate the virtual images 50 . These generated virtual images 50 are then projected on the windshield 39 (if the display 29 is a HUD) to show the virtual images 50 on the second image plane 60 along the roadway surface 62 .
- FIG. 7 is a flowchart of the method 100 for displaying lane information on the display 29 (e.g., an AR display).
- the method 100 begins at block 102 , in which the controller 34 determines that the vehicle 10 is being driven using, for example, signals generated by the sensors 40 .
- the controller 34 may receive data from one of the sensors 40 , such as a speed sensor, to determine that the vehicle 10 is moving. Then, the method 100 proceeds to block 104 .
- the controller 34 determines the location of the vehicle 10 and the environmental driving conditions around the vehicle 10 , such as weather causing poor visibility. To do so, the controller 34 may receive at least one signal from one or more sensors 40 , such as cameras 41 , GNSS transceivers 45 (e.g., Global Positioning System (GPS) transceivers) and/or data from the internet and roadway databases. For example, the controller 34 may receive vehicle-location data and roadway data from the GNSS transceivers 45 .
- vehicle-location data means data that indicates the location of the vehicle 10 .
- the term “roadway data” means data including information about a roadway 51 ( FIG.
- the roadway 51 includes a plurality of lanes 61 ( FIG. 2 ), and the roadway data also includes lane information about at least one of the lanes 61 along the route of the vehicle 10 .
- the lane information may include lane name or number and/or rules of that particular lane 61 .
- the controller 34 may determine the location of the vehicle using, for example, the vehicle-location data received from the GNSS transceiver 45 , another sensor 40 , or another suitable source of information.
- the controller 34 determines the environmental driving conditions around the vehicle 10 .
- the controller 34 receives images captured by the cameras 41 and/or weather data from a suitable source, such as the Internet.
- the images captured by the cameras 41 may include image data.
- the image data includes one or more images and information about the visibility of road signs 53 and the roadway 63 along the route of the vehicle 10 . In FIG. 3 , the road sign 53 is painted on the roadway 63 but is hidden by snow.
- the controller 34 determines the environmental driving conditions surrounding the vehicle 10 .
- the controller 34 may use the camera 41 (i.e., a front-facing camera of the vehicle 10 ) and weather data to determine that is snowing along the route of the vehicle 10 .
- weather data means data about the weather in the immediate area surrounding the vehicle 10 .
- the controller 34 determines whether the vehicle 10 is approaching a road junction 65 using the vehicle-location data and the roadway data. Further, at block 106 , the controller 34 may additionally determine whether the vehicle 10 is approaching the road junction 65 that has at least a predetermined number of pathways (e.g., three pathways). The predetermined number of pathways may be at least three pathways. Further, the controller 34 may determine that the vehicle 10 is approaching the road junction 65 with at least the predetermined number of pathways if the vehicle 10 is located at a distance that is less than a predetermined distance threshold from the road junction 65 .
- the predetermined distance threshold may be determined by testing the vehicle 10 and may be, for example, 5 meters.
- the controller 34 may determine the distance from the vehicle 10 to the road junction based on signals from one or more sensors 40 . If the vehicle 10 is approaching a road junction 65 with less than the predetermined number of pathways, then the method 100 returns to block 104 to conserve processing resources. If the vehicle 10 is approaching a road junction 65 with at least the predetermined number of pathways, then the method 100 proceeds to block 108 .
- the controller 34 detects lane markers 67 ( FIG. 3 ) using one or more inputs from the camera(s) 41 and/or another sensor 40 .
- lane marker means a marking, such as a dashed line painted on the roadway surface 62 ( FIG. 3 ), that marks the boundaries of a lane 61 ( FIG. 3 ).
- the method 100 also includes block 105 , which is performed after block 104 .
- the controller 34 determines, using the image data (e.g., images), visibility (or legibility) of one or more road signs 53 ( FIG. 2 ) along the route of the vehicle 10 is less than a predetermined minimum-confidence threshold. If the visibility (or legibility) of the road sign 53 is less than the predetermined minimum-confidence threshold, then the method 100 continues to block 110 , which is described below.
- the controller 34 determines the vehicle lane position. In other words, the controller 34 determines the position of the vehicle 10 relative to the plurality of lanes 61 of the roadway 63 along the route of the vehicle 10 . In doing so, the controller 34 may use the vehicle-location data received from GNSS transceiver 45 , the location of the roadway markers previously detected, and/or roadway databases to determine the vehicle lane position. Stated differently, at block 110 , the controller 34 determines in which lane 61 the vehicle 10 is located. Then, the method 100 continues to block 112 .
- the controller 34 determines the specific virtual images 50 to be displayed on the display 29 (e.g., the dual-focal plane augmented reality display) based on the lane information, which is part of the roadway data previously received by the controller 34 .
- the virtual images 50 may be indicated of the lane rules, lane name, the name of the road that the lane 61 leads to or other relevant lane information.
- the controller 34 may select virtual images 50 that indicate that the lanes 61 lead to a particular street or a particular interstate. Also, for instance, the controller 34 may select virtual images 50 that indicate lane rules with arrows.
- the lane rule may be that the vehicle 10 may only continue forward (i.e., it cannot turn left or right) while traveling on the central lane 54 ( FIG. 3 ) as discussed in detail above.
- the method 100 proceeds to block 114 .
- the controller 34 determines the location of the eyes 66 and/or the head 69 of the user of the vehicle 10 using at least one input from the user tracker 43 .
- the user tracker 43 may be a camera 41 configured to track the movements of the head 69 and/or the eyes 66 of the vehicle user.
- the controller 34 uses the inputs from the user tracker 43 to continuously determine, in real time, the location of the eyes 66 and/or the head 69 of the user of the vehicle 10 .
- the method 100 then proceeds to block 116 .
- the controller 34 determines, in real time, the location, type, size, shape, and color of the virtual images 50 ( FIG. 2 ) to be displayed on the display 29 (e.g., the dual-focal plane augmented reality display) based on the location of the eyes 66 and/or the head 69 of the user of the vehicle 10 and the lane information.
- the location of the virtual images 50 in the display 29 may change as the user moves the head 69 and/or eyes 66 .
- the size of the virtual images 50 may increase as the vehicle 10 gets closer to the road junction 65 .
- the method 100 proceeds to block 118 .
- the controller 34 transmits a command signal to command the display 29 to present the virtual images 50 at the previously determined locations.
- the display 29 shows the virtual images 50 at the previously determined locations.
- the virtual images 50 include the lane information concerning one or more lanes 61 along the route of the vehicle 10 as the vehicle 10 is approaching the road junction 65 .
- the location of the virtual images 50 may change in real time based on location of the eyes 66 and/or the head 69 of the vehicle user as well as the vehicle lane position as the vehicle 10 approaches the road junction 65 .
- the method 100 proceeds to block 120 .
- the controller 34 determines whether the vehicle 10 has passed the road junction 65 . To do so, the controller 34 uses the location of the vehicle 10 , which may be obtained from the GNSS transceivers 45 (e.g., Global Positioning System (GPS) transceivers) and the roadway databases. The method 100 then proceeds to block 122 .
- GPS Global Positioning System
- the method 100 ends in response to determining that the vehicle 10 has passed the road junction 65 . Further, at block 120 , the method 100 ends if the visibility (or legibility) of the road signs 53 is equal to or greater than the predetermined minimum-confidence threshold. At this juncture, the controller 34 commands the display 29 to stop showing the virtual images 50 . In response, the display 29 stops showing the virtual images 50 .
- Embodiments of the present disclosure may be described herein in terms of functional and/or logical block components and various processing steps. It should be appreciated that such block components may be realized by a number of hardware, software, and/or firmware components configured to perform the specified functions. For example, an embodiment of the present disclosure may employ various integrated circuit components, e.g., memory elements, digital signal processing elements, logic elements, look-up tables, or the like, which may carry out a variety of functions under the control of one or more microprocessors or other control devices. In addition, those skilled in the art will appreciate that embodiments of the present disclosure may be practiced in conjunction with a number of systems, and that the systems described herein are merely exemplary embodiments of the present disclosure.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Combustion & Propulsion (AREA)
- Chemical & Material Sciences (AREA)
- Transportation (AREA)
- Mechanical Engineering (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Ophthalmology & Optometry (AREA)
- Human Computer Interaction (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- Traffic Control Systems (AREA)
Abstract
Description
Claims (19)
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/692,488 US11978265B2 (en) | 2022-03-11 | 2022-03-11 | System and method for providing lane identification on an augmented reality display |
DE102022126438.6A DE102022126438A1 (en) | 2022-03-11 | 2022-10-12 | SYSTEM AND METHOD FOR DISPLAYING LANE INFORMATION ON AN AUGMENTED REALITY DISPLAY |
CN202211302123.1A CN116766924A (en) | 2022-03-11 | 2022-10-24 | System and method for displaying lane information on an augmented reality display |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/692,488 US11978265B2 (en) | 2022-03-11 | 2022-03-11 | System and method for providing lane identification on an augmented reality display |
Publications (2)
Publication Number | Publication Date |
---|---|
US20230290156A1 US20230290156A1 (en) | 2023-09-14 |
US11978265B2 true US11978265B2 (en) | 2024-05-07 |
Family
ID=87760067
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/692,488 Active 2042-09-22 US11978265B2 (en) | 2022-03-11 | 2022-03-11 | System and method for providing lane identification on an augmented reality display |
Country Status (3)
Country | Link |
---|---|
US (1) | US11978265B2 (en) |
CN (1) | CN116766924A (en) |
DE (1) | DE102022126438A1 (en) |
Citations (46)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6360168B1 (en) * | 1999-09-14 | 2002-03-19 | Alpine Electronics, Inc. | Navigation apparatus |
US20100253492A1 (en) * | 2009-04-02 | 2010-10-07 | Gm Global Technology Operations, Inc. | Daytime pedestrian detection on full-windscreen head-up display |
US20100253943A1 (en) * | 2009-04-03 | 2010-10-07 | Yongji Fu | Method and systems for particle characteruzation using optical sensor output signal fluctuation |
US20100253539A1 (en) * | 2009-04-02 | 2010-10-07 | Gm Global Technology Operations, Inc. | Vehicle-to-vehicle communicator on full-windshield head-up display |
US20100253489A1 (en) * | 2009-04-02 | 2010-10-07 | Gm Global Technology Operations, Inc. | Distortion and perspective correction of vector projection display |
US20160252363A1 (en) * | 2013-10-11 | 2016-09-01 | Tomtom Navigation B.V. | Apparatus and methods of displaying navigation instructions |
US20160305787A1 (en) * | 2014-01-27 | 2016-10-20 | Aisin Aw Co., Ltd. | Route searching system, route searching method, and computer program |
US20160328968A1 (en) * | 2016-03-16 | 2016-11-10 | Mohamed Roshdy Elsheemy | Running red lights avoidance and virtual preemption system |
US20170103272A1 (en) * | 2015-10-09 | 2017-04-13 | Denso Corporation | Information processing device |
US20180031384A1 (en) * | 2016-07-28 | 2018-02-01 | Toyota Motor Engineering & Manufacturing North America, Inc. | Augmented road line detection and display system |
US20180068559A1 (en) * | 2016-09-02 | 2018-03-08 | Volvo Car Corporation | Method and system for enabling a vehicle to discard non-pertinent road sign information |
US20180066956A1 (en) * | 2016-09-02 | 2018-03-08 | Lg Electronics Inc. | Vehicle user interface apparatus and vehicle |
US20180174368A1 (en) * | 2016-12-20 | 2018-06-21 | Harman International Industries, Incorporated | Positioning objects in an augmented reality display |
US20180218603A1 (en) * | 2017-01-30 | 2018-08-02 | Toyota Jidosha Kabushiki Kaisha | Vehicle information display system and method for displaying vehicle information |
US10217357B1 (en) * | 2017-11-03 | 2019-02-26 | Mohamed Roshdy Elsheemy | Autonomous in-vehicle virtual traffic light system |
US20190080496A1 (en) * | 2017-09-12 | 2019-03-14 | Yazaki Corporation | Vehicle display device and display control method |
US20190180485A1 (en) * | 2017-12-12 | 2019-06-13 | Lg Electronics Inc. | Vehicle control device mounted on vehicle and method of controlling the vehicle |
US10338383B2 (en) * | 2014-12-25 | 2019-07-02 | Mitsubishi Electric Corporation | Display control device |
US20200031227A1 (en) * | 2017-03-29 | 2020-01-30 | Mitsubishi Electric Corporation | Display control apparatus and method for controlling display |
US20200064138A1 (en) * | 2017-05-11 | 2020-02-27 | Hitachi Automotive Systems, Ltd. | Vehicle Control Apparatus, Vehicle Control Method, and Vehicle Control System |
US20200086888A1 (en) * | 2018-09-17 | 2020-03-19 | GM Global Technology Operations LLC | Dynamic route information interface |
US10633003B1 (en) * | 2018-12-05 | 2020-04-28 | Here Global B.V. | Method, apparatus, and computer readable medium for verifying a safe vehicle operation via a portable device |
US20200180612A1 (en) * | 2018-12-10 | 2020-06-11 | Mobileye Vision Technologies Ltd. | Navigation in vehicle crossing scenarios |
US20200254875A1 (en) * | 2019-02-11 | 2020-08-13 | Volvo Car Corporation | Remotely controlling electronic functions of a vehicle without an integrated touchscreen |
US20200292832A1 (en) * | 2019-03-14 | 2020-09-17 | Baidu Online Network Technology (Beijing) Co., Ltd. | Method and apparatus for adjusting on-vehicle projection |
US20210104212A1 (en) * | 2018-05-29 | 2021-04-08 | Denso Corporation | Display control device, and nontransitory tangible computer-readable medium therefor |
US11022458B2 (en) * | 2019-01-04 | 2021-06-01 | Telenav, Inc. | Navigation system with roadway lane guidance mechanism and method of operation thereof |
US11067995B2 (en) * | 2017-03-20 | 2021-07-20 | Mobileye Vision Technologies Ltd. | Navigation by augmented path prediction |
US20210223058A1 (en) * | 2018-12-14 | 2021-07-22 | Denso Corporation | Display control device and non-transitory computer-readable storage medium for the same |
US11169377B1 (en) * | 2020-09-16 | 2021-11-09 | E-Lead Electronic Co., Ltd. | Multi-focal plane head-up display |
US11184079B2 (en) * | 2016-03-11 | 2021-11-23 | Akoustis, Inc. | Front end module for 5.5 GHz Wi-Fi acoustic wave resonator RF filter circuit |
US20210372803A1 (en) * | 2017-09-12 | 2021-12-02 | Tomtom International B.V. | Methods and Systems of Providing Lane Information Using a Navigation Apparatus |
US20220084458A1 (en) * | 2019-05-31 | 2022-03-17 | Denso Corporation | Display control device and non-transitory tangible computer readable storage medium |
US20220172652A1 (en) * | 2019-08-23 | 2022-06-02 | Denso Corporation | Display control device and display control program product |
US20220289228A1 (en) * | 2019-12-06 | 2022-09-15 | Denso Corporation | Hmi control device, hmi control method, and hmi control program product |
US20220314886A1 (en) * | 2021-04-05 | 2022-10-06 | Canon Kabushiki Kaisha | Display control device, display control method, moving body, and storage medium |
US20220349725A1 (en) * | 2021-04-21 | 2022-11-03 | Nvidia Corporation | High definition mapping for autonomous systems and applications |
US20220383556A1 (en) * | 2020-01-06 | 2022-12-01 | Lg Electronics Inc. | Image output device and method for controlling the same |
US20230073664A1 (en) * | 2020-05-15 | 2023-03-09 | Huawei Technologies Co., Ltd. | Multi-focal-plane image generation apparatus, head-up display apparatus, related method, and device |
US11605233B2 (en) * | 2021-06-03 | 2023-03-14 | Here Global B.V. | Apparatus and methods for determining state of visibility for a road object in real time |
US11699250B1 (en) * | 2022-07-12 | 2023-07-11 | GM Global Technology Operations LLC | System and method for low visibility driving |
US20230242137A1 (en) * | 2022-01-11 | 2023-08-03 | Toyota Jidosha Kabushiki Kaisha | Notification device and notification method |
US20230298277A1 (en) * | 2022-03-18 | 2023-09-21 | GM Global Technology Operations LLC | System and method for displaying infrastructure information on an augmented reality display |
US20230306850A1 (en) * | 2022-03-22 | 2023-09-28 | GM Global Technology Operations LLC | System and method for providing a blind reveal warning on an augmented reality display |
US20230306692A1 (en) * | 2022-03-24 | 2023-09-28 | Gm Global Technlology Operations Llc | System and method for social networking using an augmented reality display |
US20230316914A1 (en) * | 2022-04-01 | 2023-10-05 | GM Global Technology Operations LLC | System and method for providing platooning information using an augmented reality display |
-
2022
- 2022-03-11 US US17/692,488 patent/US11978265B2/en active Active
- 2022-10-12 DE DE102022126438.6A patent/DE102022126438A1/en active Pending
- 2022-10-24 CN CN202211302123.1A patent/CN116766924A/en active Pending
Patent Citations (47)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6360168B1 (en) * | 1999-09-14 | 2002-03-19 | Alpine Electronics, Inc. | Navigation apparatus |
US20100253492A1 (en) * | 2009-04-02 | 2010-10-07 | Gm Global Technology Operations, Inc. | Daytime pedestrian detection on full-windscreen head-up display |
US20100253539A1 (en) * | 2009-04-02 | 2010-10-07 | Gm Global Technology Operations, Inc. | Vehicle-to-vehicle communicator on full-windshield head-up display |
US20100253489A1 (en) * | 2009-04-02 | 2010-10-07 | Gm Global Technology Operations, Inc. | Distortion and perspective correction of vector projection display |
US20100253943A1 (en) * | 2009-04-03 | 2010-10-07 | Yongji Fu | Method and systems for particle characteruzation using optical sensor output signal fluctuation |
US20160252363A1 (en) * | 2013-10-11 | 2016-09-01 | Tomtom Navigation B.V. | Apparatus and methods of displaying navigation instructions |
US20160305787A1 (en) * | 2014-01-27 | 2016-10-20 | Aisin Aw Co., Ltd. | Route searching system, route searching method, and computer program |
US10338383B2 (en) * | 2014-12-25 | 2019-07-02 | Mitsubishi Electric Corporation | Display control device |
US20170103272A1 (en) * | 2015-10-09 | 2017-04-13 | Denso Corporation | Information processing device |
US11184079B2 (en) * | 2016-03-11 | 2021-11-23 | Akoustis, Inc. | Front end module for 5.5 GHz Wi-Fi acoustic wave resonator RF filter circuit |
US20160328968A1 (en) * | 2016-03-16 | 2016-11-10 | Mohamed Roshdy Elsheemy | Running red lights avoidance and virtual preemption system |
US20180031384A1 (en) * | 2016-07-28 | 2018-02-01 | Toyota Motor Engineering & Manufacturing North America, Inc. | Augmented road line detection and display system |
US20180068559A1 (en) * | 2016-09-02 | 2018-03-08 | Volvo Car Corporation | Method and system for enabling a vehicle to discard non-pertinent road sign information |
US20180066956A1 (en) * | 2016-09-02 | 2018-03-08 | Lg Electronics Inc. | Vehicle user interface apparatus and vehicle |
US20180174368A1 (en) * | 2016-12-20 | 2018-06-21 | Harman International Industries, Incorporated | Positioning objects in an augmented reality display |
US20180218603A1 (en) * | 2017-01-30 | 2018-08-02 | Toyota Jidosha Kabushiki Kaisha | Vehicle information display system and method for displaying vehicle information |
US11067995B2 (en) * | 2017-03-20 | 2021-07-20 | Mobileye Vision Technologies Ltd. | Navigation by augmented path prediction |
US20200031227A1 (en) * | 2017-03-29 | 2020-01-30 | Mitsubishi Electric Corporation | Display control apparatus and method for controlling display |
US20200064138A1 (en) * | 2017-05-11 | 2020-02-27 | Hitachi Automotive Systems, Ltd. | Vehicle Control Apparatus, Vehicle Control Method, and Vehicle Control System |
US20190080496A1 (en) * | 2017-09-12 | 2019-03-14 | Yazaki Corporation | Vehicle display device and display control method |
US20210372803A1 (en) * | 2017-09-12 | 2021-12-02 | Tomtom International B.V. | Methods and Systems of Providing Lane Information Using a Navigation Apparatus |
US10217357B1 (en) * | 2017-11-03 | 2019-02-26 | Mohamed Roshdy Elsheemy | Autonomous in-vehicle virtual traffic light system |
US20190180485A1 (en) * | 2017-12-12 | 2019-06-13 | Lg Electronics Inc. | Vehicle control device mounted on vehicle and method of controlling the vehicle |
US20210104212A1 (en) * | 2018-05-29 | 2021-04-08 | Denso Corporation | Display control device, and nontransitory tangible computer-readable medium therefor |
US20200086888A1 (en) * | 2018-09-17 | 2020-03-19 | GM Global Technology Operations LLC | Dynamic route information interface |
US10633003B1 (en) * | 2018-12-05 | 2020-04-28 | Here Global B.V. | Method, apparatus, and computer readable medium for verifying a safe vehicle operation via a portable device |
US20200180612A1 (en) * | 2018-12-10 | 2020-06-11 | Mobileye Vision Technologies Ltd. | Navigation in vehicle crossing scenarios |
US20210223058A1 (en) * | 2018-12-14 | 2021-07-22 | Denso Corporation | Display control device and non-transitory computer-readable storage medium for the same |
US11022458B2 (en) * | 2019-01-04 | 2021-06-01 | Telenav, Inc. | Navigation system with roadway lane guidance mechanism and method of operation thereof |
US20200254875A1 (en) * | 2019-02-11 | 2020-08-13 | Volvo Car Corporation | Remotely controlling electronic functions of a vehicle without an integrated touchscreen |
US20200292832A1 (en) * | 2019-03-14 | 2020-09-17 | Baidu Online Network Technology (Beijing) Co., Ltd. | Method and apparatus for adjusting on-vehicle projection |
US20220084458A1 (en) * | 2019-05-31 | 2022-03-17 | Denso Corporation | Display control device and non-transitory tangible computer readable storage medium |
US20220172652A1 (en) * | 2019-08-23 | 2022-06-02 | Denso Corporation | Display control device and display control program product |
US20220289228A1 (en) * | 2019-12-06 | 2022-09-15 | Denso Corporation | Hmi control device, hmi control method, and hmi control program product |
US20220383556A1 (en) * | 2020-01-06 | 2022-12-01 | Lg Electronics Inc. | Image output device and method for controlling the same |
US20230073664A1 (en) * | 2020-05-15 | 2023-03-09 | Huawei Technologies Co., Ltd. | Multi-focal-plane image generation apparatus, head-up display apparatus, related method, and device |
US11169377B1 (en) * | 2020-09-16 | 2021-11-09 | E-Lead Electronic Co., Ltd. | Multi-focal plane head-up display |
US20220314886A1 (en) * | 2021-04-05 | 2022-10-06 | Canon Kabushiki Kaisha | Display control device, display control method, moving body, and storage medium |
US20220349725A1 (en) * | 2021-04-21 | 2022-11-03 | Nvidia Corporation | High definition mapping for autonomous systems and applications |
US11605233B2 (en) * | 2021-06-03 | 2023-03-14 | Here Global B.V. | Apparatus and methods for determining state of visibility for a road object in real time |
US20230242137A1 (en) * | 2022-01-11 | 2023-08-03 | Toyota Jidosha Kabushiki Kaisha | Notification device and notification method |
US20230298277A1 (en) * | 2022-03-18 | 2023-09-21 | GM Global Technology Operations LLC | System and method for displaying infrastructure information on an augmented reality display |
US20230306850A1 (en) * | 2022-03-22 | 2023-09-28 | GM Global Technology Operations LLC | System and method for providing a blind reveal warning on an augmented reality display |
US20230306692A1 (en) * | 2022-03-24 | 2023-09-28 | Gm Global Technlology Operations Llc | System and method for social networking using an augmented reality display |
US11798240B2 (en) * | 2022-03-24 | 2023-10-24 | GM Global Technology Operations LLC | System and method for social networking using an augmented reality display |
US20230316914A1 (en) * | 2022-04-01 | 2023-10-05 | GM Global Technology Operations LLC | System and method for providing platooning information using an augmented reality display |
US11699250B1 (en) * | 2022-07-12 | 2023-07-11 | GM Global Technology Operations LLC | System and method for low visibility driving |
Also Published As
Publication number | Publication date |
---|---|
CN116766924A (en) | 2023-09-19 |
US20230290156A1 (en) | 2023-09-14 |
DE102022126438A1 (en) | 2023-09-14 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10293748B2 (en) | Information presentation system | |
US10438390B2 (en) | Vehicle control device mounted on vehicle and method of controlling the vehicle | |
KR101994698B1 (en) | User interface appartus for vehicle and vehicle | |
KR102275507B1 (en) | Vehicle control device mounted on vehicle and method for controlling the vehicle | |
US11554667B2 (en) | Display device provided in vehicle and control method of display device | |
US11645919B2 (en) | In-vehicle vehicle control device and vehicle control method | |
US20200070827A1 (en) | Autonomous vehicle and operating method for autonomous vehicle | |
US20190276044A1 (en) | User interface apparatus for vehicle and vehicle including the same | |
US11279232B2 (en) | Vehicle control device mounted on vehicle and method for controlling the vehicle | |
US11699250B1 (en) | System and method for low visibility driving | |
KR102480702B1 (en) | Dashboard display and vehicle comprising the same | |
KR102611337B1 (en) | Vehicle AR display device and method of operation thereof | |
US20220388395A1 (en) | Vehicle display device and control method thereof | |
US20230398868A1 (en) | Ar display device for vehicle and method for operating same | |
US11935200B2 (en) | System and method for displaying infrastructure information on an augmented reality display | |
US11854403B2 (en) | System and method for providing a blind reveal warning on an augmented reality display | |
US20230316914A1 (en) | System and method for providing platooning information using an augmented reality display | |
US20230400321A1 (en) | Ar display device for vehicle and method for operating same | |
US11798240B2 (en) | System and method for social networking using an augmented reality display | |
US11557234B1 (en) | Windshield zone dimming and augmented reality head up display | |
US20230343210A1 (en) | Method and system for validating autonomous vehicle performance using nearby traffic patterns | |
US11978265B2 (en) | System and method for providing lane identification on an augmented reality display | |
US20230400320A1 (en) | Ar display device for vehicle and method for operating same | |
WO2023002797A1 (en) | Control device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
AS | Assignment |
Owner name: GM GLOBAL TECHNOLOGY OPERATIONS LLC, MICHIGAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SZCZERBA, JOSEPH F.;WEISS, JOHN P.;CHANG, KAI-HAN;AND OTHERS;SIGNING DATES FROM 20220309 TO 20220310;REEL/FRAME:059256/0911 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE AFTER FINAL ACTION FORWARDED TO EXAMINER |
|
ZAAB | Notice of allowance mailed |
Free format text: ORIGINAL CODE: MN/=. |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |