US20220134953A1 - Panoramic virtual environment for in-vehicle entertainment - Google Patents
Panoramic virtual environment for in-vehicle entertainment Download PDFInfo
- Publication number
- US20220134953A1 US20220134953A1 US17/083,495 US202017083495A US2022134953A1 US 20220134953 A1 US20220134953 A1 US 20220134953A1 US 202017083495 A US202017083495 A US 202017083495A US 2022134953 A1 US2022134953 A1 US 2022134953A1
- Authority
- US
- United States
- Prior art keywords
- vehicle
- entertainment
- module
- displays
- panoramic video
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R1/00—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
- B60R1/02—Rear-view mirror arrangements
- B60R1/08—Rear-view mirror arrangements involving special optical features, e.g. avoiding blind spots, e.g. convex mirrors; Side-by-side associations of rear-view and other mirrors
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R1/00—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R1/00—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
- B60R1/20—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
- B60R1/22—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/10—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used
- B60R2300/105—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used using multiple cameras
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/20—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of display used
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/30—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/30—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing
- B60R2300/303—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing using joined images, e.g. multiple camera images
Definitions
- the present disclosure relates to in-vehicle entertainment and more particularly to panoramic in-vehicle entertainment.
- Vehicles include one or more torque producing devices, such as an internal combustion engine and/or an electric motor.
- a passenger of a vehicle rides within a passenger cabin (or passenger compartment) of the vehicle.
- Vehicles may include one or more different type of sensors that sense vehicle surroundings.
- a sensor that senses vehicle surroundings is a camera configured to capture images of the vehicle surroundings. Examples of such cameras include forward facing cameras, rear facing cameras, and side facing cameras.
- Another example of a sensor that senses vehicle surroundings includes a radar sensor configured to capture information regarding vehicle surroundings.
- Other examples of sensors that sense vehicle surroundings include sonar sensors and light detection and ranging (LIDAR) sensors configured to capture information regarding vehicle surroundings.
- LIDAR light detection and ranging
- an entertainment system of a vehicle includes: a plurality of cameras configured to capture images around an exterior of the vehicle; one or more displays; and an entertainment module configured to: based on the images, generate panoramic video; and display the panoramic video within the vehicle via the one or more displays.
- the plurality of cameras include: at least one camera configured to capture images in front of the vehicle; at least one camera configured to capture images to the right of the vehicle; at least one camera configured to capture images to the left of the vehicle; and at least one camera configured to capture images behind the vehicle.
- the entertainment module is configured to perform image stabilization on the images.
- the entertainment module is configured to selectively correct ones of the images for shaking.
- the entertainment module is configured to stitch ones of the images together to create the panoramic video.
- the one or more displays include one or more heads up displays (HUDs) configured to project light onto a windshield of the vehicle and one or more displays configured to display images.
- HUDs heads up displays
- the one or more displays include at least one of an augmented reality (AR) headset and a virtual reality (VR) headset within the vehicle.
- AR augmented reality
- VR virtual reality
- the entertainment module is configured to perceive an object located outside of the vehicle and animate an item within the panoramic video based on the location.
- the entertainment module is configured to locate the item within the panoramic video based on a location of the object relative to the vehicle.
- the entertainment module is configured to animate the item within the panoramic video based on a type of the object located outside of the vehicle.
- a global positioning system (GPS) module is configured to determine a present location of the vehicle, where the entertainment module is configured to generate the panoramic video further based on the present location of the vehicle.
- GPS global positioning system
- the entertainment module is configured to display a visual identifier of an object located in front of the vehicle via at least one of the one or more displays.
- the entertainment module is configured to selectively output audio information regarding the object via one or more speakers of the vehicle.
- the entertainment module is further configured to display a visual advertisement via at least one of the one or more displays.
- the entertainment module is further configured to display a coupon via at least one of the one or more displays.
- the entertainment module is configured to selectively wirelessly transmit the coupon to a computing device of a passenger of the vehicle.
- the entertainment module is configured to selectively execute a game application stored in memory of the vehicle.
- the entertainment module is configured to generate at least one of a visual and an audible prompt for user input indicative of a rating for an entity in response to a determination that the vehicle stopped at the entity.
- an entertainment method for a vehicle includes: capturing images around an exterior of the vehicle by a plurality of cameras; based on the images, generating panoramic video; and displaying the panoramic video within the vehicle via one or more displays.
- FIG. 1 is a functional block diagram of an example vehicle system
- FIG. 2 is a functional block diagram of a vehicle including various external cameras and sensors;
- FIG. 3 is a functional block diagram of an example entertainment system of a vehicle
- FIGS. 4 and 5 are functional block diagrams of examples of the entertainment system
- FIGS. 6 and 7 are flowcharts depicting example methods of providing in-vehicle entertainment
- FIG. 8 is a flowchart depicting a portion of the example of FIG. 7 ;
- FIG. 9 includes an example illustration of information displayed in association with businesses and points of interest.
- FIG. 10 includes an example illustration of an advertisement and a coupon displayed.
- a vehicle may include an infotainment system that allows one or more users to perform infotainment tasks in vehicle via a display (e.g., a touchscreen display) and/or one or more speakers.
- a display e.g., a touchscreen display
- speakers e.g., a speaker
- the present application involves an entertainment system of a vehicle that provides panoramic and/or 360 degree views within the vehicle.
- the entertainment system may provide a panoramic view based on vehicle surroundings on one or more displays within the vehicle by stitching together images from in front of the vehicle, beside the vehicle, and behind the vehicle.
- the entertainment system may provide a 360 degree virtual environment of a game via a headset (e.g., augmented reality or virtual reality) within the vehicle.
- the entertainment system may generate objects within the virtual environment of the game based on objects in the real world around the vehicle.
- the entertainment system may also provide entertainment features, such as allowing a passenger to play games within the vehicle, outputting (visibly and/or audibly) local information to the passenger, providing advertisements to the user, and providing coupons to the passenger.
- entertainment features such as allowing a passenger to play games within the vehicle, outputting (visibly and/or audibly) local information to the passenger, providing advertisements to the user, and providing coupons to the passenger. The above may increase user experience within the vehicle.
- FIG. 1 a functional block diagram of an example vehicle system is presented. While a vehicle system for a hybrid vehicle is shown and will be described, the present application is also applicable to non-hybrid vehicles, electric vehicles, fuel cell vehicles, and other types of vehicles. The present application is applicable to autonomous vehicles, semi-autonomous vehicles, non-autonomous vehicles, shared vehicles, non-shared vehicles, and other types of vehicles.
- An engine 102 may combust an air/fuel mixture to generate drive torque.
- An engine control module (ECM) 106 controls the engine 102 .
- the ECM 106 may control actuation of engine actuators, such as a throttle valve, one or more spark plugs, one or more fuel injectors, valve actuators, camshaft phasers, an exhaust gas recirculation (EGR) valve, one or more boost devices, and other suitable engine actuators.
- engine actuators such as a throttle valve, one or more spark plugs, one or more fuel injectors, valve actuators, camshaft phasers, an exhaust gas recirculation (EGR) valve, one or more boost devices, and other suitable engine actuators.
- EGR exhaust gas recirculation
- boost devices e.g., boost devices
- the engine 102 may output torque to a transmission 110 .
- a transmission control module (TCM) 114 controls operation of the transmission 110 .
- the TCM 114 may control gear selection within the transmission 110 and one or more torque transfer devices (e.g., a torque converter, one or more clutches, etc.).
- the vehicle system may include one or more electric motors.
- an electric motor 118 may be implemented within the transmission 110 as shown in the example of FIG. 1 .
- An electric motor can act as either a generator or as a motor at a given time. When acting as a generator, an electric motor converts mechanical energy into electrical energy. The electrical energy can be, for example, used to charge a battery 126 via a power control device (PCD) 130 . When acting as a motor, an electric motor generates torque that may be used, for example, to supplement or replace torque output by the engine 102 . While the example of one electric motor is provided, the vehicle may include zero or more than one electric motor.
- a power inverter module (PIM) 134 may control the electric motor 118 and the PCD 130 .
- the PCD 130 applies power from the battery 126 to the electric motor 118 based on signals from the PIM 134 , and the PCD 130 provides power output by the electric motor 118 , for example, to the battery 126 .
- the PIM 134 may include, for example, an inverter.
- a steering control module 140 controls steering/turning of wheels of the vehicle, for example, based on driver turning of a steering wheel within the vehicle and/or steering commands from one or more vehicle control modules.
- a steering wheel angle sensor (SWA) monitors rotational position of the steering wheel and generates a SWA 142 based on the position of the steering wheel.
- the steering control module 140 may control vehicle steering via an EPS motor 144 based on the SWA 142 .
- the vehicle may include another type of steering system.
- An electronic brake control module (EBCM) 150 may selectively control brakes 154 of the vehicle.
- a horn module 156 may apply power to a horn 158 when a horn input 160 is in a first state.
- the horn 158 outputs sound when power is applied to the horn 158 .
- the horn module 156 may not apply power to the horn 158 when the horn input 160 is in a second state.
- the horn input 160 may transition from the second state to the first state, for example, in response to user application of at least a predetermined force is applied to a horn input device (e.g., located on the steering wheel).
- the horn input device may apply a bias force such that the horn input 160 is in the second state when less than the predetermined force is applied to the horn input device.
- Modules of the vehicle may share parameters via a controller area network (CAN) 162 .
- the CAN 162 may also be referred to as a car area network.
- the CAN 162 may include one or more data buses.
- Various parameters may be made available by a given module to other modules via the CAN 162 .
- the driver inputs may include, for example, an accelerator pedal position (APP) 166 which may be provided to the ECM 106 .
- a brake pedal position (BPP) 170 may be provided to the EBCM 150 .
- a position 174 of a park, reverse, neutral, drive lever (PRNDL) may be provided to the TCM 114 .
- An ignition state 178 may be provided to a body control module (BCM) 180 .
- the ignition state 178 may be input by a driver via an ignition key, button, or switch. At a given time, the ignition state 178 may be one of off, accessory, run, or crank.
- the vehicle system may also include an entertainment module 182 .
- the entertainment module 182 may be connected to, be a part of, or include an infotainment module of the vehicle.
- the entertainment module 182 selectively executes games and outputs gaming feedback via one or more output devices 184 .
- the output devices 184 may include, for example, one or more displays, one or more sets of virtual reality (VR) goggles, one or more sets of augmented reality (AR) goggles, one or more other suitable types of video output devices, one or more speakers, one or more haptic devices, and/or one or more other suitable types of output devices.
- VR virtual reality
- AR augmented reality
- the entertainment module 182 outputs gaming video via the one or more displays, one or more sets of VR goggles, and/or one or more sets of AR goggles.
- the entertainment module 182 outputs gaming audio via the one or more speakers.
- the entertainment module 182 may also output other gaming feedback via one or more haptic devices.
- haptic devices may be included with one or more seats, in the steering wheel, in one or more seat belts, etc.
- Examples of displays may include, for example, one or more displays (e.g., on a front console) of the vehicle, a head up display (HUD) that displays information via a substrate (e.g., windshield), one or more displays that drop downwardly or extend upwardly to form panoramic views, and/or one or more other suitable displays.
- HUD head up display
- one or more HUDs may project onto the windshield to use the windshield as a panoramic AR display.
- the entertainment module 182 controls play of one or more games based on user input received via one or more input devices 185 , such as one or more gaming controllers, one or more joysticks, etc. Under some circumstances, such as when the vehicle is parked, the entertainment module 182 may control gameplay based on input from vehicle components (e.g., the steering wheel, brake and accelerator pedals, horn, etc.).
- vehicle components e.g., the steering wheel, brake and accelerator pedals, horn, etc.
- the entertainment module 182 may additionally or alternatively audibly and/or visibly output other information, such as personalized advertisements, coupons, information about points of interest in front of the vehicle, etc.
- the vehicle may include a plurality of external sensors and cameras, generally illustrated in FIG. 1 by 186 .
- One or more actions may be taken based on input from the external sensors and cameras 186 .
- the infotainment module may display video, various views, and/or alerts on a display via input from the external sensors and cameras 186 during driving.
- a perception module perceives objects around the vehicle and locations of the objects relative to the vehicle.
- the ECM 106 may adjust torque output of the engine 102 based on input from the perception module.
- the PIM 134 may control power flow to and/or from the electric motor 118 based on input from the perception module.
- the EBCM 150 may adjust braking based on input from the perception module.
- the steering control module 140 may adjust steering based on input from the perception module.
- the entertainment module 182 may adjust the virtual environment of a game based on objects identified by the perception module, as discussed further below.
- the vehicle may include one or more additional control modules that are not shown, such as a chassis control module, a battery pack control module, etc.
- the vehicle may omit one or more of the control modules shown and discussed.
- the external sensors and cameras 186 include various cameras positioned to capture images and video outside of (external to) the vehicle and various types of sensors measuring parameters outside of (external to the vehicle).
- a forward facing camera 204 captures images and video of images within a predetermined field of view (FOV) 206 in front of the vehicle.
- FOV field of view
- a front camera 208 may also capture images and video within a predetermined FOV 210 in front of the vehicle.
- the front camera 208 may capture images and video within a predetermined distance of the front of the vehicle and may be located at the front of the vehicle (e.g., in a front fascia, grille, or bumper).
- the forward facing camera 204 may be located more rearward, however, such as with a rear view mirror at a windshield of the vehicle.
- the forward facing camera 204 may not be able to capture images and video of items within all of or at least a portion of the predetermined FOV of the front camera 208 and may capture images and video that is greater than the predetermined distance of the front of the vehicle. In various implementations, only one of the forward facing camera 204 and the front camera 208 may be included.
- a rear camera 212 captures images and video within a predetermined FOV 214 behind the vehicle.
- the rear camera 212 may capture images and video within a predetermined distance behind vehicle and may be located at the rear of the vehicle, such as near a rear license plate.
- a right camera 216 captures images and video within a predetermined FOV 218 to the right of the vehicle.
- the right camera 216 may capture images and video within a predetermined distance to the right of the vehicle and may be located, for example, under a right side rear view mirror.
- the right side rear view mirror may be omitted, and the right camera 216 may be located near where the right side rear view mirror would normally be located.
- a left camera 220 captures images and video within a predetermined FOV 222 to the left of the vehicle.
- the left camera 220 may capture images and video within a predetermined distance to the left of the vehicle and may be located, for example, under a left side rear view mirror.
- the left side rear view mirror may be omitted, and the left camera 220 may be located near where the left side rear view mirror would normally be located. While the example FOVs are shown for illustrative purposes, the FOVs may overlap, for example, for more accurate and/or inclusive stitching.
- the external sensors and cameras 186 may additionally or alternatively include various other types of sensors, such as ultrasonic sensors, radar sensors, etc.
- the vehicle may include one or more forward facing ultrasonic sensors, such as forward facing ultrasonic sensors 226 and 230 , one or more rearward facing ultrasonic sensors, such as rearward facing ultrasonic sensors 234 and 238 .
- the vehicle may also include one or more right side ultrasonic sensors, such as right side ultrasonic sensor 242 , and one or more left side ultrasonic sensors, such as left side ultrasonic sensor 246 .
- the locations of the cameras and ultrasonic sensors are provided as examples only and different locations could be used. Ultrasonic sensors output ultrasonic signals around the vehicle.
- the external sensors and cameras 186 may additionally or alternatively include one or more other types of sensors, such as one or more sonar sensors, one or more radar sensors, and/or one or more light detection and ranging (LIDAR) sensors.
- sensors such as one or more sonar sensors, one or more radar sensors, and/or one or more light detection and ranging (LIDAR) sensors.
- LIDAR light detection and ranging
- FIG. 3 is a functional block diagram of an example entertainment system of a vehicle.
- the entertainment module 182 includes a processor module 304 including one or more processors.
- the processor module 304 executes one or more applications 308 , such as games and other types of applications stored in memory. Each of the games is an application executable by the processor module 304 .
- the processor module 304 During execution of a game, the processor module 304 generates a panoramic virtual environment for the game based on data from the external cameras and sensors 186 . In this way, the panoramic virtual environment correlates with the real world surrounding the vehicle.
- the processor module 304 displays the virtual environment in a panoramic format or 360 degree format via one or more of the output devices 184 .
- the processor module 304 may display the virtual environment of a game in panoramic format via one or more displays that together provide panoramic views in a panoramic format (e.g., having an aspect ratio of at least 3:1 width to height or a greater width).
- the processor module 304 may display the virtual environment of a game in panoramic format using one or more HUDs that project panoramic video onto the windshield to use the windshield as a panoramic AR display.
- the processor module 304 may display a 360 degree view of the virtual environment using at least one of an AR headset in the vehicle and a VR headset in the vehicle.
- FIG. 4 is a functional block diagram for an example of the entertainment system.
- the entertainment module 182 receives images and data from the external sensors and cameras 186 from in front 404 of the vehicle, left and right sides 408 and 412 of the vehicle, and behind 416 the vehicle.
- the entertainment module 182 generates 360 degree video based on the received imaged and data and displays the video via one or more AR and/or VR headsets 420 .
- the headset 420 displays the virtual environment in a right direction 432 .
- the headset 420 displays the virtual environment in a backwards direction 436 .
- FIG. 5 is a functional block diagram for an example of the entertainment system.
- the entertainment module 182 generates panoramic video 504 (having an aspect ratio of 3:1 width to height or a greater width) based on the received imaged and data and displays the video via one or more display devices.
- the panoramic video 504 may be displayed using a left display 508 , a middle display 512 , and a right display 516 .
- the panoramic video 504 may be displayed on the windshield of the vehicle via one or more HUDs.
- the entertainment module 182 generates the panoramic or 360 degree virtual environment based on a baseline virtual environment 440 for the game.
- the entertainment module 182 adjusts the baseline virtual environment 440 based on data from the external cameras and sensors 186 to generate the panoramic or 360 degree virtual environment.
- the entertainment module 182 performs image stabilizing 444 on images received to, for example, account for vibration of the vehicle.
- the entertainment module 182 may additionally perform distortion correction 448 , for example, to reduce distortion attributable to characteristics (e.g., distortion) of the cameras of the external cameras and sensors 186 .
- the entertainment module 182 stiches 452 the images together for the panoramic or 360 degree virtual environment. For example, the entertainment module 182 stiches the left side of images from in front of the vehicle with the right side of images taken to the left of the vehicle. The entertainment module stiches the right side of images from in front of the vehicle to the left side of images taken to the right of the vehicle.
- the entertainment module 182 Based on the data received from the external sensors and cameras 186 , the entertainment module 182 perceives 456 objects (e.g., vehicles, buildings, animals, humans, etc.) located in the real world around the vehicle and locations of the objects relative to the vehicle.
- the entertainment module 182 may create objects in the virtual environment based on the types of objects perceived in the real world and the locations of the objects in the real world using animation 460 .
- FIG. 6 is a flowchart depicting an example method of providing in-vehicle entertainment.
- images from cameras 604 of the external sensors and cameras 186 are obtained.
- the images include images in front of the vehicle, to the right of the vehicle, to the left of the vehicle, and behind the vehicle.
- the entertainment module 182 corrects the images for any distortion in the images using a distortion correction algorithm.
- the entertainment module 182 enters the images into image buffers. The entertainment module 182 may determine whether one or more pixels are outside of a frame border of the images.
- the entertainment module 620 performs pixel movement tracking on the images to track motion of objects in the pixels from image to image.
- the entertainment module 182 determines movement trajectory based on the images.
- the entertainment module 628 performs extrapolation on the images.
- the entertainment module determines whether a real pixel location of an image is out of trajectory given the movement trajectory (from 624 ). If 632 is true, control continues with 636 and 640 . If 632 is false, control continues with 644 . At 644 , no image shaking correction is applied to the images, and control continues with 648 .
- the entertainment module 182 provides smoothing of the images to correct the images for shaking. The smoothing of 636 produces images corrected for shaking at 640 , and control continues with 648 . Shaking correction may be performed, for example, based on target tracking and trajectory determination.
- the entertainment module 182 stiches the images together to create a panoramic or 360 degree view.
- the entertainment module 182 may also include features and/or contours at image corners obtained at 652 .
- the entertainment module 182 may also include features and/or contours of interest obtained at 656 .
- the entertainment module 182 may also include information from matching based on points and/or contours of interest at 660 .
- the entertainment module 182 may obtain features and/or contours near corners of the images using, for example, a scale invariant feature transform (SIFT) algorithm and/or a speeded up robust features (SURF) algorithm.
- SIFT scale invariant feature transform
- SURF speeded up robust features
- the entertainment module 182 may determine the points and/or contours of interest in the images at 656 using a point of interest identification algorithm and/or a contour of interest algorithm.
- the entertainment module 182 may perform matching based on points and/or contours of interest at 660 using, for example, an affine transformation algorithm, cross-correlation algorithm, or another suitable type of algorithm.
- the panoramic or 360 degree view of the real world around the vehicle can be displayed by the entertainment module 182 at 662 via the one or more displays and/or the one or more headsets (e.g., AR and/or VR headsets). Alternatively, control may continue with 664 .
- the entertainment module 182 at 662 via the one or more displays and/or the one or more headsets (e.g., AR and/or VR headsets).
- control may continue with 664 .
- the entertainment module 182 performs perception at 664 . More specifically, the entertainment module 182 perceives objects in the images or the stitched together image.
- the entertainment module 182 may determine a type of each object and a location of each object relative to the vehicle. Examples of types of objects include, for example, vehicles, pedestrians, cyclists, animals, buildings, etc.
- the entertainment module 182 may determine the types of the objects using an object detection algorithm.
- the entertainment module 182 may determine the locations of the objects, for example, based on the FOV of the camera, the location of the object in the pixels of an image, and dimensions (e.g., pixelwise) of the object in the image.
- actors for the game e.g., vehicles, pedestrians, animals, etc.
- Structures e.g., buildings, roads, mountains, etc.
- Unrecognizable objects are also identified at 676 .
- the entertainment module 182 obtains the virtual environment 440 for the game. For example, the entertainment module 182 may determine a present location for a player of the game in the virtual environment 440 based on a (present) location 684 of the vehicle on a map 686 of the world or a predetermined area around the present location of the vehicle.
- the location 684 e.g., latitude, longitude, elevation, heading
- the entertainment module 182 may use the virtual environment as a background at 694 .
- the entertainment module 182 animates and locates the perceived objects within the virtual environment 440 .
- the entertainment module 182 may set each different type of object detected to a predetermined type of object in the virtual environment for the game. For example, the entertainment module 182 may animate pedestrians as animals in the virtual environment, vehicles as opponents within the game, and unrecognizable objects as exploratory items within the virtual environment.
- the entertainment module 182 locates the animated objects within the virtual environment based on the location of the associated objects in the real world relative to the vehicle.
- the panoramic or 360 degree view of the virtual environment including added objects can be displayed by the entertainment module 182 at 694 via the one or more displays and/or the one or more headsets (e.g., AR and/or VR headsets).
- FIG. 7 is a flowchart depicting an example method of providing in-vehicle entertainment.
- Control begins with 704 where a passenger of the vehicle enters the passenger cabin of the vehicle.
- the entertainment module 182 recognizes the passenger.
- a computing device e.g., cell phone
- the wireless connection may be, for example, using a Bluetooth transceiver, a WiFi transceiver, or another suitable type of wireless transceiver.
- the entertainment module 182 may recognize the passenger based on a unique identifier of the computing device. Additionally or alternatively, another type of recognition may be used.
- the entertainment module 182 may recognize the passenger via biometric recognition, such as facial recognition, fingerprint recognition, speech recognition, etc.
- the entertainment module 182 retrieves user information associated with the passenger. For example, the entertainment module 182 may receive the user information wirelessly from the computing device of the passenger. Alternatively, the entertainment module 182 may obtain the user information from a remote source or locally based on recognizing the passenger.
- the entertainment module 182 obtains the present location of the vehicle, such as from the GPS module 320 .
- the entertainment module 182 may also obtain a heading of the vehicle, such as from the GPS module 320 , a compass of the vehicle, and/or one or more images from one or more of the external sensors and cameras 186 .
- the entertainment module 182 activates an entertainment mode.
- the entertainment module 182 may activate the entertainment mode automatically (e.g., in response to the recognition of the passenger) or in response to receipt of user input (e.g., via a button, switch, etc.) to activate the entertainment mode.
- the entertainment module 182 may provide a particular type of entertainment, such as executing a game (e.g., a trivia game) at 724 , outputting (e.g., visibly and/or audibly) local information at 728 , or no entertainment (no visible or audible output) at 732 .
- the type of entertainment may be selected by the passenger or determined by the entertainment module 182 , for example, based on the user information.
- Local information may be obtained wirelessly from one or more remote sources (e.g., one or more websites, such as Wikipedia, one or more databases, etc.) at 736 and/or from memory 740 of the vehicle.
- the entertainment module 182 may display local information for businesses that are presently visible in front of the vehicle.
- the entertainment module 182 may, for example, project information associated with a business or point of interest onto the windshield or into the virtual environment at the location of the business or point of interest.
- FIG. 9 includes an example illustration of information displayed in association with businesses and points of interest.
- the entertainment module 182 may audibly output local information regarding businesses and/or points of interest via one or more speakers.
- the entertainment module 182 may display personalized advertisements to the passenger at 744 .
- the entertainment module 182 may determine which advertisements to display based on the user information of the passenger.
- the entertainment module 182 may determine which advertisements to display further based on the present location of the vehicle. For example, the entertainment module 182 may display advertisements for businesses that are presently visible in front of the vehicle. Additionally or alternatively, the entertainment module 182 may display advertisements in place of billboards that the vehicle is passing.
- FIG. 10 includes an example illustration of an advertisement displayed in place of a billboard.
- the entertainment module 182 may audibly output information regarding advertisements via one or more speakers.
- the entertainment module 182 may also display one or more coupons at 748 ( FIG. 7 ) with one or more of the advertisements.
- An example coupon is illustrated in FIG. 10 in association with an advertisement provided in place of a billboard.
- the advertisements and coupons may be obtained, for example, from one or more remote sources (e.g., one or more websites, such as Wikipedia, one or more databases, etc.) at 736 ( FIG. 7 ) and/or from the memory 740 of the vehicle.
- the entertainment module 182 may transmit advertisements and/or coupons to the computing device of the passenger from the vehicle at 752 .
- the entertainment module 182 may display a user interface requesting user input regarding a rating of the business or point of interest at the location on one or more displays at 756 ( FIG. 7 ).
- the location may be a planned stop of the navigation route or identified based on the vehicle being stopped at the location for at least the predetermined period.
- the user can providing a rating via the user interface.
- the entertainment module 182 may remove the user interface a predetermined period after the vehicle begins moving away from the location.
- the entertainment module 182 may determine the business or point of interest at the location from the location, from the navigation route, or in another suitable manner.
- FIG. 8 is a flowchart depicting a portion of the example of FIG. 7 . 720 is discussed above.
- the entertainment module 182 may determine whether to provide entertainment interactively (in response to user input), automatically (with or without user input), or to execute a game. For example, in response to user input, the entertainment module 182 may provide entertainment information interactively at 728 -B.
- the entertainment module 182 may, for example, output (audibly or visibly) local information to the user in response to receipt of user input regarding a location (e.g., a business or a point of interest).
- the entertainment module 182 may provide entertainment information automatically at 728 -A.
- the entertainment module 182 may, for example, output (audibly or visibly) local information to the user based on locations (e.g., businesses or points of interest) located in front of the vehicle.
- the entertainment module 182 may audibly output information via one or more speakers and display information, for example, via one or more HUDs and/or one or more other displays at 808 .
- Spatial and functional relationships between elements are described using various terms, including “connected,” “engaged,” “coupled,” “adjacent,” “next to,” “on top of,” “above,” “below,” and “disposed.” Unless explicitly described as being “direct,” when a relationship between first and second elements is described in the above disclosure, that relationship can be a direct relationship where no other intervening elements are present between the first and second elements, but can also be an indirect relationship where one or more intervening elements are present (either spatially or functionally) between the first and second elements.
- the phrase at least one of A, B, and C should be construed to mean a logical (A OR B OR C), using a non-exclusive logical OR, and should not be construed to mean “at least one of A, at least one of B, and at least one of C.”
- the direction of an arrow generally demonstrates the flow of information (such as data or instructions) that is of interest to the illustration.
- information such as data or instructions
- the arrow may point from element A to element B. This unidirectional arrow does not imply that no other information is transmitted from element B to element A.
- element B may send requests for, or receipt acknowledgements of, the information to element A.
- module or the term “controller” may be replaced with the term “circuit.”
- the term “module” may refer to, be part of, or include: an Application Specific Integrated Circuit (ASIC); a digital, analog, or mixed analog/digital discrete circuit; a digital, analog, or mixed analog/digital integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor circuit (shared, dedicated, or group) that executes code; a memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip.
- ASIC Application Specific Integrated Circuit
- FPGA field programmable gate array
- the module may include one or more interface circuits.
- the interface circuits may include wired or wireless interfaces that are connected to a local area network (LAN), the Internet, a wide area network (WAN), or combinations thereof.
- LAN local area network
- WAN wide area network
- the functionality of any given module of the present disclosure may be distributed among multiple modules that are connected via interface circuits. For example, multiple modules may allow load balancing.
- a server (also known as remote, or cloud) module may accomplish some functionality on behalf of a client module.
- code may include software, firmware, and/or microcode, and may refer to programs, routines, functions, classes, data structures, and/or objects.
- shared processor circuit encompasses a single processor circuit that executes some or all code from multiple modules.
- group processor circuit encompasses a processor circuit that, in combination with additional processor circuits, executes some or all code from one or more modules. References to multiple processor circuits encompass multiple processor circuits on discrete dies, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above.
- shared memory circuit encompasses a single memory circuit that stores some or all code from multiple modules.
- group memory circuit encompasses a memory circuit that, in combination with additional memories, stores some or all code from one or more modules.
- the term memory circuit is a subset of the term computer-readable medium.
- the term computer-readable medium does not encompass transitory electrical or electromagnetic signals propagating through a medium (such as on a carrier wave); the term computer-readable medium may therefore be considered tangible and non-transitory.
- Non-limiting examples of a non-transitory, tangible computer-readable medium are nonvolatile memory circuits (such as a flash memory circuit, an erasable programmable read-only memory circuit, or a mask read-only memory circuit), volatile memory circuits (such as a static random access memory circuit or a dynamic random access memory circuit), magnetic storage media (such as an analog or digital magnetic tape or a hard disk drive), and optical storage media (such as a CD, a DVD, or a Blu-ray Disc).
- nonvolatile memory circuits such as a flash memory circuit, an erasable programmable read-only memory circuit, or a mask read-only memory circuit
- volatile memory circuits such as a static random access memory circuit or a dynamic random access memory circuit
- magnetic storage media such as an analog or digital magnetic tape or a hard disk drive
- optical storage media such as a CD, a DVD, or a Blu-ray Disc
- the apparatuses and methods described in this application may be partially or fully implemented by a special purpose computer created by configuring a general purpose computer to execute one or more particular functions embodied in computer programs.
- the functional blocks, flowchart components, and other elements described above serve as software specifications, which can be translated into the computer programs by the routine work of a skilled technician or programmer.
- the computer programs include processor-executable instructions that are stored on at least one non-transitory, tangible computer-readable medium.
- the computer programs may also include or rely on stored data.
- the computer programs may encompass a basic input/output system (BIOS) that interacts with hardware of the special purpose computer, device drivers that interact with particular devices of the special purpose computer, one or more operating systems, user applications, background services, background applications, etc.
- BIOS basic input/output system
- the computer programs may include: (i) descriptive text to be parsed, such as HTML (hypertext markup language), XML (extensible markup language), or JSON (JavaScript Object Notation) (ii) assembly code, (iii) object code generated from source code by a compiler, (iv) source code for execution by an interpreter, (v) source code for compilation and execution by a just-in-time compiler, etc.
- source code may be written using syntax from languages including C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, Javascript®, HTML5 (Hypertext Markup Language 5th revision), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, Visual Basic®, Lua, MATLAB, SIMULINK, and Python®.
- languages including C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, Javascript®, HTML5 (Hypertext Markup Language 5th revision), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, Visual Basic®, Lua, MATLAB, SIMU
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Mechanical Engineering (AREA)
- User Interface Of Digital Computer (AREA)
- Fittings On The Vehicle Exterior For Carrying Loads, And Devices For Holding Or Mounting Articles (AREA)
Abstract
An entertainment system of a vehicle includes: a plurality of cameras configured to capture images around an exterior of the vehicle; one or more displays; and an entertainment module configured to: based on the images, generate panoramic video; and display the panoramic video within the vehicle via the one or more displays.
Description
- The information provided in this section is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.
- The present disclosure relates to in-vehicle entertainment and more particularly to panoramic in-vehicle entertainment.
- Vehicles include one or more torque producing devices, such as an internal combustion engine and/or an electric motor. A passenger of a vehicle rides within a passenger cabin (or passenger compartment) of the vehicle.
- Vehicles may include one or more different type of sensors that sense vehicle surroundings. One example of a sensor that senses vehicle surroundings is a camera configured to capture images of the vehicle surroundings. Examples of such cameras include forward facing cameras, rear facing cameras, and side facing cameras. Another example of a sensor that senses vehicle surroundings includes a radar sensor configured to capture information regarding vehicle surroundings. Other examples of sensors that sense vehicle surroundings include sonar sensors and light detection and ranging (LIDAR) sensors configured to capture information regarding vehicle surroundings.
- In a feature, an entertainment system of a vehicle includes: a plurality of cameras configured to capture images around an exterior of the vehicle; one or more displays; and an entertainment module configured to: based on the images, generate panoramic video; and display the panoramic video within the vehicle via the one or more displays.
- In further features, the plurality of cameras include: at least one camera configured to capture images in front of the vehicle; at least one camera configured to capture images to the right of the vehicle; at least one camera configured to capture images to the left of the vehicle; and at least one camera configured to capture images behind the vehicle.
- In further features, the entertainment module is configured to perform image stabilization on the images.
- In further features, the entertainment module is configured to selectively correct ones of the images for shaking.
- In further features, the entertainment module is configured to stitch ones of the images together to create the panoramic video.
- In further features, the one or more displays include one or more heads up displays (HUDs) configured to project light onto a windshield of the vehicle and one or more displays configured to display images.
- In further features, the one or more displays include at least one of an augmented reality (AR) headset and a virtual reality (VR) headset within the vehicle.
- In further features, the entertainment module is configured to perceive an object located outside of the vehicle and animate an item within the panoramic video based on the location.
- In further features, the entertainment module is configured to locate the item within the panoramic video based on a location of the object relative to the vehicle.
- In further features, the entertainment module is configured to animate the item within the panoramic video based on a type of the object located outside of the vehicle.
- In further features, a global positioning system (GPS) module is configured to determine a present location of the vehicle, where the entertainment module is configured to generate the panoramic video further based on the present location of the vehicle.
- In further features, the entertainment module is configured to display a visual identifier of an object located in front of the vehicle via at least one of the one or more displays.
- In further features, the entertainment module is configured to selectively output audio information regarding the object via one or more speakers of the vehicle.
- In further features, the entertainment module is further configured to display a visual advertisement via at least one of the one or more displays.
- In further features, the entertainment module is further configured to display a coupon via at least one of the one or more displays.
- In further features, the entertainment module is configured to selectively wirelessly transmit the coupon to a computing device of a passenger of the vehicle.
- In further features, the entertainment module is configured to selectively execute a game application stored in memory of the vehicle.
- In further features, the entertainment module is configured to generate at least one of a visual and an audible prompt for user input indicative of a rating for an entity in response to a determination that the vehicle stopped at the entity.
- In a feature, an entertainment method for a vehicle includes: capturing images around an exterior of the vehicle by a plurality of cameras; based on the images, generating panoramic video; and displaying the panoramic video within the vehicle via one or more displays.
- Further areas of applicability of the present disclosure will become apparent from the detailed description, the claims and the drawings. The detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the disclosure.
- The present disclosure will become more fully understood from the detailed description and the accompanying drawings, wherein:
-
FIG. 1 is a functional block diagram of an example vehicle system; -
FIG. 2 is a functional block diagram of a vehicle including various external cameras and sensors; -
FIG. 3 is a functional block diagram of an example entertainment system of a vehicle; -
FIGS. 4 and 5 are functional block diagrams of examples of the entertainment system; -
FIGS. 6 and 7 are flowcharts depicting example methods of providing in-vehicle entertainment; -
FIG. 8 is a flowchart depicting a portion of the example ofFIG. 7 ; -
FIG. 9 includes an example illustration of information displayed in association with businesses and points of interest; and -
FIG. 10 includes an example illustration of an advertisement and a coupon displayed. - In the drawings, reference numbers may be reused to identify similar and/or identical elements.
- A vehicle may include an infotainment system that allows one or more users to perform infotainment tasks in vehicle via a display (e.g., a touchscreen display) and/or one or more speakers.
- The present application involves an entertainment system of a vehicle that provides panoramic and/or 360 degree views within the vehicle. For example, the entertainment system may provide a panoramic view based on vehicle surroundings on one or more displays within the vehicle by stitching together images from in front of the vehicle, beside the vehicle, and behind the vehicle. As another example, the entertainment system may provide a 360 degree virtual environment of a game via a headset (e.g., augmented reality or virtual reality) within the vehicle. The entertainment system may generate objects within the virtual environment of the game based on objects in the real world around the vehicle.
- The entertainment system may also provide entertainment features, such as allowing a passenger to play games within the vehicle, outputting (visibly and/or audibly) local information to the passenger, providing advertisements to the user, and providing coupons to the passenger. The above may increase user experience within the vehicle.
- Referring now to
FIG. 1 , a functional block diagram of an example vehicle system is presented. While a vehicle system for a hybrid vehicle is shown and will be described, the present application is also applicable to non-hybrid vehicles, electric vehicles, fuel cell vehicles, and other types of vehicles. The present application is applicable to autonomous vehicles, semi-autonomous vehicles, non-autonomous vehicles, shared vehicles, non-shared vehicles, and other types of vehicles. - An engine 102 may combust an air/fuel mixture to generate drive torque. An engine control module (ECM) 106 controls the engine 102. For example, the ECM 106 may control actuation of engine actuators, such as a throttle valve, one or more spark plugs, one or more fuel injectors, valve actuators, camshaft phasers, an exhaust gas recirculation (EGR) valve, one or more boost devices, and other suitable engine actuators. In some types of vehicles (e.g., electric vehicles), the engine 102 may be omitted.
- The engine 102 may output torque to a
transmission 110. A transmission control module (TCM) 114 controls operation of thetransmission 110. For example, the TCM 114 may control gear selection within thetransmission 110 and one or more torque transfer devices (e.g., a torque converter, one or more clutches, etc.). - The vehicle system may include one or more electric motors. For example, an
electric motor 118 may be implemented within thetransmission 110 as shown in the example ofFIG. 1 . An electric motor can act as either a generator or as a motor at a given time. When acting as a generator, an electric motor converts mechanical energy into electrical energy. The electrical energy can be, for example, used to charge abattery 126 via a power control device (PCD) 130. When acting as a motor, an electric motor generates torque that may be used, for example, to supplement or replace torque output by the engine 102. While the example of one electric motor is provided, the vehicle may include zero or more than one electric motor. - A power inverter module (PIM) 134 may control the
electric motor 118 and thePCD 130. ThePCD 130 applies power from thebattery 126 to theelectric motor 118 based on signals from thePIM 134, and thePCD 130 provides power output by theelectric motor 118, for example, to thebattery 126. ThePIM 134 may include, for example, an inverter. - A
steering control module 140 controls steering/turning of wheels of the vehicle, for example, based on driver turning of a steering wheel within the vehicle and/or steering commands from one or more vehicle control modules. A steering wheel angle sensor (SWA) monitors rotational position of the steering wheel and generates aSWA 142 based on the position of the steering wheel. As an example, thesteering control module 140 may control vehicle steering via an EPS motor 144 based on theSWA 142. However, the vehicle may include another type of steering system. - An electronic brake control module (EBCM) 150 may selectively control
brakes 154 of the vehicle. Ahorn module 156 may apply power to ahorn 158 when ahorn input 160 is in a first state. Thehorn 158 outputs sound when power is applied to thehorn 158. Thehorn module 156 may not apply power to thehorn 158 when thehorn input 160 is in a second state. Thehorn input 160 may transition from the second state to the first state, for example, in response to user application of at least a predetermined force is applied to a horn input device (e.g., located on the steering wheel). The horn input device may apply a bias force such that thehorn input 160 is in the second state when less than the predetermined force is applied to the horn input device. - Modules of the vehicle may share parameters via a controller area network (CAN) 162. The
CAN 162 may also be referred to as a car area network. For example, theCAN 162 may include one or more data buses. Various parameters may be made available by a given module to other modules via theCAN 162. - The driver inputs may include, for example, an accelerator pedal position (APP) 166 which may be provided to the
ECM 106. A brake pedal position (BPP) 170 may be provided to theEBCM 150. Aposition 174 of a park, reverse, neutral, drive lever (PRNDL) may be provided to theTCM 114. Anignition state 178 may be provided to a body control module (BCM) 180. For example, theignition state 178 may be input by a driver via an ignition key, button, or switch. At a given time, theignition state 178 may be one of off, accessory, run, or crank. - The vehicle system may also include an
entertainment module 182. Theentertainment module 182 may be connected to, be a part of, or include an infotainment module of the vehicle. - The
entertainment module 182 selectively executes games and outputs gaming feedback via one ormore output devices 184. Theoutput devices 184 may include, for example, one or more displays, one or more sets of virtual reality (VR) goggles, one or more sets of augmented reality (AR) goggles, one or more other suitable types of video output devices, one or more speakers, one or more haptic devices, and/or one or more other suitable types of output devices. - The
entertainment module 182 outputs gaming video via the one or more displays, one or more sets of VR goggles, and/or one or more sets of AR goggles. Theentertainment module 182 outputs gaming audio via the one or more speakers. Theentertainment module 182 may also output other gaming feedback via one or more haptic devices. For example, haptic devices may be included with one or more seats, in the steering wheel, in one or more seat belts, etc. - Examples of displays may include, for example, one or more displays (e.g., on a front console) of the vehicle, a head up display (HUD) that displays information via a substrate (e.g., windshield), one or more displays that drop downwardly or extend upwardly to form panoramic views, and/or one or more other suitable displays. In various implementations, one or more HUDs may project onto the windshield to use the windshield as a panoramic AR display.
- The
entertainment module 182 controls play of one or more games based on user input received via one ormore input devices 185, such as one or more gaming controllers, one or more joysticks, etc. Under some circumstances, such as when the vehicle is parked, theentertainment module 182 may control gameplay based on input from vehicle components (e.g., the steering wheel, brake and accelerator pedals, horn, etc.). - The
entertainment module 182 may additionally or alternatively audibly and/or visibly output other information, such as personalized advertisements, coupons, information about points of interest in front of the vehicle, etc. - The vehicle may include a plurality of external sensors and cameras, generally illustrated in
FIG. 1 by 186. One or more actions may be taken based on input from the external sensors andcameras 186. For example, the infotainment module may display video, various views, and/or alerts on a display via input from the external sensors andcameras 186 during driving. - As another example, based on input from the external sensors and
cameras 186, a perception module perceives objects around the vehicle and locations of the objects relative to the vehicle. TheECM 106 may adjust torque output of the engine 102 based on input from the perception module. Additionally or alternatively, thePIM 134 may control power flow to and/or from theelectric motor 118 based on input from the perception module. Additionally or alternatively, theEBCM 150 may adjust braking based on input from the perception module. Additionally or alternatively, thesteering control module 140 may adjust steering based on input from the perception module. Theentertainment module 182 may adjust the virtual environment of a game based on objects identified by the perception module, as discussed further below. - The vehicle may include one or more additional control modules that are not shown, such as a chassis control module, a battery pack control module, etc. The vehicle may omit one or more of the control modules shown and discussed.
- Referring now to
FIG. 2 , a functional block diagram of a vehicle including examples of external sensors and cameras is presented. The external sensors andcameras 186 include various cameras positioned to capture images and video outside of (external to) the vehicle and various types of sensors measuring parameters outside of (external to the vehicle). For example, a forward facingcamera 204 captures images and video of images within a predetermined field of view (FOV) 206 in front of the vehicle. - A
front camera 208 may also capture images and video within apredetermined FOV 210 in front of the vehicle. Thefront camera 208 may capture images and video within a predetermined distance of the front of the vehicle and may be located at the front of the vehicle (e.g., in a front fascia, grille, or bumper). Theforward facing camera 204 may be located more rearward, however, such as with a rear view mirror at a windshield of the vehicle. Theforward facing camera 204 may not be able to capture images and video of items within all of or at least a portion of the predetermined FOV of thefront camera 208 and may capture images and video that is greater than the predetermined distance of the front of the vehicle. In various implementations, only one of theforward facing camera 204 and thefront camera 208 may be included. - A
rear camera 212 captures images and video within apredetermined FOV 214 behind the vehicle. Therear camera 212 may capture images and video within a predetermined distance behind vehicle and may be located at the rear of the vehicle, such as near a rear license plate. - A
right camera 216 captures images and video within apredetermined FOV 218 to the right of the vehicle. Theright camera 216 may capture images and video within a predetermined distance to the right of the vehicle and may be located, for example, under a right side rear view mirror. In various implementations, the right side rear view mirror may be omitted, and theright camera 216 may be located near where the right side rear view mirror would normally be located. - A left camera 220 captures images and video within a
predetermined FOV 222 to the left of the vehicle. The left camera 220 may capture images and video within a predetermined distance to the left of the vehicle and may be located, for example, under a left side rear view mirror. In various implementations, the left side rear view mirror may be omitted, and the left camera 220 may be located near where the left side rear view mirror would normally be located. While the example FOVs are shown for illustrative purposes, the FOVs may overlap, for example, for more accurate and/or inclusive stitching. - The external sensors and
cameras 186 may additionally or alternatively include various other types of sensors, such as ultrasonic sensors, radar sensors, etc. For example, the vehicle may include one or more forward facing ultrasonic sensors, such as forward facingultrasonic sensors ultrasonic sensors ultrasonic sensor 242, and one or more left side ultrasonic sensors, such as left sideultrasonic sensor 246. The locations of the cameras and ultrasonic sensors are provided as examples only and different locations could be used. Ultrasonic sensors output ultrasonic signals around the vehicle. - The external sensors and
cameras 186 may additionally or alternatively include one or more other types of sensors, such as one or more sonar sensors, one or more radar sensors, and/or one or more light detection and ranging (LIDAR) sensors. -
FIG. 3 is a functional block diagram of an example entertainment system of a vehicle. Theentertainment module 182 includes aprocessor module 304 including one or more processors. Theprocessor module 304 executes one ormore applications 308, such as games and other types of applications stored in memory. Each of the games is an application executable by theprocessor module 304. - During execution of a game, the
processor module 304 generates a panoramic virtual environment for the game based on data from the external cameras andsensors 186. In this way, the panoramic virtual environment correlates with the real world surrounding the vehicle. Theprocessor module 304 displays the virtual environment in a panoramic format or 360 degree format via one or more of theoutput devices 184. For example, theprocessor module 304 may display the virtual environment of a game in panoramic format via one or more displays that together provide panoramic views in a panoramic format (e.g., having an aspect ratio of at least 3:1 width to height or a greater width). Additionally or alternatively, theprocessor module 304 may display the virtual environment of a game in panoramic format using one or more HUDs that project panoramic video onto the windshield to use the windshield as a panoramic AR display. Additionally or alternatively, theprocessor module 304 may display a 360 degree view of the virtual environment using at least one of an AR headset in the vehicle and a VR headset in the vehicle. -
FIG. 4 is a functional block diagram for an example of the entertainment system. Theentertainment module 182 receives images and data from the external sensors andcameras 186 from infront 404 of the vehicle, left andright sides entertainment module 182 generates 360 degree video based on the received imaged and data and displays the video via one or more AR and/or VR headsets 420. As a user of the headset 420 looks forward (toward the front of the vehicle), the headset 420 displays the virtual environment in aforward direction 424. As a user of the headset 420 looks left (toward the left of the vehicle), the headset 420 displays the virtual environment in aleft direction 428. As a user of the headset 420 looks right (toward the right of the vehicle), the headset 420 displays the virtual environment in a right direction 432. As a user of the headset 420 looks backward (toward the back of the vehicle), the headset 420 displays the virtual environment in abackwards direction 436. -
FIG. 5 is a functional block diagram for an example of the entertainment system. In the example ofFIG. 5 , theentertainment module 182 generates panoramic video 504 (having an aspect ratio of 3:1 width to height or a greater width) based on the received imaged and data and displays the video via one or more display devices. In various implementations, thepanoramic video 504 may be displayed using aleft display 508, amiddle display 512, and aright display 516. In various implementations, thepanoramic video 504 may be displayed on the windshield of the vehicle via one or more HUDs. - Referring now to
FIGS. 4 and 5 , theentertainment module 182 generates the panoramic or 360 degree virtual environment based on a baselinevirtual environment 440 for the game. Theentertainment module 182 adjusts the baselinevirtual environment 440 based on data from the external cameras andsensors 186 to generate the panoramic or 360 degree virtual environment. - The
entertainment module 182 performs image stabilizing 444 on images received to, for example, account for vibration of the vehicle. Theentertainment module 182 may additionally performdistortion correction 448, for example, to reduce distortion attributable to characteristics (e.g., distortion) of the cameras of the external cameras andsensors 186. Theentertainment module 182stiches 452 the images together for the panoramic or 360 degree virtual environment. For example, theentertainment module 182 stiches the left side of images from in front of the vehicle with the right side of images taken to the left of the vehicle. The entertainment module stiches the right side of images from in front of the vehicle to the left side of images taken to the right of the vehicle. - Based on the data received from the external sensors and
cameras 186, theentertainment module 182 perceives 456 objects (e.g., vehicles, buildings, animals, humans, etc.) located in the real world around the vehicle and locations of the objects relative to the vehicle. Theentertainment module 182 may create objects in the virtual environment based on the types of objects perceived in the real world and the locations of the objects in the real world using animation 460. -
FIG. 6 is a flowchart depicting an example method of providing in-vehicle entertainment. At 608, images fromcameras 604 of the external sensors andcameras 186 are obtained. The images include images in front of the vehicle, to the right of the vehicle, to the left of the vehicle, and behind the vehicle. - At 612, the
entertainment module 182 corrects the images for any distortion in the images using a distortion correction algorithm. At 616, theentertainment module 182 enters the images into image buffers. Theentertainment module 182 may determine whether one or more pixels are outside of a frame border of the images. - At 620, the entertainment module 620 performs pixel movement tracking on the images to track motion of objects in the pixels from image to image. At 624, the
entertainment module 182 determines movement trajectory based on the images. At 628, theentertainment module 628 performs extrapolation on the images. - At 632, the entertainment module determines whether a real pixel location of an image is out of trajectory given the movement trajectory (from 624). If 632 is true, control continues with 636 and 640. If 632 is false, control continues with 644. At 644, no image shaking correction is applied to the images, and control continues with 648. At 636, the
entertainment module 182 provides smoothing of the images to correct the images for shaking. The smoothing of 636 produces images corrected for shaking at 640, and control continues with 648. Shaking correction may be performed, for example, based on target tracking and trajectory determination. - At 648, the
entertainment module 182 stiches the images together to create a panoramic or 360 degree view. Theentertainment module 182 may also include features and/or contours at image corners obtained at 652. Theentertainment module 182 may also include features and/or contours of interest obtained at 656. Theentertainment module 182 may also include information from matching based on points and/or contours of interest at 660. At 652, theentertainment module 182 may obtain features and/or contours near corners of the images using, for example, a scale invariant feature transform (SIFT) algorithm and/or a speeded up robust features (SURF) algorithm. Theentertainment module 182 may determine the points and/or contours of interest in the images at 656 using a point of interest identification algorithm and/or a contour of interest algorithm. Theentertainment module 182 may perform matching based on points and/or contours of interest at 660 using, for example, an affine transformation algorithm, cross-correlation algorithm, or another suitable type of algorithm. - The panoramic or 360 degree view of the real world around the vehicle can be displayed by the
entertainment module 182 at 662 via the one or more displays and/or the one or more headsets (e.g., AR and/or VR headsets). Alternatively, control may continue with 664. - At 664, the
entertainment module 182 performs perception at 664. More specifically, theentertainment module 182 perceives objects in the images or the stitched together image. Theentertainment module 182 may determine a type of each object and a location of each object relative to the vehicle. Examples of types of objects include, for example, vehicles, pedestrians, cyclists, animals, buildings, etc. Theentertainment module 182 may determine the types of the objects using an object detection algorithm. Theentertainment module 182 may determine the locations of the objects, for example, based on the FOV of the camera, the location of the object in the pixels of an image, and dimensions (e.g., pixelwise) of the object in the image. As a result of the perception, actors for the game (e.g., vehicles, pedestrians, animals, etc.) are identified at 668. Structures (e.g., buildings, roads, mountains, etc.) are identified at 672. Unrecognizable objects are also identified at 676. - At 680, the
entertainment module 182 obtains thevirtual environment 440 for the game. For example, theentertainment module 182 may determine a present location for a player of the game in thevirtual environment 440 based on a (present)location 684 of the vehicle on amap 686 of the world or a predetermined area around the present location of the vehicle. The location 684 (e.g., latitude, longitude, elevation, heading) of the vehicle may be determined and provided by a global positioning system (GPS)module 320 of the vehicle (FIG. 3 ). Theentertainment module 182 may use the virtual environment as a background at 694. - At 690, the
entertainment module 182 animates and locates the perceived objects within thevirtual environment 440. Theentertainment module 182 may set each different type of object detected to a predetermined type of object in the virtual environment for the game. For example, theentertainment module 182 may animate pedestrians as animals in the virtual environment, vehicles as opponents within the game, and unrecognizable objects as exploratory items within the virtual environment. Theentertainment module 182 locates the animated objects within the virtual environment based on the location of the associated objects in the real world relative to the vehicle. The panoramic or 360 degree view of the virtual environment including added objects can be displayed by theentertainment module 182 at 694 via the one or more displays and/or the one or more headsets (e.g., AR and/or VR headsets). -
FIG. 7 is a flowchart depicting an example method of providing in-vehicle entertainment. Control begins with 704 where a passenger of the vehicle enters the passenger cabin of the vehicle. At 708, theentertainment module 182 recognizes the passenger. For example, a computing device (e.g., cell phone) may wirelessly connect with the vehicle, such as via an application executing on the computing device. The wireless connection may be, for example, using a Bluetooth transceiver, a WiFi transceiver, or another suitable type of wireless transceiver. Theentertainment module 182 may recognize the passenger based on a unique identifier of the computing device. Additionally or alternatively, another type of recognition may be used. For example, theentertainment module 182 may recognize the passenger via biometric recognition, such as facial recognition, fingerprint recognition, speech recognition, etc. - At 712, the
entertainment module 182 retrieves user information associated with the passenger. For example, theentertainment module 182 may receive the user information wirelessly from the computing device of the passenger. Alternatively, theentertainment module 182 may obtain the user information from a remote source or locally based on recognizing the passenger. - At 716, the
entertainment module 182 obtains the present location of the vehicle, such as from theGPS module 320. Theentertainment module 182 may also obtain a heading of the vehicle, such as from theGPS module 320, a compass of the vehicle, and/or one or more images from one or more of the external sensors andcameras 186. - At 720, the
entertainment module 182 activates an entertainment mode. Theentertainment module 182 may activate the entertainment mode automatically (e.g., in response to the recognition of the passenger) or in response to receipt of user input (e.g., via a button, switch, etc.) to activate the entertainment mode. Also at 720, theentertainment module 182 may provide a particular type of entertainment, such as executing a game (e.g., a trivia game) at 724, outputting (e.g., visibly and/or audibly) local information at 728, or no entertainment (no visible or audible output) at 732. The type of entertainment may be selected by the passenger or determined by theentertainment module 182, for example, based on the user information. Local information may be obtained wirelessly from one or more remote sources (e.g., one or more websites, such as Wikipedia, one or more databases, etc.) at 736 and/or frommemory 740 of the vehicle. For example, theentertainment module 182 may display local information for businesses that are presently visible in front of the vehicle. Theentertainment module 182 may, for example, project information associated with a business or point of interest onto the windshield or into the virtual environment at the location of the business or point of interest.FIG. 9 includes an example illustration of information displayed in association with businesses and points of interest. In addition to a visual display, theentertainment module 182 may audibly output local information regarding businesses and/or points of interest via one or more speakers. - Additionally or alternatively to 720, the
entertainment module 182 may display personalized advertisements to the passenger at 744. Theentertainment module 182 may determine which advertisements to display based on the user information of the passenger. Theentertainment module 182 may determine which advertisements to display further based on the present location of the vehicle. For example, theentertainment module 182 may display advertisements for businesses that are presently visible in front of the vehicle. Additionally or alternatively, theentertainment module 182 may display advertisements in place of billboards that the vehicle is passing.FIG. 10 includes an example illustration of an advertisement displayed in place of a billboard. In addition to a visual display, theentertainment module 182 may audibly output information regarding advertisements via one or more speakers. - The
entertainment module 182 may also display one or more coupons at 748 (FIG. 7 ) with one or more of the advertisements. An example coupon is illustrated inFIG. 10 in association with an advertisement provided in place of a billboard. The advertisements and coupons may be obtained, for example, from one or more remote sources (e.g., one or more websites, such as Wikipedia, one or more databases, etc.) at 736 (FIG. 7 ) and/or from thememory 740 of the vehicle. Theentertainment module 182 may transmit advertisements and/or coupons to the computing device of the passenger from the vehicle at 752. - When the vehicle makes a stop at a location (e.g., a business or a point of interest) for at least a predetermined period (e.g., more than 1 minute), the
entertainment module 182 may display a user interface requesting user input regarding a rating of the business or point of interest at the location on one or more displays at 756 (FIG. 7 ). The location may be a planned stop of the navigation route or identified based on the vehicle being stopped at the location for at least the predetermined period. The user can providing a rating via the user interface. Theentertainment module 182 may remove the user interface a predetermined period after the vehicle begins moving away from the location. Theentertainment module 182 may determine the business or point of interest at the location from the location, from the navigation route, or in another suitable manner. -
FIG. 8 is a flowchart depicting a portion of the example ofFIG. 7 . 720 is discussed above. In response to the passenger activating the entertainment mode, control continues with 804. At 804, theentertainment module 182 may determine whether to provide entertainment interactively (in response to user input), automatically (with or without user input), or to execute a game. For example, in response to user input, theentertainment module 182 may provide entertainment information interactively at 728-B. Theentertainment module 182 may, for example, output (audibly or visibly) local information to the user in response to receipt of user input regarding a location (e.g., a business or a point of interest). Theentertainment module 182 may provide entertainment information automatically at 728-A. Theentertainment module 182 may, for example, output (audibly or visibly) local information to the user based on locations (e.g., businesses or points of interest) located in front of the vehicle. Theentertainment module 182 may audibly output information via one or more speakers and display information, for example, via one or more HUDs and/or one or more other displays at 808. - The foregoing description is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses. The broad teachings of the disclosure can be implemented in a variety of forms. Therefore, while this disclosure includes particular examples, the true scope of the disclosure should not be so limited since other modifications will become apparent upon a study of the drawings, the specification, and the following claims. It should be understood that one or more steps within a method may be executed in different order (or concurrently) without altering the principles of the present disclosure. Further, although each of the embodiments is described above as having certain features, any one or more of those features described with respect to any embodiment of the disclosure can be implemented in and/or combined with features of any of the other embodiments, even if that combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and permutations of one or more embodiments with one another remain within the scope of this disclosure.
- Spatial and functional relationships between elements (for example, between modules, circuit elements, semiconductor layers, etc.) are described using various terms, including “connected,” “engaged,” “coupled,” “adjacent,” “next to,” “on top of,” “above,” “below,” and “disposed.” Unless explicitly described as being “direct,” when a relationship between first and second elements is described in the above disclosure, that relationship can be a direct relationship where no other intervening elements are present between the first and second elements, but can also be an indirect relationship where one or more intervening elements are present (either spatially or functionally) between the first and second elements. As used herein, the phrase at least one of A, B, and C should be construed to mean a logical (A OR B OR C), using a non-exclusive logical OR, and should not be construed to mean “at least one of A, at least one of B, and at least one of C.”
- In the figures, the direction of an arrow, as indicated by the arrowhead, generally demonstrates the flow of information (such as data or instructions) that is of interest to the illustration. For example, when element A and element B exchange a variety of information but information transmitted from element A to element B is relevant to the illustration, the arrow may point from element A to element B. This unidirectional arrow does not imply that no other information is transmitted from element B to element A. Further, for information sent from element A to element B, element B may send requests for, or receipt acknowledgements of, the information to element A.
- In this application, including the definitions below, the term “module” or the term “controller” may be replaced with the term “circuit.” The term “module” may refer to, be part of, or include: an Application Specific Integrated Circuit (ASIC); a digital, analog, or mixed analog/digital discrete circuit; a digital, analog, or mixed analog/digital integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor circuit (shared, dedicated, or group) that executes code; a memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip.
- The module may include one or more interface circuits. In some examples, the interface circuits may include wired or wireless interfaces that are connected to a local area network (LAN), the Internet, a wide area network (WAN), or combinations thereof. The functionality of any given module of the present disclosure may be distributed among multiple modules that are connected via interface circuits. For example, multiple modules may allow load balancing. In a further example, a server (also known as remote, or cloud) module may accomplish some functionality on behalf of a client module.
- The term code, as used above, may include software, firmware, and/or microcode, and may refer to programs, routines, functions, classes, data structures, and/or objects. The term shared processor circuit encompasses a single processor circuit that executes some or all code from multiple modules. The term group processor circuit encompasses a processor circuit that, in combination with additional processor circuits, executes some or all code from one or more modules. References to multiple processor circuits encompass multiple processor circuits on discrete dies, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above. The term shared memory circuit encompasses a single memory circuit that stores some or all code from multiple modules. The term group memory circuit encompasses a memory circuit that, in combination with additional memories, stores some or all code from one or more modules.
- The term memory circuit is a subset of the term computer-readable medium. The term computer-readable medium, as used herein, does not encompass transitory electrical or electromagnetic signals propagating through a medium (such as on a carrier wave); the term computer-readable medium may therefore be considered tangible and non-transitory. Non-limiting examples of a non-transitory, tangible computer-readable medium are nonvolatile memory circuits (such as a flash memory circuit, an erasable programmable read-only memory circuit, or a mask read-only memory circuit), volatile memory circuits (such as a static random access memory circuit or a dynamic random access memory circuit), magnetic storage media (such as an analog or digital magnetic tape or a hard disk drive), and optical storage media (such as a CD, a DVD, or a Blu-ray Disc).
- The apparatuses and methods described in this application may be partially or fully implemented by a special purpose computer created by configuring a general purpose computer to execute one or more particular functions embodied in computer programs. The functional blocks, flowchart components, and other elements described above serve as software specifications, which can be translated into the computer programs by the routine work of a skilled technician or programmer.
- The computer programs include processor-executable instructions that are stored on at least one non-transitory, tangible computer-readable medium. The computer programs may also include or rely on stored data. The computer programs may encompass a basic input/output system (BIOS) that interacts with hardware of the special purpose computer, device drivers that interact with particular devices of the special purpose computer, one or more operating systems, user applications, background services, background applications, etc.
- The computer programs may include: (i) descriptive text to be parsed, such as HTML (hypertext markup language), XML (extensible markup language), or JSON (JavaScript Object Notation) (ii) assembly code, (iii) object code generated from source code by a compiler, (iv) source code for execution by an interpreter, (v) source code for compilation and execution by a just-in-time compiler, etc. As examples only, source code may be written using syntax from languages including C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, Javascript®, HTML5 (Hypertext Markup Language 5th revision), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, Visual Basic®, Lua, MATLAB, SIMULINK, and Python®.
Claims (20)
1. An entertainment system of a vehicle, comprising:
a plurality of cameras configured to capture images around an exterior of the vehicle;
one or more displays; and
an entertainment module configured to:
based on the images, generate panoramic video including a virtual environment of a game;
display the panoramic video within the vehicle via the one or more displays; and
at least one of:
perceive a pedestrian located outside of the vehicle and animate the pedestrian as an animal within the panoramic video; and
perceive a second vehicle located outside of the vehicle and animate the second vehicle as an opponent within the panoramic video.
2. The entertainment system of claim 1 wherein the plurality of cameras include:
at least one camera configured to capture images in front of the vehicle;
at least one camera configured to capture images to the right of the vehicle;
at least one camera configured to capture images to the left of the vehicle; and
at least one camera configured to capture images behind the vehicle.
3. The entertainment system of claim 1 wherein the entertainment module is configured to perform image stabilization on the images.
4. The entertainment system of claim 1 wherein the entertainment module is configured to selectively correct ones of the images for shaking.
5. The entertainment system of claim 1 wherein the entertainment module is configured to stitch ones of the images together to create the panoramic video.
6. The entertainment system of claim 1 wherein the one or more displays include one or more heads up displays (HUDs) configured to project light onto a windshield of the vehicle and one or more displays configured to display images.
7. The entertainment system of claim 1 wherein the one or more displays include at least one of an augmented reality (AR) headset and a virtual reality (VR) headset within the vehicle.
8. The entertainment system of claim 1 wherein the entertainment module is configured to perceive an object located outside of the vehicle and animate an item within the panoramic video based on the location.
9. The entertainment system of claim 8 wherein the entertainment module is configured to locate the item within the panoramic video based on a location of the object relative to the vehicle.
10. The entertainment system of claim 8 wherein the entertainment module is configured to animate the item within the panoramic video based on a type of the object located outside of the vehicle.
11. The entertainment system of claim 1 further comprising a global positioning system (GPS) module configured to determine a present location of the vehicle,
wherein the entertainment module is configured to generate the panoramic video further based on the present location of the vehicle.
12. The entertainment system of claim 1 wherein the entertainment module is configured to display a visual identifier of an object located in front of the vehicle via at least one of the one or more displays.
13. The entertainment system of claim 12 wherein the entertainment module is configured to selectively output audio information regarding the object via one or more speakers of the vehicle.
14. The entertainment system of claim 1 wherein the entertainment module is further configured to display a visual advertisement via at least one of the one or more displays.
15. The entertainment system of claim 14 wherein the entertainment module is further configured to display a coupon via at least one of the one or more displays.
16. The entertainment system of claim 1 wherein the entertainment module is further configured to display a coupon via at least one of the one or more displays.
17. The entertainment system of claim 16 wherein the entertainment module is configured to selectively wirelessly transmit the coupon to a computing device of a passenger of the vehicle.
18. The entertainment system of claim 1 wherein the entertainment module is configured to selectively execute a game application stored in memory of the vehicle.
19. The entertainment system of claim 1 wherein the entertainment module is configured to generate at least one of a visual and an audible prompt for user input indicative of a rating for an entity in response to a determination that the vehicle stopped at the entity.
20. An entertainment method for a vehicle, comprising:
capturing images around an exterior of the vehicle by a plurality of cameras;
based on the images, generating panoramic video including a virtual environment of a game; and
displaying the panoramic video within the vehicle via one or more displays,
wherein the generating the panoramic video includes at least one of:
perceiving a pedestrian located outside of the vehicle and animating the pedestrian as an animal within the panoramic video; and
perceiving a second vehicle located outside of the vehicle and animating the second vehicle as an opponent within the panoramic video.
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/083,495 US11318887B1 (en) | 2020-10-29 | 2020-10-29 | Panoramic virtual environment for in-vehicle entertainment |
CN202110516124.5A CN114506278B (en) | 2020-10-29 | 2021-05-12 | Panoramic virtual environment for in-vehicle entertainment |
DE102021112633.9A DE102021112633A1 (en) | 2020-10-29 | 2021-05-15 | Virtual panoramic environment for in-car entertainment |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/083,495 US11318887B1 (en) | 2020-10-29 | 2020-10-29 | Panoramic virtual environment for in-vehicle entertainment |
Publications (2)
Publication Number | Publication Date |
---|---|
US11318887B1 US11318887B1 (en) | 2022-05-03 |
US20220134953A1 true US20220134953A1 (en) | 2022-05-05 |
Family
ID=81184534
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/083,495 Active US11318887B1 (en) | 2020-10-29 | 2020-10-29 | Panoramic virtual environment for in-vehicle entertainment |
Country Status (3)
Country | Link |
---|---|
US (1) | US11318887B1 (en) |
CN (1) | CN114506278B (en) |
DE (1) | DE102021112633A1 (en) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115610348A (en) * | 2022-10-12 | 2023-01-17 | 蔚来汽车科技(安徽)有限公司 | Augmented reality and virtual reality equipment, on-vehicle system and vehicle |
Citations (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20090237268A1 (en) * | 2008-03-18 | 2009-09-24 | Hyundai Motor Company | Information display system for vehicle |
US20090268093A1 (en) * | 2008-04-28 | 2009-10-29 | Honda Motor Co., Ltd. | Video Display System for a Motor Vehicle |
US20130066720A1 (en) * | 2011-09-09 | 2013-03-14 | Gordon * Howard Associates, Inc. | Method and System of Providing Information to an Occupant of a Vehicle |
US20140028980A1 (en) * | 2012-07-26 | 2014-01-30 | Cloudcar, Inc. | Vehicle content projection |
US20150042751A1 (en) * | 2013-08-12 | 2015-02-12 | GM Global Technology Operations LLC | Panorama projection unit for a motor vehicle |
US20150262435A1 (en) * | 2014-03-17 | 2015-09-17 | Hti Ip, Llc | Method and System for Providing Intelligent Alerts |
US20150296140A1 (en) * | 2014-04-11 | 2015-10-15 | Grey Matter Fusion, Inc. | Panoramic view blind spot eliminator system and method |
US20180089900A1 (en) * | 2016-09-23 | 2018-03-29 | Apple Inc. | Augmented virtual display |
US20190045117A1 (en) * | 2017-08-03 | 2019-02-07 | Ford Global Technologies, Llc | Panoramic display screen assembly with integral ventilation passageways |
US20190185009A1 (en) * | 2017-12-18 | 2019-06-20 | International Business Machines Corporation | Automatic and personalized control of driver assistance components |
US10551623B1 (en) * | 2018-07-20 | 2020-02-04 | Facense Ltd. | Safe head-mounted display for vehicles |
US10694262B1 (en) * | 2019-03-12 | 2020-06-23 | Ambarella International Lp | Overlaying ads on camera feed in automotive viewing applications |
US20200282832A1 (en) * | 2017-11-17 | 2020-09-10 | Aisin Aw Co., Ltd. | Display device and computer program |
-
2020
- 2020-10-29 US US17/083,495 patent/US11318887B1/en active Active
-
2021
- 2021-05-12 CN CN202110516124.5A patent/CN114506278B/en active Active
- 2021-05-15 DE DE102021112633.9A patent/DE102021112633A1/en active Pending
Patent Citations (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20090237268A1 (en) * | 2008-03-18 | 2009-09-24 | Hyundai Motor Company | Information display system for vehicle |
US20090268093A1 (en) * | 2008-04-28 | 2009-10-29 | Honda Motor Co., Ltd. | Video Display System for a Motor Vehicle |
US20130066720A1 (en) * | 2011-09-09 | 2013-03-14 | Gordon * Howard Associates, Inc. | Method and System of Providing Information to an Occupant of a Vehicle |
US20140028980A1 (en) * | 2012-07-26 | 2014-01-30 | Cloudcar, Inc. | Vehicle content projection |
US20150042751A1 (en) * | 2013-08-12 | 2015-02-12 | GM Global Technology Operations LLC | Panorama projection unit for a motor vehicle |
US20150262435A1 (en) * | 2014-03-17 | 2015-09-17 | Hti Ip, Llc | Method and System for Providing Intelligent Alerts |
US20150296140A1 (en) * | 2014-04-11 | 2015-10-15 | Grey Matter Fusion, Inc. | Panoramic view blind spot eliminator system and method |
US20180089900A1 (en) * | 2016-09-23 | 2018-03-29 | Apple Inc. | Augmented virtual display |
US20190045117A1 (en) * | 2017-08-03 | 2019-02-07 | Ford Global Technologies, Llc | Panoramic display screen assembly with integral ventilation passageways |
US20200282832A1 (en) * | 2017-11-17 | 2020-09-10 | Aisin Aw Co., Ltd. | Display device and computer program |
US20190185009A1 (en) * | 2017-12-18 | 2019-06-20 | International Business Machines Corporation | Automatic and personalized control of driver assistance components |
US10551623B1 (en) * | 2018-07-20 | 2020-02-04 | Facense Ltd. | Safe head-mounted display for vehicles |
US10694262B1 (en) * | 2019-03-12 | 2020-06-23 | Ambarella International Lp | Overlaying ads on camera feed in automotive viewing applications |
Non-Patent Citations (1)
Title |
---|
Loz Blain, "Futurus wants to turn your car’s whole windscreen into an AR display," New Atlas, Published: December 12, 2019, Accessed at URL: https://newatlas.com/automotive/futuris-augmented-reality-windshield-hud/ (Year: 2019) * |
Also Published As
Publication number | Publication date |
---|---|
CN114506278B (en) | 2024-03-29 |
CN114506278A (en) | 2022-05-17 |
US11318887B1 (en) | 2022-05-03 |
DE102021112633A1 (en) | 2022-05-05 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9840197B2 (en) | Apparatus for providing around view and vehicle including the same | |
JP4978722B2 (en) | Speed control device | |
US10491807B2 (en) | Method to use vehicle information and sensors for photography and video viewing recording | |
US10339813B2 (en) | Display control systems and methods for a vehicle | |
US20170203692A1 (en) | Method and device for the distortion-free display of an area surrounding a vehicle | |
CN112312031A (en) | Enhanced high dynamic range imaging and tone mapping | |
US20150353010A1 (en) | Around view provision apparatus and vehicle including the same | |
US11908104B2 (en) | Weighted normalized automatic white balancing | |
US11403669B2 (en) | Vehicular advertisement providing device and vehicular advertisement providing method | |
US20220062752A1 (en) | Environment Interactive System Providing Augmented Reality for In-Vehicle Infotainment and Entertainment | |
JP7222216B2 (en) | Driving support device | |
US10421399B2 (en) | Driver alert systems and methods based on the presence of cyclists | |
US20220047951A1 (en) | In-Vehicle Gaming Systems and Methods | |
US20220207756A1 (en) | Image composition in multiview automotive and robotics systems | |
JP2020120327A (en) | Peripheral display control device | |
WO2018150642A1 (en) | Surroundings monitoring device | |
JPWO2019038914A1 (en) | Display control device, display control method, and program | |
US20230215196A1 (en) | Information processing apparatus, information processing method, and program | |
US11318887B1 (en) | Panoramic virtual environment for in-vehicle entertainment | |
US11548522B2 (en) | Speed difference indicator on head up display | |
Weber et al. | Approach for improved development of advanced driver assistance systems for future smart mobility concepts | |
US11279376B2 (en) | Vehicle control device and vehicle control method | |
JP2019079085A (en) | Drive supporting device | |
JP4493575B2 (en) | MOBILE BODY SIMULATOR, ITS CONTROL METHOD, AND CONTROL PROGRAM | |
US11348282B1 (en) | Systems and methods for calibrating vehicle cameras using external smart sensor |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: GM GLOBAL TECHNOLOGY OPERATIONS LLC, MICHIGAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ZENG, WEI;DERNOTTE, JEREMIE;LIU, KE;AND OTHERS;SIGNING DATES FROM 20201009 TO 20201029;REEL/FRAME:054208/0390 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |