EP3794563A1 - Rear view method and device using augmented reality cameras - Google Patents
Rear view method and device using augmented reality camerasInfo
- Publication number
- EP3794563A1 EP3794563A1 EP19716338.9A EP19716338A EP3794563A1 EP 3794563 A1 EP3794563 A1 EP 3794563A1 EP 19716338 A EP19716338 A EP 19716338A EP 3794563 A1 EP3794563 A1 EP 3794563A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- cameras
- vision
- retro
- camera
- scene
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 230000003190 augmentative effect Effects 0.000 title claims abstract description 10
- 238000000034 method Methods 0.000 title claims abstract description 10
- 238000009877 rendering Methods 0.000 claims description 5
- 230000007423 decrease Effects 0.000 claims description 4
- 230000003068 static effect Effects 0.000 claims description 4
- 238000009432 framing Methods 0.000 claims description 3
- 210000003128 head Anatomy 0.000 claims description 2
- 238000010276 construction Methods 0.000 claims 1
- 238000012937 correction Methods 0.000 abstract description 2
- 230000008447 perception Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 2
- 239000011521 glass Substances 0.000 description 2
- 230000001186 cumulative effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R1/00—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
- B60K35/10—Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
- B60K35/20—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor
- B60K35/21—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor using visual output, e.g. blinking lights or matrix displays
- B60K35/211—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor using visual output, e.g. blinking lights or matrix displays producing three-dimensional [3D] effects, e.g. stereoscopic images
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
- B60K35/20—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor
- B60K35/28—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
- B60K2360/20—Optical features of instruments
- B60K2360/21—Optical features of instruments using cameras
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/10—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used
- B60R2300/105—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used using multiple cameras
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/10—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used
- B60R2300/107—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used using stereoscopic cameras
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/30—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing
- B60R2300/304—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing using merged images, e.g. merging camera image with stored images
Definitions
- the present invention relates to retro vision cameras, on a motor vehicle.
- Mirror mirrors present the driver with an external vision that is framed in relation to the vehicle's outline and its interior context. On the exterior mirrors, we see most often part of the wing of the vehicle, or its door, for example the handle. On the rearview mirror, we see the rear window, the headrests. Mirror technologies also restore the three dimensions of the landscape, because the images are in direct vision for each eye.
- the retro-vision cameras eliminate blind spots, and the limitation of the field of vision by passengers, or bulky objects.
- representations can confuse drivers, deprived of visual cues to locate their vehicle and estimate distances.
- FR 2 780 230 discloses a device for retro vision by camera, delivering on the same screen a retro-vision image, and additional information provided by the engine control computer of the vehicle.
- This device automatically selects all or part of the retro-vision image, to display additional information on the screen, with dynamic framing.
- Certain driving information appears on the screen, without being embedded in the image, so as to limit the risk of confusion of the image with certain symbols displayed.
- the driver has no additional mark, relative to his environment.
- Several studies have highlighted the difficulties experienced by users with the camera back-vision, because the images returned are devoid of references on the size of the vehicle, or on its location in the environment.
- the present invention aims to overcome the disadvantages of known camera retro-vision systems.
- the device comprises a set of retro-vision cameras 1, arranged in pairs, a static or dynamic software mask, specific to each camera set, and at least one rendering screen 2.
- FIG. 1 is a diagram of an on-vehicle camera-based retro vision device
- FIG. 2 illustrates the method of the invention
- FIG. 1 diagrammatically shows a motor vehicle having a system for acquiring images by double cameras 1, lateral and rear, and a set of image reproduction screens 2, auto stereoscopic preference.
- the associated processing means comprise a retro-vision point-of-view computer A, a memory B of 3D (three-dimensional) models, texture and layout, an augmented reality image calculator C (pixels, light, user settings), and a graphic distribution unit D to the screens.
- a set of retro-vision cameras 1 arranged in pairs, for example two rear cameras, and two side cameras,
- a set of reproduction screens 2 which may be conventional (LCD, LED, TN) but preferably auto-stereoscopic,
- a user parameter interface and possibly a system for acquiring the position of the driver's eyes.
- the purpose of the mask is to add to the retro-vision some markers usually represented on mirrored mirrors (rear windows, rear headrests, wings, handles, etc.), in order to give the driver clues as to the point of view of the spectators. cameras.
- the system also provides a three-dimensional vision to maximize spatial perception.
- the software mask is preferably a graphic overlay, which is applied to the stream, of camera images, to superimpose informations.
- the proposed method of retro vision is based on the selection in the field of view of the cameras, areas corresponding to the usual retro vision mirrors mirrors.
- the shots are subject to the necessary corrections to restore the driver on a screen, a scene built by grouping the different shots of the cameras.
- the invention proposes to superimpose on the image of the outdoor scene, some reference marks of the vehicle.
- the mask brings to the user the cumulative benefits of retro-vision cameras and mirrors, without their respective disadvantages. Thanks to the mask the retro-vision device renders a complete scene of the external environment, without dead angle and without occultation of the field of vision by objects or occupants of the vehicle, while enriching it with physical references of the vehicle, to facilitate spatial perception.
- the invention provides for the insertion of three-dimensional digital models adapted to the vehicle, in the scene constructed from the cameras, and to construct an augmented reality scene, by superimposing the three-dimensional models, to the scene constructed from cameras.
- the device includes
- At least one rendering screen At least one rendering screen.
- the technology of the "auto-stereoscopic 3D” allows the driver to see directly inserted elements in three dimensions, without special equipment, such as 3D glasses.
- the depth of field provided by three-dimensional rendering helps to reinforce spatial perception.
- the cameras can advantageously be placed in their usual location on the vehicle. They are arranged in pairs, spaced apart by about 65 mm, because this distance corresponds to an average difference between two eyes, and makes it possible to restore a stereoscopic image to the user.
- a position calculator for the digital models able to work with the driver's eye positioning system to adapt the image to the latter;
- -a graphical unit able to display in 3D the numerical models, with a sufficient resolution, with or without guelgues calculations in real time to increase the realism of the scene: shadows, ambient brightness, incident lights, etc.
- the development of the masgue is done in real time with a permanent refresh. It applies to the image in several steps.
- the first step is the framing. It consists in selecting in the field the zone which corresponds to the usual retro vision of mirror mirrors, possibly applying some deformations to it if they are necessary to constitute an overview.
- the second step is the acguisition of the models. It consists of placing digital maguettes adapted to the vehicle and inserting them into the scene constructed by the cameras. In general, models reguis are headrests, doors, etc.
- the third step in building the augmented reality scene is to render the 3D models superimposed on the cameras.
- the calculator takes into account the user selected setting, and the position of the head of the user (either an average position or a precise position if the vehicle is equipped with the optional eye-tracking system).
- the return screens are preferably auto-stereoscopic, to display the content in three dimensions, without special glasses or other accessory.
- the interface is available to the driver at his driving position. Thanks to him, he can parameterize the retro vision, by regulating the transparency of the digital inserts, or the quantity of the mechanical inserts.
- the adjustment interface can advantageously be integrated into the software of a pre-existing navigation and guidance console. If the console has a touch screen, it facilitates adjustments.
- the transparency of the inserts varies depending on the external environment, for example the presence of a vehicle in a potentially hidden area, a vehicle during parking, etc.).
- the depth of the stereoscopy is also adjustable.
- Figure 3 shows an example, an image constructed from a rear camera view and two side cameras.
- the software mask does not appear on the image (100% transparency over the entire surface); on the second 12, the mask decreases the transparency of the image by 60% in the area concerned; on the third 13, this decrease is 80%.
- the invention eliminates blind spots, and improves safety. Compared to a conventional camera system, it facilitates the learning and handling phase by the driver. The location of the eyes is optional because we can content with an average position. Finally, application of the invention is not limited to motor vehicles. She also finds all her interest in other means of transport using cameras.
Landscapes
- Engineering & Computer Science (AREA)
- Mechanical Engineering (AREA)
- Transportation (AREA)
- Combustion & Propulsion (AREA)
- Chemical & Material Sciences (AREA)
- Multimedia (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Signal Processing (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computer Hardware Design (AREA)
- Computer Graphics (AREA)
- Processing Or Creating Images (AREA)
- Closed-Circuit Television Systems (AREA)
- Image Processing (AREA)
Abstract
Description
Claims
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
FR1854057A FR3081250B1 (en) | 2018-05-15 | 2018-05-15 | PROCESS AND DEVICE FOR FEEDBACK BY AUGMENTED REALITY CAMERAS |
PCT/EP2019/058316 WO2019219288A1 (en) | 2018-05-15 | 2019-04-02 | Rear view method and device using augmented reality cameras |
Publications (1)
Publication Number | Publication Date |
---|---|
EP3794563A1 true EP3794563A1 (en) | 2021-03-24 |
Family
ID=63722482
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP19716338.9A Withdrawn EP3794563A1 (en) | 2018-05-15 | 2019-04-02 | Rear view method and device using augmented reality cameras |
Country Status (5)
Country | Link |
---|---|
EP (1) | EP3794563A1 (en) |
KR (1) | KR102490465B1 (en) |
CN (1) | CN112204626A (en) |
FR (1) | FR3081250B1 (en) |
WO (1) | WO2019219288A1 (en) |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114419949B (en) * | 2022-01-13 | 2022-12-06 | 武汉未来幻影科技有限公司 | Automobile rearview mirror image reconstruction method and rearview mirror |
WO2024043588A1 (en) * | 2022-08-26 | 2024-02-29 | 엘지전자 주식회사 | Device for providing augmented reality information on basis of digital mirror and method for controlling device |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5670935A (en) * | 1993-02-26 | 1997-09-23 | Donnelly Corporation | Rearview vision system for vehicle including panoramic view |
FR2780230B1 (en) | 1998-06-18 | 2000-08-25 | Renault | MAIN REVIEW DEVICE FOR A MOTOR VEHICLE |
US20060262140A1 (en) * | 2005-05-18 | 2006-11-23 | Kujawa Gregory A | Method and apparatus to facilitate visual augmentation of perceived reality |
KR101544524B1 (en) * | 2010-12-16 | 2015-08-17 | 한국전자통신연구원 | Display system for augmented reality in vehicle, and method for the same |
US10475242B2 (en) * | 2014-01-10 | 2019-11-12 | Aisin Seiki Kabushiki Kaisha | Image display control device and image display system including image superimposition unit that superimposes a mirror image and a vehicle-body image |
JP5562498B1 (en) | 2014-03-04 | 2014-07-30 | サカエ理研工業株式会社 | Room mirror, vehicle blind spot support device using the room mirror, and display image adjustment method of the room mirror or vehicle blind spot support device |
WO2017200950A1 (en) * | 2016-05-19 | 2017-11-23 | Reald Spark, Llc | Wide angle imaging directional backlights |
JP6707007B2 (en) | 2016-09-27 | 2020-06-10 | 株式会社東海理化電機製作所 | Vehicle visibility device and vehicle visual image display method |
-
2018
- 2018-05-15 FR FR1854057A patent/FR3081250B1/en active Active
-
2019
- 2019-04-02 CN CN201980035663.8A patent/CN112204626A/en active Pending
- 2019-04-02 KR KR1020207035268A patent/KR102490465B1/en active IP Right Grant
- 2019-04-02 WO PCT/EP2019/058316 patent/WO2019219288A1/en unknown
- 2019-04-02 EP EP19716338.9A patent/EP3794563A1/en not_active Withdrawn
Also Published As
Publication number | Publication date |
---|---|
CN112204626A (en) | 2021-01-08 |
KR102490465B1 (en) | 2023-01-20 |
WO2019219288A1 (en) | 2019-11-21 |
FR3081250B1 (en) | 2021-08-13 |
KR20210008503A (en) | 2021-01-22 |
FR3081250A1 (en) | 2019-11-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP2914002B1 (en) | Virtual see-through instrument cluster with live video | |
Azuma | Augmented reality: Approaches and technical challenges | |
US20070279493A1 (en) | Recording medium, parking support apparatus and parking support screen | |
WO2005088970A1 (en) | Image generation device, image generation method, and image generation program | |
US20150325052A1 (en) | Image superposition of virtual objects in a camera image | |
KR20170135952A (en) | A method for displaying a peripheral area of a vehicle | |
JP2017191378A (en) | Augmented reality information display device and augmented reality information display method | |
EP2572164A1 (en) | Navigational aid having improved processing of display data | |
WO2019219288A1 (en) | Rear view method and device using augmented reality cameras | |
US9918066B2 (en) | Methods and systems for producing a magnified 3D image | |
WO2016102304A1 (en) | Method for presenting an image overlay element in an image with 3d information, driver assistance system and motor vehicle | |
EP3355277A1 (en) | Method for displaying on a screen at least one representation of an object, related computer program, electronic display device and apparatus | |
US10088678B1 (en) | Holographic illustration of weather | |
Poitschke et al. | Contact-analog information representation in an automotive head-up display | |
EP2469868B1 (en) | Method for correcting hyperstereoscopy and related helmet display system | |
JP2005269010A (en) | Image creating device, program and method | |
WO2019026747A1 (en) | Augmented real image display device for vehicle | |
JP2019047296A (en) | Display device, display method, control program, and electronic mirror system | |
WO2017081915A1 (en) | Image processing device, image processing method and program | |
JP6813437B2 (en) | Display system | |
CN117193530B (en) | Intelligent cabin immersive user experience method and system based on virtual reality technology | |
EP3833574B1 (en) | Method for providing an image representation of at least part of an environment of a vehicle, computer program product and driver assistance system | |
KR20180108006A (en) | Apparatus and method for trick art display | |
WO2024132688A1 (en) | Display device and method carried out in a display device | |
CA3018465C (en) | See-through based display method and system for simulators |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: UNKNOWN |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20201106 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
DAV | Request for validation of the european patent (deleted) | ||
DAX | Request for extension of the european patent (deleted) | ||
RAP3 | Party data changed (applicant data changed or rights of an application transferred) |
Owner name: RENAULT S.A.S |
|
RAP3 | Party data changed (applicant data changed or rights of an application transferred) |
Owner name: RENAULT S.A.S |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
17Q | First examination report despatched |
Effective date: 20230612 |
|
P01 | Opt-out of the competence of the unified patent court (upc) registered |
Effective date: 20230608 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN |
|
18D | Application deemed to be withdrawn |
Effective date: 20231223 |