WO2019219288A1 - Rear view method and device using augmented reality cameras - Google Patents

Rear view method and device using augmented reality cameras Download PDF

Info

Publication number
WO2019219288A1
WO2019219288A1 PCT/EP2019/058316 EP2019058316W WO2019219288A1 WO 2019219288 A1 WO2019219288 A1 WO 2019219288A1 EP 2019058316 W EP2019058316 W EP 2019058316W WO 2019219288 A1 WO2019219288 A1 WO 2019219288A1
Authority
WO
WIPO (PCT)
Prior art keywords
cameras
vision
retro
camera
scene
Prior art date
Application number
PCT/EP2019/058316
Other languages
French (fr)
Inventor
Antoine LASSAGNE
Marie Pierre BOUSSIER
Laurent TEXIER
Original Assignee
Renault S.A.S
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Renault S.A.S filed Critical Renault S.A.S
Priority to EP19716338.9A priority Critical patent/EP3794563A1/en
Priority to KR1020207035268A priority patent/KR102490465B1/en
Priority to CN201980035663.8A priority patent/CN112204626A/en
Publication of WO2019219288A1 publication Critical patent/WO2019219288A1/en

Links

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R1/00—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
    • B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
    • B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
    • B60K35/10—Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
    • B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
    • B60K35/20—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor
    • B60K35/21—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor using visual output, e.g. blinking lights or matrix displays
    • B60K35/211—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor using visual output, e.g. blinking lights or matrix displays producing three-dimensional [3D] effects, e.g. stereoscopic images
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
    • B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
    • B60K35/20—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor
    • B60K35/28—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
    • B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
    • B60K2360/20—Optical features of instruments
    • B60K2360/21—Optical features of instruments using cameras
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/10—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used
    • B60R2300/105—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used using multiple cameras
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/10—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used
    • B60R2300/107—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used using stereoscopic cameras
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/30—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing
    • B60R2300/304—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing using merged images, e.g. merging camera image with stored images

Definitions

  • the present invention relates to retro vision cameras, on a motor vehicle.
  • Mirror mirrors present the driver with an external vision that is framed in relation to the vehicle's outline and its interior context. On the exterior mirrors, we see most often part of the wing of the vehicle, or its door, for example the handle. On the rearview mirror, we see the rear window, the headrests. Mirror technologies also restore the three dimensions of the landscape, because the images are in direct vision for each eye.
  • the retro-vision cameras eliminate blind spots, and the limitation of the field of vision by passengers, or bulky objects.
  • representations can confuse drivers, deprived of visual cues to locate their vehicle and estimate distances.
  • FR 2 780 230 discloses a device for retro vision by camera, delivering on the same screen a retro-vision image, and additional information provided by the engine control computer of the vehicle.
  • This device automatically selects all or part of the retro-vision image, to display additional information on the screen, with dynamic framing.
  • Certain driving information appears on the screen, without being embedded in the image, so as to limit the risk of confusion of the image with certain symbols displayed.
  • the driver has no additional mark, relative to his environment.
  • Several studies have highlighted the difficulties experienced by users with the camera back-vision, because the images returned are devoid of references on the size of the vehicle, or on its location in the environment.
  • the present invention aims to overcome the disadvantages of known camera retro-vision systems.
  • the device comprises a set of retro-vision cameras 1, arranged in pairs, a static or dynamic software mask, specific to each camera set, and at least one rendering screen 2.
  • FIG. 1 is a diagram of an on-vehicle camera-based retro vision device
  • FIG. 2 illustrates the method of the invention
  • FIG. 1 diagrammatically shows a motor vehicle having a system for acquiring images by double cameras 1, lateral and rear, and a set of image reproduction screens 2, auto stereoscopic preference.
  • the associated processing means comprise a retro-vision point-of-view computer A, a memory B of 3D (three-dimensional) models, texture and layout, an augmented reality image calculator C (pixels, light, user settings), and a graphic distribution unit D to the screens.
  • a set of retro-vision cameras 1 arranged in pairs, for example two rear cameras, and two side cameras,
  • a set of reproduction screens 2 which may be conventional (LCD, LED, TN) but preferably auto-stereoscopic,
  • a user parameter interface and possibly a system for acquiring the position of the driver's eyes.
  • the purpose of the mask is to add to the retro-vision some markers usually represented on mirrored mirrors (rear windows, rear headrests, wings, handles, etc.), in order to give the driver clues as to the point of view of the spectators. cameras.
  • the system also provides a three-dimensional vision to maximize spatial perception.
  • the software mask is preferably a graphic overlay, which is applied to the stream, of camera images, to superimpose informations.
  • the proposed method of retro vision is based on the selection in the field of view of the cameras, areas corresponding to the usual retro vision mirrors mirrors.
  • the shots are subject to the necessary corrections to restore the driver on a screen, a scene built by grouping the different shots of the cameras.
  • the invention proposes to superimpose on the image of the outdoor scene, some reference marks of the vehicle.
  • the mask brings to the user the cumulative benefits of retro-vision cameras and mirrors, without their respective disadvantages. Thanks to the mask the retro-vision device renders a complete scene of the external environment, without dead angle and without occultation of the field of vision by objects or occupants of the vehicle, while enriching it with physical references of the vehicle, to facilitate spatial perception.
  • the invention provides for the insertion of three-dimensional digital models adapted to the vehicle, in the scene constructed from the cameras, and to construct an augmented reality scene, by superimposing the three-dimensional models, to the scene constructed from cameras.
  • the device includes
  • At least one rendering screen At least one rendering screen.
  • the technology of the "auto-stereoscopic 3D” allows the driver to see directly inserted elements in three dimensions, without special equipment, such as 3D glasses.
  • the depth of field provided by three-dimensional rendering helps to reinforce spatial perception.
  • the cameras can advantageously be placed in their usual location on the vehicle. They are arranged in pairs, spaced apart by about 65 mm, because this distance corresponds to an average difference between two eyes, and makes it possible to restore a stereoscopic image to the user.
  • a position calculator for the digital models able to work with the driver's eye positioning system to adapt the image to the latter;
  • -a graphical unit able to display in 3D the numerical models, with a sufficient resolution, with or without guelgues calculations in real time to increase the realism of the scene: shadows, ambient brightness, incident lights, etc.
  • the development of the masgue is done in real time with a permanent refresh. It applies to the image in several steps.
  • the first step is the framing. It consists in selecting in the field the zone which corresponds to the usual retro vision of mirror mirrors, possibly applying some deformations to it if they are necessary to constitute an overview.
  • the second step is the acguisition of the models. It consists of placing digital maguettes adapted to the vehicle and inserting them into the scene constructed by the cameras. In general, models reguis are headrests, doors, etc.
  • the third step in building the augmented reality scene is to render the 3D models superimposed on the cameras.
  • the calculator takes into account the user selected setting, and the position of the head of the user (either an average position or a precise position if the vehicle is equipped with the optional eye-tracking system).
  • the return screens are preferably auto-stereoscopic, to display the content in three dimensions, without special glasses or other accessory.
  • the interface is available to the driver at his driving position. Thanks to him, he can parameterize the retro vision, by regulating the transparency of the digital inserts, or the quantity of the mechanical inserts.
  • the adjustment interface can advantageously be integrated into the software of a pre-existing navigation and guidance console. If the console has a touch screen, it facilitates adjustments.
  • the transparency of the inserts varies depending on the external environment, for example the presence of a vehicle in a potentially hidden area, a vehicle during parking, etc.).
  • the depth of the stereoscopy is also adjustable.
  • Figure 3 shows an example, an image constructed from a rear camera view and two side cameras.
  • the software mask does not appear on the image (100% transparency over the entire surface); on the second 12, the mask decreases the transparency of the image by 60% in the area concerned; on the third 13, this decrease is 80%.
  • the invention eliminates blind spots, and improves safety. Compared to a conventional camera system, it facilitates the learning and handling phase by the driver. The location of the eyes is optional because we can content with an average position. Finally, application of the invention is not limited to motor vehicles. She also finds all her interest in other means of transport using cameras.

Landscapes

  • Engineering & Computer Science (AREA)
  • Mechanical Engineering (AREA)
  • Transportation (AREA)
  • Combustion & Propulsion (AREA)
  • Chemical & Material Sciences (AREA)
  • Multimedia (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Signal Processing (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Computer Graphics (AREA)
  • Processing Or Creating Images (AREA)
  • Closed-Circuit Television Systems (AREA)
  • Image Processing (AREA)

Abstract

The invention relates to a rear view method using cameras for a vehicle, based on selecting, in the field of view of the cameras, zones corresponding to the regular rear view of rear view mirrors, with the necessary corrections for reproducing to the driver, on a screen, a scene constructed by grouping together various views of the cameras, characterised in that: three-dimensional digital models adapted to the vehicle are inserted into the rear view scene constructed from the cameras; and an augmented reality scene is constructed, superimposing the three-dimensional models on the scene constructed from the cameras.

Description

PROCEDE ET DISPOSITIF DE RETRO VISION PAR CAMERAS A  METHOD AND DEVICE FOR RETRO VISION BY CAMERAS A
REALITE AUGMENTEE  AUGMENTED REALITY
La présente invention concerne la rétro vision par caméras, sur véhicule automobile. The present invention relates to retro vision cameras, on a motor vehicle.
Plus précisément, elle a pour objet un procédé de rétro vision par caméras pour véhicule, restituant au conducteur sur un écran, une scène construite par regroupement de prises de vues.  More specifically, it relates to a method of retro vision camera for vehicle, restoring the driver on a screen, a scene built by grouping shots.
L'utilisation de caméras de rétro vision numérique, avec un écran de retour dans l'habitacle, offre davantage de possibilités pour adapter le champ de vision au profil extérieur du véhicule, que les miroirs de rétro vision classique. Les caméras assurent notamment au conducteur un large champ de vision, vers l'arrière et les côtés du véhicule. Cependant, il est difficile, pour certaines personnes, d'évaluer des distances relatives sur un écran, comme la proximité latérale ou longitudinale des obstacles, des vitesses relatives, etc .  The use of digital retro-vision cameras, with a screen back in the cockpit, offers more possibilities to adapt the field of vision to the exterior profile of the vehicle, than classic retro-vision mirrors. In particular, the cameras provide the driver with a wide field of vision, towards the rear and the sides of the vehicle. However, it is difficult for some people to evaluate relative distances on a screen, such as lateral or longitudinal proximity of obstacles, relative speeds, etc.
Les rétroviseurs à miroirs présentent au conducteur une vision extérieure cadrée par rapport au contour du véhicule et à son contexte intérieur. Sur les rétroviseurs extérieurs, on voit le plus souvent une partie de l'aile du véhicule, ou de sa portière, par exemple la poignée. Sur le rétroviseur intérieur, on voit la lunette arrière, les appuie-têtes . De plus, les technologies à miroirs restituent les trois dimensions du paysage, car les images sont en vision directe pour chaque œil .  Mirror mirrors present the driver with an external vision that is framed in relation to the vehicle's outline and its interior context. On the exterior mirrors, we see most often part of the wing of the vehicle, or its door, for example the handle. On the rearview mirror, we see the rear window, the headrests. Mirror technologies also restore the three dimensions of the landscape, because the images are in direct vision for each eye.
Les caméras de rétro vision suppriment les angles morts, et la limitation du champ de vision par les passagers, ou des objets encombrants . Cependant les représentations peuvent dérouter des conducteurs, privés de repères visuels pour situer leur véhicule et estimer les distances .  The retro-vision cameras eliminate blind spots, and the limitation of the field of vision by passengers, or bulky objects. However, representations can confuse drivers, deprived of visual cues to locate their vehicle and estimate distances.
Par la publication FR 2 780 230, on connaît un dispositif de rétro vision par caméra, délivrant sur un même écran une image de rétro vision, et des informations complémentaires fournies par le calculateur de contrôle moteur du véhicule. Ce dispositif sélectionne automatiquement tout ou partie de l'image de rétro vision, pour afficher des informations additionnelles sur l'écran, avec un cadrage dynamique. Certaines informations de conduite apparaissent sur l'écran, sans être incrustées sur l'image, de manière à limiter les risques de confusion de l' image avec certains symboles affichés. Toutefois, le conducteur ne dispose d'aucun repère supplémentaire, par rapport à son environnement. Plusieurs études ont mis en évidence les difficultés rencontrées par des utilisateurs avec la rétro vision par caméra, car les images restituées sont dépourvues de repères sur l'encombrement du véhicule, ou sur son emplacement dans l'environnement. FR 2 780 230 discloses a device for retro vision by camera, delivering on the same screen a retro-vision image, and additional information provided by the engine control computer of the vehicle. This device automatically selects all or part of the retro-vision image, to display additional information on the screen, with dynamic framing. Certain driving information appears on the screen, without being embedded in the image, so as to limit the risk of confusion of the image with certain symbols displayed. However, the driver has no additional mark, relative to his environment. Several studies have highlighted the difficulties experienced by users with the camera back-vision, because the images returned are devoid of references on the size of the vehicle, or on its location in the environment.
La présente invention vise à remédier aux inconvénients des systèmes de rétro vision par caméras connus.  The present invention aims to overcome the disadvantages of known camera retro-vision systems.
Dans ce but, elle propose de créer sur une image de rétro vision, des éléments fantômes représentant les parties arrière du véhicule, pour aider l'utilisateur à retrouver les repères des rétroviseurs classiques à miroirs.  For this purpose, it proposes to create on a picture of retro vision, ghost elements representing the rear parts of the vehicle, to help the user to find the benchmarks of mirrors classic mirrors.
Ce résultat est obtenu grâce à des repères physiques en réalité augmentée :  This result is obtained thanks to physical references in augmented reality:
- on insère des maquettes numériques tridimensionnelles adaptées au véhicule, dans la scène construite à partir des caméras, et  - three-dimensional digital mockups adapted to the vehicle are inserted in the scene constructed from the cameras, and
-on construit une scène de réalité augmentée, en superposant les maquettes en trois dimensions, à la scène construite à partir des caméras .  -We built a scene of augmented reality, by superimposing the three-dimensional models, to the scene built from the cameras.
Le dispositif comprend un ensemble de caméras de rétro vision 1, disposées par paires, un masque logiciel, statique ou dynamique, spécifique à chaque jeu de caméra, et au moins un écran de restitution 2.  The device comprises a set of retro-vision cameras 1, arranged in pairs, a static or dynamic software mask, specific to each camera set, and at least one rendering screen 2.
La présente invention sera mieux comprise à la lecture de la description suivante d'un mode de réalisation non limitatif de celle- ci, en se reportant aux dessins annexés, sur lesquels : - la figure 1 est un schéma de dispositif de rétro vision par caméra embarqué sur véhicule, The present invention will be better understood on reading the following description of a non-limiting embodiment thereof, with reference to the appended drawings, in which: FIG. 1 is a diagram of an on-vehicle camera-based retro vision device,
- la figure 2 illustre le procédé de l'invention, et  FIG. 2 illustrates the method of the invention, and
- la figure 3 illustre le résultat obtenu.  - Figure 3 illustrates the result obtained.
Sur la figure 1, on a représenté de façon schématique un véhicule automobile disposant d'un système d'acquisition d'images par doubles caméras 1, latérales et arrière, et d'un ensemble d'écrans de restitution d'images 2, de préférence auto stéréoscopiques. Les moyens de traitement associés comprennent un calculateur de point de vue de rétro vision A, une mémoire B de modèles 3D (en trois dimensions), de texture et d'agencement, un calculateur C d'image de réalité augmentée (pixels, lumière, paramétrages de l'utilisateur), et une unité de distribution graphique D vers les écrans .  FIG. 1 diagrammatically shows a motor vehicle having a system for acquiring images by double cameras 1, lateral and rear, and a set of image reproduction screens 2, auto stereoscopic preference. The associated processing means comprise a retro-vision point-of-view computer A, a memory B of 3D (three-dimensional) models, texture and layout, an augmented reality image calculator C (pixels, light, user settings), and a graphic distribution unit D to the screens.
Ces composants permettent de constituer un dispositif de rétro vision par caméra disposant de :  These components make it possible to constitute a device of retro vision by camera having:
- un ensemble de caméras de rétro vision 1, disposées par paires, par exemple deux caméras arrière, et deux caméras latérales, a set of retro-vision cameras 1, arranged in pairs, for example two rear cameras, and two side cameras,
- un masque logiciel, statique ou dynamique, spécifique à chaque jeu de caméras, a static or dynamic software mask, specific to each camera set,
- un ensemble d'écrans de restitution 2, qui peuvent être classiques (LCD, LED, TN) mais de préférence auto-stéréoscopiques, a set of reproduction screens 2, which may be conventional (LCD, LED, TN) but preferably auto-stereoscopic,
- une interface de paramétrage utilisateur, et éventuellement un système d'acquisition de la position des yeux du conducteur . a user parameter interface, and possibly a system for acquiring the position of the driver's eyes.
Le masque a pour objectif de rajouter à la rétro vision, quelques repères habituellement représentés sur les rétroviseurs à miroir (lunettes arrière, appuis têtes arrière, ailes, poignées, etc.), afin de donner au conducteur des indices sur le point de vue des caméras. Le système apporte aussi une vision en trois dimensions, pour maximiser la perception spatiale. Dans le cadre de l'invention, le masque logiciel est de préférence une surcouche graphique, qui est appliquée sur le flux, d'images de caméras, pour y superposer des informations. L'application d'éléments virtuels, tels que des repères sur le flux réel d'images des caméras, améliore la perception de la scène filmée en augmentant son réalisme. The purpose of the mask is to add to the retro-vision some markers usually represented on mirrored mirrors (rear windows, rear headrests, wings, handles, etc.), in order to give the driver clues as to the point of view of the spectators. cameras. The system also provides a three-dimensional vision to maximize spatial perception. In the context of the invention, the software mask is preferably a graphic overlay, which is applied to the stream, of camera images, to superimpose informations. The application of virtual elements, such as landmarks on the actual flow of images from the cameras, improves the perception of the filmed scene by increasing its realism.
Le procédé de rétro vision proposé, repose sur la sélection dans le champ de prise de vue des caméras, de zones correspondant à la rétro vision habituelle de rétroviseurs à miroirs. Les prises de vue sont soumises aux corrections nécessaires pour restituer au conducteur sur un écran, une scène construite par regroupement des différentes prises de vues des caméras.  The proposed method of retro vision is based on the selection in the field of view of the cameras, areas corresponding to the usual retro vision mirrors mirrors. The shots are subject to the necessary corrections to restore the driver on a screen, a scene built by grouping the different shots of the cameras.
L'invention propose de superposer sur l'image de la scène extérieure, quelques repères du véhicule. Le masque apporte à l'utilisateur les avantages cumulés de la rétro vision par caméras et par miroirs, sans leurs inconvénients respectifs. Grâce au masque le dispositif de rétro vision restitue une scène complète de l'environnement extérieur, sans angle mort et sans occultation du champ de vision par des objets ou des occupants du véhicule, tout en l'enrichissant de repères physiques du véhicule, pour faciliter la perception spatiale.  The invention proposes to superimpose on the image of the outdoor scene, some reference marks of the vehicle. The mask brings to the user the cumulative benefits of retro-vision cameras and mirrors, without their respective disadvantages. Thanks to the mask the retro-vision device renders a complete scene of the external environment, without dead angle and without occultation of the field of vision by objects or occupants of the vehicle, while enriching it with physical references of the vehicle, to facilitate spatial perception.
En résumé, l'invention prévoit d'insérer des maquettes numériques tridimensionnelles adaptées au véhicule, dans la scène construite à partir des caméras, et de construire une scène de réalité augmentée, en superposant les maquettes en trois dimensions, à la scène construite à partir des caméras. Le dispositif comprend  In summary, the invention provides for the insertion of three-dimensional digital models adapted to the vehicle, in the scene constructed from the cameras, and to construct an augmented reality scene, by superimposing the three-dimensional models, to the scene constructed from cameras. The device includes
- un ensemble de caméras de rétro vision 1, disposées par paires , a set of retro-vision cameras 1, arranged in pairs,
- un masque logiciel, statique ou dynamique, spécifique à chaque jeu de caméra, et  a static or dynamic software mask, specific to each camera set, and
-au moins un écran de restitution.  at least one rendering screen.
La technologie de la « 3D auto-stéréoscopique » permet au conducteur de voir directement les éléments insérés en trois dimensions, sans équipements particuliers, tels que des lunettes 3D. La profondeur de champ assurée par la restitution en trois dimensions, contribue à renforcer la perception spatiale. The technology of the "auto-stereoscopic 3D" allows the driver to see directly inserted elements in three dimensions, without special equipment, such as 3D glasses. The depth of field provided by three-dimensional rendering helps to reinforce spatial perception.
Les caméras peuvent avantageusement être placées à leur emplacement habituel sur le véhicule. Elles sont disposées par paires, espacées d' environ 65mm, car cette distance correspond à un écart moyen entre deux yeux, et permet de restituer une image stéréoscopigue à 1' utilisateur .  The cameras can advantageously be placed in their usual location on the vehicle. They are arranged in pairs, spaced apart by about 65 mm, because this distance corresponds to an average difference between two eyes, and makes it possible to restore a stereoscopic image to the user.
Les autres ressources contribuant à la mise en œuvre de l'invention sont les suivantes :  Other resources contributing to the implementation of the invention are the following:
-une mémoire interne contenant les modèles numérigues texturés du véhicule, gui peuvent être superposés à la rétro vision ;  an internal memory containing the textured digital models of the vehicle, which can be superimposed on the retro vision;
-un calculateur de position des modèles numérigues, pouvant travailler avec le système de localisation des yeux du conducteur pour adapter l'image à celle-ci ;  a position calculator for the digital models, able to work with the driver's eye positioning system to adapt the image to the latter;
-une unité graphigue capable d'afficher en 3D les modèles numérigues, avec une résolution suffisante, avec ou sans guelgues calculs en temps réel pour augmenter le réalisme de la scène : ombres, luminosité ambiante, lumières incidentes, etc.  -a graphical unit able to display in 3D the numerical models, with a sufficient resolution, with or without guelgues calculations in real time to increase the realism of the scene: shadows, ambient brightness, incident lights, etc.
L'élaboration du masgue s'effectue en temps réel avec un rafraîchissement permanent. Il s'appligue sur l'image en plusieurs étapes .  The development of the masgue is done in real time with a permanent refresh. It applies to the image in several steps.
La première étape est le cadrage. Il consiste à sélectionner dans le champ la zone gui correspond à la rétro vision habituelle des rétroviseurs à miroirs, en lui appliguant éventuellement certaines déformations si elles sont nécessaires pour constituer une vue d'ensemble.  The first step is the framing. It consists in selecting in the field the zone which corresponds to the usual retro vision of mirror mirrors, possibly applying some deformations to it if they are necessary to constitute an overview.
La deuxième étape est l' acguisition des modèles. Elle consiste à placer des maguettes numérigues adaptées au véhicule et à les insérer dans la scène construite par les caméras. En général, les modèles reguis sont des appui-têtes, des portières, etc.  The second step is the acguisition of the models. It consists of placing digital maguettes adapted to the vehicle and inserting them into the scene constructed by the cameras. In general, models reguis are headrests, doors, etc.
La troisième étape de construction de la scène de réalité augmentée, consiste à effectuer le rendu des modèles 3D superposé à celui des caméras. Pour cela, le calculateur tient compte du paramétrage sélectionné par l'utilisateur, et de la position de la tête de celui-ci (soit une position moyenne, soit une position précise si l'on équipe le véhicule du système de localisation des yeux facultatif) . The third step in building the augmented reality scene is to render the 3D models superimposed on the cameras. For this, the calculator takes into account the user selected setting, and the position of the head of the user (either an average position or a precise position if the vehicle is equipped with the optional eye-tracking system).
Les écrans de retour sont de préférence auto-stéréoscopiques, pour visualiser le contenu en trois dimensions, sans lunettes particulières ou autre accessoire.  The return screens are preferably auto-stereoscopic, to display the content in three dimensions, without special glasses or other accessory.
L' interface est à la disposition du conducteur sur son poste de conduite. Grâce à lui, il peut paramétrer la rétro vision, en réglant la transparence des inserts numériques, ou la quantité des inserts mécaniques. L'interface de réglage peut avantageusement s'intégrer dans le logiciel d'une console de navigation et de guidage préexistante. Si la console dispose d'un écran tactile, celui-ci facilite les réglages .  The interface is available to the driver at his driving position. Thanks to him, he can parameterize the retro vision, by regulating the transparency of the digital inserts, or the quantity of the mechanical inserts. The adjustment interface can advantageously be integrated into the software of a pre-existing navigation and guidance console. If the console has a touch screen, it facilitates adjustments.
Dans un mode de réalisation particulier de l'invention, dit de « de transparence dynamique », la transparence des inserts varie en fonction de l'environnement extérieur, par exemple la présence d'un véhicule dans une zone potentiellement masquée, d'un véhicule en cours de stationnement, etc.) . Enfin, il est préférable que la profondeur de la stéréoscopie soit également réglable.  In a particular embodiment of the invention, called "dynamic transparency", the transparency of the inserts varies depending on the external environment, for example the presence of a vehicle in a potentially hidden area, a vehicle during parking, etc.). Finally, it is preferable that the depth of the stereoscopy is also adjustable.
La superposition du masque sur l'image, diminue la transparence de celle-ci dans la zone concernée. La suppression de la luminosité peut être partielle ou totale. La figure 3 montre sur un exemple, une image construite à partir d'une vue par caméra arrière et deux caméras latérales. Sur la première vue II, le masque logiciel n'apparaît pas sur l'image (100% de transparence sur toute la surface) ; sur la deuxième 12, le masque diminue la transparence de l'image de 60% dans la zone concernée ; sur la troisième 13, cette diminution est de 80%.  The superposition of the mask on the image decreases the transparency of the image in the area concerned. The removal of brightness can be partial or total. Figure 3 shows an example, an image constructed from a rear camera view and two side cameras. On the first view II, the software mask does not appear on the image (100% transparency over the entire surface); on the second 12, the mask decreases the transparency of the image by 60% in the area concerned; on the third 13, this decrease is 80%.
L'invention permet de s'affranchir des angles morts, et améliore la sécurité. Par rapport à un système de caméras classique, elle facilite la phase d'apprentissage et de prise en main par le conducteur. La localisation des yeux est facultative, car on peut se contenter d'une position moyenne. Enfin, application de l'invention n'est pas limitée aux véhicules automobiles. Elle trouve aussi tout son intérêt dans d'autres moyens de transports utilisant des caméras. The invention eliminates blind spots, and improves safety. Compared to a conventional camera system, it facilitates the learning and handling phase by the driver. The location of the eyes is optional because we can content with an average position. Finally, application of the invention is not limited to motor vehicles. She also finds all her interest in other means of transport using cameras.

Claims

REVENDICATIONS
1. Procédé de rétro vision par caméras pour véhicule, restituant au conducteur sur un écran, une scène construite par regroupement de prises de vues dans la zone d'observation habituelle de rétroviseurs à miroirs, caractérisé en ce que : 1. A method of vehicle-based retro-vision, restoring to the driver on a screen, a scene constructed by grouping shots in the usual observation zone of mirrored mirrors, characterized in that:
on insère des maquettes numériques tridimensionnelles adaptées au véhicule, dans la scène de rétro vision construite à partir des caméras, et  three-dimensional digital models adapted to the vehicle are inserted in the retro-vision scene constructed from the cameras, and
on construit une scène de réalité augmentée tenant compte d'un paramétrage sélectionné par l'utilisateur et de la position de sa tête, en superposant les maquettes en trois dimensions, à la scène construite à partir des caméras .  an augmented reality scene is constructed taking into account a user-selected parameter and the position of its head, by superimposing the three-dimensional models to the scene constructed from the cameras.
2. Procédé de rétro vision par caméras selon la revendication 1, caractérisé en ce qu'il comprend les étapes suivantes :  2. A method of back-vision cameras according to claim 1, characterized in that it comprises the following steps:
acquisition d'un flux d'images tridimensionnelles par les caméras, sélection et cadrage du point de vue du conducteur,  acquisition of a three-dimensional flow of images by the cameras, selection and framing from the point of view of the driver,
construction de la scène de réalité augmentée, affichage en trois dimensions sur des écrans de restitution .  construction of the augmented reality scene, three-dimensional display on rendering screens.
3. Procédé de rétro vision par caméras selon la revendication 1 ou 2, caractérisé en ce que le masque superposé à l'image diminue la transparence de celle-ci dans la zone concernée.  3. Method of camera back-vision according to claim 1 or 2, characterized in that the mask superimposed on the image decreases the transparency thereof in the area concerned.
4. Dispositif de rétro vision par caméras, pour la mise en œuvre du procédé selon la revendication 1 ou 2, caractérisé en ce qu'il comprend :  4. Device for retro vision by cameras, for the implementation of the method according to claim 1 or 2, characterized in that it comprises:
un ensemble de caméras de rétro vision 1, disposées par paires ,  a set of retro-vision cameras 1, arranged in pairs,
un masque logiciel, statique ou dynamique, spécifique à chaque jeu de caméra,  a static or dynamic software mask, specific to each camera game,
au moins un écran de restitution, et  at least one rendition screen, and
une interface de paramétrage pour l'utilisateur. a parameter interface for the user.
4. Dispositif de rétro vision selon la revendication 3, caractérisé en ce que l'écran de restitution est auto stéréoscopique . 4. Retro-vision device according to claim 3, characterized in that the rendering screen is auto stereoscopic.
5. Dispositif de rétro vision par caméras selon la revendication 3 ou 4 , caractérisé en ce qu'il comprend une interface de paramétraqe pour l'utilisateur.  5. Camera retro-vision device according to claim 3 or 4, characterized in that it comprises a parametriqe interface for the user.
6. Dispositif de rétro vision par caméras selon la revendication 3, 4 ou 5, caractérisé en ce qu'il comporte un système d'acquisition de la position des yeux du conducteur.  6. Camera retro-vision device according to claim 3, 4 or 5, characterized in that it comprises a system for acquiring the position of the eyes of the driver.
7. Dispositif de rétro vision par caméras selon l'une des revendications 3 à 6, caractérisé en ce qu'il comprend une paire de caméras arrière, et une paire de caméras latérales.  7. Camera retro-vision device according to one of claims 3 to 6, characterized in that it comprises a pair of rear cameras, and a pair of side cameras.
PCT/EP2019/058316 2018-05-15 2019-04-02 Rear view method and device using augmented reality cameras WO2019219288A1 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP19716338.9A EP3794563A1 (en) 2018-05-15 2019-04-02 Rear view method and device using augmented reality cameras
KR1020207035268A KR102490465B1 (en) 2018-05-15 2019-04-02 Method and apparatus for rear view using augmented reality camera
CN201980035663.8A CN112204626A (en) 2018-05-15 2019-04-02 Rear view method and apparatus using augmented reality camera

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
FR1854057 2018-05-15
FR1854057A FR3081250B1 (en) 2018-05-15 2018-05-15 PROCESS AND DEVICE FOR FEEDBACK BY AUGMENTED REALITY CAMERAS

Publications (1)

Publication Number Publication Date
WO2019219288A1 true WO2019219288A1 (en) 2019-11-21

Family

ID=63722482

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2019/058316 WO2019219288A1 (en) 2018-05-15 2019-04-02 Rear view method and device using augmented reality cameras

Country Status (5)

Country Link
EP (1) EP3794563A1 (en)
KR (1) KR102490465B1 (en)
CN (1) CN112204626A (en)
FR (1) FR3081250B1 (en)
WO (1) WO2019219288A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114419949A (en) * 2022-01-13 2022-04-29 武汉未来幻影科技有限公司 Automobile rearview mirror image reconstruction method and rearview mirror

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024043588A1 (en) * 2022-08-26 2024-02-29 엘지전자 주식회사 Device for providing augmented reality information on basis of digital mirror and method for controlling device

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5670935A (en) * 1993-02-26 1997-09-23 Donnelly Corporation Rearview vision system for vehicle including panoramic view
FR2780230A1 (en) 1998-06-18 1999-12-24 Renault Rear viewing system for automobile
US20060262140A1 (en) * 2005-05-18 2006-11-23 Kujawa Gregory A Method and apparatus to facilitate visual augmentation of perceived reality
US20120154441A1 (en) * 2010-12-16 2012-06-21 Electronics And Telecommunications Research Institute Augmented reality display system and method for vehicle
US20160350974A1 (en) * 2014-01-10 2016-12-01 Aisin Seiki Kabushiki Kaisha Image display control device and image display system
WO2017200950A1 (en) * 2016-05-19 2017-11-23 Reald Spark, Llc Wide angle imaging directional backlights

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5562498B1 (en) 2014-03-04 2014-07-30 サカエ理研工業株式会社 Room mirror, vehicle blind spot support device using the room mirror, and display image adjustment method of the room mirror or vehicle blind spot support device
JP6707007B2 (en) 2016-09-27 2020-06-10 株式会社東海理化電機製作所 Vehicle visibility device and vehicle visual image display method

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5670935A (en) * 1993-02-26 1997-09-23 Donnelly Corporation Rearview vision system for vehicle including panoramic view
FR2780230A1 (en) 1998-06-18 1999-12-24 Renault Rear viewing system for automobile
US20060262140A1 (en) * 2005-05-18 2006-11-23 Kujawa Gregory A Method and apparatus to facilitate visual augmentation of perceived reality
US20120154441A1 (en) * 2010-12-16 2012-06-21 Electronics And Telecommunications Research Institute Augmented reality display system and method for vehicle
US20160350974A1 (en) * 2014-01-10 2016-12-01 Aisin Seiki Kabushiki Kaisha Image display control device and image display system
WO2017200950A1 (en) * 2016-05-19 2017-11-23 Reald Spark, Llc Wide angle imaging directional backlights

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114419949A (en) * 2022-01-13 2022-04-29 武汉未来幻影科技有限公司 Automobile rearview mirror image reconstruction method and rearview mirror

Also Published As

Publication number Publication date
CN112204626A (en) 2021-01-08
KR102490465B1 (en) 2023-01-20
FR3081250B1 (en) 2021-08-13
KR20210008503A (en) 2021-01-22
FR3081250A1 (en) 2019-11-22
EP3794563A1 (en) 2021-03-24

Similar Documents

Publication Publication Date Title
EP2914002B1 (en) Virtual see-through instrument cluster with live video
Azuma Augmented reality: Approaches and technical challenges
US20070279493A1 (en) Recording medium, parking support apparatus and parking support screen
WO2005088970A1 (en) Image generation device, image generation method, and image generation program
US20150325052A1 (en) Image superposition of virtual objects in a camera image
KR20170135952A (en) A method for displaying a peripheral area of a vehicle
JP2017191378A (en) Augmented reality information display device and augmented reality information display method
EP2572164A1 (en) Navigational aid having improved processing of display data
WO2019219288A1 (en) Rear view method and device using augmented reality cameras
US9918066B2 (en) Methods and systems for producing a magnified 3D image
WO2016102304A1 (en) Method for presenting an image overlay element in an image with 3d information, driver assistance system and motor vehicle
EP3355277A1 (en) Method for displaying on a screen at least one representation of an object, related computer program, electronic display device and apparatus
US10088678B1 (en) Holographic illustration of weather
Poitschke et al. Contact-analog information representation in an automotive head-up display
EP2469868B1 (en) Method for correcting hyperstereoscopy and related helmet display system
JP2005269010A (en) Image creating device, program and method
WO2019026747A1 (en) Augmented real image display device for vehicle
JP2019047296A (en) Display device, display method, control program, and electronic mirror system
WO2017081915A1 (en) Image processing device, image processing method and program
JP6813437B2 (en) Display system
CN117193530B (en) Intelligent cabin immersive user experience method and system based on virtual reality technology
EP3833574B1 (en) Method for providing an image representation of at least part of an environment of a vehicle, computer program product and driver assistance system
KR20180108006A (en) Apparatus and method for trick art display
WO2024132688A1 (en) Display device and method carried out in a display device
CA3018465C (en) See-through based display method and system for simulators

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19716338

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 20207035268

Country of ref document: KR

Kind code of ref document: A

ENP Entry into the national phase

Ref document number: 2019716338

Country of ref document: EP

Effective date: 20201215