EP3663941A1 - Beurteilung einer simulierten fahrzeugbezogenen funktion - Google Patents

Beurteilung einer simulierten fahrzeugbezogenen funktion Download PDF

Info

Publication number
EP3663941A1
EP3663941A1 EP18210988.4A EP18210988A EP3663941A1 EP 3663941 A1 EP3663941 A1 EP 3663941A1 EP 18210988 A EP18210988 A EP 18210988A EP 3663941 A1 EP3663941 A1 EP 3663941A1
Authority
EP
European Patent Office
Prior art keywords
vehicle
feature
simulated
road
virtual
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP18210988.4A
Other languages
English (en)
French (fr)
Other versions
EP3663941B1 (de
Inventor
Casper Wickman
Mattias Wikenmalm
Timotei Ghiurau
Jon Seneger
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Volvo Car Corp
Original Assignee
Volvo Car Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Volvo Car Corp filed Critical Volvo Car Corp
Priority to EP18210988.4A priority Critical patent/EP3663941B1/de
Priority to US16/672,756 priority patent/US11257390B2/en
Priority to CN201911180422.0A priority patent/CN111290571B/zh
Publication of EP3663941A1 publication Critical patent/EP3663941A1/de
Priority to US17/574,726 priority patent/US11938819B2/en
Application granted granted Critical
Publication of EP3663941B1 publication Critical patent/EP3663941B1/de
Priority to US18/607,659 priority patent/US20240217339A1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F30/00—Computer-aided design [CAD]
    • G06F30/10—Geometric CAD
    • G06F30/15—Vehicle, aircraft or watercraft design
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B9/00—Simulators for teaching or training purposes
    • G09B9/02—Simulators for teaching or training purposes for teaching control of vehicles or other craft
    • G09B9/04—Simulators for teaching or training purposes for teaching control of vehicles or other craft for teaching control of land vehicles
    • G09B9/05—Simulators for teaching or training purposes for teaching control of vehicles or other craft for teaching control of land vehicles the view from a vehicle being simulated
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G02B27/0172—Head mounted characterised by optical features
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0179—Display position adjusting means not related to the information to be displayed
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F30/00—Computer-aided design [CAD]
    • G06F30/20—Design optimisation, verification or simulation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B9/00—Simulators for teaching or training purposes
    • G09B9/02—Simulators for teaching or training purposes for teaching control of vehicles or other craft
    • G09B9/04—Simulators for teaching or training purposes for teaching control of vehicles or other craft for teaching control of land vehicles
    • G09B9/042—Simulators for teaching or training purposes for teaching control of vehicles or other craft for teaching control of land vehicles providing simulation in a real vehicle
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0179—Display position adjusting means not related to the information to be displayed
    • G02B2027/0181—Adaptation to the pilot/driver
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0179—Display position adjusting means not related to the information to be displayed
    • G02B2027/0183—Adaptation to parameters characterising the motion of the vehicle
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/0179—Display position adjusting means not related to the information to be displayed
    • G02B2027/0187—Display position adjusting means not related to the information to be displayed slaved to motion of at least a part of the body of the user, e.g. head, eye

Definitions

  • the present disclosure relates to enabling evaluation of a simulated vehicle-related feature.
  • Static or dynamic evaluation rigs are commonly utilized, where the static rig for instance may be represented by a non-movable rig that may display the environment and traffic on large scale screens in front of a vehicle mock-up, and where the dynamic rig for instance may be represented by a similar rig with the difference that the dynamic rig may move in order to simulate acceleration forces and/or movement.
  • a drawback for both static and dynamic rigs is that the test subject may feel motion sickness.
  • Yet another drawback is that the environment as well as the traffic in fact is simulated.
  • the object is achieved by a method performed by a vehicle feature evaluation system for enabling evaluation of a simulated vehicle-related feature.
  • the vehicle feature evaluation system determines in relation to a road-driven vehicle, with support from a tracking system, an orientation of a head-mounted display, HMD, adapted to be worn by an occupant on-board the road-driven vehicle.
  • the vehicle feature evaluation system further determines a simulated vehicle design feature to be evaluated in the road-driven vehicle.
  • the vehicle feature evaluation system provides in real-time to a HMD display of the HMD, taking into consideration the HMD orientation, a virtual representation of the simulated vehicle design feature superimposed on a real-time surrounding-showing video stream derived from real-world image data captured with support from one or more vehicle-attached cameras adapted to capture surroundings external to the road-driven vehicle.
  • evaluation may take place in an actual real vehicle. Furthermore, that is, since there is determined a simulated vehicle design feature to be evaluated in the road-driven vehicle, there is defined - e.g. computer-generated - a virtual vehicle design feature of interest to validate in a real vehicle.
  • a virtual representation of the simulated vehicle design feature superimposed on a real-time surrounding-showing video stream derived from real-world image data captured with support from one or more vehicle-attached cameras adapted to capture surroundings external to the road-driven vehicle there is provided to the HMD display a real-time physical world view of surroundings external the road-driven vehicle - essentially as captured by the vehicle-attached camera(s) - electronically mixed with the computer-generated virtual design feature representation temporally and spatially commensurate with said physical world view such that the surrounding-showing video stream appears as a background overlaid with said virtual design feature representation.
  • the real-time surrounding-showing video stream shows surroundings and/or an environment exterior of the vehicle, hence excluding interior - and potentially at least partial exterior - views of said road-driven vehicle.
  • the HMD-wearing occupant being on-board the road-driven vehicle and with the HMD orientation being taken into consideration, provided to the HMD display is then a portion of surrounding-showing video stream corresponding to and/or spatially commensurate with a field of view of the HMD-wearing occupant.
  • the HMD-wearing occupant would accordingly be shown - on the HMD display(s) - a physical world view spatially commensurate with said occupant's field of view, but from which physical world view an interior - and potentially at least a portion of an exterior - of the road-driven vehicle would be excluded due to the vehicle-attached camera(s) capturing surroundings external of - not internal of - the road-driven vehicle.
  • the real-time surrounding-showing video stream may then represent what a driver or passenger of said road-driven vehicle would essentially see even if not wearing the HMD, but with interior - and potentially at least partial exterior - portions of the road-driven vehicle removed.
  • one or more simulated design features - e.g. represented by a simulated updated and/or new interior or exterior or a portion thereof - may be evaluated in its place.
  • a virtual representation of a simulated vehicle design feature - such as e.g. a complete interior and parts of an exterior or e.g. at least a simulated dashboard - on the surrounding-showing video stream, said simulated vehicle design feature e.g. simulated dashboard may be evaluated in a road-driven vehicle, e.g.
  • the entire test environment is authentic and/or real. Consequently, the simulated vehicle design feature may be evaluated in real traffic and/or in real road-driving scenarios, which thus enables said simulated vehicle design feature to be evaluated under more reality-resembling conditions and/or in a more reality-resembling environment than should said evaluation be performed in e.g. a commonly known physical mock-up.
  • vehicle feature evaluation system may refer to “vehicle feature validation system”, “vehicle evaluation system”, “evaluation system” and/or “evaluation supporting system”, whereas "for enabling evaluation of” may refer to “for evaluating” and/or “for providing”.
  • enabling evaluation may refer to “supporting” evaluation, whereas “evaluation” may refer to “validation”, "testing” and/or “analysis”.
  • “Simulated” vehicle-related feature may refer to “virtual”, “computer-generated”, “modelled” and/or “fictive” vehicle-related feature
  • “a simulated vehicle-related feature” may refer to "a simulated vehicle design feature", “a simulated vehicle design”, “a simulated vehicle concept”, “a simulated vehicle feature” and/or “simulated vehicle characteristics”.
  • the expression “for enabling evaluation of a simulated vehicle-related feature” may refer to “for enabling evaluation of a simulated vehicle-related feature in a road-driven vehicle”.
  • the vehicle feature evaluation system determines in relation to a road-driven vehicle, with support from a tracking system, an orientation of a head mounted display, HMD, adapted to be worn by an occupant on-board the road-driven vehicle, there is established a position and/or location of an occupant-worn HMD in view of a real vehicle e.g. adapted to and/or intended to be driven outdoors such as in real traffic and/or along actual roads.
  • HMD-wearing occupant - which may also be referred to as the test subject - being positioned in a road-driven vehicle rather than in an e.g. indoor static or dynamic rig, evaluation may take place in an actual real vehicle.
  • Determining the orientation of the HMD in relation to the road-driven vehicle, with support from the tracking system may be accomplished as commonly known in the art, for instance with input from tracking sensors such as cameras, accelerometers, magnetometers and/or gyroscopes, and/or based on software tracking e.g. aligning markers in the vehicle with corresponding markers in the virtual model.
  • the HMD may comprise and/or support any arbitrary features known in the art from HMDs commonly utilized today, with the addition of further features that will be described further on.
  • the referred to "vehicle” may be represented by any arbitrary vehicle, for instance an engine-propelled vehicle, such as e.g. a car, truck, lorry, van, tractor and/or bus.
  • road-driven vehicle may refer to "real vehicle”, “outdoor vehicle”, “vehicle adapted to be driven in real traffic” and/or “vehicle adapted to be driven along outdoor tracks and/or public roads and/or public road networks”.
  • Determining in relation to a road-driven vehicle may refer to “calculating and/or analysing” in relation to a road-driven vehicle and/or “determining optically, magnetically and/or by means of laser” in relation to a road-driven vehicle.
  • In relation to may refer to "in view of" and/or "in comparison to”.
  • “In relation to a road-driven vehicle” may further refer to “in relation to a reference point of a road-driven vehicle", which "reference point” may be arbitrarily selected as deemed adequate for the implementation at hand.
  • the expression “with support from” a tracking system may refer to “by means of” a tracking system and/or “with input from” a tracking system, whereas “a tracking system”, on the other hand, may refer to “one or more tracking sensors” and/or "a software based tracking system”.
  • “An orientation” of a head-mounted display may refer to "a position and/or location of” a head-mounted display, whereas “head-mounted display” may refer to "augmented reality and/or virtual reality head-mounted display", “goggles” and/or “electronic goggles”.
  • the expression “adapted to be worn by” an occupant may refer to “adapted to be worn at least partly in front of the eyes of” an occupant and/or merely “worn by” an occupant, whereas “an occupant” may refer to "a vehicle occupant", “a driver” and/or “a test subject”.
  • “On-board” the road-driven vehicle may refer to "positioned in”, “seated in a front seat of” and/or “seated in a driver's seat of” the road-driven vehicle.
  • the vehicle feature evaluation system determines a simulated vehicle design feature to be evaluated in the road-driven vehicle, there is defined - e.g. computer-generated - a virtual vehicle design feature of interest to validate in a real vehicle.
  • “Determining” a simulated vehicle design feature may refer to "computer-generating”, “simulating” and/or “modelling” a simulated vehicle design feature, whereas vehicle “design feature” may refer to vehicle “characteristics” and further to merely vehicle “feature” and/or vehicle “concept”.
  • “determining” a simulated vehicle design feature may refer to "determining logic, design, graphics, aesthetics, geometries and/or functionality of" a simulated vehicle design feature.
  • the vehicle feature evaluation system provides in real-time to a HMD display of the HMD, taking into consideration the HMD orientation, a virtual representation of the simulated vehicle design feature superimposed on a real-time surrounding-showing video stream derived from real-world image data captured with support from one or more vehicle-attached cameras adapted to capture surroundings external to the road-driven vehicle, there is provided to the HMD display a real-time physical world view of surroundings external the road-driven vehicle - essentially as captured by the vehicle-attached camera(s) - electronically mixed with the computer-generated virtual design feature representation temporally and spatially commensurate with said physical world view such that the surrounding-showing video stream appears as a background overlaid with said virtual design feature representation.
  • live imagery from the physical world is electronically merged with computer-generated imagery, a concept commonly referred to as mixed reality and/or augmented reality.
  • the virtual representation of the simulated vehicle design feature is overlaid the surrounding-showing video stream, a concept commonly referred to as video see-through. That is, with the vehicle-attached camera(s) provided to capture surroundings external of the road-driven vehicle, the real-time surrounding-showing video stream shows surroundings and/or an environment exterior of the vehicle, hence excluding interior - and potentially at least partial exterior - views of said road-driven vehicle. Subsequently, with the HMD-wearing occupant being on-board the road-driven vehicle - e.g.
  • HMD display is then a portion of surrounding-showing video stream - derived from real-world image data captured with support from the vehicle-attached camera(s) - corresponding to and/or spatially commensurate with a field of view of the HMD-wearing occupant.
  • the HMD-wearing occupant would accordingly be shown - on the HMD display(s) - a physical world view spatially commensurate with said occupant's field of view, but from which physical world view an interior - and potentially at least a portion of an exterior - of the road-driven vehicle would be excluded due to the vehicle-attached camera(s) capturing surroundings external of - not internal of - the road-driven vehicle.
  • the real-time surrounding-showing video stream may then represent what a driver or passenger of said road-driven vehicle would essentially see even if not wearing the HMD, but with interior - and potentially at least partial exterior - portions of the road-driven vehicle removed.
  • one or more simulated design features - e.g. represented by a simulated updated and/or new interior or exterior or a portion thereof - may be evaluated in its place.
  • a virtual representation of a simulated vehicle design feature - such as e.g. a complete interior and parts of an exterior or e.g. at least a simulated dashboard or a portion thereof - on the surrounding-showing video stream, said simulated vehicle design feature e.g. simulated dashboard may be evaluated in a road-driven vehicle, e.g.
  • the entire test environment is authentic and/or real. Consequently, the simulated vehicle design feature may be evaluated in real traffic and/or in real road-driving scenarios, which thus enables said simulated vehicle design feature to be evaluated under more reality-resembling conditions and/or in a more reality-resembling environment than should said evaluation be performed in e.g. a commonly known physical mock-up.
  • Providing "in real-time”, on the other hand may refer to providing “essentially in real-time” and/or “live”, and further to that the virtual representation superimposed on the real-world video stream is provided to the HMD display substantially close to the time of the actual capturing of the real-world image data. According to an example, “providing in real-time” may refer to merely “providing”.
  • the expression “taking into consideration” the HMD orientation may refer to “considering” and/or “taking into consideration for positioning and/or spatial positioning” the HMD orientation, whereas “taking into consideration the HMD orientation” may refer to “taking into consideration a field of view of the occupant derived from the HMD orientation” and/or “taking into consideration the HMD orientation and/or a field of view of the occupant derived from the HMD orientation”.
  • “Virtual” representation may refer to "augmented” and/or “overlaid” representation, whereas “representation” may refer to "augmentation", “data” and/or “geometrical and/or graphical model”.
  • “representation” may further refer to "graphical model", "computer-aided design, CAD, geometrics and/or design surface model, DSM, geometries” and/or “result from computer-aided engineering, CAE”.
  • "Superimposed on” a real-time surrounding-showing video stream may refer to “overlaid”, “virtually overlaid”, “placed and/or displayed in front of” a real-time surrounding-showing video stream, whereas "a real-time” surrounding-showing video stream may refer to "an essentially real-time", "a live or essentially live” and/or “instant or essentially instant” surrounding-showing video stream.
  • real-time surrounding-showing video stream may further indicate that the surrounding-showing video stream is derived in a timeframe substantially close to the time of the actual capturing of the real-world image data.
  • Real-world may refer to "physical world” and/or “real environment world”
  • video stream may refer to “streaming video” and/or merely “video”.
  • Destructived from” real-world image data may refer to "which video stream is based on” real-world image data
  • real-world image data may refer to "video data” and/or "images”.
  • Captured with support from one or more vehicle-attached cameras may refer to “taped” and/or “filmed” with support from one or more vehicle-attached cameras, whereas captured “with support from” one or more vehicle-attached cameras may refer to captured “by” and/or “by means of” one or more vehicle-attached cameras.
  • one or more vehicle-attached cameras may according to an example refer to “at least two” vehicle-attached cameras, whereas “vehicle-attached cameras” may refer to “cameras fixed to, mounted on and/or comprised in the road-driven vehicle", and further to “vehicle-attached surrounding-capturing cameras”.
  • extended of the road-driven vehicle may refer to “on the outside of” the road-driven vehicle, and further to “external to” and/or merely “external” the road-driven vehicle.
  • the expression "adapted to capture surroundings external to the road-driven vehicle” may refer to “adapted to capture surroundings external to the road-driven vehicle at least in a forward direction of the road-driven vehicle, preferably additionally in a sideway, backward, upward and/or downward direction of the road-driven vehicle”.
  • the expression "real-time surrounding-showing video stream derived from real-world image data captured with support from one or more vehicle-attached cameras adapted to capture surroundings external of the road-driven vehicle” may refer to "real-time surrounding-showing video stream derived from real-world image data captured with support from one or more vehicle-attached cameras adapted to capture surroundings external of the road-driven vehicle, wherein the real-time surrounding-showing video stream comprises a first real-time video stream from a first vehicle-attached camera adapted to capture surroundings external of the road-driven vehicle in a first direction, stitched with at least a second real-time video stream from at least a second vehicle-attached camera adapted to capture surroundings external of the road-driven vehicle in a second direction differing from the first direction".
  • the simulated vehicle design feature may comprise a simulated vehicle interior section and/or a simulated vehicle exterior section.
  • a virtual vehicle interior portion - e.g. a simulated dashboard - and/or a virtual vehicle exterior portion - e.g. a simulated hood - may be evaluated in the road-driven vehicle, such as e.g. new and/or updated design, colour, material and/or user interface thereof and/or e.g. geometrics for instance CAD and/or DSM geometrics thereof.
  • the expression simulated "vehicle interior section” may refer to simulated "at least a portion of a vehicle interior section", and further to simulated "vehicle interior portion" and/or simulated “visual and/or uncovered vehicle interior section”.
  • simulated "vehicle exterior section” may refer to simulated “at least a portion of a vehicle exterior section", and further to simulated “vehicle exterior portion” and/or simulated “visual and/or uncovered vehicle exterior section”.
  • the vehicle feature evaluation system may determine a simulated vehicle functionality feature to be evaluated in the road-driven vehicle.
  • Providing to the HMD display may then further comprise additionally providing in real-time to the HMD display of the HMD, taking into consideration the HMD orientation, a virtual representation of the simulated vehicle functionality feature superimposed on the real-time surrounding-showing video stream.
  • the simulated vehicle functionality feature may e.g. comprise a simulated vehicle human machine interface, HMI, a simulated vehicle display, a simulated vehicle head-up display, HUD, simulated light characteristics, and/or a simulated effect resulting from a simulation of the simulated vehicle functionality feature.
  • the virtual representation of the simulated vehicle functionality feature is overlaid the real-world video stream to be temporally and spatially commensurate with the video stream such that said video stream appears as a background overlaid with - in addition to the virtual design feature representation - said virtual functionality feature representation.
  • a virtual representation of a simulated vehicle functionality feature - such as for instance a virtual display with e.g. new and/or updated design and/or functionality - on the surrounding-showing video stream
  • said simulated vehicle functionality feature may be evaluated in the road-driven vehicle, e.g. while said road-driven vehicle is driven in real traffic and/or along actual roads.
  • the simulated vehicle functionality feature may be evaluated in real traffic and/or in real road-driving scenarios, which thus enables said simulated vehicle functionality feature to be evaluated under more reality-resembling conditions and/or in a more reality-resembling environment than should said evaluation be performed in e.g. a commonly known static or dynamic rig displaying the environment and traffic on large scale screens in front of a vehicle mock-up.
  • a virtual vehicle HMI may be evaluated in the road-driven vehicle, such as e.g. graphics thereof and/or new and/or updated functionality and/or design thereof.
  • a virtual vehicle display e.g. an infotainment display may be evaluated in the road-driven vehicle, such as e.g. graphics thereof and/or new and/or updated functionality and/or design thereof.
  • a virtual vehicle HUD may be evaluated in the road-driven vehicle, such as e.g. graphics thereof and/or new and/or updated functionality and/or design thereof.
  • simulated vehicle functionality feature on the other hand additionally or alternatively comprise simulated light characteristics, then virtual light characteristics e.g.
  • interior light characteristics may be evaluated in the road-driven vehicle, such as e.g. new and/or updated appearance and/or ambience of e.g. interior lighting.
  • the simulated vehicle functionality feature additionally or alternatively comprise a simulated effect resulting from a simulation of the simulated vehicle functionality feature, then a virtual effect resulting from simulation of a simulated vehicle functionality feature e.g. wiper performance, cabin climate, air flow, contamination etc. may be evaluated in the road-driven vehicle, such as e.g. the effect of a computer-aided engineering, CAE, simulation thereof.
  • CAE computer-aided engineering
  • Determining a simulated vehicle functionality feature may refer to “computer-generating”, “simulating” and/or “modelling” a simulated vehicle functionality feature, whereas vehicle “functionality feature” may refer to vehicle “characteristics” and further to merely vehicle “feature” and/or vehicle “functionality”.
  • determining a simulated vehicle functionality feature may refer to “determining logic, design, graphics, aesthetics, geometries and/or functionality of” a simulated vehicle functionality feature.
  • the expression that the simulated vehicle functionality feature “comprises” may refer to that the simulated vehicle functionality feature “is represented by”, whereas light “characteristics” may refer to light "position(s)", “positioning”, “function(s)”, “ambience” and/or “appearance”.
  • the vehicle feature evaluation system may determine a simulated object feature to be evaluated in the road-driven vehicle.
  • Providing to the HMD display may then further comprise additionally providing in real-time to the HMD display of the HMD, taking into consideration the HMD orientation, a virtual representation of the simulated object feature superimposed on the real-time surrounding-showing video stream.
  • the simulated object feature may e.g. comprise a simulated other vehicle, a simulated road user, and/or a simulated moving or fixed obstacle.
  • - e.g. computer-generated - a virtual object of interest to validate in a real vehicle.
  • the virtual representation of the simulated object feature is overlaid the real-world video stream to be temporally and spatially commensurate with the video stream such that said video stream appears as a background overlaid with - in addition to the virtual design feature representation - said virtual object feature representation.
  • a virtual representation of a simulated object feature - such as for instance a virtual pedestrian appearing to cross the road in front of the road-driven vehicle - on the surrounding-showing video stream
  • said simulated object feature and/or an effect it has on the HMD-wearing occupant - who for instance may be the driver of the road-driven vehicle - may be evaluated, e.g. while said road-driven vehicle is driven in real traffic and/or along actual roads.
  • the simulated object feature - and/or an effect it has on the HMD-wearing occupant - may be evaluated in real traffic and/or in real road-driving scenarios, which thus enables said simulated object feature and/or the effect it has on the HMD-wearing occupant to be evaluated under more reality-resembling conditions and/or in a more reality-resembling environment than should said evaluation be performed in e.g. a commonly known static or dynamic rig displaying the environment and traffic on large scale screens in front of a vehicle mock-up.
  • a virtual vehicle may be evaluated in the road-driven vehicle, such as e.g.
  • the simulated object feature additionally or alternatively comprise a simulated road user, e.g. appearing to cross the road in front of the road-driven vehicle, then a virtual road user - and/or an effect it has on the HMD-wearing occupant - may be evaluated in the road-driven vehicle.
  • the simulated object feature additionally or alternatively comprise a simulated moving or fixed obstacle, e.g. an elk appearing to e.g.
  • Determining a simulated object feature may refer to "computer-generating”, “simulating” and/or “modelling” a simulated object feature, whereas "object” may refer to "physical object”, “exterior object” and/or "object exterior of the road-driven vehicle”.
  • Object feature may further refer to merely “object”
  • determining a simulated object feature may refer to “determining logic, design, graphics, aesthetics, geometries and/or functionality of” a simulated object feature
  • the expression that the simulated object feature “comprises” may refer to that the simulated object feature "is represented by”.
  • “Other vehicle” may refer to “a vehicle” and/or “road-driven vehicle”
  • “road user” may indicate anyone who uses a road such as e.g. a pedestrian, cyclist, motorist etc.
  • the vehicle feature evaluation system may determine a fictive location of the simulated design feature relative the road-driven vehicle.
  • the above discussed providing of the virtual design feature representation taking into consideration the HMD orientation may then comprise providing the virtual design feature representation superimposed on the real-time surrounding-showing video stream such that positioning of the virtual design feature representation corresponds to the fictive design feature location.
  • the positioning of the overlaid virtual design feature representation on the real-world video stream spatially commensurate with the intended fictive vehicle location of the vehicle design feature to be simulated.
  • the fictive design feature location may be arbitrarily selected as deemed relevant for the implementation at hand, and may further be of any arbitrary form, size and/or dimensions ranging from a few millimetres up to hundreds or even thousands of millimetres across.
  • the vehicle feature evaluation system may determine a fictive location of the simulated vehicle functionality feature relative the road-driven vehicle.
  • the above discussed optional providing of the virtual functionality feature representation taking into consideration the HMD orientation may then comprise providing the virtual functionality feature representation superimposed on the real-time surrounding-showing video stream such that positioning of the virtual functionality feature representation corresponds to the fictive functionality feature location.
  • the positioning of the overlaid virtual functionality feature representation on the real-world video stream spatially commensurate with the intended fictive vehicle location of the vehicle functionality feature to be simulated.
  • the fictive functionality feature location may be arbitrarily selected as deemed relevant for the implementation at hand, and may further be of any arbitrary form, size and/or dimensions ranging from a few millimetres up to hundreds or even thousands of millimetres across.
  • the vehicle feature evaluation system may determine a fictive location of the simulated object feature relative the road-driven vehicle.
  • the above discussed optional providing of the virtual object feature representation taking into consideration the HMD orientation may then comprise providing the virtual object feature representation superimposed on the real-time surrounding-showing video stream such that positioning of the virtual object feature representation corresponds to the fictive object feature location.
  • the fictive object feature location may be arbitrarily selected as deemed relevant for the implementation at hand and may further for instance refer to a location outside the road-driven vehicle, such as e.g. in front of the road-driven vehicle, e.g. a few millimetres therefrom up to many thousands of millimetres therefrom.
  • Determining" a fictive location may refer to “mapping” a fictive location, whereas “fictive” location may refer to “intended” and/or “virtual” location.
  • Fictive “location”, on the other hand, may refer to fictive “position” and/or “placement”, whereas “relative” the road-driven vehicle may refer to “in relation to” and/or “within” the road-driven vehicle.
  • “Corresponds to” the fictive location may refer to "commensurate with", "reflects” and/or "represents” the fictive location.
  • the vehicle feature evaluation system may further comprise updating the virtual functionality feature representation based on a user interaction with the simulated vehicle functionality feature.
  • interaction by a user with the simulated vehicle functionality feature may result in that the virtual representation of the simulated vehicle functionality feature is updated in accordance with said interaction.
  • user interaction with the simulated vehicle functionality feature which feature for instance is represented by selectable options available on a simulated vehicle display, may result in that a user selected option is carried out and/or updated in accordance with the user interaction, which subsequently is reflected by the corresponding virtual functionality feature being updated in accordance therewith.
  • the user interaction may be based on detection, by means of one or more user interaction sensors, of a user in and/or at the fictive functionality feature location.
  • presence of a user e.g. the HMD-wearing occupant, and/or e.g. a finger of said user/occupant, may be sensed in or at the position in the vehicle representing the location of the simulated vehicle functionality feature, whereby the virtual representation of the simulated vehicle functionality feature subsequently may be updated in accordance with said presence, and/or in accordance with the geographical position and/or the nature of said presence.
  • User interaction may be detected in any arbitrary manner known in the art, e.g. by means of one or more user interaction sensors and/or a user interaction determining system, for instance comprising touch sensor(s), camera(s) and/or position detection sensor(s) worn by the user e.g. on his/her hand and/or finger.
  • the expression “updating” the virtual functionality feature representation may refer to “updating content, graphical data and/or geometric data of" the virtual functionality feature representation, whereas "based on a” user interaction may refer to “as a result of” user interaction and/or “based on input data and/or one or more signals derived from” user interaction.
  • “User interaction”, on the other hand, may refer to "occupant interaction”.
  • detection “by means of” one or more user interaction sensors may refer to detection “with support from” and/or “with input from” one or more user interaction sensors. Detection “of a user”, on the other hand, may refer to detection “of the occupant", and further to detection "of a hand and/or one or more fingers of a user”.
  • the expression “in and/or at” the fictive functionality feature location may refer to “in or at a location within the vehicle representing" the fictive functionality feature location.
  • the vehicle feature evaluation system may further comprise updating the virtual functionality feature representation based on a vehicle signal affecting the simulated vehicle functionality feature, derived from the road-driven vehicle.
  • a vehicle signal which affects the simulated vehicle functionality feature - e.g. resulting from manoeuvring of and/or user interaction with the road-driven vehicle - results in that the virtual representation of the simulated vehicle functionality feature is updated in accordance therewith. That is, deriving - from the road-driven vehicle - a vehicle signal affecting the simulated vehicle functionality feature, where the vehicle signal for instance comprises vehicle data such as e.g. fuel consumption and where the simulated vehicle functionality feature for instance is represented by a simulated vehicle display showing said vehicle data e.g.
  • the virtual functionality feature - in this example the virtual vehicle display - is updated in accordance with said vehicle signal, i.e. the fuel consumption value shown on the virtual display is updated along with the received vehicle signal comprising an updated fuel consumption value.
  • the vehicle signal may be derived from the road-driven vehicle in any arbitrary known manner, for instance via wired and/or wireless communication therewith, and/or with support from a vehicle signal determining system and/or unit adapted for determining which input derived from the road-driven vehicle affects the simulated vehicle functionality feature.
  • updating the virtual functionality feature representation may refer to “updating content, graphical data and/or geometric data of” the virtual functionality feature representation, whereas "based on a" vehicle signal may refer to “based on input and/or data from a" vehicle signal. "Derived from” the road-driven vehicle, on the other hand, may refer to “received from” the road-driven vehicle.
  • "updating the virtual functionality feature representation based on a vehicle signal affecting the simulated vehicle functionality feature, derived from the road-driven vehicle” may refer to “updating the virtual functionality feature representation based on a vehicle signal derived from the road-driven vehicle, which vehicle signal is based on a manoeuvring of and/or user interaction with the road-driven vehicle affecting the simulated vehicle functionality feature".
  • the vehicle feature evaluation system may update the real-time video stream, the superimposed virtual design feature representation, the superimposed virtual functionality feature representation and/or the superimposed virtual object feature representation based on updated captured image data and/or based on updated determined HMD orientation.
  • the real-time video stream and/or the virtual representation(s) are continuously updated to reflect updated captured image data and/or updated orientation of the HMD.
  • “Updating” may refer to “continuously and/or intermittently updating”.
  • the object is achieved by a vehicle feature evaluation system for - and/or adapted for - enabling evaluation of a simulated vehicle-related feature.
  • the vehicle feature evaluation system comprises a HMD orientation determining unit for - and/or adapted for - determining in relation to a road-driven vehicle, with support from a tracking system, an orientation of a head-mounted display, HMD, adapted to be worn by an occupant on-board the road-driven vehicle.
  • the vehicle feature evaluation system further comprises a design feature determining unit for - and/or adapted for - determining a simulated vehicle design feature to be evaluated in the road-driven vehicle.
  • the vehicle feature evaluation system comprises a HMD providing unit for - and/or adapted for - providing in real-time to a HMD display of the HMD, taking into consideration the HMD orientation, a virtual representation of the simulated vehicle design feature superimposed on a real-time surrounding-showing video stream derived from real-world image data captured with support from one or more vehicle-attached cameras adapted to capture surroundings external to the road-driven vehicle.
  • the simulated vehicle design feature may comprise a simulated vehicle interior section and/or a simulated vehicle exterior section.
  • the vehicle feature evaluation system may comprise a functionality feature determining unit.
  • the HMD providing unit may then be adapted for additionally providing in real-time to the HMD display of the HMD, taking into consideration the HMD orientation, a virtual representation of the simulated vehicle functionality feature superimposed on the real-time surrounding-showing video stream.
  • the simulated vehicle functionality feature may e.g. comprise a simulated vehicle human machine interface, HMI, a simulated vehicle display, a simulated vehicle head-up display, HUD, light characteristics, and/or a simulated effect resulting from a simulation of the simulated vehicle functionality feature.
  • the vehicle feature evaluation system may comprise an object feature determining unit.
  • the HMD providing unit may then be adapted for additionally providing in real-time to the HMD display of the HMD, taking into consideration the HMD orientation, a virtual representation of the simulated object feature superimposed on the real-time surrounding-showing video stream.
  • the simulated object feature may e.g. comprise a simulated other vehicle, a simulated road user, and/or a simulated moving or fixed obstacle.
  • the vehicle feature evaluation system may further comprise a location determining unit for - and/or adapted for - determining a fictive design feature location of the simulated vehicle design feature relative the road-driven vehicle.
  • the HMD providing unit may then be further adapted for providing the virtual design feature representation superimposed on the real-time surrounding-showing video stream such that positioning of the virtual design feature representation corresponds to the fictive design feature location.
  • the optional location determining unit may be for - and/or adapted for - determining a fictive functionality feature location of the simulated vehicle functionality feature relative the road-driven vehicle.
  • the HMD providing unit may then be further adapted for providing the virtual functionality feature representation superimposed on the real-time surrounding-showing video stream such that positioning of the virtual functionality feature representation corresponds to the fictive functionality feature location.
  • the optional location determining unit may be for - and/or adapted for - determining a fictive object feature location of the simulated object feature relative the road-driven vehicle.
  • the HMD providing unit may then be further adapted for providing the virtual object feature representation superimposed on the real-time surrounding-showing video stream such that positioning of the virtual object feature representation corresponds to the fictive object feature location.
  • the vehicle feature evaluation system may further comprise a user interaction updating unit for - and/or adapted for - updating the virtual functionality feature representation based on a user interaction with the simulated vehicle functionality feature.
  • the user interaction may be based on detection, by means of one or more user interaction sensors, of a user in and/or at the fictive functionality feature location.
  • the vehicle feature evaluation system may further comprise a vehicle signal updating unit for - and/or adapted for - updating the virtual functionality feature representation based on a vehicle signal affecting the simulated vehicle functionality feature, derived from the road-driven vehicle.
  • the vehicle feature evaluation system may further comprise a continuous updating unit for - and/or adapted for - updating the real-time video stream, the superimposed virtual design feature representation, the superimposed virtual functionality feature representation and/or the superimposed virtual object feature representation based on updated captured image data and/or based on updated determined HMD orientation.
  • a continuous updating unit for - and/or adapted for - updating the real-time video stream, the superimposed virtual design feature representation, the superimposed virtual functionality feature representation and/or the superimposed virtual object feature representation based on updated captured image data and/or based on updated determined HMD orientation.
  • the object is achieved by a vehicle - e.g. a road-driven vehicle - comprising a vehicle feature evaluation system as discussed above.
  • a vehicle - e.g. a road-driven vehicle - comprising a vehicle feature evaluation system as discussed above.
  • a computer program product comprising a computer program containing computer program code means arranged to cause a computer or a processor to execute the steps of the vehicle feature evaluation system discussed in the foregoing, stored on a computer-readable medium or a carrier wave.
  • the vehicle feature evaluation system 1 is here comprised in a road-driven vehicle 2, namely an exemplifying a passenger car.
  • a road-driven vehicle 2 namely an exemplifying a passenger car.
  • vehicle-attached cameras 21 - here three essentially forward-facing cameras 21 - adapted to capture surroundings external of the road-driven vehicle 2.
  • On-board said vehicle 2 is an occupant 3, wearing an HMD 4, the orientation of which - in relation to the vehicle 2 - may be determined with support from a tracking system 5 (shown in Figs. 2-3 ).
  • the HMD 4 comprises at least one HMD display 41, here two displays, on which it is displayed a real-time surrounding-showing video stream 6 derived from real-world image data 211 (shown in Figs. 2-3 ) captured with support from the one or more vehicle-attached cameras 21.
  • the simulated vehicle design feature here comprises an exemplifying simulated vehicle interior section, at least comprising a simulated dashboard and steering wheel, and the virtual representation 7 is accordingly here represented by a virtual interior section comprising an exemplifying virtual dashboard and virtual steering wheel.
  • the simulated vehicle design feature may have a fictive design feature location 70 relative the vehicle 2, and for the exemplifying simulated vehicle interior section in Figs. 1-3 , the fictive design feature location 70 is in an exemplifying manner essentially situated where the - to the HMD-wearing occupant 3 invisible - dashboard and steering wheel of the road-driven vehicle 2 are situated.
  • a simulated vehicle design feature may additionally or alternatively optionally comprise a simulated vehicle exterior section.
  • Superimposed on the surrounding-showing video stream 6 is additionally an optional virtual representation 8 of a simulated vehicle functionality feature to be evaluated in the road-driven vehicle 2.
  • the simulated vehicle functionality feature here comprises an exemplifying simulated vehicle display, more specifically a simulated infotainment display, and the virtual functionality feature representation 8 is accordingly here represented by a virtual infotainment display.
  • the simulated vehicle functionality feature may have a fictive functionality feature location 80 relative the vehicle 2, and for the exemplifying simulated infotainment display in Figs. 1-3 , the fictive functionality feature location 80 is in an exemplifying manner essentially centered on a dashboard of the road-driven vehicle 2 and/or essentially centered on the virtual dashboard 7. Additionally depicted is an exemplifying alternative virtual functionality feature representation 8' of an alternative simulated vehicle functionality feature.
  • the alternative simulated vehicle functionality feature here comprises an optional exemplifying simulated driver information display, and the alternative virtual representation 8' is then accordingly represented by an exemplifying virtual driver information display.
  • the simulated driver information display here has an exemplifying fictive functionality feature location 80' on the vehicle dashboard of the road-driven vehicle 2 and/or on the virtual dashboard 7, in front of a driver's seat of the road-driven vehicle 2.
  • a simulated vehicle functionality feature may additionally or alternatively optionally comprise e.g. a simulated vehicle HMI, a simulated HUD, simulated light characteristics and/or a simulated effect resulting from a simulation of the simulated vehicle functionality feature.
  • Fig. 3 is an optional server 10, which will be described in greater detail in conjunction with Fig. 5 .
  • Fig 4 illustrates a schematic view of an alternative vehicle feature evaluation system 1 according to embodiments of the disclosure.
  • Superimposed on the surrounding-showing video stream 6 is here additionally an optional virtual representation 9 of a simulated object feature to be evaluated in the road-driven vehicle 2.
  • the simulated object feature here comprises an exemplifying simulated road user, namely a pedestrian, and the virtual object feature representation 9 is accordingly here represented by a virtual road user 91, namely a virtual pedestrian, in an exemplifying manner crossing the street in front of the road-driven vehicle 2.
  • the simulated object feature may have a fictive object feature location 90 relative the vehicle 2, and for the exemplifying simulated pedestrian in Fig.
  • fictive object feature location 90 is in an exemplifying manner essentially a few meters in front of the road-driven vehicle 2.
  • a simulated object feature may additionally or alternatively optionally e.g. comprise a simulated other vehicle and/or a simulated moving or fixed obstacle.
  • the vehicle feature evaluation system 1 comprises a HMD orientation determining unit 101, a design feature determining unit 102, an optional functionality feature determining unit 103, an optional object feature determining unit 104, an optional location determining unit 105, a HMD providing unit 106, an optional user interaction updating unit 107, an optional vehicle signal updating unit 108, and an optional continuous updating unit 109, all of which will be described in greater detail in conjunction with Fig. 6 .
  • the embodiments herein for enabling evaluation of a simulated vehicle-related feature may be implemented through one or more processors, such as a processor 110, here denoted CPU/GPU, together with computer program code for performing the functions and actions of the embodiments herein.
  • Said program code may also be provided as a computer program product, for instance in the form of a data carrier carrying computer program code for performing the embodiments herein when being loaded into the vehicle feature evaluation system 1.
  • a data carrier may be in the form of a CD ROM disc and/or a hard drive. It is however feasible with other data carriers such as a memory stick.
  • the computer program code may furthermore be provided as pure program code on a server and downloaded to the vehicle feature evaluation system 1.
  • the vehicle feature evaluation system 1 may further comprise a memory 111 comprising one or more memory units.
  • the memory 111 may be arranged to be used to store e.g. information, and further to store data, configurations, schedulings, and applications, to perform the methods herein when being executed in the vehicle feature evaluation system 1.
  • the computer program code may be implemented in the firmware, stored in FLASH memory 111, of an embedded processor 110, and/or downloaded from online.
  • said units 101, 102, 103, 104, 105, 106, 107, 108, and/or 109 described above, the optional processor 110 and/or the optional memory 111 may at least partly be comprised in the server 10, which server 10 may be comprised in and/or carried onboard the road-driven vehicle 2.
  • said units 101, 102, 103, 104, 105, 106, 107, 108, and/or 109 described above, and which will be described in more detail later on in this description, may refer to a combination of analog and digital circuits, and/or one or more processors configured with software and/or firmware, e.g. stored in a memory such as the memory 111, that when executed by the one or more processors such as the processor 110 perform as will be described in more detail in conjunction with Fig. 6 .
  • FIG. 5 is an exemplifying optional user interaction determining system 1070, adapted for determining user interaction by a user e.g. the occupant 3 with the simulated vehicle functionality feature - here the exemplifying simulated vehicle infotainment display.
  • the user interaction determining system 1070 may for instance comprise touch sensor(s), camera(s) and/or position detection sensor(s) worn by the user e.g. on his/her hand and/or finger.
  • an exemplifying optional vehicle signal determining unit 1080 adapted for determining which input derived from the road-driven vehicle 2 affects the simulated vehicle functionality feature - here the exemplifying simulated vehicle infotainment display - e.g. during manoeuvring of said vehicle 2.
  • Fig. 6 is a flowchart depicting an exemplifying method performed by a vehicle feature evaluation system 1 according to embodiments of the disclosure. Said method is for enabling evaluation of a simulated vehicle-related feature.
  • the exemplifying method which may be continuously repeated, comprises the following actions discussed with support from Figs. 1-5 .
  • the vehicle feature evaluation system 1 determines in relation to the road-driven vehicle 2 - e.g. by means of the HMD orientation determining unit 101 - with support from the tracking system 5, an orientation of the HMD 4 adapted to be worn by the occupant 3 on-board the road-driven vehicle 2.
  • an orientation of the HMD 4 adapted to be worn by the occupant 3 on-board the road-driven vehicle 2.
  • a position and/or location of an occupant-worn HMD 4 in view of a real vehicle 2 e.g. adapted to and/or intended to be driven outdoors such as in real traffic and/or along actual roads.
  • the vehicle feature evaluation system 1 determines - e.g. by means of the design feature determining unit 102 - a simulated vehicle design feature to be evaluated in the road-driven vehicle 2.
  • a virtual vehicle design feature - optionally a simulated interior section e.g. comprising at least a simulated dashboard and a simulated steering wheel - of interest to validate in a real vehicle 2.
  • the virtual interior section 7 - here the virtual interior section and/or virtual steering wheel - may be evaluated in the road-driven vehicle 2, such as e.g. new and/or updated design, colour, material and/or user interface thereof and/or e.g. geometrics for instance CAD and/or DSM geometrics thereof.
  • the simulated vehicle design feature may comprise a simulated vehicle exterior section.
  • the vehicle feature evaluation system 1 may determine - e.g. by means of the functionality feature determining unit 103 - a simulated vehicle functionality feature to be evaluated in the road-driven vehicle 2.
  • a virtual vehicle functionality feature - optionally a simulated vehicle infotainment display - of interest to validate in a real vehicle 2.
  • the virtual functionality feature 7 - here the virtual infotainment display 7 - may be evaluated in the road-driven vehicle 2, such as e.g. graphics thereof and/or new and/or updated functionality and/or design thereof.
  • the simulated vehicle functionality feature may comprise a simulated vehicle human machine interface, HMI, a simulated vehicle head-up display, HUD, simulated light characteristics, and/or a simulated effect resulting from a simulation of the simulated vehicle functionality feature.
  • the vehicle feature evaluation system 1 may determine - e.g. by means of the object feature determining unit 104 - a simulated object feature to be evaluated in the road-driven vehicle 2.
  • a virtual object feature 9 - optionally a simulated road user 91 - of interest to validate in a real vehicle 2.
  • a virtual road user 9 - and/or an effect it has on the HMD-wearing occupant 3 - may be evaluated in the road-driven vehicle 2.
  • the simulated object feature may comprise a simulated other vehicle, and/or a simulated moving or fixed obstacle.
  • the vehicle feature evaluation system 1 may determine - e.g. by means of the location determining unit 105 - a fictive design feature location 70 of the simulated vehicle design feature relative the road-driven vehicle 2.
  • the fictive design feature location 70 of the simulated vehicle design feature - optionally the simulated interior section - may be determined to have a location in relation to the vehicle 2, e.g. in the case of the simulated dashboard and/or simulated steering wheel be located for instance where the - to the HMD-wearing occupant 3 invisible - dashboard and steering wheel of the road-driven vehicle 2 are situated.
  • the vehicle feature evaluation system 1 may in optional Action 1005 the vehicle feature evaluation system 1, additionally or alternatively, determine - e.g. by means of the location determining unit 105 - a fictive functionality feature location 80 of the simulated vehicle functionality feature relative the road-driven vehicle 2.
  • the fictive functionality feature location 80 of the simulated vehicle functionality feature - optionally the simulated vehicle display - may be determined to have a location in relation to the vehicle 2, e.g. in the case of the simulated infotainment display be located essentially centered on the dashboard of the road-driven vehicle 2 and/or essentially centered on the virtual dashboard 7.
  • the vehicle feature evaluation system 1 may in optional Action 1005 the vehicle feature evaluation system 1, additionally or alternatively, determine - e.g. by means of the location determining unit 105 -a fictive object feature location 90 of the simulated object feature relative the road-driven vehicle 2.
  • the fictive object feature location 90 of the simulated vehicle object feature - optionally the simulated road user - may be determined to have a location in relation to the vehicle 2, e.g. in the case of the simulated pedestrian be located essentially a few meters in front of the road-driven vehicle 2.
  • the vehicle feature evaluation system 1 provides in real-time to the HMD display 41 - e.g. by means of the HMD providing unit 106 - taking into consideration the HMD orientation, the virtual representation 7 of the simulated vehicle design feature superimposed on the real-time surrounding-showing video stream 6 derived from real-world image data 211 captured with support from the one or more surrounding-capturing cameras 21 adapted to capture surroundings external of the road-driven vehicle 2.
  • the virtual representation 7 of the simulated vehicle design feature superimposed on the real-time surrounding-showing video stream 6 derived from real-world image data 211 captured with support from the one or more surrounding-capturing cameras 21 adapted to capture surroundings external of the road-driven vehicle 2.
  • the HMD display 41 there is provided to the HMD display 41 a real-time physical world view 6 of surroundings external the road-driven vehicle 2 - essentially as captured by the vehicle-attached camera(s) 21 - electronically mixed with the computer-generated virtual design feature representation 7 temporally and spatially commensurate with said physical world view 6 such that the surrounding-showing video stream 6 appears as a background overlaid with said virtual design feature representation 7. That is, with the vehicle-attached camera(s) 21 provided to capture surroundings external of the road-driven vehicle 2, the real-time surrounding-showing video stream 6 shows surroundings and/or an environment exterior of the vehicle 2, hence excluding interior - and potentially at least partial exterior - views of said road-driven vehicle 2.
  • HMD-wearing occupant 3 being on-board the road-driven vehicle 2 - e.g. seated in a driver's seat or passenger's seat thereof - and with the HMD orientation being taken into consideration - provided to the HMD display 41 is then a portion of surrounding-showing video stream 6 - derived from real-world image data 211 captured with support from the vehicle-attached camera(s) 21 - corresponding to and/or spatially commensurate with a field of view of the HMD-wearing occupant 3.
  • the HMD-wearing occupant 3 would accordingly be shown - on the HMD display(s) 41 - a physical world view spatially commensurate with said occupant's field of view, but from which physical world view an interior - and potentially at least a portion of an exterior - of the road-driven vehicle 2 would be excluded due to the vehicle-attached camera(s) 21 capturing surroundings external of - not internal of - the road-driven vehicle 2.
  • the real-time surrounding-showing video stream 6 may then represent what a driver or passenger of said road-driven vehicle 2 would essentially see even if not wearing the HMD 4, but with interior portions of the road-driven vehicle 2 removed.
  • one or more simulated design features - e.g. represented by a simulated updated and/or new interior or exterior or a portion thereof - may be evaluated in its place.
  • Action 1006 may comprise additionally providing in real-time to the HMD display 41, taking into consideration the HMD orientation, a virtual representation 8 of the simulated vehicle functionality feature superimposed on the real-time surrounding-showing video stream 6.
  • the simulated vehicle functionality feature may e.g. comprise a simulated vehicle human machine interface, HMI, a simulated vehicle display, a simulated vehicle head-up display, HUD, simulated light characteristics, and/or a simulated effect resulting from a simulation of the simulated vehicle functionality feature.
  • the virtual representation 8 of the simulated vehicle functionality feature is overlaid the real-world video stream 6 to be temporally and spatially commensurate with the video stream 6 such that said video stream 6 appears as a background overlaid with - in addition to the virtual design feature representation 7 - said virtual functionality feature representation 8.
  • a virtual representation 8 of a simulated vehicle functionality feature - such as for instance a virtual display with e.g. new and/or updated design and/or functionality - on the surrounding-showing video stream 6, said simulated vehicle functionality feature may be evaluated in the road-driven vehicle 2, e.g. while said road-driven vehicle 2 is driven in real traffic and/or along actual roads.
  • the simulated vehicle functionality feature may be evaluated in real traffic and/or in real road-driving scenarios, which thus enables said simulated vehicle functionality feature to be evaluated under more reality-resembling conditions and/or in a more reality-resembling environment than should said evaluation be performed in e.g. a commonly known static or dynamic rig displaying the environment and traffic on large scale screens in front of a vehicle mock-up.
  • Action 1006 may comprise additionally providing in real-time to the HMD display 41, taking into consideration the HMD orientation, a virtual representation 9 of the simulated object feature superimposed on the real-time surrounding-showing video stream 6.
  • the simulated object feature may e.g. comprise a simulated other vehicle, a simulated road user, and/or a simulated moving and/or fixed obstacle.
  • the virtual representation 9 of the simulated object feature is overlaid the real-world video stream 6 to be temporally and spatially commensurate with the video stream 6 such that said video stream 6 appears as a background overlaid with - in addition to the virtual design feature representation 7 - said virtual functionality feature representation 9.
  • a virtual representation 9 of a simulated object feature - such as for instance a virtual pedestrian appearing to cross the road in front of the road-driven vehicle 2 - on the surrounding-showing video stream 6, said simulated object feature and/or an effect it has on the HMD-wearing occupant 3 - who for instance may be the driver of the road-driven vehicle 2 - may be evaluated, e.g.
  • the simulated object feature - and/or an effect it has on the HMD-wearing occupant 3 - may be evaluated in real traffic and/or in real road-driving scenarios, which thus enables said simulated object feature and/or the effect it has on the HMD-wearing occupant 3 to be evaluated under more reality-resembling conditions and/or in a more reality-resembling environment than should said evaluation be performed in e.g. a commonly known static or dynamic rig displaying the environment and traffic on large scale screens in front of a vehicle mock-up.
  • the vehicle feature evaluation system 1 may update - e.g. by means of the user interaction updating unit 107 - the virtual functionality feature representation 8 based on a user interaction with the simulated vehicle functionality feature.
  • interaction by a user with the simulated vehicle functionality feature may result in that the virtual representation 8 of the simulated vehicle functionality feature is updated in accordance with said interaction.
  • a user selected option is carried out and/or updated in accordance with the user interaction, which subsequently is reflected by the corresponding virtual feature 8 - here the virtual infotainment display 8 - being updated in accordance therewith.
  • said user interaction may be based on detection, by means of one or more user interaction sensors, of a user in and/or at the fictive functionality feature location 80.
  • a finger of said user/occupant 3 may be sensed in or at the position in the vehicle 2 representing the location of the simulated vehicle functionality feature, in Figs. 1-3 in or at essentially the centre of the vehicle dashboard of the road-driven vehicle 2 and/or the virtual dashboard 7, whereby the virtual representation 8 of the simulated vehicle functionality feature, here the simulated infotainment display, subsequently may be updated in accordance with said presence, and/or in accordance with the geographical position and/or the nature of said presence.
  • User interaction may be detected in any arbitrary manner known in the art, e.g. by means of one or more user interaction sensors and/or the user interaction determining system 1070 shown in Fig. 5 , for instance comprising touch sensor(s), camera(s) and/or position detection sensor(s) worn by the user e.g. on his/her hand and/or finger.
  • the vehicle feature evaluation system 1 may update - e.g. by means of the vehicle signal updating unit 1008 - the virtual functionality feature representation 8' based on a vehicle signal affecting the simulated vehicle functionality feature, derived from the road-driven vehicle 2.
  • deriving a vehicle signal which affects the simulated vehicle functionality feature - e.g. resulting from manoeuvring of the road-driven vehicle 2 - may result in that the virtual representation 8' of the simulated vehicle functionality feature, in Figs. 1 and 3 the virtual information display 8', is updated in accordance therewith.
  • deriving - from the road-driven vehicle 2 - a vehicle signal affecting the simulated vehicle functionality feature may result in that the virtual feature 8' - in Figs. 1 and 3 the exemplifying virtual information display 8' - is updated in accordance with said vehicle signal, i.e. the fuel consumption value shown on the virtual information display 8' is updated along with the received updated fuel consumption value.
  • the vehicle signal may be derived from the road-driven vehicle 2 in any arbitrary known manner, for instance via wired and/or wireless communication therewith, and/or with support from the vehicle signal determining system and/or unit 1080 adapted for determining which input derived from the road-driven vehicle 2 affects the simulated vehicle functionality feature.
  • the vehicle feature evaluation system 1 may update - e.g. by means of the continuous updating unit 109 - the real-time surrounding-showing video stream 6, the superimposed virtual design feature representation 7, the superimposed virtual functionality feature representation 8 and/or the superimposed virtual object feature representation 9 based on updated captured image data 211 and/or based on updated determined HMD orientation.
  • the real-time video stream 6 and/or the virtual representation(s) 7, 8, 9 are continuously updated to reflect updated captured image data 211 and/or updated orientation of the HMD.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Geometry (AREA)
  • General Engineering & Computer Science (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Computer Hardware Design (AREA)
  • Evolutionary Computation (AREA)
  • Optics & Photonics (AREA)
  • Business, Economics & Management (AREA)
  • Educational Administration (AREA)
  • Educational Technology (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • Automation & Control Theory (AREA)
  • Computational Mathematics (AREA)
  • Mathematical Analysis (AREA)
  • Mathematical Optimization (AREA)
  • Pure & Applied Mathematics (AREA)
  • Processing Or Creating Images (AREA)
EP18210988.4A 2018-12-07 2018-12-07 Beurteilung einer simulierten fahrzeugbezogenen funktion Active EP3663941B1 (de)

Priority Applications (5)

Application Number Priority Date Filing Date Title
EP18210988.4A EP3663941B1 (de) 2018-12-07 2018-12-07 Beurteilung einer simulierten fahrzeugbezogenen funktion
US16/672,756 US11257390B2 (en) 2018-12-07 2019-11-04 Evaluation of a simulated vehicle-related feature
CN201911180422.0A CN111290571B (zh) 2018-12-07 2019-11-27 评估模拟的车辆相关特征
US17/574,726 US11938819B2 (en) 2018-12-07 2022-01-13 Evaluation of a simulated vehicle-related feature
US18/607,659 US20240217339A1 (en) 2018-12-07 2024-03-18 Evaluation of a simulated vehicle-related feature

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
EP18210988.4A EP3663941B1 (de) 2018-12-07 2018-12-07 Beurteilung einer simulierten fahrzeugbezogenen funktion

Publications (2)

Publication Number Publication Date
EP3663941A1 true EP3663941A1 (de) 2020-06-10
EP3663941B1 EP3663941B1 (de) 2022-07-13

Family

ID=64901821

Family Applications (1)

Application Number Title Priority Date Filing Date
EP18210988.4A Active EP3663941B1 (de) 2018-12-07 2018-12-07 Beurteilung einer simulierten fahrzeugbezogenen funktion

Country Status (3)

Country Link
US (1) US11257390B2 (de)
EP (1) EP3663941B1 (de)
CN (1) CN111290571B (de)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113312776A (zh) * 2021-06-02 2021-08-27 北京五一视界数字孪生科技股份有限公司 行车轨迹的泛化方法、装置、存储介质及电子设备

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11938819B2 (en) * 2018-12-07 2024-03-26 Volvo Car Corporation Evaluation of a simulated vehicle-related feature
JP7018923B2 (ja) * 2019-12-13 2022-02-14 本田技研工業株式会社 駐車支援装置、駐車支援方法およびプログラム
US20220204009A1 (en) * 2020-12-29 2022-06-30 Waymo Llc Simulations of sensor behavior in an autonomous vehicle
US11562550B1 (en) * 2021-10-06 2023-01-24 Qualcomm Incorporated Vehicle and mobile device interface for vehicle occupant assistance
IT202200002903A1 (it) * 2022-02-16 2023-08-16 Granstudio S R L Sistema per la progettazione, revisione e/o presentazione di soluzioni prototipali per veicoli, procedimento di funzionamento e prodotto informatico corrispondenti
US20240310628A1 (en) * 2023-03-16 2024-09-19 Ford Global Technologies, Llc Vehicle hud with external screen projection

Family Cites Families (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8624924B2 (en) * 2008-01-18 2014-01-07 Lockheed Martin Corporation Portable immersive environment using motion capture and head mounted display
DE102011084596A1 (de) * 2011-10-17 2013-04-18 Robert Bosch Gmbh Verfahren zum Assistieren eines Fahrers in einer fremden Umgebung
IL308285B2 (en) * 2013-03-11 2024-11-01 Magic Leap Inc System and method for augmentation and virtual reality
DE102014206623A1 (de) * 2014-04-07 2015-10-08 Bayerische Motoren Werke Aktiengesellschaft Lokalisierung eines Head-mounted Displays (HMD) im Fahrzeug
DE102014208310A1 (de) * 2014-05-05 2015-11-05 Bayerische Motoren Werke Aktiengesellschaft Fahrerassistenzsystem zur Darstellung einer erweiterten Realität
DE102014208352A1 (de) * 2014-05-05 2015-11-05 Bayerische Motoren Werke Aktiengesellschaft System und Verfahren zum Instruieren eines Teilnehmers eines Fahrertrainings
US20180182261A1 (en) * 2014-07-02 2018-06-28 Iotmotive Ltd. Real Time Car Driving Simulator
US20160321940A1 (en) * 2015-04-29 2016-11-03 Ivan Banga Driver Education and Training System and Method for Training Drivers in Simulated Emergencies
EP3407323A4 (de) * 2016-01-22 2019-09-11 Volvo Truck Corporation Fahrbeurteilungsvorrichtung, fahrbeurteilungsverfahren, verfahren zur bereitstellung von referenzdaten, vorrichtung zur bereitstellung von referenzdaten und programm zur bereitstellung von referenzdaten
CN106274690A (zh) * 2016-08-25 2017-01-04 乐视控股（北京）有限公司 一种车辆周围景象监测方法和系统
KR102306159B1 (ko) * 2016-09-23 2021-09-29 애플 인크. 몰입형 가상 디스플레이
US10127728B2 (en) * 2016-09-30 2018-11-13 Sony Interactive Entertainment Inc. Facial feature views of user viewing into virtual reality scenes and integration of facial features into virtual reality views into scenes
EP4155867A1 (de) * 2016-11-14 2023-03-29 Logitech Europe S.A. System zum importieren von benutzerschnittstellenvorrichtungen in virtuelle/erweiterte realität
US10546560B2 (en) * 2017-10-31 2020-01-28 Uatc, Llc Systems and methods for presenting virtual content in a vehicle
US10410426B2 (en) * 2017-12-19 2019-09-10 GM Global Technology Operations LLC Augmented reality vehicle user interface

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
BEN HOBSON: "MINI's augmented-reality glasses can make cars transparent", 24 April 2015 (2015-04-24), pages 1 - 10, XP055591292, Retrieved from the Internet <URL:https://www.dezeen.com/2015/04/24/mini-augmented-reality-glasses-allow-drivers-to-see-through-the-body-of-their-car/> [retrieved on 20190523] *
FRUND J ET AL: "Cooperative design support within automobile advance development using augmented reality technology", COMPUTER SUPPORTED COOPERATIVE WORK IN DESIGN, 2004. PROCEEDINGS. THE 8TH INTERNATIONAL CONFERENCE ON XIAMEN, CHINA MAY 26-28, 2004, PISCATAWAY, NJ, USA,IEEE, vol. 2, 26 May 2004 (2004-05-26), pages 528 - 533, XP010737149, ISBN: 978-0-7803-7941-1, DOI: 10.1109/CACWD.2004.1349239 *
PURSCHKE F ET AL: "Virtual reality-new methods for improving and accelerating the development process in vehicle styling and design", COMPUTER GRAPHICS INTERNATIONAL, 1998. PROCEEDINGS HANNOVER, GERMANY 22-26 JUNE 1998, LOS ALAMITOS, CA, USA,IEEE COMPUT. SOC, US, 22 June 1998 (1998-06-22), pages 789 - 797, XP010291510, ISBN: 978-0-8186-8445-6, DOI: 10.1109/CGI.1998.694338 *
TOSHIKAZU OHSHIMA ET AL: "A Mixed Reality System with Visual and Tangible Interaction Capability - Application to Evaluating Automobile Interior Design", PROCEEDINGS / THE SECOND IEEE AND ACM INTERNATIONAL SYMPOSIUM ON MIXED AND AUGMENTED REALITY : 7 - 10 OCTOBER 2003, TOKYO, JAPAN, IEEE COMPUTER SOCIETY, LOS ALAMITOS, CALIF. [U.A.], 7 October 2003 (2003-10-07), pages 284, XP058187396, ISBN: 978-0-7695-2006-3 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113312776A (zh) * 2021-06-02 2021-08-27 北京五一视界数字孪生科技股份有限公司 行车轨迹的泛化方法、装置、存储介质及电子设备
CN113312776B (zh) * 2021-06-02 2023-11-24 万物镜像(北京)计算机系统有限公司 行车轨迹的泛化方法、装置、存储介质及电子设备

Also Published As

Publication number Publication date
CN111290571A (zh) 2020-06-16
US20200184841A1 (en) 2020-06-11
EP3663941B1 (de) 2022-07-13
CN111290571B (zh) 2023-07-21
US11257390B2 (en) 2022-02-22

Similar Documents

Publication Publication Date Title
EP3663941B1 (de) Beurteilung einer simulierten fahrzeugbezogenen funktion
US11938819B2 (en) Evaluation of a simulated vehicle-related feature
US10453256B2 (en) Lane boundary detection data generation in virtual environment
CN113474825B (zh) 用于在移动平台上提供沉浸式扩展现实体验的方法和装置
US20170109458A1 (en) Testbed for lane boundary detection in virtual driving environment
JP2021508027A (ja) 不十分な照明条件下で車両を測位するシステムおよび方法
Galko et al. Vehicle-Hardware-In-The-Loop system for ADAS prototyping and validation
US20190141310A1 (en) Real-time, three-dimensional vehicle display
EP3869302A1 (de) Fahrzeug, vorrichtung und verfahren zur verringerung des auftretens der reisekrankheit
JP2022117991A (ja) 表示装置
Miquet New test method for reproducible real-time tests of ADAS ECUs:“Vehicle-in-the-Loop” connects real-world vehicles with the virtual world
Weber et al. Approach for improved development of advanced driver assistance systems for future smart mobility concepts
US11113435B2 (en) Evaluation of a simulated vehicle functionality feature
Galazka et al. CiThruS2: Open-source photorealistic 3D framework for driving and traffic simulation in real time
KR101020505B1 (ko) 3차원 자차마크 표시 장치 및 그 방법
JP6481596B2 (ja) 車両用ヘッドアップディスプレイの評価支援装置
EP3591624B1 (de) Erweitertes sehen in einer bildsequenz, aufgenommen aus einem sich bewegenden fahrzeug
CN114327079B (zh) 试驾效果呈现装置、方法和存储介质
JP7281742B2 (ja) プログラムおよび列車運転シミュレータ
CN111108530B (zh) 信息显示装置和方法以及记录介质
Brogan et al. Automatic generation and population of a graphics-based driving simulator: Use of mobile mapping data for behavioral testing of drivers
Zair Collision warning design in automotive head-up displays
CN110929368A (zh) 用于执行计算机辅助仿真的系统
CN118485807A (zh) 基于图像显示的ar驾驶辅助方法及终端设备
CN116228587A (zh) 一种智能3d倒车影像增强系统及方法

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20201210

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602018037840

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: G06F0017500000

Ipc: G06F0030150000

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

RIC1 Information provided on ipc code assigned before grant

Ipc: G06F 30/20 20200101ALI20220214BHEP

Ipc: G06F 30/15 20200101AFI20220214BHEP

INTG Intention to grant announced

Effective date: 20220311

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602018037840

Country of ref document: DE

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1504644

Country of ref document: AT

Kind code of ref document: T

Effective date: 20220815

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20220713

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220713

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220713

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221114

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221013

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220713

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220713

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220713

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220713

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220713

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: IT

Payment date: 20221122

Year of fee payment: 5

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1504644

Country of ref document: AT

Kind code of ref document: T

Effective date: 20220713

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220713

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221113

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220713

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20221014

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602018037840

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220713

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220713

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220713

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220713

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220713

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220713

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220713

26N No opposition filed

Effective date: 20230414

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220713

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20221207

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20221231

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220713

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20221207

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20221231

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20221207

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20221207

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20221231

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20221231

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20231212

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20231122

Year of fee payment: 6

Ref country code: DE

Payment date: 20231121

Year of fee payment: 6

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20181207

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220713

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220713

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220713

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220713

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220713

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220713

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220713