EP3142902B1 - Dispositif d'affichage et vehicule - Google Patents
Dispositif d'affichage et vehicule Download PDFInfo
- Publication number
- EP3142902B1 EP3142902B1 EP15793397.9A EP15793397A EP3142902B1 EP 3142902 B1 EP3142902 B1 EP 3142902B1 EP 15793397 A EP15793397 A EP 15793397A EP 3142902 B1 EP3142902 B1 EP 3142902B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- image
- passenger
- driver
- display
- capture unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000003287 optical effect Effects 0.000 claims description 19
- 210000001508 eye Anatomy 0.000 claims description 10
- 230000000007 visual effect Effects 0.000 claims description 7
- 230000004397 blinking Effects 0.000 claims description 6
- 238000005259 measurement Methods 0.000 claims 3
- 230000001419 dependent effect Effects 0.000 claims 1
- 238000000034 method Methods 0.000 description 28
- 238000012545 processing Methods 0.000 description 23
- 230000006870 function Effects 0.000 description 12
- 230000008569 process Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 9
- 238000005516 engineering process Methods 0.000 description 7
- 238000013461 design Methods 0.000 description 5
- 230000033001 locomotion Effects 0.000 description 5
- 210000001747 pupil Anatomy 0.000 description 5
- 210000005252 bulbus oculi Anatomy 0.000 description 4
- 206010041349 Somnolence Diseases 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 2
- 230000006866 deterioration Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 206010029216 Nervousness Diseases 0.000 description 1
- 230000005856 abnormality Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 230000004424 eye movement Effects 0.000 description 1
- 210000000744 eyelid Anatomy 0.000 description 1
- 230000001815 facial effect Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012806 monitoring device Methods 0.000 description 1
- 230000000414 obstructive effect Effects 0.000 description 1
- 210000004243 sweat Anatomy 0.000 description 1
- 230000036962 time dependent Effects 0.000 description 1
- 238000002834 transmittance Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/44—Receiver circuitry for the reception of television signals according to analogue transmission standards
- H04N5/445—Receiver circuitry for the reception of television signals according to analogue transmission standards for displaying additional information
- H04N5/44504—Circuit details of the additional information generator, e.g. details of the character or graphics signal generator, overlay mixing circuits
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
- B60K35/20—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor
- B60K35/21—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor using visual output, e.g. blinking lights or matrix displays
- B60K35/23—Head-up displays [HUD]
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
- B60K35/20—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor
- B60K35/28—Output arrangements, i.e. from vehicle to user, associated with vehicle functions or specially adapted therefor characterised by the type of the output information, e.g. video entertainment or vehicle dynamics information; characterised by the purpose of the output information, e.g. for attracting the attention of the driver
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
- B60K35/60—Instruments characterised by their location or relative disposition in or on vehicles
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R1/00—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
- B60R1/008—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles allowing the driver to see passengers, e.g. for busses
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
- B60K2360/16—Type of output information
- B60K2360/178—Warnings
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
- B60K2360/20—Optical features of instruments
- B60K2360/21—Optical features of instruments using cameras
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
- B60K2360/77—Instrument locations other than the dashboard
- B60K2360/777—Instrument locations other than the dashboard on or in sun visors
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/10—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used
- B60R2300/107—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used using stereoscopic cameras
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/20—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of display used
- B60R2300/205—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of display used using a head-up display
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/80—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement
- B60R2300/8006—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement for monitoring and displaying scenes of vehicle interior, e.g. for monitoring passengers or cargo
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/0138—Head-up displays characterised by optical features comprising image capture systems, e.g. camera
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/0141—Head-up displays characterised by optical features characterised by the informative content of the display
Definitions
- the present invention relates to a display device and a vehicle including the display device.
- the current HuD technology may be appreciated for providing safe driving by reducing eye movement during manual driving. It is highly controversial as to how the human behavior inside a vehicle will be changed, or how an interior of a vehicle will be changed in a society in which the above-described autonomous driving becomes common. While it is highly controversial, it is expected that, at least, a function will be necessary which is for notifying passengers of the "the current surrounding conditions of their own vehicle.”
- a visual field can be significantly shifted, and the visual field can be deviated from the traveling direction of the vehicle. Such a situation may not be suitable for traffic safety. Even if a driver does not face a passenger, when the driver utilizes, for example, a navigation device, the driver's line of sight may be deviated from the traveling direction of the vehicle, thereby lowering safety.
- US2004/056954 A1 discloses a rear seat monitor for a vehicle.
- the vehicle monitoring system includes a camera to capture the image of the vehicle rear seat and its occupants, for example a baby/infant/child in a car seat, and a triggering system to enable the capturing and displaying of the images.
- the triggering system could be based on motion sensors, which could be image based, or sound sensors.
- the display includes a HUD.
- WO 2008/052178 A2 JOHNSON CONTROLS TECH CO [US] CHUTORASH RICHARD J [US]; SIMS MICHAEL) 2 May 2008 (2008-05-02), discloses background information;
- WO 2014/068892 A1 discloses a passenger monitoring device to monitor the status of the driver and the passengers of a vehicle. The monitoring is performed by video with an image capture device. The status of the monitored occupant is derived from the acquired images and a display, that could be a HUD, shows the condition of the passengers.
- a display device as defined in claim 1.
- a display device or the like can be provided with which, while preventing deterioration in safety, a driver can confirm a situation of a passenger.
- FIG. 1 is a diagram exemplifying a display device 1 according to an embodiment of the present invention.
- the display device 1 is included in a vehicle 120.
- the display device 1 may include a driver image capture unit 10; a forward image capture unit 15; a passenger image capture unit 20; an image processing unit 40; an image display unit 50; and a passenger display 60, for example.
- the driver image capture unit 10, the forward image capture unit 15 and the passenger display 60 can be optional components of the display device 1, and they may not be essential components of the display device 1.
- the embodiment is explained by exemplifying a car. However, the embodiment is not limited to this, and the display device 1 according to the embodiment can be included in an airplane, a ship, or robot, for example.
- a "vehicle” may refer to a car, an airplane, a ship, or a robot, for example.
- the driver image capture unit 10 may include a function for capturing an image of a driver 130 so as to determine a condition of the driver 130 of the vehicle 120.
- a monocular camera or a binocular camera (a stereo camera) may be used, for example.
- the driver image capture unit 10 can be arranged so that the driver image capture unit 10 can capture images in the vicinity of both eyes of the driver 130 as an angle of view.
- the driver image capture unit 10 may be arranged at any position in the vehicle 120 in accordance with interior design of the vehicle 120.
- the drive image capture unit 10 may be arranged at a ceiling portion of the vehicle 120.
- the driver image capture unit 10 may be arranged at a top of a dashboard of the vehicle 120, for example.
- the forward image capture unit 15 may include a function for capturing an image in front of the vehicle 120 so as to determine presence or absence of an object of which the driver 130 needs to be alerted to in front of the vehicle 120 (e.g., a preceding vehicle, or a pedestrian), and/or a distance between the vehicle 120 and a preceding vehicle.
- a monocular camera or a binocular camera a stereo camera
- the forward image capture unit 15 may be arranged at any position in the vehicle 120 in accordance with interior design of the vehicle 120.
- the forward image capture unit 15 may be arranged at a ceiling portion of the vehicle 120.
- the forward image capture unit 15 may be arranged at a top of a dashboard of the vehicle 120.
- the passenger image capture unit 20 includes a function for capturing an image of the passenger 140 so as to determine a condition of the passenger 140 that is riding in the vehicle 120 together with the driver 130.
- a monocular camera or a binocular camera may be used, for example.
- the passenger image capture unit 20 may be arranged at any position in the vehicle 120 in accordance with interior design of the vehicle 120.
- the passenger image capture unit 20 may be arranged at a rear upper portion of a front seat of the vehicle 120.
- the passenger image capture unit 20 may be arranged at a ceiling portion of the vehicle 120.
- the passenger image capture unit 20 may preferably have an angle of view that is wider than that of the driver image capture unit 10. Namely, the passenger image capture unit 20 may preferably be arranged, so that the whole body of the passenger 140 can be captured within the angle of view of the passenger image capture unit 20. That is because the passenger image capture unit 20 may be required to determine various conditions of the passenger 140. For example, when three passengers can be accommodated in a rear seat, the passenger image capture unit 20 may preferably have an angle of view that is wide enough to capture conditions of all three passengers.
- a plurality of image capture units may be arrange in the vehicle 120 as the passenger image capture unit 20.
- the image capture units may be arranged so as to ensure that an image of the whole of the rear seat can be captured.
- an image capture unit that is different from the image capture unit for the rear seat may be provided so as to determine a condition of a passenger in a front passenger seat.
- An example of a position of the image capture unit that is for determining the condition of the passenger in the front passenger seat may be a position above a dashboard at the side of the front passenger seat.
- the passenger 140 can be a child who is sitting in a child safety seat, for example.
- the embodiment is not limited to this.
- the passenger 140 may be an adult who is sitting on the rear seat of the vehicle 120.
- the passenger 140 may be sitting on the front passenger seat, instead of the rear seat.
- the image processing unit 40 may include a function for generating a superimposition image based on information that is obtained from the driver image capture unit 10 and the passenger image capture unit 20, and for outputting the superposition image on the display unit 50.
- the image processing unit 40 may be arranged at any position within the vehicle 120.
- the image processing unit 40 may have a configuration that includes a central processing unit (CPU); a read-only memory (ROM); and a main memory, for example.
- various functions of the image processing unit 40 can be implemented by reading out a program that is stored in the ROM or the like onto the main memory and executing the program by the CPU. Note that a part of or all the image processing unit 40 may be implemented only by hardware. Additionally, the image processing unit 40 may physically include a plurality of devices.
- the image display unit 50 is a so-called "heads-up display” that includes a function to display, in the vehicle 120, the image that is generated by the image processing unit 40 as a virtual image that is superimposed onto the view of the driver 130.
- the image display unit 50 may be arranged at any position in the vehicle 120 in accordance with interior design of the vehicle 120.
- the image display unit 50 may be arranged at a position above the dashboard in the vehicle 120.
- the image display unit 50 may be embedded in the dashboard.
- the image display unit 50 can enlarge, by using a mirror, a lens, and/or the like, an intermediate image that is internally generated, and the image display unit 50 can display the enlarged image as a virtual image.
- the image display unit 50 can display the virtual image such that the driver 130 can view the virtual image with a feeling as if the virtual image were separated from a visual point of the driver 130 by a predetermined distance.
- the image display unit 50 may be implemented as a panel projection type display, or a laser scanning type display, for example.
- the image display unit 50 may be any type of these displays.
- the laser scanning type display may be preferable to use in this embodiment because the laser scanning type display may allow widening of an angle of view of the virtual image, and the laser scanning type display may display a high brightness image that is robust against external light.
- a laser scanning type display is used as an example of the image display unit 50.
- FIG. 2 is a diagram exemplifying the image display unit 50 according to the embodiment.
- the image display unit 50 may include, at least, a light source 51; an optical deflector 52; a first mirror 53; a scanned surface 54; and a second mirror 55.
- the numerical reference 135 shows an eyeball of the driver 130 (which is referred to as the "eyeball 135," hereinafter), and the reference numeral 110 shows the virtual image (which is referred to as the "virtual image 110,” hereinafter).
- the light source 51 may include three laser light sources that correspond to "R,” “G” and “B,” respectively; a coupling lens; an aperture; a synthetic optical element; and a lens, for example.
- Laser beams that are emitted from the three laser light sources, respectively, are synthesized, and the synthesized light beam is guided toward a reflection surface of the optical deflector 52.
- the laser beam that is guided toward the reflection surface of the optical deflector 52 is two-dimensionally deflected by the optical deflector 52.
- one micromirror may be used that fluctuates with respect to two perpendicular axes.
- two micromirrors may be used each of which fluctuates or pivots with respect to one axis.
- the optical deflector 52 may be Micro Electro Mechanical Systems (MEMS) that can be produced by semiconductor processing.
- MEMS Micro Electro Mechanical Systems
- the optical deflector 52 can be driven by an actuator that utilizes deformation force of a piezoelectric element as the driving force.
- a concave mirror may be used as an example of the first mirror 53.
- a reflection surface of the first mirror 53 may be an anamorphic mirror, for example. Namely, the reflection surface of the first mirror 53 may be such that a curvature in a predetermined direction is different from a curvature in the direction that is perpendicular to the predetermined direction.
- the reflection surface of the first mirror 53 is an anamorphic mirror, a curved surface shape of the reflection surface can be adjusted, and aberration correction performance can be enhanced.
- the scanned surface 54 can be a transparent surface on which a two-dimensional image is formed by a light beam bundle that is reflected on the first mirror 53 and enters the scanned surface 54.
- the scanned surface 54 has a function to diverge a synthesized laser beam at a desired divergence angle.
- the scanned surface 54 may preferably have a micro-lens array structure.
- the light beam bundle that is emitted from the scanned surface 54 is enlarged and displayed by the second mirror 55 and a semi-transparent mirror 59.
- a concave mirror can be used, for example.
- the image display unit 50 may include a transmission type optical element, such as a lens or a prism.
- the semi-transparent mirror 59 can be a mirror whose transmittance in a visible region is 10 to 70%.
- the semi-transparent mirror 59 On one side of the semi-transparent mirror 59 onto which the light beam bundle that is reflected by the second mirror 55 is incident, the semi-transparent mirror 59 has a reflection surface on which a dielectric multilayer film or a wired grid is formed, for example.
- the reflection surface of the semi-transparent mirror 59 may be a mirror surface such that a light beam bundle that is emitted by a laser and that is within a predetermined wavelength range is selectively reflected.
- the reflection surface of the semi-transparent mirror 59 may be a reflection surface that is formed so that a reflection coefficient is increased for a light beam bundle having reflection peaks or reflection bandwidths that include light beams emitted from three lasers corresponding to "R,” "G” and "B.”
- the reflection surface of the semi-transparent mirror 59 may be a reflection surface that is formed so that a reflection coefficient is increased for a light beam bundle in a specific deflection direction.
- the semi-transparent mirror 59 can be integrated with a front windshield 125 (cf. FIG. 1 ) of the vehicle 120.
- a light beam bundle that is reflected by the reflection surface of the semi-transparent mirror 59 can enter the eyeball 135 of the driver 130 in a driver's seat.
- a two-dimensional image of the scanned surface 54 can be viewed by the driver 130 as an enlarged virtual image 110 at a predetermined position that is a forward position relative to the position of the reflection surface of the semi-transparent mirror 59.
- a so-called "heads-up display" can be achieved.
- the passenger display 60 may have a function for displaying an image that is captured by the driver image capture unit 10.
- the passenger display 60 can be arranged at any position in the vehicle 120 in accordance with the interior design of the vehicle 120, provided that an image on the passenger display 60 can be viewed by the passenger 140.
- the passenger display 60 may be arranged at a rear upper portion of the front seat of the vehicle 120.
- FIG. 3 an example of an algorithm for processing information that is obtained by the image processing unit 40 from the driver image capture unit 10 and the passenger image capture unit 20.
- Optical information that is obtained from the driver image capture unit 10 is transmitted to a driver's line-of-sight detector 42 of the image processing unit 40.
- the driver's line-of-sight detector 42 can detect a direction of the line of sight of the driver 130 based on the optical information that is captured by the driver image capture unit 10. Specifically, when the image information of the eyes is to be used, for example, a method can be used that is for detecting a direction of the line of sight by tracking the movement of the pupils.
- a method can be used that is for detecting a direction of the line of sight by recognizing characteristic image areas of the face (e.g., the eyes, the mouth, and/or the nose), or a method can be used that is for detecting a direction of the line of sight from facial contour information.
- the direction of the line of sight of the driver 130 that is detected by the driver's line-of-sight detector 42 is transmitted to an image generator 43.
- the optical information that is obtained by the passenger image capture unit 20 is transmitted to a passenger condition detector 41. Additionally, the optical information that is obtained by the passenger image capture unit 20 is transmitted to the image generator 43 of the image processing unit 40 via the passenger condition detector 41 of the image processing unit 40.
- the passenger condition detector 41 can detect, based on the optical information that is captured by the passenger capture unit 20, a condition of the passenger 140 (e.g., a degree of fatigue, or a degree of excitement) by image recognition, for example.
- the face of the passenger 140 is recognized and images corresponding to the areas of the eyes are extracted. From time-dependent variation of the extracted image, opening and closing of the eyelids are detected, thereby calculating blinking time (for example, a time period, per blink, within which it can be recognized that the eyes are closed) of the passenger 140. Then, by comparing the current blinking time with the normal blinking time, the degree of fatigue of the passenger 140 (e.g., a high degree of drowsiness) can be detected.
- blinking time for example, a time period, per blink, within which it can be recognized that the eyes are closed
- the degree of excitement of the passenger 140 (e.g., being irritated) can be detected.
- the condition of the passenger 140 that is detected by the passenger condition detector 41 is transmitted to the image generator 43 of the image processing unit 40.
- a condition of the passenger 140 at the start of the driving can be used, or a value that is estimated from past history or the like can be used.
- data of a normal condition can be stored, for example, in a ROM, in advance.
- FIG. 4 is a diagram exemplifying a hardware configuration when the driver image capture unit 10 is a stereo camera.
- the driver image capture unit 10 may include a first camera unit 12 and a second camera unit 13.
- One of the first camera unit 12 and the second camera unit 13 is for the left eye, and the other is for the right eye.
- the first camera unit 12 and the second camera unit 13 are arranged in parallel, depth information of a captured object can be obtained by utilizing disparity information between the first camera unit 12 and the second camera unit 13. Namely, three-dimensional information of the captured object can be obtained.
- the first camera unit 12 and the second camera unit 13 are arranged substantially parallel to each other, for example.
- the first camera unit 12 may include a lens 12a; an image sensor 12b; and a sensor controller 12c.
- the second camera unit 13 may include a lens 13a; an image sensor 13b; and a sensor controller 13c.
- an image sensor such as a charge coupled device (CCD) or a complementary material oxide semiconductor device (CMOS)
- CCD charge coupled device
- CMOS complementary material oxide semiconductor device
- the sensor controller 12c may include functions for executing exposure control of the image sensor 12b; image reading control; communication with an external circuit; and image data transmission control.
- the sensor controller 13c may include functions for executing exposure control of the image sensor 13b; image reading control; communication with an external circuit; and image data transmission control.
- Each of the sensor controllers 12c and 13c may include a CPU; a ROM; and so forth.
- the driver image capture unit 10 can be connected to the image processing unit 40 through a data bus line or a serial bus line, for example.
- the image processing unit 40 can generate a brightness image and a disparity image based on optical information that is obtained from the driver image capture unit 10. Additionally, the image processing unit 40 can recognize an object by executing various processes, such as a context recognition process.
- a portion of or all the functions of the image processing unit 40 may be included in the driver image capture unit 10. Namely, a configuration can be used such that, for the driver image capture unit 10, a CPU and the like are provided in the same housing, and thereby image processing can be executed. Note that when the forward image capture unit 15 is a stereo camera, a hardware configuration that is the same as that of FIG. 4 can be used. Similarly, when the passenger image capture unit 20 is a stereo camera, a hardware configuration that is the same as that of FIG. 4 can be used.
- the image generator 43 of the image processing unit 40 can generate an image that shows a condition of the passenger 140, and the image generator 43 can transmit the generated image to the image display unit 50.
- the image display unit 50 can display an image that is generated by the image generator 43 while superimposing the image with a view of the driver 130.
- the image generator 43 can recognize that the passenger 140 is sleepy, for example, by the information from the passenger condition detector 41, the image generator 43 can generate an image that is obtained by adding a text, "sleepy," to an image of the passenger 140 that is captured by the passenger image capture unit 20, and the image generator 43 can transmit the generated image to the image display unit 50.
- the image display unit 50 can display not only the image of the passenger 140, but also an image of the whole rear seat that includes the passenger 140.
- the image generator 43 may determine whether the generated image is to be displayed as an interrupt image by the image display unit 50, based on a change in a direction of the line of sight of the driver 130 that is detected by the driver's line-of-sight detector 42. For example, during driving in an unfamiliar place, in a situation in which a distance between vehicles is short, or in a situation in which there are many objects of which the driver 130 needs to be alerted to, the attention of the driver 130 can diverge. Thus, the trajectory of the line of sight may be complicated, and the moving range can be enlarged.
- the image generator 43 may not transmit the generated image to the image display unit 50 so that the driver 130 can focus on driving.
- the image generator 43 can determine whether the generated image is to be transmitted to the image display unit 50, based on the information that is obtained from the forward image capture unit 15.
- the image generator 43 may transmit the generated image to the image display unit 50. In this manner, the image display unit 50 can display an image that is superimposed with the view of the driver 130. However, in certain conditions, such as the above-described conditions, the image display unit 50 may not display an image that is superimposed with the view of the driver 130.
- a method of detecting a direction of a line of sight for determining whether an image is to be displayed for example, a method can be used in which pupils are detected, and the direction of the line of sight is detected in real-time from the positions of the pupils and the positions of the eyes (cf. Japanese Unexamined Patent Publication No. 2003-15816 , for example).
- another method of detecting a direction of the line of sight can be used.
- a normal direction of a plane that includes points on the iris outline is obtained from three-dimensional coordinates of the eye contour, and the center of the iris is obtained at the same time.
- the method detects, among the normal lines of the plane that includes the points on the iris outline, a line that passes through the center of the iris as a direction of a line of sight (cf. Japanese Unexamined Patent Publication No. 2004-255074 , for example).
- another method of detecting the direction of the line of sight can be used.
- a three-dimensional position of the center of a face is calculated as a reference position.
- a three-dimensional position of the center point between the center of the left pupil and the center of the right pupil in the lateral direction is calculated as a characteristic position.
- the method detects the direction of the line of sight based on a shift amount of the characteristic position relative to the reference position (cf. WO 2008/007781 , for example).
- a method other than the methods that are exemplified herein can be used as a method of detecting the direction of the line of sight.
- the image generator 43 may not transmit the generated image to the image display unit 50 (i.e., the image display unit may not display the image).
- the determination as to whether the current location is an unfamiliar place can be made by storing location information of the places that have been visited in the past and determining that the current location is a location that has not been visited in the past.
- the driver 130 may be allowed to select by using a switch whether the current location is an unfamiliar place.
- the image generator 43 may not transmit the generated image to the image display unit 50 (i.e., the image display unit may not display the image).
- the determination as to whether the distance between the vehicles is short can be made by determining whether the distance between the vehicle 120 and the preceding vehicle is greater than a threshold value, as a result that the forward image capture unit 15 of FIG. 1 recognizes the preceding vehicle and that the forward image capture unit 15 measures the distance between the preceding vehicle and the vehicle 120.
- a millimeter wave radar or a laser radar can be used for measuring a distance.
- the image generator 43 may not transmit the generated image to the image display unit 50 (i.e., the image display unit may not display the image).
- the determination as to whether there are many objects of which the driver 130 needs to be alerted to can be made by determining, as a result that the forward image capture unit 15 recognizes the objects of which the driver 130 needs to be alerted to, such as preceding vehicles and/or pedestrians, whether the number of the objects of which the driver 130 needs to be alerted to during driving of the vehicle 120 is greater than a threshold value.
- step S501 the image generator 43 generates an image.
- step S502 the image generator 43 determines whether the driver 130 is driving in an unfamiliar place.
- step S507 the image generator 43 does not transmit the generated image to the image display unit 50.
- the image display unit 50 does not display the image.
- step S502 When a determination is made at step S502 that the driver 130 is not driving in an unfamiliar place, the process proceeds to step S503, and the image generator 43 determines whether a distance between vehicles is short. Then, when a determination is made at step S503 that the distance between the vehicles is short, the process proceeds to step S507, and the image generator 43 does not transmit the generated image to the image display unit 50. As a result, the image display unit 50 does not display the image.
- step S503 When a determination is made at step S503 that the distance between the vehicles is not short, the process proceeds to step S504, and the image generator 43 determines whether there are many objects of which the driver 130 needs to be alerted to. Then, when a determination is made at step 504 that there are many objects of which the driver 130 needs to be alerted to, the process proceeds to step S507, and the image generator 43 does not transmit the generated image to the image display unit 50. As a result, the image display unit 50 does not display the image.
- step S504 When a determination is made at step S504 that there are not so many objects of which the driver 130 needs to be alerted to, the process proceeds to step S505, and the image generator 43 determines whether a direction of a line of sight of the driver 130 is moving in a complicated manner, and the image generator 43 determines whether the direction of the line of sight of the driver is moving extensively. Then, when a determination is made at step S505 that the direction of the line of sight of the drive is moving in a complicated manner, or that the direction of the line of sight of the driver is moving extensively, the process proceeds to step S507, and the image generator 43 does not transmit the generated image to the image display unit 50. As a result, the image display unit 50 does not display the image.
- step S505 When a determination is made at step S505 that the direction of the line of sight of the driver is not moving in a complicated manner, and that the direction of the line of sight of the driver is not moving extensively, the process proceeds to step S506, and the image generator 43 transmits the generated image to the image display unit 50. As a result, the image display unit 50 displays the image.
- the image generator 43 may not transmit the generated image to the image display unit 50.
- the image display unit 50 does not display a condition of the passenger 140, and the image display unit 50 can preferentially display information that may be required for driving the vehicle 120, such as vehicle speed or notice.
- the image that is generated by the image generator 43 may be displayed at a position in consideration of the view of the driver 130 that is detected by the driver's line-of-sight detector 42. Namely, when the image that is generated by the image generator 43 is displayed at the center in the direction of the line of sight of the driver 130, it is possible that the image may prevent safe driving by the driver 130. Thus, the image that is generated by the image generator 43 may preferably be displayed at a position that is within the view of the driver 130 and that is separated from the center of the direction of the line of sight of the driver 130. In this manner, safe driving by the driver 130 can be ensured.
- the image generator 43 may generate an image that is captured by the passenger image capture unit 20 as it is, and the image generator 43 may transmit the generated image to the image display unit 50.
- the text or the like may not be displayed, and only the image of the passenger 140 can be displayed in the view of the driver 130 by the image display unit 50.
- the image generator 43 may generate an image that only includes the text or the like, and the image generator 43 may transmit the generated image to the image display unit 50.
- an image of the passenger 140 may not be displayed, and the image that only includes the text or the like may be displayed in the view of the driver 130 by the image display unit 50.
- an image that is generated by the image generator 43 and that indicates a condition of the passenger 140 may be an image that only includes an image of the passenger 140; an image that only includes text or the like that indicates the condition of the passenger 140; or an image that is obtained by adding the text or or the like that indicates the condition of the passenger 140 to the image of the passenger 140.
- An image of the driver 130 that is captured by the driver image capture unit 10 may be displayed by the passenger display 60.
- the passenger 140 can view the face and body of the driver 130 through the image that is displayed by the passenger display 60.
- nervousness of the passenger 140 that is caused by being captured alone can be removed, and the passenger 140 can obtain a sense of security by presence of the driver 130.
- the driver 130 can view the image of the passenger 140 that is superimposed with and displayed in the view of the driver 130, and the passenger 140 can view the image of the driver 130 that is displayed by the passenger display 60.
- the driver 130 and the passenger 140 can comfortably communicate with each other while viewing each other's face.
- FIG. 6 is a diagram showing an example of an image that is displayed by the display device 1 according to the embodiment.
- an image 202 of the passenger 140 is displayed in the view of the driver 130.
- text or the like that indicates a condition of the passenger 140 may be displayed.
- the area in which the image is displayed is arranged at a lower part of the front windshield so that the image does not prevent driving by the driver 130.
- the area in which the image is displayed may be suitably changed based on the line of sight of the driver 130.
- the image display device 1 can display information about the passenger 140 (e.g., an image of the actual face and body of the passenger 140, or text or the like that indicates a condition of the passenger 140) in the front passenger seat or in the rear seat while superimposing the information with a view of the driver 130.
- the driver 130 can obtain information about the passenger 140 almost without moving a visual field of the driver 130, namely, almost without lowering the safety. That is, it is possible to achieve both the safe driving of the driver 130, and the smoothness of the communication with the passenger 140.
- the image generator 43 may generate an image, such as the images that are described below.
- the image generator 43 may generate an image that is obtained by adding text, "danger,” to the image of the passenger 140.
- the image generator 43 may generate an image that is obtained by adding text, "bad physical condition,” to the image of the passenger 140.
- the image generator 43 may generate an image that includes text only.
- the above-described detection of the condition of the passenger 140 can be executed by image recognition with a camera, such as the passenger image capture unit 20.
- a camera such as the passenger image capture unit 20.
- the above-described detection of the condition of the passenger 140 can be achieved by various methods, such as a method of detecting an amount of perspiration of a person sitting in the passenger seat by a sweat sensor, or a method of detecting pulse of a person sitting in the passenger seat by a pulse sensor (cf. Japanese Unexamined Patent Publication No. 2014-92965 , for example).
- the vehicle 120 is not limited to a private vehicle.
- the vehicle 120 may be a taxi.
- the vehicle 120 may be a taxi.
- a condition of a passenger i.e., the passenger 140
- the condition of the passenger can be conveyed to a taxi driver (i.e., the driver 130).
- the display device 1 may operate, only if the driver 130 instructs the display device 1 to operate. For example, when the driver 130 presses a switch that is provided in the vicinity of a steering wheel, the display device 1 my be activated; the passenger image capture unit 20 may capture an image of the passenger; the image generator 43 may generate a predetermined image; and the predetermined image can be superimposed with and displayed in the view of the driver 130. In other words, only when the driver 130 needs to know a condition of the passenger 140, the display device 1 can be operated.
- an example of the image display unit 50 is shown in which three lasers are used.
- the embodiment is not limited to this.
- a configuration can be considered such that a single laser is used and a monochrome image is formed.
- the synthetic optical element may not be required.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Mechanical Engineering (AREA)
- Combustion & Propulsion (AREA)
- Transportation (AREA)
- Chemical & Material Sciences (AREA)
- Optics & Photonics (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Human Computer Interaction (AREA)
- Computer Graphics (AREA)
- Signal Processing (AREA)
- Traffic Control Systems (AREA)
- Controls And Circuits For Display Device (AREA)
- User Interface Of Digital Computer (AREA)
- Fittings On The Vehicle Exterior For Carrying Loads, And Devices For Holding Or Mounting Articles (AREA)
- Instrument Panels (AREA)
Claims (12)
- Dispositif d'affichage (1) comprenant :une unité de capture d'image de passager (20) configurée pour capturer des informations d'un passager qui voyage dans un véhicule (120) avec un conducteur du véhicule (120) ;un générateur d'image (43) configuré pour générer une image qui montre un état du passager sur la base des informations qui sont capturées par l'unité de capture d'image de passager (20) ;un afficheur frontal (50) configuré pour superposer l'image avec une partie inférieure d'un pare-brise, et configuré pour afficher l'image dans la partie inférieure du pare-brise ;une unité de capture d'image de conducteur configurée pour capturer des informations du conducteur ; etun détecteur de ligne visuelle (42) du conducteur configuré pour détecter une direction de la ligne visuelle du conducteur sur la base des informations qui sont capturées par l'unité de capture d'image de conducteur ;dans lequel le générateur d'image (43) est configuré pour générer l'image sur la base des informations capturées par l'unité de capture d'image de passager (20) ; etdans lequel, en réponse à la détection du fait que la direction de la ligne visuelle du conducteur est stable sur la base du détecteur de ligne visuelle du conducteur, le générateur d'image est configuré pour amener l'afficheur frontal (50) à afficher l'image à une position qui est séparée d'un centre de la direction de la ligne visuelle du conducteur et à la vue du conducteur.
- Dispositif d'affichage (1) selon la revendication 1,
comprenant en outre un détecteur d'état de passager (41) configuré pour détecter l'état du passager sur la base des informations qui sont capturées par l'unité de capture d'image de passager (20). - Dispositif d'affichage (1) selon la revendication 1 ou la revendication 2,
dans lequel l'image comporte du texte qui est généré sur la base des informations qui sont obtenues auprès du détecteur d'état de passager (41). - Dispositif d'affichage (1) selon l'une quelconque des revendications précédentes,
dans lequel l'unité de capture d'image de passager (20) a un angle de vue qui est plus large que celui de l'unité de capture d'image de conducteur (10). - Dispositif d'affichage (1) selon la revendication 4,
dans lequel le passager est dans un siège arrière du véhicule (120), et
dans lequel l'unité de capture d'image de passager (20) est configurée pour capturer une image d'une intégralité du siège arrière qui comporte le passager. - Dispositif d'affichage (1) selon l'une quelconque des revendications précédentes, comprenant en outre :un afficheur passager (60),dans lequel le dispositif d'affichage (1) est configuré pour afficher une image qui est capturée par l'unité de capture d'image de conducteur (10) sur l'afficheur passager (60).
- Dispositif d'affichage (1) selon l'une quelconque des revendications précédentes,
dans lequel l'afficheur frontal (50) est configuré pour ne pas afficher l'image qui est superposée avec le champ visuel du conducteur dans une condition prédéterminée. - Dispositif d'affichage (1) selon la revendication 7,
dans lequel, en réponse à la détection du fait que la direction de la ligne visuelle du conducteur n'est pas stable sur la base du détecteur de ligne visuelle du conducteur, l'afficheur frontal (50) est configuré pour ne pas afficher l'image qui est superposée avec le champ visuel du conducteur. - Dispositif d'affichage (1) selon la revendication 8, comprenant en outre :une unité de mesure de distance configurée pour mesurer une distance entre le véhicule (120) et un autre véhicule devant le véhicule (120),dans lequel, en réponse à la détermination du fait que la distance à partir d'un résultat de mesure de distance par l'unité de mesure de distance est inférieure à une valeur seuil, l'afficheur frontal (50) est configuré pour éviter d'afficher l'image qui est superposée avec le champ visuel du conducteur.
- Dispositif d'affichage (1) selon l'une quelconque des revendications 1 à 9, dans lequel l'afficheur frontal (50) comporte une source de lumière laser, et un déflecteur optique (52) configuré pour dévier un faisceau de lumière laser qui est émis depuis la source de lumière laser.
- Dispositif d'affichage selon la revendication 2, ou toute revendication dépendante de la revendication 2,
dans lequel le générateur d'image (43) est configuré pour générer l'image avec davantage de dépendance aux informations capturées par le détecteur d'état de passager (41) ; et
le détecteur d'état de passager (41) est configuré pour recevoir, depuis l'unité de capture d'image de passager (20), des informations obtenues par l'unité de capture d'image de passager (20), et pour détecter l'état du passager en détectant un degré de fatigue du passager en comparant un temps de clignement du passager à un temps de clignement normal sur la base des informations capturées par l'unité de capture d'image de passager (20), dans lequel le temps de clignement du passager est déterminé par reconnaissance du visage du passager et extraction d'images correspondant aux régions des yeux. - Véhicule (120) comprenant :
le dispositif d'affichage (1) selon l'une quelconque des revendications 1 à 11.
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2014102093 | 2014-05-16 | ||
JP2015061053A JP6497158B2 (ja) | 2014-05-16 | 2015-03-24 | 表示装置、移動体 |
PCT/JP2015/063539 WO2015174383A1 (fr) | 2014-05-16 | 2015-04-30 | Dispositif d'affichage et vehicule |
Publications (3)
Publication Number | Publication Date |
---|---|
EP3142902A1 EP3142902A1 (fr) | 2017-03-22 |
EP3142902A4 EP3142902A4 (fr) | 2017-09-13 |
EP3142902B1 true EP3142902B1 (fr) | 2021-02-10 |
Family
ID=54479922
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP15793397.9A Active EP3142902B1 (fr) | 2014-05-16 | 2015-04-30 | Dispositif d'affichage et vehicule |
Country Status (7)
Country | Link |
---|---|
US (1) | US20170155867A1 (fr) |
EP (1) | EP3142902B1 (fr) |
JP (1) | JP6497158B2 (fr) |
CN (1) | CN106458029A (fr) |
AU (1) | AU2015260356B2 (fr) |
CA (1) | CA2944844A1 (fr) |
WO (1) | WO2015174383A1 (fr) |
Families Citing this family (33)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
DE102015218306A1 (de) * | 2015-09-23 | 2017-03-23 | Robert Bosch Gmbh | Verfahren und Vorrichtung zum Ermitteln eines Schläfrigkeitszustands eines Fahrers |
DE102016120425A1 (de) * | 2015-10-26 | 2017-04-27 | Active Knowledge Ltd. | Zusammenfügen einer ein Fahrzeugfenster imitierenden Videodurchsicht mit einem anderen Video |
US10621778B2 (en) | 2015-10-30 | 2020-04-14 | Faraday & Future Inc. | Methods and systems for generating dynamic user interface effects |
JP2017178025A (ja) * | 2016-03-30 | 2017-10-05 | 矢崎総業株式会社 | 運転支援装置 |
US20170293140A1 (en) * | 2016-04-12 | 2017-10-12 | Ostendo Technologies, Inc. | Split Exit Pupil Heads-Up Display Systems and Methods |
WO2017190092A1 (fr) * | 2016-04-29 | 2017-11-02 | Faraday&Future Inc. | Système pour fournir un équilibre de couleur dans un dispositif d'affichage d'automobile |
CN109153353B (zh) | 2016-05-24 | 2022-04-08 | 金泰克斯公司 | 具有选择性图像数据显示的车辆显示器 |
JP2018105737A (ja) * | 2016-12-27 | 2018-07-05 | トヨタ紡織株式会社 | アイポイント測定装置、及びアイポイント測定方法 |
JP6972549B2 (ja) | 2016-12-28 | 2021-11-24 | 株式会社リコー | Hud装置、車両装置及び表示方法 |
JP6922217B2 (ja) | 2016-12-28 | 2021-08-18 | 株式会社リコー | 表示装置、物体装置及び表示方法 |
JP6627787B2 (ja) | 2017-01-17 | 2020-01-08 | トヨタ自動車株式会社 | ドライバ状態検出装置 |
JP6930120B2 (ja) | 2017-02-02 | 2021-09-01 | 株式会社リコー | 表示装置、移動体装置及び表示方法。 |
JP6809262B2 (ja) | 2017-02-08 | 2021-01-06 | トヨタ自動車株式会社 | ドライバ状態検出装置 |
KR20190119093A (ko) * | 2017-03-03 | 2019-10-21 | 오스텐도 테크놀로지스 인코포레이티드 | 스플릿 사출 동공 헤드업 디스플레이 시스템 및 방법 |
DE102017204256A1 (de) | 2017-03-14 | 2018-09-20 | Bayerische Motoren Werke Aktiengesellschaft | Verfahren und Vorrichtung zur Erinnerung eines Fahrers an ein Anfahren an einer Lichtsignaleinrichtung mit variabler Ausgabefunktion |
WO2018230688A1 (fr) * | 2017-06-16 | 2018-12-20 | 本田技研工業株式会社 | Système, procédé et programme de fourniture d'expérience |
KR102397089B1 (ko) * | 2017-07-28 | 2022-05-12 | 삼성전자주식회사 | 이미지 처리 방법 및 이미지 처리 장치 |
US11265460B2 (en) * | 2017-09-20 | 2022-03-01 | Sharp Kabushiki Kaisha | Electronic device, control device, and control method |
CN111095363B (zh) * | 2017-09-22 | 2024-02-09 | 麦克赛尔株式会社 | 显示系统和显示方法 |
EP3470908B1 (fr) * | 2017-10-16 | 2021-03-24 | Volvo Car Corporation | Véhicule avec indication d'état en position supérieure |
KR102302917B1 (ko) * | 2017-12-18 | 2021-09-15 | 현대모비스 주식회사 | 안전운전 지원 장치 및 방법 |
US11430230B2 (en) | 2017-12-27 | 2022-08-30 | Pioneer Corporation | Storage device and excitement suppression device |
JP7114906B2 (ja) * | 2018-01-17 | 2022-08-09 | トヨタ自動車株式会社 | 車両用表示装置 |
JP7005112B2 (ja) * | 2018-03-05 | 2022-01-21 | 矢崎総業株式会社 | 表示システム、及び、車載システム |
CN108320496A (zh) * | 2018-03-20 | 2018-07-24 | 段宏伟 | 一种出租车精准载客与乘客快速搭车系统 |
JP2019174461A (ja) * | 2018-03-28 | 2019-10-10 | 株式会社リコー | 画像制御装置、表示装置、移動体、画像データ生成方法、及びプログラム |
JP7037764B2 (ja) | 2018-03-29 | 2022-03-17 | 株式会社リコー | 移動ルート案内装置、移動体、移動ルート案内方法及びプログラム |
JP7211673B2 (ja) * | 2018-05-25 | 2023-01-24 | 株式会社Subaru | 車両の乗員監視装置 |
KR102546441B1 (ko) * | 2018-11-07 | 2023-06-26 | 현대모비스 주식회사 | 안전운전 지원 장치 및 방법 |
US10969595B2 (en) | 2018-11-30 | 2021-04-06 | International Business Machines Corporation | In-vehicle content display apparatus |
JP7241005B2 (ja) * | 2019-11-27 | 2023-03-16 | 京セラ株式会社 | ヘッドアップディスプレイシステムおよび移動体 |
DE102020133448A1 (de) * | 2020-12-15 | 2022-06-15 | Dr. Ing. H.C. F. Porsche Aktiengesellschaft | Fahrzeug |
US20240054733A1 (en) * | 2022-08-12 | 2024-02-15 | State Farm Mutual Automobile Insurance Company | Systems and methods for enhanced outdoor displays via augmented reality |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
GB2284582A (en) * | 1993-11-22 | 1995-06-14 | Toad Innovations Ltd | Vehicle safety device to warn driver of fatigue |
US20120139816A1 (en) * | 2010-12-05 | 2012-06-07 | Ford Global Technologies, Llc | In-vehicle display management system |
Family Cites Families (23)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7887089B2 (en) * | 1992-05-05 | 2011-02-15 | Automotive Technologies International, Inc. | Vehicular occupant protection system control arrangement and method using multiple sensor systems |
JPH09301054A (ja) * | 1996-05-20 | 1997-11-25 | Honda Motor Co Ltd | 車両用前照灯装置 |
AU2003225228A1 (en) * | 2002-05-03 | 2003-11-17 | Donnelly Corporation | Object detection system for vehicle |
US20040056954A1 (en) * | 2002-09-20 | 2004-03-25 | Crandall John Christopher | System and method for monitoring within a vehicle |
JP4487633B2 (ja) * | 2004-05-24 | 2010-06-23 | 日産自動車株式会社 | 車両内コミュニケーション装置 |
JP4650349B2 (ja) * | 2005-10-31 | 2011-03-16 | 株式会社デンソー | 車両用表示システム |
WO2008052178A2 (fr) * | 2006-10-27 | 2008-05-02 | Johnson Controls Technology Company | Système de vision pour véhicule |
WO2008143079A1 (fr) * | 2007-05-22 | 2008-11-27 | Autonetworks Technologies, Ltd. | Module de commutation vidéo de voiture et dispositif de prise de vue d'image de voiture |
JP5061373B2 (ja) * | 2008-11-06 | 2012-10-31 | Necフィールディング株式会社 | 車両内犯罪防止システム、車両内犯罪防止方法および車両内犯罪防止プログラム |
KR20100063394A (ko) * | 2008-12-03 | 2010-06-11 | 한국전자통신연구원 | 차량에 탑승한 유아의 안전을 위한 제어 시스템 및 방법 |
US20100201507A1 (en) * | 2009-02-12 | 2010-08-12 | Ford Global Technologies, Llc | Dual-mode vision system for vehicle safety |
US8977489B2 (en) * | 2009-05-18 | 2015-03-10 | GM Global Technology Operations LLC | Turn by turn graphical navigation on full windshield head-up display |
JP5287746B2 (ja) * | 2009-05-21 | 2013-09-11 | 日産自動車株式会社 | 運転支援装置、及び運転支援方法 |
US20110032266A1 (en) * | 2009-08-07 | 2011-02-10 | Delphi Technologies, Inc. | Glare detection and mitigation method for a photo-sensitive display device |
JP2011116219A (ja) * | 2009-12-02 | 2011-06-16 | Koito Mfg Co Ltd | 車内監視システム |
JP5555526B2 (ja) * | 2010-04-06 | 2014-07-23 | 東芝アルパイン・オートモティブテクノロジー株式会社 | 車両用ディスプレイ装置 |
US20120224060A1 (en) * | 2011-02-10 | 2012-09-06 | Integrated Night Vision Systems Inc. | Reducing Driver Distraction Using a Heads-Up Display |
CN103314342A (zh) * | 2011-03-30 | 2013-09-18 | 本田技研工业株式会社 | 操作装置 |
JP2013061554A (ja) * | 2011-09-14 | 2013-04-04 | Ricoh Co Ltd | 画像形成装置、画像形成装置を搭載した車両 |
JP2013130832A (ja) * | 2011-12-22 | 2013-07-04 | Nippon Seiki Co Ltd | 表示装置 |
US9602779B2 (en) * | 2012-07-10 | 2017-03-21 | Christopher DeJuliis | System for passenger monitoring |
US9565402B2 (en) * | 2012-10-30 | 2017-02-07 | Baby-Tech Innovations, Inc. | Video camera device and method to monitor a child in a vehicle |
JP2014092965A (ja) * | 2012-11-05 | 2014-05-19 | Denso Corp | 乗員監視装置 |
-
2015
- 2015-03-24 JP JP2015061053A patent/JP6497158B2/ja active Active
- 2015-04-30 EP EP15793397.9A patent/EP3142902B1/fr active Active
- 2015-04-30 CN CN201580024269.6A patent/CN106458029A/zh active Pending
- 2015-04-30 US US15/129,871 patent/US20170155867A1/en not_active Abandoned
- 2015-04-30 WO PCT/JP2015/063539 patent/WO2015174383A1/fr active Application Filing
- 2015-04-30 AU AU2015260356A patent/AU2015260356B2/en active Active
- 2015-04-30 CA CA2944844A patent/CA2944844A1/fr not_active Abandoned
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
GB2284582A (en) * | 1993-11-22 | 1995-06-14 | Toad Innovations Ltd | Vehicle safety device to warn driver of fatigue |
US20120139816A1 (en) * | 2010-12-05 | 2012-06-07 | Ford Global Technologies, Llc | In-vehicle display management system |
Also Published As
Publication number | Publication date |
---|---|
WO2015174383A1 (fr) | 2015-11-19 |
JP2015231828A (ja) | 2015-12-24 |
CN106458029A (zh) | 2017-02-22 |
JP6497158B2 (ja) | 2019-04-10 |
US20170155867A1 (en) | 2017-06-01 |
EP3142902A1 (fr) | 2017-03-22 |
EP3142902A4 (fr) | 2017-09-13 |
AU2015260356A1 (en) | 2016-10-27 |
CA2944844A1 (fr) | 2015-11-19 |
AU2015260356B2 (en) | 2018-05-10 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP3142902B1 (fr) | Dispositif d'affichage et vehicule | |
US10800424B2 (en) | Driver condition detection system | |
CN108621923B (zh) | 车辆的显示系统及车辆的显示系统的控制方法 | |
JP6454368B2 (ja) | 車両の表示システム及び車両の表示システムの制御方法 | |
JP4926437B2 (ja) | 車両の運転支援装置 | |
WO2017056401A1 (fr) | Dispositif de commande, procédé de commande et programme | |
JP6047318B2 (ja) | ドライバ状態検出装置及びドライバ状態報知装置 | |
US10629106B2 (en) | Projection display device, projection display method, and projection display program | |
JP5600256B2 (ja) | 情報表示装置 | |
JP4825868B2 (ja) | 車両用警報装置 | |
JP6512475B2 (ja) | 情報提供装置、情報提供方法及び情報提供用制御プログラム | |
CN108621922B (zh) | 车辆的显示系统及车辆的显示系统的控制方法 | |
JP2006209750A (ja) | ユーザの目を検出し、ユーザの位置及び瞬き状態を決定する方法 | |
JP2014010418A (ja) | 立体表示装置及び立体表示方法 | |
US11176749B2 (en) | In-vehicle display device three-dimensional image generation | |
US11708079B2 (en) | Method, device, and system for influencing at least one driver assistance system of a motor vehicle | |
JP2010143250A (ja) | 後方視認装置 | |
JP2010039793A (ja) | 車両用表示装置 | |
JP2018116495A (ja) | ドライバ状態検出装置 | |
WO2021111765A1 (fr) | Système de conduite de véhicule | |
JP6822325B2 (ja) | 操縦支援装置、操縦支援方法、プログラム | |
US20210300183A1 (en) | In-vehicle display apparatus, method for controlling in-vehicle display apparatus, and computer program | |
US20220319200A1 (en) | Device and method for determining image data of the eyes, eye positions and/or a viewing direction of a vehicle user in a vehicle | |
JP2008162550A (ja) | 外部環境表示装置 | |
KR20200071968A (ko) | 차량 및 그 제어방법 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20161004 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G02B 27/01 20060101ALI20170502BHEP Ipc: G06F 3/01 20060101ALI20170502BHEP Ipc: B60R 16/02 20060101AFI20170502BHEP Ipc: G06F 3/048 20130101ALI20170502BHEP Ipc: B60K 35/00 20060101ALI20170502BHEP |
|
DAV | Request for validation of the european patent (deleted) | ||
DAX | Request for extension of the european patent (deleted) | ||
A4 | Supplementary search report drawn up and despatched |
Effective date: 20170811 |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G06F 3/01 20060101ALI20170807BHEP Ipc: G02B 27/01 20060101ALI20170807BHEP Ipc: B60K 35/00 20060101ALI20170807BHEP Ipc: G06F 3/048 20130101ALI20170807BHEP Ipc: B60R 16/02 20060101AFI20170807BHEP |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
17Q | First examination report despatched |
Effective date: 20181029 |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
INTG | Intention to grant announced |
Effective date: 20201007 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
RIN1 | Information on inventor provided before grant (corrected) |
Inventor name: ISHIGAKI, TOMOKO Inventor name: SAKOH, FUMIKO Inventor name: SAISHO, KENICHIROH Inventor name: YOSHIKAWA, HIROMI Inventor name: YOKOTA, SOICHIRO |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: REF Ref document number: 1360038 Country of ref document: AT Kind code of ref document: T Effective date: 20210215 Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602015065476 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG9D |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20210210 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210611 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210510 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210210 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210510 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210210 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210511 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210210 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 1360038 Country of ref document: AT Kind code of ref document: T Effective date: 20210210 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210210 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210210 Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210210 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210210 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210210 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210610 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210210 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210210 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210210 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210210 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602015065476 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210210 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210210 Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210210 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210210 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210210 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20210430 |
|
26N | No opposition filed |
Effective date: 20211111 |
|
REG | Reference to a national code |
Ref country code: BE Ref legal event code: MM Effective date: 20210430 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210210 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20210430 Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20210430 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210210 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210210 Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20210430 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210610 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20210430 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20150430 |
|
P01 | Opt-out of the competence of the unified patent court (upc) registered |
Effective date: 20230522 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210210 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20230424 Year of fee payment: 9 Ref country code: DE Payment date: 20230420 Year of fee payment: 9 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: GB Payment date: 20230419 Year of fee payment: 9 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210210 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210210 |