GB2563647A - Method of displaying an image - Google Patents

Method of displaying an image Download PDF

Info

Publication number
GB2563647A
GB2563647A GB1709972.2A GB201709972A GB2563647A GB 2563647 A GB2563647 A GB 2563647A GB 201709972 A GB201709972 A GB 201709972A GB 2563647 A GB2563647 A GB 2563647A
Authority
GB
United Kingdom
Prior art keywords
image data
vehicle
image
user
imaging means
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
GB1709972.2A
Other versions
GB201709972D0 (en
GB2563647B (en
Inventor
Richard Hancox Michael
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Jaguar Land Rover Ltd
Original Assignee
Jaguar Land Rover Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Jaguar Land Rover Ltd filed Critical Jaguar Land Rover Ltd
Priority to GB1709972.2A priority Critical patent/GB2563647B/en
Publication of GB201709972D0 publication Critical patent/GB201709972D0/en
Publication of GB2563647A publication Critical patent/GB2563647A/en
Application granted granted Critical
Publication of GB2563647B publication Critical patent/GB2563647B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R1/00—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
    • B60R1/20—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
    • B60R1/22—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle
    • B60R1/23—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with a predetermined field of view
    • B60R1/26—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with a predetermined field of view to the rear of the vehicle
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012—Head tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/60—Editing figures and text; Combining figures or text
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
    • G06T5/77—
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/30—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing
    • B60R2300/303—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing using joined images, e.g. multiple camera images
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/30—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing
    • B60R2300/304—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing using merged images, e.g. merging camera image with stored images
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/80—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement
    • B60R2300/8046—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement for replacing a rear-view mirror system
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/80—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement
    • B60R2300/8066—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement for monitoring rearward traffic
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30252—Vehicle exterior; Vicinity of vehicle

Abstract

A method of displaying an image on a vehicles rearward view display is disclosed which involves capturing first image data with a first frame of reference from a first imaging means located at a first location of the vehicle proximal to the display device, and capturing second image data with a second frame of reference from a second imaging means located at a second location of the vehicle. A transformation is then applied to the second image data to form third image data having the first frame of reference. A composite image is then formed from the first third image data which image appears to a user to have been captured at the first location. The method may also include determining a restricted portion of the first image data, e.g. obstructions such as headrests, passengers, and luggage, and may further involve overlaying a portion of the third image data corresponding to the restricted portion to provide an unrestricted composite image. The imaging means may be cameras. The method may also include eye or head tracking to determine a drivers gaze, and then forming the composite image based upon this gaze. Also disclosed is a corresponding apparatus, controller, and vehicle.

Description

METHOD OF DISPLAYING AN IMAGE
TECHNICAL FIELD
The present disclosure relates to a method of displaying an image on a display device in a vehicle. In particular, but not exclusively, the invention relates to method of displaying an image on a rearward view display system. Aspects of the invention relate to a method of displaying an image, a rearward view display system, a vehicle controller for a display system, a vehicle and a computer program product.
BACKGROUND A traditional rear view mirror system for a vehicle provides the driver with a perspective that is both natural and intuitive to use whilst driving. In its simplest form, the mirror system consists of a mirror provided on a mount which is suspended from the vehicle roofline and into which the driver of the vehicle can look to appreciate a view of the vehicle cabin to the rear, and beyond. The position of such mirrors is typically adjustable by hand. A disadvantage of traditional set-ups is that the view out of the back of the vehicle may be blocked by a variety of obstructions, including those within the vehicle such as rear occupants and luggage, and those provided by the vehicle itself including rear seats, the tailgate, a low roof and a high vehicle waistline. This obscured field of view can result in missed or misidentified visual information, making it difficult for the driver to fully assess and react to their surroundings. A digital rear view system improves on the above scenario by replacing or supplementing the traditional rear view mirror with an electronic screen fed via a camera positioned at the rear of the vehicle. This allows the driver to see a clear view out of the back of the vehicle, eliminating blind spots caused by obstructions. A disadvantage of current digital rear view systems is that the image displayed on the screen is from the perspective of the camera position rather than the mirror position. This can result in the displayed image appearing unnatural and confusing to a driver who is used to viewing a traditional rear view mirror, because the size and position of objects in the displayed image may be different to what they are expecting to see.
Furthermore, the image displayed on a conventional digital rear view system does not move in relation to the driver’s eye position in the same manner as it would in a traditional mirror, which can lead to the displayed image being unintuitive
At least in certain embodiments, the present invention seeks to provide an improved rearward view display system which overcomes or alleviates the aforementioned problem.
SUMMARY OF THE INVENTION
Aspects and embodiments of the invention provide a method, a system, a controller, a vehicle and a computer program product as claimed in the appended claims.
According to an aspect of the invention there is provided a method of displaying an image on a display device of a rearward view display system of a vehicle. The method comprises capturing first image data from a first imaging means, the first image data having a first frame of reference; capturing second image data from a second imaging means, the second imaging means having a second frame of reference; applying a transformation to the second image data to form third image data having the first frame of reference; forming a composite image from the first image data and the third image data.
According to an aspect of the invention there is provided a method of displaying an image on a display device of a rearward view display system of a vehicle. The method comprises capturing first image data from a first imaging means located at a first location of the vehicle proximal to the display device, the first image data having a first frame of reference; capturing second image data from a second imaging means located at a second location of the vehicle, the second imaging means having a second frame of reference; applying a transformation to the second image data to form third image data having the first frame of reference; forming a composite image from the first image data and the third image data, such that the composite image appears to a user to have been captured at the first location.
The rearward view display system advantageously provides a user of the vehicle, e.g. a driver of the vehicle, with a substantially unobstructed view out of the rear of the vehicle that is natural and intuitive to respond to. The first imaging means, by virtue of its location at or near the display device of the rearward view display system, enables the composite image to resemble a view produced by a traditional mirror in terms of, for example, apparent distance to objects shown in the display device.
Reference to the first imaging means being proximal to the display device is intended to mean that the first imaging means is intended to be located at or close to the location of the display device. In practice, the display device may be located in the vehicle in the position of the traditional rear view mirror, mounted at the roofline of the vehicle just in front of the driver’s eyeline. The first imaging means may be mounted at the same, or very close to the same, traditional mirror position, together with the display device.
The first imaging means may be a rearward facing camera. The second imaging means may be a rearward facing camera located proximal to the rear of the vehicle.
As above, the use of the term proximal here is intended to indicate that the second imaging means is located at or close to the rear of the vehicle. For example, the second imaging means may be located above or below the rear closure element or may be located on the roof of the vehicle towards the rear of the vehicle.
The method may comprise determining a gaze of the user. Determining the gaze of the user may comprise determining at least one of a head position and an eye direction of the user relative to the display device. Determining at least one of a head position and an eye direction of the user may include tracking the orientation of at least one of the user’s eyes to determine the eye position of the user. The composite image may be formed in dependence on the gaze of the user.
Determining the gaze of the user allows the rearward view display system to adjust the composite image in dependence on the head position and/or eye direction of the user. This allows the display device to behave similarly to a traditional mirror, with the displayed composite image changing according to the direction and position from which the display device is viewed. Thus, the display device displays an image that appears natural and intuitive to a user who is accustomed to viewing a traditional rear view mirror.
The method may comprise determining a restricted portion of the first image data. Forming the composite image may comprise overlaying an unrestricted portion of the third image data corresponding to the restricted portion of the first image data onto the first image data to provide an unrestricted composite image.
Combining a restricted portion of the first image data with an unrestricted portion of the third image data in this way enables a composite image to be formed which provides an unobstructed view out of the rear of the vehicle that is clear from obstacles within the vehicle (e.g. rear occupants and luggage) and parts of the vehicle itself (e.g. rear seats, tailgate, roof, vehicle waistline).
In one embodiment, the method comprises determining a user requirement to view an interior of the vehicle in the display device in dependence on at least one of the gaze of the user, the head position of the user and the eye direction of the user, and displaying on the display device an image formed exclusively from the first image data in dependence on said determination.
The determination of the user requirement to view an interior of the vehicle in the display device may be made in dependence on the head position of the user and the eye direction of the user.
The method may also comprise determining a user requirement to revert from viewing an interior of the vehicle in the display to viewing an image out of the rear of the vehicle in the display device. This determination may be made in dependence on at least one of the gaze of the user, the head position of the user and the eye direction of the user. The method may comprise reverting to displaying the composite image on the display device of the rearward view display system in dependence on said determination of the user reversionary requirement.
According to another aspect of the invention there is provided a method of displaying an image on a display device of a rearward view display system of a vehicle, the method comprising: capturing first image data from a first camera located at a first location of the vehicle proximal to the display device, the first image data having a first frame of reference; capturing second image data from a second camera located at a second location of the vehicle, the second image data having a second frame of reference; applying a transformation to the second image data to form third image data having the first frame of reference; forming a composite image from the first image data and the third image data, such that the composite image appears to a user to have been captured at the first location.
According to another aspect of the invention there is provided a rearward view display system for a vehicle. The rearward view display system comprises a rearward view display device comprising a visual display for displaying images formed from image data. The rearward view display system comprises a first imaging means, the first imaging means being configured to provide first image data. The rearward view display system comprises a second imaging means, configured to provide second image data. The rearward view display system comprises an image processor configured to apply a transformation to the second image data to form third image data, such that when the third image data is combined with the first image data, both the first image data and the third image data appear to have been captured from the position of the first imaging means, when displayed on the visual display. The image processor is configured to form a composite image comprising at least a part of the first image data and at least a part of the third image data.
According to another aspect of the invention there is provided a rearward view display system for a vehicle. The rearward view display system comprises a rearward view display device comprising a visual display for displaying images formed from image data. The rearward view display system comprises a first imaging means located at a first location of the vehicle proximal to the visual display, the first imaging means being configured to provide first image data. The rearward view display system comprises a second imaging means, configured to provide second image data. The rearward view display system comprises an image processor configured to apply a transformation to the second image data to form third image data, such that when the third image data is combined with the first image data, both the first image data and the third image data appear to have been captured from the position of the first imaging means, when displayed on the visual display. The image processor is configured to form a composite image comprising at least a part of the first image data and at least a part of the third image data, such that the composite image appears to the user to have been captured at the location of the first imaging means.
Thus, the rearward view display system advantageously provides a user of the vehicle, e.g. a driver of the vehicle, with a substantially unobstructed view out of the rear of the vehicle that is natural and intuitive to respond to. The first imaging means, by virtue of its location at or near the display device of the rearward view display system, enables the composite image to resemble a view produced by a traditional mirror in terms of, for example, apparent distance to objects shown in the display device.
The first imaging means may be a rearward facing camera. The second imaging means may be a rearward facing camera located proximal to the rear of the vehicle. As above, the use of the term proximal here is intended to indicate that the rearward facing camera is located at or near to the rear of the vehicle. For example, the rearward facing camera could be provided at the rear of the vehicle, e.g. located on a rear closure element of the vehicle, or could be provided near to the rear of the vehicle, e.g. on the roof of the vehicle towards the rear of the vehicle (but not necessarily at the rear-most position on the roof of the vehicle).
The rearward view display system may comprise third imaging means configured to determine a gaze of the user. The third imaging means may be configured to determine the gaze of the user by determining at least one of a head position and an eye direction of the user relative to the visual display. The third imaging means may be configured to track the orientation of at least one of the user’s eyes to determine the eye position of the user. The image processor may be configured to form the composite image in dependence on the gaze of the user.
Forming the composite image in dependence on the gaze of the user enables the display device to behave similarly to a traditional mirror, with the display device displaying a composite image that changes according to the direction and position from which the display device is viewed. This provides a natural and intuitive view for a driver who is accustomed to viewing a traditional rear view mirror.
The image processor may be configured to determine a restricted portion of the primary image data. The image processor may be configured to form the composite image by overlaying an unrestricted portion of the third image data corresponding to the restricted portion of the primary image data onto the primary image data to provide an unrestricted composite image.
In this way, the rearward view display system is configured to provide an unobstructed view out of the rear of the vehicle.
In some embodiments where the first imaging means is appropriately positioned and has an adequate field of view, the first imaging means may be configured to determine a gaze of the user. Using the first imaging means to determine a gaze of the user allows the rearward view display system to operate without the need for additional imaging means (e.g. third imaging means), thus reducing the cost and complexity of the rearward view display system.
According to a yet further aspect of the invention, there is provided a rearward view display system for a vehicle, the system comprising: a display device comprising a visual display for displaying images formed from image data; a first camera located at a first location of the vehicle proximal to the visual display, the first camera being configured to provide first image data; a second camera configured to provide second image data; an image processor configured to apply a transformation to the second image data to form third image data, such that when the third image data is combined with the first image data, both the first image data and the third image data appear to have been captured from the position of the first camera, when displayed on the visual display, wherein the image processor is configured to form a composite image comprising at least a part of the first image data and at least a part of the third image data, such that the composite image appears to the user to have been captured at the location of the first camera.
According to another aspect of the invention there is provided a controller for displaying an image on a display device of a rearward view display system of a vehicle. The controller comprises: a first input configured to receive captured first image data from a first imaging means located at a first location of a vehicle proximal to the display device, the first image data having a first frame of reference; a second input configured to receive captured second image data from a second imaging means located at a second location of the vehicle, the second imaging means having a second frame of reference; a processor; and an output. The processor is configured to apply a transformation to the second image data to form third image data having the first frame of reference, and form a composite image from the first image data and the third image data, such that the composite image appears to the user to have been captured at the first location. The output is configured to output the composite image to the display device of the rearward view display system.
In an embodiment of the abovementioned controller, the processor comprises an electronic processor having first and second electrical inputs for receiving said image data; an electronic memory device electrically coupled to the electronic processor and having instructions stored therein, wherein the processor is configured to access the memory device and execute the instructions stored therein such that it is operable to apply the transformation to the second image data to form third image data and to form the composite image from the first image data and the third image data; and an electrical output configured to output the composite image to the display device of the rearward view display system.
According to another aspect of the invention there is provided a vehicle comprising a rearward view display system as described in the above paragraphs.
The display device may be mounted to an internal roofline in a cabin of the vehicle. In this way, the display device may take the place of a traditional rear view mirror of a vehicle.
The vehicle may comprise a controller as described in the above paragraphs.
According to a further aspect of the present invention there is provided a computer program product comprising instructions which, when a program of the program product is executed by a computer, cause the computer to carry out the method of the above aspect of the present invention. The computer program product may be downloadable from a communication network and/or stored on a computer-readable and/or microprocessor-executable medium.
According to a further aspect of the present invention there is provided a non-transitory computer-readable medium having stored thereon the computer program product of a foregoing aspect of the invention.
Within the scope of this application it is expressly intended that the various aspects, embodiments, examples and alternatives set out in the preceding paragraphs, in the claims and/or in the following description and drawings, and in particular the individual features thereof, may be taken independently or in any combination. That is, all embodiments and/or features of any embodiment can be combined in any way and/or combination, unless such features are incompatible. The applicant reserves the right to change any originally filed claim or file any new claim accordingly, including the right to amend any originally filed claim to depend from and/or incorporate any feature of any other claim although not originally claimed in that manner.
BRIEF DESCRIPTION OF THE DRAWINGS
One or more embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings, in which:
Figure 1 is a schematic representation of a rearward view display system of the invention incorporated in a vehicle, wherein the rearward view display system includes a front camera, two rear cameras and a tracking camera;
Figure 2 is a schematic representation of a plan view of the rearward view display system of Figure 1, further showing a processing unit for processing image data from the front, rear and tracking cameras to form a composite image for display on a display screen of the rearward view display system;
Figure 3 is a schematic representation of a side elevation view of the rearward view display system of Figure 1;
Figure 4 is an image from a front camera of the rearward view display system, showing a perspective view corresponding to the portion of the front camera image displayed on the display screen;
Figure 5 is a set of procedural steps undertaken by the rearward view display system of Figure 1; and
Figure 6 is a set of images showing how images from the front and rear cameras of the rearward view display system of Figure 1 are combined to form a composite image of an unobstructed view out of the rear of the vehicle.
DETAILED DESCRIPTION
Referring to Figure 1, a rearward view display system 10 according to an embodiment of the invention will now be described. In use, the rearward view display system 10 is incorporated into a vehicle 12 and provides a user 14 of the vehicle 12, who in this case is a driver of the vehicle 12, with a substantially unobstructed view out of the rear of the vehicle 12 that is natural and intuitive to respond to. For example, the rearward view display system 10 is configured to provide a view out of the rear of the vehicle 12 that is clear from objects within the vehicle 12, e.g. rear occupants and luggage, and parts of the vehicle 12 itself, e.g. rear seats 16, the tailgate, a low roof and a high vehicle waistline.
Referring also to Figure 2, the rearward view display system 10 comprises a display device 18, a plurality of imaging means (for example, but not limited to cameras) 20 and a processing unit 21 comprising an image processor 22.
The display device 18 is located at the front of the vehicle cabin 24 in the position usually occupied by a traditional rear view mirror, and is configured to be used in a similar manner to a traditional rear view mirror. The display device 18 comprises a visual display 26 comprising an electronic display screen 28 for displaying an unobstructed image out of the rear of the vehicle 12, and a housing 30 for housing the display screen 28 and associated electronics (not shown). Typically the housing 30 is mounted to an interior roofline of the vehicle cabin, and may be movable manually or electronically in response to a user command to adjust the relative position of the housing relative to the user. Alternatively, the housing may be mounted to the interior of the windscreen. In this embodiment of the invention, the electronic display screen 28 comprises an LCD display screen which substantially covers the entire visual display 26. However, it will be appreciated that in other embodiments the display screen 28 may be formed of any suitable electronic display technology, e.g. an OLED display, and may form only a portion of the visual display 26. In embodiments where the electronic display screen 28 forms a portion of the visual display 26, the remainder of the visual display 26 may be formed of a traditional mirror or may be used to display additional information to the user 14. Furthermore, in addition to displaying an unobstructed view out of the rear of the vehicle 12, the display screen 28 may display other useful information to the user 14 such as a speedometer, fuel gauge, oil pressure gauge, engine temperature gauge, directions, notifications, video, chat messages etc. Alternatively or additionally, the visual display 26 may be switchable between an electronic display and a rearward view display on receipt by the system 10 of an instruction from the user 14. This enables the user 14 to switch between different views depending on their situation and/or personal preferences.
The plurality of imaging means 20 of the rearward view display system 10 include a first imaging means in the form of a front camera 32, a second imaging means in the form of two rear cameras 34 and a third imaging means in the form of a tracking camera 36. It should be noted that the terms “front” and “rear” are used throughout this specification in their conventional sense when defining relative features on the vehicle 12.
The front and rear cameras 32, 34 capture images for use in forming a composite image 38 (shown in Figure 6) for display on the display screen 28 of the display device 18, where the composite image 38 provides a substantially unobstructed view out of the rear of the vehicle 12. In this embodiment, the tracking camera 36 captures images for use in determining the gaze of the user 14, as will be explained later. However, it should be noted that images captured by the front camera 32 may alternatively or additionally be used in determining the gaze of the user in other embodiments.
The front camera 32 is a rear-facing digital camera for generating video images of the rear of the interior of the vehicle cabin 24 and out of the rear of the vehicle 12. In this embodiment, the front camera 32 is integrated into the housing 30 of the display device 18, and is centrally positioned above the electronic display screen 28. In other embodiments, the front camera 32 may be incorporated at different positions within the housing 30, may be mounted on the exterior of the housing 30, or may simply be positioned in the vicinity of the display device 18. Providing the front camera 32 in the vicinity of the display device 18, and thus ensuring that the video images captured by the front camera 32 are generated substantially from the location of the display device 18, enables the composite image 38 that is produced to more closely resemble that which would be produced by a traditional rear view mirror in terms of, for example, apparent distance to objects shown on the display device 18. It will be appreciated that the system 10 is not limited to having a single front camera 32, but may in fact comprise a plurality of front cameras located in a group or cluster, or located in different positions in the vicinity of the display device 18 to provide an enhanced field of view.
Referring to Figure 1 in particular, the rearward view display system 10 of the present embodiment includes two rear cameras 34; comprising a first rear camera 40 and a second rear camera 42. The first and second rear cameras 40, 42 are both provided at the rear of the vehicle 12 and each includes a rear-facing digital camera for generating video images. In this way, the rear cameras 34 capture a view behind the vehicle 12 that is unobstructed by the vehicle 12 itself, or by any objects contained within the vehicle 12. As will be explained in more detail later, the images from the rear cameras 34 are used in conjunction with the images from the front camera 32 to form the composite image 38 for display on the display device 18.
Still referring to Figure 1, the first rear camera 40 is positioned directly above a rear body aperture closure element 44 of the vehicle 12, at a central location across the width of the vehicle 12. The second rear camera 42 is positioned above a rear license plate (not shown) of the vehicle 12, below the rear closure element 44 (e.g. trunk / boot lid) of the vehicle 12, and is located centrally across the vehicle 12 width. It will be appreciated that the system 10 is not limited to including two rear cameras 34, but could comprise a single rear camera, or more than two rear cameras, located at different transverse positions across the rear vehicle width and/or at different heights. At least one of the rear cameras 40, 42 may be a camera that is already present on the vehicle 12. For example, a reversing camera for providing a parking aid for the vehicle 12 may function as a rear camera 34 in the rearward view display system 10. A benefit of including a greater number of rear cameras 34 is that this can provide an enhanced field of view. Note that, for simplicity, Figure 2 shows a single rear camera 34 only.
The tracking camera 36 is an interior facing digital video camera that is mounted to a dashboard 46 of the vehicle 12 and is positioned in front of the user 14. In some embodiments, the tracking camera 36 is a camera already present on the vehicle 12. For example, a vehicle camera included for monitoring driver drowsiness may also function as the tracking camera 36. The tracking camera 36 is configured for use in determining the position (referred to as Headxyz) of the head 48 of the user 14 relative to the centre of the electronic display screen 28, and the direction (referred to as Eyeey.ez) in which the eyes 50 of the user are looking relative to when the user is gazing straight ahead through the vehicle windscreen 51. As already mentioned, the front camera 32 may provide this tracking functionality in other embodiments of the invention.
Referring now to Figures 2 and 3, which show a plan (top) view and a side elevation view, respectively, of the rearward view display system 10 installed in the vehicle 12, the eye direction of the user Eyeey,ez, is defined by a horizontal eye direction angle, referred to as EyeSy, and a vertical eye direction angle, referred to as EyeSz. The horizontal eye gaze direction angle, Eyeey, defines the eye direction of the user 14 in a horizontal plane defined by X-axis 52 and Y-axis 54 (see Figure 2). The vertical eye direction angle, Eyeez, defines the eye direction of the driver in a vertical plane defined by X-axis 52 and Z-axis 56 (see Figure 3). The head position of the user 14, Headxyz, is defined by three distances, Headx, Heady, and Headz, which define the distance of the user’s head 48 from the front camera 32 along the X, Y and Z axes respectively.
The tracking camera 36 enables the rearward view display system 10 to adjust the composite image 38 displayed on the display screen 28 in dependence on the head position and eye direction of the user 14. In this way, the display device 18 behaves similarly to a traditional rear view mirror, with the displayed composite image 38 changing according to the direction and position from which the display device 18 is viewed. Thus, the display device 18 displays an image which appears natural and intuitive to a user 14 who is accustomed to viewing a traditional rear view mirror.
In addition to the image processor 22, the processing unit 21 further comprises a system memory and a wireless receiver (not shown). The processing unit 21 is configured to receive signals from the front camera 32, rear cameras 34 and tracking camera 36, where said signals comprise image data corresponding to images captured by the front camera 32, rear cameras 34 and tracking camera 36, respectively. The image processor 22 is further configured to execute one or more sets of software instructions held in the system memory corresponding to standard image processing techniques, based on the received signals, such that a composite image 38 may be formed. In this embodiment of the invention, the front camera 32, rear cameras 34 and tracking camera 36 communicate wirelessly with the processing unit 21 through associated wireless transmitters (not shown) which transmit image data signals (corresponding to images) to the wireless receiver of the processing unit 21. In other embodiments the cameras 32, 34, 36 may be connected to the processing unit 21 by means of wires. A procedure 100 by which the rearward view display system 10 generates, processes and displays images will now be described with particular reference to Figures 4, 5 and 6.
In a first step 102, the front camera 32, rear cameras 34 and tracking camera 36 each output a signal comprising image data to the processing unit 21. The signal from the front camera 32 corresponds to first image data comprising a front image 60 having a first frame of reference or field of view. The signals from the rear cameras 40, 42 correspond to second image data comprising a first rear image and a second rear image, respectively, each having what will be referred to as a second frame of reference (or field of view). The signal from the tracking camera 36 corresponds to tracking image data comprising a tracking image, whereby the tracking camera 36 provides means for determining a user’s field of view as defined by the rearward view display system 10.
At steps 104 and 106, following receipt of said image data, the processing unit 21 processes the tracking image using real-time image processing techniques to determine the head position, Headxyz, and eye direction, Eyeey,ez of the user 14. It should be noted that in embodiments of the system 10 which do not include a tracking camera 36, image data from the front camera 32 may instead be used to determine these parameters.
At subsequent steps 108 and 110, the rearward view display system 10 determines a size and position of a perspective field of view 58 of the front image 60, where the perspective field of view 58 corresponds to a perspective frame of reference 58. With particular reference to Figure 4, the perspective field of view 58 is illustrated as a rectangular region of the front image 60, which defines the field of view of the front image 60 for use in forming the composite image 38. However, the skilled person would understand that the perspective field of view 58 may have a different shape, e.g. square, circular or oval, depending on the field of view of the front camera 32 and/or the shape of the visual display 26 on which the composite image 38 is displayed. At step 108, the size of the perspective field of view 58, defined as Yb - Ya and Zb - Za, is determined based on the distance vector between the driver head position, Headxyz, and a front camera position, CamMp0Sn,xYz· At step 110, the position of the perspective field of view 58, defined by coordinates Ya, Za, is determined in dependence on the user’s head position, Headxyz, and eye direction, Eyeey,ez·
Changing the perspective field of view in dependence on the viewing position of the user 14 enables the image displayed on the display device 18 of the rearward view display system 10 to mimic the way in which the image observed in a traditional rear view mirror changes when viewed from different positions. For example, in one embodiment, the system is configured so that, when the user 14 moves closer to the display device 18, the system 10 increases the field of view displayed on the display screen 28. Conversely, when the user 14 moves further away from the display device 18, the system 10 is configured to decrease the field of view displayed on the display screen 28. When the user 14 changes their viewing angle with respect to the display device 18, the system 10 may change the region of the maximum available field of view that is displayed on the display screen 28. In this way, the rearward view display system 10 of the invention provides a more user-friendly display when compared to known rearward view display systems which do not adjust the displayed image in dependence on the user viewing position.
At step 112, the processing unit 21 determines a portion of the perspective field of view 58 of the front camera image that provides an unrestricted view 62 out of the rear of the vehicle 12. This step 112 is achieved by comparing the first image data captured by the front camera 32 with the second image data captured by the rear cameras 34. An obstruction is determined as any feature of the first image data that cannot be correlated with a feature of the second image data (which represents an unobstructed view out of the rear of the vehicle). As best seen in Figure 6, the unrestricted view 62 corresponds to the field of view that is not obstructed by obstacles within the vehicle 12 such as passengers or luggage, or by obstacles associated with the vehicle 12 itself such as passenger seats 16 or internal vehicle structure. The remaining portion of the perspective field of view 58 defines a restricted view 63 out of the rear of the vehicle 12.
At step 114, the processing unit 21 modifies a combined rear image (formed using the first rear image and second rear image from the first and second rear cameras 40, 42, respectively) to the perspective of the front image. For this, the image processor 22 applies a transformation to the second image data to form third image data having a frame of reference equivalent to the first frame of reference of the front camera 32. At a subsequent step 116, the processing unit 21 forms the composite image 38 by stitching the third image data representing the modified combined rear image onto the perspective field of view 58 of the front image, to overlay the portions of the front image that provide a restricted view 63 with an unrestricted view from the combined rear image. In this way, the composite image 38 provides an image having the first frame of reference, which appears to the user to have been captured at the location of the front camera 32.
In a further step 118, the system 10 determines if the user 14 is attempting to view an interior 64 of the vehicle 12 in the display device 18, for example to observe a passenger (e.g. a child or infant) seated in a rear seat 16 of the vehicle 12. Different characteristics of head and eye movement are indicative of a user’s focus and gaze direction, and so can be used to determine if the user 14 wishes to view the interior 64 of the vehicle 12, for example depending on the head position and eye direction of the user 14 (Headxyz, Eyeey,ez)·
If the system 10 determines that the user 14 is trying to view the interior 64 of the vehicle 12 in the display device 18, the method moves on to step 120. At step 120, the system 10 displays only the image provided by the front camera 32 on the display screen 28. In this way, the user 14 is able to view people or items within the vehicle 12 when desired, in response to certain head and eye movements of the user which indicate that they are attempting to view the interior 64 of the vehicle 12. Alternatively, if the system 10 determines that the user 14 is not attempting to view the interior 64 of the vehicle 12 in the display device 18 because the characteristics of eye and head movement indicate different preferences, then the method 100 moves on to step 122, where the composite image 38 is displayed on the display screen 28 of the display device 18.
Embodiments of the invention ensure that the rearward view display system 10 provides a natural and intuitive view out of the rear of the vehicle 12 with which the user 14 will be comfortable and familiar, and which is free from obstructions blocking the field of view of the user 14.
Many modifications may be made to the above examples without departing from the scope of the present invention as defined in the accompanying claims.

Claims (25)

1. A method of displaying an image on a display device of a rearward view display system of a vehicle, the method comprising: capturing first image data from a first imaging means located at a first location of the vehicle proximal to the display device, the first image data having a first frame of reference; capturing second image data from a second imaging means located at a second location of the vehicle, the second image data having a second frame of reference; applying a transformation to the second image data to form third image data having the first frame of reference; forming a composite image from the first image data and the third image data, such that the composite image appears to a user to have been captured at the first location.
2. A method as claimed in claim 1, wherein the first imaging means is a rearward facing camera.
3. A method as claimed in claim 1 or claim 2, wherein the second imaging means is a rearward facing camera located proximal to the rear of the vehicle.
4. A method as claimed in any preceding claim, wherein the method comprises determining a gaze of the user.
5. A method as claimed in claim 4, wherein determining the gaze of the user comprises determining at least one of a head position and an eye direction of the user relative to the display device.
6. A method as claimed in claim 5, wherein determining at least one of a head position and an eye direction of the user includes tracking the orientation of at least one of the user’s eyes to determine the eye position of the user.
7. A method as claimed in any of claims 4 to 6, wherein the composite image is formed in dependence on the gaze of the user.
8. A method as claimed in any preceding claim, wherein the method comprises determining a restricted portion of the first image data.
9. A method as claimed in claim 8, wherein forming the composite image comprises overlaying a portion of the third image data corresponding to the restricted portion of the first image data onto the first image data to provide an unrestricted composite image.
10. A rearward view display system for a vehicle, the system comprising: a display device comprising a visual display for displaying images formed from image data; a first imaging means located at a first location of the vehicle proximal to the visual display, the first imaging means being configured to provide first image data; a second imaging means, configured to provide second image data; an image processor configured to apply a transformation to the second image data to form third image data, such that when the third image data is combined with the first image data, both the first image data and the third image data appear to have been captured from the position of the first imaging means, when displayed on the visual display, wherein the image processor is configured to form a composite image comprising at least a part of the first image data and at least a part of the third image data, such that the composite image appears to the user to have been captured at the location of the first imaging means.
11. A rearward view display system as claimed in claim 10, wherein the first imaging means is a rearward facing camera.
12. A rearward view display system as claimed in claim 10 or claim 11, wherein the second imaging means is a rearward facing camera located proximal to the rear of the vehicle.
13. A rearward view display system as claimed in any of claims 10 to 12, comprising third imaging means configured to determine a gaze of the user.
14. A rearward view display system as claimed in claim 13, wherein the third imaging means are configured to determine the gaze of the user by determining at least one of a head position and an eye direction of a user relative to the visual display.
15. A rearward view display system as claimed in claim 14, wherein the third imaging means are configured to track the orientation of at least one of the user’s eyes to determine the eye position of the user.
16. A rearward view display system as claimed in any of claims 13 to 15, wherein the image processor is configured to form the composite image in dependence on the gaze of the user.
17. A rearward view display system as claimed in any of claims 10 to 16, wherein the image processor is configured to determine a restricted portion of the primary image data.
18. A rearward view display system as claimed in claim 17, wherein the image processor is configured to form the composite image by overlaying a portion of the third image data corresponding to the restricted portion of the first image data onto the first image data to provide an unrestricted composite image.
19. A rearward view display system as claimed in any of claims 10 to 18, wherein the first imaging means is configured to determine a gaze of the user.
20. A controller for displaying an image on a display device of a rearward view display system of a vehicle, the controller comprising: a first input configured to receive first image data from a first imaging means located at a first location of a vehicle proximal to the display device, the first image data having a first frame of reference; a second input configured to receive second image data from a second imaging means located at a second location of the vehicle, the second image data having a second frame of reference; a processor configured to: apply a transformation to the second image data to form third image data having the first frame of reference; and form a composite image from the first image data and the third image data, such that the composite image appears to the user to have been captured at the first location; and an output configured to output the composite image to the display device of the rearward view display system.
21. A vehicle comprising a rearward view display system as claimed in any one of claims 10 to 19.
22. A vehicle as claimed in claim 21, wherein the rearward view display device is mounted to an internal roofline in a cabin of the vehicle.
23. A vehicle as claimed in claim 21 or claim 22, comprising a controller as claimed in claim 20.
24. A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any one of claims 1 to 9.
25. A non-transitory computer-readable medium having stored thereon the computer program product of claim 24.
GB1709972.2A 2017-06-22 2017-06-22 Method of displaying an image on a display of a rearward view display system Active GB2563647B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
GB1709972.2A GB2563647B (en) 2017-06-22 2017-06-22 Method of displaying an image on a display of a rearward view display system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
GB1709972.2A GB2563647B (en) 2017-06-22 2017-06-22 Method of displaying an image on a display of a rearward view display system

Publications (3)

Publication Number Publication Date
GB201709972D0 GB201709972D0 (en) 2017-08-09
GB2563647A true GB2563647A (en) 2018-12-26
GB2563647B GB2563647B (en) 2020-03-25

Family

ID=59523675

Family Applications (1)

Application Number Title Priority Date Filing Date
GB1709972.2A Active GB2563647B (en) 2017-06-22 2017-06-22 Method of displaying an image on a display of a rearward view display system

Country Status (1)

Country Link
GB (1) GB2563647B (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2586389A (en) * 2018-03-29 2021-02-17 Mitsubishi Electric Corp Image processing device, image processing method, and monitor system
US11833968B2 (en) 2019-03-02 2023-12-05 Jaguar Land Rover Limited Imaging system and method

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014130049A1 (en) * 2013-02-25 2014-08-28 Johnson Controls Technology Company Systems and methods for augmented rear-view displays
US20160288717A1 (en) * 2013-03-29 2016-10-06 Aisin Seiki Kabushiki Kaisha Image display control apparatus, image display system and display unit

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014130049A1 (en) * 2013-02-25 2014-08-28 Johnson Controls Technology Company Systems and methods for augmented rear-view displays
US20160288717A1 (en) * 2013-03-29 2016-10-06 Aisin Seiki Kabushiki Kaisha Image display control apparatus, image display system and display unit

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2586389A (en) * 2018-03-29 2021-02-17 Mitsubishi Electric Corp Image processing device, image processing method, and monitor system
GB2586389B (en) * 2018-03-29 2022-03-30 Mitsubishi Electric Corp Image processing device, image processing method, and monitoring system
US11833968B2 (en) 2019-03-02 2023-12-05 Jaguar Land Rover Limited Imaging system and method

Also Published As

Publication number Publication date
GB201709972D0 (en) 2017-08-09
GB2563647B (en) 2020-03-25

Similar Documents

Publication Publication Date Title
CN108621937B (en) In-vehicle display apparatus, control method of in-vehicle display apparatus, and storage medium storing control program of in-vehicle display apparatus
US8781731B2 (en) Adjusting method and system of intelligent vehicle imaging device
US9376059B2 (en) Method for presenting an image in a vehicle
US20160297362A1 (en) Vehicle exterior side-camera systems and methods
JP6035298B2 (en) Visual system
US20190100145A1 (en) Three-dimensional image driving assistance device
US11034242B2 (en) Motor vehicle with a display arrangement and method for operating a display arrangement of a motor vehicle
US10793069B2 (en) Method for assisting the driver of a motor vehicle in maneuvering the motor vehicle with a trailer, driver assistance system as well as vehicle/trailer combination
US11794667B2 (en) Image processing apparatus, image processing method, and image processing system
WO2015013311A1 (en) Vehicle imaging system
WO2019187283A1 (en) Image processing device, image display system, and image processing method
JP2023536976A (en) Vehicle vision system and method including electronic image display for driver rearview
CN211468310U (en) Vehicle display system and vehicle
US20220001803A1 (en) Image processing apparatus, image processing method, and image processing system
GB2563647A (en) Method of displaying an image
JP2010208359A (en) Display device for vehicle
JP2017111739A (en) Driving support apparatus and driving support method
JP2007290570A (en) Vehicular display device
US20220030178A1 (en) Image processing apparatus, image processing method, and image processing system
JP7073237B2 (en) Image display device, image display method
CN111669543A (en) Vehicle imaging system and method for parking solutions
US20200278745A1 (en) Vehicle and control method thereof
US20180316868A1 (en) Rear view display object referents system and method
US10933812B1 (en) Outside-vehicle environment monitoring apparatus
JP4696825B2 (en) Blind spot image display device for vehicles