WO2019034916A1

WO2019034916A1 - System and method for presentation and control of virtual camera image for a vehicle

Info

Publication number: WO2019034916A1
Application number: PCT/IB2017/054999
Authority: WO
Inventors: Gustavo Fortmann
Original assignee: Harman International Industries, Incorporated
Priority date: 2017-08-17
Filing date: 2017-08-17
Publication date: 2019-02-21

Abstract

The present disclosure relates to system and method for presentation and control of a virtual camera view for a vehicle. The method includes receiving image data from a wide-angle camera mounted to the vehicle in a fixed position. A section of the image data is selected based on a vehicle control setting to generate a virtual camera view presenting image data of the scene from the perspective of different position or orientation than the cameras position. Output of a virtual camera view is performed on display of the vehicle. The vehicle operation is detected and graphical elements are included to the virtual camera views to aid vehicle operation while driving in reverse.

Description

SYSTEM AND METHOD FOR PRESENTATION AND CONTROL OF

VIRTUAL CAMERA IMAGE FOR A VEHICLE

FIELD

[0001] The present disclosure relates to systems, methods and devices for driver assistance, and more particularly to presentation and control of virtual camera views for a vehicle.

BACKGROUND

[0002] In recent years, vehicles have been outfitted with cameras to provide a view of space and objects surrounding a vehicle. By way of example, many vehicles include a single back-up camera for parking assistance. Often these cameras are used for driver assistance and to improve safety. While cameras are known to provide views for parking assistance, use of these devices is often limited due to the fixed view or limited display area presented by captured image data. Existing vehicle systems typically configure backup cameras which present a view that either faces straight back or provides a separate hitch view (e.g., downward view). These types of systems are limited in that they each do not allow for viewing objects which may be in vehicle blind spots or that are outside of the presented view. Existing vehicle systems may use multiple cameras to stitch images which can lead to higher costs due to multiple cameras and may require user operation. There exists a need to improve vehicle systems to provide additional features and different functions. There also exists a need to improve image data for presentation in vehicles.

BRIEF SUMMARY OF THE EMBODIMENTS

[0003] Disclosed and claimed herein are methods, devices and systems for presentation and control of a virtual camera view for a vehicle. One embodiment is directed to a method including receiving, by a control device, image data from a wide-angle camera, wherein the wide-angle camera is mounted to the vehicle in a fixed position at a first location of the vehicle, and wherein the image data captures a scene from the first location. The method also includes selecting, by the control device, a section of the image data from the wide-angle camera based on a vehicle control setting. The method also includes generating, by the control device, a virtual camera view based on the section the image data, wherein the virtual camera view presents image data of the scene from the perspective of a second location. The method also includes controlling, by the control device, output of the virtual camera view to a display of the vehicle. [0004] In one embodiment, the image data includes at least a 180 degree representation of the scene.

[0005] In one embodiment, selecting a section includes selecting a portion of the scene laterally displaced from a center of the image data.

[0006] In one embodiment, the vehicle control setting is an intended direction of the vehicle, and wherein the virtual camera view is generated to present a view of the scene associated with a direction of travel of the vehicle based on a turn angle measurement.

[0007] In one embodiment, generating includes extracting pixel data for the section of image data and mapping extracted pixels to a modified layout for correction of wide-angle curvature.

[0008] In one embodiment, the virtual camera view is generated to appear from a different position and different orientation using the image data detected by the wide-angle camera.

[0009] In one embodiment, the first position is laterally offset from a centerline of the vehicle and the virtual camera view provides a view from a perspective of the centerline.

[0010] In one embodiment, the first position is associated with a centerline of the vehicle and the virtual camera view provides a view from a perspective that is at least one of laterally offset from the centerline and vertically offset from the first position.

[0011] In one embodiment, the first position is associated with a centerline of the vehicle and the virtual camera view provides a view from a perspective that is laterally offset from the centerline and an orientation of the virtual camera view presents a view that is angularly offset from the centerline.

[0012] In one embodiment, generating the virtual camera position is based on at least one of a turning angle measurement and change in angular of displacement of the vehicle.

[0013] Another embodiment is directed to vehicle system including a wide-angle camera and a control device coupled to the camera. The control device is configured to receive image data from a wide-angle camera, wherein the wide-angle camera is mounted to the vehicle in a fixed position at a first location of the vehicle, wherein the image data captures a scene from the first location. The control device is configured to select a section of the image data from the wide-angle camera based on a vehicle control setting and generate a virtual camera view based on the section the image data, wherein the virtual camera view presents image data of the scene from the perspective of a second location. The control device is configured to control output of the virtual camera view to a display of the vehicle.

[0014] Another embodiment is directed to a method for presentation and control of a virtual camera view for a vehicle. The method includes receiving, by a control device, image data from a wide-angle camera, wherein the wide-angle camera is mounted to the vehicle in a fixed position at a first location of the vehicle, wherein the image data captures a scene from the first location. The method also includes selecting, by the control device, a section of the image data from the wide- angle camera based on a turning angle measurement, and generating, by the control device, a virtual camera view based on the section the image data, wherein the virtual camera view presents image data of the scene from the perspective of a second location and an updated viewing direction, the second location displaced laterally from the first location. The method also includes controlling, by the control device, output of the virtual camera view to a display of the vehicle.

[0015] In one embodiment, the virtual camera view is updated by the control device based on at least one of vehicle displacement and changes in turning angle measurement.

[0016] Other aspects, features, and techniques will be apparent to one skilled in the relevant art in view of the following detailed description of the embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The features, objects, and advantages of the present disclosure will become more apparent from the detailed description set forth below when taken in conjunction with the drawings in which like reference characters identify correspondingly throughout and wherein:

[0018] FIGs. 1A-1C depict graphical representations of vehicle configurations according to one or more embodiments;

[0019] FIG. 2 depicts a process for presentation of a virtual camera view for a vehicle according to one or more embodiments;

[0020] FIG. 3 depicts a graphical representation of device components according to one or more embodiments;

[0021] FIG. 4 depicts a process for presentation and control of a virtual camera view for a vehicle according to one or more embodiments; [0022] FIGs. 5A-5C depict graphical representations of virtual camera views according to one or more embodiments;

[0023] FIGs. 6A-6C depict graphical representations of virtual camera views according to one or more embodiments;

[0024] FIG. 7 depicts a graphical representation of virtual camera view generation according to one or more other embodiments; and

[0025] FIG. 8 depicts a graphical representation of a vehicle configuration according to one or more embodiments.

DETAILED DESCRIPTION OF THE EXEMPLARY EMBODIMENTS

Overview and Terminology

[0026] One aspect of the disclosure is directed to selectively positioning and orienting virtual camera views based on the image data captured by a wide-angle camera. In one embodiment a camera mounted to a vehicle, such as a rear view camera includes a wide-angle lens that allows for the camera to acquire an image of a scene behind the vehicle with a large viewing angle. The scene provides image data of objects in the vicinity of the vehicle, either directly behind, or behind and to either side of the vehicle. In one embodiment, a wide-angle camera is utilized to capture visual information proximate the vehicle and to capture an entire scene including elements behind and at both sides of a vehicle. Wide-angle cameras create an image which may be subject to lens distortion. As such, presentation of raw image data (e.g., unmodified) may be difficult to interpret, especially for a driver. According to one embodiment, image data from the wide-angle camera may be processed to select a portion of the image data and present a view that is easier to understand and which may be more "natural" to view.

[0027] In one embodiment, a process is provided for presentation and control of a virtual camera view. The process can include receiving image data, selection of a portion of image data, and generating a virtual camera view from the selected portion of image data. Generating the virtual camera view can include one or more operations to generate an output image that appears to be taken from at least one of a different position and different orientation (e.g., viewing angle) than the actual position of the camera that detects image data. In addition, control of the generation may be based on one or more detected vehicle settings. Virtual camera views may be created to enhance driving experience of a vehicle in reverse. Virtual camera views may be controlled based on vehicle operation and settings such that the view may be updated during operation of the vehicle based on one more control parameters. The initial display characteristics may allow for a backward facing view where virtual camera position is then adjusted to one or more other positions and directions.

[0028] As used herein, a virtual camera view relates to an image generated from a selection of image data received from a wide-angle camera, the image data altered in such a manner so as to appear to have been taken from a camera positioned and/or oriented differently (e.g., in any angular degree of freedom) from the wide-angled camera that generated the source image data and with an apparent different field of view of the wide-angle camera. A virtual camera view may be generated from the perspective of one or more virtual camera positions. A virtual camera position may be characterized as having a different location or different orientation from the actual camera position and orientation. Similarly, a virtual camera view may be characterized as having a different field of view than the actual camera position and orientation.

[0029] A vehicle relates to one or more of a car, passenger vehicle, van, sport utility vehicle, trailer, bus, and vehicle with a backup camera in general and is not limited to on-road vehicles and may include one or more of mining equipment, farming equipment and recreational off-road vehicles.

[0030] Another embodiment is directed to presenting image data based on turning angle of a vehicle. In one embodiment, detecting vehicle operation includes detecting angular changes to the vehicle during vehicle operation to determine an intended direction of the vehicle. One or more turning angle measurements may be performed during vehicle operation to determine intended direction. Turning angle measurements may be performed based on one or more vehicle sensors. In certain embodiments, turning angle measurements may be determined using steering wheel position and/or tire/wheel direction. In certain embodiments, a virtual camera position may be modified based on detected changes in vehicle position and/or vehicle controls (e.g., turning a steering wheel before reversing) such that motion in a backwards direction that is not directly backwards, controls presentation of a virtual camera image.

[0031] As used herein, the terms "a" or "an" shall mean one or more than one. The term "plurality" shall mean two or more than two. The term "another" is defined as a second or more. The terms "including" and/or "having" are open ended (e.g., comprising). The term "or" as used herein is to be interpreted as inclusive or meaning any one or any combination. Therefore, "A, B or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". An exception to this definition will occur only when a combination of elements, functions, steps or acts are in some way inherently mutually exclusive.

[0032] Reference throughout this document to "one embodiment," "certain embodiments," "an embodiment," or similar term means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of such phrases in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner on one or more embodiments without limitation.

Exemplary Embodiments

[0033] Referring now to the figures, FIGs. 1 A- 1C depict graphical representations of vehicle configurations according to one or more embodiments. In one embodiment, a vehicle configuration 100 is provided for vehicle 105 including a rear mounted camera 110. Vehicle configuration 100 allows for camera 110 to be mounted to vehicle 105 in a fixed location pointing in a fixed direction. Camera 110 may be a vehicle backup camera. According to one

embodiment, camera 110 is a wide-angle camera mounted to a rear portion of vehicle 105, such as the bumper, trunk hood, lift gate, rear hatch, license plate housing, etc. FIGs. 1A-1C depict operation of vehicle components with respect to an exemplary mounting location of a camera according to one or more embodiments. As will be discussed herein, principles of the disclosure may be applied to other mounting locations and may be applied to other functions. In addition, although FIGs. 1A-1C depict vehicle 105 as a passenger vehicle, it should be appreciated that the processes, configurations and systems described herein may apply to other vehicle types, including, vans, sport utility vehicles, trucks, and vehicles in general including back up cameras.

[0034] FIG. 1A depicts a graphical representation of vehicle configuration 100 including a side profile of vehicle 105. According to one embodiment, vehicle configuration 100 may be configured to output a virtual camera view for vehicle 105. As shown in FIG. 1A, camera 110 is located on a rear portion of vehicle 105, such as the vehicle bumper. It may be appreciated that the principles of the disclosure do not limit the placement of camera 110 on a bumper, as other locations of the vehicle may be employed to mount camera 110. Camera 110 may be configured as a rear facing camera to capture image data for a scene behind vehicle 105, which may be used to present image data within vehicle 105 to assist a driver. According to one embodiment, camera faces directly behind vehicle 105 as shown by direction 120. According to another embodiment, camera 110 is a wide-angle camera configured with a large angle of view in vertical and horizontal planes. An exemplary representation of the detection area of camera 110 is shown by upper bound 115a and lower bound 115b. Upper bound 115a and lower bound 115b are exemplary and may provide a field of view greater than at least 180 degrees according to one or more embodiments.

[0035] According to one embodiment, a virtual camera view may be output to a display (not shown in FIG. 1 A) of vehicle 105 from the perspective of a virtual camera position. The virtual camera view allows for at least one of the viewing direction and position of camera 110 to appear as if the camera has been repositioned or redirected. According to an exemplary embodiment, virtual camera position 125 may be a position associated with the back of vehicle 105 above the position of camera 110, such as the top portion of a lift gate, spoiler, etc. Because camera 110 includes a wide-angle of detection in the vertical plane, as shown by exemplary boundaries 115a- 115b, image data detected by camera 110 may be used to generate an output image from the perspective of virtual camera position 125. It should be appreciated that having a view from a higher location may allow for greater visibility and thus, improve driver view. Virtual camera position 125 relates to a virtual vertical displacement of camera 110 such that image date output by the camera appears to be generated from virtual camera position 125. For example, boundaries 126a-126b may characterize the field of view associated with virtual camera position 125. As such, the field of view of the virtual camera position is a selection of a scene within image data captured within boundaries 115a-115b. The area behind vehicle 105 in the direction 120 may make up a scene detected by image data generated by camera 110.

[0036] FIG. IB depicts a graphical representation of vehicle configuration 100 including a plan view (e.g., birdseye view) of vehicle 105. As shown in FIG. IB, camera 110 is located on a rear portion of vehicle 105, such as the vehicle bumper facing direction 120. According to one embodiment, a virtual camera view may be presented from one or more positions laterally displaced from the position of camera 110, such as position 130 and position 135. In addition to presenting the image for a location that is laterally displaced, the direction of the camera 110 may be virtually oriented (e.g., directed). According to another embodiment, an exemplary representation of the detection area of camera 110 is shown by bounds 140a and 140b.

[0037] According to another embodiment, sections of image data captured by camera 110 may be presented. In one embodiment, a section of the image data may relate to a view associated with the direction of travel and angular displacement of the vehicle. According to one embodiment, although image data may be collected for an entire scene within bounds 140a-140b, the virtual camera position may be presented for the area within bounds 145a-145b. As will be discussed in more detail below, the portion of the scene presented as a virtual camera view may be modified from the scene within bounds 145a-145b when a virtual camera view is generated.

[0038] As described herein, virtual camera position may be displaced from the actual camera position in one or more of a lateral direction (e.g., displaced to the side) and vertical direction (e.g., vertically displaced, raised above the car, etc.). According to another embodiment, virtual camera position may be displaced a distance from the actual camera position in direction 120, such that the virtual camera position is moved away from the vehicle. As such the virtual camera position may include three degrees of freedom in translation to include up and down, left and right, and projection back behind the vehicle. The virtual camera view can then be angled from the virtual camera position to include a field of view includes the rear of the vehicle and the entire bumper as well as portions of a scene behind and/or to the side of the vehicle that is of interest. By way of example, when the virtual camera position is projected back from the vehicle, the virtual camera view may be similar to the vantage point of a person standing behind the vehicle. According to another embodiment, graphical representations, such as digital representation of the vehicle, can be used to complete the scene if the virtual camera view aims partially towards the vehicle, where captured image data may not include portions of a vehicle behind the actual camera position.

[0039] FIG. 1C depicts a graphical representation of vehicle configuration 100 including a rear view of vehicle 105. As shown in FIG. 1C, camera 110 is located on a rear portion of vehicle 105, such as the vehicle bumper. According to one embodiment, a virtual camera view may be presented from one or more positions. Exemplary virtual camera positions are shown in FIG. 1C at position 125 at top center of the vehicle rear, position 130 at passenger side bumper, position 135 are driver (e.g., left hand drive vehicle) side bumper, position 150 at passenger side top and position 155 at driver side top. Positions shown in FIG. 1C are exemplary, it should be appreciated that the virtual camera position may be located in other locations, either based on the vehicle type, vehicle height etc. It should also be appreciated that depending on the wide-angle lens employed, a virtual camera position may be located above the actual vehicle such as position 160. In some embodiments, the wide-angle lens (and processing) may allow for position of the virtual camera to be displaced a lateral distance from the vehicle side and even extend beyond limits of the vehicle's width. In addition, the wide-angle lens may allow for adjusting the orientation (e.g., direction) of the virtual camera view. [0040] FIG. 2 depicts a process for presentation of a virtual camera view for a vehicle according to one or more embodiments. Process 200 may be employed by a device, such as a vehicle control unit of a system and one or more other components to output a virtual camera view. Process 200 may be performed by a single device or multiple devices. As will be described herein, a vehicle (e.g., vehicle 105) may include one or more modules, such as a view generator module to generate one or more virtual camera views. The view generator module of the vehicle may be a software module executed by a processor. Process 200 may be executed by one or more components of a vehicle, such as a control device.

[0041] According to one embodiment, process 200 may be initiated by a control device receiving image data from a wide-angle camera at block 205. In one embodiment, the image data is detected by a wide-angle camera mounted to a vehicle in a fixed position. The camera may be mounted in a location associated with the rear portion of the vehicle to allow for back up assist and/or act as a backup camera. The camera is mounted on the vehicle at a location such that the image data from the camera captures a scene from the first location. According to one embodiment, because the camera is a wide-angle camera, the detection area of the camera can allow for a wide view to be generated, such as a 180 degree representation of the scene behind the vehicle. According to another embodiment, image data received at block 205 requires at least one of processing and correction to allow for a useable display. As will be discussed in more detail below, the detected image data may appear to include a substantial degree of curvature due to the wide-angle lens and will not be suitable for display by the vehicle. However, the wide-angle image may allow for a greater representation of the scene from a fixed camera.

[0042] At block 210, the control device selects a section of the image data from the wide-angle camera. By selecting a section of the image data, a portion of the scene may be presented as a displayed image in the vehicle after one or more operations. Usually, vehicle displays for backup cameras are limited in size (e.g., less than 10 inches/25 cm). Accordingly representation of the entire scene at a single time may not be useful for an operator of the vehicle to view when not driving backwards. In one embodiment, a section relates to a portion of the image data defined by a boundary. Sections may relate to a middle portion of a scene, driver side portion of the scene, and passenger side portion of the scene. According to another embodiment, each section may be based on the wide-angle lens features, focal length of the camera and a desired output size/zoom for image data to be presented. By way of example, each section may relate to a boundary having a shape or design which accounts for the camera features such that a middle portion of the scene includes a first shape for the section, and lateral portions of the scene will require different shaped sections.

[0043] In one embodiment, selection of a section at block 210 may be based on a vehicle control setting, such as one or more of steering wheel position, amount of vehicle displacement, user control setting of a vehicle control interface, and object detection. In one exemplary embodiment, when a vehicle steering wheel is rotated, selecting a section includes selecting a portion of the scene laterally displaced from a center of the image data. By way of example, an operator of the vehicle may put the vehicle in reverse triggering display of a back up camera. By turning the steering wheel in a particular direction (e.g., backing out to the right/passenger side), objects within the scene of the passenger side or oncoming objects (e.g., traffic, pedestrians, etc.) are of interest. Accordingly, in this example, a section of the scene corresponding to the direction of travel of the vehicle rear end may be selected.

[0044] According to another embodiment, selection of the image data may be based on virtual camera position. The virtual camera position may be selected based on, or in response to, a vehicle setting, such that steering wheel position may trigger at least one of a virtual camera position laterally displaced from the camera location and a virtual change in camera direction. Image data may be selected at block 210 based on at least one of a desired virtual camera position and virtual direction. In that fashion, the vehicle control setting is steering wheel position, and the virtual camera view is generated to present a view of the scene corresponding to the direction of travel of the vehicle based on the steering wheel position.

[0045] At block 215, the control device generates a virtual camera view based on a selected section the image data. Generating the virtual camera view at block 215 can account for one or more of a selected virtual camera position and virtual camera direction. The virtual camera view presents image data of the scene from the perspective of a second location (e.g., the virtual camera position), such as a position of the vehicle, or locations which extend beyond the width and height of the vehicle. In one embodiment, generating at block 215 includes extracting pixel data for the section of image data selected and mapping extracted pixels to a modified layout for correction of wide-angle curvature. As mentioned above, raw image data of a wide-angle camera will allow for a wide detection area of a scene, such as a 180 degree representation, however, the raw image data will not be suitable for presentation in the vehicle. Generating at block 215 may include performing a correction for curvature of a wide-angle lens and to allow for presentation of the image such that perspective of the image appears from at least one of a different location and direction (e.g., orientation). Accordingly, the virtual camera view is generated to appear from at least one of a different position, different orientation (e.g., direction), field of view, and combinations thereof, using the image data detected by the wide-angle camera by processing pixel location from a first coordinate in the received image data to a second coordinate system associated with the virtual camera location and orientation. Generating at block 215 allows for one or more virtual camera positions, such as a camera at a first position associated with a centerline of the vehicle and the virtual camera view provides a view from a perspective that is at least one of laterally offset from the centerline and vertically offset from the first position.

[0046] According to one embodiment, generating at block 215 allows for providing a more informative view of a scene in comparison to a rear facing camera with a fixed view. According to another embodiment, generating at block 215 may allow for image data detected by a camera offset from a vehicle centerline to appear as from the vehicle middle. For example, generating at block 215 can include modifying output of an offset camera to appear as having a virtual camera position along a centerline of the vehicle and the virtual camera view provides a view from a perspective of the centerline.

[0047] At block 220, output of virtual camera view may be controlled to allow for presentation of the virtual camera view on a vehicle display. In one embodiment, the virtual camera view is controlled to be updated based on changes to steering wheel position. According to another embodiment, the virtual camera view may be controlled based on objects detected by one or more sensors of the vehicle such that a view is modified to highlight near objects. According to another embodiment, multiple virtual camera positions may be defined for vehicle maneuvers, such that different sections of the image scene are identified during a vehicle maneuver. At block 220, the virtual camera view may be updated based on changes in received image data.

[0048] FIG. 3 depicts a graphical representation of device components according to one or more embodiments. According to one embodiment, device 300 relates to a control device of a vehicle configured to provide image data for a virtual camera view. Device 300 includes processor 305, camera 310, data storage unit 315, input/output module 320 and display 325. According to one embodiment, device 300 may interoperate with one or more components of a vehicle system 330, which may be separate from device 300, to receive operation characteristics of the vehicle (e.g., steering wheel position, angular adjustments, selection of reverse gear, etc.)

[0049] Processor 305 is configured to control operation of device 300. According to one embodiment, processor 305 may be configured to run a view generator module 306 to generate virtual camera views. In other embodiments, view generator module 306 may be a physical hardware unit of device 300.

[0050] In one embodiment, view generator module 306 is configured to generate virtual camera views based on image data received from camera 310. Processor 305 may also function as a control unit to control output and updating of virtual camera views based on one or more signals from vehicle system 330. Processor 305 may operate based on executable code of view generator 306 and data storage unit 315 to perform and control functions of device 300. By way of example, processor 305 may execute process 200 of FIG. 2 to output virtual camera views.

Processor 305 may execute and direct one or more processes and functional blocks described herein to present virtual camera views and control presentation of elements.

[0051] Camera 310 may be a wide-angle camera configured to detect image data with a 180 degree detection range in a horizontal plane. Data storage unit 315 may be configured to store executable code to operate processor 305 and device 300. Input/output (I/O) module 320 may be configured to receive inputs from a controller or input surface (e.g., touch screen, input buttons, etc.) and to output image data to display 325.

[0052] In certain embodiments, device 300 and processor 305 may be configured to communicate with components of a vehicle, such as vehicle system 330. By way of example, vehicle system 330 may relate to a user interface system of a vehicle including one or more sensors, functions and data capabilities. In that fashion, vehicle system 330 may provide one or more of location data, and vehicle operating data to processor 305, including but not limited to GPS data, heading, acceleration, map information, points of interest, etc.

[0053] FIG. 4 depicts a process for presentation and control of a virtual camera view for a vehicle according to one or more embodiments. According to one embodiment, vehicle operation may be detected to update or change virtual camera position and thus, parameters for displaying image data within a vehicle using process 400.

[0054] Process 400 may be initiated by detection of image data at block 405. A wide-angle camera mounted to a vehicle may detect image data for a scene which may be used to generate a virtual camera view. At block 410, the virtual camera view may be output to a display. Process

400 also includes detecting vehicle operation at block 415. In certain embodiments, steering wheel position (e.g., turned right from a center position from 0.25 turn to 1.5 turns) may translate to one or more virtual camera positions. In one embodiment, the amount of steering wheel displacement is directly correlated with the displacement of a virtual camera from a vehicle centerline. According to another embodiment, any amount of steering wheel displacement in a certain direction is directly correlated with displacement of a virtual camera from a vehicle centerline to select a virtual camera position that adjusts the camera position and direction to show as much of the scene (e.g., lateral portion of image data received). Vehicle operation such as speed, acceleration and displacement may be considered and determined to select image data and/or generate a virtual camera view. In one embodiment, when steering settings are tied to the view such that the generated virtual view is updated as the steering wheel is turned. In other embodiments, the virtual view may be generated to maximize presentation of a scene to the side of the vehicle. In certain embodiments, the virtual camera view output at block 410 may be displaced from the actual camera position, rather than just rotating the view, so that a portion of the vehicle can be provided in the view that provides a frame of reference for an operator of the vehicle. In addition, the virtual camera view may include more of a vehicle bumper than a rear facing camera. Providing image data of the vehicle bumper, or in the alternative a graphical element representing the location of the vehicle bumper, can act as a frame of reference for the operator and thus, provide a reference for signaling vehicle location with respect to other object (e.g., vehicles, obstructions, etc.). Rotating and translating the position of reference can allow for a representation of an entire bumper, or the majority of a bumper.

[0055] At decision block 420, process 400 determines whether an initial virtual display output 410 requires modification based on vehicle operation detected at block 415. When the virtual display is not to be updated (e.g., "NO" path out of decision block 420), process 400 continues to output virtual display at block 410. When the virtual display is to be updated (e.g., "YES" path out of decision block 420), process 400 continues with generating a display output at block 415.

[0056] According to one embodiment, generating the display output at block 425 includes one or more operations for generating a virtual camera view using detected image data. The display output generated at block 425 may be output at block 410. Following output of image data at block 410, process 400 may monitor vehicle operation at block 415.

[0057] In one embodiment, image data received at block 405 is corrected. Correction may be for one or more image effects, such as removal of barrel distortion. The control unit may also perform a zoom/enlargement operation to preserve image elements. In addition to selection of one or more regions of received image data, pixel location may be reoriented.

[0058] FIGs. 5A-5C depict graphical representations of virtual camera views according to one or more embodiments. In particular, FIGs. 5A-5C depict one or more virtual camera views that may be presented for a vehicle operation, such as a vehicle attempting a maneuver between objects.

[0059] In FIG. 5A, vehicle 505 is presented including camera 510 mounted to a rear portion of the vehicle such as bumper 506. Vehicle 505 is depicted as performing a maneuver in reverse, shown by direction 525 between barriers 530a-530b. Barriers 530a-530b may be obstacles, other vehicles, part of a structure, parking space dividers, etc. Representation of the type of vehicle 505 and barriers 530a-530b in FIG. 5A are for illustration. Virtual camera views may be presented for other driving maneuvers, vehicle types and obstacles. Camera 510 faces direction 515.

According to one embodiment virtual camera position 520 may be selected.

[0060] FIGS. 5B and 5C depict operation of a control unit of vehicle 505 using image data from camera 510.

[0061] FIG. 5B depicts a representation of back view of vehicle 505 including a bumper 506 and camera 510. According to one embodiment, camera 510 is a wide-angle camera configured to detect image data. Virtual camera view 540 is an exemplary representation of a portion of a scene behind vehicle 505 which includes obstacle 530b. Virtual camera view may relate to an initial output of a vehicle display, such that the virtual camera view provides a backup camera function oriented directly behind vehicle 505, such as direction 515. Virtual camera view 540 also relates to a portion of the detected image data and is rectified to remove most if not all of the curvature from the wide-angle lens of camera 505. According to one embodiment, virtual camera view 540 may optionally include a graphical representation of a camera, shown as 545, in the display and a graphical representation 550 of the bumper of vehicle 505. Graphical representation 545 may be formatted and displayed to notify an operator of the vehicle a vantage point of the virtual camera view. Similarly, graphical representation 550 may not be the actual bumper 506 of the vehicle, but a graphical element added to the virtual camera view to provide a user with a view of the vehicle with respect to objects in the detected image data. Generating a virtual view may include adding one or more graphical elements, such as graphical representation 545 and 550. For purposes of illustration, a desired vehicle path (e.g., path 525) is shown in the virtual camera view as path 551. According to one embodiment, virtual camera view 540 may be modified based on one or more vehicle settings.

[0062] FIG. 5C depicts a modified virtual camera view according to one or more embodiments. FIG. 5C depicts a representation of the back of vehicle 505 including a bumper 506 and camera 510. According to one embodiment, based on detection of one or more vehicle settings which may include steering wheel rotation, a control unit of vehicle 505 may identify virtual camera position 511. Based on virtual camera position 511, virtual camera view 555 may be generated. According to one embodiment, virtual camera view 540 and virtual camera view 555 are each generated from the same physical location of vehicle 505 and the same image data detected by camera 510. However, a control unit of vehicle 505 generates virtual camera view 555 by selecting a different section of the image data and performing one or more operations on the selected image data to generate a virtual camera view that appears to be taken from a different location and/or having a different direction. According to one embodiment, virtual camera view includes a graphical representation of a camera in the display, shown as 546 and a graphical representation 551 of the bumper of vehicle 505. Graphical representations 546 and 551 may be shown differently than graphical representations 546 and 551 to illustrate to an operator the position and orientation of a virtual camera position associated with the virtual camera view. In FIG. 5C, virtual camera view appears to be taken from a laterally displaced location (e.g., virtual camera position 511) as such the graphical elements 546 and 551 for the camera and bumper respectively are rotated in the display to signify position of vehicle 505 in the displayed image. For purposes of illustration, an estimated vehicle path (e.g., path 525) is shown in the virtual camera view as path 552.

[0063] As can be seen from FIGs. 5A-5C, a virtual camera view can include a virtual image generated from actual image data that allows for viewing objects, such as object 530a, that would not be visible with a conventional back facing implementation.

[0064] FIGs. 6A-6C depict graphical representations of virtual camera views according to one or more embodiments. FIG. 6A depicts a graphical representation of image data 605 received from a wide-angle camera and virtual camera view 655 which may be generated based on a selection of image data 605. Image data 605 is shown including a selection area 610. Image data 605 is captured for a scene 615. Scene 615 relates to an area within the detection area of a vehicle camera. In FIG. 6A, scene 615 includes a roadway 620 and objects 625, 630, and 645 associated with a passenger side of a vehicle and objects 640 an 650 associated with a driver side of a vehicle. According to one embodiment, a control unit of a vehicle selects area 610 for presentation of virtual camera view 655.

[0065] Virtual camera view 655 represents an area behind a vehicle and employs pixel data from area 610 to generate a display image within a vehicle. Virtual camera view modifies the pixel location and image data of area 610 such that the appearance of roadway 620, and objects, 625, 630, 640 and 650 are corrected (e.g., dewarped, etc.). By way of example, the effects of curvatures can be rectified to straighten elements to appear more naturally with distortion from the wide-angle lens reduced or removed. FIG. 6A depicts a virtual camera position directly behind a vehicle.

[0066] FIG. 6B, depicts selection of area 661 of image data 605 that is different from area 610. As shown in FIG. 6B, image data 605 includes scene 615 relates to an area within the detection area of a vehicle camera. In FIG. 6B, scene 615 includes a roadway 620 and objects 625, 630, and 645 associated with a passenger side of a vehicle and objects 640 an 650 associated with a driver side of a vehicle. Area 661 is associated with a passenger side (left side of image data 605 in FIG. 6B) which may correlate to a vehicle control or steering wheel control directing the vehicle towards the portion of scene 615 associated with objects 625, 630 and 645. According to one embodiment, area 661 may be selected based one or more vehicle controls and a control unit of the vehicle may generate virtual camera view 665.

[0067] Virtual camera view 665 includes roadway 620 and objects 625, 630, 640and 645 associated with a passenger side of a vehicle. Objects depicted in virtual camera view may be enlarged to show additional detail, such as object 625 for example.

[0068] FIG. 6C depicts selection of area 671 of image data 605 that is different from area 610. As shown in FIG. 6C, image data 605 includes scene 615 relates to an area within the detection area of a vehicle camera. In FIG. 6C, scene 615 includes a roadway 620 and objects 625, 630, and 645 associated with a passenger side of a vehicle and objects 640 an 650 associated with a driver side of a vehicle. Area 671 is associated with a driver side (right side of image data 605 in FIG. 6C) which may correlate to a vehicle control or steering wheel control directing the vehicle towards the portion of scene 615 associated with objects 630, 640 and 650. According to one embodiment, area 671 may be selected based one or more vehicle controls and a control unit of the vehicle may generate virtual camera view 675. Virtual camera view 675 includes roadway 620 and objects 630, 640 and 650 associated with a driver side of a vehicle. Objects depicted in virtual camera view may be enlarged to show additional detail, such as object 650 for example.

[0069] According to one embodiment, the shape of a selection area, such as area 610, area 661, and are 671, may be modified based on changes to a virtual camera position. By way of example, in one embodiment, the source camera captures a multitude of pixels for scene 615. According to one embodiment, pixels of scene 615 are reoriented to simulate that a resulting image comes from a camera in a different position. The source pixels used are determined by mathematical calculations that include the extrinsic parameters (location and orientation of the camera), the intrinsic parameters (resolution of the camera and shape of the lens) and a mathematical translation of the view.

[0070] FIG. 7 depicts a graphical representation of virtual camera view generation according to one or more other embodiments. FIG. 7 depicts a graphical representation of image data 700 which may be received from a wide angle camera and virtual camera view 705. According to one embodiment, image data 700 received form a camera includes a plurality of pixels shown as 710 and 720, for example. Image data 700 may include a plurality of pixels (e.g., more than what is shown in FIG. 7), wherein the pixels are arranged in the image data based on location within a scene and lens characteristics. A wide-angle lens may be characterized as imparting curvature, shown by curved grid lines 710, to image elements. According to one embodiment, generating a virtual camera view, corrects for curvature induced effects to image elements.

[0071] According to another embodiment, elements of image data 700 may be described as being associated with an inner torroidal surface based on the wide-angle lens of the vehicle camera. The image data can be provided to the control unit based on a two dimensional coordinate system or three-dimensional coordinate system. As such, the control unit may convert received image data from the coordinate system image data is received to a coordinate system for presentation on a vehicle display.

[0072] Virtual camera view 705 includes pixel elements 715 and 720 from image data 705, wherein the pixel elements are arranged on a grid 725. In that fashion, pixel element location maybe modified to allow for removal of one or more wide-angle lens effects. Removal of wide- angle effects may not remove all curvature /lens effects from the image data.

[0073] According to one embodiment, conversion of image data 700 to virtual camera view 705 may be based on a transition table such that for each point in virtual camera view 705 the table provides source information from image data 700. In some cases, pixel data of image data 700 may be interpolated to calculate a blended color for an output pixel of virtual camera view 705, such as pixel element 715. One or more other operation may be used in addition to, or instead of a transition table, such as interpolation, and spherical projection. According to one embodiment, conversion of image data 700 to virtual camera view 705 may include one or more operations for determining output color (e.g., hue, saturation, and luminance) of the output pixel from one or more input pixels of image data 700. [0074] FIG. 8 depicts a graphical representation of a vehicle configuration according to one or more embodiments. According to one embodiment, principles of the disclosure allow for presenting a virtual camera view as part of a vehicle system, such as a backup camera, to appear as being generated by a camera positioned at one location (e.g., vehicle centerline) different from the actual camera position, such as a distance from the vehicle centerline. Vehicle 800 of FIG. 8 includes camera 805 displaced from centerline 815. According to one embodiment, generating a virtual camera image includes generating an image using image data received from camera 805 as if detected by a camera positioned at virtual camera position 810.

[0075] While this disclosure has been particularly shown and described with references to exemplary embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the claimed embodiments.

Claims

CLAIMS What is claimed is:

1. A method for presentation and control of a virtual camera view for a vehicle, the method comprising:

receiving, by a control device, image data from a wide-angle camera, wherein the wide- angle camera is mounted to the vehicle in a fixed position at a first location of the vehicle, wherein the image data captures a scene from the first location;

selecting, by the control device, a section of the image data from the wide-angle camera based on a vehicle control setting;

generating, by the control device, a virtual camera view based on the section the image data, wherein the virtual camera view presents image data of the scene from the perspective of a second location; and

controlling, by the control device, output of the virtual camera view to a display of the vehicle.

2. The method of claim 1 , wherein the image data includes at least a 180 degree representation of the scene.

3. The method of claim 1, wherein selecting a section includes selecting a portion of the scene laterally displaced from a center of the image data.

4. The method of claim 1 , wherein the vehicle control setting is intended direction of the vehicle, and wherein the virtual camera view is generated to present a view of the scene resulting from a direction of travel of the vehicle based on a turn angle measurement.

5. The method of claim 1, wherein generating includes extracting pixel data for the section of image data and mapping extracted pixels to a modified layout for correction of wide- angle curvature.

6. The method of claim 1, wherein the virtual camera view is generated to appear from a different position and different orientation using the image data detected by the wide-ang camera.

7. The method of claim 1, wherein the first position is laterally offset from a centerline of the vehicle and the virtual camera view provides a view from a perspective of the centerline.

8. The method of claim 1, wherein the first position is associated with a

centerline of the vehicle and the virtual camera view provides a view from a perspective that is at least one of laterally offset from the centerline and vertically offset from the first position.

9. The method of claim 1, wherein the first position is associated with a

centerline of the vehicle and the virtual camera view provides a view from a perspective that is laterally offset from the centerline and an orientation of the virtual camera view presents a view that is angularly offset from the centerline.

10. The method of claim 1, wherein generating the virtual camera position is based on at least one of a turning angle measurement and change in angular of displacement of the vehicle.

11. A vehicle system comprising:

a wide-angle camera; and

a control device coupled to the camera, wherein the control device is configured to:

receive image data from a wide-angle camera, wherein the wide-angle camera is mounted to the vehicle in a fixed position at a first location of the vehicle, wherein the image data captures a scene from the first location; select a section of the image data from the wide-angle camera based on a vehicle control setting;

generate a virtual camera view based on the section the image data, wherein the virtual camera view presents image data of the scene from the perspective of a second location; and control output of the virtual camera view to a display of the vehicle.

12. The system of claim 11, wherein the image data includes at least a 180 degree representation of the scene.

13. The system of claim 11, wherein selecting a section includes selecting a portion of the scene laterally displaced from a center of the image data.

14. The system of claim 11, wherein the vehicle control setting is intended direction of the vehicle, and wherein the virtual camera view is generated to present a view of the scene resulting from a direction of travel of the vehicle based on a turn angle measurement.

15. The system of claim 11, wherein generating includes extracting pixel data for the section of image data and mapping extracted pixels to a modified layout for correction of wide-angle curvature.

16. The system of claim 11, wherein the virtual camera view is generated to appear from a different position and different orientation using the image data detected by the wide-angle camera.

17. The system of claim 11, wherein the first position is laterally offset from a centerline of the vehicle and the virtual camera view provides a view from a perspective of the centerline.

18. The system of claim 11, wherein the first position is associated with a

19. The system of claim 11, wherein the first position is associated with a

20. The system of claim 11 , wherein generating the virtual camera position is based on at least one of a turning angle measurement and change in angular of displacement of the vehicle.

21. A method for presentation and control of a virtual camera view for a

vehicle, the method comprising:

selecting, by the control device, a section of the image data from the wide-angle camera based on a turning angle measurement;

generating, by the control device, a virtual camera view based on the section the image data, wherein the virtual camera view presents image data of the scene from the perspective of a second location and an updated viewing direction, the second location displaced laterally from the first location; and

22. The method of claim 21, wherein the virtual camera view is updated by the control device based on at least one of vehicle displacement and changes turning angle measurement.