EP2471689A1 - Fahrhilfen-anzeigevorrichtung - Google Patents

Fahrhilfen-anzeigevorrichtung Download PDF

Info

Publication number
EP2471689A1
EP2471689A1 EP10848628A EP10848628A EP2471689A1 EP 2471689 A1 EP2471689 A1 EP 2471689A1 EP 10848628 A EP10848628 A EP 10848628A EP 10848628 A EP10848628 A EP 10848628A EP 2471689 A1 EP2471689 A1 EP 2471689A1
Authority
EP
European Patent Office
Prior art keywords
image display
correction image
display region
leftward
rightward
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP10848628A
Other languages
English (en)
French (fr)
Other versions
EP2471689A4 (de
EP2471689B1 (de
Inventor
Tadao Ichinose
Koji Takahashi
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Panasonic Intellectual Property Management Co Ltd
Original Assignee
Panasonic Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Panasonic Corp filed Critical Panasonic Corp
Publication of EP2471689A1 publication Critical patent/EP2471689A1/de
Publication of EP2471689A4 publication Critical patent/EP2471689A4/de
Application granted granted Critical
Publication of EP2471689B1 publication Critical patent/EP2471689B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/80—Geometric correction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/06—Topological mapping of higher dimensional structures onto lower dimensional surfaces
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/30—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/30—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing
    • B60R2300/306—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing using a re-scaling of images

Definitions

  • the present invention relates to a drive assist display apparatus for performing image processing to provide a driver with an image taken backwardly or forwardly from a vehicle by a vehicle-mounted camera.
  • the conventional drive assist display apparatus is required to take the image of a wider area, and to display a wider image.
  • the conventional drive assist display apparatus is designed in various ways such as, for example, taking the image with a plurality of vehicle-mounted cameras or taking the image by using a wide-angle lens, and displaying the wide image in a limited display area.
  • Patent Literature 1 the conventional drive assist display apparatus taking the image by using the wide-angle lens is disclosed by a Patent Literature 1.
  • the Patent Literature 1 discloses an image taken by using the wide-angle lens.
  • the reference sign 101-1 is regarded as indicating a crosswise straight line crossing in front of the vehicle-mounted camera on the ground surface
  • the reference sign 101-2 is regarded as indicating a vertical straight line extending from the ground surface with forming a building.
  • Patent Literature 2 the conventional drive assist display apparatus designed to perform a correcting operation of a wide-angle taken image is disclosed by a Patent Literature 2.
  • the Patent Literature 2 discloses a corrected omnidirectional image developed by a Mercator projection from an omnidirectional taken image.
  • FIGS. 59 and 60 each shows an image clipped horizontally about a range of 180 degrees from the corrected omnidirectional image shown in FIG. 58 .
  • the reference sign 102-2-1 is regarded as indicating a vertical straight line extending from the ground surface with forming a building.
  • the reference sign 102-2-2 is regarded as indicating parallel lines extending on the ground surface with crossing from the near side toward the far side of the vehicle-mounted camera, the parallel lines constituting a vertically-striped pattern of a crosswalk.
  • the reference sign 102-3-1 is regarded as indicating a crosswise straight line crossing in front of the vehicle-mounted camera on the ground surface.
  • the conventional drive assist display apparatus designed to take the image with the plurality of vehicle-mounted cameras and to display the image taken by the vehicle-mounted cameras is disclosed by a Patent Literature 3.
  • the Patent Literature 3 discloses a display image including three images respectively taken by three vehicle-mounted cameras.
  • the display image has a backward left image display region 71 for displaying the image taken by the vehicle-mounted camera mounted at an angle toward a backward left direction, a backward right image display region 72 for displaying the image taken by the vehicle-mounted camera mounted at an angle toward a backward right direction, and a backward straight image display region 73 for displaying the image taken by the vehicle-mounted camera mounted at an angle toward a backward straight direction.
  • the image regions 71 to 73 are located in a manner that the backward left image display region 71 and the backward right image display region 72 are covered by the top side of the backward straight image display region 73.
  • Patent Literature 2 discloses an image correction method.
  • the image correction method By the image correction method, a vertical subject such as, for example, the vertical straight line 102-2-1 is displayed straight vertically on the display image. In other word, the image correction method improves visibility of the vertical subject.
  • the crosswise straight line 102-3-1 is curved in similar to the line 101-1.
  • the straight line 102-3-1 is displayed in a horizontal direction in the vicinity of a front face portion, and displayed in a nearly vertical direction in left and right far portions.
  • an image of an edge portion of the vehicle such as, for example, a bumper is taken on the downside of the image to make the driver recognize relationships between the vehicle and the circumference of the vehicle for the purpose of a drive assist
  • the edge portion of the vehicle is curved beyond recognition on the display image.
  • the lines 102-2-2 constituting the vertically-striped pattern of the crosswalk with extending a front-back direction from the position of the camera are especially near to the vehicle-mounted camera.
  • the front-back direction straight lines among the lines 102-2-2 are obviously curved and spread toward the left or right side on the display image. This leads to the fact that the disclosure of the Patent Literature 2 encounters such a problem of difficulty to make the driver recognize the width of the vehicle and a travelling direction of the vehicle for the purpose of the drive assist.
  • the straight line is not obviously carved on the taken image without the image correction method disclosed in the Patent Literature 2.
  • the example of the taken image is disclosed in the Patent Literature 3.
  • the eye line of the driver is forced to be moved from the right side of the backward left image display region 71 to an image appearance position Q on the left side of the backward straight image display region 73 in the reverse direction of movement of the moving object T on the backward straight image display region 73.
  • the disclosure of the Patent Literature 3 is not designed to guide the movement of the eye line to the backward straight image display region 73 by displaying the auxiliary image M'.
  • the disclosure of the Patent Literature 3 is not designed to guide to the complete image appearance position Q and not designed to correct the direction and length of the movement of the eye line to the image appearance position Q. This results in the fact that the disclosure of the Patent Literature 3 encounters such a problem that the driver needs time to consider about the positional relationships. As a result, there is a possibility that the driver falls behind in vehicle handling.
  • the disclosure of the Patent Literature 3 has a difficulty to take and display the image of the edge portion of the vehicle such as, for example, the bumper. This results from the fact that there is a large angular difference between the edge portion of the vehicle and the straight backward direction.
  • a subject of the present invention to provide a drive assist display apparatus which can realize, from imaging to displaying the front, left, right and down wide area of the backward or frontward of the vehicle, displaying in high visibility of a form of an imaged subject in accordance with purposes, improving visibility of directional and positional relationships between the imaged subject and the vehicle, and facilitating the movement of the eye line targeting to the subject on the display.
  • a drive assist display apparatus comprises: one or more imaging sections operable to take an image of a front, left, right and down wide area of a backward or frontward of a vehicle from an imaging position; an image processing section operable to correct the image taken by the imaging section, operable to divide the corrected image, operable to locate the divided images respectively on arbitrary correction image display regions, and operable to combine mask pictures with the located images on arbitrary display positions; an image display section for displaying the image processed by the image processing section, wherein the image processing section is operable to create: a forward correction image corrected in a manner that, when an image of square cells formed by horizontal and vertical lines with respect to a horizontal ground surface on a forward plane is taken, the square cells on the forward plane are displayed as squares or rectangles formed by horizontal and vertical lines with each having an area equal to one another on a display image, the forward plane being in perpendicular relationship with a forward virtual sight line, the forward virtual sight line being in parallel relationship
  • the whole taken image of the front, left, right and down wide area of the backward or frontward of the vehicle is simultaneously displayed.
  • each correction image display region is corrected with reducing the deformation of the image of the subject on the plane, the plane being in perpendicular to the leftward, rightward and downward directions with respect to the vehicle front-back direction.
  • the display regions are displayed on directional relationships similar to the actuality directional relationships.
  • straight line subjects are respectively displayed as the straight lines in the correction image display regions.
  • straight line subjects perpendicular to the ground surface in forward, leftward and rightward directions are respectively displayed as the vertical straight lines in the correction image display regions.
  • straight line subjects parallel to the ground surface in forward and downward directions and parallel to the vehicle left-right direction are respectively displayed as the horizontal straight lines in the correction image display regions.
  • vehicle front-back direction on the ground surface is indicated by forms of the left and right sides of the downward correction image display region and the bottom side of the leftward and rightward correction image display regions.
  • boundaries of the display regions adjacent to one another are respectively displayed.
  • the drive assist display apparatus may adjust the inclined angle of the leftward and rightward virtual sight lines, or adjusting the degree of expanding horizontally the square cells on the leftward and rightward planes so that the straight line corresponding to the bottom side of the leftward or rightward correction image display region and being parallel relationship with the vehicle front-back direction on the ground surface is displayed more horizontal on the leftward or rightward correction image display region than on the forward correction image display region.
  • straight lines displayed on the leftward or rightward correction image display region toward the vehicle front-back direction except having a height equal to an imaging position is reduced in inclination with retained.
  • the drive assist display apparatus may display in a manner that, when the image of the square cells on the leftward plane is taken, the square cells are appeared as parallelogram cells each having an area equal to one another and inclined down toward left gradually at a degree that straight lines parallel to the ground surface are visible in the leftward correction image display region, and that, when the image of the square cells on the rightward plane is taken, the square cells are appeared as parallelogram cells each having an area equal to one another and inclined down toward right gradually at a degree that the straight lines parallel to the ground surface is visible in the rightward correction image display region.
  • a horizon is displayed horizontally in the forward correction image display region, inclined down toward left in the leftward correction image display region, and inclined down toward right in the rightward correction image display region.
  • the drive assist display apparatus may locate the correction image display regions in a manner that the bottom side of the leftward correction image display region is adjacent to the left side of the downward correction image display region, and the bottom side of the rightward correction image display region is adjacent to the right side of the downward correction image display region, fit the taken images to the respective correction image display regions in a manner that the same images are appeared in the vicinity of the sides of the respective correction image display regions adjacent to each other, and draw the mask pictures of lines or dotted lines in arbitrary color on the sides of the respective correction image display regions adjacent to each other.
  • the subject moving between the leftward or rightward direction and downward direction is displayed without discontinuity.
  • the drive assist display apparatus may display in a manner that, when the image of the square cells on the leftward plane is taken, the square cells are appeared as parallelogram cells each having an area equal to one another and inclined down toward left gradually at a degree that lines parallel to the ground surface are visible in the leftward correction image display region, and that, when the image of the square cells on the rightward plane is taken, the square cells are appeared as parallelogram cells each having an area equal to one another and inclined down toward right gradually at a degree that lines parallel to the ground surface are visible in the rightward correction image display region.
  • the horizon is displayed horizontally in the forward correction image display region, displayed inclined down toward left in the leftward correction image display region, and displayed inclined down toward right in the rightward correction image display region.
  • the drive assist display apparatus to be operated with: a vehicle front-back direction straight line being in parallel relationship with the vehicle front-back direction on the ground surface, and corresponding to the left or right side of the downward correction image display region or corresponding to the bottom side of the leftward or rightward correction image display region; a vehicle left-right direction straight line being in parallel relationship with the vehicle left-right direction on the ground surface, and corresponding to the top side of the downward correction image display region or corresponding to the bottom side of the forward correction image display region; an intersection point between the vehicle front-back direction straight line and the vehicle left-right direction straight line; a circle being an innermost of a vehicle travelling way on the ground surface, the vehicle travelling way travelled thereon the vehicle at full lock; and a tangent line tangent to the circle with passing on an image origin being under the imaging position on the ground surface, wherein the intersection point may be in the vicinity of the tangent line.
  • the circle being the innermost of the vehicle travelling way is located in mainly the leftward or rightward correction image display region under the relationships between the forward correction image display region and the leftward or rightward correction image display region, and targeted the boundaries between the forward correction image display region and the leftward and rightward correction image display regions.
  • the front, left, right and down wide area is displayed without a missing part; the whole image is displayed similar in form and location to the real sight; and the horizon is displayed horizontally across three display regions. This leads to the fact that the whole image is improved visibility as subjects from an adjacent place to a distant place are overlooked from a place.
  • the subject moving along the front-back direction or the distantly-positioned left-right direction is displayed without discontinuity and misalignment. This results in the fact that the movement of the eye line useful for the drive assist toward the front-back direction or the left-right direction is facilitated.
  • the straight line subjects are respectively displayed as the straight lines in display regions, thereby resulting in improving visibility of directionality.
  • the straight lines perpendicular to the ground surface is displayed as the vertical lines in the image useful for watching a left-right direction, thereby resulting in improving visibility of up and down relationships useful for the drive assist between the ground surface and an obstacles.
  • the straight lines directed along an crosswise direction of the vehicle is displayed as the horizontal lines in the image useful for watching an up-down direction, thereby resulting in reducing the deformation of the image of the edge portion of the vehicle such as, for example, the bumper and improving visibility of relationships of directionality of the vehicle with respect to the crosswise direction.
  • the display regions are matched in sides and forms, thereby resulting in improving visibility of the vehicle front-back direction useful for the drive assist.
  • the vehicle front-back direction straight lines on the ground surface are displayed on the leftward or rightward correction image display region as inclined closely to the horizon with increasing distance from the vehicle, thereby resulting in maintaining visibility of the depth of the vehicle front-back direction and decreasing the inclination of the image of the subject in the vehicle front-back direction.
  • the horizon is appeared as arcuate shape on the display image. This leads to the fact that the forward, leftward and rightward correction image display regions are respectively recognized to display the forward, leftward and rightward directions.
  • the image is displayed without discontinuity and misalignment from the left-right direction to the down direction, thereby resulting in improving visibility of the relationships in the whole image, and facilitating the movement of the eye line targeting to the subject moving in the whole display image.
  • the forward, leftward and rightward correction image display regions are respectively recognized to display the forward, leftward and rightward directions without influence of visibility of the downward correction image display region under the state that the downward correction image display region is in contact with each leftward and rightward correction image display region.
  • the straight line formed mask pictures indicating the boundary between the leftward or rightward correction image display region and the forward correction image display region acts as a guideline for judging whether or not the vehicle at full lock is in contact with the solid subject, thereby resulting in improving visibility of the relationships between the vehicle, for example, moved into the garage and the solid subject.
  • FIG. 1 is a configuration diagram of the first embodiment of the present invention.
  • the reference sign 1 indicates a vehicle mounting a drive assist display apparatus
  • the reference sign 2 indicates a vehicle-mounted camera having a wide-angle lens for taking an image of a wide area
  • the reference sign 3 indicates an image processing section accommodated in the vehicle-mounted camera 2.
  • the image processing section 3 is operable to correct the image taken by the vehicle-mounted camera 2, operable to divide the corrected image, operable to locate the divided images respectively on arbitrary correction image display regions, and operable to draw mask pictures on arbitrary display positions.
  • the reference sign 4 indicates a monitor for displaying the image processed by the image processing section 3.
  • the image processing section 3 may be accommodated in the monitor 4 as shown in FIG. 2 .
  • the image processing section 3 may also be accommodated in an image processing apparatus 5 as shown in FIG. 3 .
  • the vehicle-mounted camera 2 may be replaced by a plurality of vehicle-mounted cameras 2-1 to 2-n each having a non wide-angle lens.
  • the vehicle-mounted cameras 2-1 to 2-n are, therefore, mounted adjacently to each other to make the declinations be inconspicuous on the image to be displayed.
  • the vehicle-mounted camera 2 While there has been shown in FIGS. 1 to 3 about the fact that the vehicle-mounted camera 2 is mounted on the rear side of the vehicle, the vehicle-mounted camera 2 may be mounted on the front side of the vehicle. In the first and following embodiments of the present invention, the vehicle-mounted camera 2 is described as mounted on the rear side of the vehicle.
  • FIGS. 4 and 5 each shows a detailed mounted state of the vehicle-mounted camera 2 with respect to the vehicle 1.
  • FIG. 4 The overhead view of the vehicle 1 is shown in FIG. 4 .
  • the ground surface is assumed to be horizontal.
  • the reference sign 10 indicates a rear bumper of the vehicle 1
  • the reference sign y indicates a y-axis representing a front-back direction of the vehicle 1.
  • the y-axis is in parallel relationship with the ground surface, and passes through a center of a left-right direction of the vehicle 1.
  • the reference sign x indicates an x-axis representing the left-right direction of the vehicle 1.
  • the x-axis is in parallel relationship with the ground surface, and is tangent to the edge of the bumper, i.e., the rear edge portion of the vehicle 1.
  • the reference sign 6 indicates an origin of the vehicle 1 hereinafter referred to as "vehicle origin", the vehicle origin 6 being an intersection point of the y-axis with the x-axis.
  • the vehicle-mounted camera 2 is located on the y-axis.
  • the side view of the vehicle 1 is shown in FIG. 5 .
  • the reference sign y indicates the y-axis shown in FIG. 4 .
  • the reference sign z indicates a z-axis being in perpendicular relationship with the ground surface, the z-axis being tangent to the edge of the bumper, i.e., the rear edge portion of the vehicle 1 across the vehicle origin 6 shown in FIG. 4 .
  • the reference sign 7 indicates an optical axis of the lens constituting the vehicle-mounted camera 2, the optical axis 7 being assumed to be inclined with respect to the y-axis at an angle of 40 degrees.
  • the reference sign 30 indicates an imaging origin representing an imaging position, the imaging position being assumed to be at -0.1m in the direction of the y-axis and at 0.9m in the direction of the z-axis from the vehicle origin 6.
  • the rear edge of the bumper i.e., the rear edge portion of the vehicle 1 is assumed to be at 0.6m in the direction of the z-axis.
  • angles and positional relationships are mentioned as an example. In actuality, the angles and positional relationships are varied in response to a type of vehicle.
  • FIGS. 6 , 7 , 9 and 13 each shows a subject imaged by the vehicle-mounted camera 2 on the basis of the vehicle origin 6.
  • the reference sign 21-1 indicates a forward virtual sight line being in parallel relationship with the y-axis.
  • the reference sign 21-2 indicates a forward plane being in perpendicular relationship with the forward virtual sight line 21-1, the forward plane 21-2 being at 1.2m in the direction of the y-axis.
  • the reference sign 21 indicates cells each aligned with one another at equal intervals of 0.5m on the illustrated part of the forward plane 21-2. Each of the cells 21 is formed a square having a 0.5m side being in parallel relationship with the ground surface, and a 0.5m side being in perpendicular relationship with the ground surface.
  • the reference sign 21G indicates a vehicle left-right direction straight line on the ground surface, the vehicle left-right direction straight line 21G being included in the cells 21 on the forward plane.
  • the reference sign 21L indicates a vertical line being at 1.5m in the direction of the x-axis from the vehicle origin 6, the vertical line 21L being included in the cells 21 on the forward plane.
  • the reference sign 21R indicates a vertical line being at -1.5m in the direction of the x-axis from the vehicle origin 6, the vertical line 21R being included in the cells 21 on the forward plane.
  • the reference sign 21G indicates the vehicle left-right direction straight line 21G shown in FIG. 6 .
  • the reference sign 22-1-1 indicates a downward virtual sight line inclined with respect to the y-axis at a downward angle of 90 degrees.
  • the reference sign 22-1-2 indicates a downward plane being in perpendicular relationship with the downward virtual sight line 22-1-1 on the ground surface.
  • the reference sign 22-1 indicates cells each aligned with one another at equal intervals of 0.5m on the illustrated part of the downward plane 22-1-2.
  • Each of the cells 22-1 is formed a square having a 0.5m side being in parallel relationship with the vehicle left-right direction straight line 21G, and a 0.5m side being in perpendicular relationship with the vehicle left-right direction straight line 21G.
  • the reference sign 22-1 further indicates cells being on the downward plane.
  • Each of the cells formed by the vehicle left-right direction straight line 21G and a straight line being in parallel relationship with the x-axis passing just under the vehicle-mounted camera 2 on the ground surface.
  • the reference signs 21L and 21R respectively indicate the vertical lines 21L and 21R shown in FIG. 6 .
  • the reference sign 23-1-1L indicates a leftward virtual sight line inclined with respect to the y-axis at a leftward angle of 45 degrees.
  • the reference sign 23-1-2L indicates a leftward plane being in perpendicular relationship with the leftward virtual sight line 23-1-1L, and including a point at 1.5m in the direction of the x-axis and 1.2m in the direction of the y-axis from the vehicle origin 6.
  • the reference sign 23-1L indicates cells each aligned with one another at equal intervals of 0.5m on the illustrated part of the leftward plane 23-1-2L including the vertical line 21L.
  • Each of the cells 23-1L is formed with a square having a 0.5m side being in parallel relationship with the ground surface, and a 0.5m side being in perpendicular relationship with the ground surface.
  • the reference sign 23-1-1R indicates a rightward virtual sight line inclined with respect to the y-axis at a rightward angle of 45 degrees.
  • the reference sign 23-1-2R indicates a rightward plane being in perpendicular relationship with the rightward virtual sight line 23-1-1R, and including a point at -1.5m in the direction of the x-axis and 1.2m in the direction of the y-axis from the vehicle origin 6.
  • the reference sign 23-1R indicates cells each aligned with one another at equal intervals of 0.5m on the illustrated part of the rightward plane 23-1-2R including the vertical line 21R.
  • Each of the cell 23-1R is formed with a square having a 0.5m side being in parallel relationship with the ground surface, and a 0.5m side being in perpendicular relationship with the ground surface.
  • the reference sign 24 indicates 9 straight lines respectively passing at -0.1m and points from 0m to 3.5m at 0.5m intervals in the direction of the y-axis on the ground surface, and each having an infinite length in parallel relationship with the x-axis.
  • the reference sign 24 further indicates 13 straight lines respectively passing at 10m, -10m, and points from -2.5m to 2.5m at 0.5m intervals in the direction of the x-axis on the ground surface, and each having an infinite length toward the positive direction of the y-axis in parallel relationship with the y-axis.
  • the reference sign 24-15L indicates a vehicle front-back direction straight line included in the straight lines 24 on the ground surface at 1.5m in the direction of the x-axis.
  • the reference sign 24-15R indicates a vehicle front-back direction straight line included in the straight lines 24 on the ground surface at -1.5m in the direction of the x-axis.
  • FIG. 15 shows a taken image before operated by an image processing.
  • the reference sign 9 indicates an image circle assumed to be taken with an angle at 90 degrees around the optical axis of the lens.
  • FIGS. 16 to 25 each shows a display image displayed by the monitor 4.
  • the display image is obtained from the taken image operated by the same operation of the image processing section 3.
  • the taken image of the backward of the vehicle should be displayed as a mirrored image horizontally flipped the whole.
  • the taken image will be described as not horizontally flipped the whole for better understanding of the horizontal relationships between the taken image and the display image.
  • FIG. 16 shows an image arrangement of the display image operated by of the image processing section 3 for a display area of the monitor 4.
  • the reference sign 41-1 indicates a forward correction image display region allocated in the upside of the center of the display area.
  • the reference sign 41-2 indicates a downward correction image display region allocated below the forward correction image display region 41-1.
  • the reference sign 41-3L indicates a leftward correction image display region allocated to the left of the forward correction image display region 41-1.
  • the reference sign 41-3R indicates a rightward correction image display region allocated to the right of the forward correction image display region 41-1.
  • the reference sign 41-4 indicates a dotted line mask picture constantly displayed as a dotted line in arbitrary color at a display position of adjacent sides of the forward and downward correction image display regions 41-1 and 41-2.
  • the reference sign 41-5L indicates a dotted line mask picture constantly displayed as a dotted line in arbitrary color at a display position of adjacent sides of the forward and leftward correction image display regions 41-1 and 41-3L.
  • the reference sign 41-5R indicates a dotted line mask picture constantly displayed as a dotted line in arbitrary color at a display position of adjacent sides of the forward and rightward correction image display regions 41-1 and 41-3R.
  • the reference signs 41-6L, 41-6R, 41-7L and 41-7R indicate plane mask pictures displayed on arbitrary color and allocated the display regions except the forward, downward, leftward and rightward correction image display regions 41-1, 41-2, 41-3L and 41-3R.
  • FIG. 17 shows the operated image of the taken image shown in FIG. 15 under the allocation shown in FIG. 16 .
  • an image displayed on the forward correction image display region 41-1 is corrected in a manner that each of the cell 21 is displayed as a square formed by horizontal and vertical lines and having an area equal to one another.
  • the cells may be corrected to be displayed as rectangles.
  • the forward correction image display region 41-1 has a bottom side coincided with the vehicle left-right direction straight line 21G, a left side coincided with the vertical line 21L, and a right side coincided with the vertical line 21R.
  • an image displayed on the downward correction image display region 41-2 is corrected in a manner that each of the square cells is displayed as a square formed by the horizontal and vertical lines and having an area equal to one another.
  • the cells may be corrected to be displayed as rectangles.
  • the downward correction image display region 41-2 has a top side coincided with the vehicle left-right direction straight line 21G, located adjacent to the bottom side of the forward correction image display region 41-1, and coincided in the position on the x-axis direction with the vehicle left-right direction straight line 21G corresponding to the bottom side of the forward correction image display region 41-1.
  • the downward correction image display region 41-2 has a left side coincided with the vehicle front-back direction straight line 24-15L positioned at the same distance in the x-axis direction with the vertical line 21L.
  • the downward correction image display region 41-2 has a right side coincided with the vehicle front-back direction straight line 24-15R positioned at the same distance in the x-axis direction with the vertical line 21L.
  • the bottom side of the downward correction image display region 41-2 is coincided with the position at -0.1m in the direction of the y-axis, the bottom side of the downward correction image display region 41-2 may be coincided with arbitrary position in the direction of the y-axis.
  • an image displayed on the leftward correction image display region 41-3L is corrected in a manner that each of the cells is displayed as a vertically long rectangle formed by the horizontal and vertical lines and having an area equal to one another.
  • an image displayed on the leftward correction image display region 41-3L is corrected in a manner that rectangles are displayed on the whole of the leftward correction image display region 41-3L.
  • the cells may be corrected to be displayed as squares or horizontally long rectangles.
  • the leftward correction image display region 41-3L has a right side coincided with the vertical line 21L, located adjacent to the left side of the forward correction image display region 41-1, and coincided in the position on the z-axis direction with the vertical line 21L corresponding to the left side of the forward correction image display region 41-1.
  • the leftward correction image display region 41-3L has a bottom side coincided with the vehicle front-back direction straight line 24-15L positioned at the same distance in the x-axis direction with the vertical line 21L.
  • the leftward correction image display region 41-3L has a left-top side inclined down toward left in the left-top area and corresponding to the image circle 9.
  • the left-top area over the left-top side has no taken image, thereby the left-top side being inclined as with a side of a plane mask picture 41-7L. If the left-top area has the taken image depending on the mounted state or the characteristic of the lens, the left-top side is not needed.
  • the left side of the leftward correction image display region 41-3L is coincided with the position at -0.1m in the direction of the y-axis
  • the left side of the leftward correction image display region 41-3L may be coincided with arbitrary position in the direction of the y-axis.
  • an image displayed on the rightward correction image display region 41-3R is corrected in a manner that each of the cells is displayed as a vertically long rectangle formed by the horizontal and vertical lines and having an area equal to one another.
  • an image displayed on the rightward correction image display region 41-3R is corrected in a manner that rectangles are displayed on the whole of the rightward correction image display region 41-3R.
  • the cells may be corrected to be displayed as squares or horizontally long rectangles.
  • the rightward correction image display region 41-3R has a left side coincided with the vertical line 21R, located adjacent to the right side of the forward correction image display region 41-1, and coincided in the position on the z-axis direction with the vertical line 21R corresponding to the right side of the forward correction image display region 41-1.
  • the rightward correction image display region 41-3R has a bottom side coincided with the vehicle front-back direction straight line 24-15R positioned at the same distance in the x-axis direction with the vertical line 21R.
  • the rightward correction image display region 41-3R has a right-top side inclined down toward right in the right-top area and corresponding to the image circle 9.
  • the right-top area over the right-top side has no taken image, thereby the right-top side being inclined as with a side of a plane mask picture 41-7R. If the right-top area has the taken image depending on the mounted state or the characteristic of the lens, the left-top side is not needed.
  • the right side of the rightward correction image display region 41-3R is coincided with the position at -0.1m in the direction of the y-axis
  • the right side of the rightward correction image display region 41-3R may be coincided with arbitrary position in the direction of the y-axis.
  • the display image is shown as FIG. 18 .
  • the image is displayed as following.
  • the display image is shown as FIG. 19 .
  • the display image is shown as FIG. 20 .
  • the display image is shown as FIG. 20 .
  • the display image is shown as FIG. 22 .
  • the display image is shown as FIG. 23 .
  • the display image is shown as FIG. 24 .
  • each cuboids 12-1, 12-2 and 12-3 having a width of 0.5m, a depth of 0.1m and a height of 0.1m; one of the pairs of cuboids 12-1, 12-2 and 12-3 being at 0.5m in the direction of the x-axis; other of the pairs of cuboids 12-1, 12-2 and 12-3 being at -0.5m in the direction of the x-axis; the pair of cuboids 12-1 being at 1.5m in the direction of the y-axis; the pair of cuboids 12-2 being at 1.1m in the direction of the y-axis; and the pair of cuboids 12-3 being at 0.5m in the direction of the y-axis, the image is displayed as FIG. 25 .
  • the cells 21 on the forward plane, the cells 22-1 on the downward plane, the cells 23-1L on the leftward plane and the cells 23-1R on the rightward plane is displayed as the squares or the rectangles.
  • the forward correction image display region 41-1, the downward correction image display region 41-2, the leftward correction image display region 41-3L and the rightward correction image display region 41-3R are displayed on relationships similar to the real sight.
  • All of the straight lines are displayed as the straight lines.
  • the straight lines perpendicular to the ground surface are displayed as the vertical lines in the image displayed on the forward correction image display region 41-1, the leftward correction image display region 41-3L and the rightward correction image display region 41-3R useful for watching a left-right direction.
  • the straight lines and the cuboids in the vehicle left-right direction parallel to the ground surface are displayed as the horizontal lines in the image displayed on the forward correction image display region 41-1 and the downward correction image display region 41-2 useful for watching an up-down direction.
  • the infinite straight line being parallel to the x-axis and being at -0.1m on the y-axis in the straight lines 24 on the ground surface is displayed, and the intersection point of the infinite straight line with the horizon 8 is displayed, thereby resulting in the fact that the wide angle area within a horizontal angle of 180 degrees from the position of the vehicle-mounted camera 2 is displayed.
  • the straight line being parallel to the x-axis, being at -0.1m on the y-axis, and forming the cells 22-1 on the downward plane is displayed, thereby resulting in the fact that the position under the vehicle-mounted camera 2 is displayed.
  • the vehicle front-back direction straight lines 24-15L and 24-15R are displayed on the downward correction image display region 41-2, the leftward correction image display region 41-3L and the rightward correction image display region 41-3R are displayed, thereby resulting in the fact that the areas in the vicinity of the vehicle front-back direction straight lines 24-15L and 24-15R are displayed without a missing part.
  • FIGS. 19 to 20 each shows an example of the display image when the cuboid 11 moves in the back of the vehicle 1 from left to right as similar to the moving object T shown in FIG. 61 , thereby resulting in the fact that the display position of the cuboid 11 moves from the leftward correction image display region 41-3L to the forward correction image display region 41-1.
  • the sides of the cuboid 11 are polygonal at the boundary between the leftward correction image display region 41-3L and the forward correction image display region 41-1, the display position of the cuboid 11 moves without misalignment of the sides, and therefore the eye line continuously moves from left to right.
  • the relationship between the rightward correction image display region 41-3R and the forward correction image display region 41-1 is similar to the relationship between the leftward correction image display region 41-3L and the forward correction image display region 41-1.
  • FIGS. 22 to 24 each shows an example of the display image when the cuboid 11 being back and lateral to the vehicle 1 approaches the vehicle 1 in the vehicle front-back direction, or where the vehicle 1 approaches the cuboid 11 being back and lateral to the vehicle 1 in the vehicle front-back direction.
  • the display position of the cuboid 11 moves without misalignment as similar to the example shown in FIGS. 19 to 21 , and therefore the eye line continuously moves from right to left.
  • the image of the cuboid 11 grows in height in response to the shortened distance between the cuboid 11 and the vehicle 1, thereby maintaining visibility of the distance.
  • the direction of the movement of the cuboid 11 is easily recognized on the basis of the inclination of the bottom side of the leftward correction image display region 41-3L.
  • FIG. 25 shows an example of the display image when the vehicle 1 backwardly approaches barrier curbs for the purpose of parking.
  • the display image represents a case that the distance between the cuboids 12-1 to 12-3 and the vehicle 1 is shortened.
  • the lines are polygonal, the display position of the cuboids 12-1 to 12-3 move without misalignment of the lines, and therefore the eye line continuously moves from up to down as each shown in FIGS. 19 to 21 .
  • the directions of the movement of the cuboids 12-1 to 12-3 are easily recognized on the basis of the inclination of the left and right sides of the downward correction image display region 41-2.
  • the forward virtual sight line 21-1, the downward virtual sight line 21-1-1, the leftward virtual sight line 21-1-1L and the rightward virtual sight line 21-1-1R may respectively pass through any positions.
  • the forward plane 21-2, the downward plane 21-1-2, the leftward plane 23-1-2L and the rightward plane 23-1-2R may respectively pass through any positions under conditions of maintaining the angle with respect to the virtual sight lines and conditions of the cells 21, the cells 21-1, the cells 23-1L and the cells 23-1R respectively formed squares on the forward plane 21-2, the downward plane 22-1-2, the leftward plane 23-1-2L and the rightward plane 23-1-2R.
  • the directions of the leftward virtual sight line 21-1-1L and the rightward virtual sight line 21-1-1R are not respectively leftward and rightward inclined at an angle of 0 or 90 degrees with respect to the y-axis.
  • the cells corresponding to the cells 23-1L on the leftward plane 23-1-2L and the cells 23-1R on the rightward plane 23-1-2R as shown in FIG. 9 are respectively appeared as the cells 23-3L on the leftward plane and the cells 23-1R on the rightward plane as shown in FIG. 11 .
  • the reference signs 41-1, 41-2 and 41-4 are respectively equal to the reference signs 41-1, 41-2 and 41-4 in FIG. 16 .
  • the reference sign 42-3L indicates a leftward correction image display region displayed therein the corrected image corresponding to the cells 23-3L on the leftward plane.
  • the reference sign 42-3R indicates a rightward correction image display region displayed therein the corrected image corresponding to the cells 23-3R on the rightward plane.
  • the reference sign 42-5L indicates a dotted line mask picture equivalent to the dotted line mask picture 41-5L.
  • the reference sign 42-5R indicates a dotted line mask picture equivalent to the dotted line mask picture 41-5R.
  • the reference sign 42-6L and 42-6R respectively indicates plane mask pictures equivalent to the plane line mask pictures 41-6L and 41-6R.
  • the taken picture shown in FIG. 26 is displayed as shown in FIG. 28 .
  • the leftward correction image display region 42-3L displays therein the image to the left-top
  • the rightward correction image display region 42-3R displays therein the image to the right-top. Therefore, the leftward correction image display region 42-3L has no left-top area such as the area displaying the plane mask picture 41-7L
  • the rightward correction image display region 42-3R has no right-top area such as the area displaying the plane mask picture 41-7R.
  • FIG. 29 shows the display image when the straight lines 24 on the ground surface shown in FIG. 13 , the horizon 8 and the bumper 10 of the vehicle 1 are simultaneously imaged.
  • leftward and rightward display regions are narrow.
  • FIG. 29 it is to be understood that the leftward and rightward display regions shown in FIG. 29 are clearly narrower than the leftward and rightward display regions shown in FIG. 18 .
  • the image corresponding to the direction of a horizontal eye line is corrected to the image in the direction of a front eye line, thereby resulting in the fact that the image is unnaturally formed as expanding horizontally.
  • the cells corresponding to the cells 23-1L on the leftward plane 23-1-2L and the cells 23-1R on the rightward plane 23-1-2R as shown in FIG. 9 are respectively appeared as the cells 23-4L on the leftward plane and the cells 23-4R on the rightward plane as shown in FIG. 12 .
  • the reference signs 41-1, 41-2 and 41-4 are respectively equal to the reference signs 41-1, 41-2 and 41-4 in FIG. 16 .
  • the reference sign 43-3L indicates a leftward correction image display region displayed therein the corrected image corresponding to the cells 23-4L on the leftward plane.
  • the reference sign 43-3R indicates a rightward correction image display region displayed therein the corrected image corresponding to the cells 23-4R on the rightward plane.
  • the reference sign 43-5L indicates a dotted line mask picture equivalent to the dotted line mask picture 41-5L.
  • the reference sign 43-5R indicates a dotted line mask picture equivalent to the dotted line mask picture 41-5R.
  • the reference sign 43-6L, 43-6R, 43-7L and 43-7R respectively indicates plane mask pictures equivalent to the plane mask pictures 41-6L, 41-6R, 41-7L and 41-7R.
  • the taken picture shown in FIG. 30 is displayed as shown in FIG. 32 .
  • FIG. 33 shows the display image when the straight lines 24 on the ground surface shown in FIG. 13 , the horizon 8 and the bumper 10 of the vehicle 1 are simultaneously imaged.
  • the direction of the downward virtual sight line 22-1-1 is not downward inclined at an angle of 0 degree with respect to the y-axis.
  • the front, left, right and down wide area is displayed without a missing part; the whole image is displayed similar in form and location to the real sight; and the horizon is displayed horizontally across three display regions. This leads to the fact that the whole image is improved visibility as subjects from an adjacent place to a distant place are overlooked from a place.
  • the subject moving along the front-back direction or the distantly-positioned left-right direction is displayed without discontinuity and misalignment. This results in the fact that the movement of the eye line useful for the drive assist toward the front-back direction or the left-right direction is facilitated.
  • the straight line subjects are respectively displayed as the straight lines in the display regions, thereby resulting in improving visibility of directionality.
  • the straight lines perpendicular to the ground surface is displayed as the vertical lines in the image useful for watching a left-right direction, thereby resulting in improving visibility of up and down relationships useful for the drive assist between the ground surface and an obstacles.
  • the straight lines directed along an crosswise direction of the vehicle is displayed as the horizontal lines in the image useful for watching an up-down direction, thereby resulting in reducing the deformation of the image of the edge portion of the vehicle such as, for example, the bumper and improving visibility of relationships of directionality of the vehicle with respect to the crosswise direction.
  • the display regions are matched in sides and forms, thereby resulting in improving visibility of the vehicle front-back direction useful for the drive assist.
  • FIG. 37 shows the display image of the second embodiment of the present invention, which adjusted the leftward and rightward correction image display regions of the first embodiment of the present invention.
  • the reference sign 23-2-1L indicates a leftward virtual sight line adjusted from the leftward virtual sight line 23-1-1L inclined with respect to the y-axis at a leftward angle of 45 degrees as inclined with respect to the y-axis at a leftward angle of 60 degrees.
  • the reference sign 23-2-2L indicates a leftward plane being in perpendicular relationship with the leftward virtual sight line 23-2-1L, and including a point at 1.5m in the direction of the x-axis and 1.2m in the direction of the y-axis as similar to the leftward plane 23-1-2L.
  • the reference sign 23-2-1R indicates a rightward virtual sight line adjusted from the rightward virtual sight line 23-1-1R inclined with respect to the y-axis at a rightward angle of 45 degrees as inclined with respect to the y-axis at a rightward angle of 60 degrees.
  • the reference sign 23-2-2R indicates a rightward plane being in perpendicular relationship with the rightward virtual sight line 23-2-1R, and including a point at -1.5m in the direction of the x-axis and 1.2m in the direction of the y-axis as similar to the rightward plane 23-1-2R.
  • the reference sign 23-2L indicates cells on the illustrated part of the leftward plane 23-2-2L, the cells 23-2L equivalent to the cells 23-1L on the leftward plane.
  • the reference sign 23-2R indicates cells on the illustrated part of the rightward plane 23-2-2R, the cells 23-2R equivalent to the cells 23-1R on the rightward plane.
  • the reference signs 41-1, 41-2 and 41-4 are respectively equal to the reference signs 41-1, 41-2 and 41-4 in FIG. 16 .
  • the reference sign 44-3L indicates a leftward correction image display region displayed therein the corrected image corresponding to the cells 23-2L on the leftward plane.
  • the reference sign 44-3R indicates a rightward correction image display region displayed therein the corrected image corresponding to the cells 23-3R on the rightward plane.
  • the reference sign 44-5L indicates a dotted line mask picture equivalent to the dotted line mask picture 41-5L.
  • the reference sign 44-5R indicates a dotted line mask picture equivalent to the dotted line mask picture 41-5R.
  • the reference sign 44-6L, 44-6R, 44-7L and 44-7R respectively indicates plane mask pictures equivalent to the plane mask pictures 41-6L, 41-6R, 41-7L and 41-7R.
  • the taken picture shown in FIG. 34 is displayed as shown in FIG. 36 .
  • FIG. 37 shows the display image when the straight lines 24 on the ground surface shown in FIG. 13 , the horizon 8 and the bumper 10 of the vehicle 1 are simultaneously imaged.
  • the leftward correction image display region 44-3L has a bottom side coincided with the vehicle front-back direction straight line 24-15L, and inclined more gradual than the bottom side of the leftward correction image display region 41-3L.
  • the rightward correction image display region 44-3R has a bottom side coincided with the vehicle front-back direction straight line 24-15R, and inclined more gradual than the bottom side of the rightward correction image display region 41-3R.
  • the vehicle front-back direction straight line is inclined gradually with respect to the forward correction image display region without changing the inclination of the horizon 8 in the display image.
  • FIG. 39 shows the display image under the imaging condition similar to the imaging condition of the display image shown in FIG. 24 concerned with the first embodiment of the present invention.
  • the cuboid 11 shown in FIG. 24 is a trapezoid leftward opened in form
  • the cuboid 11 shown in FIG. 39 is resembled to a rectangle in form with remaining the inclination in the vehicle front-back direction.
  • the relationship of the inclination of the vehicle front-back direction straight line with respect to the display image is adjustable, without adjusting the angles of the leftward virtual sight line and the rightward virtual sight line, by expanding horizontally the leftward and rightward correction image display regions 41-3L and 41-3R, and expanding horizontally the corrected images displayed on the respective expanded leftward and rightward correction image display regions 41-3L and 41-3R with maintaining the state that the corrected image of each cells on the planed is formed a rectangle having the same area with one another.
  • the relationship of the inclination of the vehicle front-back direction straight line with respect to the display image is adjustable by both of adjusting the angles of the leftward virtual sight line and the rightward virtual sight line and expanding the correction image display regions toward the leftward and rightward direction.
  • the vehicle front-back direction straight lines on the ground surface are displayed on the leftward or rightward correction image display region as inclined closely to the horizon with increasing distance from the vehicle, thereby resulting in maintaining visibility of the depth of the vehicle front-back direction and decreasing the inclination of the image of the subject in the vehicle front-back direction.
  • FIG. 42 shows the display image of the third embodiment of the present invention, which adjusted the leftward and rightward correction image display regions of the second embodiment of the present invention.
  • the reference signs 41-1, 41-2 and 41-4 are respectively equal to the reference signs 41-1, 41-2 and 41-4 in FIG. 16 .
  • the reference sign 45-3L indicates a leftward correction image display region deformed from the leftward correction image display region 44-3L.
  • the reference sign 45-3R indicates a rightward correction image display region deformed from the rightward correction image display region 44-3R.
  • the reference sign 45-5L indicates a dotted line mask picture equivalent to the dotted line mask picture 44-5L.
  • the reference sign 45-5R indicates a dotted line mask picture equivalent to the dotted line mask picture 44-5R.
  • the reference sign 45-6L, 45-6R, 45-7L and 45-7R respectively indicates plane mask pictures equivalent to the plane mask pictures 44-6L, 44-6R, 44-7L and 44-7R.
  • the taken picture shown in FIG. 34 is displayed as shown in FIG. 41 .
  • the corrected image displayed on the leftward correction image display region 45-3L is deformed in a manner that the cells 23-2L on the leftward plane are appeared as parallelogram cells each having an area equal to one another and inclined down toward left gradually at a degree that the cells 23-2L are visible.
  • the leftward correction image display region 45-3L is deformed in response to the deformation of the cells 23-2L.
  • the corrected image displayed on the rightward correction image display region 45-3R is deformed in a manner that the cells 23-2R on the leftward plane are appeared as parallelogram cells each having an area equal to one another and inclined down toward right gradually at a degree that the cells 23-2R are visible.
  • the rightward correction image display region 45-3R is deformed in response to the deformation of the cells 23-2R.
  • FIG. 42 shows the display image deformed as previous mentioned, when the straight lines 24 on the ground surface shown in FIG. 13 , the horizon 8 and the bumper 10 of the vehicle 1 are simultaneously imaged.
  • the horizon 8 is displayed horizontally in the forward correction image display region 41-1.
  • the horizon 8 displayed on the leftward correction image display region 45-3L is connected to the horizon 8 displayed on the forward correction image display region 41-1 and inclined down toward left gradually at a degree that the horizon 8 is visible.
  • the horizon 8 displayed on the rightward correction image display region 45-3R is connected to the horizon 8 displayed on the forward correction image display region 41-1 and inclined down toward right gradually at a degree that the horizon 8 is visible.
  • FIG. 43 shows the display image under the imaging condition similar to the imaging condition of the display image shown in FIG. 19 concerned with the first embodiment of the present invention.
  • FIG. 38 shows the before deformed display image concerned with the second embodiment of the present invention.
  • the top side of the cuboid 11 in the vehicle front-back direction is displayed as inclined gradually down toward right in similar to the real sight in FIG.38 , but the top side of the cuboid 11 in the vehicle front-back direction is displayed on horizontal in FIG. 43 .
  • the form in the vehicle front-back direction is slightly different from the real sight in a limited area on the horizon 8, but the whole leftward or rightward correction image display region is almost unaffected in visibility.
  • FIG. 47 shows the display image of the fourth embodiment of the present invention, which adjusted the downward correction image display region of the first embodiment of the present invention.
  • the reference sign 22-2-1 indicates a downward virtual sight line adjusted from the downward virtual sight line 22-1-1 inclined with respect to the y-axis at a downward angle of 90 degrees as inclined with respect to the y-axis at a downward angle of 59 degrees.
  • the reference sign 22-2-2 indicates a downward plane being in perpendicular relationship with the downward virtual sight line 22-2-1, and including a point at 1.2m in the direction of the y-axis and 0m in the direction of the z-axis as similar to the downward plane 22-1-2.
  • the reference sign 22-2 indicates cells on the illustrated part of the downward plane 22-2-2, the cells 22-2 equivalent to the cells 22-1 on the downward plane.
  • Each of the vehicle left-right direction straight lines is assumed to be at 0.5m intervals with respect to the vehicle left-right direction straight line 21G.
  • the cells 22-2 on the downward plane are assumed to be imaged through the ground surface.
  • the reference signs 41-1, 41-3L, 41-3R, 41-5L, 41-5R, 41-7L and 41-7R are respectively equal to the reference signs 41-1, 41-3L, 41-3R, 41-5L, 41-5R, 41-7L and 41-7R in FIG. 16 .
  • the reference sign 46-2 indicates a downward correction image display region displayed therein the corrected image corresponding to the cells 22-2 on the downward plane.
  • the reference sign 46-4 indicates a dotted line mask picture equivalent to the dotted line mask picture 41-4.
  • the reference sign 46-8L indicates a dotted line mask picture constantly displayed as a dotted line in arbitrary color at a display position of adjacent sides of the leftward and downward correction image display regions 41-3L and 46-2.
  • the reference sign 46-8R indicates a dotted line mask picture constantly displayed as a dotted line in arbitrary color at a display position of adjacent sides of the rightward and downward correction image display regions 41-3R and 46-2.
  • FIG. 46 shows the operated image of the taken image shown in FIG. 44 under the allocation shown in FIG. 45 .
  • an image displayed on the downward correction image display region 46-2 is corrected in a manner that each of the cell 22-2 is displayed as a horizontally long rectangle formed by horizontal and vertical lines and having an area equal to one another.
  • an image displayed on the downward correction image display region 46-2 is corrected in a manner that rectangles are displayed on the whole of the downward correction image display region 46-2.
  • the cells may be corrected to be displayed as squares or vertically long rectangles depending on the relationships among the forward correction image display region 41-1, the leftward correction image display region 41-3L and the rightward correction image display region 41-3R.
  • the downward correction image display region 46-2 has a top side coincided with the vehicle left-right direction straight line 21G, located adjacent to the bottom side of the forward correction image display region 41-1, and coincided in the position on the x-axis direction with the vehicle left-right direction straight line 21G corresponding to the bottom side of the forward correction image display region 41-1.
  • FIG. 47 shows the display image when the straight lines 24 on the ground surface shown in FIG. 13 , the horizon 8 and the bumper 10 of the vehicle 1 are simultaneously imaged.
  • the downward correction image display region 46-2 has a left side coincided with the vehicle front-back direction straight line 24-15L, located adjacent to the bottom side of the leftward correction image display region 41-3L, and coincided in the position on the y-axis direction with the vehicle front-back direction straight line 24-15L corresponding to the bottom side of the leftward correction image display region 41-3L.
  • the downward correction image display region 46-2 has a right side coincided with the vehicle front-back direction straight line 24-15R, located adjacent to the bottom side of the rightward correction image display region 41-3R, and coincided in the position on the y-axis direction with the vehicle front-back direction straight line 24-15R corresponding to the bottom side of the rightward correction image display region 41-3R.
  • the image is displayed as following.
  • the display image is shown as FIG. 49 .
  • the display image is shown as FIG. 50 .
  • the display image is shown as FIG. 51 .
  • FIGS. 49 to 51 each shows an example of the display image when the cuboid 13 moves in the back of the vehicle 1 from left to right, thereby resulting in the fact that the display position of the cuboid 13 moves from the leftward correction image display region 41-3L to the downward and forward correction image display regions 46-2 and 41-1.
  • the sides of the cuboid 13 are polygonal at the boundary among the leftward correction image display region 41-3L and the downward correction image display region 46-2 and the forward correction image display region 41-1, the display position of the cuboid 13 moves without misalignment of the sides, and therefore the eye line toward a bottom part of the cuboid 13 continuously moves from left to bottom.
  • the relationship between the rightward correction image display region 47-3R and the downward or forward correction image display region 46-2, 41-1 is similar to the relationship between the leftward correction image display region 41-3L and the downward correction image display region 46-2.
  • the downward virtual sight line is not inclined downward at an angle of 90 degrees with respect to the y-axis.
  • the downward virtual sight line is inclined downward at an angle of 90 degrees with respect to the y-axis, the left and right sides of the downward correction image display region are displayed in vertical as similar to the first to third embodiments of the present invention, thereby resulting in the fact that each of the left and right sides of the downward correction image display region is not adjacent to the bottom side of the leftward or rightward correction image display region.
  • the downward virtual sight line is therefore inclined downward at an angle under 90 degrees with respect to the y-axis. This leads to the fact that the image is approximately the view in the direction of the front eye line as seen from the vehicle front-back direction straight lines displayed on the downward correction image display region 46-2 shown in FIG. 47 .
  • the direction of eye line toward the downward correction image display region is limited to be inclined forward, the image is displayed without discontinuity and misalignment from the left-right direction to the down direction, thereby resulting in improving visibility of the relationships in the whole image, and facilitating the movement of the eye line targeting to the subject moving in the whole display image.
  • FIG. 54 shows the display image of the fifth embodiment of the present invention, which adjusted the leftward, rightward and downward correction image display regions of the fourth embodiment of the present invention.
  • the reference sign 41-1 is equal to the reference sign 41-1 in FIG. 16 .
  • the reference sign 47-3L indicates a leftward correction image display region deformed from the leftward correction image display region 41-3L.
  • the reference sign 47-3R indicates a rightward correction image display region deformed from the rightward correction image display region 41-3R.
  • the reference sign 47-2 indicates a downward correction image display region deformed from the downward correction image display region 46-2
  • the reference sign 47-5L indicates a dotted line mask picture equivalent to the dotted line mask picture 41-5L.
  • the reference sign 47-5R indicates a dotted line mask picture equivalent to the dotted line mask picture 41-5R.
  • the reference sign 47-4 indicates a dotted line mask picture equivalent to the dotted line mask picture 46-4.
  • the reference sign 47-8L indicates a dotted line mask picture equivalent to the dotted line mask picture 46-8L.
  • the reference sign 45-8L indicates a dotted line mask picture equivalent to the dotted line mask picture 46-8R.
  • the reference signs 47-7L and 47-7R respectively indicate plane mask pictures equivalent to the plane mask pictures 41-7L and 41-7R.
  • the taken image shown in FIG. 44 is displayed as FIG. 53 .
  • the corrected image displayed on the leftward correction image display region 47-3L is deformed in a manner that the cells 23-1L on the leftward plane are appeared as parallelogram cells each having an area equal to one another and inclined down toward left gradually at a degree that the cells 23-1L are visible.
  • the leftward correction image display region 47-3L is deformed in response to the deformation of the cells 23-1L.
  • the corrected image displayed on the rightward correction image display region 47-3R is deformed in a manner that the cells 23-1R on the rightward plane are appeared as parallelogram cells each having an area equal to one another and inclined down toward right gradually at a degree that the cells 23-1R are visible.
  • the rightward correction image display region 47-3R is deformed in response to the deformation of the cells 23-1R.
  • Each of the parallelogram rightward cells is equal in form and size to each parallelogram rightward cell as if horizontally flipped.
  • the corrected image displayed on the downward correction image display region 47-2 is downwardly expanded in response to the deformation of the leftward correction image display region 47-3L and the rightward correction image display region 47-3R.
  • the vertical sides of the rectangle cells displayed as the rectangle cells 22-2 on the downward correction image display region are respectively deformed longer than that of the rectangle cells displayed on the downward correction image display region 46-2.
  • FIG. 54 shows the display image deformed as previous mentioned, when the straight lines 24 on the ground surface shown in FIG. 13 , the horizon 8 and the bumper 10 of the vehicle 1 are simultaneously imaged.
  • the displayed horizon 8 is inclined as similar in the third embodiment of the present invention.
  • FIG. 55 shows the display image under the imaging condition similar to the imaging condition of the display image shown in FIG. 19 concerned with the first embodiment of the present invention.
  • FIG. 48 shows the before deformed display image concerned with the fourth embodiment of the present invention.
  • the inclination of the top side of the cuboid 11 in the vehicle front-back direction is similar to that of the third embodiment of the present invention.
  • the corrected image displayed on the downward correction image display region 47-2 as shown in FIG. 54 is not visibly different from the corrected image displayed on the downward correction image display region 46-2 as shown in FIG. 47 .
  • the fifth embodiment has visibility similar to the third embodiment without influence of visibility of the downward correction image display region under the state that the downward correction image display region is in contact with each leftward and rightward correction image display region.
  • FIG. 56 shows the display image of the sixth embodiment of the present invention, which adjusted the positions of the boundaries between the frontward correction image display region and the leftward or rightward correction image display region on the basis of the relationships shown in FIG. 14 .
  • the reference sign 25-3 indicates a circle being an innermost of a vehicle travelling way on the ground surface, the vehicle travelling way travelled thereon the vehicle at full lock.
  • the reference sign 25-4 indicates a circle being an outermost of a vehicle travelling way on the ground surface, the vehicle travelling way travelled thereon the vehicle at full lock.
  • the innermost circle 25-3 is concentric with the outermost circle 25-4.
  • the reference sign 25-2 indicates a common center of the innermost circles 25-3 with the outermost circles 25-4.
  • the reference sign 25-1 indicates an image origin being under an imaging position on the ground surface.
  • the reference sign 25-5 indicates a tangent line tangent to the innermost circles 25-3 with passing on the image origin 25-1 on the ground surface.
  • the reference sign 25-6 indicates a tangent point of the tangent line 25-5 with respect to the innermost circles 25-3.
  • the reference sign 25-7 indicates an image separation reference point on the tangent line 25-5 at an arbitrary distance such as 1.5m in the direction of the x-axis.
  • the second embodiment of the present invention is not adjusted on the basis of the positional relationship as shown in FIG. 14
  • the sixth embodiment of the present invention will be explained as operating in the same manner as the second embodiment of the present invention as shown in FIGS. 35 and 36 in the result of the adjustment on the basis of the positional relationship as shown in FIG. 14 .
  • FIG. 56 shows the display image operated in the same manner as the second embodiment of the present invention as shown in FIGS. 35 and 36 .
  • the innermost circle 25-3 is displayed on the leftward correction image display region 44-3L with the subjects shown in FIG. 37 .
  • the innermost circle 25-3 is not displayed in real, but shown for better understanding of the positional relationship.
  • the reference sign 25-8 indicates an image separation point representing a boundary among the downward correction image display region 41-2, the forward correction image display region 41-1 and the leftward correction image display region 44-3L.
  • the image separation point is the same position in the directions of the x-axis, y-axis and z-axis directions on the downward correction image display region 41-2, the forward correction image display region 41-1 and the leftward correction image display region 44-3L.
  • the direction of the y-axis is adjusted to the relationship 1.2m vicinity of the 1.16m of the image separation reference point 25-7.
  • the dotted mask picture 44-5L displayed at any height in the direction of z-axis indicates the direction toward 1.5m in the direction of x-axis and 1.2m in the direction of the y-axis from the image origin 25-1 on the ground surface.
  • the solid subject having a height in the direction of the z-axis from the innermost circle 25-3 is displayed in vertical with respect to the innermost circle 25-3.
  • the cuboid 11 is corresponding to a solid subject having an edge in the vicinity of the tangent point 25-6, and a vertical side of the edge is displayed in the vicinity of the dotted mask picture 44-5L. At this time, the vehicle 1 at full lock passes beside the cuboid 11.
  • the vehicle 1 at full lock for the purpose of parking is then recognized to be in the vicinity of the cuboid 11 without contacting with the cuboid 11.
  • the straight line formed mask pictures indicating the boundary between the leftward or rightward correction image display region and the forward correction image display region acts as a guideline for judging whether or not the vehicle at full lock is in contact with the solid subject, thereby resulting in improving visibility of the relationships between the vehicle, for example, moved into the garage and the solid subject.
  • the drive assist display apparatus of the present invention is available for a drive assist display apparatus which can realize, from imaging to displaying the front, left, right and down wide area of the backward or frontward of the vehicle, displaying in high visibility of the form of the imaged subject in accordance with purposes, improving visibility of the directional and positional relationships between the imaged subject and the vehicle, and facilitating the movement of the eye line targeting to the subject on the display.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Closed-Circuit Television Systems (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)
  • Fittings On The Vehicle Exterior For Carrying Loads, And Devices For Holding Or Mounting Articles (AREA)
EP10848628.3A 2010-04-08 2010-07-23 Fahrhilfen-anzeigevorrichtung Active EP2471689B1 (de)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2010089724A JP5451497B2 (ja) 2010-04-08 2010-04-08 運転支援表示装置
PCT/JP2010/004722 WO2011125137A1 (ja) 2010-04-08 2010-07-23 運転支援表示装置

Publications (3)

Publication Number Publication Date
EP2471689A1 true EP2471689A1 (de) 2012-07-04
EP2471689A4 EP2471689A4 (de) 2014-09-10
EP2471689B1 EP2471689B1 (de) 2018-07-04

Family

ID=44762126

Family Applications (1)

Application Number Title Priority Date Filing Date
EP10848628.3A Active EP2471689B1 (de) 2010-04-08 2010-07-23 Fahrhilfen-anzeigevorrichtung

Country Status (5)

Country Link
US (1) US8363103B2 (de)
EP (1) EP2471689B1 (de)
JP (1) JP5451497B2 (de)
CN (1) CN102341269B (de)
WO (1) WO2011125137A1 (de)

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP5880916B2 (ja) * 2011-06-03 2016-03-09 ソニー株式会社 情報処理装置、情報処理方法、およびプログラム
JP5999043B2 (ja) * 2013-07-26 2016-09-28 株式会社デンソー 車両周辺監視装置およびプログラム
DE102014107235A1 (de) * 2014-05-22 2015-11-26 Dr. Ing. H.C. F. Porsche Aktiengesellschaft Verfahren zur Darstellung einer Fahrzeugumgebung auf einer Anzeigevorrichtung; eine Anzeigevorrichtung; ein System aus einer Mehrzahl an Bilderfassungseinheiten und einer Anzeigevorrichtung; ein Computerprogramm
JP2016225865A (ja) * 2015-06-01 2016-12-28 東芝アルパイン・オートモティブテクノロジー株式会社 俯瞰画像生成装置
CN108432241B (zh) * 2015-12-28 2022-11-29 住友建机株式会社 挖土机
US10861338B2 (en) * 2016-05-05 2020-12-08 Harman International Industries, Incorporated Systems and methods for driver assistance
CN106915295B (zh) * 2017-03-21 2019-09-17 青岛海信移动通信技术股份有限公司 车辆大灯状态的控制方法及装置
US11866042B2 (en) 2018-08-20 2024-01-09 Indian Motorcycle International, LLC Wheeled vehicle adaptive speed control method and system
WO2020090512A1 (ja) * 2018-10-31 2020-05-07 ソニー株式会社 撮影装置、制御方法、及び、プログラム

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1288618A2 (de) * 2001-08-09 2003-03-05 Matsushita Electric Industrial Co., Ltd. Anzeigevorrichtung zur Fahrunterstützung
US20050012685A1 (en) * 2002-03-05 2005-01-20 Tsuyoshi Okada Image display controller
EP1816869A1 (de) * 2004-11-24 2007-08-08 Aisin Seiki Kabushiki Kaisha Kamera-kalibrationsverfahren und kamera-kalibrationseinrichtung
US20090322878A1 (en) * 2007-04-23 2009-12-31 Sanyo Electric Co., Ltd. Image Processor, Image Processing Method, And Vehicle Including Image Processor

Family Cites Families (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3824237B2 (ja) * 2003-09-05 2006-09-20 ソニー株式会社 画像処理装置および方法、記録媒体、並びにプログラム
JP4766841B2 (ja) * 2003-09-08 2011-09-07 株式会社オートネットワーク技術研究所 車両に搭載されるカメラ装置及び車両周辺監視装置
JP2006159933A (ja) * 2004-12-02 2006-06-22 Denso Corp 車両周辺表示支援システム
JP4682664B2 (ja) * 2005-03-29 2011-05-11 アイシン精機株式会社 周辺監視システム
JP2007022176A (ja) 2005-07-13 2007-02-01 Auto Network Gijutsu Kenkyusho:Kk 車両周辺視認装置
JP4862321B2 (ja) * 2005-08-30 2012-01-25 ソニー株式会社 車載カメラ装置
JP4014612B2 (ja) * 2005-11-09 2007-11-28 シャープ株式会社 周辺光量補正装置、周辺光量補正方法、電子情報機器、制御プログラムおよび可読記録媒体
JP4762698B2 (ja) * 2005-11-30 2011-08-31 アルパイン株式会社 車両周辺画像表示装置
JP4731392B2 (ja) * 2006-04-26 2011-07-20 アルパイン株式会社 車載周辺状況提示装置
JP4926822B2 (ja) 2006-05-24 2012-05-09 株式会社岩根研究所 湧きだし点基準ｃｖ領域画像前置装置
JP4257356B2 (ja) * 2006-09-26 2009-04-22 株式会社日立製作所 画像生成装置および画像生成方法
JP5088669B2 (ja) * 2007-03-23 2012-12-05 株式会社デンソー 車両周辺監視装置
JP4714174B2 (ja) * 2007-03-27 2011-06-29 富士フイルム株式会社 撮像装置
JP4793307B2 (ja) * 2007-04-03 2011-10-12 株式会社デンソー 車両周辺監視装置
JP4924896B2 (ja) * 2007-07-05 2012-04-25 アイシン精機株式会社 車両の周辺監視装置
JP5075674B2 (ja) * 2008-02-26 2012-11-21 本田技研工業株式会社 駐車支援装置
JP2010009235A (ja) * 2008-06-25 2010-01-14 Toyota Motor Corp 画像表示装置

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1288618A2 (de) * 2001-08-09 2003-03-05 Matsushita Electric Industrial Co., Ltd. Anzeigevorrichtung zur Fahrunterstützung
US20050012685A1 (en) * 2002-03-05 2005-01-20 Tsuyoshi Okada Image display controller
EP1816869A1 (de) * 2004-11-24 2007-08-08 Aisin Seiki Kabushiki Kaisha Kamera-kalibrationsverfahren und kamera-kalibrationseinrichtung
US20090322878A1 (en) * 2007-04-23 2009-12-31 Sanyo Electric Co., Ltd. Image Processor, Image Processing Method, And Vehicle Including Image Processor

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
See also references of WO2011125137A1 *
Yu-Chih Liu ET AL: "Bird's-Eye View Vision System for Vehicle Surrounding Monitoring" In: "Robot Vision", 1 January 2008 (2008-01-01), Springer Berlin Heidelberg, Berlin, Heidelberg, XP055004046, ISBN: 978-3-54-078157-8 vol. 4931, pages 207-218, DOI: 10.1007/978-3-540-78157-8_16, * figures 4-6 * *

Also Published As

Publication number Publication date
JP2011221760A (ja) 2011-11-04
CN102341269A (zh) 2012-02-01
EP2471689A4 (de) 2014-09-10
US20110279674A1 (en) 2011-11-17
JP5451497B2 (ja) 2014-03-26
WO2011125137A1 (ja) 2011-10-13
CN102341269B (zh) 2015-06-17
EP2471689B1 (de) 2018-07-04
US8363103B2 (en) 2013-01-29

Similar Documents

Publication Publication Date Title
EP2471689B1 (de) Fahrhilfen-anzeigevorrichtung
US10212342B2 (en) Panoramic windshield viewer system
US7365653B2 (en) Driving support system
US8199975B2 (en) System and method for side vision detection of obstacles for vehicles
US8009868B2 (en) Method of processing images photographed by plural cameras and apparatus for the same
US8477191B2 (en) On-vehicle image pickup apparatus
EP2939211B1 (de) Verfahren und system zur erzeugung einer rundumansicht
JP5715778B2 (ja) 車両用画像表示装置
JP7263731B2 (ja) 画像処理装置
US8848050B2 (en) Drive assist display apparatus
JPWO2013046593A1 (ja) 俯瞰画像生成装置、俯瞰画像生成方法、および俯瞰画像生成プログラム
KR102057021B1 (ko) 패널 변환
CN106436544A (zh) 机动建筑机械和用于显示机动建筑机械周围环境的方法
US11945306B2 (en) Method for operating a visual field display device for a motor vehicle
CN107004250B (zh) 图像生成装置及图像生成方法
WO2019034916A1 (en) SYSTEM AND METHOD FOR PRESENTING AND CONTROLLING VIRTUAL CAMERA IMAGE FOR A VEHICLE
CN114077306A (zh) 实现内容可视化的装置和方法
JP4247066B2 (ja) 車両周辺視認装置
CN112802109B (zh) 一种汽车鸟瞰全景图生成方法
US20130044217A1 (en) Drive assist display apparatus
CN113379605B (zh) 一种车载360°全景影像系统及计算机存储介质
TWM518653U (zh) 行車影像輔助裝置
US20220222947A1 (en) Method for generating an image of vehicle surroundings, and apparatus for generating an image of vehicle surroundings
CN204845719U (zh) 三维行车影像辅助装置
JP6007773B2 (ja) 画像データ変換装置並びにナビゲーションシステムおよびカメラ装置並びに車両

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20120516

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

RIN1 Information on inventor provided before grant (corrected)

Inventor name: TAKAHASHI, KOJI

Inventor name: ICHINOSE, TADAO

DAX Request for extension of the european patent (deleted)
A4 Supplementary search report drawn up and despatched

Effective date: 20140808

RIC1 Information provided on ipc code assigned before grant

Ipc: B60R 11/02 20060101ALI20140804BHEP

Ipc: H04N 7/18 20060101ALI20140804BHEP

Ipc: B60R 21/00 20060101ALI20140804BHEP

Ipc: G06T 3/00 20060101ALI20140804BHEP

Ipc: B60R 1/00 20060101AFI20140804BHEP

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: PANASONIC INTELLECTUAL PROPERTY MANAGEMENT CO., LT

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: PANASONIC INTELLECTUAL PROPERTY MANAGEMENT CO., LT

17Q First examination report despatched

Effective date: 20160919

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20180409

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1014154

Country of ref document: AT

Kind code of ref document: T

Effective date: 20180715

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602010051730

Country of ref document: DE

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20180704

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1014154

Country of ref document: AT

Kind code of ref document: T

Effective date: 20180704

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180704

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180704

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181004

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181104

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180704

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181004

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180704

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180704

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180704

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20181005

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180704

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180704

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180704

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180704

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180704

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180723

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20180731

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602010051730

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

REG Reference to a national code

Ref country code: DE

Ref legal event code: R084

Ref document number: 602010051730

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180723

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180704

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180704

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180704

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180704

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180731

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180731

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180704

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180731

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180704

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180704

26N No opposition filed

Effective date: 20190405

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20181004

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180704

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180904

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20181004

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180723

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180704

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20100723

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180704

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180704

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20180704

REG Reference to a national code

Ref country code: DE

Ref legal event code: R081

Ref document number: 602010051730

Country of ref document: DE

Owner name: PANASONIC AUTOMOTIVE SYSTEMS CO., LTD., YOKOHA, JP

Free format text: FORMER OWNER: PANASONIC INTELLECTUAL PROPERTY MANAGEMENT CO., LTD., OSAKA, JP

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20240719

Year of fee payment: 15