Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/77—Retouching; Inpainting; Scratch removal
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
  • B60R1/00—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
  • B60R1/20—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
  • B60R1/22—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle
  • B60R1/28—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with an adjustable field of view
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/20—Image enhancement or restoration using local operators
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
  • G06V20/50—Context or environment of the image
  • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
  • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
  • G06V20/586—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads of parking space
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
  • G06V20/60—Type of objects
  • G06V20/64—Three-dimensional objects
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20081—Training; Learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20084—Artificial neural networks [ANN]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30248—Vehicle exterior or interior
  • G06T2207/30252—Vehicle exterior; Vicinity of vehicle
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30248—Vehicle exterior or interior
  • G06T2207/30252—Vehicle exterior; Vicinity of vehicle
  • G06T2207/30264—Parking

Definitions

• the present invention relates to a method in which a scenery is recorded as at least one raw image by at least one optical detection means, which is attached in particular to a means of transport, and in which image data of the scenery in the subsequently rendered Reder image is incomplete in at least one area and / or with errors.
• the invention also relates to an image processing system, a use of the same and a means of locomotion.
• Means of transport, and among these in particular land vehicles, are provided with an increasing number of Assistenzsyste men.
• These also include camera devices that are intended to give the occupants and, among them, preferably the respective driver, a visual impression of one or more scenes outside a passenger compartment, for example to support, facilitate or monitor a parking process.
• an image processing system according to claim 12, its use according to claim 13 and a means of locomotion according to claim 14 represent solutions to the mentioned problem.
• the solution therefore consists initially in a modular method, in which in a first step an identification of missing parts in the rendered image based on the visibility restrictions. In a next step, masks are created from the missing parts of the rendered image, and the rendered image is only to be reconstructed in these masks.
• the correction image is optimized by post-processing to form an optimization image and this is displayed instead of the respective correction image.
• Possible artifacts can be smoothed and the appearance can be made more pleasant to the eye after the image reconstruction, for which the rendered image can be post-processed, for example to increase the sharpness, reduce the contrast and / or harmonize colors.
• the rendered images, reconstructed correction images or optimized optimization images are displayed to the viewer in real time or with a negligible delay to allow the user to react promptly to the scenes of the images shown to him.
• Preferably carries the repetition frequency of the displayable images at least 5fps.
• the visibility restrictions are determined at least using a three-dimensional model of the respective vehicle as well as using an accommodation of the optical detection means.
• Different scenarios can also be described in a suitable manner using known visibility restrictions. Concretely related to a scenario with a motor vehicle, those parts of the rendered scene that do not have any image data can be identified using a 3D model of the motor vehicle and the camera housing.
• data on visibility restrictions, geometric models of environments (and patterns) and (in particular) scenery data that have already been generated can be stored in at least one database in an advantageous variant of the method.
• parts of the respective render image that are not to be reconstructed for the mask or masks can then be protected or hidden from further processing in a simple manner from the start.
• the image data to be reconstructed can be generated with the aid of a machine learning approach.
• the image data can be reconstructed with the aid of an artificial neural network, which accesses the at least one database and is trained on the basis of its data.
• the incomplete and / or faulty image data can be reconstructed using edge-based methods. Thereby, edges or object transitions are searched for in the rendered images. Processing with algorithms often does not yet provide closed edges; these have to be joined using additional processes to enclose objects.
• the edge-based method can preferably be a level set method, in particular a fast marching method.
• the former represents a numerical method for approximating geometric objects and their movement, with curves and surfaces advantageously being able to be calculated on a spatially fixed coordinate system without having to use parameterizations of the objects in question.
• a special method for solving boundary value problems numerically is the fast marching method, which solves boundary value problems of the eikonal equation, here the development of a closed surface as a function of time with a certain speed.
• the edge-based method can use a diffusion approach that is used for dimension reduction or feature extraction and is later propagated to predict the information in areas of incomplete and / or faulty mapping.
• a machine learning approach can be used in a suitable manner, for example a Markov Random Field (MRF) method, whereby the MRFs can be used to segment digital images or classified areas and assume an interaction or mutual influencing of elements of a field.
• MRF Markov Random Field
• an image processing system that executes the above method in one of its variants, whose use in a parking assistance system is a
• a means of locomotion which, for example, carries out a parking process at low speed, and a means of locomotion itself, in particular a land vehicle, is provided with such a system for the task at hand.
• Fig. La, lb schematic perspective views of a rear outer area on a motor vehicle, recorded with an optical detection means, shown as a render image (Fig. La) and as a correction image (Fig. Lb), which was generated according to the inventive method; and
• FIG. 2a, 2b are schematic perspective views of an outside area on a motor vehicle, recorded with an optical detection means, shown as a render image (FIG. 2a) and as a correction image (FIG. 2b) that was generated according to the method according to the invention.
• La and lb show schematic perspective views of a rear scenery in the outside area of a motor vehicle, which is detected with an optical detection means.
• the rendered image of the scene shown in Fig. La one recognizes an essentially rectangular area which is not completely mapped, since there image data of the scene are missing due to the housing of the optical detection means designed as a camera, not shown.
• the missing image data can be reconstructed by means of the method according to the invention by means of digital impainting.
• the aim is to create a coherent image that is based on the entirety of the image itself and then gives the user a better visual experience when looking at it.
• Said area of missing image data 10 can be recognized by an edge 20 which separates this area from that area 30 known image data, which represents the introductory, identifying step.
• the area 30 of known image data is provided with a mask, i.e. masks are generated which, with masked areas 30 that are not to be processed, enclose the area or areas 10 of incomplete and / or faulty imaging, so that although these areas 10, but not the areas 30 of known image data that are correctly captured and rendered, are reconstructed.
• Contour lines of the rendered image that touch the edge 20 of the mask are continued along their imaginary extension into the non-masked area 10 of the image, as well as the structure of an area around the mask edge 20. Different sections are defined in the unmasked area by the contour lines, which are filled with the respective color of the edge assigned to them, after which the respective area is optionally textured.
• FIGS. 1 a and 1 b it can be seen that the edges 40a, 40b, 40c are shown correctly in the correction image, while the continuation of the edge 40d shows a negligible discontinuity due to the darkness of the upper area.
• FIG. 2a and 2b which in turn show a render image (FIG. 2a) and a correction image (FIG. 2b) of another, here a lateral scene on a motor vehicle, seen from above, show that those are missing in the render image Image data of the again rectangular area 10 in the correction image, on the one hand, the depicted structure 50, but also the shadow area 60 facing away from a light source, not shown, can be accurately reproduced by the reconstruction according to the invention.
• the invention described above relates to a method for processing images, wherein a scene is recorded as at least one raw image by at least one optical detection means, which is attached in particular to egg nem means of transport, and image data of the scenery in the subsequently rendered render image at least incomplete and / or in at least one area be mapped with errors.
• the process comprises the following steps:
• the visual experience of a user of a system provided with the optical detection means is improved in a suitable manner, since a spot-free, continuous representation of the scenery in the correction image is made available.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Multimedia (AREA)
• Mechanical Engineering (AREA)
• Image Analysis (AREA)
• Image Processing (AREA)
• Closed-Circuit Television Systems (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=69723762

Family Applications (1)

Country Status (6)

Families Citing this family (4)

Family Cites Families (20)

• 2019
  • 2019-02-11 DE DE102019201702.9A patent/DE102019201702A1/de active Pending
• 2020
  • 2020-01-30 US US17/310,566 patent/US11961215B2/en active Active
  • 2020-01-30 WO PCT/DE2020/200008 patent/WO2020164671A1/de not_active Ceased
  • 2020-01-30 CN CN202080009784.8A patent/CN113330480A/zh active Pending
  • 2020-01-30 EP EP20707560.7A patent/EP3924932A1/de active Pending
  • 2020-01-30 JP JP2021542481A patent/JP2022517849A/ja active Pending
• 2024
  • 2024-01-11 JP JP2024002421A patent/JP2024041895A/ja active Pending

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events