JPWO2019223840A5 - - Google Patents

Description

The present invention relates to a method and a device for displaying the vehicle surroundings in a vehicle at a current point in time.

The vehicle may be equipped with a driver assist system that displays or reproduces the vehicle surroundings on a display device or the like for the vehicle occupants inside the vehicle. Such driver assist systems, also called surround view systems , are usually installed in the vehicle and include one or more on-board cameras with different viewing areas or allowing different viewing angles. are doing.

Although multiple viewing areas are captured by different onboard cameras, at least one blind spot area, also known as a blind spot, remains, which is outside all the viewing areas and is therefore not included within the camera image . An example of such a blind spot area is a ground surface that a vehicle passes over at the moment of observation , and as a result becomes invisible . This is because the various onboard cameras are usually placed, for example, in front of the vehicle , at the rear of the vehicle , and on the side of the vehicle to see far away , so that the ground surface is always covered by the vehicle itself or its body. . Therefore , this area is usually not displayed and is occupied by placeholders, for example. This placeholder is not only visually unsightly , but also irritating to vehicle occupants .

It is therefore an object of the invention to provide a method which allows an improved display of the vehicle surroundings inside a vehicle.

This problem is solved by the subject matter of the independent claims. Advantageous embodiments and developments are indicated by the dependent claims, the description and the drawings.

A first aspect of the invention relates to a method for displaying a vehicle surroundings within a vehicle at a current, second point in time. "Current " can be understood to mean that the display of the surroundings of the vehicle is basically done in real time while the vehicle is moving . The vehicle may move in any direction, with the vehicle speed determined , for example, from the odometry data , with real-time display occurring at an appropriate frame rate.

The method according to the invention contemplates the following steps.
- First , a first blind spot area image, for example a composite image or a rendered image, is provided with a plurality of blind spot area pixels, which image represents the field of view of the on-board camera at a first point in time before a second point in time; This includes image synthesis, for example rendering, of the blind spot area around the vehicle located outside . That is , the first point in time is earlier in time than the current second point in time being observed.
- Then each first blind spot area pixel of the blind spot area is placed, eg transformed, into a new position estimated, eg predicted, for the second point in time ; That is, the blind spot area image has a plurality of blind spot area pixels arranged from an old position included in the blind spot area image to a new position based on estimation, prediction, transformation, interpolation, etc. A combination of blind area pixels may also be used instead of each single blind area pixel to reduce aliasing effects .
- subsequently determining whether the new position of each first blind spot region pixel is within the blind spot region at a second point in time; In this way , the considered and newly placed blind spot area pixels may be located outside the blind spot area due to the movement of the vehicle.
- Subsequently, if the new position is determined to be within the blind spot area, each first blind spot area pixel is synthesized with motion compensation based on the vehicle's motion data, so that the first blind spot area pixel is Generation ( eg, compositing, rendering, etc.) of a second blind spot region pixel (eg , rendered pixel) for each of the two points in time is performed . Since the vehicle may move from blind spot area image to blind spot area image while driving , the blind spot area pixels may also change their position , and the vehicle's motion data may change the second blind spot area pixel. considered for generating. If the vehicle is not moving, the second blind spot area pixel corresponds to the first one. Motion data can be obtained , for example, from odometry of a vehicle system or movement estimation from multiple camera images.

According to this configuration , the proposed method offers several advantages . In this way , a temporary reconstruction of the blind spot area of the vehicle surroundings is performed without a placeholder permanently covering the blind spot area without reference to the vehicle surroundings to be displayed. Furthermore, since it is basically a recursive method, it is necessary to retain and provide only one first blind spot area image, which reduces the available memory of the memory unit and/ or data processing unit, for example. It is possible to save resources. This makes it possible to reduce the cost of providing a driver assistance system that operates in this way.

In an advantageous development, each second blind spot area pixel at the second point in time is changed to the first point in time if the specific new position of each first blind spot area pixel is determined to be outside the blind spot area image. It is contemplated that each first blind spot region pixel can be generated by combining using at least one provided camera image taken at . In this case, the already synthesized images ( e.g. rendered images ) are no longer used as a basis for further image synthesis ( e.g. rendering ) , but rather the camera images taken from the vehicle periphery. use This also significantly saves memory resources because only one camera image needs to be stored and provided . For example, if a vehicle has a plurality of cameras oriented in the direction of line of sight to the front of the vehicle, the direction of line of sight to the rear of the vehicle, and the direction of line of sight to the side of the vehicle, one Each vehicle-mounted camera can provide its own camera image.

In a further embodiment, the movement vector of each first blind spot region pixel obtained from the motion data is used to select one of a plurality of differently positioned and/or aligned vehicle cameras , each having a different field of view. , it is possible to determine which one vehicle-mounted camera provides the camera image used for pixel compositing . Thereby, for example, a first blind spot area pixel that moves into the viewing area directed towards the front of the vehicle due to the movement of the vehicle can be generated or synthesized, for example rendered , by one camera image of this on-vehicle camera. You can .

Depending on the direction of movement of the vehicle, ie straight forward, backward, and/ or curved driving, the movement vector may be located within the respective field of view of the corresponding on-vehicle camera. In this case , the corresponding camera image of the very camera in whose field of view the movement vector is located can be selected for presentation .

In order to significantly save memory resources , it is advantageous for each existing vehicle camera to be able to provide exactly one camera image, or one frame.

In this regard, it is also advantageous to ensure that exactly one blind spot area image is provided for a first point in time.

For example, in order to be able to reproduce a display image of the vehicle surroundings to be displayed on the display of the vehicle, a rendering image generated or synthesized from respective second blind spot area pixels at a second point in time, e.g. The second blind spot region image obtained can be combined with at least one current camera image at a second point in time . In this case, the viewing area is represented by a real- time camera image, and the blind spot area is represented by a second blind spot area image at a second current point in time .

In particular, at the initial startup of a device and/ or a driver assist system that operates according to this method, a situation may occur in which the first blind spot area image of the blind spot area is not yet available. The display image to be displayed inside the vehicle can then be shifted to the viewing area of the respective on-board camera, for example the front camera, in order to hide the blind spot area.

This method can be used particularly advantageously when the blind spot area is the ground surface on which the vehicle travels or the ground surface covered by the vehicle body . Instead of placeholders, a composite , e.g. rendered, display image can be displayed.

The method described above can be implemented, for example , in a control means such as a control device of a vehicle . The control means may include a storage device and a data processing device .

A further aspect of the invention relates to a device for displaying a vehicle surroundings within a vehicle at a current, second point in time . The device is equipped with :
- at least one onboard camera for determining a corresponding viewing area around the vehicle;
- Exactly one blind spot area image is maintained , including an image composition of the blind spot area around the vehicle located outside the viewing area at the first point in time before the second point in time. storage device .
- to carry out the following, namely:
i) placing each first blind spot region pixel of the blind spot region in a new position estimated with respect to a second point in time; ii) the new position of each blind spot region pixel at the second point in time continues to be in the blind spot region; and iii) if the new position is determined to be within the blind spot region, each first blind spot region pixel is motion compensated based on motion data of the vehicle. generating each second blind spot region pixel at a second point in time by ;
data processing equipment adapted to carry out .

The device can be developed according to the method described above and has the particular advantage that, in principle, a recursive method can save memory resources significantly.

The invention is particularly suitable for a driver assist system for a vehicle equipped with a display device arranged inside the vehicle, for example a display.

BRIEF DESCRIPTION OF THE DRAWINGS Advantageous embodiments of the invention will be explained in detail below using the accompanying figures.

Figure 3 shows a vehicle equipped with a driver assist system according to an aspect , comprising a device for displaying the vehicle surroundings according to a further aspect of the invention; FIG. 6 shows an illustration of generating a blind spot area image using a previously synthesized blind spot area image as a base; FIG. Fig. 3 shows an illustration of generating a blind spot area image using a previously captured camera image as a base; Fig. 3 shows an illustration of generating a blind spot area image using a previously captured camera image as a base; 5 shows a flowchart for a method according to a further aspect of the invention.

The drawings are only schematic illustrations and are used only to explain the invention. Elements that are the same or have the same effect are consistently given the same reference numbers .

FIG. 1 shows a vehicle 1 on the ground surface , which can essentially move in the x and y directions and is equipped with a device 100 in the form of a driver assist system or a surround view system. Be prepared . This allows the occupants of the vehicle to see a surround view of the current vehicle surroundings within the vehicle as a display or visual representation .

For this purpose , the device 100 comprises a display device 110 in the form of a display, which is arranged inside the vehicle 1 and visually displays the display image I. Furthermore , the device 100 comprises a data processing device 120 , comprising at least one processor (not described in detail), which acts together with a display device 110, and a storage device 130, which also acts together. In addition, the device 100 includes a plurality of on-vehicle cameras 140F, 140R, 140LL, 140LR, which are installed at different positions of the vehicle 1 and have different viewing angles or viewing areas. It has 141F, 141R, 141LL, and 141LR . Therefore , the vehicle-mounted camera 140F is disposed at the front of the vehicle , the vehicle-mounted camera 140R is disposed at the rear of the vehicle , the vehicle-mounted camera 140LL is disposed laterally to the left , and the vehicle-mounted camera 140LR is disposed laterally to the right. The viewing areas 141F, 141R, 141LL, and 141LR can grasp the respective camera images I_F, I_R, I_LL, and I_LR, and can be directly reproduced on the display device 110, and may be (temporarily) stored in the storage device 130. be done. For display, camera images I_F, I_R, I_LL, I_L R are composited or combined into a display image I by data processing device 120 (see FIGS. 2A-2C).

From FIG. 1, on the ground surface , there is a blind spot area B that is not within any of the viewing areas 141F, 141R, 141LL, 141LR due to the vehicle 1 located on the ground surface or the body of the vehicle 1 , and therefore, It can be seen that none of the vehicle-mounted cameras 140F, 140R, 140LL, and 140LR can detect this because the vehicle body blocks their respective fields of view. As a result , a direct camera image of the blind spot area B cannot be reproduced as the blind spot area image I_B of the display image I on the display device 110.

Nevertheless, in order to display the blind spot area B in an essentially photorealistic representation in the form of a blind spot area image I_B on the display device 110 , the device 100 is configured to may operate by the method described below with reference to .

In principle , displaying the surrounding area of the vehicle on the display device 110 should be performed basically in real time at the current time t, so the display image I is the display of the vehicle at the current time t. It should include the surrounding area. This is indicated by the time variable [t 2 ] with the same reference number in the following description and in FIGS. 2A-2C. Accordingly , the current first time point t-1, which is earlier than the second time point t, is designated as [t-1] in FIGS. 2A-2C.

At the time of initial startup of the device 100, in an arbitrary step S0, the blind spot area B in the display image I of the display device 110 converts the display image I of the vehicle periphery into the viewing area 141F of the vehicle-mounted cameras 140F, 140R, 140LL, and 140LR. By shifting blind area B to one of 141R, 141LL, and 141LR to such an extent that it becomes blank , it becomes blank . The selection of the viewing areas 141F, 141R, 141LL, and 141L R is performed depending on vehicle data such as vehicle speeds v_x, v_y, steering angle, transmission gear selection, etc., for example, depending on the traveling directions x and y of the vehicle. It can be done by Expressed concretely , the display image I precedes the actual surrounding area of the vehicle at time t by, for example, the length of one vehicle body, and the blind spot area B becomes blank . As a result, instead of the blind spot area B, an initial blind spot area including a portion of each camera image I F[t-1], IR[t-1], ILL[t-1], ILR[t-1] is created. Image I_B[t-1] can be stored and kept available .

In step S1, the blind spot area image I_B[t-1] at time t-1 is provided from the storage device 130 at the current time t, and this is the content of the above-mentioned blind spot only at the first time immediately after the initial startup of the device 100. During operation , it is continuously updated as described below . If the vehicle 1 moves , for example, at 15 m /s in the Time includes the image content of the preceding time point t-1.

In step S2, the data processing device 120 determines that each first blind spot area pixel IB_SP[t-1] of the blind spot area image I_B[t-1] is estimated or predicted for the current time point t. position, for example by means of a transformation or a method resembling a transformation, taking into account, in particular, the motion data v_x, v_y of the vehicle 1. These are provided, for example, by a vehicle system, for example a vehicle bus, or determined from camera images I_F, I_R, I_LL, I_LR. The estimation or prediction of the new position may be performed , for example, by a suitable image processing method. If the vehicle 1 moves further in the x direction, for example with v_x=15 m/s, a corresponding movement vector SP_V is determined for each blind spot area pixel IB_SP[t-1], which As region pixels IB_SP[t] are placed at respective new positions as suggested in FIG. 2A.

Subsequently, in step S3, the data processing device 120 determines whether the new position of each blind spot area pixel IB_SP remains within the blind spot area B at the current time t. In the example shown in FIG. 2A, each blind spot region pixel IB_SP that is shifted or transformed by the movement vector SP_V continues to be within the blind spot region B.

In this case ,
Each first blind spot area pixel IB_SP[t-1] is motion compensated in step S5A on the basis of the motion data of the vehicle 1, i.e. in particular by being shifted by the movement vector SP_V .
In step S4, each second blind spot region pixel IB_SP[t] is generated , e.g. rendered, in the data processing device 120 for the current time point t . As shown in FIG. 2A for two exemplary blind spot area pixels IB_SP, this preferably means that each first blind spot area pixel IB_SP[t-1] is placed in its new position and from there A second blind spot area image I_B[t] is generated , eg rendered , and repeated until it is displayed in the display image I[t] instead of the blind spot area B. Subsequently, the second blind spot area image I_B[t] is stored in the storage device 130 and, in the next execution of the method starting from step S1, the first blind spot area image I_B[t] is updated in content for a further second point in time t+n. It is provided as a region image I_B[t-1].

This can be generalized as follows .
I_B[x, y, t] = I_B[x+v_x, y+v_y, t-1],
In the formula, I_B is a blind spot area image, v_x and v_y are motion data in the x direction and y direction, t-1 is the first time point, and t is the current second time point.

In FIG. 2B, another possible case is shown , which is determined in step S3, in which the new position of each first blind spot region pixel IB_SP[t-1] is determined at the current time t in the blind spot . It is not within area B and is outside blind spot area B. Thus, in the example shown in FIG. 2B, each blind spot region pixel IB_SP shifted or transformed by the movement vector SP_V enters the field of view 141F of the onboard camera 140F due to the movement of the vehicle in the x direction. ing.

In this case, each first blind spot area pixel IB_SP[t] is synthesized in step S5B based on the camera image I_F[t-1] taken and provided at the first time point t-1, for example Each second blind spot area pixel IB_SP[t] for the current time point t is generated in step S4 by being rendered , for example rendered .

FIG. 2C shows the case determined in step S3 in the example of right curve travel , which includes movement in the x and y directions according to motion data v_x, v_y. As shown in FIG. 2C, each blind spot area pixel IB_SP shifted or transformed by the movement vector SP_V enters the viewing area 141LR of the on-vehicle camera 140LR due to the movement of the vehicle in the x direction.

In this case, in step S4, each first blind spot region pixel IB[t-1] is determined based on the camera image I_LR[t-1] taken and provided at the first time point t-1 in step S5B. , each second blind spot region pixel IB_SP[t] is generated for the current time point t by compositing, e.g. rendering , the respective first blind spot region pixel IB[t-1], e.g. rendered . _ As shown in FIG. 2B for two exemplary blind spot area pixels IB_SP, this preferably means that each first blind spot area pixel IB_SP[t-1] is placed in its new position and from there A second blind spot area image I_B[t] is generated , eg rendered , and repeated until it is displayed in the display image I[t] instead of the blind spot area B. Subsequently, the second blind spot area image I_B[t] is stored in the storage device 130 and, in the next execution of the method starting from step S1, the first blind spot area image I_B[t] is updated in content for a further second point in time t+n. It is provided as a region image I_B[t-1].

According to FIGS. 2B and 2C, in step S5B, the generation of the displayed blind spot area pixel IB_SP[t] is performed here by image synthesis, for example the camera image I_F[t-1] taken at time t-1. Alternatively, it is implemented by rendering I_LR[t-1]. Needless to say, this principle can be implemented similarly to other camera images I_R and I_LL.

This can be generalized as follows:
I_B [x, y, t] = I_F, I_R, I_LL, I_LR [x+v_x, y+v_y, t-1],
In the formula, I_B is the blind spot area image, I_F-I_LR is the camera image of the in-vehicle camera 140F-140LR, v_x and v_y are the motion data in the x direction and the y direction, t-1 is the first time point, and t is the , currently at the second point in time.

Furthermore, in step S6, which is an option, from the second blind spot area image I_B[t] and the current camera images I_F[t], I_R[t], I_LL[t], I_LR[t] at the second time t. , display image I for display device 110 are combined .

FIG. 3 summarizes the method described above, including optional step S0, steps S1-S5, and optional step S6, again as a flowchart.

Claims (10)

A method for displaying a vehicle surrounding area within a vehicle (1) at a current second time point (t), the method comprising:
This method consists of the following steps:
- At the first time point (t-1) before the second time point (t) , outside the viewing area (141F, 141R, 141LL, 141LR) of the in-vehicle camera (140F, 140R, 140LL, 140LR) a step (S1) of providing a first blind spot area image (I_B[t]) including image synthesis of a blind spot area (B) in the peripheral area of the vehicle where the vehicle is located;
- placing each first blind spot area pixel (IB_SP[t-1]) of the blind spot area (B) in a new position estimated for the second time point (t) (S2);
- determining (S3) whether the new position of each first blind spot area pixel (IB_SP[t-1]) is within the blind spot area (B) at a second time point (t);
If the new position is determined to be within the blind spot area (B), each first blind spot area pixel (IB_SP[t-1]) is added to the motion data (v_x, v_y) of the vehicle (1). a step (S4) of generating each second blind spot area pixel (IB_SP[t]) at the second time point (t) by combining (S5A) based on the second time point (t) ;
A method characterized by comprising : If the new position is determined to be outside the blind spot area (B), each first blind spot area pixel (IB_SP[t-1]) is captured at least one first time point (t-1). , the second time point is synthesized (S5B) using the provided camera images (I_F[t-1], I_R[t-1], I_LL[t-1], I_LR[t-1]). A method according to claim 1, characterized in that each second blind spot region pixel (IB_SP[t]) of (t) is generated (S4). Based on the motion vector (SP_V) of each first blind spot area pixel (IB_SP[t-1]) obtained from the motion data (v_x, v_y), each different viewing area (141F, 141R, 141LL, 141LR) ) Camera image (I_F[t-1], I_R[t-1], I_LL[t-1], I_LR[ 3. The method according to claim 2, characterized in that it is determined whether t-1]) is provided. In providing the camera image (I_F[t-1 ], I_R[t-1], I_LR[t-1], I_RR[t-1]) are selected. Exactly one camera image (I_F[t-1], I_R[t-1], I_LR[t-1], I_RR[t-1]) for each in-vehicle camera (140F, 140R, 140LL, 140LR) The method according to claim 3 or 4, characterized in that the method is maintained for providing only. according to one of the preceding claims, characterized in that exactly one blind spot area image (IB_SP[t-1]) at a first point in time (t-1) is maintained for providing. Method. A second blind spot area image (IB_SP[t]) generated for the second time point (t) from each second blind spot area pixel (IB_SP[t]) and at least one current image of the second time point (t). Any one of the preceding claims characterized in that the camera images (I_F[t], I_R[t], I_LL[t], I_LR[t]) are combined for display in the vehicle (1). The method described in section. At the time of initial startup , the display image (I) of the vehicle periphery is shifted into the visual field (141F, 141R, 141LL, 141LR) to the extent that the blind spot area (B) is hidden . A method according to any one of the claims. Method according to any one of the preceding claims, characterized in that the blind spot area (B) is arranged on the ground surface traveled by the vehicle (1). A device (100) for displaying a vehicle surrounding area within a vehicle (1) at a current second time point (t), the device (100) comprising:
This device (100) is
- at least one onboard camera (140F, 140R, 140LL, 140LR) for understanding the corresponding viewing area (141F, 141R, 141LL, 141LR) around the vehicle;
- The blind spot area (B) of the vehicle peripheral area located outside the visual field (141F, 141R, 141LL, 141LR) at the first time point (t-1) before the second time point (t) a storage device (130) in which exactly one blind spot area image (I_B[t-1] ) is maintained, including image composition;
- to carry out the following, namely:
i) placing each first blind spot region pixel (IB_SP[t-1]) of the blind spot region (B) in a new position estimated with respect to a second time point (t);
ii) determining whether the new position of each first blind spot region pixel (IB_SP[t-1]) at the second time point (t ) is still within the blind region (B);
iii) If it is determined that the new position is within the blind spot area (B), each first blind spot area pixel (IB_SP[t-1]) is assigned the motion data (v_x, v_y) of the vehicle (1). generating each second blind spot area pixel (IB_SP[t]) at a second time point (t) by combining (S5) based on ;
a data processing device (120) adapted to perform
An apparatus (100) comprising :