DE102020132825A1 - METHOD OF OPERATING A CAMERA DEVICE, VEHICLE AND COMPUTER PROGRAM PRODUCT - Google Patents

Abstract

A method for operating a camera device (3) arranged in a vehicle (1) behind a vehicle window (2) is proposed. The method comprises the steps: a) recording an image of an object (O) arranged in front of the vehicle window (2); andb) focusing the image while correcting for vehicle window optical properties.

Description

The present invention relates to a method for operating a camera device, a vehicle with the camera device and a computer program product.

Conventionally, vehicles include a front camera mounted on an inside of a windshield (vehicle front glass) of a vehicle. The front camera captures stationary or moving objects outside the vehicle through the windshield. Environmental influences, such as sunshine or outside temperatures below freezing, can cause changes in the material properties of the windshield. The changed material properties can in turn lead to falsifications in the recorded image and/or video data. This is particularly problematic in the case of high-resolution cameras, which are particularly sensitive to these falsifications.

Against this background, it is an object of the present invention to provide an improved camera device.

1. a) Aufnehmen eines Bildes eines vor der Fahrzeugscheibe angeordneten Objekts; und
2. b) Scharfstellen des Bildes unter Korrektur von optischen Fahrzeugscheiben-Eigenschaften.

According to a first aspect, a method for operating a camera device arranged in a vehicle behind a vehicle window is proposed. The procedure includes the steps:

1. a) recording an image of an object arranged in front of the vehicle window; and
2. b) Focusing the image while correcting optical vehicle window properties.

Due to the correction of the optical properties of the vehicle pane, more precise sensor data, such as image and/or video data, can be obtained. Since this sensor data serves as the basis for further processing in a driver assistance system in the vehicle, its reliability and accuracy are increased.

Advantageously, high-resolution cameras (high definition cameras (HD)) can also be integrated in the vehicle behind the vehicle window. Although the HD cameras have high sensitivity to deviations in vehicle glass optical characteristics, this integration is possible due to the above correction of vehicle glass optical characteristics. Their use improves the database for the data processing of the driver assistance system.

It is also advantageously possible to install vehicle windows in the vehicle with lower demands on their material and their production, since the optical vehicle window properties of these vehicle windows are corrected. This in turn leads to a reduction in manufacturing costs.

In particular, the camera device is mounted in a vehicle, such as a motor vehicle. The vehicle is, for example, a passenger car, a truck, a bus, a motorcycle, an aircraft and/or a watercraft. The camera device can also be used in vehicles that are operated autonomously or at least partially autonomously.

The camera device is arranged, for example, on or under a vehicle window, in particular a windshield (front vehicle window) of the vehicle. The camera device is preferably arranged on an inside of the windshield and is directed in the direction of travel of the vehicle. The windshield is located on an optical path between the camera and an object located outside of the vehicle. The camera device can also be arranged and oriented differently. The camera device can also be arranged on or under a rear window (rear vehicle window) of the vehicle and thus be directed in the opposite direction to the direction of travel of the vehicle.

The camera device has a camera for recording an image, in particular an image of the surroundings of the vehicle. Appropriate monitoring areas in front of, behind, next to, above and/or below the vehicle can be monitored with the camera device by a suitable arrangement of the camera device. In particular, stationary objects (or static objects) and/or moving objects (or dynamic objects) in the area surrounding the vehicle can be detected, such as vehicles, people, animals, plants, obstacles, bumps in the road (e.g. potholes or stones). ), lane boundaries, traffic signs or open spaces (e.g. parking spaces). In particular, the image can reproduce or represent a real and/or current scene in the area surrounding the vehicle. In particular, the image can be recorded when the vehicle is not moving (or stationary). Alternatively or cumulatively, however, the image can also be recorded when the vehicle is driving (or moving).

A driver assistance system has, for example, speed assistance (adaptive cruise control, distance control, ACC: Adaptive Cruise Control), lane assistance, lane keeping assistance (LKA: Lane Keep Assist), lane change assistance, emergency braking assistance (AEB: Automatic Emergency Braking), steering assistance, emergency steering assistance (AES: Automatic Emergency Steering), traffic jam assistance (TJA: Traffic Jam Assist), light assistance and/or high beam assistance. The driver assistance system(s) is (are) implemented in particular by software. In the case of a software implementation, the driver assistance system can be embodied as a computer program product, as a function, as a calculation routine, as an algorithm, as part of a program code or as an executable object.

The camera device is installed in particular behind the windshield in the interior of the vehicle by a vehicle original equipment manufacturer (OEM). At the time of installation of the camera device, the camera of the camera device is preferably in an out-of-focus state. In particular, the method according to steps a) and b) is carried out only after the windshield has been installed.

In particular, focusing occurs over a range from half a hyperfocal distance to infinity.

Correcting the vehicle pane optical properties can also be referred to as calibrating out or compensating for the vehicle pane optical properties using the camera device.

According to one embodiment, the vehicle pane optical properties include an index of refraction.

The refractive index can in particular be an optical material property of the vehicle pane. Furthermore, the following optical vehicle pane properties, for example, can also be taken into account in the correction: a reflection index, a transmission index, an absorption index and/or an extinction coefficient.

According to a further embodiment, the focusing takes place by means of an autofocus lens of the camera device.

Advantageously, by using (adjusting) the autofocus lens, automatic focusing can take place through the vehicle window onto an object in front of the vehicle window. In particular, the camera device does not require a predefined test image for this automatic process.

In particular, during automatic focussing using the autofocus lens, the image is focused by control signals until it has a desired (depth) focus, which is defined using a threshold value within the image processing algorithm. The image processing algorithm uses in particular a Fourier transformation and/or an edge contrast measurement (edge detection). In particular, the position of the autofocus lens within the camera device is moved back and forth by control signals until the image has the desired sharpness. This can also be referred to as mechanical focusing.

Furthermore, the camera has in particular an image sensor, e.g. B. a CCD sensor (Charged Coupled Device) or a CMOS sensor (Complementary Metal-Oxide Semiconductor sensor). The camera has, for example, a printed circuit board on which electrical and/or electronic components, such as the image sensor, are arranged. The camera device also has, for example, a memory unit for storing the images, videos and/or other information recorded by the camera, and an image processor which, in particular, carries out the image processing of images and/or videos recorded by the camera using the image processing algorithm.

According to a further embodiment, focusing takes place exclusively when the camera device is put into operation for the first time and/or when it is put into operation again.

Initial start-up is understood in particular to mean that the focusing takes place after the camera device is first installed behind the vehicle window, when the camera device is switched on for the first time using an external energy supply, such as a power supply unit, or when the vehicle engine is started for the first time.

Restarting is preferably understood to mean that the focusing is carried out during maintenance of the vehicle, for example in a workshop, after a software update of the vehicle and/or the camera device, after a vehicle window has been replaced or when the vehicle ignition is activated to switch the vehicle on again he follows.

According to a further embodiment, focusing occurs continuously while the vehicle is in operation.

The continuous focusing has the technical effect that when correcting the optical properties of the vehicle pane, thermal properties of the vehicle pane are also taken into account can be taken into account. For example, the vehicle window expands in strong sunlight and contracts in temperatures below freezing. The expansion and contraction change the optical vehicle window properties and thus also the recording of an image and/or video and are also corrected by the automatic focusing.

In particular, continuous focus includes real-time automatic focusing while the vehicle is in operation. The continuous focusing can also be done at predetermined time intervals, such as every 5, 10, 15, 20, 30 seconds or even every minute, every two minutes, every three minutes, every five minutes or a longer period of time. In particular, the continuous focusing takes place at predetermined time intervals of 10 seconds to five minutes.

In particular, before continuous focusing (or in a mode that is not yet focused), the autofocus lens can be moved to a predetermined position. In particular, this position can serve as a starting point for continuous focusing.

In particular, the continuous focusing can include starting an image processing algorithm in the camera device, by means of which the (continuous) focusing method is carried out. This may include first determining a section (or “section of interest”) in which the object or image to be focused is located. In particular, a focusing result (for example a so-called focus score) can be calculated, in particular using a Fourier transformation, and stored. In particular, the focusing result can be calculated based on the predetermined position of the autofocus lens. Since the predetermined position is only a (rough) starting point for the focusing process, the focusing result can be correspondingly imprecise when it is processed for the first time.

In particular, it can then be determined whether the determined focusing result is greater than or equal to a specific minimum value. If this is not the case, the autofocus lens can be moved to a next possible (predefined) position. In particular, this can mean that the autofocus lens is or was in a position that is or was still a long way from optimal focusing. A focusing result can then be calculated again for the next possible position of the autofocus lens and it can then be determined again whether the focusing result determined is greater than or equal to a specific minimum value. A stored focusing result can in particular correspond to a specific position of the autofocus lens.

In particular, if the determined focusing result is greater than or equal to a specific minimum value, it can be checked whether the determined focusing result is better than a (previously) stored focusing result. If this is the case, the position of the auto focus lens that corresponds to the determined in-focus result can be stored as the new position of the auto focus lens.

In particular, the autofocus lens can then be moved from the new position to a position that is close to the new position. This can be done in order to find out whether a (even) better focusing result can be achieved than the determined focusing result. For example, the position of the autofocus lens can be shifted by a few millimeters or micrometers in relation to the new position.

In particular, when the detected focus result is not better than a memorized focus result, the autofocus lens can be moved back to the position corresponding to the memorized focus result that last gave the best focus result (last memorized focus result). In particular, it is then possible to initially wait until a specific time has elapsed. The autofocus lens can then be set to the setting of the last focus result stored. The specific time can be several seconds or several minutes, for example. In this specific time, it is possible for the vehicle and/or the object O to move. This may require refocusing. Specifically, after the specified time has elapsed, the position of the auto focus lens may be moved back a predetermined amount or distance. The image processing algorithm or the (continuous) focusing can then be carried out again. For this purpose, the steps of the image processing algorithm or the (continuous) focusing can be carried out again for the autofocus lens that has been moved back. This can be done in order to calculate a focusing result, particularly for the autofocus lens that has been moved back.

In one embodiment, the method may include determining a focus mode of the camera device. In particular, it can there are at least two focusing modes to which the camera device can (alternatively) be set, for example an initial focusing mode (INI) and a continuous focusing mode (KON). The initial focus mode can be used to focus the camera device once. Continuous focus mode can be used to focus the camera device multiple times. In particular, after it has been switched on, the camera device can check which of the two focusing modes the camera device is currently in. If it is in the initial focus mode, it can be checked whether the camera device has already been focused once. If this is the case, the autofocus lens can be moved to the position that was set when it was previously focused. After the auto focus lens has been moved to this position, the normal operating mode (NOM) of the camera device can then occur. In this, the camera device can preferably be set up to record an image and/or a video.

According to a second aspect, a vehicle with a camera device arranged in the vehicle behind a vehicle window of the vehicle is proposed. The camera device is set up to record an image of an object arranged in front of the vehicle window and to focus the image by correcting optical vehicle window properties.

Furthermore, a computer program product is proposed which comprises instructions which, when the program is executed by a computer, cause the latter to execute the method described above.

A computer program product, such as a computer program means, can be made available or supplied by a server in a network, for example, as a storage medium such as a memory card, USB stick, CD-ROM, DVD, or in the form of a downloadable file. This can be done, for example, in a wireless communication network by transferring a corresponding file with the computer program product or the computer program means.

The embodiments and features described for the proposed method apply accordingly to the proposed vehicle.

Further possible implementations of the invention also include combinations of features or embodiments described above or below with regard to the exemplary embodiments that are not explicitly mentioned. The person skilled in the art will also add individual aspects as improvements or additions to the respective basic form of the invention.

• 1 zeigt eine Seitenansicht eines Fahrzeuges mit einer Kameravorrichtung;
• 2 zeigt ein Flussablaufdiagramm eines Verfahrens zum Betreiben der Kameravorrichtung aus 1; und
• 3 zeigt ein detailliertes Flussablaufdiagramm eines Scharfstellverfahrens zum Betreiben der Kameravorrichtung gemäß 2.

Further advantageous refinements and aspects of the invention are the subject matter of the dependent claims and of the exemplary embodiments of the invention described below. The invention is explained in more detail below on the basis of preferred embodiments with reference to the enclosed figures.

• 1 shows a side view of a vehicle with a camera device;
• 2 FIG. 12 shows a flowchart of a method for operating the camera device 1 ; and
• 3 FIG. 12 shows a detailed flow chart of a focusing method for operating the camera device according to FIG 2 .

Elements that are the same or have the same function have been provided with the same reference symbols in the figures, unless otherwise stated.

1 shows a side view of a designed as a passenger vehicle vehicle 1. In 1 the camera device 3 is arranged behind a vehicle window 2 in the vehicle 1 . The vehicle pane 2 is designed here as a front vehicle pane (windshield). The camera device 3 is set up to record an image of an object O arranged in front of the vehicle window 2 in the direction of travel R and to focus the image by correcting optical properties of the vehicle window. The camera device 3 can be used for various driver assistance systems, such as for lane keeping assistance.

In other embodiments, the camera device 3 can also be attached to the rear vehicle window in the vehicle 1 and be oriented in the opposite direction to the direction R of travel.

The camera device 3 also has a housing with an autofocus lens 4 . The camera device 3 is attached to the vehicle front window 2, for example by means of a frame or a bracket. In 1 For example, the camera device 3 captures at least one image (or a video) of an object O located outside the vehicle through the windshield 2 . The image (or multiple images in the case of a video) is then focussed using the autofocus lens 4 of the camera device 3, with the optical properties of the vehicle pane being corrected in the course of the focussing.

2 1 shows a flow chart of a method for operating a camera device 3 arranged in a vehicle 1 behind a vehicle window 2 1 with steps S1 and S2.

In step S1 , the camera device 3 takes an image of an object O arranged in front of the vehicle window 2 in the vicinity of the vehicle 1 . Next, in step S2, the image is focused while correcting optical vehicle window properties.

The focussing takes place in particular by means of an autofocus lens 4 of the camera device 3. For example, the optical vehicle window properties include a refractive index.

Furthermore, the focusing takes place exclusively when the camera device 3 is started up for the first time and/or when it is started up again. Alternatively, the focusing can be carried out continuously during operation of the vehicle 1 .

3 FIG. 12 shows a detailed flow chart of a focusing method, hereinafter referred to as an algorithm, for operating the camera device 3 according to FIG 2 with steps S10 - S22. 3 is divided in particular into a first section 10 and a second section 20 . A focusing mode of the camera device 3 is determined in the first section 10 . After that, in the second section 20, a focusing process for focusing an image by means of the autofocus lens 4 is carried out.

The first section 10 includes steps S10 to S13 and starts with step S10. In step S10, the focusing mode of the camera device 3 is determined. In particular, there are at least two focusing modes, namely an initial focusing mode INI and a continuous focusing mode KON, to which the camera device 3 can be set. The initial focusing mode INI is used to focus the camera device 3 once. The continuous focus mode KON is used to focus the camera device 3 multiple times. For this purpose, the camera device 3 is first switched on in S10. The algorithm then checks which of the two focusing modes the camera device 3 is currently in.

If the initial focus mode INI is present, the algorithm jumps to step S11. In S11 it is checked whether the camera device 3 has already been focused once. If this is the case (YES), the algorithm jumps to step S12, in which the autofocus lens 4 is moved to the position which was set in the previous focusing. After the auto focus lens 4 is moved to this position, step S12 is ended. The algorithm then jumps to the normal NOM operating mode of the camera device 3. In this mode, the camera device 3 is preferably set up to record an image and/or a video.

If the camera device 3 has not yet been focused once, the algorithm jumps from S11 to step S13 (NO). Then, in S13, the auto-focus lens 4 is moved to a predetermined position serving as a starting point for the focusing process in the second section 20. FIG. After the algorithm has processed S13, it jumps to a step S14, which will be explained later, in the second section 20.

Alternatively, if the continuous focus mode KON is present, the algorithm jumps straight from S10 to step S13, in which the camera device 3 is first set in an out-of-focus mode. Then, the auto-focus lens 4 is moved to a predetermined position serving as a starting point for the focusing process in the second section 20. FIG. After the algorithm S13 has processed, it jumps to step S14 in the second section 20.

In the second section 20, the actual focusing process is now carried out using steps S14 to S22. In step S14, an image processing algorithm is started in the camera device 3, by means of which the focusing method is carried out.

In the course of step S14, a “section of interest” is defined, ie a section in which the object O or the image that is to be focused is located. The algorithm then jumps from S14 to S15. In step S15, a focusing result (for example a so-called focus score) is calculated, in particular using a Fourier transformation, and stored. Specifically, the in-focus result is calculated based on the predetermined position of the autofocus lens 4 from S13. Since the predetermined position from S13 is only a (rough) starting point for the focusing method, the focusing result is correspondingly imprecise when step S14 is processed for the first time. After S15 has been processed, the algorithm jumps to step S16.

In step S16, it is now determined whether the in-focus result determined in step S15 is greater than or equal to a certain minimum value. If not (NO), the algorithm jumps to S18. "NO" means in particular that the autofocus lens 4 is at a position that of optimal focus is still far away. In S18, therefore, the autofocus lens 4 is moved to a closest possible (predefined) position. The algorithm then returns to step S15. In S15, a focusing result is calculated again for the closest possible position of the autofocus lens 4, and then step S16 is carried out again.

If the focusing result determined in S16 is greater than or equal to a certain minimum value (YES), the algorithm jumps to S17. In particular, "YES" means that the autofocus lens 4 is in a position that is close to optimum focusing.

In steps S17 to S22, it is checked in particular whether the determined focusing result, which was determined in S16, is better or worse than a focusing result stored in the camera device 3 in the past. A saved focusing result corresponds to a specific position of the autofocus lens 4.

In S17 it is checked whether the determined focusing result is better than a stored focusing result. If so (YES), the algorithm jumps to step S19. In S19, the position of the auto focus lens 4, which corresponds to the detected in-focus result, is stored as a new position of the auto focus lens 4. In order to find out whether an (even) better focusing result can be achieved than the determined focusing result, the autofocus lens 4 is shifted from the new position to a position that is close to the new position. For example, the position of the autofocus lens 4 is shifted by a few millimeters or micrometers in relation to the new position. When this has happened, the algorithm jumps back to step S15. Then, in S15, a picture is taken again with this shifted setting of the autofocus lens 4, and a focus result is calculated. Steps S16, S17 and S19 or step S20 described later are then run through again.

If in S17 the detected in-focus result is not better than a memorized in-focus result (NO), the algorithm jumps to step S20. In S20, the auto-focus lens 4 is returned to the position corresponding to the stored in-focus result which last gave the best in-focus result (last stored in-focus result).

At this point, it is then checked in S20 whether the initial focusing mode INI of the camera device 3 mentioned in the first section 10 is present. If this is the case (YES), the camera device 3 is declared as being “focused” once or initially with the last saved focussing result, and the camera device 3 ends the focussing process. The algorithm then jumps to the above-explained normal operating mode NOM of the camera device 3. The camera device 3 can then record at least one image and/or video with the autofocus lens 4, which corresponds to the last focussing result stored.

If in S20 the mentioned initial focus mode INI is not present (NO), the algorithm jumps to step S21. Since the initial focusing mode INI is not present, this means that the camera device 3 is in the continuous operating mode KON—that is, in a mode in which continuous focusing is performed. Thus, step S21 and subsequent step S22 are performed in the course of continuous focusing.

In S21, there is first a wait until a specific time has elapsed. Furthermore, in S21, the autofocus lens 4 is set to the setting of the last memorized focusing result. The specific time is, for example, several seconds or several minutes. In this specific time, it is possible that the vehicle 1 and/or the object O will move, which may require refocusing. After the specified time has elapsed, the algorithm jumps to step S22. In S22, the position of the auto focus lens 4 is moved back a predetermined amount or distance. The algorithm then returns to step S14. Steps S14 et seq. are then carried out again for the autofocus lens 4 that has been moved back in order to calculate a focusing result for the autofocus lens 4 that has been moved back.

Although the present invention has been described using exemplary embodiments, it can be modified in many ways.

Reference List

1

vehicle

2

vehicle window

3

camera device

4

auto focus lens

10

first section

20

second part

CON

continuous focus mode

INI

initial focus mode

NOM

normal operating mode

O

object

R

driving direction

S1 - S2

process steps

S10 - S13

Procedure Steps (Focusing Mode)

S14 - S22

Process steps (focusing process)

Claims (9)

Method for operating a camera device (3) arranged in a vehicle (1) behind a vehicle window (2), with the steps: a) recording an image of an object (O) arranged in front of the vehicle window (2); and b) Focusing the image while correcting optical vehicle window properties. procedure after claim 1 , wherein the vehicle pane optical properties include an index of refraction. procedure after claim 1 or 2 , wherein the focusing takes place by means of an autofocus lens (4) of the camera device (3). procedure after claim 3 , wherein the focusing takes place exclusively when the camera device (3) is put into operation for the first time and/or when it is put into operation again. procedure after claim 3 , wherein the focusing during operation of the vehicle (1) takes place continuously. procedure after claim 5 , wherein the continuous focusing comprises real-time automatic focusing during operation of the vehicle (1). procedure after claim 5 or 6 , wherein the continuous focusing occurs at predetermined time intervals. Computer program product, comprising instructions which, when the program is executed by a computer, cause the latter to carry out the method according to one of Claims 1 - 7 to execute. Vehicle (1) with a camera device (3) arranged behind a vehicle window (2) of the vehicle (1) in the vehicle (1), the camera device (3) being set up to take an image of an object ( O) and is also set up to sharpen the image by correcting optical vehicle window properties.

Images