EP2374269A1

EP2374269A1 - Control device for a camera arrangement, camera arrangement for a vehicle and method for controlling a camera in a vehicle

Info

Publication number: EP2374269A1
Application number: EP09747802A
Authority: EP
Inventors: Martin Rous
Original assignee: Robert Bosch GmbH
Current assignee: Robert Bosch GmbH
Priority date: 2008-12-03
Filing date: 2009-10-14
Publication date: 2011-10-12
Also published as: WO2010063509A1; US20110261212A1; DE102008044322A1; US8797401B2

Abstract

The invention relates to a control device (10) for a camera arrangement (29), wherein the control device (10) has: at least one regulator (16) for recording image signals (1 1 ) from a camera (6) and for outputting or adjusting camera parameters (19, 20) for the camera (6), wherein the image signals (1 1) have a sequence of frames (Fi, i = 1, 2, ). The invention provides that the control device (10) subdivides the sequence of frames (Fi) into at least two sub-sequences (12, 14), and the at least one regulator (16) separately regulates the sub-sequences (12, 14) and outputs various camera parameters (19, 20) for regulating the sub-sequences. Separate regulators or a common regulator can be used in this case. Function modules (22, 23, 24, 25) preferably transmit state signals via interfaces to the at least one regulator. Furthermore, the camera arrangement is provided for a vehicle, and a method is provided for controlling the camera.

Description

description

title

Control device for a camera arrangement, camera arrangement for a vehicle and method for controlling a camera of a vehicle

The invention relates to a control device for a camera arrangement, a camera arrangement for a vehicle and a method for controlling a camera of a vehicle.

State of the art

In video systems of vehicles, the integration time or exposure time of a camera is adapted to the ambient light level. For this purpose, the video system or camera system is equipped with a camera control, the z. B. by evaluating a histogram or analysis of the image data, the ambient light brightness determined and concludes from the measured values to an integration time, which ensures a good image capture. As measuring criteria serve z. B. Contrast, mean, moments and percentiles.

In video systems in which a camera is permanently assigned to an application or function, the image quality optimization can be targeted by the camera control to the needs of the one application or function. Such applications or functions are z. B. lane detection, traffic sign recognition, light assistant or Trajektorienbildung or detection of moving objects (Moving Object Detection MOD), person detector,

Night vision.

The introduction of multi-purpose camera systems or MPC (Multi Purpose Camera) systems uses the image signals for multiple functions. As long as the functions or applications have the same or similar requirements for the image quality, the optimization of the image quality is directly feasible. It becomes more difficult when different functions make mutually exclusive demands on the image quality.

DE 698 08 024 T2 describes a method and an image recording device for generating a video signal with extended dynamic range. Such systems are also known as High Dynamic Range (HDR) images in which successive images are generated with different parameter setting, in particular with different exposure times, so that subsequently from image areas of different brightness, an optimized image can be created by z. B. dark areas from the longer exposed image and brighter areas are taken from the shorter exposed image. Here, the consecutive frames of the output image signal can be set differently, so that a sequence of different frames is generated. For this purpose different exposure times are fixed and subsequently the images are evaluated.

Patent Abstracts of Japan JP 2002165138 A describes a control device which records the continuous output of two images with different exposure times.

Disclosure of the invention

The invention is based on the idea of forming subsequences or subsequences from an image signal sequence or sequence of frames which the camera or its imager chip outputs, and to regulate these subsequences separately. The sequence of frames is output with different camera settings or camera parameters, preferably alternately or modulo the number of subsequences, so that z. B. each even frame is set with a first control and each odd frame with a second control. Thus, according to the invention of z. B. an HDR camera output sequence of frames as subsequences regulated differently. Here are the subsequences z. T. evaluated and then set the camera parameters. Preferably, the regulations with different control objectives work completely separate from each other. According to the invention, it is possible according to an embodiment that a common controller is used for the multiple controls, ie a multimodal control. In this case, in particular, a bimodal control for controlling two subsequences and a trimodal control for controlling three subsequences can be carried out. According to an alternative embodiment, arrangements with two separately operating controllers are also possible, which work synchronized on the different frames. Each controller regulates for itself with its control targets, z. B. two controllers in dual mode or three controllers in triplemode.

Furthermore, according to a preferred embodiment, the quality of the functions performed with the image signals is also assessed. According to the invention, more functions than subsequences or regulations can also be provided here, which is advantageous in particular in MPC systems. According to the invention, it can be provided that the functions or their functional modules output a signal about their state or the quality of their state or the quality of their output signals to the regulations. The controls pick up these output signals and use them to adjust the camera parameters. This can be considered in particular when a function module indicates a very poor quality or performance. Thereupon, one or more controls can change the camera parameters accordingly, e.g. For example, shorten or extend the integration time to improve the quality of the function.

The function modules can be designed, in particular, as software modules which input an input into the control algorithm via a suitable software interface, for B. a calculated quality measure of the function. This parameter transfer can also z. B. be the control window, which is to be used for the specific function for frame-synchronous camera control. Thus, the function can obtain a suitable exposure of the scene in its currently viewed image area.

The functional module can also be provided in another control device or to a different computer than the controller or the controller and intervene in the camera control via a suitable data connection. The invention also provides a computer program or computer program product for carrying out the method according to the invention on a computer, in particular also a data carrier which is written or stored with such a computer program.

Brief description of the drawings

1 shows a block diagram of a camera arrangement according to an embodiment of the invention with bimodal control and four functional modules;

Fig. 1 a further details of the camera arrangement in addition to FIG.

1 ;

2 shows a camera arrangement according to a further embodiment with dual-mode control; FIG. 33 shows a camera system according to a further embodiment with trimodal control; FIG.

4 shows a flowchart of a method according to the invention;

5 shows a street scene with a vehicle with the camera arrangement.

Figure 5 shows a street scene 1, in which a vehicle 2 on a road 3 with z. B. tracks 3a and 3b, which are bounded by lane markers 4a, 4b, 4c, which are designed as solid or dashed lines. The vehicle 2 has z. B. behind its windshield 5, a camera 6, in the detection area 7 at least a portion of the tracks 3a, 3b, the lane markers 4a, 4b, 4c, other vehicles 8 and arranged on or next to the lane 3 traffic sign 9 is located. The camera 6 outputs image signals 1 1 to a controller 10; As a camera 6 or part of the camera 6 is used in particular a CMOS or CCD chip, the output in a known manner, the image signals 1 1 as frames F1, F2, F3, ... consecutively. The camera 6 and the

Control device 10 form a camera assembly 29 which is provided in the vehicle 2.

According to the embodiment shown in Figure 1, the image signals 1 1 are output as a sequence of frames F1, F2, F3, F4, F5 ..., ie Fi with i = 1, 2, 3, .. successively from the camera 6 are read out. According to the invention, subsequences are formed from the sequence of frames Fi. In the embodiment of FIG. 1 with bimodal control, in a subdivision device 15 of the control device 10, a first subsequence 12 of the frames F1, F3, F5,..., Ie the odd-numbered frames i, and a second subsequence

Subsequence 14 of frames F2, F4, ..., i. the frames with even number i formed. Thus, the frames of the two subsequences 12, 14 alternate so that in every subsequence, every other frame or only the even or odd frames are always. The subdivision device can be designed purely in terms of software. Separate regulations are carried out on the two subsequences 12, 14. In the bimodal embodiment of FIG. 1, a common regulator 16 is provided which performs both controls and subsequently outputs or adjusts camera parameters 19, 20 with which the camera 6 is adjusted. The controller 16 is defined in a manner known per se by a control algorithm which is designed as software in the control device 10. He evaluates the frames Fi z. B. by means of a histogram or an analysis, wherein z. B. an ambient light brightness, contrast, mean, median, moments and percentiles, z. B. the gray tone stages, determined and assessed; From this, the camera parameters 19, 20 are determined which, in particular, the integration time, also z. B. offset and electrical gain (gain) of

Set the camera or its pixels; Further, the aperture, black level, and the shape of the exposure sensitivity characteristic can be used.

Here, the camera parameters 19 are used to set the frames for the first

Subsequence 12, d. H. the odd frames F1, F3, F5, and correspondingly the second camera parameters 20 for setting the frames F2, F4,... of the second subsequence 14. The camera 6 is thus set alternately according to the camera parameters 19 and 20.

The controller 16 pursues different control objectives for the two regulations. They are defined by predetermined functions, which are shown as function modules 22, 23, 24, 25, in particular software function modules 22, 23, 24, 25. You can z. B. be realized in a memory device 17 as a program. Such function modules can, in particular, be the lane recognition 22, the traffic sign recognition 23, the light assistant 24 and the trajec- Torienbildung or detection of moving objects (Moving Object Detection MOD) 25, which are stored in the software of the controller 10 in a conventional manner as software modules, each function module 22, 23, 24, 25 from the frames Fi, i = 1, 2 ... works. If only two functional modules 22, 23 are provided, each control module 22, 23 can be assigned a control and a control target, so that z. B. the lane detection 22 on the first subsequence 12 works. Since the lane detection for detecting the lane markings 4a, 4b, 4c in Figure 5 in particular over greater distances from the vehicle 2 is carried out and thus a higher intensity is advantageous here z. B. no restrictions on the integration time provided; relevant is a lower screen area. The function module of the traffic sign recognition 23 tends to require different integration of different traffic signs 9 rather short integration times, z. Up to a maximum of 15 ms, so that the contours are detected sharply, so that in the control of the second subsequence 14 short integration times are to be striven for, with laterally external image areas being relevant. Thus, the camera parameters 19, 20 will generally differ. In such a simple embodiment with only two functional modules 22, 23, the control targets can be integrated directly into the camera control algorithm and a respective quality target can be achieved frame-synchronously in the sub-sequences 12, 14. The recording of the image data is shown in FIG. 1 a for the sake of clarity, according to which the function module 22 receives the frames of the first subsequence 12 and the function module 23 receives the frames of the second subsequence 14, the function modules 22, 23 corresponding to their evaluation or evaluation of the Output image data S5, S6, z. B. for display to the driver and / or for a vehicle intervention.

In the embodiment shown in FIG. 1, however, two further functional modules 24, 25 are provided, which accordingly pursue further or overlapping goals. The functional module of the light assistant 24 is designed to detect other road users 8 even at a greater distance in order to switch off the main beam, if necessary. Thus, longer integration times are advantageous here. In the functional module of the detection of moving objects 25 is provided to recognize structures even in dark areas; For calculating the trajectory of the moving objects, however, a sufficiently precise contour detection makes sense. Thus, the function modules 22, 23, 24, 25 according to FIG. 1 a basically each access both subsequences 12, 14. You can hereby z. B. different image areas of the frames Fi, i = 1, 2, 3 ... examine, in which relevant contours are expected. They output accordingly output signals S7, S8, z. B. as an indication to the driver or for automatic vehicle intervention, z. B. Dimming the high beam.

According to the invention, as further functional modules, it is also possible to provide a person detection and / or a night vision instead of the functional modules mentioned here or also in addition to the functional modules mentioned here.

According to the invention, the function modules 22, 23, 24, 25 give status signals

S1, S2, S3, S4 to the controller 16 and its control algorithm, which takes into account when creating the camera parameters 19, 20 accordingly. For this purpose, in particular one or more software interfaces 26-1, 26-2, 26-3, 26-3 in the control algorithm of the controller 16 and one or more software interfaces 22a, 23a, 24a, 25a in the function modules 22, 23 , 24,

25 be formed. The status signals S1, S2, S3, S4 can in particular contain a quality assessment which contains an evaluation of their own status and / or the quality of their output signals S5, S6, S7, S8. You can thus z. For example, it may just be a simple scalar value between a minimum and a maximum; however, they can also contain more complex data. Of the

In this case, controller 16 takes into account in particular if a status signal S1 to S4 assumes a very poor value in order to possibly modify the control target of one or both of the subsequences 12, 14 in such a way that this functional target improves and, if appropriate, the functional target of one of the other functional modules, the is better according to its state signal is limited. In this case, 16 criteria for the consideration of the state signals S1 to S4 can be set in the controller. In particular, priorities can be set so that one of the function modules can work preferentially. Furthermore, z. B. lower limit values for the state signals S1 to S4 are set, below which an improvement of this functional module is sought.

According to the invention, the functional modules 22, 23, 24, 25 can also be provided outside the control device 10, in which case the data and signal transmission to the interfaces 26-1, 26-2, 26-3, 26-4 via a suitable data connection he follows. FIG. 2 shows a further embodiment with dual-mode control, in which two regulators 16a, 16b are provided for the two controls instead of the regulator 16 from FIG. The status signals S1 to S4 can hereby be output completely or partially to both controllers 16a, 16b. According to FIG. 2, z. B. provided that the light assistant 24 and the detection of moving objects 25 their state signals S 3 and S4 to both controllers 16a, 16b output. The regulators 16a, 16b thus each operate independently and can, in particular, represent different control algorithms or can also be designed as different controllers in terms of hardware. The function modules 22, 23, 24, 25 can also output different status signals to the different controllers 16a, 16b if different criteria are relevant to the controllers 16a, 16b.

FIG. 3 shows a corresponding subdivision of the image signals 1 1 in a trimodal control with three subsequences 12, 13, 14 and a common controller 16, camera parameters 21 correspondingly being additionally output to the camera 6, which thus comprise three successive frames in each case sets different.

In an embodiment with triplemode control, not shown, three separate regulators are used; with which three subsequences 12, 13, 14 are regulated in accordance with FIG.

According to the method according to the invention of FIG. 4, the camera 6 thus acquires images in the control circuits in step 30 and outputs image signals 1 1 as a result of frames Fi, which are subsequently divided into two or more subsequences 12 and 14 or 12, 13 in step 31 , 14 are divided. In step 32, the evaluation of the subsequences 12, 14 or 12, 13, 14 follows by the function modules 22, 23, 24, 25. In step 33, status signals S1, S2, S3, S4 are output. In step 34, the output of the control parameters 19, 20 or 19, 20, 21 by the controller 16 and the controller 16 a, 16 b, taking into account the

State signals S1 to S4.

Claims

claims

1 . Control device (10) for a camera arrangement (29), wherein the control device (10) comprises: at least one controller (16; 16a, 16b) for recording image signals (1 1) of a camera (6) and output or setting of camera parameters (19 .

20, 21) for the camera (6), the image signals (1 1) comprising a sequence of frames (Fi, i = 1, 2, ...), characterized in that the control device (10) is the sequence of frames (Fi) is subdivided into at least two subsequences (12, 13, 14) and the at least one controller (16; 16a, 16b) controls the subsequences (12, 13, 14) separately and different camera parameters (19, 20, 21) for the Regulation of subsequences (12, 13, 14) outputs.

2. Control device according to claim 1, characterized in that the subsequences (12, 13, 14) are completely disjoint and the frames (Fi) of the different subsequences (12, 13, 14) in the sequence of frames (Fi) of the image signal ( 1 1) are arranged alternately one behind the other.

3. Control device according to one of the preceding claims, characterized in that it further comprises: at least two internal function modules (22, 23, 24, 25) for receiving and evaluating the frames (Fi) and / or the subsequences (12, 13, 14 ).

4. Control device according to one of the preceding claims, characterized in that the at least one controller (16, 16a, 16b) further comprises interfaces (26-1, 26-2, 26-3, 26-4) for the internal function modules (22 , 23, 24, 25) or for external function modules for receiving and evaluating the frames (Fi) and / or the subsequences (12, 13, 14).

5. Control device according to claim 3 or 4, characterized in that the at least one controller (16; 16a, 16b) or an evaluation device connected to the controller of the internal function modules (22, 23, 24, 25) or external function modules state signals (S1 , S2, S3, S4) about their own condition and / or condition or quality of theirs

Output signals (S5, S6, S7, S8) receives, and the at least one controller (16; 16a, 16b) considered in at least one control of at least one of the state signals (S1, S2, S3, S4).

6. Control device according to claim 5, characterized in that the function modules (22, 23, 24, 25) are designed as software function modules at least one in a memory device (17) of the control device (10) stored program, and in a control algorithm of at least a controller (16, 16a, 16b) at least one software interface (26-1, 26-2, 26-3, 26-4) for receiving the state signals (S1, S2, S3, S4) is formed.

7. Control device according to claim 5 or 6, characterized in that the at least one controller (16, 16a, 16b) in the case that a state signal (S1, S2, S3, S4) of a function module (22, 23, 24 , 25) indicates a bad state of the functional module and / or a bad state or a poor quality of its output signal (S5, S6, S7, S8), the at least one controller (16, 16a, 16b) at least one of the controls changed such that the state of the first functional module and / or the state or the quality of its output signal (S5, S6, S7,

S8) is improved.

8. Control device according to claim 7, characterized in that in the case that a state signal (S1, S2, S3, S4) of a first functional module (22, 23, 24, 25) a bad state of the functional module and / or a bad state or indicates a poor quality of its output signal (S5, S6, S7, S8) and a state signal of a second functional module indicates a better state of the functional module and / or a better state or a better quality of its output signal, the at least one controller (16, 16a, 16b) at least one of the regulations changed such that the state of the first functional module and / or the state or quality of its output signal (S5, S6, S7, S8) is improved.

9. Control device according to claim 8, characterized in that the function modules (22, 23, 24, 25) are assigned priorities and the control device to the status signals (S1, S2, S3, S4) of the function modules (22, 23, 24, 25 ) Responding to their priority.

10. Control device according to one of the preceding claims, character- ized in that it comprises a single common regulator (16) for the at least two controls of the at least two subsequences (12, 13, 14).

1 1. Control device according to one of Claims 1 to 9, characterized in that, for each subsequence (12, 13, 14), it has its own controller (16a,

16b) which respectively outputs camera parameters (19, 20, 21) for setting the camera (6) for the respective subsequence (12, 13, 14).

12. Control device according to one of the preceding claims, character- ized in that the function modules (22, 23, 24, 25) from the following

Group of function modules are formed: a track recognition module (22), a traffic sign recognition module (23), a light assist module (24), a moving object detection module (25).

13. Control device according to one of the preceding claims, characterized in that the number of function modules (22, 23, 24, 25) is greater than the number of subsequences (12, 13, 14) and at least one function module (22, 23, 24 , 25) accesses frames (Fi) from at least two subsequences (12, 13, 14).

14. Control device according to one of the preceding claims, characterized in that the camera parameters (19, 20, 21) of the control in each case comprise at least one element from the following group: integration time, aperture, offset, electrical gain (gain),

Black level, the shape of the exposure sensitivity curve.

15. Control device according to one of the preceding claims, characterized in that the at least one controller determines (16, 16a, 16b) in the evaluation of the frames: a list of a histogram of the brightness and / or gray levels, a contrast, an average, a Moment and / or a percentile.

16. Camera arrangement (29) with a camera (6) and a control device (10) according to one of the preceding claims.

17. Camera arrangement according to claim 16, characterized in that it has a further memory device separate from the control device (10), in which external functional modules (22, 23, 24, 25) are stored, wherein between the further memory device and the control device (10 ) a data connection is provided.

18. Vehicle with a camera arrangement according to claim 16 or 17, wherein the camera (6) in the region of a vehicle window (15) or a bumper is arranged for monitoring an outdoor area, in particular a street scene (3), or an interior area in the vehicle.

19. Method for controlling a camera (6) with at least the following steps:

Capturing an image by the camera (6) and outputting image signals (1 1) as a result of frames (Fi, i = 1, 2, ...) (30),

Formation of subsequences (12, 13, 14) in the image signals (1 1) (31), evaluation or evaluation of the subsequences (12, 13, 14) (32), regulation of the subsequences (12, 13, 14) by issuing Camera parameters (19, 20, 21) to the camera (6) for outputting the image signals (1 1) (34).

20. The method according to claim 19, characterized in that functions are performed in which the image data are judged according to different measurement criteria, for. B. lane detection, traffic sign recognition, light assistant, detection of moving objects / T jektjektbildung, person detection, night vision, and by the functions state signals (S1,

S2, S3, S4) are issued to the regulations (33), whereby the regulations gene in dependence of the recorded state signals (S1, S2, S3, S4) are performed.

A computer program comprising computer-executable instructions for carrying out a method according to claim 19 or 20 on a computer or for operating a control device according to any one of claims 1 to 15.