WO2014019602A1 - Method and system for optimizing image processing in driver assistance systems - Google Patents

Method and system for optimizing image processing in driver assistance systems Download PDF

Info

Publication number
WO2014019602A1
WO2014019602A1 PCT/EP2012/064885 EP2012064885W WO2014019602A1 WO 2014019602 A1 WO2014019602 A1 WO 2014019602A1 EP 2012064885 W EP2012064885 W EP 2012064885W WO 2014019602 A1 WO2014019602 A1 WO 2014019602A1
Authority
WO
WIPO (PCT)
Prior art keywords
image data
imager
data
unit
driver assistance
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/EP2012/064885
Other languages
French (fr)
Inventor
Albrecht Neff
Stefan Singer
Jochen GERSTER
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Bayerische Motoren Werke AG
NXP USA Inc
Original Assignee
Bayerische Motoren Werke AG
Freescale Semiconductor Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Bayerische Motoren Werke AG, Freescale Semiconductor Inc filed Critical Bayerische Motoren Werke AG
Priority to PCT/EP2012/064885 priority Critical patent/WO2014019602A1/en
Publication of WO2014019602A1 publication Critical patent/WO2014019602A1/en
Priority to US14/608,357 priority patent/US20150139500A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/98—Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns
    • G06V10/993—Evaluation of the quality of the acquired pattern
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/44—Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/46—Embedding additional information in the video signal during the compression process
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/90—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/14—Picture signal circuitry for video frequency region

Definitions

  • the present document relates to image processing.
  • the present document relates to the optimization of image processing in a driver assistance system of a vehicle.
  • Camera-based image/video applications e.g. automotive driver assistance camera functions
  • image/video compression schemes in order to reduce the bandwidth required for the transmission of the image/video data from the camera through a network of the vehicle to an image/video analysis unit.
  • Vehicles such as cars or trucks
  • Lossy image compression schemes e.g. Joint Picture Group (JPEG), Motion JPEG (MJPEG), Moving Picture Experts Group (MPEG), H.264, etc.
  • JPEG Joint Picture Group
  • MJPEG Motion JPEG
  • MPEG Moving Picture Experts Group
  • H.264 H.264
  • lossy image compression schemes typically lead to the creation of artifacts, the extent of which increases with an increasing compression ratio (i.e. with a reducing data-rate).
  • the image/video analysis unit may be configured to apply image/video analysis algorithms to the (compressed) image/video data (e.g. for the detection of a pedestrian).
  • the artifacts comprised within the (compressed) image/video data may impact the performance of the image/video analysis algorithms.
  • the detection rate for pedestrian detection may be negatively affected, when increasing the compression ratio of the image/video compression scheme.
  • the parameter settings of the image/video compression schemes may affect the performance of the image/video analysis algorithms which are applied to the (compressed) image/video data.
  • an imager simulator configured to be used in lieu of an imager within a vehicle.
  • the imager simulator may be used in lieu of an imager within a driver assistance system of the vehicle.
  • the imager simulator may be used in lieu of an imager within an infotainment system (e.g. a video communication system) of the vehicle.
  • the driver assistance system may be a camera-based (also referred to as imager-based) driver assistance system.
  • the driver assistance system may comprise an imager (also referred to as an image sensor) configured to record an optical signal and to thereupon generate (digital) image data according to a particular data format.
  • the imager simulator may be used within the driver assistance system in lieu of (i.e. in place of or instead of) the imager, in order to replace the image data, which is generated by the imager based on optical signals which are recorded by the image, by image data, which is generated based on pre-determined (and possibly pre-stored) reference data.
  • the imager simulator may comprise an image source configured to store the pre- determined reference data.
  • the pre-determined reference data may comprise one or more of the following: image data recorded using the imager of the driver assistance system (in other words, image data which has been pre-recorded using the imager which is to be simulated by the imager simulator); image data generated by a pattern generator
  • the imager simulator may further comprise an imager interface unit configured to generate image data based on the pre-determined reference data.
  • the image data generated by the imager simulator is adapted to be used within the driver assistance system (instead of image data generated by the imager of the driver assistance system).
  • the image data may conform to a pre-determined format, wherein the pre-determined format corresponds to the particular format of image data generated by the imager of the driver assistance system.
  • the pre-determined format may be in conformity with a pre-determined standard, e.g. with the ITU-R BT.601 or the ITU-R BT.709 standard.
  • the predetermined format may specify one or more of the following: a resolution of a frame of the image data (e.g.
  • a number of pixels per line and/or a number of pixels per column a frame-rate of succeeding frames of the image data (e.g. a number of frames per second); a color space used to represent color information comprised within the image data (e.g. a luma component indicative of a brightness of a pixels, in conjunction with chrominance components indicative of relative color information of the pixel); a chroma subsampling scheme (e.g. indicative of a reduced spatial resolution for one or more components of a pixel); a number of bits used to encode a pixel of the image data (e.g.
  • a number of bits used to encode each component of a pixel a rate at which the bits which encode a pixel of the image data are transmitted (also referred to as a pixel clock); the use of synchronization signals, e.g. of Hsync, Href and/or Vsync signals, which may be used to indicate an end of a line and/or an end of a frame; an interlaced or de-interlaced operation of the imager; timing information for synchronization of the imager simulator with one or more components of the driver assistance system downstream of the imager (e.g. with a compression unit of the driver assistance system); and a serial or parallel transmission of the image data.
  • synchronization signals e.g. of Hsync, Href and/or Vsync signals
  • the imager simulator may comprise a physical connector (e.g. a coax connector) which conforms to the physical connector of the imager, such that the imager simulator may be integrated within the driver assistance system using the physical connector destined for the imager.
  • the imager simulator comprises a physical connector which corresponds to the physical connector of the imager and generates image data in the predetermined format which corresponds to the particular format of the image data generated by the imager.
  • the imager simulator may be configured to completely take the place of the imager within the driver assistance system without the need of making any modifications to the rest of the driver assistance system.
  • the imager simulator may be used to test the complete driver assistance system downstream of the output of the imager (wherein downstream refers to the flow direction of the image data through the driver assistance system).
  • an evaluation system for determining a performance indicator of an imager-based driver assistance system may comprise an imager simulator according to any of the aspects described in the present document.
  • the imager simulator may be used in lieu of (e.g. by replacing) the imager of the driver assistance system.
  • the imager simulator may be configured to simulate and replace a plurality of imagers of the driver assistance system (e.g. in case of stereo vision).
  • the imager simulator may be configured to generate image data based on pre-determined reference data.
  • the evaluation system may further comprise an image analysis unit configured to analyze data derived from the image data, thereby generating analyzed image data.
  • the image analysis unit may make use of one or more of a plurality of image analysis algorithms to generate the analyzed image data.
  • An image analysis algorithm may e.g. be configured to perform line detection and/or obstacle detection.
  • the analyzed image data may be indicative of zero or more objects detected within the data derived from the image data.
  • the data derived from the image data may e.g. be the image data (in cases where the imager output is directly coupled to the image analysis unit, e.g. via dedicated bus).
  • the data derived from the image data may e.g. be a compressed and subsequently decompressed version of the image data (in cases where the image data at the imager output is compressed for the purpose of bandwidth reduction).
  • the evaluation system may further comprise an evaluation unit configured to determine the performance indicator of the imager-based driver assistance system (or a performance indicator of the one or more image analysis algorithms used within the image analysis unit) based on the analyzed image data (in particular based on the zero or more objects detected within the data derived from the image data).
  • the evaluation unit may be configured to determine the performance indicator also based on benchmark data derived from the predetermined reference data.
  • the benchmark data may e.g. be determined manually from the pre-determined reference data.
  • the benchmark data may be indicative of one or more benchmark objects.
  • the analyzed image data may be indicative of zero or more detected objects.
  • the performance indicator may comprise a detection rate of the one or more benchmark objects within the analyzed image data (e.g. by comparing the one or more benchmark objects with the zero or more detected objects).
  • the evaluation system (and the driver assistance system to be evaluated) may further comprise a compression unit configured to encode the image data using a lossy
  • the compression scheme which is applied within the compression unit may be adjusted using one or more settings, or, one or more sets of settings.
  • the evaluation system (and the driver assistance system to be evaluated) may comprise a corresponding decompression unit configured to decode the compressed image data, thereby yielding decoded image data.
  • the compression unit may be located at the imager / at the imager simulator and the decompression unit may be located at the image analysis unit.
  • the compression unit and the decompression unit may be coupled via an on-board network (e.g. an Ethernet network) of the vehicle.
  • the compression unit and the decompression unit may be used to reduce the bandwidth of the image data sent from the imager simulator to the image analysis unit.
  • the image analysis unit may then be configured to analyze the decoded image data to provide the analyzed image data (i.e. the data derived from the image data may correspond to the decoded image data).
  • the compression unit may comprise one or more settings (or one or more sets of settings) which may be used to adjust the compression scheme which is applied to the image data.
  • the one or more settings may relate to one or more of the following: the compression scheme, e.g. an MJPEG, a JPEG, an MPEG, or an H.264 algorithm, used within the compression unit; an overall or average target bit-rate or an overall or average compression ratio to be achieved by the compression scheme; a maximum bit-rate per frame of the image data or a maximum bit-rate per subregion of a frame of the image data; a quality criteria applied by the compression scheme when encoding the image data; a target average bit-rate per pre-determined time interval (e.g.
  • the evaluation system may further comprise a reference image analysis unit configured to analyze the image data directly (without prior compression and decompression), thereby generating reference analyzed image data. Furthermore, the evaluation system may comprise a comparison unit configured to determine a performance deterioration of the imager-based driver assistance system (and/or of the image analysis unit) due to the lossy compressing scheme applied to the image data, based on the reference analyzed image data and based on the analyzed image data.
  • comparison unit (or an additional parameter tuning unit) may be configured to determine the one or more settings of the compression scheme used within the compression unit, which reduce (e.g. minimize) the performance deterioration of the imager-based driver assistance system (and/or of the image analysis unit).
  • a method for determining a performance indicator of an imager-based driver assistance system (or of an image analysis unit comprised within the driver assistance system) is described.
  • the method may comprise generating image data based on pre-determined reference data e.g. using an imager simulator in lieu of an imager of the driver assistance system.
  • the imager simulator may be configured as described in the present document.
  • the method may proceed in transmitting the image data over an on- board network of the vehicle within which the driver assistance system is used.
  • the method may comprise analyzing data derived from the image data using a first image analysis algorithm, thereby generating analyzed image data.
  • the first image analysis algorithm may be selected from a plurality of different image analysis algorithms.
  • the performance indicator may be determined based on the analyzed image data.
  • the method may further comprise encoding the image data using a lossy compressing scheme.
  • the compression scheme may be adjusted using a set of settings for the compression scheme.
  • the set of settings may comprise e.g. one or more settings of compression schemes described in the present document.
  • the method may comprise repeating the generating step, the encoding step, the decoding step, the analyzing step and the determining step using the same pre-determined reference data but using different ones of a plurality of different sets of settings for the compression scheme, thereby yielding a corresponding plurality of performance indicators.
  • the above mentioned method may be iterated using different sets of settings for the compression scheme.
  • a plurality of performance indicators of the driver assistance system or of the image analysis unit
  • the method may then comprise selecting a set of settings from the plurality of different sets of settings based on the plurality of performance indicators.
  • the set of settings may be selected which results in the highest performance indicator (e.g. in the highest detection rate).
  • the method may also be iterated for different image analysis algorithms used within the image analysis unit. As such, different sets of settings for the compression scheme may be selected, in dependence of the image analysis algorithm used within the image analysis unit.
  • the method may comprise determining the plurality of performance indicators using a second image analysis algorithm, different from the first image analysis algorithm, thereby yielding a possibly different set of settings to be used in conjunction with the second image analysis algorithm.
  • the imager may comprise an optical camera configured to capture optical signals, thereby generating the (digital) image data.
  • the driver assistance system may comprise a compression unit configured to encode the image data using a lossy compressing scheme, thereby yielding compressed image data.
  • the compression scheme may be adjustable using one or more settings.
  • the compressed image data may be transmitted over an on-board network (e.g. an Ethernet network) of the vehicle.
  • the driver assistance system may further comprise a decompression unit configured to decode the compressed image data, thereby yielding decoded image data.
  • the driver assistance system may comprise an image analysis unit configured to analyze the decoded image data using at least one of a plurality of image analysis algorithms, thereby yielding analyzed image data.
  • the analyzed image data may be usable for assisting a driver of the vehicle.
  • the driver assistance system may be configured to generate feedback information (e.g. a warning message) to the driver of the vehicle in dependence of the analyzed image data (e.g. when the analyzed image data comprises an object corresponding to a pedestrian).
  • the driver assistance system may comprise a memory unit configured to store one or more settings of the compression scheme for the plurality of image analysis algorithms, respectively.
  • the memory unit may be configured to store for each of the plurality of image analysis algorithms a corresponding set of one or more settings (e.g. in the form of a table).
  • the driver assistance system may be configured to select and use the one or more settings for the compression scheme, stored in the memory unit, in dependence of the at least one image analysis algorithm used by the image analysis unit. As such, for different image analysis algorithms, different settings may be used for the compression scheme, thereby improving the overall performance of the driver assistance system.
  • a software program is described.
  • the software program may be adapted for execution on a processor and for performing the method steps outlined in the present document when carried out on the processor.
  • the storage medium may comprise a software program adapted for execution on a processor and for performing the method steps outlined in the present document when carried out on the processor.
  • the computer program may comprise executable instructions for performing the method steps outlined in the present document when executed on a computer.
  • Figs, la and lb show example systems for evaluating the performance of a driver assistance system
  • Figs. 2a and 2b show example systems for evaluating the performance of a driver assistance system using hardware-in-the-loop techniques
  • Fig. 3 illustrates a block diagram of an example imager simulator
  • Fig. 4 shows a flow chart of an example method for evaluating the performance of a driver assistance system
  • Fig. 5 illustrates another example system for evaluating the performance of a driver assistance system.
  • Vehicles may comprise a driver assistance system which comprises one or more image sensors (also referred to as imagers) configured to capture image data (e.g. a still image or a video).
  • image data e.g. a still image or a video
  • the captured image data may be transferred to an image analysis unit which analyzes the image data using one or more image analysis algorithms. Examples for image analysis algorithms are
  • ⁇ a lane detection algorithm configured to detect whether the vehicle stays within a current lane of the road that the vehicle is driving on;
  • an obstacle detection algorithm configured to detect whether an obstacle (e.g. a pedestrian or an animal) appears on the trajectory of the vehicle.
  • the image analysis algorithms are typically directed at the detection of one or more objects within the image data provided by one or more imagers.
  • the image analysis algorithms typically make use of computer vision techniques, in particular object recognition techniques, such as CAD-like object-based methods, appearance-based methods and/or feature-based methods.
  • Figs, la and lb illustrate systems 100, 120 for evaluating the performance of such camera- based driver assistance systems.
  • the system 100 comprises an image sensor (also referred to as an imager) 101.
  • the imager 101 generates image data 111 which is transmitted to an image analysis unit 102.
  • the image data 111 may be in a pre-determined format, e.g. in a format which is in accordance to the ITU-R BT.601 specification (for video signals with 525-line or 625-line television systems) or in accordance to the ITU-R BT.709
  • the imager 101 may comprise an appropriate interface for generating the image data 111 in the pre-determined format.
  • the image analysis unit 102 receives the image data 111 and performs one or more image analysis algorithms in order to detect one or more objects within the received image data 111.
  • the analyzed image data 112 (e.g. comprising one or more detected objects) may then be passed to an evaluation unit 103.
  • the evaluation unit 103 may be configured to evaluate the performance of the image analysis unit 102.
  • the evaluation unit 103 may have access to benchmark data, and may be configured to compare the benchmark data with the analyzed image data 112.
  • the benchmark data may e.g. be determined by a person who analyzes the image data 111 provided by the imager 101 and who performs a manual detection of one or more benchmark objects within the image data 111.
  • the benchmark data may comprise one or more benchmark objects, and can be compared to the analyzed image data 112 comprising one or more detected objects, in order to determine a performance indicator of the image analysis unit 102 (e.g. in order to determine a detection rate of the image analysis unit 102).
  • the system 100 of Fig. la does not make use of image compression such that the image data 111 generated by the imager 101 is provided - as is - to the image analysis unit 102. Due to the increase of the amount of data transmission within a vehicle and due to the use of a shared network infrastructure (such as Gigabit Ethernet), the image data 111 may be submitted to lossy compression schemes, in order to reduce the bandwidth required for transmitting the image data 111 to the image analysis unit 102. This is illustrated in Fig. lb which shows an example system 120 for evaluating the performance of a driver assistance system 140 comprising lossy image compression.
  • the image data 111 is encoded (typically directly at the location of the imager 101 prior to transmission over the on-board network of the vehicle) within a compression unit 121 (also referred to as an encoding unit 121).
  • the compression unit 121 may be configured to apply a lossy compression algorithm (e.g. JPEG, MJPEG, MPEG, H.264) to the image data 111, thereby providing compressed image data 131.
  • a lossy compression algorithm e.g. JPEG, MJPEG, MPEG, H.264
  • the compressed image data 131 is transmitted over the on-board network of the vehicle to a decompression unit 122 (also referred to as a decoding unit 122) which is configured to apply a decompression algorithm (which corresponds to the compression algorithm applied within the compression unit 121) to the compressed image data 131, thereby providing decoded image data 132.
  • the decoded image data 132 typically differs from the (original) image data 1 11 generated by the imager 101.
  • the deviations of the decoded image data 132 depend on the settings of the compression unit 121, wherein the settings may e.g. relate to one or more of:
  • the compression algorithm e.g. MJPEG, JPEG, MPEG, H.264
  • the compression algorithm e.g. MJPEG, JPEG, MPEG, H.264
  • the compression ratio i.e. the ratio of the size (e.g. measured in bits) of the image data 111 and the size (e.g. measured in bits) of the compressed image data 131);
  • a quality factor e.g. a signal-to-noise ratio or a psycho-visually motivated signal- to-noise ratio
  • the quality factor may be defined as a target (e.g. average) quality factor, as a minimum quality factor or as a maximum quality factor
  • a maximum bit-rate per image block (e.g. per macro-block in JPEG);
  • an amount of cutoff of bits in an encoded frame may occur before transmission of the image data 111 (notably for bandwidth limitation of the image data 111);
  • the one or more settings of the compression unit 121 which may be selected and/or modified typically depend on the compression algorithm used within the compression unit 121.
  • the compression algorithm applied within the compression unit 121 is directed at achieving a target compression ratio or a target bit-rate subject to psycho-visual quality criteria, thereby reducing artifacts comprised within the decoded image data 132, which are visible to a human being.
  • the psycho-visual quality criteria may not be optimal when using the decoded image data 132 for image analysis within the image analysis unit 102. As such, a further setting of the compression unit 121 may be
  • the image analysis unit 102 applies one or more image analysis algorithms to the decoded image data 102 in order to generate the analyzed image data 133.
  • the analyzed image data 133 may comprise one or more detected objects.
  • the evaluation unit 103 determines a performance indicator of the image analysis unit 102 or of the driver assistance system 140 comprising the imager 101, the compression unit 121, the decompression unit 122 and the image analysis unit 102, based on the analyzed image data 133 and based on benchmark data comprising one or more benchmark objects.
  • the performance indicator may e.g. be directed at a rate of detection of objects captured by the imager 101.
  • the lossy compression algorithm applied to the image data 111 typically affects the performance of the driver assistance system 140. It is desirable to improve the performance of the driver assistance system 140 by adjusting the various settings of the compression unit 121 depending on the image processing algorithm used within the image analysis unit 102. This may be achieved using a trial-and-error approach by adjusting the settings of the compression unit 121 in conjunction with a particular image processing algorithm during a number of test drives.
  • the use of test drives is, however, time consuming and typically does not lead to optimal results, because the conditions during the test drives are not reproducible. Furthermore, the results obtained during different test drives are usually not comparable, as the conditions during the different test drives (e.g. the conditions of the light and of the objects which are to be detected) vary.
  • Figs. 2a and 2b show block diagrams of example systems 200, 220 which may be used to determine and/or tune the driver assistance system of a vehicle in an efficient and reproducible manner.
  • the systems 200, 220 correspond to the systems 100, 120 of Figs, la, lb, respectively.
  • the imagers 101 have been replaced by imager simulators 201, respectively.
  • the imager simulators 201 are configured to provide image data 111 in the same format (e.g. according to the same protocol) as the imagers 101 of Figs, la, lb.
  • the imager simulators 201 may be coupled to the network (system 200) or to the compression unit 121 (system 220) in the same manner as the actual imagers 101.
  • the imager simulators 201 are further configured to render pre-recorded reference data as image data 111.
  • the same image data 111 can be re-produced multiple times (using the pre-recorded reference data), thereby allowing for reproducible test sequences of the driver assistance system of a vehicle.
  • the driver assistance system remains unchanged, apart from the imager 101 which is replaced by the imager simulator 201, thereby allowing for Hardware-in-the-Loop testing which is as close as possible to the actual driver assistance system used within the vehicle.
  • Fig. 3 shows a block diagram of an example imager simulator 201.
  • the imager simulator 201 of Fig. 3 comprises an image source 301 (e.g. a personal computer, a logger of video data, or a pattern generator) which is configured to store the pre-recorded reference data 302 and which is configured to provide the pre-recorded reference data 302 to a processing unit 310.
  • the processing unit 310 may be implemented e.g. using an FPGA (field- programmable gate array).
  • the processing unit 310 comprises a reference data interface unit 312 which is configured to request and receive the pre-recorded reference data 302 from the image source 301.
  • the reference data interface unit 312 comprises a Gigabit-Ethernet interface to communicate with the image source 301.
  • the pre-recorded reference data 302 may be stored in a variable buffer 313.
  • the reference data interface unit 312 may be configured to send control data (e.g. Ethernet SYNC- frames) to the image source 301 in order to control the transmission rate of the prerecorded reference data 302 from the image source 301.
  • the control data may be sent to the image source 301 in dependence of the fill level of the buffer 313 and/or in dependence of the frame rate of the image data 111 generated by the imager simulator 201, thereby regulating the data flow from the image source 301 to the processing unit 310.
  • the buffer 313 may be configured to compensate for variations in the data flow, thereby ensuring that the image data 111 can be generated at a stable frame rate and/or with a timing which is adjustable to the timing of the compression unit 121 or of the on-board network.
  • the processing unit 310 comprises an imager interface unit 314 configured to generate the image data 111 according to the pre-determined format.
  • the format may e.g. be in accordance to the ITU-R BT.601 or ITU-R BT.709 standards.
  • the imager interface unit 314 may be configured to take the pre-recorded reference data 302 from the buffer 313 and format the pre-recorded reference data 302 in accordance to the pre-determined format, thereby generating the image data 111.
  • the pre-determined format of the image data 111 may relate to one or more of the following aspects: • the underlying specification (e.g. ITU-R BT.601 or ITU-R BT.709);
  • the color space used to represent color information e.g. the YCbCr color space using a luma component, a blue-difference component and a red-difference component, or the YUV color space using a luma component and two chrominance components;
  • chroma subsampling i.e. subsampling of chroma information with respect to luma information, e.g. YCbCr 4:2:2 or YUV 4:2:2);
  • bits per component e.g. 8 bits, 10 bits or 12 bits per component
  • the frame rate e.g. 60Hz, 50Hz, 30Hz, 25Hz, or 24Hz
  • Hsync signal e.g. pulse
  • Href signal indicating the length of a line
  • the imager interface unit 314 may provide a physical interface (e.g. a connector) which is in accordance to the physical interface of the imager 101.
  • the imager interface unit 314 may comprise a 25-pin Sub-D connector, in case of parallel transmission format, or a BNC (Bayonet Neill-Concelman) connector for connecting a coaxial cable, in case of serial transmission.
  • BNC Boyonet Neill-Concelman
  • the processing unit 310 may further comprise a control unit 311 configured to control the operation of the reference data interface unit 312, the buffer 313 and/or the imager interface unit 314.
  • the control unit 311 may be used to configure other interfaces of the processing unit 310, such as a serial interface, an I2C interface and/or an SPI interface (not shown).
  • the system 220 of Fig. 2b which comprises the imager simulator 201 can be used in a method 400 for determining settings of the compression unit 121, which improve the performance of the image analysis performed within the image analysis unit 102 downstream of the compression unit 121.
  • the method 400 makes use of pre-determined reference data 302 stored on the image source 301.
  • the pre-determined reference data 302 is rendered as image data 111 using the imager simulator 201.
  • the image data 111 is processed by the compression unit 121 and the de-compression unit 122 using a first set of settings (step 402), thereby generating first decoded image data 132.
  • the image analysis unit 102 analyzes the first decoded image data 132 using one or more image analysis algorithms, thereby yielding first analyzed image data 133 (step 403).
  • the evaluation unit 103 determines a first performance indicator for the image analysis unit 102 based on the first analyzed image data 133 (and possibly based on benchmark data) (step 404).
  • the above mentioned scheme i.e.
  • method steps 401 to 404) may be repeated (step 405) for a plurality of different sets of settings of the compression / de-compression units 121, 122 using the same pre-determined reference data 302, thereby determining a corresponding plurality of performance indicators for the image analysis unit 102 (using the one or more image analysis algorithms).
  • the set of settings for a current iteration of the method 400 may be determine based on the sets of settings used for the one or more preceding iterations, and/or based on the performance indicators determined within the one or more preceding iterations.
  • optimization techniques such as gradient decent algorithms or heuristic algorithms
  • the plurality of sets of settings may be pre-determined in accordance to a pre-determined test protocol.
  • the method 400 may terminate, e.g. after a pre-determined number of iterations, and/or after having determined the plurality of performance indicators for a plurality of pre- determined sets of settings, and/or after having determined a (local) optimum for the performance indicator.
  • the method 400 may then comprise the step 406 of selecting a set of settings from the plurality of sets of settings which corresponds to the maximum performance indicator of the plurality of determined performance indicators.
  • the method 400 allows to determine a set of settings for the compression / decompression units 121, 122 which allows to increase (e.g. maximize) the performance of the image analysis unit 102 (e.g. which allows to maximize the detection rate) when using the one or more image analysis algorithms.
  • the method 400 may be performed for different image analysis algorithms (e.g. for lane detection and/or for pedestrian detection), thereby determining improved sets of settings for the compression / decompression units 121, 122 for different image analysis algorithms.
  • the respective improved sets of settings may then be used in conjunction with the different image analysis algorithms, thereby improving the performance of the diver assistance system 140 (comprising the actual imager 101), depending on the one or more image analysis algorithms in use.
  • the driver assistance system 140 may be configured to perform pedestrian detection up to a pre-determined maximum speed (e.g. of 60 km/h).
  • the driver assistance system 140 may be configured to perform lane detection for speeds higher than the pre-determined maximum speed.
  • the driver assistance system 140 may be configured to use the set of settings which were optimized for pedestrian detection at speeds up to the pre-determined maximum speed, and configured to use the set of settmgs which were optimized for lane detection at speeds above the pre-determined maximum speed.
  • the method 400 using the imager simulator 201 allows for a time- and cost-efficient, as well as a reproducible scheme for determining the set of settings for a compression algorithm to be used in conjunction with a particular image analysis algorithm, thereby improving the performance of the particular image analysis algorithm.
  • the design and the optimization of camera-based driver assistance systems can be accelerated and improved.
  • the systems 200, 220 and/or the method 400 may be used to simulate the behavior of the driver assistance system 140 in response to a faulty imager 101.
  • the imager simulator 201 may be configured to generate erroneous image data 111 (e.g. a freeze frame, or frames of a video comprising faulty pixels). The performance of the image analysis unit 102 subject to such erroneous image data 111 may be evaluated, thereby providing insights with regards to the robustness of the driver assistance system 140.
  • the systems 200, 220 and/or the method 400 may be extended to a plurality of imager simulators 201, thereby simulating the presence of a corresponding plurality of imagers 101 within the driver assistance system 140.
  • the plurality of imagers 101 may be used for stereo vision or for parking cameras, and the plurality of imager simulators 201 may be configured to generate appropriate image data 111 (based on appropriate pre-determined reference data 302) in order to simulate such stereo or parking situations.
  • the pre-determined reference data 302 for the plurality of imager simulators 201 may have been recorded using the respective imagers 101 of the actual driver assistance system 140.
  • the plurality of imager simulators 201 may be configured to generate the respective image data 111 in a synchronized manner, thereby simulating the real-life situation of the driver assistance system 140. If required, a predetermined temporal offset between the image data 111 generated by the different imager simulators 201 may be implemented.
  • the pre-determined reference data 302 is not limited to image data pre-recorded by an actual imager 101 within the actual driver assistance system 140.
  • artificial image data e.g. generated by a pattern generator
  • the pre-determined reference data 302 may be adapted to test extreme situations with regards to the compression algorithms applied within the compression unit 121.
  • the pre-determined reference data 302 may be selected in order to test a "worst case” scenario with regards to compression aspects, thereby allowing to appropriately design the compression unit 121 (e.g. memory and/or processing capacity of the compression unit 121) for such a "worst case” scenario.
  • Fig. 5 illustrates a block diagram of another example system 500 for evaluating and/or optimizing the performance of a driver assistance system.
  • the system 500 is particularly advantageous for tuning the one or more settings of the video compression unit 121 (and of the video compression scheme used therein).
  • the system 500 comprises a reference system 510 (which - in conjunction with the imager simulator 201 - corresponds to the system 200 described in Fig.
  • the imager simulator 201 comprising the image source 301 (e.g. a database) and the processing unit 310, provides (raw) image data 111 which is analyzed directly by the image analysis unit 102, thereby providing reference analyzed image data 112.
  • the evaluation unit 103 of the reference system 510 may be configured to determine a performance indicator of the driver assistance system without compression, based on the reference analyzed image data 112. As outlined above, the evaluation unit 103 of the reference system 510 may make use of benchmark data 502 (e.g. benchmark data 502 provided by the imager simulator 201) for this purpose.
  • the system 500 comprises the evaluation system 520 which - in conjunction with the imager simulator 201 - corresponds to the system 220 that has already been described in the context of Fig. 2b.
  • the evaluation unit 103 of the evaluation system 520 determines a performance indicator of the driver assistance system including compression, based on the analyzed image data 112 (and possibly based on the benchmark data 502).
  • the comparison unit 501 of system 500 may be configured to compare the performance indicators provided by the reference system 510 and by the evaluation system 520. By using the information provided by the reference system 510, the comparison unit 501 is enabled to clearly identify the performance deteriorations of the image analysis unit 102 which are due to the image compression applied within the driver assistance system.
  • the comparison unit 501 may exclude performance issues which are inherent to the image analysis algorithm used within the image analysis unit 102 (and which are not necessarily due to compression).
  • the system 500 may further comprise a parameter tuning unit 511 which is configured to tune the one or more settings of the compression scheme used within the compression unit 121 (using the feedback provided by the comparison unit 501), thereby improving the overall performance of the driver assistance system comprising image compression (e.g. using the method of Fig. 4).
  • an imager simulator has been described which may be used in lieu of an actual imager of the camera-based driver assistance system, thereby allowing to test the camera-based driver assistance system in an efficient and reproducible manner.
  • a setup comprising the imager simulator may be used to adjust various parameters of the camera-based driver assistance system (in particular, to adjust the settings of a compression unit comprised within the camera-based driver assistance system), thereby improving the overall performance of the camera-based driver assistance system.
  • the methods and systems described in the present document may be implemented as software, firmware and/or hardware. Certain components may e.g. be implemented as software running on a digital signal processor or microprocessor. Other components may e.g. be implemented as hardware and or as application specific integrated circuits.
  • the signals encountered in the described methods and systems may be stored on media such as random access memory or optical storage media. They may be transferred via networks, such as radio networks, satellite networks, wireless networks or wireline networks, e.g. an Ethernet network.
  • Typical devices making use of the methods and systems described in the present document are driver assistance systems in a vehicle (e.g. in a car or a truck).
  • the software may be referred to as a computer program and the invention may also be implemented in a computer program for running on a programmable apparatus, at least including code portions for performing steps of a method according to the invention when run on a programmable apparatus, such as a computer system or enabling a programmable apparatus to perform functions of a device or system according to the invention.
  • the computer program may be stored, e.g. internally in the programmable apparatus, on a computer readable storage medium or transmitted to the programmable apparatus via a readable transmission medium. All or some of the computer program may be provided on tangible or non-tangible computer readable media permanently, removably or remotely coupled to an information processing system.
  • the computer readable media may be transitory or non transitory and include, for example and without limitation, any number of the following: magnetic storage media including disk and tape storage media; optical storage media such as compact disk media (e.g., CD-ROM, CD-R, etc.) and digital video disk storage media; nonvolatile memory storage media including semiconductor-based memory units such as FLASH memory, EEPROM, EPROM, ROM; ferromagnetic digital memories; MRAM; volatile storage media including registers, buffers or caches, main memory, RAM, etc.; and data transmission media including computer networks, point-to- point telecommunication equipment, and carrier wave transmission media, just to name a few.
  • magnetic storage media including disk and tape storage media
  • optical storage media such as compact disk media (e.g., CD-ROM, CD-R, etc.) and digital video disk storage media
  • nonvolatile memory storage media including semiconductor-based memory units such as FLASH memory, EEPROM, EPROM, ROM
  • ferromagnetic digital memories such as FLASH memory,

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Signal Processing (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Quality & Reliability (AREA)
  • Evolutionary Computation (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Software Systems (AREA)
  • Medical Informatics (AREA)
  • General Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Description

Method and System for Optimizing Image Processing in Driver Assistance Systems
TECHNICAL FIELD
The present document relates to image processing. In particular, the present document relates to the optimization of image processing in a driver assistance system of a vehicle.
BACKGROUND
Camera-based image/video applications (e.g. automotive driver assistance camera functions) often use image/video compression schemes in order to reduce the bandwidth required for the transmission of the image/video data from the camera through a network of the vehicle to an image/video analysis unit. Vehicles (such as cars or trucks) may comprise shared networks such as Ethernet networks, with limited bandwidth for the transmission of image/video data. Lossy image compression schemes (e.g. Joint Picture Group (JPEG), Motion JPEG (MJPEG), Moving Picture Experts Group (MPEG), H.264, etc.) may be used to reduce the data-rate for the transmission of image/video data (e.g. by a factor of 20 to 50). On the other hand, lossy image compression schemes typically lead to the creation of artifacts, the extent of which increases with an increasing compression ratio (i.e. with a reducing data-rate).
The image/video analysis unit may be configured to apply image/video analysis algorithms to the (compressed) image/video data (e.g. for the detection of a pedestrian). The artifacts comprised within the (compressed) image/video data may impact the performance of the image/video analysis algorithms. By way of example, the detection rate for pedestrian detection may be negatively affected, when increasing the compression ratio of the image/video compression scheme. As such, the parameter settings of the image/video compression schemes may affect the performance of the image/video analysis algorithms which are applied to the (compressed) image/video data. The present document addresses the above mentioned problem of a driver assistance system which makes use of image/video analysis algorithms in conjunction with lossy image/video compression schemes. In particular, the present document describes methods and systems which are directed at improving the performance of image/video analysis algorithms when used in conjunction with lossy image/video compression schemes. It should be noted that for conciseness, reference is made in the following to the term "image" only, wherein the term "image" is understood to comprise still images, as well as video (i.e. moving images).
SUMMARY
According to an aspect an imager simulator configured to be used in lieu of an imager within a vehicle is described. In particular, the imager simulator may be used in lieu of an imager within a driver assistance system of the vehicle. Alternatively or in addition, the imager simulator may be used in lieu of an imager within an infotainment system (e.g. a video communication system) of the vehicle. The driver assistance system may be a camera-based (also referred to as imager-based) driver assistance system. This means that the driver assistance system may comprise an imager (also referred to as an image sensor) configured to record an optical signal and to thereupon generate (digital) image data according to a particular data format. The imager simulator may be used within the driver assistance system in lieu of (i.e. in place of or instead of) the imager, in order to replace the image data, which is generated by the imager based on optical signals which are recorded by the image, by image data, which is generated based on pre-determined (and possibly pre-stored) reference data.
The imager simulator may comprise an image source configured to store the pre- determined reference data. The pre-determined reference data may comprise one or more of the following: image data recorded using the imager of the driver assistance system (in other words, image data which has been pre-recorded using the imager which is to be simulated by the imager simulator); image data generated by a pattern generator
(representing e.g. artificially created objects to be detected by an image analysis unit of the driver assistance system); and image data representing a freeze image.
The imager simulator may further comprise an imager interface unit configured to generate image data based on the pre-determined reference data. The image data generated by the imager simulator is adapted to be used within the driver assistance system (instead of image data generated by the imager of the driver assistance system). The image data may conform to a pre-determined format, wherein the pre-determined format corresponds to the particular format of image data generated by the imager of the driver assistance system. The pre-determined format may be in conformity with a pre-determined standard, e.g. with the ITU-R BT.601 or the ITU-R BT.709 standard. Alternatively or in addition, the predetermined format may specify one or more of the following: a resolution of a frame of the image data (e.g. a number of pixels per line and/or a number of pixels per column); a frame-rate of succeeding frames of the image data (e.g. a number of frames per second); a color space used to represent color information comprised within the image data (e.g. a luma component indicative of a brightness of a pixels, in conjunction with chrominance components indicative of relative color information of the pixel); a chroma subsampling scheme (e.g. indicative of a reduced spatial resolution for one or more components of a pixel); a number of bits used to encode a pixel of the image data (e.g. a number of bits used to encode each component of a pixel); a rate at which the bits which encode a pixel of the image data are transmitted (also referred to as a pixel clock); the use of synchronization signals, e.g. of Hsync, Href and/or Vsync signals, which may be used to indicate an end of a line and/or an end of a frame; an interlaced or de-interlaced operation of the imager; timing information for synchronization of the imager simulator with one or more components of the driver assistance system downstream of the imager (e.g. with a compression unit of the driver assistance system); and a serial or parallel transmission of the image data.
The imager simulator may comprise a physical connector (e.g. a coax connector) which conforms to the physical connector of the imager, such that the imager simulator may be integrated within the driver assistance system using the physical connector destined for the imager. In an embodiment, the imager simulator comprises a physical connector which corresponds to the physical connector of the imager and generates image data in the predetermined format which corresponds to the particular format of the image data generated by the imager. As such, the imager simulator may be configured to completely take the place of the imager within the driver assistance system without the need of making any modifications to the rest of the driver assistance system. Furthermore, the imager simulator may be used to test the complete driver assistance system downstream of the output of the imager (wherein downstream refers to the flow direction of the image data through the driver assistance system).
According to another aspect, an evaluation system for determining a performance indicator of an imager-based driver assistance system is described. The evaluation system may comprise an imager simulator according to any of the aspects described in the present document. The imager simulator may be used in lieu of (e.g. by replacing) the imager of the driver assistance system. It should be noted that the imager simulator may be configured to simulate and replace a plurality of imagers of the driver assistance system (e.g. in case of stereo vision). As outlined above, the imager simulator may be configured to generate image data based on pre-determined reference data.
The evaluation system may further comprise an image analysis unit configured to analyze data derived from the image data, thereby generating analyzed image data. The image analysis unit may make use of one or more of a plurality of image analysis algorithms to generate the analyzed image data. An image analysis algorithm may e.g. be configured to perform line detection and/or obstacle detection. The analyzed image data may be indicative of zero or more objects detected within the data derived from the image data. The data derived from the image data may e.g. be the image data (in cases where the imager output is directly coupled to the image analysis unit, e.g. via dedicated bus).
Alternatively, the data derived from the image data may e.g. be a compressed and subsequently decompressed version of the image data (in cases where the image data at the imager output is compressed for the purpose of bandwidth reduction). The evaluation system may further comprise an evaluation unit configured to determine the performance indicator of the imager-based driver assistance system (or a performance indicator of the one or more image analysis algorithms used within the image analysis unit) based on the analyzed image data (in particular based on the zero or more objects detected within the data derived from the image data). The evaluation unit may be configured to determine the performance indicator also based on benchmark data derived from the predetermined reference data. The benchmark data may e.g. be determined manually from the pre-determined reference data. The benchmark data may be indicative of one or more benchmark objects. Furthermore, the analyzed image data may be indicative of zero or more detected objects. In such cases, the performance indicator may comprise a detection rate of the one or more benchmark objects within the analyzed image data (e.g. by comparing the one or more benchmark objects with the zero or more detected objects). The evaluation system (and the driver assistance system to be evaluated) may further comprise a compression unit configured to encode the image data using a lossy
compressing scheme, thereby yielding compressed image data. The compression scheme which is applied within the compression unit may be adjusted using one or more settings, or, one or more sets of settings. In addition, the evaluation system (and the driver assistance system to be evaluated) may comprise a corresponding decompression unit configured to decode the compressed image data, thereby yielding decoded image data. The compression unit may be located at the imager / at the imager simulator and the decompression unit may be located at the image analysis unit. The compression unit and the decompression unit may be coupled via an on-board network (e.g. an Ethernet network) of the vehicle. As such, the compression unit and the decompression unit may be used to reduce the bandwidth of the image data sent from the imager simulator to the image analysis unit. As already indicated above, the image analysis unit may then be configured to analyze the decoded image data to provide the analyzed image data (i.e. the data derived from the image data may correspond to the decoded image data).
The compression unit may comprise one or more settings (or one or more sets of settings) which may be used to adjust the compression scheme which is applied to the image data. The one or more settings may relate to one or more of the following: the compression scheme, e.g. an MJPEG, a JPEG, an MPEG, or an H.264 algorithm, used within the compression unit; an overall or average target bit-rate or an overall or average compression ratio to be achieved by the compression scheme; a maximum bit-rate per frame of the image data or a maximum bit-rate per subregion of a frame of the image data; a quality criteria applied by the compression scheme when encoding the image data; a target average bit-rate per pre-determined time interval (e.g. per second); a maximum bit-rate per pre- determined time interval (e.g. per second); and a minimum value for a quality of the compressed image data with respect to the image data (the quality of the compressed image data may be indicated with respect to the quality criteria applied by the compression scheme). The evaluation system may further comprise a reference image analysis unit configured to analyze the image data directly (without prior compression and decompression), thereby generating reference analyzed image data. Furthermore, the evaluation system may comprise a comparison unit configured to determine a performance deterioration of the imager-based driver assistance system (and/or of the image analysis unit) due to the lossy compressing scheme applied to the image data, based on the reference analyzed image data and based on the analyzed image data. Furthermore, the comparison unit (or an additional parameter tuning unit) may be configured to determine the one or more settings of the compression scheme used within the compression unit, which reduce (e.g. minimize) the performance deterioration of the imager-based driver assistance system (and/or of the image analysis unit).
According to another aspect, a method for determining a performance indicator of an imager-based driver assistance system (or of an image analysis unit comprised within the driver assistance system) is described. The method may comprise generating image data based on pre-determined reference data e.g. using an imager simulator in lieu of an imager of the driver assistance system. The imager simulator may be configured as described in the present document. The method may proceed in transmitting the image data over an on- board network of the vehicle within which the driver assistance system is used.
Furthermore, the method may comprise analyzing data derived from the image data using a first image analysis algorithm, thereby generating analyzed image data. The first image analysis algorithm may be selected from a plurality of different image analysis algorithms. The performance indicator may be determined based on the analyzed image data.
The method may further comprise encoding the image data using a lossy compressing scheme. The compression scheme may be adjusted using a set of settings for the compression scheme. The set of settings may comprise e.g. one or more settings of compression schemes described in the present document. By applying the compression scheme (customized using a particular set of settings) to the image data, compressed image data is provided. The compressed image data may be decoded, thereby yielding decoded image data. In cases where the method comprises encoding and decoding, the analyzed image data may be determined based on the decoded image data.
The method may comprise repeating the generating step, the encoding step, the decoding step, the analyzing step and the determining step using the same pre-determined reference data but using different ones of a plurality of different sets of settings for the compression scheme, thereby yielding a corresponding plurality of performance indicators. In other words, the above mentioned method may be iterated using different sets of settings for the compression scheme. As such, a plurality of performance indicators of the driver assistance system (or of the image analysis unit) may be determined for different sets of settings for the compression scheme. The method may then comprise selecting a set of settings from the plurality of different sets of settings based on the plurality of performance indicators. By way of example, the set of settings may be selected which results in the highest performance indicator (e.g. in the highest detection rate). It should be noted that the method may also be iterated for different image analysis algorithms used within the image analysis unit. As such, different sets of settings for the compression scheme may be selected, in dependence of the image analysis algorithm used within the image analysis unit. In particular, the method may comprise determining the plurality of performance indicators using a second image analysis algorithm, different from the first image analysis algorithm, thereby yielding a possibly different set of settings to be used in conjunction with the second image analysis algorithm.
According to a further aspect, an imager-based driver assistance system for a vehicle is described. The driver assistance system may comprise an imager configured to generate image data. The imager may comprise an optical camera configured to capture optical signals, thereby generating the (digital) image data. Furthermore, the driver assistance system may comprise a compression unit configured to encode the image data using a lossy compressing scheme, thereby yielding compressed image data. As described above, the compression scheme may be adjustable using one or more settings. The compressed image data may be transmitted over an on-board network (e.g. an Ethernet network) of the vehicle. The driver assistance system may further comprise a decompression unit configured to decode the compressed image data, thereby yielding decoded image data. In addition, the driver assistance system may comprise an image analysis unit configured to analyze the decoded image data using at least one of a plurality of image analysis algorithms, thereby yielding analyzed image data. The analyzed image data may be usable for assisting a driver of the vehicle. In particular, the driver assistance system may be configured to generate feedback information (e.g. a warning message) to the driver of the vehicle in dependence of the analyzed image data (e.g. when the analyzed image data comprises an object corresponding to a pedestrian).
The driver assistance system may comprise a memory unit configured to store one or more settings of the compression scheme for the plurality of image analysis algorithms, respectively. By way of example, the memory unit may be configured to store for each of the plurality of image analysis algorithms a corresponding set of one or more settings (e.g. in the form of a table). The driver assistance system may be configured to select and use the one or more settings for the compression scheme, stored in the memory unit, in dependence of the at least one image analysis algorithm used by the image analysis unit. As such, for different image analysis algorithms, different settings may be used for the compression scheme, thereby improving the overall performance of the driver assistance system.
According to a further aspect, a software program is described. The software program may be adapted for execution on a processor and for performing the method steps outlined in the present document when carried out on the processor.
According to another aspect, a storage medium is described. The storage medium may comprise a software program adapted for execution on a processor and for performing the method steps outlined in the present document when carried out on the processor.
According to a further aspect, a computer program product is described. The computer program may comprise executable instructions for performing the method steps outlined in the present document when executed on a computer.
It should be noted that the methods and systems including its preferred embodiments as outlined in the present patent application may be used stand-alone or in combination with the other methods and systems disclosed in this document. Furthermore, all aspects of the methods and systems outlined in the present patent application may be arbitrarily combined. In particular, the features of the claims may be combined with one another in an arbitrary manner. SHORT DESCRIPTION OF THE FIGURES
The invention is explained below in an exemplary manner with reference to the
accompanying drawings, wherein
Figs, la and lb show example systems for evaluating the performance of a driver assistance system;
Figs. 2a and 2b show example systems for evaluating the performance of a driver assistance system using hardware-in-the-loop techniques;
Fig. 3 illustrates a block diagram of an example imager simulator;
Fig. 4 shows a flow chart of an example method for evaluating the performance of a driver assistance system; and
Fig. 5 illustrates another example system for evaluating the performance of a driver assistance system.
DETAILED DESCRIPTION
Vehicles may comprise a driver assistance system which comprises one or more image sensors (also referred to as imagers) configured to capture image data (e.g. a still image or a video). The captured image data may be transferred to an image analysis unit which analyzes the image data using one or more image analysis algorithms. Examples for image analysis algorithms are
· a lane detection algorithm configured to detect whether the vehicle stays within a current lane of the road that the vehicle is driving on; and/or
• an obstacle detection algorithm configured to detect whether an obstacle (e.g. a pedestrian or an animal) appears on the trajectory of the vehicle. The image analysis algorithms are typically directed at the detection of one or more objects within the image data provided by one or more imagers. The image analysis algorithms typically make use of computer vision techniques, in particular object recognition techniques, such as CAD-like object-based methods, appearance-based methods and/or feature-based methods.
Figs, la and lb illustrate systems 100, 120 for evaluating the performance of such camera- based driver assistance systems. The system 100 comprises an image sensor (also referred to as an imager) 101. The imager 101 generates image data 111 which is transmitted to an image analysis unit 102. The image data 111 may be in a pre-determined format, e.g. in a format which is in accordance to the ITU-R BT.601 specification (for video signals with 525-line or 625-line television systems) or in accordance to the ITU-R BT.709
specification (for high definition video signals of e.g. 1080-line HD television systems). The imager 101 may comprise an appropriate interface for generating the image data 111 in the pre-determined format.
The image analysis unit 102 receives the image data 111 and performs one or more image analysis algorithms in order to detect one or more objects within the received image data 111. The analyzed image data 112 (e.g. comprising one or more detected objects) may then be passed to an evaluation unit 103. The evaluation unit 103 may be configured to evaluate the performance of the image analysis unit 102. For this purpose, the evaluation unit 103 may have access to benchmark data, and may be configured to compare the benchmark data with the analyzed image data 112. The benchmark data may e.g. be determined by a person who analyzes the image data 111 provided by the imager 101 and who performs a manual detection of one or more benchmark objects within the image data 111. In any case, the benchmark data may comprise one or more benchmark objects, and can be compared to the analyzed image data 112 comprising one or more detected objects, in order to determine a performance indicator of the image analysis unit 102 (e.g. in order to determine a detection rate of the image analysis unit 102).
The system 100 of Fig. la does not make use of image compression such that the image data 111 generated by the imager 101 is provided - as is - to the image analysis unit 102. Due to the increase of the amount of data transmission within a vehicle and due to the use of a shared network infrastructure (such as Gigabit Ethernet), the image data 111 may be submitted to lossy compression schemes, in order to reduce the bandwidth required for transmitting the image data 111 to the image analysis unit 102. This is illustrated in Fig. lb which shows an example system 120 for evaluating the performance of a driver assistance system 140 comprising lossy image compression. The image data 111 is encoded (typically directly at the location of the imager 101 prior to transmission over the on-board network of the vehicle) within a compression unit 121 (also referred to as an encoding unit 121). The compression unit 121 may be configured to apply a lossy compression algorithm (e.g. JPEG, MJPEG, MPEG, H.264) to the image data 111, thereby providing compressed image data 131.
The compressed image data 131 is transmitted over the on-board network of the vehicle to a decompression unit 122 (also referred to as a decoding unit 122) which is configured to apply a decompression algorithm (which corresponds to the compression algorithm applied within the compression unit 121) to the compressed image data 131, thereby providing decoded image data 132. The decoded image data 132 typically differs from the (original) image data 1 11 generated by the imager 101. The deviations of the decoded image data 132 depend on the settings of the compression unit 121, wherein the settings may e.g. relate to one or more of:
• the compression algorithm (e.g. MJPEG, JPEG, MPEG, H.264) used within the compression unit 121 / decompression unit 122;
• the compression ratio (i.e. the ratio of the size (e.g. measured in bits) of the image data 111 and the size (e.g. measured in bits) of the compressed image data 131);
• the block-size (in case of a block-based image encoder/decoder) used by the
compression algorithm;
• a target bit-rate per frame vs. target bit-rate per block (this setting allows to impose a total number of bits for a complete frame of the image data 111, thereby enabling an overall distribution of the total number of bits to a complete frame of the image data 111 , vs. to impose a number of bits per block of a frame of the image data 11 1, thereby ensuring a minimum quality per block of a frame of the image data 111).
• a target average bit-rate per second or per pre-determined time interval;
• a maximum bit-rate per second or per pre-determined time interval (thereby
ensuring that an instantaneous bandwidth does not exceed a pre-determined bandwidth limit); a quality factor (e.g. a signal-to-noise ratio or a psycho-visually motivated signal- to-noise ratio) to be achieved; the quality factor may be defined as a target (e.g. average) quality factor, as a minimum quality factor or as a maximum quality factor;
a maximum bit-rate per image block (e.g. per macro-block in JPEG);
an amount of cutoff of bits in an encoded frame; such a cutoff may occur before transmission of the image data 111 (notably for bandwidth limitation of the image data 111);
an amount of adjustment and/or a type/algorithm of adjustment of the value for any of the above mentioned settings for the encoding of a current frame, based on the knowledge from the encoding of one or more previous frames;
an amount of adjustment and/or a type/algorithm of adjustment of the value of any of the above mentioned settings before or while encoding the next frame, the next macroblock, and/or the next block;
an amount of adjustment of the value of any of the above mentioned settings, based on the available resources at the encoder, the decoder, and/or on the on-board transmission network;
an amount of adjustment of the value of any of the above mentioned settings, based on one or more active image analysis algorithms used within the image analysis unit 102.
It should be noted that the one or more settings of the compression unit 121 which may be selected and/or modified typically depend on the compression algorithm used within the compression unit 121.
Typically, the compression algorithm applied within the compression unit 121 is directed at achieving a target compression ratio or a target bit-rate subject to psycho-visual quality criteria, thereby reducing artifacts comprised within the decoded image data 132, which are visible to a human being. The psycho-visual quality criteria may not be optimal when using the decoded image data 132 for image analysis within the image analysis unit 102. As such, a further setting of the compression unit 121 may be
• the quality criteria applied by the compression algorithm. The image analysis unit 102 applies one or more image analysis algorithms to the decoded image data 102 in order to generate the analyzed image data 133. The analyzed image data 133 may comprise one or more detected objects. The evaluation unit 103 determines a performance indicator of the image analysis unit 102 or of the driver assistance system 140 comprising the imager 101, the compression unit 121, the decompression unit 122 and the image analysis unit 102, based on the analyzed image data 133 and based on benchmark data comprising one or more benchmark objects. The performance indicator may e.g. be directed at a rate of detection of objects captured by the imager 101.
The lossy compression algorithm applied to the image data 111 typically affects the performance of the driver assistance system 140. It is desirable to improve the performance of the driver assistance system 140 by adjusting the various settings of the compression unit 121 depending on the image processing algorithm used within the image analysis unit 102. This may be achieved using a trial-and-error approach by adjusting the settings of the compression unit 121 in conjunction with a particular image processing algorithm during a number of test drives. The use of test drives is, however, time consuming and typically does not lead to optimal results, because the conditions during the test drives are not reproducible. Furthermore, the results obtained during different test drives are usually not comparable, as the conditions during the different test drives (e.g. the conditions of the light and of the objects which are to be detected) vary.
Figs. 2a and 2b show block diagrams of example systems 200, 220 which may be used to determine and/or tune the driver assistance system of a vehicle in an efficient and reproducible manner. The systems 200, 220 correspond to the systems 100, 120 of Figs, la, lb, respectively. However, the imagers 101 have been replaced by imager simulators 201, respectively. The imager simulators 201 are configured to provide image data 111 in the same format (e.g. according to the same protocol) as the imagers 101 of Figs, la, lb. As such, the imager simulators 201 may be coupled to the network (system 200) or to the compression unit 121 (system 220) in the same manner as the actual imagers 101.
However, the imager simulators 201 are further configured to render pre-recorded reference data as image data 111. As such, the same image data 111 can be re-produced multiple times (using the pre-recorded reference data), thereby allowing for reproducible test sequences of the driver assistance system of a vehicle. The driver assistance system remains unchanged, apart from the imager 101 which is replaced by the imager simulator 201, thereby allowing for Hardware-in-the-Loop testing which is as close as possible to the actual driver assistance system used within the vehicle.
Fig. 3 shows a block diagram of an example imager simulator 201. The imager simulator 201 of Fig. 3 comprises an image source 301 (e.g. a personal computer, a logger of video data, or a pattern generator) which is configured to store the pre-recorded reference data 302 and which is configured to provide the pre-recorded reference data 302 to a processing unit 310. The processing unit 310 may be implemented e.g. using an FPGA (field- programmable gate array). The processing unit 310 comprises a reference data interface unit 312 which is configured to request and receive the pre-recorded reference data 302 from the image source 301. By way of example, the reference data interface unit 312 comprises a Gigabit-Ethernet interface to communicate with the image source 301.
The pre-recorded reference data 302 may be stored in a variable buffer 313. The reference data interface unit 312 may be configured to send control data (e.g. Ethernet SYNC- frames) to the image source 301 in order to control the transmission rate of the prerecorded reference data 302 from the image source 301. The control data may be sent to the image source 301 in dependence of the fill level of the buffer 313 and/or in dependence of the frame rate of the image data 111 generated by the imager simulator 201, thereby regulating the data flow from the image source 301 to the processing unit 310. The buffer 313 may be configured to compensate for variations in the data flow, thereby ensuring that the image data 111 can be generated at a stable frame rate and/or with a timing which is adjustable to the timing of the compression unit 121 or of the on-board network.
The processing unit 310 comprises an imager interface unit 314 configured to generate the image data 111 according to the pre-determined format. As indicated above, the format may e.g. be in accordance to the ITU-R BT.601 or ITU-R BT.709 standards. In particular, the imager interface unit 314 may be configured to take the pre-recorded reference data 302 from the buffer 313 and format the pre-recorded reference data 302 in accordance to the pre-determined format, thereby generating the image data 111. The pre-determined format of the image data 111 may relate to one or more of the following aspects: • the underlying specification (e.g. ITU-R BT.601 or ITU-R BT.709);
• the resolution of an image (e.g. 525-lines or 625-lines in case of ITU-R BT.601, or 1080-lines or 1052-lines in case of ITU-R BT.709);
• the color space used to represent color information (e.g. the YCbCr color space using a luma component, a blue-difference component and a red-difference component, or the YUV color space using a luma component and two chrominance components);
• chroma subsampling (i.e. subsampling of chroma information with respect to luma information, e.g. YCbCr 4:2:2 or YUV 4:2:2);
• the number of bits per component (e.g. 8 bits, 10 bits or 12 bits per component);
• the frame rate (e.g. 60Hz, 50Hz, 30Hz, 25Hz, or 24Hz);
• the pixel clock (PCLK), indicating the rate at which a block of data (e.g. 8 bits) is output as image data 111 ;
• a length of the Hsync signal (e.g. pulse) indicating the end of a line; or a length of a Href signal indicating the length of a line;
• a polarity of the Vsync signal, indicating the end of a frame;
• interlaced or de-interlaced operation of the imager;
• timing information to synchronize with the subsequent compression unit 102;
• a serial or parallel transmission format.
The imager interface unit 314 may provide a physical interface (e.g. a connector) which is in accordance to the physical interface of the imager 101. By way of example, the imager interface unit 314 may comprise a 25-pin Sub-D connector, in case of parallel transmission format, or a BNC (Bayonet Neill-Concelman) connector for connecting a coaxial cable, in case of serial transmission.
The processing unit 310 may further comprise a control unit 311 configured to control the operation of the reference data interface unit 312, the buffer 313 and/or the imager interface unit 314. The control unit 311 may be used to configure other interfaces of the processing unit 310, such as a serial interface, an I2C interface and/or an SPI interface (not shown). The system 220 of Fig. 2b which comprises the imager simulator 201 can be used in a method 400 for determining settings of the compression unit 121, which improve the performance of the image analysis performed within the image analysis unit 102 downstream of the compression unit 121. The method 400 makes use of pre-determined reference data 302 stored on the image source 301. In step 401 , the pre-determined reference data 302 is rendered as image data 111 using the imager simulator 201. The image data 111 is processed by the compression unit 121 and the de-compression unit 122 using a first set of settings (step 402), thereby generating first decoded image data 132. The image analysis unit 102 analyzes the first decoded image data 132 using one or more image analysis algorithms, thereby yielding first analyzed image data 133 (step 403). Subsequently, the evaluation unit 103 determines a first performance indicator for the image analysis unit 102 based on the first analyzed image data 133 (and possibly based on benchmark data) (step 404). The above mentioned scheme (i.e. method steps 401 to 404) may be repeated (step 405) for a plurality of different sets of settings of the compression / de-compression units 121, 122 using the same pre-determined reference data 302, thereby determining a corresponding plurality of performance indicators for the image analysis unit 102 (using the one or more image analysis algorithms). The set of settings for a current iteration of the method 400 may be determine based on the sets of settings used for the one or more preceding iterations, and/or based on the performance indicators determined within the one or more preceding iterations. In particular, optimization techniques (such as gradient decent algorithms or heuristic algorithms) may be used to determine the set of settings for the current iteration based on the sets of settings and the performance indicators of the one or more preceding iterations. Alternatively or in addition, the plurality of sets of settings may be pre-determined in accordance to a pre-determined test protocol.
The method 400 may terminate, e.g. after a pre-determined number of iterations, and/or after having determined the plurality of performance indicators for a plurality of pre- determined sets of settings, and/or after having determined a (local) optimum for the performance indicator. The method 400 may then comprise the step 406 of selecting a set of settings from the plurality of sets of settings which corresponds to the maximum performance indicator of the plurality of determined performance indicators. As such, the method 400 allows to determine a set of settings for the compression / decompression units 121, 122 which allows to increase (e.g. maximize) the performance of the image analysis unit 102 (e.g. which allows to maximize the detection rate) when using the one or more image analysis algorithms.
The method 400 may be performed for different image analysis algorithms (e.g. for lane detection and/or for pedestrian detection), thereby determining improved sets of settings for the compression / decompression units 121, 122 for different image analysis algorithms. The respective improved sets of settings may then be used in conjunction with the different image analysis algorithms, thereby improving the performance of the diver assistance system 140 (comprising the actual imager 101), depending on the one or more image analysis algorithms in use. By way of example, the driver assistance system 140 may be configured to perform pedestrian detection up to a pre-determined maximum speed (e.g. of 60 km/h). Furthermore, the driver assistance system 140 may be configured to perform lane detection for speeds higher than the pre-determined maximum speed. As such, the driver assistance system 140 may be configured to use the set of settings which were optimized for pedestrian detection at speeds up to the pre-determined maximum speed, and configured to use the set of settmgs which were optimized for lane detection at speeds above the pre-determined maximum speed.
The method 400 using the imager simulator 201 allows for a time- and cost-efficient, as well as a reproducible scheme for determining the set of settings for a compression algorithm to be used in conjunction with a particular image analysis algorithm, thereby improving the performance of the particular image analysis algorithm. As a result, the design and the optimization of camera-based driver assistance systems can be accelerated and improved.
It should be noted that various variants of the systems 220 and the method 400 may be implemented:
By way of example, the systems 200, 220 and/or the method 400 may be used to simulate the behavior of the driver assistance system 140 in response to a faulty imager 101. The imager simulator 201 may be configured to generate erroneous image data 111 (e.g. a freeze frame, or frames of a video comprising faulty pixels). The performance of the image analysis unit 102 subject to such erroneous image data 111 may be evaluated, thereby providing insights with regards to the robustness of the driver assistance system 140. The systems 200, 220 and/or the method 400 may be extended to a plurality of imager simulators 201, thereby simulating the presence of a corresponding plurality of imagers 101 within the driver assistance system 140. By way of example, the plurality of imagers 101 may be used for stereo vision or for parking cameras, and the plurality of imager simulators 201 may be configured to generate appropriate image data 111 (based on appropriate pre-determined reference data 302) in order to simulate such stereo or parking situations. By way of example, the pre-determined reference data 302 for the plurality of imager simulators 201 may have been recorded using the respective imagers 101 of the actual driver assistance system 140. The plurality of imager simulators 201 may be configured to generate the respective image data 111 in a synchronized manner, thereby simulating the real-life situation of the driver assistance system 140. If required, a predetermined temporal offset between the image data 111 generated by the different imager simulators 201 may be implemented.
As already indicated above, the pre-determined reference data 302 is not limited to image data pre-recorded by an actual imager 101 within the actual driver assistance system 140. By way of example, artificial image data (e.g. generated by a pattern generator) may be used as pre-determined reference data 302, in order to test the driver assistance system 140 under extreme conditions. Alternatively or in addition, the pre-determined reference data 302 may be adapted to test extreme situations with regards to the compression algorithms applied within the compression unit 121. By way of example, the pre-determined reference data 302 may be selected in order to test a "worst case" scenario with regards to compression aspects, thereby allowing to appropriately design the compression unit 121 (e.g. memory and/or processing capacity of the compression unit 121) for such a "worst case" scenario.
It should be noted that the systems 200, 220 and/or the method 400 may be used
(alternatively or in addition) to optimize other parameters of the driver assistance system 140. Overall, the imager simulator 201 creates a cost/time-efficient and reproducible test environment for the driver assistance system 140 and may therefore be used to tune the various parameters of the driver assistance system 140 (including those not mentioned in the present document). Fig. 5 illustrates a block diagram of another example system 500 for evaluating and/or optimizing the performance of a driver assistance system. The system 500 is particularly advantageous for tuning the one or more settings of the video compression unit 121 (and of the video compression scheme used therein). The system 500 comprises a reference system 510 (which - in conjunction with the imager simulator 201 - corresponds to the system 200 described in Fig. 2a), which evaluates the performance of the driver assistance system (and in particular, the performance of the image analysis unit 102) without the impact of image compression. The imager simulator 201, comprising the image source 301 (e.g. a database) and the processing unit 310, provides (raw) image data 111 which is analyzed directly by the image analysis unit 102, thereby providing reference analyzed image data 112. The evaluation unit 103 of the reference system 510 may be configured to determine a performance indicator of the driver assistance system without compression, based on the reference analyzed image data 112. As outlined above, the evaluation unit 103 of the reference system 510 may make use of benchmark data 502 (e.g. benchmark data 502 provided by the imager simulator 201) for this purpose.
Furthermore, the system 500 comprises the evaluation system 520 which - in conjunction with the imager simulator 201 - corresponds to the system 220 that has already been described in the context of Fig. 2b. The evaluation unit 103 of the evaluation system 520 determines a performance indicator of the driver assistance system including compression, based on the analyzed image data 112 (and possibly based on the benchmark data 502). The comparison unit 501 of system 500 may be configured to compare the performance indicators provided by the reference system 510 and by the evaluation system 520. By using the information provided by the reference system 510, the comparison unit 501 is enabled to clearly identify the performance deteriorations of the image analysis unit 102 which are due to the image compression applied within the driver assistance system. In particular, the comparison unit 501 may exclude performance issues which are inherent to the image analysis algorithm used within the image analysis unit 102 (and which are not necessarily due to compression). The system 500 may further comprise a parameter tuning unit 511 which is configured to tune the one or more settings of the compression scheme used within the compression unit 121 (using the feedback provided by the comparison unit 501), thereby improving the overall performance of the driver assistance system comprising image compression (e.g. using the method of Fig. 4).
In the present document methods and systems for improving the performance of a camera- based driver assistance system have been described. In particular, an imager simulator has been described which may be used in lieu of an actual imager of the camera-based driver assistance system, thereby allowing to test the camera-based driver assistance system in an efficient and reproducible manner. A setup comprising the imager simulator may be used to adjust various parameters of the camera-based driver assistance system (in particular, to adjust the settings of a compression unit comprised within the camera-based driver assistance system), thereby improving the overall performance of the camera-based driver assistance system.
The methods and systems described in the present document may be implemented as software, firmware and/or hardware. Certain components may e.g. be implemented as software running on a digital signal processor or microprocessor. Other components may e.g. be implemented as hardware and or as application specific integrated circuits. The signals encountered in the described methods and systems may be stored on media such as random access memory or optical storage media. They may be transferred via networks, such as radio networks, satellite networks, wireless networks or wireline networks, e.g. an Ethernet network. Typical devices making use of the methods and systems described in the present document are driver assistance systems in a vehicle (e.g. in a car or a truck).
The software may be referred to as a computer program and the invention may also be implemented in a computer program for running on a programmable apparatus, at least including code portions for performing steps of a method according to the invention when run on a programmable apparatus, such as a computer system or enabling a programmable apparatus to perform functions of a device or system according to the invention.
The computer program may be stored, e.g. internally in the programmable apparatus, on a computer readable storage medium or transmitted to the programmable apparatus via a readable transmission medium. All or some of the computer program may be provided on tangible or non-tangible computer readable media permanently, removably or remotely coupled to an information processing system. The computer readable media may be transitory or non transitory and include, for example and without limitation, any number of the following: magnetic storage media including disk and tape storage media; optical storage media such as compact disk media (e.g., CD-ROM, CD-R, etc.) and digital video disk storage media; nonvolatile memory storage media including semiconductor-based memory units such as FLASH memory, EEPROM, EPROM, ROM; ferromagnetic digital memories; MRAM; volatile storage media including registers, buffers or caches, main memory, RAM, etc.; and data transmission media including computer networks, point-to- point telecommunication equipment, and carrier wave transmission media, just to name a few.

Claims

CLAIMS 1 ) An imager simulator (201) configured to be used in lieu of an imager (101) within a vehicle, the imager simulator (201) comprising
- an image source (301) configured to store pre-determined reference data (302); and
- an imager interface unit (314) configured to generate image data (111) based on the pre-determined reference data (302); wherein the image data (111) conforms to a pre-determined format; and wherein the pre-determined format corresponds to a format of image data generated by the imager (101) .
2) The imager simulator (201) of claim 1 , wherein the pre-determined format specifies one or more of the following:
- a pre-determined standard, e.g. the ITU-R BT.601 or the ITU-R BT.709
standard;
- a resolution of a frame of the image data (111);
- a frame-rate of succeeding frames of the image data (111);
- a color space used to represent color information comprised within the image data (111);
- a chroma subsampling scheme;
- a number of bits used to encode a pixel of the image data (111);
- a rate at which the bits which encode a pixel of the image data (111) are
transmitted;
- a use of synchronization signals, e.g. of Hsync, Href and/or Vsync signals;
- an interlaced or de-interlaced operation of the imager (101);
- timing information for synchronization of the imager simulator (201) with one or more components downstream of the imager (101); and - a serial or parallel transmission of the image data (111).
3) The imager simulator (201) of any previous claim, wherein the imager simulator (201) comprises a physical connector which conforms to the physical connector of the imager (101).
4) An evaluation system (200, 220, 500) for determining a performance indicator of an imager-based driver assistance system, the evaluation system (200, 220, 500) comprising
- an imager simulator (201) according to any of claims 1 to 3, in lieu of an
imager (101) of the driver assistance system; wherein the imager simulator (201) is configured to generate image data (111) based on pre-determined reference data (302);
- an image analysis unit (102) configured to analyze data derived from the image data (111), thereby generating analyzed image data (112, 133); and
- an evaluation unit (103) configured to determine the performance indicator based on the analyzed image data (112, 133).
5) The evaluation system (200, 220, 500) of claim 4, wherein the evaluation unit (103) configured to determine the performance indicator also based on benchmark data derived from the pre-determined reference data (302).
6) The evaluation system (200, 220, 500) of claim 5, wherein
- the benchmark data is indicative of one or more benchmark objects;
- the analyzed image data (112, 133) is indicative of zero or more detected
objects; and
- the performance indicator comprises a detection rate of the one or more
benchmark objects within the analyzed image data (112, 133).
7) The evaluation system (220, 500) of any of claims 4 to 6, further comprising
- a compression unit (121) configured to encode the image data (111) using a lossy compressing scheme, thereby yielding compressed image data (131); and - a decompression unit (122) configured to decode the compressed image data (131), thereby yielding decoded image data (133); wherein the image analysis unit (102) is configured to analyze the decoded image data (133) to provide the analyzed image data (133).
8) The evaluation system (220, 500) of claim 7, wherein the compression unit (121)
comprises one or more of the following adjustable settings:
- the compression scheme, e.g. a MJPEG, a JPEG, an MPEG, or an H.264
algorithm, used within the compression unit (121);
- an overall target bit-rate or an overall compression ratio implemented by the compression scheme;
- a maximum bit-rate per frame of the image data (111) or a maximum bit-rate per subregion of a frame of the image data (111);
- a target average bit-rate per pre-determined time interval;
- a maximum bit-rate per pre-determined time interval;
- a minimum value for a quality of the compressed image data (131) with respect to the image data (111); and
- a quality criteria applied by the compression scheme when encoding the image data (111).
9) The evaluation system (500) of any of claims 7 to 8, further comprising
- a reference image analysis unit (102) configured to analyze the image data
(111) , thereby generating reference analyzed image data (112); and
- a comparison unit (501) configured to determine a performance deterioration of the imager-based driver assistance system due to the lossy compressing scheme applied to the image data (111), based on the reference analyzed image data
(112) and based on the analyzed image data (133).
10) A method (400) for determining a performance indicator of an imager-based driver assistance system, the method (400) comprising - generating (401) image data (111) based on pre-determined reference data (302) using an imager simulator (201) in lieu of an imager (101) of the driver assistance system;
- analyzing (403) data derived from the image data (111) using a first image analysis algorithm, thereby generating analyzed image data (112, 133); and
- determining (404) the performance indicator based on the analyzed image data (112, 133).
11) The method (400) of claim 10, wherein the pre-determined reference data (302)
comprises one or more of the following:
- image data recorded using the imager (101) of the driver assistance system
(140);
- image data generated by a pattern generator; and
- image data representing a freeze image.
12) The method (400) of any of claims 10 to 11, further comprising
- encoding (402) the image data (111) using a lossy compressing scheme and a set of settings for the compression scheme, thereby yielding compressed image data (131); and
- decoding (402) the compressed image data (131), thereby yielding decoded image data (133); wherein the analyzed image data (133) is determined based on the decoded image data (133).
13) The method (400) of claim 12, further comprising
- repeating (405) the generating (401), the encoding (402), the decoding (402), the analyzing (403) and the determining (404) using the same pre-determined reference data (302) and using a plurality of different sets of settings of the compression scheme, thereby yielding a corresponding plurality of performance indicators; and
- selecting (406) a set of settings from the plurality of different set of settings based on the plurality of performance indicators. 14) The method (400) of claim 13, further comprising determining the plurality of performance indicators using a second image analysis algorithm, different from the first image analysis algorithm.
15) An imager-based driver assistance system (140) for a vehicle, comprising
- an imager (101) configured to generate image data (111);
- a compression unit (121) configured to encode the image data (111) using a lossy compressing scheme, thereby yielding compressed image data (131); wherein the compression scheme is adjustable using one or more settings;
- a decompression unit (122) configured to decode the compressed image data (131), thereby yielding decoded image data (133);
- an image analysis unit (102) configured to analyze the decoded image data (133) using at least one of a plurality of image analysis algorithms, thereby yielding analyzed image data (133); wherein the analyzed image data (133) is usable for assisting a driver of the vehicle; and
- a memory unit configured to store one or more settings of the compression scheme for each of the plurality of image analysis algorithms, respectively; wherein the driver assistance system (140) is configured to set the one or more settings stored in the memory unit, in dependence of the at least one image analysis algorithm used by the image analysis unit (102).
PCT/EP2012/064885 2012-07-30 2012-07-30 Method and system for optimizing image processing in driver assistance systems Ceased WO2014019602A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/EP2012/064885 WO2014019602A1 (en) 2012-07-30 2012-07-30 Method and system for optimizing image processing in driver assistance systems
US14/608,357 US20150139500A1 (en) 2012-07-30 2015-01-29 Method and System for Optimizing Image Processing in Driver Assistance Systems

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/EP2012/064885 WO2014019602A1 (en) 2012-07-30 2012-07-30 Method and system for optimizing image processing in driver assistance systems

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US14/608,357 Continuation US20150139500A1 (en) 2012-07-30 2015-01-29 Method and System for Optimizing Image Processing in Driver Assistance Systems

Publications (1)

Publication Number Publication Date
WO2014019602A1 true WO2014019602A1 (en) 2014-02-06

Family

ID=46640009

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2012/064885 Ceased WO2014019602A1 (en) 2012-07-30 2012-07-30 Method and system for optimizing image processing in driver assistance systems

Country Status (2)

Country Link
US (1) US20150139500A1 (en)
WO (1) WO2014019602A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112249022A (en) * 2020-10-29 2021-01-22 北京罗克维尔斯科技有限公司 Performance analysis method and device of advanced vehicle driving assistance system

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150120035A1 (en) * 2013-10-25 2015-04-30 Infineon Technologies Ag Systems and Methods for Linking Trace Information with Sensor Data
KR102432804B1 (en) 2015-06-02 2022-08-16 한화테크윈 주식회사 Video capture device using MJPEG
WO2017017972A1 (en) * 2015-07-24 2017-02-02 オリンパス株式会社 Image data transmission system
GB2552511A (en) 2016-07-26 2018-01-31 Canon Kk Dynamic parametrization of video content analytics systems
US10757320B2 (en) 2017-12-28 2020-08-25 Waymo Llc Multiple operating modes to expand dynamic range
US20190208136A1 (en) * 2017-12-29 2019-07-04 Waymo Llc High-speed image readout and processing
CN108399596B (en) * 2018-02-07 2020-12-18 深圳奥比中光科技有限公司 Depth image engine and depth image calculation method
EP3570062B1 (en) * 2018-05-18 2022-06-29 Aptiv Technologies Limited Radar system and method for receiving and compressing radar signals
US12405888B2 (en) * 2022-06-27 2025-09-02 Advanced Micro Devices, Inc. Dynamic performance adjustment

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120053754A1 (en) * 2010-08-31 2012-03-01 Karen Pease Electronic communications and control module

Non-Patent Citations (5)

* Cited by examiner, † Cited by third party
Title
M. RAHMANI ET AL: "Real-Time Video Compression for Driver Assistance Camera Systems", 22 July 2008 (2008-07-22), Dublin, Ireland, XP055064649, ISBN: 978-9-63-979927-1, Retrieved from the Internet <URL:http://eudl.eu/article.php?id=3553> [retrieved on 20130529] *
MEHRNOUSH RAHMANI ET AL: "A novel network design for future IP-based driver assistance camera systems", NETWORKING, SENSING AND CONTROL, 2009. ICNSC '09. INTERNATIONAL CONFERENCE ON, IEEE, PISCATAWAY, NJ, USA, 26 March 2009 (2009-03-26), pages 457 - 462, XP031455554, ISBN: 978-1-4244-3491-6 *
WOLFGANG HINTERMAIER ET AL: "A novel real-time video data scheduling approach for driver assistance services", INTELLIGENT VEHICLES SYMPOSIUM (IV), 2011 IEEE, IEEE, 5 June 2011 (2011-06-05), pages 1 - 6, XP031998901, ISBN: 978-1-4577-0890-9, DOI: 10.1109/IVS.2011.5940391 *
WOLFGANG HINTERMAIER ET AL: "A system architecture for IP-camera based driver assistance applications", INTELLIGENT VEHICLES SYMPOSIUM (IV), 2010 IEEE, IEEE, PISCATAWAY, NJ, USA, 21 June 2010 (2010-06-21), pages 540 - 547, XP031732264, ISBN: 978-1-4244-7866-8 *
YANIV ALON: "off-vehicle evaluation of camera based pedestrian detection", PROCEEDINGS INTELLIGENT VEHICLES SYMPOSIUM 2012, 3 June 2012 (2012-06-03), pages 352 - 358, XP055058294, Retrieved from the Internet <URL:ieeexplore.ieee.org> [retrieved on 20130403], DOI: 10.1109/IVS.2012.6232160 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112249022A (en) * 2020-10-29 2021-01-22 北京罗克维尔斯科技有限公司 Performance analysis method and device of advanced vehicle driving assistance system

Also Published As

Publication number Publication date
US20150139500A1 (en) 2015-05-21

Similar Documents

Publication Publication Date Title
US20150139500A1 (en) Method and System for Optimizing Image Processing in Driver Assistance Systems
KR102230776B1 (en) Linear encoder for image/video processing
CN112913237B (en) Artificial intelligence encoding and decoding methods and apparatus using deep neural networks
US10362306B2 (en) Image communication apparatus, image transmission apparatus, and image reception apparatus
CN105306883B (en) Image receiving apparatus, image transmission system, and image receiving method
KR101735025B1 (en) Method, device, and system for pre-processing a video stream for subsequent motion detection processing
US10085015B1 (en) Method and system for measuring visual quality of a video sequence
US10148963B2 (en) Methods of and apparatus for encoding data arrays
CN112383778B (en) Video coding method and device and decoding method and device
CN104756497B (en) image delivery system
KR101686155B1 (en) A method for configuration of video stream output from a digital video camera
US20130235931A1 (en) Masking video artifacts with comfort noise
US11082711B2 (en) Image coding device, image coding method, and image falsification identification program
US20130121422A1 (en) Method And Apparatus For Encoding/Decoding Data For Motion Detection In A Communication System
EP2782344A1 (en) Inter-image prediction method and device and corresponding coding method and apparatus
US12113957B2 (en) Program, device, and method for generating significant video stream from original video stream
JP2013229666A (en) Abnormality inspection device and remote monitoring inspection system
JP7533601B2 (en) Information processing device, information processing system, and information processing method
JP7269134B2 (en) Program, server, system, terminal and method for estimating external factor information affecting video stream
US12518400B2 (en) Information processing device, information processing system, and information processing method
US20240420275A1 (en) Image compression/decompression in a computer vision system
EP2466889A2 (en) Multi-camera system for an automobile, automobile and method for operating a multi-camera system in an automobile
CN113727073A (en) Method and system for realizing vehicle-mounted video monitoring based on cloud computing
CN116916030A (en) Method and corresponding device for adaptive video data compression in a vehicle
KR20110087859A (en) Method, apparatus and computer readable medium for adjusting quantization coefficients

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12745448

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12745448

Country of ref document: EP

Kind code of ref document: A1