WO2016209877A1

WO2016209877A1 - System and method for processing streamed video images to correct for flicker of amplitude-modulated lights

Info

Publication number: WO2016209877A1
Application number: PCT/US2016/038639
Authority: WO
Inventors: Tom B. Sherman; Gregory S. Bush; Daniel G. MCMILLAN; Dale M. Radcliff; Igor Majdandzic
Original assignee: Gentex Corporation
Priority date: 2015-06-22
Filing date: 2016-06-22
Publication date: 2016-12-29
Also published as: KR20180019189A; JP2021002869A; JP2018526851A; JP7296350B2; CN107709096B; JP7060958B2; CN107709096A; EP3310618A4; KR102135427B1; EP3310618A1; US11178353B2; US20160373684A1

Abstract

A display system is provided for a vehicle equipped with a camera for supplying streamed video images of a scene rearward of the vehicle. The display system includes an image processing unit for receiving the streamed video images and processing the streamed video images, and a display for displaying the processed streamed video images. To perform processing of the streamed video images, the image processing unit is configured to: detect amplitude-modulated light sources in the streamed video images, classify the detected amplitude-modulated light sources into one of several possible classifications, select the streamed video images in which an amplitude-modulated light source is detected that flickers based upon the classification of the amplitude-modulated light source, and modify the selected streamed video images to correct for flicker of any amplitude-modulated light sources in the selected streamed video images.

Description

SYSTEM AND METHOD FOR PROCESSING STREAMED VIDEO IMAGES TO CORRECT FOR FLICKER OF AMPLITUDE-MODULATED LIGHTS

BACKG ROUND OF THE I NVENTION

[0001] The present invention generally relates to processing of video images streamed to a display, and more specifically to processing of streamed video images of scenes exterior to a vehicle. In some embodiments the present invention pertains even more specifica lly to processing of video images obtained from a rearward facing camera in a vehicle that are streamed to a display serving as a replacement for a rea rview mirror.

SUM MARY OF TH E INVENTION

[0002] According to one aspect of the invention, a display system is provided for a vehicle equipped with a ca mera for supplying streamed video images of a scene rearward of the vehicle. The display system comprising: an image processing unit for receivi ng the streamed video images and processing the streamed video images; and a display for displaying the processed streamed video images. To perform processing of the streamed video images, the image processing unit is configured to: detect amplitude-modulated light sources in the strea med video images, classify the detected amplitude-modulated light sources into one of several possible classifications, select the streamed video images in which an amplitude-modulated light source is detected that flickers based upon the classification of the am plitude-modulated light source, a nd modify the selected streamed video images to correct for flicker of any amplitude-modulated light sources in the selected streamed video images.

[0003] According to one aspect of the invention, a display system is provided that comprises: a n image processing unit for receiving streamed video images and processing the streamed video images; and a display for displaying the processed streamed video images. To perform processing of the streamed video images, the image processing unit is configured to: detect amplitude-modulated light sources in the streamed video images, classify the detected amplitude-modulated light sources into at least two classes where a first class of detected am plitude-modulated light sources having a flicker not perceivable by a human when viewed directly by the huma n, and a second class of detected amplitude-modulated light sources having a flicker that is perceivable by a human when viewed directly by the human, track the detected amplitude-modulated light sources through image frames of the streamed video images, and modify the streamed video images in which an amplitude-modulated light source is detected that is classified in the first class by substituting pixels representing each of the detected amplitude-modulated light sources that is classified in the first class such that the pixels representing the detected amplitude-modulated light source are always at a state so that when the processed streamed video images are displayed, each of the detected amplitude- modulated light sources that is classified in the first class appears to have no perceivable flicker.

[0004] According to one aspect of the invention, a method of processing streamed video images is provided that comprises: detecting am plitude-modulated light sources in the streamed video images; classifying the detected amplitude-modulated light sources into at least two classes where a first class of detected amplitude-modulated light sources having a flicker not perceivable by a human when viewed directly by the human, and a second class of detected amplitude-modulated light sources having a flicker that is perceivable by a human when viewed directly by the human; tracking the detected amplitude-modulated light sources through image frames of the streamed video images; and modifying the streamed video images in which an amplitude-modulated light source is detected that is classified in the first class by substituting pixels representing each of the detected amplitude-modulated light sources that is classified in the first class such that the pixels representing the detected amplitude-modulated light source are always at a state so that when the processed streamed video images are displayed, each of the detected amplitude-modulated light sources that is classified in the first class appears to have no perceivable flicker.

[0005] These and other features, advantages, and objects of the present invention will be further understood and appreciated by those skilled in the art by reference to the following specification, claims, and appended drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0006] In the drawings:

[0007] Fig. 1 is a block diagram of an imaging system according to an embodiment of the invention;

[0008] Fig. 2 is a flow cha rt showing the method steps performed by the image processing unit shown in Fig. 1;

[0009] Fig. 3 is a cut-away plan view of a vehicle in which the imaging system may be implemented;

[0010] Fig. 4A is a front a nd side perspective view of a vehicle rea rview assembly in which various components of the imaging system may be implemented; and

[0011] Fig. 4B is a front elevationa l view of the vehicle rearview assem bly shown in Fig.

4A.

DETAI LED DESCRI PTION OF PREFERRED EMBODI MENTS

[0012] Reference will now be made in detail to the present preferred embodiments of the invention, examples of which are illustrated in the accompa nying drawings. Wherever possible, the sa me reference numera ls will be used throughout the drawings to refer to the same or like parts. I n the drawings, the depicted structural elements are not to sca le and certain components a re en larged relative to the other components for purposes of emphasis a nd understanding.

[0013] A common problem in rendering streaming video data captured from a n imager occurs when the object being imaged is a n amplitude modulated (AM) light source. A very common example of this type of light source is one which is pu lsing on/off at some periodic rate, like a vehicle lamp assembly constructed with light emitting diodes (LEDs), where the LEDs are pulse-width-modulated (PWM), which is a subset of possible amplitude modulation methods. The PWM period and duty cycle result in the LEDs being turned on and off at some periodic rate, and a camera taking streaming images of this lamp assembly will ca pture successive images where an LED may be On' in one or more consecutive images, and then 'off in one or more subsequent images. Other examples of AM light sources include the flashers on an emergency vehicle (which may also be comprised of PWM LEDs), a turn signal on a vehicle, or a fluorescent light source in a tunnel or parking garage.

[0014] For many of the exemplary AM light sources listed above, a human observer of the light source does not perceive any flicker in the 'on/off pattern since the frequency of the on/off pattern is higher than the human vision system can perceive (PWM LED headlamp/tail lamp assemblies being a prime example). But in imaging the AM light element with an electronic camera system, the exposure time, frame rate, and shutter scheme (rolling or globa l) used when capturing the light element at a particular pixel in the imager array may result in some images showing this pixel to be imaging an On' state of the light element, and successive images showing this pixel capturing the 'off state of the light element. In attempting to render these images to a display, at some display frame rate, the display system may end up presenting the human observer an 'on/off pattern that is discernible as a 'flickering' light.

[0015] Fig. 1 shows an image system 10 according to a first embodiment. As shown, image system 10 includes a camera 26 that captures images of a scene and outputs streamed video images of the scene, and a display system 12, which includes an image processing unit 30 that receives the streamed video images and processes the images (as discussed in detail below) and outputs the processed streamed video images, and a display 32 that displays the processed streamed video images.

[0016] The methods and processing sequences described herein are intended to mitigate the 'flickering' phenomena seen in rendered AM headlamps and tail lamps (especially targeted to PWM LED assemblies, but not limited to lighting of that technology). As described below, the platform on which these methods may be implemented is part of an automotive mirror replacement system, where a vehicle mirror is replaced by camera (lens plus digital imager) 26, image processing unit (serial processor and/or ASIC/FPGA) 30, and electronic display (LCD/LED panel) 32. The methods described herein may be incorporated in the image processing unit 30 in the above system 10. As shown in Fig. 2, the method steps may be performed in the following sequence (as would occur on the image processing unit 30): 1) receiving the streamed video images (step 100); 2) detection of the PWM LED (or AM) lights in a succession of the streamed video images (step 102); 3) differentiation/classification of the PWM LED (or AM) elements (which are part of a headlamp or tail lamp assembly) from other illuminating objects in the scene which have time-varying brightness levels (e.g. emergency vehicle lights) (step 104); 4) tracking of the pulsed lights over time (step 106); 5) correction of the flicker artifact associated with these rendered lights in a way that is appropriate to the specific type of light source (step 108); and 6) supplying the processed video streamed images to display 32 (step 110). Possible techniques for each of these steps are detailed below.

[0017] Multiple methods exist for performing step 102 involving detection of time- varying lights in a sequence of captured images. In the problem area of a rearview mirror replacement system (based on an electronic camera 26, an image processing unit 30, and a display system 32), PWM LED lights that may need to be detected are those originating from vehicle headlamp and tail lamp systems. These lights are related to vehicles, which are on the same roadway as the vehicle outfitted with the mirror replacement system. The search space for the PWM LED lights of interest thus can be influenced by roadway detection, where an auto-aim or lane detection system can narrow the light search space to the vertical region above the detected road boundaries (from a lane detection system), or around the focus of expansion (from an auto aim system), and discriminated from stationary non-vehicle light sources. I n this reduced search space, methods exist in existing high beam control systems to detect PWM LED lights as disclosed in commonly- owned U.S. Patent Nos. 6,587,573, 6,593,698, 6,611,610, 6,631,316, 6,653,614, 6,728,393, 6,774,988, 6,861,809, 6,906,467, 6,947,577, 7,321,112, 7,417,221, 7,565,006, 7,567,291, 7,653,215, 7,683,326, 7,881,839, 8,045,760, 8,120,652, and 8,543,254.

[0018] Additionally, detection methods such as frame subtraction may be used for detecting time-varying light sources, where successive images are subtracted from one another to produce temporal difference maps. The resultant maps are then processed by routines (implemented in software or in ASIC/FPGA fabric), which perform some combination of thresholding and/or filtering to identify spatial areas in the map where there were significant changes in pixel brightness between the two source images. The absolute value of the difference data indicates the magnitude of the change in pixel intensity between frames, and the sign of the difference data indicates whether the change in a pixel value between frames is associated with a light source brightening or darkening. The frame data used to generate these temporal difference maps may be raw data from a Bayer patterned image, luminance data extracted from the image, or some other image form extracted from the image processing path. On a typical roadway scene, the most significant deltas in pixel values between a pair of frames (referenced to a single pixel location), tend to be related to these PWM LED (AM) lights which are going from extremely bright, to fully off. Motion artifacts can also contribute to temporal changes in image values at the pixel locations, but in the search space of the roadway imaged by the vehicle, this motion is quite small -- as the image capture rate is rapid compared to vehicle dynamics, and the brightness changes related to objects which do not produce their own illumination is also quite reduced (imaging a vehicle body at a pixel in the first frame to a part of the vehicle bumper in the next frame does not produce as significant a luminance cha nge than the PWM LED is exhibiting in its on/off sequencing).

[0019] Other methods of detecting the presence of AM lights may be leveraged from the imager implementation, where some imagers may supply information (to the pixel level) on whether the scene brightness changed state during the pixel exposure time (especially for an imager such as an HDR CMOS imager).

[0020] As described below, the methods for correctly rendering pulsed lights tend to fall in the category of adding image content to 'brighten' the pulsed light location for durations when the light is captured as 'off and addressing incorrect color measurements induced by the time-varying nature of the lights. The classification operation (step 104) is applied to discriminate between the types of time-varying light sources that introduce the brightness and /or color errors. To ensure only the desired pulsed lights are corrected (and not, for example, motion artifacts), light source classification maybe performed to influence the correction step 108. Methods of classifying PWM LED lights are known in high beam control systems such as those disclosed in commonly-owned U.S. Patent Nos. 6,587,573, 6,593,698, 6,611,610, 6,631,316, 6,653,614, 6,728,393, 6,774,988, 6,861,809, 6,906,467, 6,947,577, 7,321,112, 7,417,221, 7,565,006, 7,567,291, 7,653,215, 7,683,326, 7,881,839, 8,045,760, 8,120,652, and 8,543,254. But other options exist in the use of temporal changes, color, brightness and location. The options for use of brightness and color for classification are greatly enhanced by the use of a Bayer patterned, High Dynamic Range (HDR) imager in the camera system, since bright objects are not saturated with an HDR imager, and the Bayer pattern contributions can be demosaiced to determine color of very bright lights. Object detection systems that classify vehicles can also be used to influence the classification of PWM LED headlamps/tail lamps, by limiting search windows to areas associated with the identified vehicles.

[0021] Basically, the classification can be used to distinguish between those flickering lights that are humanly perceivable when viewing the lights directly from those lights that are not humanly perceivable as flickering when viewing the lights directly. This way, the images of the light sources may be selectively modified based upon such classification so that the light sources will appear in the displayed scenes as they would otherwise appear to a human viewing the lights directly.

[0022] The step of temporal tracking of pulsed lights (step 106) can be performed using the techniques for tracking vehicle lights as described in known high beam control systems such as those disclosed in commonly-owned U.S. Patent Nos. 6,587,573, 6,593,698, 6,611,610, 6,631,316, 6,653,614, 6,728,393, 6,774,988, 6,861,809, 6,906,467, 6,947,577, 7,321,112, 7,417,221, 7,565,006, 7,567,291, 7,653,215, 7,683,326, 7,881,839, 8,045,760, 8,120,652, 8,543,254, and 9,185,363. This temporal and spatial tracking is useful when selectively modifying the images in order to brighten pixels corresponding to the expected location of the flickering light source in those modified images.

[0023] Step 108 involves resolving light flicker for rendering on display 32. With the AM

(or pulsed) lights which need to be addressed for display flicker reduction identified, the method of flicker reduction can be performed by substituting low pixel values (from 'off situations), with values which correspond to levels associated with On' situations. The pixel value replacements can be performed at the raw level (a Bayer pattern color associated with the replaced pixel), or at a later processing step in the processing subsystem. There are advantages to performing pixel replacement at the post-demosaic step, and color can be preserved for the PWM light by creating the correct balance of red, green and blue contributions. To maintain the displayed bounda ries of AM light objects when pixel substitution is being performed, some image processing steps may be used to predict the object outline in an upcoming frame by using the tracking information of step 106, and an object shape detection routine.

[0024] Alternatively, the temporal difference maps from the detection step can be used to define the region of pixels to be substituted (since they represent the pixels which have changed state between frames), with better results possible from using maps that incorporate more than just a two frame difference. One possible implementation of this method would involve creating difference maps of pixel values (by location) across sequences of frames, replacing pixel values that have been determined to be producing images of pulsed PWM LED lights with an average of the highest M values in an N frame sequence (M less than N), if the average of highest M values exceeds some threshold. If analysis of overall image luminance and object color is used, this replacement method may also be used to replace PWM LED detection, classification, tracking and replacement.

[0025] A forward -facing turn signal is one example of a time-varying light source that could be detected, classified, and corrected using the ideas disclosed here. Unlike a PWM light, whose row-to-row values on the imager may vary greatly due to beat frequencies, a turn signal's frequency is significantly lower than a camera's frame rate (1-2 Hz as opposed to 15-120 Hz). This results in areas of the light turning on and off at the turn signal's frequency. This spatial consistency within the boundaries of the light, coupled with a detected frequency that is indicative of a turn signal and a yellowish hue, could allow classification of a light as a turn signal as it is tracked. Once the system knows what kind of light it is, the system can fix it by increasing its yellow saturation, creating more visual appeal, but leave its on/off behavior alone.

[0026] A PWM LED tail lamp is a difficult object to image and visualize correctly beca use it is typically not relatively bright compared to the background. In addition, for rolling shutter cameras, each row of pixels may have a sharply different level of brightness, and this can be exacerbated by the spatial effects of the Bayer filter, leading to many artifacts in both chrominance and luminance. However, some of these characteristics-row-to- row variation, local colors that are wildly different, and colors and intensities that change drastically from frame to frame, etc.-in addition to other characteristics-location in the image, predominance of brighter red pixels, motion toward the focus of expansion, frequency estimation on the light modulation, etc.-could allow classification of these lights with high accuracy. Fixing PWM LED tail lamps could be performed by making the colors a uniformly saturated red while choosing a luminance from the detected range, which would end up being visually appealing and remove harsh artifacts.

[0027] Referring now to Fig. 3, a schematic diagram of a vehicular implementation of the above embodiment is shown. A vehicle 20 is shown that is driven by operator 22. One or more cameras 26 are operative to view a scene 24. In the example shown, scene 24 is generally behind vehicle 20. However, camera 26 may be oriented in a variety of ways to view scenes at other locations about vehicle 20 including, but not limited to, the sides, back, front, bottom, top, a nd inside. In the example shown, signals representative of the scene are sent via channel 28 to an image processing unit 30. Image processing unit 30 produces an enhanced image of scene 24 on one or more displays 32. Input from an optional ambient light sensor 34 and one or more direct glare sensors 36 is also available to image processing unit 30.

[0028] In a particularly useful embodiment, a rearview assembly 50 (Figs. 4A and 4B) is augmented or replaced by imaging system 10 having cameras 26 which cover a wide field of view to the back and sides so that pedestrians or other objects directly in back of vehicle 20 may be seen and so that oncoming traffic from the sides may be seen. The system is designed so that, when backing out of a parking spot, oncoming vehicles may be seen before backing into the lane of travel. This requires camera system 26 with a near 180° field of view or several camera systems 26 mounted near the rear of the vehicle. An ana logous system with a camera or cameras 26 mounted near the front of the vehicle 20 is adapted to view cross traffic at a "blind" intersection before entering the lane of travel of the cross traffic. These are desirable applications for the present invention which supplement the viewing function of conventional rearview mirrors.

[0029] Figs. 4A and 4B show an example of a rearview assembly 50 having a housing 54 with a display 32 and an optional mirror element 52 positioned in front of the display 32. A user switch 56 may optionally be provided for tilting of the mirror element 52 and/or display 32 to reduce glare on the display 32 when activated. Examples of such a rearview assembly 50 are known and are disclosed in commonly-owned U.S. Patent Application Publication Nos. 2015/0219967 Al, 2015/0266427 Al, and 2015/0277203 Al. The optional ambient light sensor 34 and a direct glare sensor 36 may be incorporated in rearview assembly 50 as is known in the art. Further, image processing unit 30 may be disposed in the rearview assembly 50. Rearview assembly 50 may be an interior rearview assembly as shown in Figs. 4A and 4B, or may be an exterior rearview assembly.

[0030] The above description is considered that of the preferred embodiments only.

Modifications of the invention will occur to those skilled in the art and to those who make or use the invention. Therefore, it is understood that the embodiments shown in the drawings and described above are merely for illustrative purposes and not intended to limit the scope of the invention, which is defined by the claims as interpreted according to the principles of patent law, including the doctrine of equivalents.

Claims

CLAIMS What is claimed is:

1. A display system for a vehicle equipped with a camera for supplying streamed video images of a scene rearward of the vehicle, the display system comprising:

an image processing unit for receiving the streamed video images and processing the streamed video images; and

a display for displaying the processed streamed video images,

wherein to perform processing of the streamed video images, the image processing unit is configured to:

detect amplitude-modulated light sources in the streamed video images, classify the detected amplitude-modulated light sources into one of several possible classifications,

select the streamed video images in which an amplitude-modulated light source is detected that flickers based upon the classification of the amplitude- modulated light source, and

modify the selected streamed video images to correct for flicker of any amplitude-modulated light sources in the selected streamed video images.

2. The display system of claim 1, wherein the image processing unit modifies the selected streamed video images such that the pixels representing each of the detected amplitude-modulated light sources are maintained at a state so that when the processed streamed video images are displayed, each of the detected amplitude-modulated light sources that is represented by the pixels appears to have no perceivable flicker.

3. The display system of claim 2, wherein each of the detected amplitude- modulated light sources are maintained by substituting low pixel values from off periods with higher pixel values from on periods.

4. The display system of any one of claims 1-3, wherein the image processing unit is further configured to track the detected amplitude-modulated light sources through image frames of the streamed video images.

5. The display system of claim 4, wherein the image processing unit modifies the selected streamed video images such that the pixels representing each of the detected amplitude-modulated light sources are maintained at a state so that when the processed streamed video images are displayed, each of the detected amplitude-modulated light sources that is represented by the pixels appears to have no perceivable flicker and appears at the expected locations in the images based upon the tracking of each of the detected amplitude-modulated light sources.

6. The display system of any one of claims 1-5, wherein the image processing unit does not modify the streamed video images to correct for flicker from light sources classified as a turn signal or emergency vehicle light.

7. The display system of any one of claims 1-6, wherein the image processing unit classifies the detected amplitude-modulated light sources into at least two classes where a first class of detected amplitude-modulated light sources having a flicker not perceivable by a human when viewed directly by the human, and a second class of detected amplitude-modulated light sources having a flicker that is perceivable by a human when viewed directly by the human.

8. The display system of claim 7, wherein the streamed video images in which an amplitude-modulated light source is detected that is classified in the first class is modified by substituting pixels representing each of the detected amplitude-modulated light sources that is classified in the first class such that the pixels representing each of the detected amplitude-modulated light sources are always at a state so that when the processed streamed video images are displayed, the detected amplitude-modulated light source that is classified in the first class appears to have no perceivable flicker.

9. The display system of any one of claims 7-8, wherein the image processing unit classifies the detected amplitude-modulated light sources into the first class when a frequency of the flicker in the light sources is above a threshold frequency and classifies the detected amplitude-modulated light sources into the second class when a frequency of the flicker in the light sources is below the threshold frequency.

10. A rearview assembly for mounting to the vehicle, the rearview assembly comprising the display system of any one of claims 1-9.

11. A display system comprising:

an image processing unit for receiving streamed video images and processing the streamed video images; and

a display for displaying the processed streamed video images,

detect amplitude-modulated light sources in the streamed video images, classify the detected amplitude-modulated light sources into at least two classes where a first class of detected amplitude-modulated light sources having a flicker not perceivable by a human when viewed directly by the human, and a second class of detected amplitude-modulated light sources having a flicker that is perceivable by a human when viewed directly by the human,

track the detected amplitude-modulated light sources through image frames of the streamed video images, and

modify the streamed video images in which an amplitude-modulated light source is detected that is classified in the first class by substituting pixels representing each of the detected amplitude-modulated light sources that is classified in the first class such that the pixels representing the detected amplitude-modulated light source are always at a state so that when the processed streamed video images are displayed, each of the detected amplitude- modulated light sources that is classified in the first class appears to have no perceivable flicker.

The display system of claim 11, wherein the image processing unit classifies the detected amplitude-modulated light sources into the first class when a frequency of the flicker in the light sources is above a threshold frequency and classifies the detected amplitude-modulated light sources into the second class when a frequency of the flicker in the light sources is below the threshold frequency.

13. The display system of claim 12, wherein each of the detected amplitude- modulated light sources are maintained by substituting low pixel values from off periods with higher pixel values from on periods.

14. The display system of any one of claims 11-13, wherein the image processing unit does not modify the streamed video images to correct for flicker from light sources classified in the second class.

15. The display system of claim 14, wherein light sources classified in the second class include turn signals and emergency vehicle lights.

16. A rearview assembly for mounting to the vehicle, the rearview assembly comprising the display system of any one of claims 11-15.

17. A method of processing streamed video images comprising:

detecting amplitude-modulated light sources in the streamed video images;

classifying the detected amplitude-modulated light sources into at least two classes where a first class of detected amplitude-modulated light sources having a flicker not perceivable by a human when viewed directly by the human, and a second class of detected amplitude-modulated light sources having a flicker that is perceivable by a human when viewed directly by the human;

tracking the detected amplitude-modulated light sources through image frames of the streamed video images; and

modifying the streamed video images in which an amplitude-modulated light source is detected that is classified in the first class by substituting pixels representing each of the detected am plitude-modulated light sources that is classified in the first class such that the pixels representing the detected amplitude-modulated light source are always at a state so that when the processed streamed video images are displayed, each of the detected amplitude-modulated light sources that is classified in the first class appears to have no perceivable flicker.

18. The method of claim 17, wherein the detected amplitude-modulated light sources are classified into the first class when a frequency of the flicker in the light sources is above a threshold frequency and are classified into the second class when a frequency of the flicker in the light sources is below the threshold frequency.

19. The method of any one of claims 17-18, wherein the light sources classified in the second class are not corrected for flicker.

20. The method of claim 19, wherein light sources classified in the second class include turn signals and emergency vehicle lights.