JP2008507449A - Panorama vision system and method - Google Patents

Abstract

  The distorted panoramic vision system and method provides a visually correct composite image that is acquired through a wide-angle optic and projected onto a display surface. The system uses an image acquisition device to capture scenes up to 360 ° or 4π steradian width. The image processor corrects for luminance or chrominance non-uniformities and applies a spatial transformation to each image frame. The spatial transformation is convolved by concatenating the display transformation, the acquired geometry, and the optical distortion transformation, and the display geometry and optical transformation. The distortion correction is applied separately for the red component, the green component and the blue component to eliminate chromatic aberration in the lateral direction of the optical component. A display system is then used to display the resulting composite image on the display device that is then projected through the projection optics onto a display surface. The resulting image is clearly undistorted and matches the characteristics of the display surface.

Description

  The present invention relates to a vision system, and more particularly to a panoramic vision system with image distortion correction.

  There is an increasing need for improved situational awareness for many applications, including surveillance systems, videoconferencing vision systems, and vehicle vision systems. Essential for such an application is the need to monitor a large operating area and form a composite image for easy understanding in different user modes. To minimize the number and cost of cameras, a camera with a panoramic lens allows a wide field of view, but will have distortion due to its inherent geometry and optical nonlinearities. For surveillance applications, multiple panoramic cameras may cover the entire exterior and interior area of the building, and the system can provide a continuous view of the area and track subjects through the area either manually or automatically. .

  In a vehicle vision system, multiple panoramic cameras can provide a full 360 ° and disturbing view of the area. Such a system can adapt the display view to specific operator modes such as turn, reverse, and lane change to improve situational awareness. An additional advantage of the vehicle vision system is that it reduces the air resistance and noise caused by the side mirrors and reduces the vehicle's wide span by eliminating such protruding mirrors. These systems can have the ability to detect moving objects, alert nearby objects, and track such objects through multiple display zones. The vision system could also greatly enhance night vision through various technologies such as infrared devices, radar devices, and photosensitive devices.

  The vision system consists of one or more image acquisition devices coupled to one or more visual display devices. The image acquisition device can incorporate various lenses with different focal lengths and depths, such as planar, panoramic, fish-eye, and annular. Lenses such as fisheye and ring have a wide field of view and a large depth of focus. They can capture a wide and deep field of view. However, they tend to deform the image, especially the edges. The resulting image looks disproportionate. For any type of lens, there are also optical distortions caused by tangential and radial lens imperfections, lens excursions, lens focal length, and light reduction near the outer portion of the lens. In the visual system, there is yet another type of image distortion caused by luminance variations and chromatic aberration. These distortions affect image quality and sharpness.

  Prior art panoramic vision systems do not remove image distortion while accurately mixing multiple images. One such panoramic system, that is, a rearview mirror vision system for a vehicle, is disclosed in Patent Document 1. This system comprises an image capture device, an image synthesizer, and a display system. There are electronic image processing systems that correct for geometric, optical, chromatic aberration, and luminance image distortion, both in this document and in references to it. In Patent Document 2, the vehicle vision system includes not only pre-calibration-based luminance correction, but also other optical corrections and geometric corrections. Full brightness and chrominance corrections must be adapted to the changing ambient environment based on the input and output light geometric parameters.

The distorted image and the breaks between multiple images slows down the operator's visual perception, thus her / his situational perception and creates a potential error. This is one of the most important obstacles to launching an efficient panoramic vision system. Thus, providing a panoramic vision system with an accurate representation of the view of the situation through removing geometric, optical, chromatic aberration, brightness and other image distortions and providing a composite image of multiple views desirable. Such correction supports visualization and recognition and improves visual image quality.
US Pat. No. 6,498,620B2 US Patent Application Publication No. 2003/0103141 A1 US Pat. No. 5,949,331 George Wolberg, "Digital Image Warping", IEEE Computer Society Press, IEEE Computer Society Press, 1988

  The present invention provides a panoramic vision system having an associated camera, display optics, and geometric characteristics.

The invention in one aspect provides a panoramic vision system having an associated camera, display optics, and geometric characteristics, the system comprising:
(A) a plurality of image acquisition devices for capturing image frame data from a scene and generating an image sensor input, the image frame data collectively covering a field of view up to 360 °, and geometric distortion A plurality of image acquisition devices having parameters and optical distortion parameters;
(B) a digitizer coupled to the plurality of image acquisition devices to sample and convert image frame data and image sensor input to digital image data;
(C) (i) an image measuring device for receiving digital image data and measuring an image brightness histogram and an ambient light source level associated with the digital image data;
(Ii) Receive digital image data along with camera, display optics and geometric characteristics, image illumination histogram and ambient light source level, correct brightness non-uniformities, and brightness of selected areas in the digital image data A brightness correction module coupled to the image measuring device to optimize the range;
(Iii) image sensor inputs, cameras, display optics and geometric distortion parameters and optical distortion parameters, including geometric properties and imperfections, associated therewith to form convolved distortion parameters; A convolution stage coupled to the brightness correction module for coupling;
(Iv) a distortion correction module that generates a distortion correction transform based on the convolved distortion parameter, applies it to the digital image data, and is coupled to the convolution stage to generate corrected digital image data;
(V) a display controller coupled to the distortion correction module to synthesize a composite image from the corrected digital image data;
An image processor coupled to a digitizer comprising:
(D) a display system coupled to the image processor for displaying a composite image on a display surface for display, the composite image being visually distorted;
Is provided.

In another aspect, the present invention includes not only the camera, display optics, and geometric distortion parameters as well as geometric characteristics, but also geometric characteristics, to generate composite images that cover up to 360 ° or 4π steradians. Providing a method for providing panoramic vision using a panoramic vision system comprising:
(A) obtaining image frame data from a scene, the image frame data collectively covering a field of view of up to 360 ° or 4π steradians and generating an image sensor input;
(B) converting image frame data and image sensor input into digital image data, the digital image data being associated with an image luminance histogram and an ambient light source level;
(C) Acquire camera, display optics and geometric characteristics, image brightness histogram and ambient light source level and correct brightness non-uniformities to optimize the brightness range of selected areas in the digital image data. And
(D) convolution of geometric distortion parameters and optical distortion parameters, including image sensor inputs, cameras, display optics, and geometric properties and imperfections associated therewith, to form a convolved distortion parameter; ,
(E) generating and applying a distortion correction transform to the digital image data, the distortion correction conversion being based on the geometric distortion parameters and the optical distortion parameters convolved to generate corrected digital image data; And
(F) compositing a composite image from the corrected digital image data;
(G) displaying the composite image on a display surface for display, and the previous composite image being visually distorted;
Is provided.
In another aspect, the present invention provides an image processor for use in a panoramic vision system that has not only geometric and optical distortion parameters, but also associated cameras, display optics, and geometric characteristics. The panoramic vision system captures image frames from the scene and generates digital image data and image sensor inputs, and converts the digital image data and image sensor inputs into digital image data. And using a digitizer, the image processor
(A) an image measuring device for receiving digital image data and for measuring a peripheral light source level associated with the image luminance histogram and the digital image data;
(B) to receive digital image data with camera, display optics and geometric properties, image brightness histogram, and ambient light source level, and to correct brightness non-uniformity, and selection among digital image data A brightness correction module coupled to the image measuring device to optimize the brightness range of the generated region;
(C) geometrical and optical distortion parameters, including geometrical characteristics and imperfections associated therewith to form the image sensor input, camera, display optics, and convolved distortion parameters; A convolution stage coupled to the brightness correction module for coupling;
(D) a distortion correction module coupled to a convolution stage that generates distortion correction transforms based on the convolved distortion parameters and applies to the digital image data to generate corrected digital image data;
(E) a display controller coupled to the distortion correction module to synthesize a composite image from the corrected digital image data;
Is provided.

  In its first embodiment, the present invention provides a vehicle vision system that enhances rear view, side view, and corner view and covers up to 360 ° horizontal view and up to 180 ° vertical view coverage. The status display image can be integrated with other vehicle information data such as vehicle status or navigation data. The image is preferably displayed on the front panel adjacent to the driver's field of view. Additional views can be displayed to improve coverage and flexibility, such as interior dashboard corner displays that replace the mirror and eliminate exposure to external elements and air resistance. The system is adapted to reconfigure the display and toggle between views based on control inputs when one view becomes more critical than the other.

  For example, the display shows a wide view of the rear lower part of the vehicle when the vehicle is backing, or a clockwise signal is turned on, and shows a right view when the vehicle is driving at higher speeds or a clockwise signal Reconfigure to show the right corner view when the vehicle is active or the vehicle is traveling at low speed. In a preferred embodiment, the system provides functions such as parking assistance or lane crossing alerts through not only distance determination but also through pattern recognition of curves, lanes and objects.

  In another embodiment, the present invention provides a surveillance system that covers up to 360 ° horizontal, ie, a 4π steradian view of the exterior and / or interior of structures such as buildings, bridges, and the like. The system provides a plurality of distortion corrected views and can provide a plurality of distortion corrected views to generate a continuous strip or scan scanning panoramic view. The system can perform motion detection and object tracking in the surveillance area, and to decode people, vehicles, etc.

  In yet another embodiment, the present invention provides a video conference vision system that produces a wide view up to 180 ° or a circular view up to 360 °. In this embodiment, the present invention facilitates displaying conference or gathering participants and provides the ability to zoom in on a speaker with a single camera.

  Further details of the different aspects and advantages of the embodiments of the invention will become apparent in the following description, taken in conjunction with the accompanying drawings.

  Constructed in accordance with the present invention, FIG. 1 shows the overall structure of the vision system 100. It has a camera 110 for capturing image frames, a digitizer 118 for converting image frame data into digital image data, adjusts brightness, corrects image distortion, and forms a composite image from digital image data and control parameters Multiple image acquisitions such as an image processor 200 for displaying, a controller 101 for relaying user parameters and sensor parameters to the image processor 200, and a display device 120 for displaying an image created for display on the display surface 160 Equipment. The final image produced by the present invention covers up to 360 ° including multiple specific regions of interest and is not significantly distorted. It considerably enhances situational awareness.

  The camera 110 shown in FIG. 1 includes an image optical component 112, a sensor array 114, and an image capture control mechanism 116. For applications that require large area coverage, the imaging optics 112 may use wide angle lenses to minimize the number of cameras required. Wide angle lenses also have a wider depth of focus, reducing the need for focus adjustment and its cost realization. Such lenses typically have a field of view in excess of 100 ° and a depth of focus from a few feet to infinity. Both of these characteristics make such a lens desirable for a panoramic vision system. Wide-angle lenses are difficult to optically correct and have even greater optical distortion that is expensive. In the present invention, electronic correction is used to compensate for optical and geometric distortions as well as other nonlinearities and imperfections in the optical component or projection path.

  In the present invention, the camera lens need not be complete or produce a complete image. All image distortions, including image distortions caused by lens geometry or imperfections, are mathematically represented in shape and corrected electronically. The importance of this feature lies in the fact that it makes it possible to use an inexpensive wide-angle lens with an increased depth of focus. The disadvantages of such lenses are the image distortion and brightness non-uniformity they cause. They tend to stretch the subject especially near the edge of the scene. This effect is well known for fisheye lenses or annular lenses, for example. They make the subject appear unbalanced. There are bulk optic solutions that can be used to compensate for these distortions. However, with the addition of bulk optics, such devices will be larger, heavier and more expensive. The cost of bulk optical component components is fixed by labor costs for functions such as polishing and alignment. However, electronic components are constantly becoming cheaper and more efficient. It is also well known that bulk optics can never completely remove a specific distortion.

  Another drawback of bulk optic solutions is sensitivity to adjustment and failure on impact. Therefore, there is a significant need for electronic image correction that makes the visual system robust and relatively economical while allowing it to represent true settings. This is evidenced by the fact that in the visual system, electronic components are likely to be needed anyway to accomplish other tasks. These tasks include panorama conversion and display control. The image processor 200 described later in this section provides these essential functions.

  Image sensor array 114 is depicted in camera 110. The sensor array is adapted to moderate extreme variations in scene light intensity through the camera system. In order for a visual system to operate efficiently in different lighting conditions, there is a need for a large dynamic range. This dynamic range requirement stems from daytime and nighttime variations in ambient light intensity, as well as wide variations from the incident light source at nighttime. The camera sensor is basically a mosaic of segments each exposed to light from a part of the scene, and registers the light intensity as the output voltage. The image capture control mechanism 116 transmits image frame luminance information, and based on these data, it receives commands from the image processor 200 for integrated optimization, lens aperture control, and white balance control.

  The digitizer 118 receives image data from the image capture control mechanism 116 and converts these data into digital image data. In the alternative, the functionality of the digitizer 118 may be integrated with the image sensor 114 or the image processor 200. In the case of existing audio data acquisition, the digitizer 118 receives these data from the audio device array 117. It then generates digital audio data and combines these data with the digital image data. The digital image data is then transmitted to the image processor 200.

  FIG. 2A shows the image processor 200 in detail. The image processor function receives digital image data, measures image statistics, improves image quality, compensates for brightness non-uniformities, corrects for various distortions in these data, and ultimately results in significant distortion Generating one or more composite images without It includes not only the control data interface 202, the image measurement module 210, the brightness correction module 202, the distortion convolution stage 250, the distortion correction module 260 and the display controller 280, but also camera optical component data, projection optical component data, and projection geometric shape data. An optical component geometry data interface 236 is also provided. In order to understand the function of the image processor 200, it is important to briefly describe the nature and cause of these distortions.

  Ambient light and spot illumination can cause extreme fluctuations in luminance levels, resulting in poor image quality and difficulty in image element recognition. In the present invention, the image processor 200 measures the complete image area and the selected image area and analyzes them to control the exposure for the best image quality in the area of interest. High light source sources such as headlights and spotlights are substantially reduced in luminance, and low level objects are improved in detail to support element recognition.

  The image sensor may be of any type and resolution, but for applications specified in the present invention, a high resolution solid state CCD or CMOS image sensor will be appropriate in terms of size, cost, integration, and flexibility. I will. Typical applications include adjusting to ambient light levels that vary widely in aperture control and integration time. In addition to aperture control and integrated time control, the brightness correction module 220 performs histogram analysis of the region of interest, expands the contrast in the low light region, and achieves enhanced light element recognition to achieve enhanced image element recognition. Reduce contrast.

  Brightness non-uniformity is an undesirable brightness variation across the image. The main physical reason for brightness non-uniformity is the fact that rays passing through various parts of the optical system travel different distances and area densities. The brightness of a light beam decreases with the distance it travels. In addition to this purely physical cause, the imperfection of the optical system also causes brightness non-uniformities. Examples of such imperfections are projection lens vignettes and lateral variations, and non-uniformities in the projection light produced. If the brightness variations for three different color components are different, they are called chrominance non-uniformities.

  Another type of optical distortion is chromatic aberration. It is due to the fact that optical components such as lenses have different refractive indices for different wavelengths. Light rays propagating through the optical component are refracted at different angles at different wavelengths. This results in a side shift of the color in the image. Different color components of the point object are separated and branched by the aberration in the lateral direction. On the display surface, the point will appear to have an edge. Another type of chromatic aberration is inherently along the axis and is caused by the fact that the lens has different focal points at different light wavelengths. This type of chromatic aberration may not be corrected electronically.

  Various other distortions may exist in the vision system. Tangential or radial lens imperfections, lens excursions, projector imperfections, and keystone distortion from off-axis projections are some of the common distortions.

  In addition to the noted distortion, the image processor 200 maps the captured scene to a display surface with certain characteristics such as shape, size, and aspect ratio. For example, an image is formed from various parts of a captured scene and projected onto a part of a car windshield, in which case it is not only for a specific size of the display surface but also for a shape that is not flat Suffer from distortion. The image processor in the present invention corrects all these distortions as described below.

  Referring now to the details of the image processor 200 of FIG. 2A, digital image data is received by an image measurement module 210 where a histogram of image contrast and brightness within the region of interest is measured. These histograms are analyzed by the brightness correction module 220 to control sensor exposure and to adjust digital image data and improve image content for visualization and detection. Some adjustments include highlight compression, contrast enhancement, detail enhancement and noise reduction.

  Along with this measurement information, the brightness correction module 220 receives camera and projection optics data from the optics geometry data interface 236. These data are determined from accurate light propagation calculations and optical component calibration. These data are critical for brightness correction because they provide the optical path distances of different rays that pass through the camera as well as the projection system. Separate or combined cameras and projection correction maps are generated to calculate the correction for each pixel.

  The correction map can be obtained offline or it can be calculated dynamically according to the environment. Therefore, the function of the brightness correction module 220 is to receive digital image data and generate digital image data with adjusted brightness. In the case of chrominance non-uniformity, the luminance correction module 220 preferably applies the luminance correction to three different color components separately. The physical implementation of the brightness correction module 220 may be by software processing or a digital signal processor, or dedicated processing circuitry such as computational logic in an integrated circuit.

  The brightness-adjusted image data needs to be corrected for geometrical distortions, optical distortions, and other spatial distortions. These distortions are referred to as “warp” correction, and such correction techniques are referred to in the literature as “image warping”. A description of image warping is published in Non-Patent Document 1, which is incorporated herein by reference.

  Basically, image warping is an efficient parameterization of coordinate transformation, mapping output pixels to input pixels. Ideally, the grid data set represents the mapping of every output pixel to the input pixel. However, the grid data representation is very strict in terms of hardware implementation due to the shear size of the lookup table. Image warping in the present invention provides an efficient way to represent a pixel grid data set through several parameters. In one example, this parameterization is performed by a polynomial of degree n, where n is determined by the complexity of the combined distortion.

  Various regions of the output space of another example of the present invention are divided into patches with inherent geometric properties to reduce the frequency of the polynomial. As a general rule, the higher the number of patches and the higher the frequency of the fitting polynomial per patch, the more accurate the parameterization of the grid data set. However, this must be balanced with the processing power for real-time applications. Thus, such a warp map represents the mapping of output pixels to input pixels and represents the display geometry, including the camera optic, display optic, and final composite image designation properties and display surface shape. Furthermore, any control parameters including user input parameters are combined with the parameters and expressed in a single transformation.

  In addition to coordinate transformation, sampling or filtering functions are often required. Once the output image pixel is mapped onto the input pixel, the area around this input pixel is designated for filtering. This region is called the filter installation area. Filtering is essentially a weighted averaging function that produces the intensity of the constituent colors of the output pixel based on all pixels within the footprint. In a particular example, an anisotropic elliptical footprint is used for optical image quality. It is well known that the larger the size of the installation area, the higher the quality of the output image. The image processing 200 in the present invention performs image filtering together with simultaneous coordinate conversion.

  To correct for image distortion, all the geometric distortion parameters and optical distortion parameters described above are concatenated in the distortion convolution stage 250 for different color components. These parameters include camera optical component data, projection optical component data, and projection geometric shape data via optical component geometry interface 236 and control input via control interface 202. The concatenated optical distortion parameters and geometric distortion parameters are then obtained by the distortion correction module 260. The function of this module is to convert the luminance of the position, shape and color of each element of the scene into display pixels. The shape of the display surface 160 is taken into account in the projection geometry data 234. This surface is not necessarily flat and may be any general shape as long as a surface map is obtained and coupled with other distortion parameters. The display surface map is convolved with the rest of the distortion data.

  In one example of the present invention, the distortion correction module 260 obtains a warp map that covers the entire space of distortion parameters. The process is described in detail in co-pending US Patent Application Nos. 2003 / 0020732-A1 and 2003 / 0043303-A1, which are incorporated herein by reference. For each set of distortion parameters, a transform is calculated to compensate for the distortion experienced when the image propagates through the camera optics and through the display optics onto a particular shape of the display surface 160. The creation of the distortion parameter set and transformation calculation may be performed off-line and stored in memory for access by the image processor 200 via the interface. They could also be implemented at least partly dynamically in the case of changing parameters.

  Display images can consist of view windows that can be independent views or linked views. For each view window, the distortion correction module 260 interpolates and calculates spatial transformation and filtering parameters from the warped surface equation and performs image transformation for the display image.

  For each image frame, the distortion correction module 260 first finds the closest grid data point in the distortion parameter space. It then interpolates the existing transform corresponding to that set of parameters and matches the actual distortion parameters. The correction module 260 then applies the transformation to the digital image data to compensate for any distortion. The digital image data from each frame of each camera is combined to form a composite image that conforms to the display surface and its underlying structure. The corresponding digital data is corrected so that when the image is formed on the display surface, it is clearly undistorted and fits an optimized display area on the display surface. The physical implementation of the distortion correction module 260 may be by a software program on a general purpose digital signal processor or by dedicated processing circuitry such as an application specific integrated circuit. The actual example of the image processor 200 is incorporated into a Silicon Optics Inc. sxW1 and REON chip.

  FIG. 2B shows the flow logic of the image processor 200 in an example of the invention. The digital data flow of this chart is shown by bold lines, whereas the calculated data flow is drawn through thin lines. As can be seen in this figure, a brightness histogram and a contrast histogram are measured from the digital data in step (10). Camera optics along with display optics and display geometry parameters are obtained in steps (14) and (16). These data are then acquired along with the brightness histogram and contrast histogram in step (20) where the image brightness non-uniformity is adjusted. Not only the control parameters obtained in step (26) but also the optical components and geometric data from steps (14) and (16) are then collected in step (30). In this step, all distortion parameters are concatenated. A transform that reverses the effects of distortion is then calculated in step (40). This compensation transformation is then applied to the brightness adjusted digital image data obtained from step (20).

  As explained above, the pixel data is filtered in this step for higher image quality. Therefore, step (50) constitutes simultaneous coordinate transformation and image processing. In step (50), the distortion compensated digital image data is used to generate a composite image for display.

  Once the digital image data for the frame is processed and created, the display controller 280 generates an image from these data. In the present example, the image is formed on a display device 120, which may be a direct view display or a projection display device. In one example, the image from projection display device 120 is projected onto display surface 160 through display engineering component 122. The display optical component 122 is a bulk optical component component for directing the optical display device 120 projected onto the display surface 160. Certain display systems have additional optical and geometric distortions that need to be corrected. In this example, image processor 200 concatenates these distortions and corrects the combined distortion.

  It should be noted that the present invention is not limited to a particular choice of display system. The display device could be provided as a liquid crystal, light emitting diode, cathode ray tube, electroluminescent, plasma, or any other viable display option. The display device could be displayed directly, or it could be projected through the display optics onto an integrated compartment and act as a display screen. Preferably, the brightness of the display system is adjusted via an ambient light sensor, an independent signal and a user dimmer switch. The size of the display surface is also preferably adjustable by the user according to the distance from the operator's eyes.

  The controller 101 is connected to the image processor 200. It obtains user parameters from the user interface 150 along with inputs from the sensor 194 and sends them to the image processor 200 for display control. User parameters are specific to applications corresponding to various embodiments of the present invention. Sensor 194 preferably includes an ambient light sensor and a direct glare sensor. Data from these sensors is used for display adjustment. Control parameters are convolved with other parameters to provide the desired image.

  For some applications of the present invention, it may be desirable to record a data stream or send it to different clients over a network. In both cases, it is desirable and necessary to compress the video data stream first to achieve storage and bandwidth limitations. The compression stage 132, in one example, receives a video stream from the image processor 200 and compresses the digital video data. The compressed data is stored in storage device 142 for future use. In another embodiment, the data is encoded at the encryption stage 134 and transmitted over the network via the network interface 144.

  Although the general features of the present invention have been described in detail, many example implementations of the vision system 100 will be described individually. In one embodiment, the present invention provides a vehicle vision system. FIG. 3A shows a conventional self-propelled vehicle 900 with two side mirrors and one in-vehicle mirror covering a right view 902, a left view 904, and a rear view 906. It is well known that blind spots can occur due to conventional mirror position and field of view and can have distortion due to wide vision. It is difficult to get full situational awareness from the images provided by the three mirrors.

  In the example of the invention illustrated in FIG. 3B, the vehicle 900 'is equipped with a camera 110' and a camera 111 'in front of the driver's seat. Coverage is expanded by overlapping these positions with the driver's direct forward view. The camera 113 'in this example is located at the rear of the vehicle or at the middle rear of the roof. Specific regions of different views are selected to provide a continuous image that does not overlap. This prevents driver confusion.

  In another example illustrated in FIG. 3C, the vehicle 900 "has a camera 110" and a camera 111 "at the front corner of the vehicle, and a camera 113" at the center rear. In this example, the emphasis is on views that can be adjusted by user input or by control parameters. For example, the turning side view 903 "is made available when the turning signal is active. The coverage of this view is a function of user input, and in principle covers the entire area between the dashed lines. It should be noted that the driving side view 905 ″ is also used in the standard operating mode, which can be expanded to cover the entire area between the dashed lines. The rear view of this example has two modes depending on the control parameters. When the vehicle is reversed, the reverse rear view 907 "is used for display. This view causes complete coverage of the rear of the vehicle including objects on the paved road. Guarantees and greatly facilitates parallel parking, but a narrower driver rear view 906 "is used when the vehicle is in driving mode. These positions and angles ensure a convenient view of the exterior of the vehicle by the driver. This example greatly facilitates functions such as parallel parking and lane change.

  FIG. 4A shows an example of the display surface 160 as seen by the driver. A right view 162 and a left view 164 are at the upper right and upper left of the display surface 160, respectively, while a rear view 166 is at the bottom of the display surface. FIG. 5A shows an example of a display surface 160 'where a wider display is used and the side display is on two sides of the rear view 166'. Image processor 200 has panorama conversion and image stitching functions to create this particular display. 4B and 5B show examples of the reconstructed display of FIGS. 4A and 5A when the vehicle turns to the right or is reversed, respectively.

  FIG. 6A is a diagram of an example of a reconstructed display of the display surface 160 when the vehicle changes lanes and enters the right lane. In this example, the right front and rear of the vehicle are fully visible and cause situational recognition for all of the right side of the vehicle. FIG. 6B is an illustration of an example display configured to turn right when a right turn signal is active. The side change view 903 "of FIG. 3C is now displayed as the side change view 163" in the upper middle part of the display. The rear view 166 "and the right view 166" are at the bottom of the display.

  FIG. 7 shows another example of the display surface 160. This particular configuration of the display can be achieved by collecting visual information from the image acquisition device, as well as distance determination from the ultrasonic and radar sensors integrated with the vision system 100. In this illustrated example, a bold and gray shaded vehicle is provided with a vision system 100 and is shown with respect to ambient settings, ie other vehicles. This example implementation of the present invention greatly enhances situational awareness and facilitates vehicle driving. Two of the dimensions of each object are captured directly by the camera. The shape and size of these vehicles and objects are reconstructed by extrapolating the camera view according to the size and shape of the lanes and curves. A more accurate visual account of the driving scene could be reconstructed with pattern recognition and look-up tables for specific objects in the database.

  It should be noted that the present invention is not limited to these illustrated examples. Variations in the number and position of cameras as well as reconstruction of the display surface are within the scope of the present invention. Projection geometry data, projection optics data, and camera optics data cover all these alternative implementations of image processor 200.

  Preferably, the display brightness is adjustable via an ambient light sensor, via a signal from a vehicle headlight or via a manual dimmer switch. In addition, the size of the display surface is preferably adjustable via a driver. It is also preferred that the focal length of the displayed image is substantially within the field of focus of the driver, as described in US Pat. However, it is also preferred that the focal length of the display system is adjusted according to the speed of the vehicle in order to ensure that images are always formed within the driver's depth of focus. At higher speeds, the driver naturally focuses on longer distances. Such adjustment is accomplished via speedometer signal 156 or transmission signal 154.

  FIG. 6 shows an example of user input 150 in the vehicle vision system. The signal interface 151 receives signals from different components and connects to the controller 101. For example, a turn signal 152 that is active while turning to the right relays the signal to the signal interface 151 and onto the controller 101 and ultimately to the image processor 200. Once the image processor 200 receives this signal, it configures the display to highlight the right display 162 on the display surface 160. FIG. 4B shows the display screen 160 under such conditions. The right view display occupies half of the display surface 160 and the other half is dedicated to the rear view display 166. FIG. 5B shows the same situation with a wide display surface embodiment. Other signals call other functions.

  For example, while parking the vehicle, the transmission preferably generates a signal to enhance the rear view. The other signals are a steering signal that is active when the steering wheel is turned to one side more than a preset limit, and a brake signal that is active when the brake pedal is pressed. It includes a transmission signal that conveys information about the moving speed and direction, and a speedometer signal that is measured at various set speeds to input various signals. The display system adjusts to different situations depending on any of these control parameters and automatically reconfigures the critical image to be usable. The various control signals are incorporated with the image processor 200, here only a few examples are mentioned.

  A variety of useful information could be displayed on the display surface 160, including road maps, roadside information, GPS information, and local weather forecasts. The vision system 100 accesses these data either through downloaded files or through wireless communication at the driver's request. It then superimposes these data on the image frame data to create an image for the display system. Important alert systems are also preferably received and displayed by the visual system and broadcast via the display system interface and the audio signal interface. The vision system 100 could also be integrated with an intelligent highway drive system via exchanging image data information. It is also integrated with the distance limiter and the object identifier according to US Pat.

  In one example implementation, the vehicle vision system 100 includes a compression stage 132 and a recording device 142 for continuous recording of events as detected by a camera and audio device. These features incorporate the ability to save specific segments prior to past impacts in addition to events intended by the driver to be reviewed after unfortunate events such as accidents, car takeovers and traffic violations. Such information could be used by law enforcement agencies, judicial authorities, and insurance companies.

  In another embodiment, the present invention provides a video conference vision system that covers up to 180 ° or 360 °, depending on the default settings. In this embodiment, an audio signal is required along with a video signal. In the illustrated example of FIG. 1 of vision system 100, audio device array 117 provides audio input in addition to video input from a camera. The audio signal data is converted into digital data via the digitizer 118 and is superimposed on the digital image frame data. Pan functionality is provided based on triangulation of audio signals to pan to individual speakers or interrogators. The pan and zoom functions are provided digitally by the image processor 200. In the absence of optical zoom, digital zoom is provided.

  FIG. 7A relates to an example of a video conference system. It shows the view of the camera 110 at the center of the conference table. In this particular example, the camera 110 has a hemispherical lens system and captures everything on and including the table surface. The raw image is stretched upside down and unbalanced. However, the image displayed on the display surface 160 is not distorted and is panorama converted.

  FIG. 7B shows a display surface 160 where the speaker image is magnified and the rest of the conference is visible in a straight line on the same surface.

  In yet another embodiment, the present invention provides a surveillance system with motion detection and object recognition. FIG. 2A shows an example of an image processor 200 used in a surveillance system. The motion detector 270 evaluates successive input image frames based on the detected motion and preset levels of signal alerts and sends motion region coordinates to the distortion convolution stage 250. The input image frame is used to monitor an area outside the current view window. The tracking coordinates are used by the distortion convolution stage 250 to calculate a view window for the corrected display of the detected object. Distorted object images can be enhanced in resolution by guaranteed motion of objects and temporal interpolation. Object recognition and classification can also be performed.

  FIG. 8 shows an illustrative example of a surveillance system. Cameras 110 and 111 are in the passage where they are attached to the wall. These two cameras monitor the traffic through the passage. The image processor 200 in this embodiment uses a motion detector and tracking device 270 to track the motion of the subject as the subject passes through the passage. The display screen 160 of this example is shown in FIG. 9A at a specific time and later in FIG. 9B. The passage of the subject is fully captured as he moves from the view of one camera to the other. The vision system can also perform resolution enhancement by temporal extraction to improve subject detail and recognition, displaying it at the top of both FIGS. 9A and 9B. The top of the figure is provided at full resolution or by digitally zooming on a moving subject. Object recognition and tracking is accomplished through comparing a moving object from one frame to the next frame. The outline highlights the tracked object to facilitate recognition.

1 represents a general view of a vision system and its components constructed in accordance with the present invention. 1 represents the structure of the image processor of FIG. 1 as part of a vision system. 2 represents the flow logic of the image processor of FIG. 1 represents a conventional vehicle with mirrors used for side and rear views. The setting value of the example of a camera and its view is represented in a vehicle example of a visual system. FIG. 4 represents a selected view for a camera in a vehicle example of a vision system. Fig. 4 represents a visual surface of a visual system display in an example vehicle vision system. FIG. 4B shows the same display surface as in FIG. 4A where the right turn signal is active in the example vehicle vision system. The vehicle vision system example represents a broad alternative of the display surface of the vision system display. 5B represents the reconstructed display surface of FIG. 5A when the vehicle is running in reverse. Represents a reconstructed display when the vehicle is changing lanes. Represents a reconstructed display when the vehicle turns to the right and the driver's view disappears. Represents an upper view display showing objects adjacent to a vehicle. Fig. 3 represents an example of a control input in a vehicle vision system. FIG. 3 represents a hemispherical 180 ° (2π steradian) view camera set at the center of a conference table in the example video conference vision system. Fig. 4 represents a display of an example video conference vision system. Represents a passage or building wall with two cameras mounted on a wall as part of an example surveillance system. 1 represents a display of an example surveillance system of the present invention. It represents the same display as FIG. 9A at a later time.

Claims (56)

A panoramic vision system having an associated camera, display optics and geometric properties,
(A) a plurality of image acquisition devices for capturing image frame data from a scene and generating image sensor input, the image frame data collectively covering a field of view of up to 360 °, The image acquisition device having a dynamic distortion parameter and an optical distortion parameter;
(B) a digitizer coupled to the plurality of image acquisition devices for sampling and converting the image frame data and the image sensor input into digital image data;
(C) (i) an image measuring device for receiving the digital image data and receiving the image luminance histogram and the ambient light source level associated with the digital image data;
(Ii) receiving the digital image data along with the camera, display optics and geometric characteristics, the image luminance histogram, and the ambient light source level, and correcting luminance inhomogeneities, and within the digital image data A brightness correction module coupled to the image measuring device to optimize the brightness range of selected areas of
(Iii) the geometry including the image sensor input, the camera, display optics, and geometric properties and imperfections associated therein to form a convolved distortion parameter; A convolution stage coupled to the brightness correction module to combine a distortion parameter and an optical distortion parameter;
(Iv) a distortion correction module coupled to the convolution stage to generate a distortion correction transform, apply to the digital image data based on the convolved distortion parameter, and generate corrected digital image data; ,
(V) a display controller coupled to the distortion correction module to synthesize a composite image from the corrected digital image data;
(D) a display system coupled to the image processor for displaying the composite image on a display surface for display, wherein the composite image is free of visual distortion;
An image processor coupled to the digitizer comprising:
A system comprising:   The image processor of claim 1, wherein the image processor is adapted to separately process a red component, a green component, and a blue component associated with the digital image data to correct lateral chromatic aberration and distortion. Panoramic vision system.   The panoramic vision system of claim 2, wherein the brightness correction module processes the red component, the green component, and the blue component separately to correct chrominance non-uniformity.   The panoramic vision system according to claim 1, wherein the plurality of image acquisition devices and the digitizer are realized by a plurality of digital cameras.   The panorama of claim 3, wherein the display system is a light projection system, and the geometric distortion parameters and optical distortion parameters are coupled to precorrect the associated geometric distortion and optical distortion. Visual system.   At least one ambient light sensor, at least one illumination signal, and at least one user dimmer switch, wherein the brightness of the display system includes the ambient light sensor, a separate illumination signal, and a user dimmer switch. 4. The panoramic vision system of claim 3, wherein the panoramic vision system is adjusted according to at least one of the following.   A controller further comprising: a controller adapted to relay the set of user parameters to the panoramic vision system to link the user parameters to a distortion parameter based on the display function and priority set selected. Item 4. The panoramic vision system according to item 3.   The panoramic vision system of claim 3, further comprising a control interface coupled to the image processor adapted to receive and relay user input and control parameters.   4. The panoramic vision system of claim 3, wherein the image processor is further adapted to send an exposure control command to the image acquisition device for aperture and exposure time control.   4. The panoramic vision system of claim 3, further comprising a display surface, wherein the position and size of the composite image on the display surface is adjustable according to a set of user inputs.   4. The panoramic vision system according to claim 3, provided as a vehicle vision system.   The display system is adapted to form a composite image in a forward view of a driver located within the vehicle, i.e., on at least one of the group consisting of a dashboard, windshield, roof and fascia. The vehicle vision system according to claim 11.   13. The vehicle of claim 12, wherein the display system is adapted to form a composite image on two display surfaces on the interior corners of the windshield and fascia to mimic a vehicle side mirror. Visual system.   The vehicle vision system of claim 11, wherein the focus of the composite image falls within the forward focus field of view associated with a driver located within the vehicle.   The vehicle vision system of claim 11, wherein the focus of the display system is substantially infinite.   12. The vehicle vision system of claim 11, wherein the vehicle vision system is reconfigured in different configurations according to inputs from a group consisting of a turn signal, a transmission system, a brake system, a steering system, and a speedometer.   The display system improves the right view from the vehicle when the vehicle turn signal is set to turn right and the left view when the vehicle turn signal is set to turn left The vehicle vision system of claim 16, adapted to improve the vehicle.   The display system improves the right view from the vehicle when the vehicle steering wheel is turned to the right by a preset angle, and the vehicle steering wheel is turned to the left by a preset angle. 17. The vehicle vision system of claim 16 adapted to improve the left view from the vehicle when operated.   17. The vehicle vision system of claim 16, wherein the display system is adapted to improve the rear view from the vehicle while the vehicle is operating in reverse.   17. The vision system is adapted to improve at least one of the front corner side view and the rear corner side view from the vehicle when a direct view is obstructed by an adjacent object. Vehicle vision system as described in.   At least one of a laser and a radar sensor, wherein the image processor uses the sensor to detect a distance between the exterior of the vehicle and the object, and between the exterior of the vehicle and the object 21. The vehicle vision system of claim 20, further adapted to provide an audio alert and a visual alert when the distance is within a range defined by the speed of the vehicle.   The display system is configured to capture the 2D through the camera and show the vehicle and its surrounding environment as seen from above by extrapolating in 3D according to lane and curve patterns and object lookup database The vehicle vision system of claim 21, wherein:   The image processor is further adapted to detect at least one of road, lane, and curve edge markings and provide audio and visual warnings when the exterior of the vehicle approaches the edge markings 21. The vehicle vision system of claim 20, wherein:   24. The vehicle of claim 23, wherein the image processor is further adapted to provide a direction indicator on the display system to detect the position of the designated area and guide movement of the vehicle to the designated area. Visual system.   At least one of the group consisting of an ambient light sensor, an approaching headlight sensor, and a driver dimmer switch, the brightness and contrast associated with the display system being the ambient light sensor, the approaching head 12. The vehicle vision system of claim 11, wherein the vehicle vision system is adjusted according to at least one reading of a light sensor and a driver dimmer switch.   Further comprising a safety black box device and a trigger device for recording all digital image data associated with a period defined around the trigger activation, wherein the trigger comprises a group of users, collisions, hijacking, and theft 12. The vehicle vision system of claim 11 adapted to be activated by at least one of the following.   The vehicle vision system of claim 11, further comprising an integrated GPS navigation device that allows roadmaps and roadside information to be displayed on the display system.   4. The panoramic vision system according to claim 3, comprising a plurality of surveillance sensors, wherein the system is provided as a surveillance vision system.   30. The surveillance vision system of claim 28, wherein the image processor is further adapted to perform motion detection and object tracking studies using the surveillance sensor.   30. The surveillance vision system of claim 28, wherein the image processor is adapted to provide a set of visual and audio signals when motion is detected by a surveillance sensor.   The image processor is further adapted to define a specified set of protected areas, track a plurality of objects, and generate audio and visual alerts when such objects enter the specified protected area; 27. The surveillance vision system of claim 26, wherein the display system is reconfigured to highlight such objects when such objects enter the designated protected area.   4. The panoramic visual system of claim 3, further comprising a set of audio sensors, wherein the digitizer is further adapted to convert the audio sensor input into digital audio data, and the system is provided as a video conference visual system.   The image processor and the display system are further adapted to generate and display a zoomed image of the video conference user according to the intensity and direction of the audio signal generated by the video conference user. The videoconferencing vision system described. Panorama vision using a camera, display optics and a panoramic vision system with geometric characteristics as well as geometric distortion parameters and optical distortion parameters to generate composite images covering up to 360 ° or 4π steradians A method for providing
(A) obtaining image frame data from a scene, the image frame data collectively covering a field of view of up to 360 ° or 4π steradians and generating a set of image sensor inputs;
(B) converting the image frame data and the image sensor input into digital image data, the digital image data being associated with an image luminance histogram and an ambient light source level;
(C) obtaining the camera, display optics, and geometric properties, the image luminance histogram and the ambient light source level to optimize the luminance range of a selected region in the digital image data; Correcting the non-uniformity of
(D) the geometric distortion parameters and optical including the geometric characteristics and imperfections associated therein to form the image sensor input, the camera, display optics, and convolved distortion parameters; Convolving the distortion parameters;
(E) generating and applying the distortion correction transform to the digital image data, the convolved geometric and optical distortion parameters to generate digital image data in which the distortion correction conversion is corrected; Based on
(F) compositing a composite image from the corrected digital image data;
(G) displaying the composite image on a display surface for display, the composite image being visually distorted;
Said method comprising.   35. The method of claim 34, further comprising processing the red component, the green component, and the blue component of the digital image data separately to correct lateral chromatic aberration distortion.   36. The method of claim 35, wherein brightness adjustment is performed separately for each of the red, green and blue components to correct for chrominance non-uniformity. The correction conversion is
(I) obtaining a grid data set covering the entire space of the geometric and optical distortion parameters;
(Ii) calculating a correction transform corresponding to each data set on the grid of (i);
(Iii) interpolating adjacent grid transforms to conform to the actual image frame transform parameters;
40. The method of claim 36, achieved by:   38. The method of claim 37, wherein (i) and (ii) are performed once offline and the transform grid is accessed for each image frame correction.   38. The method of claim 37, wherein the method is performed dynamically under conditions in which (i) and (ii) vary.   38. The method of claim 37, wherein the pixel map associated with the transform is represented by a warp map.   41. The method of claim 40, wherein the warp map is parameterized with respect to one of an incremental polynomial and a position polynomial.   37. The method of claim 36, further generating an exposure control command and relaying it to a plurality of image acquisition devices.   37. The method of claim 36, further comprising receiving control parameters and user parameters for convolution with distortion parameters.   40. The method of claim 36, further comprising motion detection and tracking.   44. The method of claim 43, further comprising adjusting the brightness, contrast, and size of the composite image in accordance with a sensor and user input.   44. The method of claim 43, further reconfiguring the display system according to a set of control inputs.   44. The method of claim 43, further adjusting the focal length of the composite image according to a set of control parameters. An image processor for use with an associated camera, display optics, and a panoramic vision system having geometric characteristics as well as geometric distortion parameters and optical distortion parameters, wherein the panoramic vision system images from a scene Using a plurality of image acquisition devices for capturing frames and generating digital image data and image sensor input, and a digitizer for converting the digital image data and the image sensor input into digital image data;
(A) an image measuring device for receiving the digital image data and for measuring the ambient light source level associated with the image luminance histogram and the digital image data;
(B) receiving the digital image data with the camera, display optics and geometric characteristics, image luminance histogram, and ambient light source level, correcting luminance non-uniformity, and in the digital image data A brightness correction module coupled to the image measuring device to optimize the brightness range of a selected region;
(C) the geometric distortion parameters and optical including the geometric characteristics and imperfections associated therein to form the image sensor input, the camera, the display optics, and convolved distortion parameters; A convolution stage coupled to the brightness correction module to combine distortion parameters;
(D) generating a distortion correction transform to generate corrected digital image data, and applying the distortion correction transform to the digital image data based on the convolved distortion parameter; A combined distortion correction module;
(E) a display controller coupled to the distortion correction module to synthesize a composite image from the corrected digital image data;
An image processor.   49. The image processor of claim 48, further adapted to separately process a red component, a green component, and a blue component of the digital image data to correct lateral chromatic aberration distortion.   50. The image processor of claim 49, wherein the brightness correction module processes a red component, a green component, and a blue component of the digital image data separately to correct chrominance non-uniformity.   51. The image processor of claim 50, further adapted to generate an exposure control command and relay to an image acquisition device.   51. The image processor of claim 50, further adapted to receive control parameters and user parameters for convolution with distortion parameters.   51. The image processor of claim 50, further adapted for a motion detection component and a tracking component.   51. The image processor of claim 50, further adapted to adjust the brightness, contrast, and size of the synthesized image according to a set of sensors and user input.   53. The image processor of claim 52, further adapted to reconfigure the display system according to a set of control inputs.   53. The image processor of claim 52, further adapted to adjust the focal length of the composite image according to a set of control parameters.

Images