JP2009267924A - Imaging system and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging system which processes an image, photographed by a camera, by a central controller, the total delay time being prevented from increasing while a time necessary for the image processing is secured. <P>SOLUTION: The camera 10 includes a video processing circuit 13 which generates image feature information to be used for the image processing from the photographed image, and a transmission processing circuit 14 which transmits image information 21 including the image feature information to the central controller 30. The central controller 30 includes an image processing circuit 33 which performs image processing for extracting and recognizing a specific subject using the transmitted image feature information. The transmission processing circuit 14 when transmitting the image information to the central controller transmits the image feature information to be used for the image processing precedently in the video frame. The image feature information is preferably a reduced image of the photographed image or a luminance signal of the photographed image. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an imaging system and an image processing method for performing image processing such as extraction and recognition of a specific subject from an image signal captured by a camera.

  An imaging system that provides various driving assistance functions has been put into practical use, such as installing a camera on an automobile body, monitoring the surroundings of the vehicle, and displaying a driver's blind spot image on the monitor. Furthermore, an imaging system having a function of not only displaying a captured image on a monitor but also extracting a specific subject from the captured image using an image processing technique is possible. As a specific subject, for example, a lane white line (lane) on a road can be detected from luminance information and contour information in a captured image, or a person can be detected from an infrared image by a pattern matching method or the like. An imaging system that implements such a function has a configuration in which one or a plurality of cameras are mounted on an automobile body, an image signal from the cameras is transmitted to a central control device, processed, and an extracted subject image is displayed. It has become.

  In an in-vehicle imaging system, it is necessary to instantly display an image captured by a camera on a monitor, and it is important to reduce the delay time from image formation on the image sensor of the camera to image display on the monitor. It is. A general imaging device such as a CCD or MOS scans pixel signals on a light receiving surface and sequentially outputs image signals. Therefore, in order to perform transmission from the camera to the central control device with the minimum delay time, it is most efficient to transmit and display the pixel signals in the scanning order of the image sensor. Further, various methods have been proposed for a method of synchronizing the camera and the central control device and an efficient method of transmitting an image signal. For example, Patent Document 1 discloses an in-vehicle camera system that switches the output format of a captured image in accordance with the number of cameras and the running situation. Patent Document 2 discloses a video signal processing apparatus that starts address conversion processing in units of fields when non-synchronous videos captured by a plurality of cameras are arranged in one frame of video.

JP 2004-216984 A JP 2006-180340 A

  According to the techniques described in Patent Document 1 and Patent Document 2, it is possible to efficiently transmit an image signal captured by a camera to a central control device. However, when image processing such as white line (lane) detection and person extraction described above is performed in the central control device, processing time increases because complicated arithmetic processing is required. As a result, the delay time until the processing result image is displayed becomes a factor. As advanced functions are added for image processing, the delay time tends to increase, and there has been a conflicting demand between the enhancement of image processing functionality and the reduction of display delay time. In order to solve this, it is necessary to prevent an increase in the delay time in consideration of the entire processing flow from imaging with the camera to transmission and image processing with the central control unit. In the prior art, sufficient consideration has not been made in this respect.

  An object of the present invention is to provide an imaging system capable of preventing an increase in delay time of the entire imaging system while securing a time required for image processing in a central control device.

  The present invention is an imaging system that transmits an image captured by a camera to a central control device, performs image processing for extraction and recognition of a specific subject, and displays the image. A video processing circuit for generating image feature information used for processing, and a transmission processing circuit for transmitting image information including the image feature information to the central control unit. The central control unit uses the image feature information transmitted from the camera. An image processing circuit that performs image processing for extracting and recognizing a specific subject, and a display processing unit that synthesizes the processed images and displays them on the image display unit.

  Here, when transmitting image information to the central control unit, the transmission processing circuit transmits image feature information used for image processing in advance in the video frame. Preferably, the image feature information is a reduced image of the captured image or a luminance signal of the captured image.

  The present invention is an image processing method for transmitting an image captured by a camera to a central control device, performing image processing for extraction and recognition of a specific subject, and displaying the image. Image feature information used for image processing is generated, and image information including the image feature information is transmitted to the central control device. The central control device uses the image feature information transmitted from the camera to extract and recognize a specific subject. Processing is performed, and the processed images are combined and displayed on the image display unit.

  ADVANTAGE OF THE INVENTION According to this invention, the image processing of the image image | photographed with the camera can be performed comfortably, and the imaging system excellent in real-time property can be provided.

FIG. 1 is a system configuration diagram showing an embodiment of an imaging system according to the present invention.
In the imaging system of the present embodiment, four cameras A, B, C, and D (reference numerals 10A, 10B, 10C, and 10D) are installed in a vehicle, and an image signal from each camera 10 is transmitted via a transmission path 20 to the center. The image data is transmitted to the control device 30, and the central control device 30 performs composition of each image and extraction processing of a specific subject, and displays them on the image display unit 40. In the embodiment, four cameras are installed. However, the number of cameras may be appropriately set (including the case of one). As an image processing function, image processing such as white line (lane) detection and person extraction is performed, and white lines (lanes) and markers indicating the positions of persons as shown by reference numerals 41 and 42 are displayed on the image display unit 40. indicate.

  In order to execute image processing efficiently, the cameras 10A to 10D and the central control device 30 are synchronously controlled based on a common video frame synchronization signal. This synchronization signal is issued from the central control device 30, but can also be issued from the camera side. Further, in order to execute image processing quickly in the central control device 30, partial information of the image or information extracted from the image (hereinafter referred to as image feature information) necessary for executing the image processing is obtained from each camera. Added or prioritized transmission in terms of time. Hereinafter, an image signal transmitted from the camera to the central controller 30 is also referred to as image information.

  In each of the cameras 10 </ b> A to 10 </ b> D, an image formed on the image sensor 12 from the optical system 11 is processed by the image processing circuit 13. The video processing circuit 13 performs wide dynamic range processing and reduced image generation processing on the captured image.

  First, the wide dynamic range process will be described. In order for the central control device 30 to perform image processing effectively, it is necessary to provide an image in which an appropriate contrast without overexposure or darkening is ensured. Particularly in automobile applications, overexposure and darkening are likely to occur when the headlight of an oncoming vehicle is taken or when the vehicle approaches the exit of a tunnel, and it is important to deal with such high-contrast subjects. Wide dynamic range technology is a technique for dealing with this. Specifically, the exposure time is changed, for example, the image is taken twice, the two shot images are temporarily stored in the memory 15, and an output image with a wide dynamic range is generated while adjusting the mixing ratio. .

  Next, reduced image generation creates an image with a reduced image size (number of pixels). Since the image processing performed by the central controller 30 is to extract a specific subject, it is possible to speed up the extraction processing by providing a reduced image having a resolution corresponding to the size of the subject. There is a sampling method as a simple method for generating a reduced image. Although this method simply thins out pixel values, there are cases in which a noise component is mixed and a false signal is generated. There is a scaling method as another method of generating a reduced image. This method obtains and represents the average of the pixel values of the reduction source region, and can reduce the generation of false signals. What is necessary is just to select the production | generation method suitably according to the kind of to-be-photographed object. Since the reduced image generation processing is sequentially performed according to the pixel arrangement in the memory 15, the input image data can be processed almost in real time with little delay time.

  The image processed by the video processing circuit 13 is stored in the memory 15 again. The transmission processing circuit 14 reads out the image data stored in the memory 15 in a predetermined order, and transmits it to the transmission path 20. The image information 21 to be transmitted includes image feature information (reduced image, luminance signal) used for image processing, and has a data format to be described later.

  In the central controller 30, the image information 21 of each camera received from the transmission path 20 is temporarily stored in the memory 32 via the transmission processing circuit 31. The image processing circuit 33 performs image processing such as white line (lane) detection and person extraction from the received image. At that time, image processing is efficiently executed by using image feature information (reduced image, luminance signal) included in the image information 21. The display processing circuit 34 performs viewpoint conversion processing, image synthesis processing, and the like on the image from each camera, generates a composite image including the extracted subject, and outputs the composite image to the image display unit 40.

The present embodiment is characterized by the format of the image information 21 transmitted from the camera 10 to the central control device 30. This will be described below.
FIG. 2 is a diagram showing an example of a conventional image information format for comparison. Here, a signal arrangement in the case of conforming to the CCIR656 format which is the most standard digital video transmission standard is shown.

  As the image information in this case, the luminance signal (Y) and the color difference signals (U, V) are alternately transmitted based on the timing T200 of the video frame synchronization signal. Then, at the timing T201 when all the signals of one video frame 01 are transmitted, all the data of the video frame 01 are prepared in the memory 32 of the central control device 30.

  At this timing T201, the image processing circuit 33 starts recognition extraction processing 210 for white line recognition and person extraction. Further, the display processing circuit 34 starts an image combining process 220 for each camera in order to generate a display image. When these processing results are to be reflected in the display image in the next video frame 02, each processing must be completed by the timing T202 (display start time) of the synchronization signal of the next video frame 02.

  Among these, in the display processing 220 by the display processing circuit 34, it is not necessary to frequently change the screen synthesis method and the calculation method, so that most of the processing circuits are easily implemented as hardware. Therefore, even if the processing time is short, it does not become a big problem. However, in the recognition extraction process 210 by the image processing circuit 33, various processes such as branching of various processes, comparison with a reference image, and derivation of a motion vector by referring to a past image are combined with an image that changes every moment. Done. Therefore, the ratio of software processing by the microprocessor inevitably increases. Accordingly, it is difficult to cope with a short processing time, and image display delay or failure occurs.

  On the other hand, FIG. 3 is a diagram showing an example of a format of image information in the present embodiment. In this case, the image information transmitted from the camera 10 to the central control device 30 is first transmitted in advance with a reduced image (S) used for pattern extraction of person extraction based on the timing T300 of the synchronization signal of the video frame 01. To do. Next, the luminance signal (Y) used for white line detection is transmitted at the timing T301, and finally the color difference signal (U, V) is transmitted at the timing T302. Here, the reduced image (S) is created by using the image data of the video frame 01 by the video processing circuit 13, that is, belongs to the same frame as the luminance signal (Y) and the color difference signals (U, V). is there. Further, the transmission processing circuit 14 reads the reduced image (S) first, the luminance signal (Y), and finally the color difference signals (U, V) in this order from the image signal stored in the memory 15 at each timing described above. Thus, the above data format is obtained.

  In the case of the present embodiment, the image processing circuit 33 can quickly start the person extraction processing 310 using the reduced image (S) at T301. Next, at T302, the white line recognition process 320 can be quickly started using the luminance signal (Y). As a result, as compared with FIG. 2, the processing time until the timing T304 of the synchronization signal of the next video frame 02 is greatly extended, and the recognition extraction processing 310, 320 by the image processing circuit 33 is executed with a margin. It is possible to prevent the processing from failing. In this embodiment, the reduced image (S) is transmitted with the first priority and the luminance signal (Y) is transmitted with the second priority. However, the order may be changed depending on the purpose of the image processing.

  In this embodiment, the timing T303 at which all the data of the video frame 01 are prepared is shifted to the latter half by the time for transmitting the reduced image (S). Therefore, the display processing 330 by the display processing circuit 34 starts from the delayed timing T303, and the processing time is shorter than that in FIG. 2, but the influence is small because there is a margin for the above reason.

  According to the present embodiment, since the start timing of the recognition / extraction processing in the central control device 30 is advanced and sufficient calculation processing time is secured, an increase in the overall delay time from imaging to display is suppressed. An imaging system can be provided.

  In the present embodiment, the central control apparatus has been described as performing person extraction processing by pattern matching using a reduced image and white line recognition using a luminance signal. However, the present invention is not necessarily limited thereto. For example, it is possible to extract edge information of a subject from a captured image and use it as information for image processing. It goes without saying that the same effect can be obtained by preferentially transmitting the image feature information necessary for the processing in the image information in accordance with the extraction / recognition processing method performed by the central control unit.

1 is a system configuration diagram showing an embodiment of an imaging system according to the present invention. The figure which shows the example of the format of the conventional image information. The figure which shows an example of the format of the image information in a present Example.

Explanation of symbols

10 ... Camera,
11 ... Optical system,
12: Image sensor,
13 ... Video processing circuit,
14 ... transmission processing circuit,
15 ... Memory,
20 ... transmission line,
21 ... Image information,
30 ... Central control unit,
31 ... Transmission processing circuit,
32 ... Memory,
33. Image processing circuit,
34 ... display processing circuit,
40. Image display unit,
41. Marker signal (white line),
42: Marker signal (person).

Claims (6)

In an imaging system that transmits an image captured by a camera to a central control device and performs image processing for extraction and recognition of a specific subject,
The camera includes a video processing circuit that generates image feature information used for image processing in the central control device from a captured image, and a transmission processing circuit that transmits image information including the image feature information to the central control device. ,
The central control device includes an image processing circuit that performs image processing for extracting and recognizing a specific subject using the image feature information transmitted from the camera, and a display process for combining the processed images and displaying the combined images on an image display unit. An imaging system comprising a unit. The imaging system according to claim 1,
The camera and the central control unit process the captured image for each video frame based on a common synchronization signal,
When the transmission processing circuit transmits the image information to the central control device, the image characteristic information used for image processing is transmitted in advance in the video frame. The imaging system according to claim 1 or 2,
The imaging system, wherein the image feature information is a reduced image of the captured image or a luminance signal of the captured image. In an image processing method for transmitting an image captured by a camera to a central control device, performing image processing for extraction and recognition of a specific subject, and displaying an image,
The camera generates image feature information used for image processing in the central control device from a captured image, transmits image information including the image feature information to the central control device,
The central control device performs image processing for extracting and recognizing a specific subject using the image feature information transmitted from the camera, synthesizes the processed images, and displays them on an image display unit Processing method. The image processing method according to claim 4,
The camera and the central control unit process the captured image for each video frame based on a common synchronization signal,
An image processing method according to claim 1, wherein when the image information is transmitted to the central controller, the camera transmits the image feature information used for image processing in advance in the video frame. The image processing method according to claim 4 or 5,
The image processing method, wherein the image feature information is a reduced image of the captured image or a luminance signal of the captured image.

Images