JP2013206356A - Image processing device and image processing camera device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it possible to simultaneously operate an application using a low-resolution image and an application using a high-resolution image by using one frame of images.SOLUTION: An image processing device that performs processing on a picked up image 121 picked up by a camera unit 100 includes: a super-resolution processing unit 102 as a resolution processing unit; and an image recognition engine 111 as an application image processing unit. The super-resolution processing unit 102 converts the picked up image 121 into a restoration image 122 as a high-resolution image, and the application image processing unit 111 includes: a light spot detection processing unit 113 as an application processing unit using the restoration image 122; and a lane and vehicle detection processing unit 112 as the application processing unit using the restoration image 122. In the image recognition engine 111, a plurality of applications are simultaneously executed.

Description

  The present invention relates to an apparatus that processes an image captured by an in-vehicle camera or the like, and can simultaneously perform an application using high-resolution image information such as lane recognition and an application using low-resolution image information, for example. The present invention relates to an image processing apparatus and a camera apparatus using the image processing apparatus.

  As a background art in this technical field, a camera device with an image quality degradation function has a video quality degradation means that can be restored in the camera, and has been degraded to the minimum image quality necessary for state judgment in normal monitoring work. When a clear image is necessary, there is a camera device that restores and uses the deteriorated image quality. In this camera device, degradation information restoration technology for improving image quality using the distribution of degradation information is used as the image degradation means that can be restored, and a special file necessary for restoration of the image quality is generated during the process of degrading the image quality. Or encryption key management is not required. Further, the image deterioration means is configured to have an image quality deterioration level switching means for setting the image quality deterioration level according to the purpose of use (see, for example, Patent Document 1).

JP 2010-103867 A

  The camera device having the above-described structure describes a mechanism that restores a recoverable image quality degraded image to a clear image using degradation information restoration technology. However, the image capturing means of Patent Document 1 cannot simultaneously output a deteriorated image and a clear image. For example, normally, it is necessary to fix and use a deteriorated image and to restore it separately when a clear image is required. For example, image processing that requires a degraded image (headlight control application) and image processing that requires a clear image (automatic inter-vehicle control application) cannot be operated simultaneously.

  The present invention has been made in view of such problems, and an object of the present invention is to use an application using a low-resolution image such as a headlight control application using an image of one frame, and an inter-vehicle distance. An object of the present invention is to provide an image processing apparatus capable of simultaneously operating an application using a high resolution image such as an automatic control application. Another object of the present invention is to provide an image processing camera device using the image processing device. In the present specification, in the following description, the technique for restoring a deteriorated image to a clear image is referred to as super-resolution processing.

  In order to achieve the above object, an image processing apparatus according to the present invention is an image processing apparatus that processes a captured image captured by a camera unit, and the image processing apparatus includes a resolution processing unit and an application image processing unit. The resolution processing unit converts the captured image into a high resolution image, and the application image processing unit includes an application processing unit using the captured image and an application processing unit using the high resolution image. It is characterized by providing. The captured image is a low-resolution image, and the image processing unit performs light spot detection processing using the low-resolution image.

  The image processing apparatus of the present invention configured as described above performs application processing using a low-resolution captured image captured by the camera unit, converts the low-resolution captured image into a high-resolution image, and converts the high-resolution image. Other application processing can be performed simultaneously using the image. For example, it is possible to perform the light spot detection process using the captured image as it is at a low resolution, and simultaneously perform the lane detection process and the preceding vehicle detection process using the high resolution image.

  According to the present invention, it is possible to simultaneously operate a headlight control application using low-resolution image information and an inter-vehicle automatic control application using high-resolution image information using one frame image captured by an in-vehicle camera or the like. An image processing device and an image processing camera device can be provided.

1 is a block diagram showing the configuration of Embodiment 1 of an image processing camera apparatus using an image processing apparatus according to the present invention. The figure explaining the angle of view and angle resolution of the image processing camera apparatus of FIG. The figure explaining angle resolution required for light spot detection, such as a preceding vehicle taillight. It is a figure explaining angle resolution and required resolution, (a) is a schematic diagram which shows the relationship between a light source and an image sensor, (b) is a schematic diagram which blurred and reflected the light source, (c) is a color of an image sensor. The schematic diagram of a pattern. It is a figure explaining a focus and a flange back, (a) is a characteristic figure of MTF, (b) is an explanatory view of a flange back, (c) is an explanatory view of other examples which lower MTF. The figure explaining the focus recovery | restoration by focus adjustment range and super-resolution. The block diagram of the vehicle-mounted image processing stereo camera apparatus which shows Example 2 of the image processing camera apparatus which concerns on this invention.

[Example 1]
Hereinafter, an embodiment of an image processing camera apparatus using an image processing apparatus according to the present invention will be described in detail with reference to the drawings. In the first embodiment, an example of an in-vehicle image processing camera device capable of simultaneously operating a headlight control application using a low resolution image and an automatic inter-vehicle control application using a high resolution image using an image of one frame. Will be explained.

  FIG. 1 is a block diagram illustrating a configuration of an image processing camera apparatus 10 using the in-vehicle camera of the present embodiment. In FIG. 1, an in-vehicle image processing camera device 10 includes a camera unit 100, an image processing unit 101, a super-resolution processing unit 102, for example, an image recognition engine 111 mounted on an image processing CPU. A lane / vehicle detection processing unit 112 is provided as a vehicle-to-vehicle automatic control application, and a light spot detection processing unit 113 is provided as a headlight control application.

  The camera unit 100, for example, captures information in front of the vehicle with a lens system and an image sensor, which will be described later, and outputs image information. In the first embodiment, the camera unit 100 is not focused high-resolution image information. The shifted and blurred unfocused image information (low-resolution image) is output and output to the image processing unit 101. The image processing unit 101 separates the unfocused image information into two, and outputs one image information as it is to an image recognition engine 111 as an application processing unit described later. The image processing unit 101 includes a super-resolution processing unit 102 that converts the other image information into high-resolution image information. Details of the super-resolution processing unit 102 will be described later.

  The high-resolution image information output from the super-resolution processing unit 102 is supplied to a lane / vehicle detection processing unit 112 as an inter-vehicle automatic control application of the image engine 111. The low-resolution image information (low-resolution image that is a captured image) separated by the image processing unit 101 is supplied to a light spot detection processing unit 113 as a headlight control application of the image recognition engine 111. The light spot detection processing unit 113 performs image processing such as tail light detection of a preceding vehicle and headlight detection of an oncoming vehicle, for example. Various applications are executed by the image recognition engine 111. The low-resolution image is a captured image itself, and the high-resolution image is a recovered image obtained by performing super-resolution processing on the captured image.

  Here, the super-resolution processing unit 102 will be described. The super-resolution processing unit 102 performs super-resolution processing on the unfocused blurred image information 121 captured by the camera unit 100 and outputs a recovered image 122 in which blur is recovered. Various known methods are used for this super-resolution processing. One technique is to model and digitize the blur characteristics of an imaging system such as an image sensor or lens, and use the digitized characteristics to restore the original image that would otherwise be obtained without blur. There is.

  Next, with reference to FIG. 2, the angular resolution determined by the relationship between the image sensor 200 and the angle of view 201 in the camera unit 100 will be described. In FIG. 2, for example, when the angle of view 201 that is incident on the lens 202 is 60 degrees, in other words, when the image processing target area is 60 degrees, the number of pixels of the image sensor is 1200 pixels. In this case, the angular resolution is 60 degrees / 1200 pixels = 0.05 degrees / pixel. Information for 0.1 degree can be obtained with 2 pixels, and conversely, it is narrower than 0.05 degree, for example, far-distance information is smaller than the resolution that can be obtained with 1 pixel, so it is useful for image processing. Necessary imaging cannot be performed. That is, in this example, it can be said that the amount of information obtained per pixel of the image sensor 200 is 0.05 degrees.

  In addition, with reference to FIG. 3, the angular resolution necessary for detecting the taillight of the preceding vehicle mainly handled in the headlight control application will be described. In FIG. 3, the preceding vehicle taillight 302 of the preceding vehicle 301 is imaged by the camera device 300 typified by the in-vehicle image processing device, and the imaging device 200 and the lens 202 provided in the camera device 300. For example, when the taillight of a preceding vehicle 500 m ahead has a diameter (φ) of 0.1 m, the angular resolution required to detect the taillight of the preceding vehicle is ArcTan (0.1 m / 500 m) ≈ 0.011 degree.

  Furthermore, with reference to FIG. 4, the angular resolution described with reference to FIGS. 2 and 3 and the angular resolution necessary for detecting the taillight will be specifically described. In FIG. 4 (a), for example, when the taillight of the preceding vehicle 500m ahead is used as the light source 400, and the image is picked up by the imaging system having the above-described 1200-pixel imaging device and the angle of view 201 of 60 degrees, The resolution 401 necessary for light detection is 0.011 degree. The angular resolution is 0.05 degrees. At this time, since the information for 0.05 degree is imaged within one pixel size 404, the taillight reflection area 403 having the information for only 0.011 degree is smaller than the one pixel size 404.

  On the other hand, in taillight detection, the characteristic red color is often detected. Therefore, color imaging is often used for the image sensor 200. As shown in FIG. 4C, the general Bayer pattern 406 includes a red filter (R), a green filter (G1, G2), and a blue filter (B) for 4 pixels of 2 × 2 pixels. . At this time, as described above, if the taillight reflection area 403 is present in an area smaller than one pixel, a so-called false color is generated when the color demosaicing process is performed, and red cannot be reproduced correctly. Become. Therefore, in order to obtain red, as shown in FIG. 4B, it is necessary to blur and capture the taillight reflection region 407 with respect to at least the region of 2 × 2 pixel size 405. There are various methods for the color demosaicing processing here, but it is only necessary to capture a blurred image in an area where red can be reproduced correctly.

  In addition, for example, a blurred image captured in an optical system including a lens will be described with reference to FIG. 5A and 5B, generally, an MTF (Modulation Transfer Function) characteristic 500 is a constant distance at which the focal length is matched to the distance between the lens 510 and the image sensor 200, that is, the flange back. In this case, the MTF maximum (just focus) is obtained. Even if the distance between the lens 510 and the image sensor 200 is short with respect to the just-focus flange back, conversely, the MTF (focus) is small even if the distance between the lens 510 and the image sensor 200 is long. That is, a blurred image is obtained.

  As described above, the focus 501 required for the light spot processing (headlight control application) represented by the preceding vehicle taillight cannot be detected if the MTF (focus) is too high, and therefore the MTF (focus) needs to be small to some extent. is there. On the other hand, the higher the focus MTF (502) that is needed for lane detection or vehicle detection (automatic inter-vehicle control application), for example, that requires farther information ahead of travel, is better.

  Thus, simultaneously operating the headlight control application and the inter-vehicle automatic control application using one frame image captured by the camera unit 100 is a contradictory MTF (focus), that is, a low-resolution image and a high-resolution image. In the first embodiment, a plurality of applications can be executed simultaneously by obtaining a high-resolution recovery image from a low-resolution captured image captured by the camera unit 100. In addition, although the example which moves the image pick-up element 200 was shown in order to make a flange back variable, you may comprise so that an image pick-up element may be fixed and the lens 510 may be moved.

  Further, as an alternative to the method of making the flange back variable in order to obtain a low-resolution captured image, as shown in FIG. 5C, by fixing the flange back and inserting a phase plate 511 or the like, The MTF (focus) can also be manipulated by physically changing the focal position.

  Furthermore, with reference to FIG. 6, it will be described that the headlight control application and the vehicle-to-vehicle automatic control application are simultaneously operated using one frame image by super-resolution processing. FIG. 6 illustrates that a high-resolution recovery image is obtained from a low-resolution captured image by changing the flange back.

  In FIG. 6, a region where the MTF (focus) is higher than the focus recoverable region 601 by the super-resolution processing and the MTF (focus) is smaller than the focus 501 necessary for the light spot detection processing is the flange back adjustment range. 602. By adjusting the flange back of the lens 510 and the image sensor 200 to the adjustment range 602, an image with a small MTF (focus) suitable for a light distribution application is used as it is as a blurred image, and image processing is performed on the other hand. An image suitable for a control application can be subjected to image processing after the MTF (focus) is recovered by super-resolution processing 603 and the MTF (focus) is increased.

  The operation of the image processing camera apparatus 10 of the present embodiment configured as described above will be described below. An unfocused blurred image 121 captured by the camera unit 100 is created and input to the image processing unit 101. In the image processing unit 101, the blurred image 121 is separated into two, and one piece of image information is input to the light spot detection processing unit 113 of the image recognition engine 111 while being blurred. Further, the other blurred image information 121 is converted into high-resolution image information by the super-resolution processing unit 102 in the image processing unit 101, and the blurred image 121 is subjected to super-resolution processing to recover blur. The restored image 122 is output.

  In the image recognition engine 111, the light spot detection processing unit 113 performs light spot detection processing using the blurred image 121, and operates the headlight control application. At the same time, the lane / vehicle detection processing unit 112 uses the high-resolution recovery image 122 to perform lane detection processing and vehicle detection processing, and activates the inter-vehicle automatic control application. As described above, the image recognition engine 111 can simultaneously operate the headlight control application and the inter-vehicle automatic control application.

[Example 2]
In the second embodiment, an application example of the image processing apparatus to the in-vehicle image processing stereo camera will be described with reference to FIG. In other words, the image processing camera device 20 according to the second embodiment uses a stereo camera including a right camera unit and a left camera unit. FIG. 7 is a block diagram of the in-vehicle image processing stereo camera device according to the second embodiment. In this embodiment, components substantially equivalent to those of the first embodiment are denoted by the same reference numerals, and detailed description thereof is omitted.

  In FIG. 7, the image processing camera device 20 according to the second embodiment includes a plurality of cameras of a right camera unit 700 and a left camera unit 701, and configures a stereo camera device. Further, the image processing unit 101 that performs image processing on images input from a plurality of cameras separates the image information output from the right camera unit 700 into three, and performs subsequent application processing with one image information as it is. Supply to the department. The image processing unit 101 includes a super-resolution processing unit 102 that processes another piece of image information into high-resolution image information, and a light spot detection processing unit 703 that processes another piece of image information.

  The image processing unit 101 further includes a parallax calculation unit 704 that performs stereo image processing. The parallax calculation unit 704 outputs a parallax image 710 for performing parallax calculation for stereo image processing based on image information obtained by increasing the resolution of the captured images of the right camera unit 700 and the left camera unit 701.

  The light spot detection processing unit 703 outputs a light spot image 711 for image processing represented by headlight control. The light spot detection processing unit 703 is not always necessary. As described with reference to FIG. 1, the headlight control application may be operated by mounting a light spot detection processing unit on the image recognition engine 111.

  The image recognition engine 111 as an application processing unit includes an inter-vehicle automatic control application 721 and a headlight control application 722. The inter-vehicle automatic control application 721 executes lane / vehicle detection processing, etc., as in the first embodiment, and the headlight control application 722, as in the first embodiment, detects taillights of the preceding vehicle or is opposed to the vehicle. Executes light spot detection processing such as car headlight detection. Alternatively, image processing using the light spot image 711 obtained from the light spot detection processing unit 703 is executed. Note that the image recognition engine 111 may execute other image processing.

  In the image processing camera device 20 according to the second embodiment configured as described above, the low-resolution blurred captured image 121 captured by the right camera unit 700 is separated into three pieces of image information, and one piece of image information is an image processing unit. 101 is converted into high-resolution image information by the super-resolution processing unit 102 and output as a restored image 122 from which blur is recovered, and the other image information remains at low resolution, and an image recognition engine 111 as a subsequent-stage application processing unit. Is output. Further, one piece of image information is input to the light spot detection processing unit 703 as a light spot detection process, and a light spot process for detecting a light spot is executed.

  The low-resolution blurred captured image 121 captured by the left camera unit 701 is input to the super-resolution processing unit 102, converted into high-resolution image information, and output as a recovered image from which blur is recovered. The left and right high-resolution images converted into high-resolution image information by the super-resolution processing unit 102 are input to the parallax calculation unit 704, and the parallax calculation unit 704 performs parallax calculation based on the left and right images. The light spot detection processing by the light spot detection processing unit 703 and the parallax calculation by the parallax calculation unit 704 are performed on the same image captured in one frame.

  The parallax image 710 obtained in the image processing by the parallax calculation unit 704 and the recovered image 122 that has been recovered from blur by the super-resolution processing unit 102 are used in the inter-vehicle automatic control application 721 in the image recognition engine 111. Further, the light spot image 711 and the blurred image 121 obtained in the image processing by the light spot detection processing unit 703 are used for the headlight control application 722 in the image recognition engine 111. Thus, also in the image processing stereo camera device 20 using the in-vehicle stereo camera, the headlight control application and the inter-vehicle automatic control application can be operated simultaneously.

  In the parallax calculation of the parallax calculation unit 704, the calculated parallax image 710 is used, for example, to detect the inter-vehicle distance from the preceding vehicle. When the detected inter-vehicle distance is kept constant or the inter-vehicle distance is small It is used for the above-mentioned automatic inter-vehicle control application such as control for slowing down the vehicle. In addition, although an example of an automatic inter-vehicle control application is lane / vehicle detection, it is also used to detect an obstacle such as a pedestrian. In addition to the inter-vehicle automatic control application, it is also used for pre-crash control (collision avoidance / reduction control for obstacles) and the like.

  In the first and second embodiments described above, taillight detection is given as an example of the headlight control application, but applications that require color information such as signal detection and sign detection are also included.

  As mentioned above, although embodiment of this invention was explained in full detail, this invention is not limited to an above-described Example, Various modifications are included. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Further, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

  Each of the above-described configurations, functions, processing units, processing means, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit. Each of the above-described configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files for realizing each function can be stored in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

  In addition, the control lines and information lines are those that are considered necessary for the explanation, and not all the control lines and information lines on the product are necessarily shown. Actually, it may be considered that almost all the components are connected to each other. An example of an in-vehicle camera is shown as a camera device, and an example of two stereo camera devices is shown as a plurality of camera devices. However, an image picked up by a fixed camera such as a fixed point camera or an image picked up by three or more camera devices A plurality of application processes may be performed using an image.

  As an application example of the present invention, it is possible to simultaneously perform a plurality of application processes with image signals captured by various camera devices using this image processing device, and it can be applied to various uses of vehicles by attaching an in-vehicle camera. At the same time, it can be applied to various uses such as local safety by attaching a fixed point camera.

10: Image processing camera device 20: Image processing stereo camera device 100: Camera unit 101: Image processing unit 102: Super-resolution processing unit (resolution processing unit)
111: Image recognition engine (application image processing unit)
112: Lane / vehicle detection processing unit 113: Light spot detection processing unit 121: Blurred image (low resolution image)
122: Recovery image (high resolution image)
200: imaging device 201: angle of view 202: lens 300: camera device 301: preceding vehicle 302: preceding vehicle taillight 400: light source 401: necessary taillight detection resolution 402: angular resolution 403: taillight reflection region 404: 1 pixel Size 405: 2 × 2 pixel size 406: Bayer pattern 407: Blurred taillight reflected area 500: MTF characteristic 501: Focus necessary for light spot detection processing 502: Focus necessary for lane / vehicle detection 510: Lens 511 Phase plate 601: recoverable region 602 by super-resolution: flange back adjustment range 603: super-resolution processing 700: right camera unit 701: left camera unit 703: light spot detection processing unit 704: parallax calculation unit 710: parallax image 711 : Light spot image 721: Inter-vehicle automatic control application 722: Headlight control Application

Claims (7)

An image processing apparatus that processes a captured image captured by a camera unit,
The image processing apparatus includes a resolution processing unit and an application image processing unit,
The resolution processing unit converts the captured image into a high-resolution image,
The application image processing unit includes an application processing unit using the captured image and an application processing unit using the high-resolution image.   The image processing apparatus according to claim 1, wherein the captured image is a low-resolution image, and the application processing unit performs light spot detection processing using the low-resolution image. The camera unit includes a lens and an image sensor,
The image processing apparatus according to claim 2, wherein the low-resolution image is captured by adjusting a flange back of the lens and the imaging element.   The flange back has a fixed adjustment range so that the low resolution image can be acquired and the high resolution image can be acquired by the super-resolution processing unit. Image processing device.   The image processing apparatus according to claim 1, wherein the image processing unit includes a light spot detection processing unit.   The said camera part is provided with the right camera part and the left camera part, and the said image process part is provided with the parallax calculation part which calculates | requires a parallax from the image imaged with the left and right camera part. The image processing apparatus according to any one of the above.   An image processing camera apparatus comprising: the image processing apparatus according to claim 1; and the camera unit that captures the captured image.

Images