JPWO2014174765A1 - Imaging apparatus and imaging method - Google Patents

Abstract

In the imaging apparatus and the imaging method according to the present invention, a composite image is generated based on the first and second images obtained by imaging in the first and second wavelength bands, respectively. When the image composition is generated, the object image in the second image is synthesized with the composition target image in the first image. For this reason, the image pickup apparatus and the image pickup method according to the present invention can supplement the image quality that is insufficient in one image with another image, and thus can further improve the visibility of the object.

Description

  The present invention relates to an imaging apparatus and an imaging method for imaging a first image and a second image and performing image processing.

  2. Description of the Related Art Conventionally, an imaging device that captures a predetermined subject at night or in an environment with poor visibility such as rain or fog has been used for monitoring or in-vehicle use. In Patent Document 1, an object such as a pedestrian is detected by using an infrared image obtained by imaging infrared light in order to support driving of the driver under the above environment. A technique for highlighting an object is disclosed.

  In the technique described in the above prior art, the object is highlighted, but the shape or the like of the object itself is changed and displayed, which may give an unnatural impression to the driver. This prior art has room for improvement in visibility in this respect, for example.

JP 2001-76298 A

  The present invention has been made in view of the above circumstances, and an object thereof is to provide an imaging apparatus and an imaging method that can further improve the visibility of an object.

  In the imaging apparatus and the imaging method according to the present invention, a composite image is generated based on the first and second images obtained by imaging in the first and second wavelength bands, respectively. When the image composition is generated, the object image in the second image is synthesized with the composition target image in the first image. For this reason, the image pickup apparatus and the image pickup method according to the present invention can supplement the image quality that is insufficient in one image with another image, and thus can further improve the visibility of the object.

  The above and other objects, features and advantages of the present invention will become apparent from the following detailed description and the accompanying drawings.

It is a block diagram which shows an example of a structure of the imaging device concerning 1st Embodiment. It is a fragmentary diagram of the optical filter group concerning an embodiment. It is an operation | movement flowchart of the imaging device concerning 1st Embodiment. It is a figure for demonstrating the visible image, near-infrared image, and synthesized image concerning 1st Embodiment. It is a block diagram which shows an example of a structure of the imaging device concerning 2nd Embodiment. It is a figure for demonstrating the visible image, near-infrared image, and synthesized image concerning 2nd Embodiment. It is a block diagram which shows an example of a structure of the imaging device concerning 3rd Embodiment.

  Hereinafter, an embodiment according to the present invention will be described with reference to the drawings. In addition, the structure which attached | subjected the same code | symbol in each figure shows that it is the same structure, The description is abbreviate | omitted suitably. In the present specification, when referring generically, it is indicated by a reference symbol without a suffix, and when referring to an individual configuration, it is indicated by a reference symbol with a suffix.

[First Embodiment]
FIG. 1 is a block diagram illustrating an example of the configuration of the imaging apparatus 1000 according to the first embodiment. FIG. 2 is a partial view of the optical filter group according to the embodiment. In FIG. 2, each optical filter is partially shown by looking at the optical filter group arranged in a matrix with a predetermined arrangement from the incident side. FIG. 2A is a diagram illustrating the RGBIr mode optical filter group 103 according to the first embodiment, and FIG. 2B is a diagram illustrating the WYRIr mode optical filter group 103 according to the first embodiment. FIG. 2C is a diagram illustrating a WYR type optical filter group 103a according to a second embodiment described later, and FIG. 2D is a diagram illustrating a CMYG type optical filter group 103a according to a second embodiment described later.

  In FIG. 1, an imaging apparatus 1000 according to the present embodiment includes, for example, an imaging unit 100, an image processing unit 200, an output unit 300, and an input unit 400. Imaging data is input from the imaging unit 100 to the image processing unit 200, and data after image processing is output from the image processing unit 200 to the output unit 300. The input unit 400 is connected to the image processing unit 200.

  The imaging unit 100 is an apparatus that captures an image of a predetermined subject (object), for example, for monitoring or in-vehicle use. The imaging unit 100 includes, for example, an optical system 101, an optical filter group 103, and an image sensor 105. Light from the predetermined subject enters the optical filter group 103 through the optical system 101, and only light in a predetermined wavelength range is transmitted through the optical filter group 103 and enters the image sensor 105. More specifically, light in a predetermined wavelength range transmitted through the optical filter group 103 enters a plurality of photoelectric conversion elements included in the image sensor 105, and an analog electrical signal corresponding to the intensity of the incident light is imaged. The image is output from the sensor 105 to the image processing unit 200.

  The optical system 101 includes an optical element such as one or a plurality of lenses, and forms, for example, light from the predetermined subject on the light receiving surface of the image sensor 105 via the optical filter group 103. The optical filter group 103 includes a plurality of optical filters that transmit light in a predetermined wavelength range with different center wavelengths. In this embodiment, the optical filter group 103 is, for example, a red (R) filter that transmits red light, a green (G) filter that transmits green light, a blue (B) filter that transmits blue light, and infrared light. There are provided four types of optical filters, infrared (Ir) filters that transmit light (in the present embodiment, near-infrared filters). Each of these four types of optical filters has a predetermined arrangement, for example, as shown in FIG. 2A. Thus, they are arranged in a matrix in two directions, the x direction and the y direction, which are orthogonal to each other. The optical filter group 103 of the present embodiment is similar to a so-called Bayer array type optical filter group, and one of two green filters included in the Bayer array type optical filter group is replaced with a near infrared filter. In the matrix arrangement, for example, when the position of s rows and t columns is represented as (t, s), the configuration of the optical filter group 103 is specifically as follows. That is, the optical filter group 103 includes, for example, the G filter at the (1,1) position, the R filter at the (2,1) position, the B filter at the (1,2) position, and the Ir filter at (2). , 2) are configured by arranging basic patterns (rectangular portions indicated by A in the figure) arranged in a matrix in two directions, the x direction and the y direction (hereinafter also referred to as RGBIr mode). Call). As shown in FIG. 2B, the optical filter group 103 includes, for example, a yellow (Y) filter that transmits yellow light (2, 2) at the position of a white (W) filter (1, 1) that transmits white light. The basic pattern (rectangular portion indicated by B in the figure) in which the R filter is disposed at the position (1,2) and the Ir filter is disposed at the position (2,2) at the position 1) is the x direction and y It may be configured by being arranged in a matrix in two directions (hereinafter also referred to as a WYRIr mode).

  The optical filters (R, G, B, Ir or W, Y, R, Ir filters) arranged in a matrix in the optical filter group 103 are arranged in a matrix included in the image sensor 105 described later. Each of the plurality of photoelectric conversion elements has a one-to-one correspondence. In the case of the RGBIr mode, on the light receiving surface of the image sensor 105, the photoelectric conversion element that receives green light is located at the position (1, 1), the photoelectric conversion element that receives red light is located at the position (2, 1), and blue. A basic pattern in which a photoelectric conversion element that receives light is disposed at a position (1, 2) and a photoelectric conversion element that receives near-infrared light is disposed at a position (2, 2), respectively, has the x direction and y It is arranged in a matrix in two directions. As described above, each of the plurality of photoelectric conversion elements receives light of a single color. For example, the photoelectric conversion element located at (2, 2) in FIG. 2A receives only near-infrared light. Thus, since one photoelectric conversion element receives light of a single color, it generates a pixel value of that color, but does not generate pixel values of other colors. For this reason, the pixel value of the other color in the photoelectric conversion element is the other color generated by the photoelectric conversion element that receives the light of the other color disposed around the photoelectric conversion element, for example. It is generated by interpolation using the pixel values. For example, the blue pixel value in the photoelectric conversion element that receives near-infrared light located at (2, 2) is the photoelectric value existing at the positions (1, 2) and (3, 2) adjacent to this. It is calculated by performing an interpolation operation using the blue pixel value generated by the conversion element.

  The image sensor 105 includes a plurality of photoelectric conversion elements arranged in a matrix that photoelectrically converts incident light and outputs a value corresponding to the intensity of the incident light. Among the plurality of photoelectric conversion elements in the image sensor 105, the plurality of photoelectric conversion elements that receive light through the R filter, the G filter, and the B filter respectively receive light in a wavelength region included in the first wavelength band. A plurality of first photoelectric conversion elements L1 that pick up an image and output first image pickup data D1. Among the plurality of photoelectric conversion elements in the image sensor 105, the plurality of photoelectric conversion elements that receive light through the Ir filter capture images of light in a second wavelength band that includes a wavelength band different from the first wavelength band. The plurality of second photoelectric conversion elements L2 that output the second imaging data D2. In this embodiment, for example, the light in the first wavelength band is visible light, and there are a plurality of light in the wavelength region included in the first wavelength band, which are red light, green light, and blue light. The light in the second wavelength band is infrared light (near infrared light in the present embodiment).

  In the present embodiment, the image sensor 105 is one image sensor, and the plurality of first photoelectric conversion elements L1 and the plurality of second photoelectric conversion elements L2 are formed on one image sensor 105. The image sensor 105 outputs the first imaging data D1 and the second imaging data D2 to the image processing unit 200 as analog electrical signals. The first image and the second image are images based on the first imaging data D1 and the second imaging data D2 output from one image sensor 105, respectively, and have the same size (same image size) and the same shape. The number of pixels of the first image and the second image is the same after the interpolation operation or the like as described above. Note that the image sensor 105 of the present embodiment is, for example, a CMOS (Complementary Metal Oxide Semiconductor) type or a CCD (Charge Coupled) type in which a photoelectric conversion characteristic representing a characteristic of an electric output with respect to the intensity of incident light is a linear characteristic or a linear logarithmic characteristic. Device) type image sensor. When the image sensor 105 has a linear logarithmic characteristic, since the dynamic range is wide, the dynamic range is appropriately compressed according to the dynamic range of the output unit 300. For this dynamic range compression, known conventional means are used, for example, disclosed in Japanese Patent No. 47367792.

  The image processing unit 200 detects a predetermined type of object from the second image based on the second imaging data D2, and includes the first image based on the first imaging data D1 and the second type including the predetermined type of object. Image processing for generating a composite image based on the second image based on the imaging data D2 is performed. In the present embodiment, the first image based on the first imaging data D1 is a so-called visible image, and the second image based on the second imaging data D2 is a so-called near infrared image. The image processing unit 200 includes a control calculation unit 211, for example. The control calculation unit 211 is connected to the image sensor 105, and the output from the control calculation unit 211 is output to the output unit 300.

  The control calculation unit 211 performs various calculations such as overall control of the imaging unit 100, the image processing unit 200, the output unit 300, and the input unit 400 and image synthesis. The control calculation unit 211 includes, for example, a CPU (Central Processing Unit), an EEPROM (Electrically Erasable Programmable Read Only Memory), a ROM (Read Only Memory), and a RAM (Random Access Memory). In addition, the control calculation unit 211 functionally includes a control unit 215, an object extraction unit 209, and an image composition unit 205 by executing an image processing program stored in the ROM. The object extraction unit 209 receives the second imaging data D2 from the image sensor 105, and the output of the object extraction unit 209 is output to the image composition unit 205. The image composition unit 205 receives the first imaging data D1 from the image sensor 105. The image composition unit 205 is connected to the object extraction unit 209, and the output of the image composition unit 205 is output to the output unit 300.

  The control unit 215 governs overall control of the imaging unit 100, the image processing unit 200, the output unit 300, and the input unit 400. The object extraction unit 209 detects a predetermined type of object T (hereinafter simply referred to as the object T) from the near-infrared image, and includes the object T from the near-infrared image based on the second imaging data D2. The object image G2 in the second image area is extracted. In the present embodiment, for example, the object T is set in advance, and is, for example, a constant temperature organism such as a person or an animal, and may be plural. The shape of the object image G2 including the object T is a predetermined appropriate shape, for example, a rectangular shape. The shape of the object image G2 may be, for example, the shape of the object T itself. The object extraction unit 209 includes, for example, an object detection unit 201 and an object image region extraction unit 203. The object detection unit 201 receives the second imaging data D <b> 2 from the image sensor 105, and the output of the object detection unit 201 is output to the object image region extraction unit 203. The object image region extraction unit 203 receives the second imaging data D2 from the image sensor 105. The object image region extraction unit 203 is connected to the object detection unit 201, and the output of the object image region extraction unit 203 is output to the image composition unit 205.

  The object detection unit 201 detects a predetermined type of object T from the second image based on the second imaging data D2. More specifically, the object detection unit 201 detects the object T from the near-infrared image. The detection method of the target T in the target detection unit 201 uses, for example, a so-called pattern matching method that detects the target T using a predetermined template in the form of the target T (a constant temperature organism such as a person or an animal). (For example, refer to Japanese Patent Application Laid-Open No. 2004-145660). Another method for detecting the object T is, for example, creating an edge direction histogram indicating the degree of sharpness of the edge of the image for all pixels of the infrared image, and the degree of sharpness of the edge and the corresponding pixel. May be a method of detecting the target T by estimating the shape (see, for example, JP 2010-117772 A). In this method, a boosting method machine learning algorithm or the like may be used to improve detection accuracy (for example, Takayoshi Yamashita, Hironobu Fujiyoshi, “Features Effective for Specific Object Recognition”, Information Processing Society of Japan) Research report, CVIM 165, November, 2008, pp. 221-236). By these techniques, it is possible to avoid detection of, for example, an automobile engine as a heat source other than a thermostat such as a human or an animal from a near-infrared image, and the target T can be detected with higher accuracy.

  In the present embodiment, the object detection unit 201 detects the object T as described above, and includes the detected object T, for example, an area of the object image G2 in the second image based on the second imaging data D2. Information is output to the object image region extraction unit 203. The area information is information representing the range of the object image G2 in the second image. The area information is, for example, each pixel coordinate (hereinafter, also referred to as vertex coordinate) corresponding to each of the four vertices in the rectangle of the object image G2. Here, the coordinate value of the pixel in the captured image based on the captured image data is, for example, the pixel at one of the four vertices in the rectangle of the captured image as the origin, the longitudinal direction of the image as the x-axis direction, and the shortness of the image. It is a coordinate value in an xy orthogonal coordinate system or the like in which the hand direction is the y-axis direction.

  The object image region extraction unit 203 extracts the object image G2 from the second image based on the second imaging data D2. More specifically, the object image region extraction unit 203 extracts the object image G2 from the near-infrared image. In the present embodiment, the object image area extraction unit 203, for example, sets a pixel value of an area excluding the second image area r2 (object image G2) in the near-infrared image based on the area information to a predetermined constant value. A mask image M is created by converting to. As a result, the object image region extraction unit 203 extracts the object image G2 from the near-infrared image, and outputs a mask image M including the object image G2 to the image composition unit 205. Since the object image G2 included in the near-infrared image has only the luminance value as the pixel value, in the present embodiment, the predetermined constant value is, for example, zero. In the mask image M, the region information other than the second image region r2 of the object image G2 is masked (blackened), so that the region information is substantially superimposed on the mask image M. The object image region extraction unit 203 stores, for example, a near-infrared image corresponding to the region information until the region information is input from the object detection unit 201 to the object image region extraction unit 203. The region information and the near-infrared image corresponding to the region information are synchronized.

  The image synthesis unit 205 generates a synthesized image based on the first image based on the first imaging data D1 and the second image based on the second imaging data. More specifically, the image composition unit 205 transfers the object image G2 in the second image area r2 including the object T in the second image to the second image area r2 in the first image. The image is combined with the image to be combined G1 in the corresponding first image region r1.

  Based on the region information, the image composition unit 205 in the present embodiment synthesizes the object image G2 extracted by the object extraction unit 209 with the composition target image G1, and outputs the combined image data to the output unit 300. . More specifically, the image synthesizing unit 205 synthesizes the mask image M (the same shape, the same size, and the same number of pixels as the first image) for each corresponding pixel to the visible image that is the first image. Thus, the object image G2 is synthesized with the synthesis target image G1 between the rectangular second image region r2 and the rectangular first image region r1 having the same vertex coordinates as the second image region r2. . By using the mask image M, the image composition unit 205 adds pixel values of pixels having the same coordinates between the visible image and the mask image M, and synthesizes the object image G2 with the visible image. . The image composition unit 205 stores, for example, a visible image corresponding to the mask image M until the mask image M is input from the object extraction unit 209 to the image composition unit 205, and the mask image M and the mask image M are stored. Synchronize the visible image corresponding to.

  In this embodiment, the image composition unit 205 includes, for example, a luminance composition unit 205a and a chromaticity correction unit 205b. In the image composition based on the first image and the second image, the image composition unit 205 performs first brightness data representing the brightness included in the first image data D1 and second representing the brightness included in the second image data D2. The luminance data is synthesized by the luminance synthesis unit 205a. In the present embodiment, the luminance data is, for example, a luminance value calculated in the following (Expression 1), and the luminance synthesis unit 205a performs, for example, the first luminance of the synthesis target image G1 in the first image for each corresponding pixel. The composite luminance value is calculated by adding the second luminance value of the object image G2 extracted from the second image to the value. The image composition unit 205 converts the chromaticity data representing the chromaticity included in the first imaging data D1 into the color according to the composition of the second brightness data of the object image G2 and the first brightness data of the composition target image G1. Correction is performed by the degree correction unit 205b. In the present embodiment, the chromaticity data is, for example, the chromaticity value calculated in the following (Expression 1), and the chromaticity correction unit 205b performs synthesis in the first image, which is a visible image, for each corresponding pixel. The first chromaticity value of the target image G1 is converted (corrected) into a second chromaticity value corresponding to a change rate of the composite luminance value with respect to the first luminance value. More specifically, the chromaticity correction unit 205b obtains the second chromaticity value by an arithmetic expression of “second chromaticity value = first chromaticity value × combined luminance value / first luminance value”, for example.

  The output unit 300 is a display device that outputs the image synthesized by the image synthesis unit 205. The display device is, for example, an LCD (Liquid Crystal Display) or the like, and is disposed, for example, on a dashboard in front of a driver's seat of a car, in a monitoring room, or the like. For example, the display device is a head mounted display. The input unit 400 is an input device such as a dip switch or a push switch for inputting information to the imaging apparatus 1000.

  FIG. 3 is an operation flowchart of the imaging apparatus 1000 according to the first embodiment. FIG. 4 is a diagram for explaining a visible image, a near-infrared image, and a composite image according to the first embodiment. First, when light from a predetermined subject (object) enters the imaging unit 100, the optical system 101 forms an optical image of the subject on the light receiving surface of the image sensor 105 via the optical filter group 103. . Then, the first photoelectric conversion element L1 captures an optical image of the subject with visible light and outputs first imaging data D1, and the second photoelectric conversion element L2 captures an optical image of the subject with near-infrared light. The second imaging data D2 is output. In the present embodiment, since the optical filter group 103 is in the RGBIr mode, red light, green light, and blue light signals (analog outputs) are output from the plurality of first photoelectric conversion elements L1 of the image sensor 105 as the first imaging data D1. Each is output and A / D converted (analog-digital converted). A luminance value (first luminance value) Y and chromaticity values (first chromaticity values) Cb and Cr are expressed by the following (formulas) using the red value R, the green value G, and the blue value B, which are the respective digital values. Conversion is performed using the conversion formula represented by 1) (step S1).

  When the optical filter group 103 is in the WYRIr mode, the white light, yellow light, and red light signals (analog output) are A in the conversion formulas expressed by the following (Expression 2) to (Expression 4). The red value R, the green value G, and the blue value B are calculated using the digital values w, y, and r that have been subjected to the / D conversion, and then the first luminance value and the first color are calculated using (Equation 1). A degree value is calculated.

R = r-ir (Formula 2)
G = y−r (Formula 3)
B = wy (Formula 4)
Note that ir is a digital value obtained by A / D converting a detection signal (analog output) of infrared light.

  Next, the target object detection unit 201 detects the target object T from the near-infrared image, and the target object image region extraction unit 203 uses the predetermined constant value to determine a region excluding the target object image G2 from the near-infrared image. A masked mask image M is generated. Thereafter, the image composition unit 205 composites the mask image M with the visible image. In this image synthesis, the synthesized brightness value is calculated in the brightness synthesizer 205a (step S3). That is, the luminance composition unit 205a adds the luminance value of the synthesis target image G1 and the luminance value of the target image G2 (by synthesizing the first luminance data and the second luminance data), thereby synthesizing. A luminance value is calculated. For example, the combined luminance value Y ′ is calculated by the equation “Y ′ = R + G + B + IR”. The combined luminance value Y ′ may be calculated by the equation “Y ′ = Y + IR” after the luminance value (first luminance value) Y is calculated. Here, IR is a digital value obtained by A / D-converting a near-infrared light signal (analog output) output from the image sensor 105, and more specifically, a luminance value in the mask image M. When the optical filter group 103 is in the WYRIr mode, the combined luminance value Y ′ is calculated by the equation “Y ′ = w + y + r + ir”. Further, in each calculation formula for the combined luminance value Y ′, weighting may be performed for addition of digital values (IR, ir) of near infrared light. In this case, for example, when the optical filter group 103 is in the RGBIr mode, the combined luminance value Y ′ is calculated by the expression “Y ′ = (1−t) (R + G + B) + t · IR”. When 103 is a WYRIr mode, it may be calculated by the equation “Y ′ = (1−t) (w + y + r) + t · ir” (0 <t <1).

  Next, the chromaticity correction unit 205b corrects the chromaticity as the luminance of the combined image is converted from the first luminance value to the combined luminance value (step S5). In step S5, the second chromaticity is calculated using the formula “second chromaticity value = first chromaticity value × combined luminance value / first luminance value”. Specifically, if the second chromaticity values are Cb ′ and Cr ′, the second chromaticity is expressed by the formulas “Cb ′ = Cb × Y ′ / Y” and “Cr ′ = Cr × Y ′ / Y”. Calculated.

  Next, the image composition unit 205 converts the luminance value Y ′, the chromaticity values Cb ′, and Cr ′ into the red value R ′, the green value G ′, and the conversion expression represented by (Equation 5) below. Conversion to a blue value B ′ (step S7).

  Thereafter, the image composition unit 205 performs gamma correction according to the display characteristics of the display device (output unit 300) for each of the red value R ′, the green value G ′, and the blue value B ′ (step S9). The red value, the green value, and the blue value after the gamma correction are output to the output unit 300.

  The output unit 300 serving as a display device receives the input of the red value, the green value, and the blue value after the gamma correction, and displays a composite image (step S11).

  FIG. 4 shows a first image (FIGS. 4A and 4D) which is a visible image based on the first imaging data D1, and a near red color based on the second imaging data D2 including the target T (broken line rectangular portion (second image region)). The second image (FIGS. 4B and 4E), which is an outer image, and the image (FIGS. 4C and 4F) in which the first image and the mask image M are combined in the image combining unit 205 are illustrated. 4A to 4C are actual captured images, and FIGS. 4D to 4F are schematic diagrams in which characteristic portions in the captured images of FIGS. 4A to 4C are extracted. FIG. 4A is a visible image obtained by imaging light in the first wavelength band, and therefore is an image in which the road and bushes on one side of the road are imaged in the dark. The part that is illuminated is imaged. In FIG. 4D, a portion illuminated by the road, the bush, and the illumination, which is a characteristic portion of the captured image of FIG. 4A, is represented by a solid line. In FIG. 4D, hatching indicating darkness is added. FIG. 4B is a near-infrared image obtained by imaging light in the second wavelength band, and thus is an image in which the road and the bushes on both sides of the road are dimly captured. Human and animal types are imaged with good brightness. In FIG. 4E, the road, the bush, and the human and animal types, which are characteristic portions of the captured image of FIG. 4B, are represented by solid lines. Note that the dim points in FIG. 4B are not represented in FIG. 4E. FIG. 4C is an image obtained by combining FIG. 4A and a mask image M (not shown) obtained by masking the region excluding the broken-line rectangular portion of FIG. 4B with the predetermined constant value. Human and animal patterns are projected with high brightness in an image in which a bush on the side of the road and a portion illuminated by light near the center of the image appear. FIG. 4F shows the characteristic portions of the captured image of FIG. 4C, that is, the road, the bush, the portion illuminated by the illumination, and the types of the person and the animal by solid lines. In FIG. 4F, hatching indicating darkness is added.

  4C and FIG. 4F displays the object T, which has poor visibility in the visible images of FIG. 4A and FIG. 4D, with higher visibility (the broken-line rectangular portion in FIG. 4C and FIG. 4F (first 1 image area)).

  Next, another embodiment will be described.

[Second Embodiment]
FIG. 5 is a block diagram illustrating an example of a configuration of the imaging apparatus 1000a according to the second embodiment. FIG. 6 is a diagram for explaining a visible image, a near-infrared image, and a composite image according to the second embodiment. In FIG. 5, the imaging apparatus 1000 a according to the second embodiment is different from the imaging apparatus 1000 according to the first embodiment described above in that, for example, the imaging unit 100 a is used instead of the imaging unit 100, and the image processing is performed instead of the image processing unit 200. Part 200a. Hereinafter, the following description will be focused on differences from the first embodiment. In addition, in each structure of 2nd Embodiment, the same thing as the structure of 1st Embodiment is attached | subjected the same code | symbol, and the description is abbreviate | omitted suitably unless there is a special reason.

  In the present embodiment, the imaging unit 100a has two separate image sensors. Accordingly, the imaging unit 100a includes, for example, an optical system 101a, an optical system 101b, an optical filter group 103a, a first image sensor 105a, and a second image sensor 105b. Light from the predetermined subject enters the optical filter group 103a through the optical system 101a, and only the light in the predetermined wavelength range is transmitted through the optical filter group 103a and enters the first image sensor 105a. Light from the predetermined subject enters the second image sensor 105b through the optical system 101b.

  The optical system 101a is the same as the optical system 101. The optical system 101b is, for example, an infrared lens. In this embodiment, the optical filter group 103a is, for example, a so-called RGGB type optical filter group having a Bayer array. The optical filter group 103a may be, for example, a so-called WYR type optical filter group illustrated in FIG. 2C. The WYR type optical filter group includes, for example, a white (W) filter at a position (1, 1), a yellow (Y) filter at a position (2, 1), and a red (R) filter (1, 2). And a basic pattern (rectangular portions indicated by C in the figure) in which white (W) filters are respectively arranged at positions (2, 2) are arranged in a matrix in the two directions of the x direction and the y direction. That's it. The optical filter group 103a may be, for example, a so-called CMYG type complementary color filter group shown in FIG. 2D. C indicates cyan, M indicates magenta, Y indicates yellow, and G indicates green. The CMYG type optical filter group has, for example, a cyan (C) filter that transmits cyan light at a position (1, 1) and a magenta (M) filter that transmits magenta light at a position (2, 1). , A basic pattern (rectangular portion indicated by D in the figure) in which the yellow (Y) filter is disposed at the position (1, 2) and the green (G) filter is disposed at the position (2, 2), respectively, in the x direction And arranged in a matrix in two directions of the y direction.

  The plurality of photoelectric conversion elements included in the first image sensor 105a receive light in the wavelength region included in the first wavelength band via the optical filter group 103a, thereby imaging the light in the wavelength region and performing first imaging. The plurality of first photoelectric conversion elements L1 that output the imaging data D1. The first image sensor 105a is, for example, an image sensor similar to the image sensor 105, in which the photoelectric conversion characteristic is a linear characteristic or a linear logarithmic characteristic. The plurality of photoelectric conversion elements included in the second image sensor 105b receive light in a second wavelength band that includes a wavelength band different from the first wavelength band via the infrared lens, so that the second wavelength A plurality of second photoelectric conversion elements L2 that pick up light of a band and output second imaging data D2. In the present embodiment, the first image data 105 from the first image sensor 105a and the second image data D2 from the second image sensor 105b are output to the control calculation unit 211, respectively. In the present embodiment, the light in the wavelength region included in the first wavelength band is visible light, and the light in the second wavelength band is far infrared light. The plurality of first photoelectric conversion elements L1 captures light (red, green, and blue) in a plurality of wavelength regions included in the first wavelength band, and outputs first imaging data D1. The second image sensor 105b is a so-called quantum type or thermal type far infrared image sensor. The first image sensor 105a and the second image sensor 105b have substantially the same object to be imaged. For example, the first image sensor 105a and the second image sensor 105b are fixed with a predetermined distance (baseline length) at an interval so that the optical axes are parallel to each other. Is done. In the present embodiment, the first image based on the first imaging data D1 of the first image sensor 105a and the second image based on the second imaging data D2 of the second image sensor 105b have the same size, the same shape, and the same pixel. Is a number.

  The image processing unit 200a includes an object extraction unit 209a instead of the object extraction unit 209. The object extraction unit 209a further includes a parallax correction unit 207 with respect to the object extraction unit 209. That is, the object extraction unit 209a includes, for example, the object detection unit 201, the object image region extraction unit 203, and the parallax correction unit 207, and the output of the object image region extraction unit 203 is input to the parallax correction unit 207. The output of the parallax correction unit 207 is input to the image composition unit 205. In this case, the control arithmetic unit 211 further functionally configures the parallax correction unit 207 by executing an image processing program corresponding to the parallax correction stored in the ROM.

  The parallax correction unit 207 has a visible image (first image) generated by the first imaging data D1 generated because the second image sensor 105b and the first image sensor 105b are separated and fixed by the base line length as in a so-called stereo camera. The parallax between the 1 image) and the far-infrared image (2nd image) by the 2nd imaging data D2 is correct | amended by a well-known conventional means. Note that this parallax correction may be executed, for example, by shifting a far-infrared image in the horizontal direction within the image. This shift amount is appropriately set in advance according to the baseline length. The parallax correction unit 207 shifts the object image G2 extracted by the object image region extraction unit 203 in the horizontal direction in the mask image M by the shift amount, blackens the part other than the object image G2, and horizontally A mask image M ′ after correcting the deviation is generated, and the mask image M ′ is output to the image composition unit 205 instead of the mask image M.

  Similar to FIG. 4 of the first embodiment, the imaging apparatus 1000a according to the present embodiment displays the target T with poor visibility in the visible image as shown in FIG.

  More specifically, FIG. 6 includes a first image (FIGS. 6A and 6D) that is a visible image based on the first imaging data D1, and a second object T (a broken-line rectangular portion (second image region)). Examples include a second image (FIGS. 6B and 6E) that is a far-infrared image based on the imaging data D2, and an image (FIGS. 6C and 6F) in which the first image and the mask image M are combined in the image combining unit 205. Has been. 6A to 6C are actual captured images, and FIGS. 6D to 6F are schematic diagrams in which characteristic portions in the captured images of FIGS. 6A to 6C are extracted. Since FIG. 6A is a visible image obtained by imaging light in the first wavelength band, it is an image in which the road and bushes on one side of the road are imaged in the dark. The part that is illuminated is imaged. FIG. 6D shows a solid line indicating a portion of the captured image of FIG. 6A that is illuminated by the road, the bush, and the illumination. In FIG. 6D, hatching indicating darkness is added. FIG. 6B is a far-infrared image obtained by imaging light in the second wavelength band, and thus is an image in which the road and the bushes on both sides of the road are dimly captured. Human and animal types are imaged with good brightness. In FIG. 6E, the road, the bush, and the human and animal types, which are characteristic portions of the captured image of FIG. 6B, are represented by solid lines. Note that the dim points in FIG. 6B are not represented in FIG. 6E. As can be seen by comparing FIG. 6A and FIG. 6B, the first image shown in FIG. 6A and the second image shown in FIG. 6B are shifted by a parallax corresponding to the baseline length. FIG. 6C is an image obtained by combining FIG. 6A and a mask image M (not shown) obtained by masking the region excluding the broken-line rectangular portion of FIG. 6B with the predetermined constant value. Human and animal patterns are projected with high brightness in an image in which a bush on the side of the road and a portion illuminated by light near the center of the image appear. FIG. 6F shows the characteristic portions of the captured image of FIG. 6C, that is, the road, the bush, the portion that is illuminated, and the human and animal types in solid lines. In FIG. 6F, hatching indicating darkness is added.

  As described above, in the synthesized images of FIGS. 6C and 6F, the object T that is poorly visible in the visible images of FIGS. 6A and 6D is displayed with higher visibility (the broken-line rectangles in FIGS. 6C and 6F). Part (first image area)).

  Next, another embodiment will be described.

[Third Embodiment]
FIG. 7 is a block diagram illustrating an example of a configuration of an imaging apparatus 1000b according to the third embodiment. In FIG. 7, the imaging apparatus 1000b according to the third embodiment is different from the imaging apparatus 1000 according to the first embodiment in, for example, an imaging unit 100b instead of the imaging unit 100 and an image processing unit 200b instead of the image processing unit 200. Is provided. Hereinafter, the following description will be focused on differences from the first embodiment or the second embodiment. In addition, in each structure of 3rd Embodiment, the same thing as the structure of 1st Embodiment or 2nd Embodiment is attached | subjected the same code | symbol, and the description is abbreviate | omitted suitably unless there is a special reason.

  The imaging unit 100b includes, for example, an optical system 101c, an optical system 101d, an optical filter group 103c, a third image sensor 105c, and a fourth image sensor 105d. In the present embodiment, the imaging unit 100b has two separate image sensors. Light from the predetermined subject enters the optical filter group 103c through the optical system 101c, and only light in a predetermined wavelength range is transmitted through the optical filter group 103c and enters the third image sensor 105c. Further, light from the predetermined subject enters the fourth image sensor 105d through the optical system 101d.

  The optical system 101c is an imaging optical system having a near infrared lens. The optical system 101d is an imaging optical system having a far infrared lens. The optical filter group 103c is a near infrared filter.

  The plurality of photoelectric conversion elements included in the third image sensor 105c receive the light in the wavelength region included in the first wavelength band via the optical filter group 103c, thereby imaging the light in the wavelength region and performing first imaging. The plurality of first photoelectric conversion elements L1 that output the imaging data D1. The third image sensor 105c is, for example, an image sensor whose photoelectric conversion characteristics are linear characteristics or linear logarithmic characteristics. The plurality of photoelectric conversion elements included in the fourth image sensor 105d receive the second wavelength band including a wavelength band different from the first wavelength band via the far-infrared lens, whereby the second A plurality of second photoelectric conversion elements L2 that pick up light in the wavelength band and output second image data D2. In the present embodiment, the first image data D1 from the third image sensor 105c and the second image data D2 from the fourth image sensor 105d are output to the control calculation unit 211, respectively. In the present embodiment, for example, the light in the wavelength region included in the first wavelength band is near infrared light, and the light in the second wavelength band is far infrared light. In this case, the fourth image sensor 105d is a so-called quantum type or thermal type far infrared image sensor. As in the second embodiment, the third image sensor 105c and the fourth image sensor 105d have substantially the same imaging target. For example, the third image sensor 105c and the fourth image sensor 105d have a predetermined distance (the base length) so that the optical axes are parallel to each other. ) Are fixed at intervals. In this embodiment, the first image and the second image based on the first image data D1 of the third image sensor 105c and the second image data D2 of the fourth image sensor 105d have the same size, the same shape, and the same number of pixels. is there.

  The image processing unit 200b includes, for example, an image composition unit 205d instead of the image composition unit 205, and the object extraction unit 209a instead of the object extraction unit 209. The image composition unit 205d extracts the object image G2 from the second image that is a far-infrared image based on the second imaging data D2, and applies the mask image M to the first image that is a near-infrared image based on the first imaging data D1. Perform image processing to synthesize '.

  The image composition unit 205d includes a luminance composition unit 205a. More specifically, the luminance composition unit 205a includes third luminance data representing the luminance included in the first imaging data D1 (specifically, in this embodiment, the pixel value of the near-infrared image, that is, the luminance value), and the first luminance data. The fourth luminance data representing the luminance included in the two imaging data D2 (specifically, in this embodiment, the pixel value of the far-infrared image, that is, the luminance value) is combined to obtain a combined luminance value. Since the near-infrared image and the far-infrared image do not have chromaticity information, the image composition unit 205d does not have the chromaticity correction unit 205b.

  In the present embodiment, in the imaging unit 100b, the plurality of first photoelectric conversion elements L1 and the plurality of second photoelectric conversion elements L2 may exist on one image sensor. In this case, the imaging unit 100b includes a plurality of the near-infrared filters arranged in a matrix with a predetermined arrangement and an optical filter group having a far-infrared filter that transmits far-infrared light. Among the plurality of photoelectric conversion elements included in the one image sensor, the plurality of photoelectric conversion elements that receive light through the plurality of near-infrared filters are near-infrared light in a wavelength region included in the first wavelength band. Are a plurality of first photoelectric conversion elements L1 that output first imaging data D1. Among the plurality of photoelectric conversion elements included in the image sensor, a plurality of photoelectric conversion elements that receive light through the plurality of far-infrared filters include a second wavelength band that is different from the first wavelength band. A plurality of second photoelectric conversion elements L2 that image far-infrared light in the wavelength band and output second imaging data D2.

  Although the imaging apparatus 1000b according to the present embodiment is not specifically illustrated, the object T having poor visibility in the near-infrared image is similar to FIG. 4 of the first embodiment and FIG. 6 of the second embodiment. It is displayed with better visibility.

  In the imaging apparatus and imaging method described in each of the above embodiments, the object image G2 extracted from the second image based on the second imaging data D2 is added to the synthesis target image G1 in the first image based on the first imaging data D1. Synthesize. For this reason, since the image quality that is insufficient in one image can be supplemented by another image, the visibility of the object can be further improved.

  Note that in the first and second embodiments, the imaging units 100 and 100a use a near-infrared light projector to capture near-infrared light more clearly, and aim at an imaging target at night. Near infrared light may be irradiated.

  Further, the output from the object image area extraction unit 203 is not limited to the mask image M, and may be an output of the object image G2 and the area information, for example. In this case, the image composition unit 205 has a rectangular second shape at the same coordinates as the vertex coordinates in the first image based on the first imaging data D1, based on the vertex coordinates of the object image G2 as the region information. A rectangular image portion having the same dimensions as the image region r2 (object image G2) is used as a first image region r1 (composition target image G1), and the object image G2 is image-synthesized with the composition target image G1.

  The present specification discloses various aspects of the technology as described above, and the main technologies are summarized below.

  An imaging apparatus according to an aspect includes a plurality of first photoelectric conversion elements that capture light in a wavelength region included in a first wavelength band and output first imaging data, and a wavelength band different from the first wavelength band. Synthesis based on a plurality of second photoelectric conversion elements that capture light in the second wavelength band and output second imaging data, a first image based on the first imaging data, and a second image based on the second imaging data An image composition unit for generating an image, wherein the image composition unit converts an object image of a second image region including a predetermined type of object of the second image into the object of the first image. The image is combined with the image to be combined in the first image area corresponding to the second image area.

  In the above-described imaging device, the imaging device further includes an object detection unit that detects an object of a predetermined type in the second image based on the second imaging data, and the image synthesis unit is detected by the object detection unit. Preferably, the object image in the second image region including the target object is combined with the composition target image in the first image region corresponding to the second image region in the first image. .

  An imaging method according to another aspect includes a first imaging step of acquiring first imaging data by imaging light in a wavelength region included in a first wavelength band by a plurality of first photoelectric conversion elements; and the first wavelength A second imaging step in which light of a second wavelength band including a wavelength band different from the band is captured by a plurality of second photoelectric conversion elements to obtain second imaging data; a first image based on the first imaging data; An image composition step for generating a composite image based on the second image based on the two imaging data, wherein the image composition step includes a target of a second image region including a predetermined type of object of the second image. The object image is synthesized with the synthesis target image in the first image area corresponding to the second image area in the first image.

  In such an imaging apparatus and imaging method, the object image in the second image based on the second imaging data is synthesized with the synthesis target image in the first image based on the first imaging data. For this reason, the image pickup apparatus and the image pickup method can supplement the image quality that is insufficient in one image with another image, and thus can further improve the visibility of an object.

  In another aspect, in the imaging apparatus, it is preferable that the second wavelength band is a band having a longer wavelength than the first wavelength band. In such an imaging apparatus, the object can be accurately extracted from the second imaging data.

  In another aspect, in the above-described imaging devices, the first imaging data includes first luminance data representing luminance, the second imaging data includes second luminance data representing luminance, and the image synthesis The unit preferably synthesizes the second luminance data of the object image and the first luminance data of the composition target image.

  According to this configuration, the first luminance data included in the first imaging data and the second luminance data included in the second imaging data are synthesized. For this reason, such an imaging device can display the target object brightly in the combined image, and the visibility of the target object is further improved.

  In another aspect, in the above-described imaging device, the light in the first wavelength band is visible light, the first imaging data includes chromaticity data representing chromaticity, and the light in the second wavelength band. Is infrared light, and the image synthesizing unit obtains chromaticity data of the first imaging data in accordance with the synthesis of the second luminance data of the object image and the first luminance data of the synthesis target image. It is preferable to correct.

  According to this configuration, the first luminance data and the second luminance data are combined, and the chromaticity value is further corrected. For this reason, such an imaging device can suppress a change in hue before and after synthesis.

  In another aspect, in the above-described imaging devices, the light in the first wavelength band is near infrared light, the first imaging data includes third luminance data representing luminance, and the second wavelength. The band light is far-infrared light, the second imaging data includes fourth luminance data representing luminance, and the image synthesis unit includes fourth luminance data of the object image and the synthesis target image. It is preferable to synthesize the third luminance data.

  According to this configuration, the third luminance data of the synthesis target image based on the first imaging data and the fourth luminance data of the target image based on the second imaging data are synthesized. For this reason, since such an imaging device can supplement image quality that is insufficient in one image with another image, the visibility of the object can be further improved.

  In another aspect, it is preferable that the above-described imaging devices include one image sensor in which the plurality of first photoelectric conversion elements and the plurality of second photoelectric conversion elements are formed.

  According to this configuration, the plurality of first photoelectric conversion elements and the plurality of second photoelectric conversion elements are formed in the one image sensor. For this reason, such an imaging device can be miniaturized.

  In another aspect, in the above-described imaging devices, the first image sensor in which the plurality of first photoelectric conversion elements are formed and the first image sensor in which the plurality of second photoelectric conversion elements are formed are separate from each other. The second image sensor is preferably provided.

  According to this configuration, separate first image sensor and second image sensor are used. For this reason, such an imaging apparatus can use an image sensor suitable for each wavelength band, and can detect light in each wavelength band more appropriately.

  This application is based on Japanese Patent Application No. 2013-93535 filed on Apr. 26, 2013, the contents of which are included in this application.

  In order to express the present invention, the present invention has been properly and fully described through the embodiments with reference to the drawings. However, those skilled in the art can easily change and / or improve the above-described embodiments. It should be recognized that this is possible. Therefore, unless the modifications or improvements implemented by those skilled in the art are at a level that departs from the scope of the claims recited in the claims, the modifications or improvements are not covered by the claims. To be construed as inclusive.

  According to the present invention, an imaging device and an imaging method can be provided.

Claims (9)

A plurality of first photoelectric conversion elements for imaging light in a wavelength region included in the first wavelength band and outputting first imaging data;
A plurality of second photoelectric conversion elements for imaging light of a second wavelength band including a wavelength band different from the first wavelength band and outputting second imaging data;
An image composition unit that generates a composite image based on the first image based on the first image data and the second image based on the second image data;
The image synthesizing unit converts a target image in a second image region including a predetermined type of target in the second image into a first image region corresponding to the second image region in the first image. To be combined with the target image of
Imaging device. The second wavelength band is a wavelength band longer than the first wavelength band.
The imaging device according to claim 1. The first imaging data includes first luminance data representing luminance,
The second imaging data includes second luminance data representing luminance,
The image synthesis unit synthesizes the second luminance data of the object image and the first luminance data of the synthesis target image;
The imaging device according to claim 1 or 2. The light in the first wavelength band is visible light, and the first imaging data includes chromaticity data representing chromaticity,
The light in the second wavelength band is infrared light,
The image composition unit corrects chromaticity data of the first imaging data in accordance with the composition of the second luminance data of the object image and the first luminance data of the composition target image.
The imaging device according to claim 3. The light in the first wavelength band is near-infrared light, and the first imaging data includes third luminance data representing luminance,
The light in the second wavelength band is far-infrared light, and the second imaging data includes fourth luminance data representing luminance,
The image composition unit synthesizes the fourth luminance data of the object image and the third luminance data of the composition target image;
The imaging device according to claim 1 or 2. An object detection unit for detecting a predetermined type of object in the second image based on the second imaging data;
The image compositing unit corresponds to the first image region corresponding to the second image region of the first image, the object image of the second image region including the target object detected by the target object detection unit. Synthesizing the image to be synthesized in the image area
The imaging device according to any one of claims 1 to 5. Comprising one image sensor formed with the plurality of first photoelectric conversion elements and the plurality of second photoelectric conversion elements;
The imaging device according to any one of claims 1 to 6. A first image sensor formed with the plurality of first photoelectric conversion elements;
A second image sensor separate from the first image sensor in which the plurality of second photoelectric conversion elements are formed,
The imaging device according to any one of claims 1 to 6. A first imaging step of capturing light of a wavelength region included in the first wavelength band with a plurality of first photoelectric conversion elements to obtain first imaging data;
A second imaging step of acquiring second imaging data by imaging light of a second wavelength band including a wavelength band different from the first wavelength band by a plurality of second photoelectric conversion elements;
An image synthesis step of generating a synthesized image based on the first image based on the first imaging data and the second image based on the second imaging data;
In the image composition step, an object image in a second image area including a predetermined type of object in the second image is converted into a first image area corresponding to the second image area in the first image. To be combined with the target image of
Imaging method.

Images