JP2018107731A - Image generating apparatus and imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image generating apparatus capable of generating a suitable image while suppressing deterioration of resolution.SOLUTION: An imaging system as an image generating apparatus, comprises: a random color filter array 202 including a plural kinds of color filters 202a to 202c; a photo diode 203a that receives a light permitting the random color filter array 202; an AD conversion part that converts the light received in the photo diode 203a into digital data; and a color image generation circuit 105 that generates an image by using the digital data and modulation information of the random color filter array 202. At least two color filters in the random color filter array 202 are positioned in front of the photo diode 203a. The color filter that is the smallest in the random color filter array 202 is smaller than the photo diode 203a.SELECTED DRAWING: Figure 4

Description

  The present disclosure relates to an image generation apparatus and the like using a compressed sensing technique.

  In order to capture a color image, it is necessary to acquire information of three different wavelength ranges of red (R), green (G), and blue (B) corresponding to the three primary colors of light. There is also a color imaging device that acquires R, G, and B information with three image sensors. However, many color imaging devices have only one image sensor for miniaturization and cost reduction. For this reason, many color imaging devices acquire R, G, and B information with a single image sensor.

  Conventionally, information on one wavelength region of R, G, and B is acquired for each pixel, and information on three wavelength regions of R, G, and B is acquired for each pixel by performing a process called demosaicing. Techniques are known.

  FIG. 32 is a schematic diagram showing a widely used Bayer array (for example, Patent Document 1). In the Bayer array, G pixels close to human visual characteristics occupy 1/2 of the total pixels, and R and B pixels occupy 1/4 of the total pixels. Then, information on the three wavelength ranges of R, G, and B is acquired for all the pixels by the demosaicing process.

  On the other hand, Patent Document 2 discloses a technique for performing demosaicing by arranging optical filter elements in a random color pattern and applying a compressed sensing technique to a sample data set.

US Pat. No. 5,629,734 Special table 2013-511924 gazette

Rudin L. I. Osher S .; J. et al. , And Fatemi E .; Nonlinear total variation based noise removal algorithms. Physica D, vol. 60, pp. 259-268, 1992. Shuunsuke Ono, Isao Yamada, “Decorrelated Vector Total Variation”, IEEE Conferencing on Computer Vision and Pattern Recognition, 2014. J. et al. Ma, “Improved Iterative Curved Thresholding for Compressed Sensing and Measurement”, IEEE Transactions on Measurement & Measurement, vol. 60, no. 1, pp. 126-136, 2011. M.M. Aharon, M.M. Elad, and A.A. M.M. Brookstein, “K-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation,” IEEE Transactions on Image Processing, Vol. 54, no. 11, pp. 4311-4322, 2006. D. Kiku, Y. et al. Monno, M.M. Tanaka and M.M. Okutomi, “Minimized-Laplacian residual interpolation for color image demosking”, IS & T / SPIE Electronic Imaging (EI), 2014. Manya V. Afonso, Jose M .; Biocas-Dias, and Mario A. et al. T.A. Figueiredo, “Fast Image Recovery Using Variable Splitting and Constrained Optimization”, IEEE Transactions on Image Processing, VOL. 19, NO. 9, pp. 2345-2356, 2010.

  However, in the methods of Patent Documents 1 and 2, only information on one wavelength region among R, G, and B is acquired in each pixel of the image sensor. For this reason, the resolution of the color image after demosaicing is lowered, and an artifact called false color may occur.

  The present disclosure provides, as an exemplary non-limiting example, an image generation apparatus capable of generating an appropriate image while suppressing a decrease in resolution. The present disclosure also provides an imaging device used to generate an appropriate image. Additional benefits and advantages of one aspect of the present disclosure will become apparent from the specification and drawings. This benefit and / or advantage may be provided individually by the various aspects and features disclosed in the specification and drawings, and not all are required to obtain one or more thereof.

  An image generation apparatus according to an aspect of the present disclosure includes a random optical filter array having a plurality of types of optical filters, wherein the plurality of types of optical filters are randomly arranged and transmitted through the random optical filter array A generation circuit that generates an image using a photodiode that receives light, an AD conversion unit that converts light received by the photodiode into digital data, the digital data, and modulation information of the random optical filter array Wherein at least two optical filters in the random optical filter array are positioned in front of the photodiode, and the optical filter having the smallest size in the random optical filter array is smaller than the photodiode.

  An imaging apparatus according to an aspect of the present disclosure includes a random optical filter array having a plurality of types of optical filters, wherein the plurality of types of optical filters are randomly arranged, and passes through the random optical filter array. A photodiode that receives the received light, and an AD converter that converts the light received by the photodiode into digital data, wherein at least two optical filters in the random optical filter array are located in front of the photodiode. The optical filter having the smallest size in the random optical filter array is smaller than the photodiode.

  The general and specific aspects described above may be implemented using a system, method and computer program, or may be implemented using a combination of system, method and computer program.

  According to the image generation device or the like according to an aspect of the present disclosure, a reduction in resolution can be suppressed and an appropriate image can be generated.

FIG. 1 is a schematic diagram illustrating a configuration of an imaging system according to the first embodiment. FIG. 2 is a diagram illustrating details of the configuration of the wavelength modulation unit and the modulated image acquisition unit in the imaging system. FIG. 3 is a schematic diagram illustrating a configuration of a random color filter array in the imaging system. FIG. 4 is a schematic diagram of an imaging system including a random color filter array. FIG. 5 is a diagram showing the transmittance as the wavelength characteristic of the three types of filters according to the embodiment. FIG. 6 is a schematic diagram showing the arrangement on a two-dimensional plane of a random color filter array composed of three types of filters and an image sensor. FIG. 7 is a schematic diagram showing a cross section of a random color filter array constituted by three types of filters and an image sensor. FIG. 8 is a diagram showing the transmissivity as 28 wavelength characteristics realized by the random color filter array shown in FIG. FIG. 9 is a diagram showing transmittance as wavelength characteristics of the four types of filters according to the embodiment. FIG. 10 is a schematic diagram showing the arrangement on a two-dimensional plane of a random color filter array composed of four types of filters and an image sensor. FIG. 11 is a schematic diagram showing a cross section of a random color filter array composed of four types of filters and an image sensor. FIG. 12 is a diagram showing the transmittance as 84 wavelength characteristics realized by the random color filter array shown in FIG. FIG. 13 is a schematic diagram showing the arrangement on the two-dimensional plane of the random color filter array and the imaging device in a form in which light transmitted through nine of the three types of filters is received by one photodiode 203a. FIG. 14 is a diagram showing transmittances as 55 wavelength characteristics realized by the random color filter array shown in FIG. FIG. 15 is a schematic diagram showing an arrangement on a two-dimensional plane of a random color filter array and an image sensor in which light transmitted through nine of four types of filters is received by one photodiode 203a. FIG. 16 is a diagram showing transmittances as 220 wavelength characteristics realized by the random color filter array shown in FIG. FIG. 17 is a diagram showing transmittance as wavelength characteristics of three types of complementary color filters. FIG. 18 is a flowchart illustrating a main processing procedure of the image generation apparatus in the imaging system according to the first embodiment. FIG. 19 is a schematic diagram illustrating a modulated image and a generated image when the number of pixels of the image sensor is N = 16. FIG. 20 is a diagram illustrating R images generated by various methods. FIG. 21 is a diagram showing G images generated by various methods. FIG. 22 is a diagram showing B images generated by various methods. FIG. 23 is a diagram showing color images generated by various methods. FIG. 24 is a diagram illustrating G images generated by various methods. FIG. 25 is an example of an image obtained by enlarging a partial region of the images in FIGS. FIG. 26 is a schematic diagram showing an example of an arrangement in which the phase of the random color filter array and the arrangement of the photodiodes are shifted. FIG. 27A is a schematic diagram illustrating an example of a random color filter array including color filters that overlap with a plurality of photodiodes in an arrangement projected onto a two-dimensional plane. FIG. 27B is a schematic diagram showing an example of a random color filter array different from FIG. 27A. FIG. 28 is a schematic diagram illustrating a configuration of an imaging system according to the second embodiment. FIG. 29 is a flowchart illustrating a main processing procedure of the image generation apparatus in the imaging system according to the second embodiment. FIG. 30 is a schematic diagram illustrating a generated image (multiband image) when the number of pixels of the image sensor is N = 16. FIG. 31 is a schematic diagram showing an arrangement on a two-dimensional plane of a random color filter array composed of three types of filters and an image sensor. FIG. 32 is a schematic diagram showing a Bayer array.

(Knowledge that forms the basis of this disclosure)
In the techniques described in Patent Documents 1 and 2, only one information of R (red), G (green), and B (blue) is acquired in one pixel. For this reason, the acquired information is not always sufficient, and an appropriate image with high resolution may not be generated.

  On the other hand, for example, information of each wavelength band of R, G, and B is mixed and given to each pixel, and the mixing is performed randomly on the pixel group, so that more information can be obtained from each pixel. From the obtained information, an appropriate image can be generated by a compression sensing technique. For example, a small number of optical filters such as an R filter that mainly transmits the R wavelength band, a G filter that mainly transmits the G wavelength band, and a total of three types of color filters such as a B filter that mainly transmits the B wavelength band. It is useful to realize pseudo-random information mixing for the pixel group. For example, by installing a random optical filter array configured by randomly arranging relatively few types of optical filters on the optical path of light received by the imaging device, an appropriate image with high resolution can be obtained. Generation may be possible. Based on the above knowledge, an image generation apparatus and the like according to the present disclosure will be described below.

  An image generation apparatus according to an aspect of the present disclosure includes a random optical filter array having a plurality of types of optical filters, wherein the plurality of types of optical filters are randomly arranged and transmitted through the random optical filter array A generation circuit that generates an image using a photodiode that receives light, an AD conversion unit that converts light received by the photodiode into digital data, the digital data, and modulation information of the random optical filter array Wherein at least two optical filters in the random optical filter array are positioned in front of the photodiode, and the optical filter having the smallest size in the random optical filter array is smaller than the photodiode. Random here includes pseudo-randomness. Optical filters of the same type have the same wavelength characteristics related to the relationship between the wavelength of light and light transmittance, and optical filters of different types have wavelength characteristics related to the relationship between the wavelength of light and light transmittance. Is different. The plurality of optical filters are positioned in front of the photodiode, that is, in front of the light receiving surface that is the light receiving region of the photodiode, and are on the optical path from the light passing through the optical system member such as a lens to the light receiving surface. Be placed. The modulation information of the random optical filter array is information regarding the light transmittance in a random optical filter array in which a plurality of optical filters are arranged in a substantially plane, and the light transmittance is determined according to the position on the plane and the wavelength of light. Can change.

  As a result, the photodiode as each pixel has sufficient information through a random optical filter array in which a plurality of types of optical filters having different wavelength characteristics related to the relationship between the wavelength of light and the transmittance are arranged at random. Since it can be acquired, a decrease in resolution is suppressed and an appropriate image can be generated.

  Further, for example, the optical filter may be a color filter, the random optical filter array may be a random color filter array, and the image generated by the generation circuit may be a color image.

  Thereby, for example, light transmitted through a plurality of sets of color filters such as an R filter, a G filter, and a B filter is added by a photodiode, and an appropriate color image having a high resolution is obtained based on the modulation information by, for example, a compression sensing technique. Can be generated.

  For example, when viewed from the front of the photodiode, each of the at least two optical filters may overlap the photodiode at different positions.

  Thereby, in the arrangement projected onto the two-dimensional plane parallel to the light receiving surface of the photodiode, the photodiode can receive the light transmitted through two or more optical filters at each position superimposed on the photodiode. The amount of information of light acquired in (1) becomes larger than that of the conventional Bayer array. For this reason, an appropriate image with high resolution can be generated. In addition, since the two or more optical filters are arranged so as to overlap the photodiodes at positions different from each other rather than at completely overlapping positions, a decrease in the amount of light received by the photodiodes can be suppressed.

  Further, for example, the number of optical filters in the random optical filter array may be six times or more larger than the number of the photodiodes.

  As a result, a random optical filter array having practically sufficient randomness (randomness) can be realized, so that an appropriate image can be generated by compressed sensing.

  Further, for example, the plurality of types of optical filters include a first type of optical filter, a second type of optical filter, and a third type of optical filter, and the first type of optical filter, The two types of optical filters and the third type of optical filter may have different wavelength characteristics.

  Thus, a useful random optical filter array can be configured relatively easily using three types of optical filters such as an R filter, a G filter, and a B filter.

  In addition, for example, the photodiode receives light including a first wavelength band, a second wavelength band, and a third wavelength band, and the first type optical filter in the first wavelength band. Different from the light transmittance of the second type optical filter in the first wavelength band and the light transmittance of the third type optical filter in the first wavelength band. Light transmittance of the first type optical filter in the second wavelength band, light transmittance of the second type optical filter in the second wavelength band, and third type in the second wavelength band. Unlike the light transmittance of the optical filter, the light transmittance of the first type optical filter in the third wavelength band, and the light transmittance of the second type optical filter in the third wavelength band, In the third wavelength band It may differ from the three light transmittance of the optical filter.

  As a result, a useful random optical filter array can be configured relatively easily using three types of optical filters having different light transmittances in each wavelength band such as the wavelength bands of R, G, and B, for example.

  Further, for example, the image generated by the generation circuit may be a multiband image.

  Thereby, a multiband image with high resolution can be generated. For example, the random optical filter array may include an optical filter that transmits light in a wavelength band other than visible light.

  Further, for example, the generation circuit may generate the image using a compression sensing technique.

  By this compressed sensing, an image can be generated appropriately.

  An imaging apparatus according to an aspect of the present disclosure includes a random optical filter array having a plurality of types of optical filters, wherein the plurality of types of optical filters are randomly arranged, and passes through the random optical filter array. A photodiode that receives the received light, and an AD converter that converts the light received by the photodiode into digital data, wherein at least two optical filters in the random optical filter array are located in front of the photodiode. The optical filter having the smallest size in the random optical filter array is smaller than the photodiode.

  As a result, the photodiode as each pixel has sufficient information through a random optical filter array in which a plurality of types of optical filters having different wavelength characteristics related to the relationship between the wavelength of light and the transmittance are arranged at random. Be able to get. For this reason, an appropriate image having a relatively high resolution can be generated using the information of each pixel.

  These comprehensive or specific various aspects include one or a plurality of combinations of an apparatus, a system, a method, an integrated circuit, a computer program, a computer-readable recording medium, and the like.

  Hereinafter, embodiments of an imaging system according to an image generation device of the present disclosure will be described with reference to the drawings. Any of the embodiments shown here is merely an example. Therefore, numerical values, shapes, materials, components, arrangement and connection forms of components, and steps and order of steps shown in the following embodiments are merely examples, and are not limited. Among the constituent elements in the following embodiments, constituent elements that are not described in the independent claims can be arbitrarily added. Each figure is a mimetic diagram and is not necessarily illustrated strictly.

(Embodiment 1)
FIG. 1 shows a configuration of an imaging system 10 according to the present embodiment. The imaging system 10 includes an imaging device 11 and an image generation device 12.

  The imaging device 11 includes a wavelength modulation unit 101, a modulated image acquisition unit 102, and a transmission circuit 103. On the other hand, the image generation device 12 includes a reception circuit 104, a color image generation circuit 105, and an output I / F (interface) device 106. The imaging device 11 and the image generation device 12 may be integrated. Of course, when the imaging device 11 and the image generation device 12 are integrated, the transmission circuit 103 and the reception circuit 104 may be omitted.

(Imaging device 11)
First, the imaging device 11 will be described with reference to FIG.

  FIG. 2 shows details of the configuration of the wavelength modulation unit 101 and the modulated image acquisition unit 102.

  As shown in FIG. 2, the wavelength modulation unit 101 corresponds to the imaging optical system 201 and the random color filter array 202. The modulated image acquisition unit 102 corresponds to the image sensor 203.

(Imaging optical system 201)
The imaging optical system 201 includes at least one or a plurality of lenses and a lens position adjusting mechanism (none of which is shown). One or more lenses collect light from the subject and form an optical signal. The optical signal indicates an image of the subject. The lens position adjustment mechanism is, for example, an actuator for adjusting the imaging position by the lens and a control circuit (controller) that controls the drive amount of the actuator. Note that when the focus of one or more lenses is fixed, the lens position adjusting mechanism is unnecessary. Further, like a pinhole camera, an optical image may be formed without using a lens.

  The imaging optical system 201 may be called an optical system.

(Image sensor 203)
FIG. 3 shows an example of the detailed configuration of the random color filter array 202 and the image sensor 203.

  The image sensor 203 includes a plurality of photodiodes 203a and at least one AD converter 203b.

  The optical signal formed by the imaging optical system 201 is received by the photodiode 203a, and converted into an electrical signal by the AD conversion unit 203b. The electrical signal indicates a modulated image that is a modulated image for each pixel. That is, the image sensor 203 captures a modulated image. The image sensor 203 is disposed at the focal length of the lens.

  The image sensor 203 includes a plurality of pixels that receive light collected by the optical system and output an electrical signal. The pixel corresponds to the photodiode 203a. The plurality of photodiodes 203a of the image sensor 203 and the plurality of electrical signals correspond one-to-one.

(Random color filter array 202)
The random color filter array 202 has a plurality of types of color filters 202a to 202c. In the random color filter array 202, for example, the plurality of color filters 202a to 202c are at mutually different positions on a substantially plane, and are randomly arranged (for example, a color filter is randomly selected from a plurality of types of color filters). This is a collection of color filters. The random color filter array 202 is disposed in front of the photodiode 203a, that is, on the optical path where the light incident from the imaging optical system 201 reaches the image sensor 203 (for example, before the incident light reaches the light receiving surface of the photodiode 203a). The

  The color filter 202a, the color filter 202b, and the color filter 202c are color filters of different types (that is, wavelength characteristics related to the relationship between light wavelength and transmittance). As an example, one type of color filter 202a is an R filter that mainly transmits the R wavelength band, and another type of color filter 202b is a G filter that mainly transmits the G wavelength band. Another type of color filter 202c is a B filter that mainly transmits the B wavelength band.

  In the example of FIG. 3, the random color filter array 202 includes six color filters randomly selected from a set of three types of color filters 202a to 202c for each photodiode 203a. Details of the arrangement of the color filters 202a to 202c with respect to the photodiodes 203a (that is, the arrangement of individual color filters in a set of six color filters) will be described later with reference to FIGS.

  Typically, the random color filter array 202 is disposed in contact with the front surface of the image sensor 203, that is, the light receiving surface. The random color filter array 202 may be disposed in front of the image sensor 203 with a predetermined interval from the front surface.

  The random color filter array 202 is used for filtering light of a plurality of wavelength bands incident on the image sensor 203, and an image captured through the random color filter array 202 is referred to as a modulated image. The random color filter array 202 has different light transmittances at arbitrary positions where light of a plurality of wavelength bands is transmitted. Here, “position” means the position of a minute region having a certain area. Assuming that the random color filter array 202 composed of the substantially planar color filters 202a to 202c has a substantially planar shape, the light transmittance may be different for each position of each minute region on the plane. The light transmittance at each position is determined by the wavelength characteristics of the plurality of types of color filters 202a to 202c constituting the random color filter array 202 and the arrangement of the random color filter array 202, and is referred to as modulation information. The area of each minute region may be equal to or smaller than the light receiving area of each photodiode 203a of the image sensor 203, for example. The light transmittance may vary depending on the wavelength (wavelength band) to be transmitted.

  The plurality of photodiodes 203a and the positions of the plurality of minute regions related to the random color filter array 202 may correspond one-to-one or one-to-many.

  Further details of the random color filter array 202 will be described later.

(Transmission circuit 103)
The transmission circuit 103 transmits the modulation image captured by the image sensor 203 and the modulation information set corresponding to the random color filter array 202 to the image generation device 12. Transmission may be performed by either wired communication or wireless communication.

  In the present embodiment, it is assumed that the imaging system 10 includes the transmission circuit 103 and the reception circuit 104 and performs processing by transmitting and receiving modulated images and modulation information in substantially real time. However, the imaging system 10 may include a storage device (for example, a hard disk drive) that stores a modulated image and modulation information, and may perform processing in non-real time.

(Image generation device 12)
With reference to FIG. 1 again, the receiving circuit 104, the color image generating circuit 105, and the output interface device 106 of the image generating device 12 will be described.

(Receiving circuit 104)
The receiving circuit 104 receives the modulated image and the modulation information output from the imaging device 11. Communication between the reception circuit 104 and the transmission circuit 103 may be wired communication or wireless communication. Even if the transmission circuit 103 transmits the modulated image and the modulation information through wired communication, the reception circuit 104 may receive the information wirelessly through a device that converts the wired communication into wireless communication. . The reverse is also true.

(Color image generation circuit 105)
The color image generation circuit 105 generates a color image using the modulation image and modulation information received by the reception circuit 104. Details of the color image generation process (color image generation process) will be described later. The color image generation circuit 105 sends the generated color image to the output interface device 106.

(Output interface device 106)
The output interface device 106 is a video output terminal or the like. The output interface device 106 outputs the color image to the outside of the image generation device 12 as a digital signal or an analog signal.

(Details of the random color filter array 202)
Next, the random color filter array 202 will be described in more detail with reference to FIGS.

  FIG. 4 is a schematic diagram of the imaging system 10 including the random color filter array 202.

  As described above, the random color filter array 202 has a light transmittance that can vary depending on the position and the wavelength band as the wavelength characteristic. That is, the combinations of the light transmittances of the respective wavelength bands are different from each other at arbitrary positions where the light of the plurality of wavelength bands is transmitted. In order to realize such optical characteristics, the random color filter array 202 of the present embodiment includes a plurality of types of color filters 202a to 202c having different light transmittances according to the wavelength band of light. A plurality of color filters randomly selected from a plurality of types of color filters 202a to 202c are arranged so as to correspond to the photodiode 203a. Therefore, the number of color filters constituting the random color filter array 202 is greater than the number of photodiodes 203a constituting the image sensor 203. Further, the types of color filters arranged at each position in the random color filter array 202 can be different. As illustrated in FIG. 4, the random color filter array 202 includes, for example, a plurality of random color filter arrays 202 as viewed from the front of the photodiode 203a (that is, as viewed from the upstream imaging optical system 201 in the optical path of incident light). Each color filter is arranged so as to overlap the photodiode 203a at a different position. As seen from the front of the photodiode 203a, the color filters may partially overlap each other, but as shown in the example of FIG. 4, the amount of light received by the photodiode 203a is suppressed by not overlapping each other. Can be done.

  In FIG. 4, three color filters in which three types of color filters 202a to 202c are combined in association with one photodiode 203a are described, but this is only an example, and the color filters to be combined are shown. Both the number and the number of types are not limited to three.

  FIG. 5 is a diagram illustrating the light transmittance as each wavelength characteristic of the three types of color filters 202a to 202c. In the figure, the filter 1 is a color filter 202a, the filter 2 is a color filter 202b, and the filter 3 is a color filter 202c.

  The wavelength characteristic of the filter indicates how much light incident on the filter is reflected, transmitted, and absorbed according to the wavelength. The sum of the reflected light, transmitted light and absorbed light is equal to the incident light. The ratio of transmitted light to incident light is called “transmittance”. This transmittance is also referred to as light transmittance. The ratio of absorbed light to incident light is called “absorption rate”. The absorptance can also be obtained by subtracting the amount of reflected light and the amount of transmitted light from the amount of incident light, and further dividing by the amount of incident light. FIG. 5 shows the relationship between the light transmittance and the wavelength of light in each filter.

  6 and 7 are schematic views of a random color filter array 202 composed of three types of color filters 202a to 202c. FIG. 6 shows the arrangement of the color filters of the random color filter array 202 and the photodiodes 203a of the image sensor 203 on a two-dimensional plane. This two-dimensional plane is a plane parallel to the light receiving surface of the image sensor 203, for example. FIG. 7 shows a cross section of the random color filter array 202 and the image sensor 203. In FIG. 6 and the like, the color filters 202a, 202b, and 202c are represented by R, G, and B, respectively.

  In this embodiment, light that has passed through six color filters selected from the three types of color filters 202a, 202b, and 202c is received by one photodiode 203a. As shown in FIG. 6, there are 3H6 = 28 combinations in which six color filters are selected from the three types of color filters 202a to 202c while allowing duplication.

  Here, for the wavelength λ, the light transmittance of the color filter 202a is C1 (λ), the light transmittance of the color filter 202b is C2 (λ), and the light transmittance of the color filter 202c is C3 (λ). The number of color filters corresponding to the photodiode 203a related to the pixel at the position (u, v) is NP (u, v), of which the number of color filters 202a is NR (u, v) and the number of color filters 202b. Is NG (u, v), and the number of color filters 202c is NB (u, v), the light transmittance CA (λ, u, v) as the wavelength characteristic at a certain position (u, v) of the random color filter array 202. v) is expressed by the following equation.

  CA (λ, u, v) = {NR (u, v) · C1 (λ) + NG (u, v) · C2 (λ) + NB (u, v) · C3 (λ)} / NP (u, v (1)

  Here, NR (u, v) + NG (u, v) + NB (u, v) = NP (u, v). Further, NP (u, v) = 6.

  FIG. 8 shows the light transmittance according to the wavelength as the 28 wavelength characteristics realized by the random color filter array 202 described above.

  Thus, although the random color filter array 202 in this embodiment uses only three types of color filters 202a to 202c, six filters selected from the three types are used for one photodiode 203a. By making it correspond, 28 wavelength characteristics can be given to each pixel, so that a random color filter array having a certain degree of randomness (that is, an optical filter array that enables random sampling with a certain degree of randomness) Is realized.

(Variation 1 of the random color filter array 202)
In the above description, the filters constituting the random color filter array 202 are three types of color filters 202a to 202c. However, four or more types of filters may be used and function as optical filters having different light transmittances as wavelength characteristics. If it does, it is not necessarily a color filter. Hereinafter, a mode (Modification 1) in which four types of filters are used as the random color filter array 202 described above will be described.

  FIG. 9 is a diagram showing the light transmittance as each wavelength characteristic of the four types of filters. In the figure, filters 1 to 3 are color filters 202a to 202c, and a filter 4 is a filter that transmits the entire region. The filter 4 can be realized by not including a member as a filter when generating a random color filter array. By providing a portion that does not contain a member in this way and including a filter that transmits the entire region in the random color filter array 202, the cost can be reduced. Here, in the random color filter array 202, the filter that transmits the entire region will be referred to as a color filter for the sake of convenience, and the description will be continued.

  10 and 11 are schematic views of a random color filter array 202 composed of four types of filters. FIG. 10 shows the arrangement of the filters of the random color filter array 202 and the photodiodes 203a of the image sensor 203 on a two-dimensional plane. In FIG. 10, the color filter 202d that transmits the entire region is denoted by W. FIG. 11 shows a cross section of the random color filter array 202 and the image sensor 203. In FIG. 11 and the like, the color filter 202d is represented by W.

  In this modification, light that has passed through six filters selected from four types of color filters 202a, 202b, 202c, and 202d is received by one photodiode 203a. The color filter 202d is a filter that transmits the entire area. As shown in FIG. 10, there are 4H6 = 84 combinations in which 6 filters are selected from the four types of filters while allowing overlap.

  Here, for the wavelength λ, the light transmittance of the color filter 202a is C1 (λ), the light transmittance of the color filter 202b is C2 (λ), the light transmittance of the color filter 202c is C3 (λ), and the color filter 202d. Is assumed to be C4 (λ). The number of color filters corresponding to the photodiode 203a related to the pixel at the position (u, v) is NP (u, v), of which the number of color filters 202a is NR (u, v) and the number of color filters 202b. Is NG (u, v), the number of color filters 202c is NB (u, v), and the number of color filters 202d is NW (u, v), the random color filter array 202 is located at a certain position (u, v). The light transmittance CA (λ, u, v) as the wavelength characteristic is expressed by the following equation.

  CA (λ, u, v) = {NR (u, v) · C1 (λ) + NG (u, v) · C2 (λ) + NB (u, v) · C3 (λ) + NW (u, v) · C4 (λ)} / NP (u, v) (Formula 2)

  Here, NR (u, v) + NG (u, v) + NB (u, v) + NW (u, v) = NP (u, v). Further, NP (u, v) = 6.

  FIG. 12 shows light transmittance according to wavelength as 84 wavelength characteristics realized by the random color filter array 202 according to this modification.

  As described above, the random color filter array 202 according to the present modification uses only four types of color filters 202a to 202d. However, six filters selected from the four types are selected for one photodiode 203a. By making it correspond, 84 kinds of wavelength characteristics can be given to each pixel, so that a random color filter array having a certain degree of randomness is realized.

(Modification 2 of the random color filter array 202)
In the above description, the random color filter array 202 is configured such that light that has passed through six filters selected from three or more types of filters is received by one photodiode 203a. Although shown, the number of filters through which light received by one photodiode 203a passes may be seven or more. Hereinafter, a mode (Modification 2) in which the random color filter array 202 is configured such that light that has passed through nine of the three types of filters is received by one photodiode 203a will be described.

  FIG. 13 illustrates a two-dimensional plane of each filter of the random color filter array 202 and each photodiode 203a of the image sensor 203 in a form in which light transmitted through nine of the three types of filters is received by one photodiode 203a. The above arrangement is shown.

  As shown in FIG. 13, there are 3H9 = 55 combinations in which nine filters are selected from three types of filters while allowing duplication.

  The light transmittance CA (λ, u, v) as a wavelength characteristic at a certain position (u, v) of the random color filter array 202 according to the present modification is expressed by the above-described Expression 1. However, NP (u, v) = 9.

  FIG. 14 shows the light transmittance according to the wavelength as 55 kinds of wavelength characteristics realized by the random color filter array 202 according to this modification.

  As described above, the random color filter array 202 according to this modification uses only three types of color filters, but nine photodiodes selected from the three types are associated with one photodiode 203a. Thus, since 55 types of wavelength characteristics can be provided for each pixel, a random color filter array having a certain degree of randomness is realized.

(Modification 3 of the random color filter array 202)
Further, the random color filter array 202 is configured so that light that has passed through seven or more filters selected from four or more types of filters 202a, 202b, 202c, and 202d is received by one photodiode 203a. Also good. Hereinafter, a mode (Modification 3) in which the random color filter array 202 is configured such that light that has passed through nine of the four types of filters is received by one photodiode 203a will be described.

  FIG. 15 shows a two-dimensional plane of each filter of the random color filter array 202 and each photodiode 203a of the image sensor 203 in a form in which light transmitted through nine of the four types of filters is received by one photodiode 203a. The above arrangement is shown.

  As shown in FIG. 15, there are 4H9 = 220 combinations for selecting nine filters from the four types of filters while allowing duplication.

  The light transmittance CA (λ, u, v) as a wavelength characteristic at a certain position (u, v) of the random color filter array 202 according to this modification is expressed by the above-described Expression 2. However, NP (u, v) = 9.

  FIG. 16 shows the light transmittance according to the wavelength as 220 wavelength characteristics realized by the random color filter array 202 according to this modification.

  As described above, the random color filter array 202 according to this modification uses only four types of filters, but nine photodiodes selected from the four types correspond to one photodiode 203a. , 220 wavelength characteristics can be provided for each pixel, so that a random color filter array having a certain degree of randomness is realized.

(Modification 4 of the random color filter array 202)
In the above-described embodiment, the three types of color filters 202a to 202c constituting the random color filter array 202 are an R (red) filter, a G (green) filter, and a B (blue) filter, respectively. However, the filter characteristics are not limited to this. Here, a mode in which a complementary color filter is included in the random color filter array 202 will be described.

  For example, in the random color filter array 202, a complementary color filter (for example, a magenta, cyan, or yellow filter) having a wider transmission band than the primary color filter such as an R filter, a G filter, or a B filter is used.

  FIG. 17 is a diagram showing the light transmittance as each wavelength characteristic of the three types of complementary color filters. In the figure, a filter 1 is, for example, a magenta filter, a filter 2 is, for example, a cyan filter, and a filter 3 is, for example, a yellow filter.

  By using such a complementary color filter, the transmission band is widened, so that an image with reduced noise can be acquired.

  Of course, R, G, and B color filters, that is, primary color filters, filters that transmit the entire wavelength band, and complementary color filters may be combined as filters constituting the random color filter array 202. For example, the random color filter array 202 may be configured by combining the filters shown in FIGS. 9 and 17. In this case, the random color filter array 202 is configured such that light transmitted through six of the seven types of filters is received by one photodiode 203a. There are 7H6 = 924 combinations in which 6 filters are selected from the 7 types of filters while allowing duplication. In this case, the random color filter array 202 uses only seven types of filters, but by associating six filters selected from the seven types with one photodiode 203a, 924 wavelength characteristics can be obtained. Since it can be given for each pixel, a random color filter array having a certain degree of randomness is realized.

  Similarly, the random color filter array 202 may be configured by combining the filters shown in FIGS. 5 and 17. In this case, the random color filter array 202 is configured such that light transmitted through six of the six types of filters is received by one photodiode 203a. There are 6H6 = 462 combinations in which 6 filters are selected from 6 types of filters while allowing overlap. In this case, the random color filter array 202 uses only six types of filters, but by associating six filters selected from the six types with one photodiode 203a, 462 kinds of wavelength characteristics can be obtained. Since it can be given for each pixel, a random color filter array having a certain degree of randomness is realized.

(Processing of image generation device 12)
Next, processing in the image generation apparatus 12 (see FIG. 1) will be described with reference to FIG.

  FIG. 18 is a flowchart showing a main processing procedure of the image generation apparatus 12.

  The reception circuit 104 of the image generation device 12 receives the modulated image and the modulation information transmitted by the transmission circuit 103 of the imaging device 11 (step S101).

  Next, the color image generation circuit 105 generates a color image from the modulated image and the modulation information using an image restoration technique (for example, a compression sensing technique) (step S102).

  Next, the output interface device 106 outputs the color image generated by the color image generation circuit 105 for display on a display or use for image processing such as human detection (step S103).

(Color image generation processing)
Hereinafter, the color image generation processing by the color image generation circuit 105 in step S102 will be described in more detail.

  The color image generation process can be formulated as follows, where y is a captured modulated image and x is a generated image that is a color RGB image to be generated.

  Here, the matrix A is a sampling matrix indicating modulation information. The sampling matrix A shows the relationship between the modulated image y and the generated image x. Further, for example, when the number of pixels is N, the modulated image y is represented by an N × 1 matrix, the generated image x is represented by a 3N × 1 matrix, and the sampling matrix A is an N × 3N matrix. It is expressed by

  Hereinafter, a method for obtaining the sampling matrix A will be described. Here, a method using color calibration by the Macbeth color checker will be described. The Macbeth Color Checker is a 24-color sample based on the Munsell color system. In the Macbeth color checker, XYZ values and sRGB values of each color sample are defined.

  For example, the sRGB values of each color sample j (j = 1, 2, 3,..., 24) are expressed by R ′ (j), G ′ (j), and B ′ (j). Further, by imaging each color sample j with the imaging device 11 of the present embodiment, the pixel signal I (j, i) of the modulated image is obtained at each pixel i (i = 1, 2, 3,..., N). To be acquired. In this case, the following relational expression holds.

  c (1, i) .R (j) + c (2, i) .G (j) + c (3, i) .B (j) = I (j, i) (Formula 4)

  However, c (x, i) (x = 1, 2, 3) is an element of i rows (3 (i−1) + x) columns of the sampling matrix A. Elements other than i rows (3 (i−1) + x) columns of the sampling matrix A are zero. R (j), G (j) and B (j) are obtained by linearly converting R ′ (j), G ′ (j) and B ′ (j). Specifically, it is obtained by the following calculation.

  From the relationship of Equation 4, 24 equations are derived for the three unknowns c (1, i), c (2, i), and c (3, i). Therefore, it is possible to derive c (1, i), c (2, i) and c (3, i) by the least square method. By performing this process on all the pixels i, the sampling matrix A can be acquired.

  Next, a method for obtaining the generated image x from the sampling matrix A and the modulated image y in the color image generating circuit 105 will be described. In order to simplify the description, a case where the number of pixels of the image sensor 203 is N = 16 will be described.

  FIG. 19 shows a modulated image y when the number of pixels of the image sensor 203 is N = 16 and an R image r, a G image g, and a B of a generated image (color RGB image) x generated based on the modulated image y. It is a schematic diagram which shows the image b. In this figure, (a) is a modulated image y, (b) is an R image r which is a red (R) channel of a color RGB image generated based on the modulated image y, and (c) is based on the modulated image y. G images g and (d) which are green (G) channels of the generated color image indicate B images b which are blue (B) channels of the color image generated based on the modulated image y. The modulated image y and the generated image x are expressed by the following equations.

  As is apparent from Equation 6, in Equation 3, the number of elements of x, which is an unknown number, is 48, and the number of elements of y, which is the number of observations, is 16. That is, the number of equations is smaller than the unknown. Therefore, Expression 3 is a defect setting problem.

  The imaging system 10 uses a compressed sensing technique to solve this defect setting problem. The compression sensing technique is a technique for compressing the amount of data by performing addition processing (encoding) at the time of signal sensing and decoding (restoring) the original signal by performing restoration processing using the compressed data. . In the compressed sensing process, prior knowledge is used to solve the defect setting problem.

  As prior knowledge about a natural image, a total variation (for example, Non-Patent Document 1 and Non-Patent Document 2) that is the sum of absolute values of luminance changes between neighboring positions on the image may be used. Further, sparsity (for example, Non-Patent Document 3) that many coefficients become 0 in linear transformation such as wavelet transformation, DCT transformation, and curvelet transformation may be used. Moreover, dictionary learning (Dictionary Learning) (for example, nonpatent literature 4) etc. which acquire the conversion factor of the above-mentioned linear transformation by learning may be utilized.

  Here, Decorrelated Vectorial Total Variation, which is a technique classified as a kind of total variation, will be described. This technique suppresses the occurrence of artifacts called false colors by separately calculating the luminance component and the gradient of the color difference component of a color image. This is realized by minimizing the following evaluation function.

  This evaluation function is composed of the following three terms.

1. Data Fidelity term: || Ax−y || ² ₂ : A constraint term for satisfying Equation 3.
2. Dynamic range term: a range of x for calculating the minimum value min ([0, 255] ^{3 × N} ): a constraint term for a pixel value being 0 or more and 255 or less.
3. De correlated vectorial total variation term: J (x): Total variation term in which the gradient of the luminance component and the color difference component of the color image is separated.

Here, || Ax−y || ² ₂ represents the square sum of Ax−y (the square of the L2 norm). J (x) corresponds to the difference between neighboring pixels regarding the luminance component and the color difference component in the entire image, and is expressed by the following Expressions 8 to 12. In Expressions 8 to 12, R represents a real number, and R ₊ represents a non-negative real number.

  20 to 22 show an example of a color image generated by the color image generation circuit 105 in this embodiment for each color. 20 shows an R image, FIG. 21 shows a G image, and FIG. 22 shows a B image. The inventors of the present application actually generated and compared color images. In FIGS. 20 to 22, each image is shown by shading according to the luminance value of the image.

  In these drawings, (a) shows a correct image (correct color image) captured by a three-plate camera. (B) is a general demosaicing technique and shows a demosaicing image by the ACPI (Adaptive Color Plane Interpolation) method described in Patent Document 1. (C) shows the demosaicing image by MLRI (Minimized-Laplacian Residual Interpolation) method of the nonpatent literature 5. FIG. (D) shows a generated image (restored image) generated by the color image generation circuit 105 according to the present embodiment using de-correlated vector total variation.

  Hereinafter, an example will be described with reference to FIG. 20, but the same description can be applied to FIGS. 21 and 22.

  A window having a lattice pattern is shown in the center of FIG.

  20B and 20C, images captured using the conventional Bayer array shown in FIG. 32 are used. The inventors of the present application have verified and confirmed that an artifact called a false color exists in the vicinity of the edge in the demosaicing image by the ACPI method shown in FIG. Specifically, the presence of a false color was confirmed at the edge of the window. The inventors of the present application have also confirmed that such a false color does not exist in the restored image according to the present embodiment shown in FIG.

  In the demosaicing image by the MLRI method shown in FIG. 20C, high-frequency components such as a lattice pattern of windows cannot be restored. On the other hand, these images indicate that the color image generation circuit 105 according to the present embodiment can restore up to a high frequency region without artifacts as compared with the prior art.

  FIG. 23 is an example of a color image including all colors. In FIG. 23, each image is shown by shading according to the luminance value of the image (that is, the luminance value calculated by adding the R, G, and B component values multiplied by the coefficient for each color). . As apparent from a glance at the grid pattern of the window, the generated image (restored image) according to the present embodiment is closest to the correct image.

  Note that when the signal received by the image sensor 203 is saturated, Equation 3 does not hold, and the data fidelity term in Equation 7 causes a reduction in the restored image quality, which is the image quality of the generated image. Therefore, the color image generation circuit 105 improves the restored image quality by changing the data fidelity term in Equation 7 for saturated pixels. This can be done by changing the second term in Equation 7 as follows.

  Here, the second term of Equation 13 is a data fidelity term for a saturated pixel and is called saturation constraint.

Also,
Indicates unsaturated pixels in the captured modulated image y.

Also,
Indicates saturated pixels of the captured modulated image y.

Also,
Indicates a vector obtained by transposing the i-th row of the sampling matrix A.

Also,
It is. This is a function that returns 0 when the value obtained by multiplying the estimated value x by the sampling matrix A is saturated, that is, when y _i or more, and ∞ in other cases. Here, the proximity operator of Equation 14 is
It is expressed by Therefore, it can be solved by using a convex optimization technique such as ADMM (Alternating Direction Method of Multipliers) (see Non-Patent Document 6).

  24 and 25 show G images generated by various methods. The inventors of the present application actually generated and compared color images. In FIG. 24 and FIG. 25, each image is shown by shading according to the luminance value of the G image.

  24 and 25, (a) shows a correct image captured by a three-plate camera. (B) and (c) show the generated image (restored image) generated by the color image generating circuit 105. Specifically, (b) shows an image restored by the evaluation function of Equation 7, that is, a restored image without saturation constraint, and (c) is restored by changing the second term in the evaluation function of Equation 7 to Equation 13. In other words, a restored image with saturation constraint is shown. Also, (a) to (c) in FIG. 25 are “enlargement target areas” indicated by the rectangular frames in FIG. 24 (d) of the images in (a) to (c) in FIG. The image which expanded the corresponding range is shown.

  Some pixels in FIG. 24A have high luminance values, and some pixels in the observed modulated image are saturated. Therefore, as shown in FIGS. 24B and 25B, in the restored image without saturation constraint, the data fidelity term does not function correctly, and artifacts such as sesame salt noise occur. On the other hand, it was confirmed that such an artifact did not occur in the restored image shown in FIG. 24C and FIG. 25C in which the restoration image quality was improved using the saturation constraint. The generated image (restored image) according to the present embodiment shown in (c) of FIG. 25 is very close to the correct image of (a).

  As described above, in the imaging system 10 of the present embodiment, imaging is performed using the random color filter array 202, and color image generation processing is performed by introducing constraints due to saturated pixels in the compression sensing technique, thereby producing artifacts. And a high-definition color image can be acquired. That is, in the imaging system 10, only one image sensor (image sensor) is used for imaging, but as shown in FIGS. 25A and 25C, a high-quality image close to an image captured by a three-plate camera. It is possible to obtain

(Variation related to the positional relationship between the random color filter array 202 and the image sensor 203)
In the above description, an example in which a plurality of (for example, six, nine, etc.) color filters constituting the random color filter array 202 is arranged corresponding to one photodiode 203a in the image sensor 203 has been shown. The correspondence relationship between the number of color filters and one photodiode 203a is not necessarily constant. Further, although the example in which the number of color filters constituting the random color filter array 202 is larger than the number of photodiodes 203a in the image sensor 203 is shown, the number of color filters is not necessarily larger than the number of photodiodes 203a. May be.

  For example, the arrangement of the color filter group in the random color filter array 202 and the arrangement of the photodiode 203a group in the image sensor 203 may be shifted from each other. FIG. 26 shows an example of an arrangement in which the phases of the random color filter array 202 and the photodiode 203a are shifted in this way.

  In the random color filter array 202 shown in FIG. 26, some color filters 202a to 202c are arranged to cover a plurality of photodiodes 203a so that the positional relationship between the color filters 202a to 202c and the photodiodes 203a is not aligned. Has been. That is, in the example of FIG. 26, the random color filter array 202 includes color filters 202a to 202c that overlap with the plurality of photodiodes 203a in an arrangement in which the plurality of photodiodes 203a are projected on a two-dimensional plane arranged two-dimensionally. Including.

  Here, the area of the photodiode 203a related to the pixel at the position (u, v) is SP (u, v), of which the area where the color filter 202a is arranged is SR (u, v), and the color filter 202b is arranged. If the area where SG (u, v) is arranged and the area where the color filter 202c is arranged is SB (u, v), the light transmittance CA (λ, u) as the wavelength characteristic at the position (u, v) , V) is expressed by the following equation. For the wavelength λ, the light transmittance of the color filter 202a is C1 (λ), the light transmittance of the color filter 202b is C2 (λ), and the light transmittance of the color filter 202c is C3 (λ).

  CA (λ, u, v) = {SR (u, v) · C1 (λ) + SG (u, v) · C2 (λ) + SB (u, v) · C3 (λ)} / SP (u, v (Equation 16)

  Here, SR (u, v) + SG (u, v) + SB (u, v) ≦ SP (u, v).

  As described above, when the random color filter array 202 is configured such that the arrangement phases of the color filters 202a to 202c and the photodiode 203a do not coincide with each other, the color filters 202a to 202c are necessarily smaller than the photodiode 203a. For example, the color filters 202a to 202c may be as large as approximately the size of the photodiode 203a.

  An example of the random color filter array 202 in which the size of the color filters 202a to 202c is approximately equal to the size of the photodiode 203a is shown in FIGS. 27A and 27B. In the example of FIGS. 27A and 27B, the random color filter array 202 includes a color filter that overlaps the plurality of photodiodes 203a in an arrangement projected onto a two-dimensional plane. FIG. 27B shows an example in which the random color filter array 202 is larger than the range of all the photodiodes 203 a of the image sensor 203. As shown in FIG. 27A or FIG. 27B, when the color filters 202a to 202c are arranged so that the ranges of the photodiodes 203a do not completely coincide with each other, the light transmittance CA (λ as a wavelength characteristic at the position (u, v)). , U, v) is expressed by Equation 16.

  According to the imaging system 10 that performs imaging using the random color filter array 202 having various configurations described above, a high-definition color image with reduced artifacts can be obtained by performing color image generation processing using a compression sensing technique. Can be acquired.

(Embodiment 2)
FIG. 28 shows a configuration of the imaging system 10 according to the present embodiment. Of the components of the imaging system 10 according to the present embodiment, the same components as those of the imaging system 10 shown in the first embodiment are denoted by the same reference numerals in FIG. 28 as those in FIG. Is omitted as appropriate. The imaging system 10 according to the present embodiment includes a multiband image generation circuit 107 in the image generation apparatus 12 instead of the color image generation circuit 105 shown in the first embodiment. The imaging system 10 can generate a multiband image that is not limited to an RGB color image of the three primary colors. A multiband image is an image expressed by a signal obtained by dividing the wavelength band of light into four or more regions. The wavelength band is not limited to visible light, and may be a wavelength band such as near infrared, infrared, and ultraviolet. A multiband image is, for example, a near infrared light image, an infrared light image, an ultraviolet light image, or the like. possible.

  The multiband image generation circuit 107 generates a multiband image from the modulated image and the modulation information using an image restoration technique (for example, a compression sensing technique). The multiband image generation circuit 107 sends the generated multiband image to the output interface device 106.

  The output interface device 106 according to the present embodiment outputs a multiband image to the outside of the image generation device 12 as a digital signal or an analog signal. The output interface device 106 may switch the output image so that a color image of visible light is output in a bright scene and a near-infrared light image is output in a dark scene.

  In addition, the random color filter array 202 in the wavelength modulation unit 101 (see FIG. 28) of the imaging system 10 according to the present embodiment includes, for example, mainly near-infrared in addition to color filters such as an R filter, a G filter, and a B filter. It may be a random optical filter array configured as an array of a plurality of types of optical filters including filters that transmit light. For example, when a near-infrared light image is acquired as a multiband image, the random optical filter array can include a filter that transmits near-infrared light.

  The processing in the image generation apparatus 12 (see FIG. 28) according to the present embodiment will be described below with reference to FIG.

  FIG. 29 is a flowchart showing a main processing procedure of the image generation apparatus 12 in the present embodiment. In the figure, the same elements as those in FIG. 18 are denoted by the same reference numerals.

  The receiving circuit 104 of the image generating device 12 receives the modulated image and the modulation information transmitted by the transmitting circuit 103 (step S101).

  Next, the multiband image generation circuit 107 performs multiband image generation processing for generating a multiband image from the modulated image and the modulation information using an image restoration technique (for example, a compression sensing technique) (step S104).

  Next, the output interface device 106 outputs the multiband image generated by the multiband image generation circuit 107 for display on a display or use for image processing such as human detection (step S105).

(Multiband image generation processing)
Hereinafter, the multiband image generation process in step S104 will be described in more detail.

  The multiband image generation process can be formulated as follows, where y ′ is a captured modulated image and x ′ is an M band multiband image to be generated (M is an integer of 4 or more).

  Here, the matrix A is a sampling matrix indicating modulation information.

  In order to simplify the description, assuming that the number of pixels of the image sensor 203 is N = 16 (see FIG. 19), this modulated image y ′ and a multiband image generated based on the modulated image y ′ A generated image x ′ is expressed by the following equation. FIG. 30 shows the generated image x ′ divided into images of M channels.

  As is clear from Equation 18, the number of elements of x ′ that is an unknown number in Equation 17 is 16M, and the number of elements of y ′ that is the number of observations is 16. That is, the number of equations is smaller than the unknown. Therefore, Expression 17 is a defect setting problem. However, as shown in the first embodiment, this defect setting problem can be solved by using the compression sensing technique. Various techniques shown in the first embodiment can also be applied to the imaging system 10 that generates a multiband image by imaging according to the present embodiment.

  As described above, the imaging system 10 of the present embodiment performs imaging using the random color filter array 202 or the random optical filter array, and reduces artifacts by performing multiband image generation processing using a compression sensing technique. In addition, a high-definition multiband image can be acquired.

(Other embodiments)
As described above, Embodiments 1 and 2 have been described as examples of the technology according to the present disclosure. However, the technology according to the present disclosure is not limited to these embodiments, and can also be applied to embodiments in which changes, replacements, additions, omissions, and the like have been made as appropriate. That is, unless departing from the gist of the present disclosure, the forms in which various modifications conceived by those skilled in the art have been applied to the above-described embodiments, the forms constructed by combining the components in the different embodiments, etc. In one embodiment. For example, the following modifications are also included in one embodiment of the technology according to the present disclosure.

  In the above-described embodiment, an example in which the imaging system 10 includes the imaging device 11 and the image generation device 12 has been described. However, the image generation device according to the present disclosure may include the imaging device 11 described above. That is, the imaging system 10 described above may be used.

  In the above embodiment, the imaging system 10 that generates a color image or a multiband image is shown. However, the imaging system 10 generates an image that is expressed by a signal obtained by dividing an arbitrary wavelength band of light into two or more regions. For example, the imaging system 10 may generate an image represented by a signal obtained by dividing the infrared light region into three regions.

  In addition, the imaging optical system 201 described above is not limited to the one that uses a lens for imaging, and may be one that uses a reflecting mirror, for example.

  In addition, each component (particularly each circuit) of the imaging system 10 shown in the above embodiment is configured by dedicated hardware, or executes software (program) suitable for each component. It may be realized by. Each component may be realized by a program execution unit such as a microprocessor reading and executing a program recorded in a storage medium (or recording medium) such as a hard disk or a semiconductor memory.

  The plurality of circuits included in the imaging device 11 may constitute one circuit as a whole, or may be separate circuits. Similarly, the plurality of circuits included in the image generation device 12 may constitute one circuit as a whole, or may be separate circuits. Each of these circuits may be a general-purpose circuit or a dedicated circuit. Further, for example, another constituent element may execute the process executed by a specific constituent element in the above embodiment instead of the specific constituent element. In addition, the execution order of the various processes in the above embodiment may be changed, and a plurality of processes may be executed in parallel.

  As described above, the image generation apparatus according to the present disclosure has a random optical filter array (for example, a color filter 202a to 202d, or a complementary color filter, a filter that mainly transmits light other than visible light). A random color filter array 202), and the plurality of types of optical filters are randomly arranged. The photodiode 203a that receives light transmitted through the random optical filter array and the light received by the photodiode 203a are digital. An AD conversion unit 203b for converting data, and a generation circuit (for example, a color image generation circuit 105, a multiband image generation circuit 107) that generates an image using the digital data and the modulation information of the random optical filter array. Random optical filter array At least two optical filters are positioned in front of the photodiode 203a (for example, in front of the light receiving region such as on the optical path until the light that has passed through the imaging optical system 201 reaches the light receiving region of the photodiode 203a). The optical filter having the smallest size in the optical filter array is smaller than the photodiode 203a. Thereby, the photodiode 203a as each pixel has sufficient information through the random optical filter array in which a plurality of types of optical filters having different wavelength characteristics related to the relationship between the wavelength of light and the transmittance are mutually arranged. Therefore, a decrease in resolution can be suppressed and an appropriate image can be generated. By the way, since the color filter array is generally formed by applying a color resist, the individual color filters 202a to 202c are not separated as shown in FIG. 6, and adjacent color filters of the same type as shown in FIG. Is a large size. In this case, the optical filter having the smallest size indicates the size of the minimum component of the color filter array. Specifically, in FIG. 31, the size of the color filter 202a in the upper right corner is the size of the optical filter having the smallest size.

  Further, for example, the optical filter may be the color filters 202a to 202c, the random optical filter array may be the random color filter array 202, and the generation circuit may be the color image generation circuit 105.

  For example, when viewed from the front of the photodiode 203a, at least two optical filters (for example, color filters 202a to 202d) may overlap the photodiode 203a at different positions.

  In addition, for example, the number of optical filters (for example, color filters 202a to 202d) in the random optical filter array (for example, the number for each arrangement position of the arranged optical filters) is more than six times the number of the photodiodes 203a. Also good.

  Further, for example, the plurality of types of optical filters include a first type optical filter (for example, a color filter 202a), a second type optical filter (for example, a color filter 202b), and a third type optical filter (for example, a color filter). The first type optical filter, the second type optical filter, and the third type optical filter may have different wavelength characteristics.

  Further, for example, the photodiode 203a receives light including the first wavelength band, the second wavelength band, and the third wavelength band, and the light of the first type optical filter in the first wavelength band. Unlike the transmittance, the light transmittance of the second type optical filter in the first wavelength band, and the light transmittance of the third type optical filter in the first wavelength band, the first in the second wavelength band is different. The light transmittance of the seed optical filter, the light transmittance of the second kind optical filter in the second wavelength band, and the light transmittance of the third kind optical filter in the second wavelength band differ from the third kind. Light transmittance of the first type optical filter in the third wavelength band, light transmittance of the second type optical filter in the third wavelength band, and light transmittance of the third type optical filter in the third wavelength band. It may be different.

  For example, the generation circuit may be the multiband image generation circuit 107.

  Further, for example, the generation circuit (for example, the color image generation circuit 105, the multiband image generation circuit 107, etc.) may generate an image using a compression sensing technique.

  In addition, the imaging apparatus according to the present disclosure includes a random optical filter array (for example, the random color filter array 202) having a plurality of types of optical filters (for example, the color filters 202a to 202d), and the plurality of types of optical filters are randomly selected. And a photodiode 203a that receives the light transmitted through the random optical filter array, and an AD converter 203b that converts the light received by the photodiode 203a into digital data, and includes at least two in the random optical filter array. One optical filter is located above the photodiode 203a, and the optical filter having the smallest size in the random optical filter array is smaller than the photodiode 203a. Thereby, the photodiode 203a as each pixel has sufficient information through the random optical filter array in which a plurality of types of optical filters having different wavelength characteristics related to the relationship between the wavelength of light and the transmittance are mutually arranged. You can get.

  The imaging system according to the present disclosure can be applied to various cameras.

DESCRIPTION OF SYMBOLS 10 Imaging system 11 Imaging apparatus 12 Image generation apparatus 101 Wavelength modulation part 102 Modulated image acquisition part 103 Transmission circuit 104 Reception circuit 105 Color image generation circuit 106 Output I / F (interface) apparatus 107 Multiband image generation circuit 201 Imaging optical system 202 Random Color Filter Array 202a, 202b, 202c, 202d Color Filter 203 Image Sensor 203a Photodiode 203b AD Converter

Claims (9)

A random optical filter array having a plurality of types of optical filters;
Here, the plurality of types of optical filters are randomly arranged,
A photodiode for receiving light transmitted through the random optical filter array;
An AD converter that converts light received by the photodiode into digital data;
Using the digital data and the modulation information of the random optical filter array, a generation circuit that generates an image,
At least two optical filters in the random optical filter array are located in front of the photodiode;
The optical filter having the smallest size in the random optical filter array is smaller than the photodiode. The optical filter is a color filter,
The random optical filter array is a random color filter array;
The image generation apparatus according to claim 1, wherein the image generated by the generation circuit is a color image. The image generation apparatus according to claim 1, wherein each of the at least two optical filters overlaps the photodiode at a position different from each other when viewed from the front of the photodiode. The image generation device according to claim 1, wherein the number of optical filters in the random optical filter array is six times or more than the number of the photodiodes. The plurality of types of optical filters include a first type optical filter, a second type optical filter, and a third type optical filter,
The image generation device according to any one of claims 1 to 4, wherein the first type optical filter, the second type optical filter, and the third type optical filter have different wavelength characteristics. . The photodiode receives light including a first wavelength band, a second wavelength band, and a third wavelength band,
The light transmittance of the first type optical filter in the first wavelength band, the light transmittance of the second type optical filter in the first wavelength band, and the third wavelength in the first wavelength band. Unlike the light transmittance of some optical filters,
The light transmittance of the first type optical filter in the second wavelength band, the light transmittance of the second type optical filter in the second wavelength band, and the third wavelength in the second wavelength band. Unlike the light transmittance of some optical filters,
The light transmittance of the first type optical filter in the third wavelength band, the light transmittance of the second type optical filter in the third wavelength band, and the third wavelength band in the third wavelength band. The image generation apparatus according to claim 5, wherein the image generation apparatus is different from a light transmittance of a seed optical filter. The image generation apparatus according to claim 1, wherein the image generated by the generation circuit is a multiband image. The image generation apparatus according to claim 1, wherein the generation circuit generates the image using a compression sensing technique. A random optical filter array having a plurality of types of optical filters;
Here, the plurality of types of optical filters are randomly arranged,
A photodiode for receiving light transmitted through the random optical filter array;
An AD conversion unit that converts light received by the photodiode into digital data;
At least two optical filters in the random optical filter array are located in front of the photodiode;
The optical filter having the smallest size in the random optical filter array is smaller than the photodiode.