WO2022020989A1 - 一种滤光阵列、移动终端以及设备 - Google Patents

一种滤光阵列、移动终端以及设备 Download PDF

Info

Publication number
WO2022020989A1
WO2022020989A1 PCT/CN2020/104797 CN2020104797W WO2022020989A1 WO 2022020989 A1 WO2022020989 A1 WO 2022020989A1 CN 2020104797 W CN2020104797 W CN 2020104797W WO 2022020989 A1 WO2022020989 A1 WO 2022020989A1
Authority
WO
WIPO (PCT)
Prior art keywords
filter array
type
image
filter
processing
Prior art date
Application number
PCT/CN2020/104797
Other languages
English (en)
French (fr)
Inventor
黄韬
杨晖
伍朝晖
张友明
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to CN202080012899.2A priority Critical patent/CN114287127B/zh
Priority to EP20946880.0A priority patent/EP4181509A4/en
Priority to PCT/CN2020/104797 priority patent/WO2022020989A1/zh
Publication of WO2022020989A1 publication Critical patent/WO2022020989A1/zh
Priority to US18/159,724 priority patent/US20230170363A1/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
    • H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
    • H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
    • H04N25/131—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements including elements passing infrared wavelengths
    • H—ELECTRICITY
    • H01—ELECTRIC ELEMENTS
    • H01L—SEMICONDUCTOR DEVICES NOT COVERED BY CLASS H10
    • H01L27/00—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate
    • H01L27/14—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate including semiconductor components sensitive to infrared radiation, light, electromagnetic radiation of shorter wavelength or corpuscular radiation and specially adapted either for the conversion of the energy of such radiation into electrical energy or for the control of electrical energy by such radiation
    • H01L27/144—Devices controlled by radiation
    • H01L27/146—Imager structures
    • H01L27/14601—Structural or functional details thereof
    • H01L27/1462—Coatings
    • H01L27/14621—Colour filter arrangements
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B5/00—Optical elements other than lenses
    • G02B5/20—Filters
    • G02B5/201—Filters in the form of arrays
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4015—Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
    • H—ELECTRICITY
    • H01—ELECTRIC ELEMENTS
    • H01L—SEMICONDUCTOR DEVICES NOT COVERED BY CLASS H10
    • H01L27/00—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate
    • H01L27/14—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate including semiconductor components sensitive to infrared radiation, light, electromagnetic radiation of shorter wavelength or corpuscular radiation and specially adapted either for the conversion of the energy of such radiation into electrical energy or for the control of electrical energy by such radiation
    • H01L27/144—Devices controlled by radiation
    • H01L27/146—Imager structures
    • H01L27/14601—Structural or functional details thereof
    • H01L27/14625—Optical elements or arrangements associated with the device
    • H01L27/14627—Microlenses
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
    • H04N23/11—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths for generating image signals from visible and infrared light wavelengths

Definitions

  • the present application relates to the technical field of image processing, and in particular, to a filter array, a mobile terminal, and a device.
  • Hyperspectral imaging technology or multispectral imaging technology is based on very multi-band image data technology. It concentrates advanced technologies in the fields of optics, optoelectronics, electronics, information processing, and computer science. It is an emerging technology that organically combines traditional two-dimensional imaging technology and spectroscopy technology. Hyperspectral imaging technology or multispectral imaging technology obtains the spectral information of the measured object while obtaining the spatial characteristic imaging of the object.
  • Hyperspectral imaging technology or multi-spectral imaging technology has the characteristics of ultra-multi-band, high spectral resolution, narrow band, wide spectral range, and unified spectrum, and the collected image information is rich. Because hyperspectral imaging or multispectral imaging technology can more accurately describe the spectral information of the physical world and achieve more accurate spectral detection, it is widely used in the fields of accurate color and material detection. However, the current solution for improving the resolution of a hyperspectral image or a multispectral image is expensive, so how to improve the resolution of the hyperspectral image or the multispectral image without increasing the cost needs to be solved urgently.
  • the embodiments of the present application provide an optical filter array, which can acquire high-resolution hyperspectral images at low cost.
  • a first aspect of the present application provides an optical filter array, which may include: a first type of filter array and a second type of filter array, the first type of filter array covers spectral information in a first wavelength range, and the second type of filter array
  • the filter array of the includes the spectral information of the second waveband range, and the spectral information of the second waveband range is different from the spectral information of the first waveband range.
  • the first type of color filter array covers the spectral information of the visible light band
  • the second type of color filter array can cover the spectral information of the near-infrared band
  • the first type of color filter array covers the first part of the spectral information of the visible light band
  • the second type of color filter array can cover the spectral information of the second part of the visible light band, and the first part of the visible light band is different from the second part of the visible light band.
  • the spectral information of the second waveband range and the spectral information of the first waveband range overlap within the preset waveband range.
  • the first type of color filter array covers the spectral information of the first part of the visible light band
  • the second type of color filter array may cover the spectral information of the second part of the visible light band
  • the first part of the visible light band and the second part of the visible light band partially overlap.
  • the first type of color filter array covers red spectral information, green spectral information, and blue spectral information
  • the second type of color filter array can cover red spectral information, as well as spectral information in the near-infrared band, that is, the second type
  • the spectral information covered by the color filter array may overlap with the spectral information covered by the color filter array of the first type within a preset range. The following will be explained from the perspective of channels.
  • One channel represents a spectral response function. In this application, it is also sometimes referred to as a channel representing a spectral information.
  • the channels included in the second type of color filter array and the first type of color filter array include The channels of the color filter array are completely different, or the color filter array of the second type includes the same part of the channels as the color filter array of the first type.
  • a preferred embodiment in this design is that the color filter array of the second type includes the same part of the channels as the color filter array of the first type.
  • the spectral information covered by the second type of color filter array is different from the spectral information covered by the first type of color filter array, or the second type of filter array covers different spectral information.
  • the spectral information covered by the color filter array of the first type coincides with the spectral information covered by the color filter array of the first type within a preset range, and the spectral information covered by the color filter array of the second type can be used for the spectrum covered by the color filter array of the first type. to supplement.
  • the solution provided by the present application does not require that the entire color filter array be a hyperspectral filter array. Since the more hyperspectral color filter arrays, the higher the cost of improving the resolution.
  • the entrance pupil radiance must reach a certain value, and its value is positively correlated with the image resolution and spectral resolution. If the spectral resolution is very high (hyperspectral or multispectral), in order to ensure sufficient energy imaging, the image resolution must be reduced, but currently on mobile phones or cameras, it is hoped to reduce the area of the sensor or filter array as much as possible. Therefore, the more hyperspectral color filter arrays, the more difficult it is to improve the resolution, and the higher the cost. Or it can also be understood that the narrow-band filter for hyperspectral imaging is difficult to process and small in size, so high-resolution hyperspectral cameras have great challenges and high costs.
  • the present application can reduce the number of hyperspectral color filter arrays included in the color filter array, and improve the resolution of the image through the hypospectral color filter array and the corresponding algorithm.
  • the second type of color filter array includes more channels than the first type of color filter array, and a low-resolution hyperspectral image can be obtained through the second type of color filter array, A high-resolution low-spectral image is obtained through the first type of color filter array, and then the data obtained through the first type of color filter array and the data obtained through the second type of color filter array are fused to obtain a high-resolution image hyperspectral images.
  • the number of channels included in the second type of color filter array is the same as the number of channels included in the first type of color filter array, or the number of channels included in the second type of color filter array is smaller than that of the first type of color filter array.
  • the number of channels included in the color filter array of the type, but the spectral information covered by the color filter array of the second type can be supplemented by the spectral information covered by the color filter array of the first type.
  • the data obtained by the color array and the data obtained by the second type of color filter array are fused to obtain a low-resolution hyperspectral image, and then the resolution of the image is improved according to the relevant algorithm to obtain a high-resolution hyperspectral image. image.
  • the filter array of the first type may include a filter array of three channels, and the filter array of the first type may include three channels of Each channel in is used to represent a kind of spectral information in the first band range. It can be known from the first possible implementation manner of the first aspect that a specific filter array of the first type is provided, which increases the diversity of solutions.
  • the three channels are the red channel R, the green channel G, the blue channel B, the yellow channel Y and the near-infrared channel. Any three channels in channel IR.
  • the number of channels that the filter array of the second type may include is greater than 3.
  • a positive integer, one channel that the filter array of the second type may include is used to represent a kind of spectral information in the second wavelength range.
  • the three channels are any one of an RGB channel, a RYB channel, and an RBGIR channel.
  • the first type of filter array may include a four-channel filter array, or the first type of filter array may include a three-channel filter array.
  • the number of channels of the second type of filter arrays is a positive integer greater than 4.
  • the four channels are the red channel R, the green channel G, the blue channel B, the yellow channel Y and the near-infrared channel. Any four channels in channel IR.
  • the number of channels of the filter array of the second type is different.
  • the filter array of the first type and the filter array of the second type are The filter arrays are all matrix structures.
  • the filter array may include multiple groups of filters of the first type.
  • the filter array may include multiple groups of the first type of filters.
  • the filter array may include multiple groups of the first type of filters.
  • a filter array and a plurality of groups of the second type of filter arrays are arranged in parallel.
  • the second type of filter array is Periodically distributed or aperiodically distributed.
  • the total area of the multiple groups of filter arrays of the first type is greater than The total area of the filter array of the second type of group.
  • the filter array of the first type may include color filters, A lens and a photoelectric conversion unit.
  • the photoelectric conversion unit is used to convert an optical signal into an electrical signal.
  • the color filter is located between the microlens and the photoelectric conversion unit.
  • one microlens corresponds to one color filter.
  • a microlens corresponds to a color filter, and corresponds to a plurality of photoelectric conversion units
  • a color filter corresponds to a plurality of microlenses, and corresponds to a plurality of photoelectric conversion units.
  • the filter array of the second type may include color filters, A lens and a photoelectric conversion unit.
  • the photoelectric conversion unit is used to convert an optical signal into an electrical signal.
  • the color filter is located between the microlens and the photoelectric conversion unit.
  • one microlens corresponds to one color filter. , and corresponds to a photoelectric conversion unit, or a microlens corresponds to a color filter, and corresponds to a plurality of photoelectric conversion units, or a color filter corresponds to a plurality of microlenses, and corresponds to a plurality of photoelectric conversion units.
  • a second aspect of the present application provides a camera module, the camera module may include a base and a filter array, the filter array is mounted on the base, and the filter array is in the first aspect or any possible implementation of the first aspect The described filter array.
  • the camera module may further include a processor and a memory, the processor and the memory are coupled, and program instructions related to image processing are stored in the memory.
  • program instructions stored in the memory are executed by the processor, illuminance estimation, white balance processing and demosaic processing are performed on the image obtained by the first type of filter array to obtain the first image, and the image obtained by the second type of filter array is processed.
  • the image is subjected to illumination estimation, white balance processing and demosaic processing to obtain a second image, and the first image and the second image are fused to obtain a high-resolution hyperspectral image.
  • a third aspect of the present application provides a camera, the camera may include a housing and a camera module, the camera module is accommodated in the housing, and the camera module is the camera module described in the second aspect.
  • the camera may further include a processor and a memory, the processor and the memory are coupled, and program instructions related to image processing are stored in the memory.
  • program instructions are executed by the processor, illuminance estimation, white balance processing and demosaic processing are performed on the image obtained by the first type of filter array to obtain the first image, and the image obtained by the second type of filter array is subjected to illuminance. estimation, white balance processing, and demosaicing processing to obtain a second image, and fusion processing is performed on the first image and the second image to obtain a high-resolution hyperspectral image.
  • a fourth aspect of the present application provides a mobile terminal, the mobile terminal may include a lens, a lens holder and a filter array, the lens is mounted on the lens holder, and the lens holder is arranged between the filter array and the lens.
  • the filter array is the filter array described in the first aspect or any possible implementation manner of the first aspect.
  • the mobile terminal may further include a processor and a memory, the processor and the memory are coupled, and program instructions related to image processing are stored in the memory.
  • program instructions are executed by the processor, illuminance estimation, white balance processing and demosaic processing are performed on the image obtained by the filter array of the first type to obtain the first image, and the image obtained by the filter array of the second type is processed.
  • a fifth aspect of the present application provides a smart car, the smart car may include a lens, a lens holder and a filter array, the lens is mounted on the lens holder, and the lens holder is arranged between the filter array and the lens.
  • the filter array is the filter array described in the first aspect or any possible implementation manner of the first aspect.
  • the smart car may further include a processor and a memory, the processor and the memory are coupled, and program instructions related to image processing are stored in the memory.
  • program instructions are executed by the processor, illuminance estimation, white balance processing and demosaic processing are performed on the image obtained by the filter array of the first type to obtain the first image, and the image obtained by the filter array of the second type is processed.
  • a sixth aspect of the present application provides a monitoring device, the monitoring device may include a lens, a lens holder and a filter array, the lens is mounted on the lens holder, and the lens holder is arranged between the filter array and the lens.
  • the filter array is the filter array described in the first aspect or any possible implementation manner of the first aspect.
  • the monitoring device may further include a processor and a memory, the processor and the memory are coupled, and program instructions related to image processing are stored in the memory, and when the memory is When the stored program instructions are executed by the processor, illuminance estimation, white balance processing and demosaic processing are performed on the image obtained by the filter array of the first type to obtain the first image, and the image obtained by the filter array of the second type is processed. Perform illumination estimation, white balance processing and demosaicing processing to obtain a second image, and perform fusion processing on the first image and the second image to obtain a high-resolution hyperspectral image.
  • a seventh aspect of the present application provides an electronic device, the electronic device may include a lens, a lens holder and a filter array, the lens is mounted on the lens holder, and the lens holder is arranged between the filter array and the lens.
  • the filter array is the filter array described in the first aspect or any possible implementation manner of the first aspect.
  • the electronic device may further include a processor and a memory, the processor and the memory are coupled, and program instructions related to image processing are stored in the memory.
  • program instructions are executed by the processor, illuminance estimation, white balance processing and demosaic processing are performed on the image obtained by the filter array of the first type to obtain the first image, and the image obtained by the filter array of the second type is processed.
  • Fig. 1 is the principle schematic diagram of Bayer array
  • FIG. 2 is a schematic structural diagram of a filter array according to an embodiment of the present application.
  • FIG. 3 is a schematic structural diagram of another filter array provided by an embodiment of the present application.
  • FIG. 4 is a schematic structural diagram of another filter array provided by an embodiment of the present application.
  • FIG. 5 is a schematic structural diagram of another filter array provided by an embodiment of the present application.
  • FIG. 6 is a schematic structural diagram of another filter array provided by an embodiment of the present application.
  • FIG. 7 is a schematic structural diagram of another filter array provided by an embodiment of the present application.
  • FIG. 8 is a schematic structural diagram of another filter array provided by an embodiment of the present application.
  • FIG. 9 is a schematic structural diagram of another filter array provided by an embodiment of the present application.
  • FIG. 10 is a schematic structural diagram of another filter array provided by an embodiment of the present application.
  • FIG. 11 is a schematic structural diagram of another filter array provided by an embodiment of the present application.
  • FIG. 12 is a schematic structural diagram of another filter array provided by an embodiment of the present application.
  • FIG. 13 is a schematic structural diagram in which the first type of array provided by the embodiment of the present application is 3-channel or 4-channel;
  • FIG. 14 is a schematic structural diagram of a first type of filter array or a second type of filter array provided by an embodiment of the present application;
  • 15 is a schematic structural diagram of a first type of filter array or a second type of filter array provided by an embodiment of the present application;
  • 16 is a schematic structural diagram of a first type of filter array or a second type of filter array provided by an embodiment of the present application;
  • 17 is a schematic structural diagram of a color filter
  • Fig. 18 is another kind of structural representation of color filter
  • Fig. 19 is another kind of structural representation of color filter
  • FIG. 20 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • the terms “installed”, “connected”, “connected”, “fixed”, “arranged” and other terms should be understood in a broad sense, for example, it may be a fixed connection, or It can be a detachable connection, or integrated; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, or it can be the internal connection of two elements or the interaction between the two elements. .
  • installed e.g., it may be a fixed connection, or It can be a detachable connection, or integrated; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, or it can be the internal connection of two elements or the interaction between the two elements.
  • Bayer array refers to a common method used to collect digital images when a charge coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) is used as a light sensor.
  • CCD charge coupled device
  • CMOS complementary metal-oxide semiconductor
  • the image sensor converts light into current. The brighter the light, the larger the current value, and the darker the light, the smaller the current value. However, the image sensor has a serious defect: it can only sense the intensity of light, but not the wavelength of light. Since the color of light is determined by the wavelength, the image propagator cannot record the color.
  • the scheme is to use only one image sensor, and in front of the image sensor, a layer of color filter array (CFA) is set, and the Bayer array does not put three on each pixel (pixel) A color filter, but a filter that places a single color on each pixel at intervals.
  • each pixel of the image sensor is an independent photoelectric conversion unit, which can only output the collected light from intensity to voltage signal.
  • the pixel itself is a monochromatic image sensor. It can record color because before photoelectric conversion, the light intensity collection is completed by projecting light of a specific color onto the pixel, and thus a specific "monochromatic light" is established. "Corresponding relationship with the light intensity obtained by the photoelectric conversion unit. In this way, each channel can get a picture with partial value vacancies, and the values of these vacancies can be filled by various interpolation methods.
  • the Bayer array will be described in detail below with reference to FIG. 1 .
  • the implementation scheme is a color filter Bayer array composed of filters of different colors.
  • the filter is sometimes referred to as a color filter, and the filter is referred to as a color filter.
  • the filter is referred to as a color filter.
  • the Bayer array usually has only three color filters of red, green and blue.
  • the color filters correspond to the pixels of the image sensor one-to-one, and each color filter can only pass one of the three types of light: red (red, R), green (green, G), and blue (blue, B).
  • the arrangement of the dots that can be filtered by different colors is regular.
  • the green dots are distributed in the upper right and lower left of each green dot, the blue dots are distributed up and down, and the red dots are distributed to the left and right.
  • There are twice as many green dots as red or blue dots because studies have found that the human eye is most sensitive to green.
  • the image obtained now has only three colors, which obviously cannot express the colors of the real world. Therefore, it is also necessary to approximately restore its original value according to the color around each pixel, and finally obtain a color image. This process is called "demosaicing".
  • Demosaicing can be implemented in different ways, simple interpolation, such as a green filter pixel where the green component is accurately measured, while the red and blue components of that pixel are obtained from the neighborhood.
  • the red value of a green pixel can be calculated by interpolating two adjacent red pixels, and similarly, the blue value can be calculated by interpolating two adjacent blue pixels.
  • the red component R33, the green component G33 and the blue component B33 in R33 can be expressed by the following formula:
  • the red, green and blue components of each pixel can be obtained, and the final output color image can be obtained.
  • the hyperspectral imaging technology has the characteristics of ultra-multi-band, high spectral resolution, narrow band, wide spectral range, and integrated spectrum, and the collected image information is rich. Therefore, it has a wide range of applications in the fields of accurate color and material detection.
  • the resolution of the image obtained by hyperspectral imaging technology is low, and the cost of improving the resolution is high.
  • the present application provides an optical filter array, which includes two types of type arrays.
  • the first type of filter array is used to obtain high-resolution low-spectral images.
  • the first type of filter array may be the above-mentioned Bayer array
  • the second type of filter array is used for To acquire low-resolution hyperspectral images, such as this second type of filter array is a multispectral filter array.
  • a filter array with such a structure can obtain a high-resolution hyperspectral image in one imaging process, which not only has a simple structure, but also reduces the cost of improving the resolution of the hyperspectral image.
  • An optical filter array provided by an embodiment of the present application includes: a first type of filter array and a second type of filter array, the first type of filter array covers spectral information of a first wavelength range, and the second type of filter array The light array covers the spectral information of the second waveband range, and the spectral information of the second waveband range is different from the spectral information of the first waveband range.
  • the first type of color filter array covers the spectral information of the visible light band
  • the second type of color filter array can cover the spectral information of the near-infrared band
  • the first type of color filter array covers the first part of the spectral information of the visible light band
  • the second type of color filter array can cover the spectral information of the second part of the visible light band, and the first part of the visible light band is different from the second part of the visible light band.
  • the spectral information of the second waveband range and the spectral information of the first waveband range overlap within the preset waveband range.
  • a channel represents a spectral response function, and this application is sometimes referred to as a channel representing a spectral information
  • the channels included in the color filter array of the second type are the same as those included in the color filter array of the first type.
  • the channels are completely different, or the color filter array of the second type includes channels that are partially the same as the channels included in the color filter array of the first type.
  • the color filter array of the second type includes more channels than the color filter array of the first type, and a low-resolution hyperspectral image can be obtained by the color filter array of the second type, by The first type of color filter array obtains high-resolution low-spectral images, and then the data obtained through the first type of color filter array and the data obtained through the second type of color filter array are fused to obtain a high-resolution image. Hyperspectral image.
  • the number of channels included in the second type of color filter array is the same as the number of channels included in the first type of color filter array, or the number of channels included in the second type of color filter array is smaller than that of the first type of color filter array.
  • the data obtained by the color array and the data obtained by the second type of color filter array are fused to obtain a low-resolution hyperspectral image, and then the resolution of the image is improved according to the relevant algorithm to obtain a high-resolution hyperspectral image. image.
  • the filter array of the first type includes less spectral information
  • the filter array of the second type includes more spectral information.
  • the filter array of the first type can be considered to be used for acquiring hypospectral images and the filter array of the second type is used to acquire hyperspectral images.
  • the number of channels included in the filter array of the first type is 3 or 4 channels.
  • the second type of filter array includes a large number of channels, such as 8 or 10 channels. If the number of channels is large, it can be considered that the filter array of the second type is used to obtain hyperspectral images or multispectral images. In addition, the large number of channels requires high-resolution images to be acquired at a high cost.
  • the first type of filter array listed above includes 3 or 4 channels
  • the second type of filter array includes 8 or 10 channels for illustration only, and does not mean that the first type
  • the number of channels of the filter array and the second type of filter array is limited.
  • the above example is to illustrate that the smaller the number of channels, the lower the cost of making a high-resolution image, but the less spectral information it contains, that is, a low-spectral image; the more the number of channels, the greater the cost of making a high-resolution image, However, it includes more spectral information, that is, hyperspectral images.
  • a first type of filter array with a small number of channels is used to obtain a high-resolution low-spectral image, and the resolution is increased while reducing the cost
  • a second type of filter array with a large number of channels is used to obtain the image.
  • Low-resolution hyperspectral images for more spectral information.
  • a high-resolution image refers to an image with a vertical resolution greater than or equal to 720, which can also be called a high-definition image, such as 1280 ⁇ 720 and 1920 ⁇ 108, where the width (that is, the horizontal resolution), followed by the multiplication sign to indicate the height (ie vertical resolution).
  • Low-resolution images are those with a vertical resolution of less than 720.
  • the hyperspectral image may cover part of visible light information, or the hyperspectral image may cover the entire visible light information, or the hyperspectral image may cover all visible light information and near-infrared light information, or the hyperspectral image may cover part of the visible light information information and near-infrared light information.
  • the low-spectral image can cover part of the visible light information, or the low-spectral image can cover the entire visible light information, and when the hyperspectral image covers part of the visible light information, or the hyperspectral image covers the entire visible light information, the low-spectral image can cover less spectral information than the hyperspectral image. Spectral information that the image can cover.
  • Both hyperspectral and low spectrum are related to the number of channels, that is, hyperspectral is acquired through multiple channels, and low spectrum is acquired through fewer channels. Since the higher the number of channels, the higher the cost of increasing the resolution of the image, the smaller the number of channels, the lower the cost of increasing the resolution of the image. Therefore, it can be considered that the resolution of the image and the level of the spectrum are related to the number of channels.
  • the number of channels of the filter array of the first type is a positive integer not greater than M
  • the number of channels of the filter array of the second type is a positive integer greater than M, where M is a positive integer.
  • M is 1, the number of channels of the first type of filter array is 1, and the number of channels of the second type of filter array is a positive integer greater than 1.
  • the number of channels of the second type of filter array can be 2 , 3, 8, 10, 20 and so on.
  • M is 7, the number of channels of the first type of filter array is a positive integer not greater than 7, such as 1, 3, 4, 6, etc., and the number of channels of the second filter array is greater than 7 A positive integer, such as 9, 10, 30, etc.
  • the selection of the number of channels of the first type of filter array determines the cost of improving the image resolution
  • the selection of the number of channels of the second type of filter array determines the spectral information included in the image.
  • the first type of filter array is a 3-channel filter array, wherein, in a possible embodiment, the three channels are red channels (red, R), the green channel (green, G), the blue channel (blue, B), the yellow channel Y, and any three of the near-infrared channel IR.
  • the number of channels of the second type of filter array is 4 to 25.
  • the filter array of the second type is shown with a channel number of 16.
  • the first type of filter arrays may all be a certain type of 3-channel filter arrays. For example, as shown in FIG.
  • the first type of filter arrays are RGB three channel
  • the filter arrays of the first type may all be filter arrays of three channels of RGB.
  • the filter array of the first type may include multiple types of filter arrays with 3 channels.
  • the filter array of the first type includes three channels of RGB and three channels of RYB.
  • the filter array provided in this application which can include a first-type filter array with three RGB channels, or a first-type filter array with three RYB channels.
  • the filter array of the first type is all displayed with three channels of RGB.
  • the number of channels of the second type of filter array is 16. In a possible implementation, it can be considered that each channel represents a color, and the 16 colors are different from each other.
  • the filter array includes a plurality of groups of the first type of filter arrays and a plurality of groups of the second type of filter arrays, and the filter arrays of the first type and the second type of filter arrays
  • the positional relationship is that a plurality of groups of filter arrays of the first type surround the filter arrays of the second type.
  • the filter array of the first type surrounds the filter array of the second type, and the closer to the center of the filter array, the higher the distribution density of the filter array of the second type.
  • the larger the density the smaller the distribution density of the first type of filter array.
  • the filter array of the second type is arranged in the center of the filter array, and the filter array of the first type is arranged around the filter array.
  • the positional relationship between the filter arrays of the first type and the filter arrays of the second type is that a plurality of groups of filter arrays of the second type surround the filter arrays of the first type.
  • the light array as shown in FIG. 4 , is shown with the first type of filter array as an RGB three-channel, and the second filter array as an 8-channel filter array.
  • the closer to the center of the filter array the greater the distribution density of the first type of filter array, and the smaller the distribution density of the second type of filter array.
  • the filter array of the first type is arranged in the center of the filter array
  • the filter array of the second type is arranged around the filter array.
  • the filter array of the second type surrounding the filter array of the first type may include the filter array of the second type half surrounding the filter array of the first type.
  • the filter array of the second type is located in the first type of filter array. The left, right and lower sides of the filter array of the type, but not the upper side without the filter array of the first type.
  • the positional relationship between the filter arrays of the first type and the filter arrays of the second type is multiple groups of filter arrays of the first type and multiple groups of the filter arrays of the second type
  • the filter arrays are arranged side by side.
  • the filter array of the second type is periodically distributed.
  • any two groups of filter arrays of the second type in the filter array are separated by the same number of filter arrays of the first type.
  • FIG. 6 a schematic diagram of the structure of a second type of filter array with periodic distribution is shown.
  • the filter array of the second type may also not be periodically distributed.
  • FIG. 7 it can be embodied that any two groups of filter arrays of the second type in the filter array are separated by different numbers of filter arrays of the first type.
  • the total area of the plurality of groups of the first type of filter arrays is greater than the total area of the plurality of groups of the second type of filter arrays.
  • the total area of all the filter arrays of the first type is larger than the total area of the filter arrays of the second type, and the total area of the filter arrays of the first type is larger, so the cost can be better saved , to obtain hyperspectral high-resolution images.
  • the filter array of the first type is a matrix structure, and the specific description is given by a matrix structure of 2 times 2.
  • the filter array of the second type is also a matrix structure. As shown in FIG. 2, the description is made with a matrix structure of 4 by 4.
  • the filter array of the second type may also have a 1-by-2 matrix structure, a 3-by-3 matrix structure, a 5-by-5 matrix structure, and so on.
  • the filter array of the second type in the embodiment of the present application may also have other structures.
  • the filter array of the second type is an “L” type structure, or as shown in FIG. 9 , the filter array of the second type can be a “concave” type structure.
  • the second-type filter array can also be other structures, for example, the second-type filter array can also be an “E” type structure .
  • the first type of filter array is a 4-channel filter array, wherein, in a possible embodiment, the 4 channels are red channel R, green Channel G, blue channel B, yellow channel Y, and any four of the near-infrared channel IR.
  • the number of channels of the second type of filter array is 4 to 25.
  • the filter array of the second type is shown with a channel number of 16.
  • the filter arrays of the first type can all be a certain type of 4-channel filter arrays. For example, as shown in FIG. 10 , the filter arrays of the first type are four RGBIR filter arrays. Channels, in the filter array shown in FIG.
  • the first type of filter arrays may all be filter arrays of four channels of RGBIR.
  • the first type of filter array may include multiple types of 4-channel filter arrays.
  • the first type of filter array includes four channels of RGBIR and four channels of RYBIR.
  • the provided filter array may include a first-type filter array with four channels of RGBIR, or a first-type filter array with four channels of RYBIR.
  • the filter array may include a three-channel filter array and a four-channel filter array.
  • the filter array includes both a three-pass filter array and a four-channel filter array. Four-channel filter array.
  • the first type of filter array includes a 4-channel filter array
  • the structure of the second type of filter array and the positional relationship between the first filter array and the second filter array please refer to Section 1 above. It should be understood that one type of filter array includes a 3-channel filter array, which will not be repeated in this embodiment of the present application.
  • FIG. 13 a to FIG. 13 c several typical first-type filter arrays are structural schematic diagrams with 3 channels or 4 channels.
  • the number of channels included in the second type of color filter array may be less than the number of channels included in the first type of color filter array, or the number of channels included in the second type of color filter array
  • the number of channels included can be the same as the number of channels included in the first type of color filter array, but the spectral information covered by the second type of color filter array can be supplemented by the spectral information covered by the first type of color filter array.
  • the correlation algorithm fuses the data obtained through the first type of color filter array and the data obtained through the second type of color filter array to obtain a hyperspectral high-resolution image.
  • the positional relationship between the first type of filter array and the second type of filter array can refer to the first type of filter array and the first type of filter array in the first design method.
  • the positional relationship between the second type of filter arrays should be understood, and details will not be repeated here.
  • the filter array provided by the present application may include any one of the structures described in FIGS. 2 to 12 , or may include any of the structures described in FIGS. 2 to 12 . combination.
  • the filter array provided by the present application may include both the structure described in FIG. 3 and the structure described in FIG. 4 .
  • the above mainly introduces the number of channels of the first type of filter array, the number of channels of the second type of filter array, the positional relationship between the first type of filter array and the second type of filter array, the first The structure of the filter array of the type, and the structure of the filter array of the second type.
  • the specific structures of the filter array of the first type and the filter array of the second type are described below.
  • the first type of filter array includes a color filter, a microlens and a photoelectric conversion unit, the photoelectric conversion unit is used to convert an optical signal into an electrical signal, and the color filter is located between the microlens and the photoelectric conversion unit
  • the filter array of the first type one microlens corresponds to one color filter and one photoelectric conversion unit, or one microlens corresponds to one color filter and corresponds to multiple photoelectric conversion units, or one filter
  • the color sheet corresponds to a plurality of microlenses and corresponds to a plurality of photoelectric conversion units. The following description will be made with reference to FIGS. 14 to 16 .
  • a color filter is arranged above the photoelectric conversion unit, and a microlens is arranged above the color filter, and the microlens, the color filter and the photoelectric conversion unit are in a one-to-one correspondence, that is, a photoelectric conversion unit
  • a color filter is arranged above the color filter, and a microlens is arranged above the color filter.
  • the purpose and function of the microlens is to improve the light energy received by the photoelectric conversion unit, which can also be said to be the number of photons in the microscopic field.
  • Each photoelectric conversion unit is limited in structure, and it is impossible to seamlessly connect with other photoelectric conversion units, so a large amount of photosensitive area will be lost on the entire sensor.
  • a color filter is provided above the plurality of photoelectric conversion units.
  • a color filter is provided above every two photoelectric conversion units, and a color filter is provided above each color filter.
  • a microlens As shown in FIG. 16 , a color filter is provided above the plurality of photoelectric conversion units. In a preferred embodiment, a color filter is provided above every two photoelectric conversion units, and a color filter is provided above each color filter.
  • the second type of filter array includes a color filter, a microlens and a photoelectric conversion unit, the photoelectric conversion unit is used to convert the optical signal into an electrical signal, and the color filter is located between the microlens and the photoelectric conversion unit
  • the filter array of the second type one microlens corresponds to one color filter and one photoelectric conversion unit, or one microlens corresponds to one color filter and corresponds to multiple photoelectric conversion units, or one filter
  • the color sheet corresponds to a plurality of microlenses and corresponds to a plurality of photoelectric conversion units.
  • the filter array of the second type can be understood with reference to FIG. 14 to FIG. 16 , and details are not repeated here.
  • Color filters are optical devices used to select the desired wavelength band of radiation.
  • the color filter is made of chemical dye materials.
  • the color filter is made of plastic or glass and added with special dyes.
  • the red color filter can only pass red light, and so on. .
  • the transmittance of the glass sheet is similar to that of air originally, and all colored light can pass through, so it is transparent, but after dyeing, the molecular structure changes, the refractive index also changes, and the passage of some colored light changes. For example, when a beam of white light passes through a blue filter, a beam of blue light is emitted, while very little green and red light is emitted, and most of it is absorbed by the filter.
  • the color filter is made of structured optics.
  • the color filter can be a nano-hole structure, a waveguide grating structure, a multi-layer interference film structure, etc., any of which can be The structure that realizes the color filtering function can be used in any embodiment of the present application.
  • the color filter can also be an FP cavity (fabry-perot cavity) structure and so on.
  • the FP cavity is a device that utilizes the phenomenon of multi-beam interference.
  • FIG. 17 it is a schematic diagram of the structure of the color filter fabricated by the nano-hole structure.
  • the interference of the surface plasmon polaritons of adjacent holes acts as a trigger for the selective transmission of light.
  • the nanopore type can design different filters by adjusting the pore size of the nanopores processed by metal materials and the spacing between the nanopores. As shown in FIG. 18 , it is a color filter made of a waveguide grating structure.
  • the introduction of surface plasmon metal into the micro-nano grating structure makes the micro-nano grating superimpose the absorption and scattering of surface plasmon on the basis of optical diffraction.
  • the micro-nano periodic grating structure can be used as a dispersion medium to select the wavelength band.
  • this filtering structure is a color filter made from a multilayer interference film structure.
  • this filtering structure can be divided into two categories: based on localized surface The metal/medium/metal (MIM) type of plasmon resonance, and the Fabry-Perot (FP) type based on multi-beam interference.
  • the top metal of the MIM filter structure generally has a micro-nano structure.
  • the local surface plasmon resonance of the top layer and the anti-parallel surface current of the upper and lower layers of the metal work together to make the MIM structure absorb light in a specific wavelength band, thereby realizing the filtering function.
  • the FP filter structure uses two layers of semi-transparent and semi-reflective films to form a reflective layer, and a cut-off layer in the middle to form a resonator, and uses multi-beam interference to transmit or reflect constructive light. This mode does not require the nanostructure of the top layer, and generally only considers the optical properties and manufacturing process of each layer.
  • High-resolution hypospectral images and low-resolution hyperspectral images can be obtained through the filter array provided by the above embodiments of the present application, and fusion processing is performed on the high-resolution hypospectral images and the low-resolution hyperspectral images , that is, high-resolution hyperspectral images can be obtained.
  • the solution provided by the present application does not require multiple imaging, and only needs one imaging to obtain a high-resolution hyperspectral image with high resolution.
  • ISP integrated signal processors
  • ISP refers to hardware dedicated to processing the signal from the sensor and generating the final image, usually integrated as a module into a system-on-chip. The various processing that will be described next is carried out at the ISP.
  • the light enters the lens and reaches the sensor with the filter array provided by the embodiments of the present application to obtain the most primitive electrical signal.
  • Illumination estimation, demosaicing and noise reduction are performed on this signal, and a picture in raw format will be obtained, and then white balance processing is performed on it to obtain an image.
  • the illuminance estimation is to make the signal meet the preset standard (international standardization organization, ISO), for example, the standard can be the standard of the camera's sensitivity, in order to achieve the corresponding setting, the camera will receive the signal to gain .
  • ISO international standardization organization
  • the demosaic process can be understood with reference to the demosaic process mentioned above in the introduction of the Bayer array, and details are not repeated here. There are many ways to reduce noise.
  • Color deviation can be corrected through white balance processing.
  • white balance is to correct the point (white point) of equal RGB, which can be corrected by scaling the three channels respectively.
  • White balance processing is to adjust the color circuit inside the camera to offset the color cast by adjusting the color circuit inside the camera under different color temperature conditions, which is closer to the visual habits of the human eye.
  • White balance can also be simply understood as under any color temperature conditions, the standard white captured by the camera is adjusted by the circuit, so that it is still white after imaging.
  • the multispectral image can be regarded as blurring and downsampling the spectral image to be reconstructed, and the hypospectral image can be regarded as spectrally downsampling the spectral image to be reconstructed. Therefore, the spectral image fusion problem can also be regarded as using two degraded spectral images to reconstruct the spectral image before the degradation.
  • the observed multispectral image Yh can be regarded as the blurred, downsampled and noised data of the image Z to be reconstructed.
  • the observed hypospectral image Ym can be viewed as spectrally downsampled and noised data of the image Z to be reconstructed. Therefore, the spectral image degradation model connecting Z with Yh and Ym can be expressed as:
  • Nm and Nh represent Gaussian noise.
  • the spectral downsampling matrix R and the spatial blur matrix B are preset.
  • the spectral downsampling matrix and the spatial blur matrix can be estimated by using methods such as spectral decomposition, and high-resolution hyperspectral images can be obtained.
  • spectral decomposition is introduced into spectral fusion because it can decompose the spectral image into the form of multiplying two low-dimensional matrices.
  • Spectral decomposition aims to decompose each pixel in a spectral image into a series of component spectral features, called endmembers and abundance coefficients. Endmembers are usually assumed to be pure substances in the image, and the abundance coefficient of each pixel represents the proportion of different substances in this pixel.
  • the spectral image Z to be reconstructed can be expressed as:
  • ⁇ h represents the spectral band of the multispectral image.
  • ns is usually small, which means that E and X are relatively low-dimensional
  • ⁇ m represents the low spectral spectral band.
  • the spectral image Z to be reconstructed is a linear combination of an endmember matrix E representing spectral information and an abundance matrix X representing spatial information. Also because the observed multispectral images have high spatial resolution, the observed hyperspectral images have high spectral resolution. Therefore, when reconstructing spectral images with high spatial and high spectral resolution, two linear optimization models are used to extract the abundance matrix of the multispectral image and the endmember matrix of the hyperspectral image, respectively.
  • the image obtained by the first type of filter array is used as the low-spectral image Ym
  • the image obtained by the second type of filter array is used as the multi-spectral image Yh. Protocol methods and algorithms to obtain high-resolution multispectral images.
  • the image obtained by the first type of filter array is taken as the low spectral image Ym
  • the second type of filter array and the adjacent one of the second type of filter array The image jointly acquired by the first type of filter array is taken as the multispectral image Yh, and the high-resolution multispectral image is acquired by the above scheme method and algorithm.
  • the present application also provides an electronic device, the electronic device may include a lens, a lens holder and a filter array, the lens is mounted on the lens holder, and the lens holder is arranged between the filter array and the lens.
  • the filter array is the filter array described in the embodiments of the present application.
  • the electronic device provided in this application can be any device that needs to install a filter array, such as a mobile terminal, a smart car, and a monitoring device.
  • FIG. 20 is a schematic diagram of a hardware structure of a communication device provided by an embodiment of the present application. It includes: a processor 2001 and a memory 2002 .
  • the processor 2001 includes but is not limited to a central processing unit (CPU), a network processor (NP), an application-specific integrated circuit (ASIC) or a programmable logic device (programmable logic device, PLD) one or more.
  • the above-mentioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a general-purpose array logic (generic array logic, GAL) or any combination thereof.
  • CPLD complex programmable logic device
  • FPGA field-programmable gate array
  • GAL general-purpose array logic
  • Memory 2002 may be read-only memory (ROM) or other type of static storage device that can store static information and instructions, random access memory (RAM) or other type of static storage device that can store information and instructions It can also be an electrically erasable programmable read-only memory (electrically programmable read-only memory, EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, Optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or capable of carrying or storing desired program code in the form of instructions or data structures and capable of Any other medium that can be accessed by a computer, but is not limited to this.
  • ROM read-only memory
  • RAM random access memory
  • EEPROM electrically erasable programmable read-only memory
  • CD-ROM compact disc read-only memory
  • Optical disc storage including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs
  • the memory may exist independently and be connected to the processor 2001 through a communication line.
  • the memory 2002 may also be integrated with the processor 2001. If the memory 2002 and the processor 2001 are independent devices, the memory 2002 and the processor 2001 are connected, for example, the memory 2002 and the processor 2001 can communicate through a communication line.
  • the electronic device may be a smart car
  • the smart car may include a processor and a memory
  • the processor and the memory are coupled
  • the memory stores program instructions related to image processing, when the memory stores program instructions
  • illuminance estimation, white balance processing and demosaic processing are performed on the image obtained by the first type of filter array to obtain a high-resolution low-spectral image
  • the image obtained by the second type of filter array is processed.
  • the electronic device may be a monitoring device, the monitoring device may include a processor and a memory, the processor and the memory are coupled, and the memory stores program instructions related to image processing, when the memory stores program instructions
  • illuminance estimation, white balance processing and demosaic processing are performed on the image obtained by the first type of filter array to obtain a high-resolution low-spectral image
  • the image obtained by the second type of filter array is processed.
  • the electronic device may be a mobile terminal, and the mobile terminal may include a processor and a memory, the processor and the memory are coupled, and program instructions related to image processing are stored in the memory, when the program instructions stored in the memory are When executed by the processor, illuminance estimation, white balance processing and demosaic processing are performed on the image obtained by the first type of filter array to obtain a high-resolution low-spectral image, and the image obtained by the second type of filter array is processed. Perform illumination estimation, white balance processing and demosaicing to obtain low-resolution hyperspectral images, and perform fusion processing of high-resolution hypospectral images and low-resolution hyperspectral images to obtain high-resolution hyperspectral images image.

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Power Engineering (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Hardware Design (AREA)
  • Condensed Matter Physics & Semiconductors (AREA)
  • Electromagnetism (AREA)
  • Microelectronics & Electronic Packaging (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Optics & Photonics (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • Theoretical Computer Science (AREA)
  • Color Television Image Signal Generators (AREA)

Abstract

本申请公开了一种滤光阵列，包括：第一类型的滤光阵列和第二类型的滤光阵列，第一类型的滤光阵列涵盖第一波段范围的光谱信息，第二类型的滤光阵列涵盖第二波段范围的光谱信息，第二波段范围的光谱信息与第一波段范围的光谱信息不同，或者第二波段范围的光谱信息与第一波段范围的光谱信息在预设波段范围内重叠。本申请实施例还提供一种移动终端，智能汽车，监控设备以及电子设备。通过本申请提供的方案，可以以低成本获取高分辨率的高光谱图像。

Description

一种滤光阵列、移动终端以及设备 技术领域
本申请涉及图像处理技术领域，尤其涉及一种滤光阵列、移动终端以及设备。
背景技术
高光谱成像技术或者多光谱成像技术是基于非常多波段的影像数据技术。它集中了光学、光电子学、电子学、信息处理、计算机科学等领域的先进技术，是传统的二维成像技术和光谱技术有机地结合在一起的一门新兴技术。高光谱成像技术或者多光谱成像技术在获得物体空间特征成像的同时，也获得了被测物体的光谱信息。
高光谱成像技术或者多光谱成像技术具有超多波段、高的光谱分辨率、波段窄、光谱范围广和图谱合一等特点，采集到的图像信息量丰富。由于高光谱成像或者多光谱成像技术可以更准确刻画物理世界的光谱信息，实现更准确光谱检测，从而在准确色彩、物质检测等领域有广泛应用。但是，目前提升高光谱图像或者多光谱图像的分辨率的方案成本高，因此如何提升高光谱图像或者多光谱图像的分辨率且不增加成本亟待解决。
发明内容
本申请实施例提供一种滤光阵列，以低成本获取高分辨率的高光谱图像。
为达到上述目的，本申请实施例提供如下技术方案：
本申请第一方面提供一种滤光阵列，可以包括：第一类型的滤光阵列和第二类型的滤光阵列，第一类型的滤光阵列涵盖第一波段范围的光谱信息，第二类型的滤光阵列涵盖第二波段范围的光谱信息，第二波段范围的光谱信息与第一波段范围的光谱信息不同。比如第一类型的滤色阵列涵盖可见光波段的光谱信息，第二类型的滤色阵列可以涵盖近红外波段的光谱信息，再比如，第一类型的滤色阵列涵盖第一部分可见光波段的光谱信息，第二类型的滤色阵列可以涵盖第二部分可见光波段的光谱信息，且第一部分可见光波段不同于第二部分可见光波段。或者第二波段范围的光谱信息与第一波段范围的光谱信息在预设波段范围内重叠。比如，第一类型的滤色阵列涵盖第一部分可见光波段的光谱信息，第二类型的滤色阵列可以涵盖第二部分可见光波段的光谱信息，且第一部分可见光波段与第二部分可见光波段部分重叠。举例说明，假设第一类型的滤色阵列涵盖红色光谱信息，绿色光谱信息，蓝色光谱信息，第二类型的滤色阵列可以涵盖红色光谱信息，以及近红外波段的光谱信息，即第二类型的滤色阵列涵盖的光谱信息可以和第一类型的滤色阵列涵盖的光谱信息在预设范围内重合。下面从通道的角度进行解释，一个通道代表一种光谱响应函数，本申请有时也称一个通道代表一种光谱信息，则第二类型的滤色阵列包括的通道与第一类型的滤色阵列包括的通道完全不同，或者第二类型的滤色阵列包括的通道与第一类型的滤色阵列包括的通道部分相同。在第二类型的滤色阵列包括的通道与第一类型的滤色阵列包括的通道部分相同这种设计中一个优选的实施方式是第二类型的滤色阵列和第一类型的滤色阵列有一个相同的通道，或者有两个相同的通道。由第一方面可知，本申请提供的滤光阵列中包括两种类型的滤光阵列，第二类型的滤色阵列涵盖的光谱信息与第一类型滤色阵列涵盖的光谱信息不同，或者第二类型的滤色阵列涵盖的光谱信息与第一类型滤色阵列涵盖的光谱信息在预设范围内重合，则第二类型的滤色阵列涵盖的光谱信息可以对第一类型 滤色阵列涵盖的光谱进行补充。为了获取高光谱高分辨率的图像，本申请提供的方案，不需要整个滤色阵列全部都是高光谱的滤光阵列。由于高光谱的滤色阵列越多，提升分辨率的成本越高。这是因为传感器要想成像,入瞳辐亮度必须达到一定的值,其值与图像分辨率和光谱分辨率成正相关关系。如果光谱分辨率很高(高光谱或者多光谱),为了保证足够的能量成像那么图像分辨率就必须得降低，但是目前手机上或者相机上，希望尽可能减小传感器或者滤光阵列的面积，所以，高光谱的滤色阵列越多，提升分辨率的难度越大，成本越高。或者也可以理解为高光谱成像的窄带滤光片加工难度大，做小尺寸难度大，因此高分辨高光谱相机具有很大挑战，成本较高。本申请可以减少滤色阵列中包括的高光谱滤色阵列的数目，通过低光谱的滤色阵列以及相应的算法提升图像的分辨率。比如在一种设计中，第二类型的滤色阵列包括的通道数多于第一类型的滤色阵列包括的通道数，可以通过第二类型的滤色阵列获取低分辨率高光谱的图像，通过第一类型的滤色阵列获取高分辨率低光谱的图像，再对通过第一类型的滤色阵列获取的数据以及通过第二类型的滤色阵列获取的数据进行融合，以得到高分辨率的高光谱图像。再比如在另一种设计中，第二类型的滤色阵列包括的通道数虽然和第一类型的滤色阵列包括的通道数相同，或者第二类型的滤色阵列包括的通道数小于第一类型的滤色阵列包括的通道数，但是第二类型的滤色阵列涵盖的光谱信息可以对第一类型的滤色阵列涵盖的光谱信息进行补充，则可以通过相关算法将通过第一类型的滤色阵列获取的数据，以及通过第二类型的滤色阵列获取的数据进行融合，以得到高光谱的低分辨率图像，再根据相关算法提升图像的分辨率，以得到的高光谱的高分辨率图像。
可选地，结合上述第一方面，在第一种可能的实施方式中，第一类型的滤光阵列可以包括三个通道的滤光阵列，第一类型的滤光阵列可以包括的三个通道中的每一个通道分别用于表示第一波段范围内的一种光谱信息。由第一方面第一种可能的实现方式可知，给出了一种具体的第一类型的滤光阵列，增加了方案的多样性。
可选地，结合上述第一方面第一种可能的实施方式，在第二种可能的实现方式中，三个通道为红色通道R，绿色通道G，蓝色通道B，黄色通道Y以及近红外通道IR中的任意三个通道。
可选地，结合上述第一方面第一种或第一方面第二种可能的实施方式，在第三种可能的实施方式中，第二类型的滤光阵列可以包括的通道数目为大于3的正整数，第二类型的滤光阵列可以包括的一个通道用于表示第二波段范围内的一种光谱信息。
可选地，结合上述第一方面第二种可能的实施方式，在第四种可能的实施方式中，三个通道为RGB通道，RYB通道，RBGIR通道中的任意一种通道。
可选地，结合上述第一方面，在第五种可能的实施方式中，第一类型的滤光阵列可以包括四通道的滤光阵列，或者第一类型的滤光阵列可以包括三通道的滤光阵列和四通道的滤光阵列，第二类型的滤光阵列的通道数目为大于4的正整数。
可选地，结合上述第一方面第三种可能的实施方式，在第六种可能的实施方式中，四个通道为红色通道R，绿色通道G，蓝色通道B，黄色通道Y以及近红外通道IR中的任意四个通道。
可选地，结合上述第一方面第五种可能的实施方式或第一方面第六种可能的实施方式， 在第七种可能的实施方式中，第二类型的滤光阵列的通道数目为不小于4且不大于25的正整数，第二类型的滤光阵列可以包括的一个通道用于表示第二波段范围内的一种光谱信息。
可选地，结合上述第一方面或第一方面第一种至第一方面第七种可能的实施方式，在第八种可能的实施方式中，第一类型的滤光阵列和第二类型的滤光阵列均为矩阵结构。
可选地，结合上述第一方面或第一方面第一种至第一方面第八种可能的实施方式，在第九种可能的实施方式中，滤光阵列可以包括多组第一类型的滤光阵列和多组第二类型的滤光阵列，多组第一类型的滤光阵列包围多组第二类型的滤光阵列。
可选地，结合上述第一方面第九种可能的实施方式，在第十种可能的实施方式中，越靠近滤光阵列的中心，第二类型的滤光阵列分布密度越大，第一类型的滤光阵列的分布密度越小。
可选地，结合上述第一方面或第一方面第一种至第一方面第八种可能的实施方式，在第十一种可能的实施方式中，滤光阵列可以包括多组第一类型的滤光阵列和多组第二类型的滤光阵列，多组第二类型的滤光阵列包围多组第一类型的滤光阵列。
可选地，结合上述第一方面或第一方面第一种至第一方面第八种可能的实施方式，在第十二种可能的实施方式中，滤光阵列可以包括多组第一类型的滤光阵列和多组第二类型的滤光阵列，多组第一类型的滤光阵列和多组第二类型的滤光阵列并列设置。
可选地，结合上述第一方面第九种至第一方面第十二种可能的实施方式，在第十三种可能的实施方式中，在滤光阵列中，第二类型的滤光阵列是周期性分布的或者非周期性分布的。
可选地，结合上述第一方面第九种至第一方面第十三种可能的实施方式，在第十四种可能的实施方式中，多组第一类型的滤光阵列的总面积大于多组第二类型的滤光阵列的总面积。
可选地，结合上述第一方面第一种至第一方面第十四种可能的实施方式，在第十五种可能的实施方式中，第一类型的滤光阵列可以包括滤色片，微透镜以及光电转换单元，光电转换单元用于将光信号转换为电信号，滤色片位于微透镜和光电转换单元的中间，针对一个第一类型的滤光阵列，一个微透镜对应一个滤色片，且对应一个光电转换单元，或者一个微透镜对应一个滤色片，且对应多个光电转换单元，或者一个滤色片对应多个微透镜，且对应多个光电转换单元。
可选地，结合上述第一方面第一种至第一方面第十四种可能的实施方式，在第十六种可能的实施方式中，第二类型的滤光阵列可以包括滤色片，微透镜以及光电转换单元，光电转换单元用于将光信号转换为电信号，滤色片位于微透镜和光电转换单元的中间，针对一个第二类型的滤光阵列，一个微透镜对应一个滤色片，且对应一个光电转换单元，或者一个微透镜对应一个滤色片，且对应多个光电转换单元，或者一个滤色片对应多个微透镜，且对应多个光电转换单元。
本申请第二方面提供一种摄像模组，摄像模组可以包括底座和滤光阵列，滤光阵列安装在底座上，滤光阵列为第一方面或第一方面任意一种可能的实施方式中描述的滤光阵列。
可选地，结合上述第二方面，在第一种可能的实施方式中，摄像模组还可以包括处理器和存储器，处理器和存储器耦合，存储器中存储有与图像处理相关的程序指令，当存储 器存储的程序指令被处理器执行时，对第一类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到第一图像，对第二类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到第二图像，对第一图像和第二图像进行融合处理，以得到高分辨率的高光谱图像。
本申请第三方面提供一种摄像机，摄像机可以包括壳体和摄像模组，摄像模组收容于壳体内，摄像模组为第二方面描述的摄像模组。
可选地，结合第三方面，在第一种可能的实施方式中，摄像机还可以包括处理器和存储器，处理器和存储器耦合，存储器中存储有与图像处理相关的程序指令，当存储器存储的程序指令被处理器执行时，对第一类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到第一图像，对第二类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到第二图像，对第一图像和第二图像进行融合处理，以得到高分辨率的高光谱图像。
本申请第四方面提供一种移动终端，移动终端可以包括镜头，镜头支架和滤光阵列，镜头安装在镜头支架上，镜头支架设置于滤光阵列和镜头之间。滤光阵列为第一方面或第一方面任意一种可能的实施方式中描述的滤光阵列。
可选地，结合上述第四方面，在第一种可能的实施方式中，移动终端还可以包括处理器和存储器，处理器和存储器耦合，存储器中存储有与图像处理相关的程序指令，当存储器存储的程序指令被处理器执行时，对第一类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到第一图像，对第二类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到第二图像，对第一图像和第二图像进行融合处理，以得到高分辨率的高光谱图像。
本申请第五方面提供一种智能汽车，智能汽车可以包括镜头，镜头支架和滤光阵列，镜头安装在镜头支架上，镜头支架设置于滤光阵列和镜头之间。滤光阵列为第一方面或第一方面任意一种可能的实施方式中描述的滤光阵列。
可选地，结合上述第五方面，在第一种可能的实施方式中，智能汽车还可以包括处理器和存储器，处理器和存储器耦合，存储器中存储有与图像处理相关的程序指令，当存储器存储的程序指令被处理器执行时，对第一类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到第一图像，对第二类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到第二图像，对第一图像和第二图像进行融合处理，以得到高分辨率的高光谱图像。
本申请第六方面提供一种监控设备，监控设备可以包括镜头，镜头支架和滤光阵列，镜头安装在镜头支架上，镜头支架设置于滤光阵列和镜头之间。滤光阵列为第一方面或第一方面任意一种可能的实施方式中描述的滤光阵列。
可选地，结合上述第六方面，在第一种可能的实施方式中，监控设备还可以包括处理器和存储器，处理器和存储器耦合，存储器中存储有与图像处理相关的程序指令，当存储器存储的程序指令被处理器执行时，对第一类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到第一图像，对第二类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到第二图像，对第一图像和第二图像进 行融合处理，以得到高分辨率的高光谱图像。
本申请第七方面提供一种电子设备，电子设备可以包括镜头，镜头支架和滤光阵列，镜头安装在镜头支架上，镜头支架设置于滤光阵列和镜头之间。滤光阵列为第一方面或第一方面任意一种可能的实施方式中描述的滤光阵列。
可选地，结合上述第七方面，在第一种可能的实施方式中，电子设备还可以包括处理器和存储器，处理器和存储器耦合，存储器中存储有与图像处理相关的程序指令，当存储器存储的程序指令被处理器执行时，对第一类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到第一图像，对第二类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到第二图像，对第一图像和第二图像进行融合处理，以得到高分辨率的高光谱图像。
附图说明
图1为拜耳阵列的原理示意图；
图2为本申请实施例提供的一种滤光阵列的结构示意图；
图3为本申请实施例提供的另一种滤光阵列的结构示意图；
图4为本申请实施例提供的另一种滤光阵列的结构示意图；
图5为本申请实施例提供的另一种滤光阵列的结构示意图；
图6为本申请实施例提供的另一种滤光阵列的结构示意图；
图7为本申请实施例提供的另一种滤光阵列的结构示意图；
图8为本申请实施例提供的另一种滤光阵列的结构示意图；
图9为本申请实施例提供的另一种滤光阵列的结构示意图；
图10为本申请实施例提供的另一种滤光阵列的结构示意图；
图11为本申请实施例提供的另一种滤光阵列的结构示意图；
图12为本申请实施例提供的另一种滤光阵列的结构示意图；
图13为本申请实施例提供的第一类型的阵列是3通道或者4通道的结构示意图；
图14为本申请实施例提供的第一类型的滤光阵列或第二类型的滤光阵列的结构示意图；
图15为本申请实施例提供的第一类型的滤光阵列或者第二类型的滤光阵列的结构示意图；
图16为本申请实施例提供的第一类型的滤光阵列或者第二类型的滤光阵列的结构示意图；
图17为滤色片的结构示意图；
图18为滤色片的另一种结构示意图；
图19为滤色片的另一种结构示意图；
图20为本申请实施例提供的一种电子设备的结构示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。
本申请的说明书和权利要求书及上述附图中的术语“第一”、“第二”、“第三”、 “第四”等(如果存在)是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换，以便这里描述的实施例能够以除了在这里图示或描述的内容以外的顺序实施。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，例如，包含了一系列步骤或单元的过程、方法、系统、产品或设备不必限于清楚地列出的那些步骤或单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它步骤或单元。
在本申请的描述中，“多个”的含义是两个或两个以上，除非另有明确具体的限定。
在本申请中，除非另有明确的规定和限定，术语“安装”、“相连”、“连接”、“固定”、“设置”等术语应做广义理解，例如，可以是固定连接，也可以是可拆卸连接，或成一体；可以是机械连接，也可以是电连接；可以是直接相连，也可以通过中间媒介间接相连，还可以是两个元件内部的连通或两个元件的相互作用关系。对于本领域的普通技术人员而言，可以根据具体情况理解上述术语在本申请中的具体含义。
在本申请的描述中，需要理解的是，术语“长度”、“宽度”、“上”、“下”、“前”、“后”、“左”、“右”、“竖直”、“水平”、“顶”、“底”、“内”、“外”等指示的方位或位置关系为基于附图所示的方位或位置关系，仅是为了便于描述本申请和简化描述，而不是指示或暗示所指的装置或元件必须具有特定的方位、以特定的方位构造和操作，因此不能理解为对申请的限制。
在说明书及权利要求当中使用了某些词汇来指称特定的组件。本领域技术人员应可理解，硬件制造商可能会用不同的名词来称呼同一个组件。本说明书及后权利要求并不以名称的差异来作为区分组件的方式，而是以组件在功能上的差异来作为区分的准则。在说明书及权利要求当中所提及的包含或者包括是为一开放式的用语，故应解释成包含但不限定于或者包括但不限于。
为了便于更好的理解本申请，下面具体阐述本申请所描述的技术方案的研究思路：
首先介绍一下现有技术中的拜耳阵列。拜耳阵列指的是电荷藕合器件(charge coupled device，CCD)或者互补性氧化金属半导体(complementary metal-oxide semiconductor，CMOS)作为光传感器的时候，采集数字图像时用到的一种常见的方法。
图像传感器将光线转化成电流，光线越亮，电流的数值就越大，光线越暗，电流的数值就越小。但是，图像传感器有一个很严重的缺陷：它只能感受光的强弱，无法感受光的波长。由于光的颜色由波长决定，所以图像传播器无法记录颜色。为了获得彩色图像，其中方案是只用一块图像传感器，在图像传感器前面，设置一层彩色滤光片阵列(color filter array，CFA)，拜耳阵列并没有在每个像素(pixel)上放三个颜色的滤镜，而是有间隔的在每个像素上放置单一颜色的滤镜。换句话说，图像传感器的每一个像素都是一个独立的光电转换单元，只能将采集到的光线由强度转换为电压信号后输出。也就是说像素本身为单色的图像传感器，它能记录彩色是因为在进行光电转换之前，通过让特定颜色的光线投射到像素上完成光线强度的采集，并由此建立起特定“单色光”与光电转换单元得到的光线强度的相对应关系。这样以来，每个通道能得到一个部分值空缺的图片，这些空缺的值可以通过各种插值手段进行填充。下面结合图1对拜耳阵列进行具体的说明。该实现方案是由不同颜色的滤光片组成的滤色片拜尔阵列。把光分别滤成“单色光”后，再 传递给像素单独记录强度，并建立由拜尔阵列形成的“单色光”与由光电转换单元得到的电压的相对应关系。需要说明的，本申请有时也将滤光片称为滤色片，将滤光称为滤色，在不强调二者的区别之时，二者表示相同的意思。如图1所示，通常拜尔阵列只有红绿蓝三种颜色的滤光片。滤色片与图像传感器的像素一一对应，每个滤色片只能通过红(red,R)、绿(green,G)、蓝(blue,B)三种光其中之一。这种可以通过不同颜色滤光点的排列是有规律的，每个绿色点的右上、左下分布的绿点，上下分布的是蓝点，左右分布的是红点。绿点的数量是红色或者蓝色数量的两倍，是因为研究发现人眼对绿色是最为敏感的。现在得到的图像也只有三种颜色，显然还无法表达真实世界的颜色。所以还需要根据每个像素周围的颜色来近似恢复它原来的值，最后得到彩色图像。这个过程叫做“去马赛克(demosaicing)”。去马赛克有不同的实现方法，简单的内插法，比如绿色过滤器的像素精确测量了绿色成分，而该像素红色和蓝色的成分则是从邻区获取。一个绿色像素的红色数值可由相邻两个红色像素内插计算出来，同样的，内插相邻两个蓝色像素也能计算出蓝色数值。举例说明，以图1中所示的R33为例,R33中的红色成分R33，绿色成分G33以及蓝色成分B33可以通过如下公式表示：
R33＝R33
Figure PCTCN2020104797-appb-000001
Figure PCTCN2020104797-appb-000002
通过去马赛克可以获得每个像素包括的红色成分，绿色成分以及蓝色成分，并获取最终输出的彩色图像。
但是，仅通过RGB三个通道采集到的图像信息量不够丰富，无法更准确刻画物理世界的光谱信息。而高光谱成像技术具有超多波段、高的光谱分辨率、波段窄、光谱范围广和图谱合一等特点，采集到的图像信息量丰富。从而在准确色彩、物质检测上等领域有广泛应用。但是，通过高光谱成像技术获取的图像的分辨率低，想要提高分辨率成本又高，比如，通过去马赛克获取彩色图像，如果只有RGB三个通道，对于一个像素，需要计算其他两种颜色的成分，如果是高光谱成像，对于一个像素，可能需要猜测大量的其他颜色的成分，计算量大大增加，提升了成本。因此如何提升高光谱图像的分辨率且不增加成本亟待解决。
针对上述问题，本申请提供一种滤光阵列，该滤光阵列包括两种类型的类型阵列。其中，第一种类型的滤光阵列用于获取高分辨率的低光谱图像，比如该第一种类型的滤光阵列可以是上述提到的拜耳阵列，第二种类型的滤光阵列用于获取低分辨率的高光谱图像，比如该第二种类型的滤光阵列是多光谱滤光阵列。这样结构的滤光阵列可以在一次成像过程中，获取到高分辨率的高光谱图像，不仅结构简单，还可以降低提升高光谱图像的分辨率的成本。
基于上面的研究思路，下面对本申请提供的技术方案进行具体的介绍。
本申请实施例提供的一种滤光阵列包括：第一类型的滤光阵列和第二类型的滤光阵列，第一类型的滤光阵列涵盖第一波段范围的光谱信息，第二类型的滤光阵列涵盖第二波段范围的光谱信息，第二波段范围的光谱信息与第一波段范围的光谱信息不同。比如第一类型 的滤色阵列涵盖可见光波段的光谱信息，第二类型的滤色阵列可以涵盖近红外波段的光谱信息，再比如，第一类型的滤色阵列涵盖第一部分可见光波段的光谱信息，第二类型的滤色阵列可以涵盖第二部分可见光波段的光谱信息，且第一部分可见光波段不同于第二部分可见光波段。或者第二波段范围的光谱信息与第一波段范围的光谱信息在预设波段范围内重叠。
从通道的角度进行解释，一个通道代表一种光谱响应函数，本申请有时也称一个通道代表一种光谱信息，则第二类型的滤色阵列包括的通道与第一类型的滤色阵列包括的通道完全不同，或者第二类型的滤色阵列包括的通道与第一类型的滤色阵列包括的通道部分相同。
在一种设计中，第二类型的滤色阵列包括的通道数多于第一类型的滤色阵列包括的通道数，可以通过第二类型的滤色阵列获取低分辨率高光谱的图像，通过第一类型的滤色阵列获取高分辨率低光谱的图像，再对通过第一类型的滤色阵列获取的数据以及通过第二类型的滤色阵列获取的数据进行融合，以得到高分辨率的高光谱图像。再比如在另一种设计中，第二类型的滤色阵列包括的通道数虽然和第一类型的滤色阵列包括的通道数相同，或者第二类型的滤色阵列包括的通道数小于第一类型的滤色阵列包括的通道数，但是第二类型的滤色阵列涵盖的光谱信息可以对第一类型的滤色阵列涵盖的光谱信息进行补充，则可以通过相关算法将通过第一类型的滤色阵列获取的数据，以及通过第二类型的滤色阵列获取的数据进行融合，以得到高光谱的低分辨率图像，再根据相关算法提升图像的分辨率，以得到的高光谱的高分辨率图像。以下针对这两种设计分别进行说明。
在一个可能的实施方式中，可以认为第一类型的滤光阵列包括的光谱信息少，第二类型的滤光阵列包括的光谱信息多。可以认为第一类型的滤光阵列用于获取低光谱图像，第二类型的滤光阵列用于获取高光谱图像。在一个优选的实施方式中，第一类型的滤光阵列包括的通道数目为3种通道或者4种通道。第二类型的滤光阵列包括的通道数目多，比如可以包括8种或者10种通道。通道数目多，则可以认为，可以认为第二类型的滤光阵列用于获取高光谱图像，或者多光谱图像。此外，通道数目多，则需要以高成本获取高分辨率的图像。需要说明的是，上述列举的第一类型的滤光阵列包括3种通道或者4种通道，第二类型的滤光阵列包括8种或者10种通道仅为举例说明，并不代表对第一类型的滤光阵列和第二类型的滤光阵列的通道数目的限制。上述举例是为了说明，通道数目越少，做成高分辨率的图像成本越小，但是包括的光谱信息少，即低光谱图像；通道数目越多，做成高分辨率的图像成本越大，但是包括的光谱信息多，即高光谱图像。所以，本申请提供的方案，用通道数目少的第一类型的滤光阵列获取高分辨率的低光谱图像，增加分辨率的同时降低成本，用通道数目多的第二类型的滤光阵列获取低分辨率的高光谱图像，获取更多的光谱信息。当两种滤光阵列获取的数据进行融合后，可以获得高分辨率的高光谱图像，且不会增加额外的成本。
在一个可能的实施方式中，高分辨率图像是指垂直分辨率大于等于720的图像，也可以称为高清图像，比如可以是1280×720和1920×108，其中乘号前面表示宽度(即水平分辨率)，乘号后面表示高度(即垂直分辨率)。低分辨率图像是指垂直分辨率小于720的图像。
在一个可能的实施方式中，高光谱图像可以涵盖部分可见光信息，或者高光谱图像可以涵盖整个可见光信息，或者高光谱图像可以涵盖全部可见光信息和近红外光信息，或者高光谱图像可以涵盖部分可见光信息和近红外光信息。低光谱图像可以涵盖部分可见光信息，或者低光谱图像可以涵盖整个可见光信息，且当高光谱图像涵盖部分可见光信息，或者高光谱图像涵盖整个可见光信息时，低光谱图像可以涵盖的光谱信息小于高光谱图像可以涵盖的光谱信息。
高光谱和低光谱均和通道数目相关，即高光谱通过多通道获取，低光谱通过少一些的通道获取。由于通道数目越多，增加图像的分辨率的成本越大，通道数目越小，增加图像的分辨率的成本越低。所以可以认为图像的分辨率以及光谱的高低均和通道数目相关。
在一个可能的实施方式中，第一类型的滤光阵列的通道数目为不大于M的正整数，第二类型的滤光阵列的通道数目为大于M的正整数，M为正整数。比如M为1时，第一类型的滤光阵列的通道数目为1，第二类型的滤光阵列的通道数目为大于1的正整数，比如第二类型的滤光阵列的通道数可以是2，3，8，10，20等等。再比如，M为7时，第一类型的滤光阵列的通道数目为不大于7的正整数，比如可以是1，3，4，6等等，第二滤光阵列的通道数目为大于7的正整数，比如可以是9，10，30等等。
需要说明的是，第一类型的滤光阵列的通道数目的选取决定了提升图像分辨率的成本，第二类型的滤光阵列的通道数目的选取决定了图像包括的光谱信息，下面结合本申请实施例提供的几个优选方案对第一类型的滤光阵列和第二类型的滤光阵列进行说明。
如图2中所示，在一个优选的实施方式中，第一类型的滤光阵列为3通道的滤光阵列，其中，在一个可能的实施方式中，该三个通道为红色通道(red，R)，绿色通道(green，G)，蓝色通道(blue，B)，黄色通道Y以及近红外通道IR中的任意三个通道。第二类型的滤光阵列的通道数目为4至25。以图2中所示，以该第二类型的滤光阵列的通道数目为16进行了展示。如图2中所示，第一类型的滤光阵列可以都是某一种类型的3通道的滤光阵列，比如如图2中所示，以该第一类型的滤光阵列是RGB三个通道，第一类型的滤光阵列可以全部都是RGB三个通道的滤光阵列。或者，第一类型的滤光阵列可以包括多种类型的3通道的滤光阵列，比如第一类型的滤光阵列是RGB三个通道和RYB三个通道，则本申请提供的滤光阵列中，可以包括RGB三个通道的第一类型的滤光阵列，也可以包括RYB三个通道的第一类型的滤光阵列。图2所示的滤光阵列中，以第一类型的滤光阵列全部都是RGB三个通道进行的展示。第二类型的滤光阵列的通道数目为16，在一个可能的实施方式中，可以认为每一个通道代表一种颜色，该16种颜色各不相同。
关于第一类型的滤光阵列和第二类型的滤光阵列之间的位置关系可能有多种情况，除了本申请实施例展示出的几种位置关系之外，还可能存在其他的位置关系，应当理解，除了本申请实施例展示出的几种位置关系之外，其他的位置关系也应当属于本申请实施例的保护范围。如图2中所示，滤光阵列包括多组第一类型的滤光阵列以及多组第二类型的滤光阵列，并且第一类型的滤光阵列和第二类型的滤光阵列之间的位置关系为多组第一类型的滤光阵列包围第二类型的滤光阵列。
在一个优选的实施方式中，如图3所示，第一类型的滤光阵列包围第二类型的滤光阵列，且越靠近滤光阵列的中心，第二类型的滤光阵列的分布密度越大，第一类型的滤光阵 列分布密度越小。换句话说，第二类型的滤光阵列设置在滤光阵列的中心，第一类型的滤光阵列设置在滤光阵列的周围。这种设计的好处在于通常通过镜头拍照，拍照的对象一般会位于镜头的中心位置，则将第二类型的滤光阵列设置在滤光阵列的中心位置，可以提升图像中目标对象包括的光谱信息。
在一个可能的实施方式中，如图4所示，第一类型的滤光阵列和第二类型的滤光阵列之间的位置关系为多组第二类型的滤光阵列包围第一类型的滤光阵列，如图4中所示，以第一类型的滤光阵列为RGB三通道，第二滤光阵列为8通道的滤光阵列进行展示。在一个可能的实施方式中，越靠近滤光阵列的中心，第一类型的滤光阵列的分布密度越大，第二类型的滤光阵列分布密度越小。换句话说，第一类型的滤光阵列设置在滤光阵列的中心，第二类型的滤光阵列设置在滤光阵列的周围。这种设计的好处在于通常通过镜头拍照，拍照的对象一般会位于镜头的中心位置，则将第二类型的滤光阵列设置在滤光阵列的中心位置，可以提升图像中目标对象区域的图像分辨率。需要说明的是，第二类型的滤光阵列包围第一类型的滤光阵列可以包括第二类型的滤光阵列半包围第一类型的滤光阵列，比如第二类型的滤光阵列位于第一类型的滤光阵列的左侧，右侧以及下侧，但是没有没有第一类型的滤光阵列的上侧。
在一个可能的实施方式中，如图5所示，第一类型的滤光阵列和第二类型的滤光阵列之间的位置关系为多组第一类型的滤光阵列和多组第二类型的滤光阵列并列设置。
在一个可能的实施方式中，第二类型的滤光阵列是周期分布的。比如具体可以表现在，滤光阵列中任意两组第二类型的滤光阵列中间间隔了相同数量的第一类型的滤光阵列。如图6所示，展示了一种第二类型的滤光阵列是周期分布的结构示意图。在一个可能的实施方式中，第二类型的滤光阵列也可以不是周期分布的。比如如图7所示，具体可以表现在滤光阵列中任意两组第二类型的滤光阵列中间间隔了不同数量的第一类型的滤光阵列。
在一个可能的实施方式中，多组第一类型的滤光阵列的总面积大于多组第二类型的滤光阵列的总面积。在这种实施方式中，全部第一类型的滤光阵列的总面积大于全部第二类型的滤光阵列的总面积，第一类型的滤光阵列的总面积大，则可以更好的节约成本，以得到高光谱的高分辨率图像。
如图2至图7所示的结构中，第一类型的滤光阵列是矩阵结构，具体的以2乘以2的矩阵结构进行的说明，第二类型的滤光阵列也是矩阵结构，具体的如图2所示，以4乘以4的矩阵结构进行的说明。需要说明的是，除了上述提到的第一类型的滤光阵列是2乘以2的矩阵结构，第二类型的滤光阵列是4乘以4的矩阵结构，还可以是其他的矩阵结构，比如第二类型的滤光阵列还可以是1乘以2的矩阵结构，3乘以3的矩阵结构，5乘以5的矩阵结构等等。此外，需要说明的是，除了矩阵结构，本申请实施例中的第二类型的滤光阵列还可能有其他的结构。比如，如图8所示，第二类型的滤光阵列是“L”型的结构，或者如图9所示，第二类型的滤光阵列可以是“凹”型的结构。当然，除了本申请给出的几种第二类型的滤光阵列的结构，第二类型的滤光阵列还可以是其他结构，比如第二类型的滤光阵列还可以是“E”型的结构。
如图10中所示，在一个优选的实施方式中，第一类型的滤光阵列为4通道的滤光阵列，其中，在一个可能的实施方式中，该4个通道为红色通道R，绿色通道G，蓝色通道B，黄 色通道Y以及近红外通道IR中的任意四个通道。第二类型的滤光阵列的通道数目为4至25。以图10中所示，以该第二类型的滤光阵列的通道数目为16进行了展示。如图10中所示，第一类型的滤光阵列可以都是某一种类型的4通道的滤光阵列，比如如图10中所示，以该第一类型的滤光阵列是RGBIR四个通道，图10所示的滤光阵列中，第一类型的滤光阵列可以全部都是RGBIR四个通道的滤光阵列。或者，第一类型的滤光阵列可以包括多种类型的4通道的滤光阵列，比如如图11所示，第一类型的滤光阵列是RGBIR四个通道和RYBIR四个通道，则本申请提供的滤光阵列中，可以包括RGBIR四个通道的第一类型的滤光阵列，也可以包括RYBIR四个通道的第一类型的滤光阵列。在一个可能的实施方式中，该滤光阵列可以包括三通道的滤光阵列和四通道的滤光阵列，比如如图12所示，该滤光阵列中既包括三通的滤光阵列也包括四通道的滤光阵列。当第一类型的滤光阵列包括4通道的滤光阵列时，关于第二类型的滤光阵列的结构，以及第一滤光阵列和第二滤光阵列之间的位置关系可以参照上文第一类型的滤光阵列包括3通道的滤光阵列进行理解，本申请实施例对此不再重复赘述。
如图13中的a至图13中的c所示，给出了几种典型的第一类型的滤光阵列是3通道或者4通道的结构示意图。
以上主要针对本申请提供的第一种设计方式进行了介绍，即在第一种设计方式中，第二类型的滤色阵列包括的通道数多于第一类型的滤色阵列包括的通道数，可以通过第二类型的滤色阵列获取低分辨率高光谱的图像，通过第一类型的滤色阵列获取高分辨率低光谱的图像，再对通过第一类型的滤色阵列获取的数据以及通过第二类型的滤色阵列获取的数据进行融合，以得到高分辨率的高光谱图像。需要说明的，在本申请提供的第二种设计方式中，第二类型的滤色阵列包括的通道数可以少于第一类型的滤色阵列包括的通道数，或者第二类型的滤色阵列包括的通道数可以和第一类型的滤色阵列包括的通道数目相同，但是第二类型的滤色阵列涵盖的光谱信息可以对第一类型的滤色阵列涵盖的光谱信息进行补充，则可以通过相关算法将通过第一类型的滤色阵列获取的数据，以及通过第二类型的滤色阵列获取的数据进行融合，以得到的高光谱的高分辨率图像。在第二种设计方式中，除了通道数目的限定，第一类型的滤光阵列和第二类型的滤光阵列之间的位置关系可以参照第一种设计方式中第一类型的滤光阵列和第二类型的滤光阵列之间的位置关系进行理解，这里不再重复赘述。
需要说明的是，本申请提供的滤光阵列可以包括图2至图12中所描述的结构中的任意一种，也可以包括图2至图12中所描述的结构中的任意多种结构的组合。比如本申请提供的滤光阵列可以同时包括图3所描述的结构以及图4所描述的结构。
上文主要介绍了第一类型的滤光阵列的通道数目，第二类型的滤光阵列的通道数目，第一类型的滤光阵列和第二类型的滤光阵列之间的位置关系，第一类型的滤光阵列的结构，以及第二类型的滤光阵列的结构。下面对第一类型的滤光阵列以及第二类型的滤光阵列的具体结构。
在一个可能的实施方式中，第一类型的滤光阵列包括滤色片，微透镜以及光电转换单元，光电转换单元用于将光信号转换为电信号，滤色片位于微透镜和光电转换单元的中间，针对一个第一类型的滤光阵列，一个微透镜对应一个滤色片，且对应一个光电转换单元， 或者一个微透镜对应一个滤色片，且对应多个光电转换单元，或者一个滤色片对应多个微透镜，且对应多个光电转换单元。下面结合图14至图16进行说明。如图14所示，光电转换单元的上方设置有滤色片，滤色片的上方设置有微透镜，并且微透镜，滤色片和光电转换单元是一一对应的关系，即一个光电转换单元的上方设置一个滤色片，一个滤色片的上方设置有一个微透镜。微透镜的目的和作用是提高光电转换单元接受的光能,在微观领域,也可以说是光子数量。每个光电转换单元限于结构,是不可能与其它光电转换单元无缝衔接的,那么在整个传感器上,就会损失大量的感光面积。通过微透镜,将整个表面接受的光,汇聚到一个个光电转换单元上,大大提高了光的利用效率。如图15所示，多个光电转换单元的上方设置有一个滤色片，在一个优选的实施方式中，每两个光电转换单元的上方设置一个滤色片，每一个滤色片的上方设置一个微透镜。如图16所示，多个光电转换单元的上方设置有一个滤色片，在一个优选的实施方式中，每两个光电转换单元的上方设置一个滤色片，每一个滤色片的上方设置有多个微透镜，在一个优选的实施方式中，每一个滤色片的上方设置有两个微透镜。
在一个可能的实施方式中，第二类型的滤光阵列包括滤色片，微透镜以及光电转换单元，光电转换单元用于将光信号转换为电信号，滤色片位于微透镜和光电转换单元的中间，针对一个第二类型的滤光阵列，一个微透镜对应一个滤色片，且对应一个光电转换单元，或者一个微透镜对应一个滤色片，且对应多个光电转换单元，或者一个滤色片对应多个微透镜，且对应多个光电转换单元。关于第二类型的滤光阵列可以参照图14至图16进行理解，这里不再重复赘述。
滤色片有时也被称为滤光片，是用来选取所需辐射波段的光学器件。在一个可能的实施方式中，滤色片是通过化学染料材料制作而成，比如滤光片是塑料或玻璃片再加入特种染料做成的，红色滤光片只能让红光通过，如此类推。玻璃片的透射率原本与空气差不多，所有有色光都可以通过，所以是透明的，但是染了染料后，分子结构变化，折射率也发生变化，对某些色光的通过就有变化了。比如一束白光通过蓝色滤光片，射出的是一束蓝光，而绿光、红光极少，大多数被滤光片吸收了。在一个可能的实施方式中，滤色片是由结构光学制作而成的，比如滤色片可以是纳米孔洞结构，可以是波导光栅结构，可以是多层干涉薄膜结构等等，任何一种可以实现滤色功能的结构，本申请实施例都可以采用，比如滤色片还可以是FP腔(fabry-perot cavity)结构等等。FP腔是一种利用多光束干涉现象来工作的装置。为了更好的理解由结构光学制作的滤色片，下面结合图17至图19举例说明。如图17所示，是通过纳米孔洞结构制作而成的滤色片的结构示意图。相邻孔洞的表面等离极化激元的干涉作为引发对光的选择透过。纳米孔洞型通过调节金属材料加工的纳米孔的孔径大小，以及纳米孔之间的间距来设计不同的滤光片。如图18所示，是通过波导光栅结构制作而成的滤色片。微纳光栅结构引入表面等离激元金属，使得微纳光栅在光学衍射基础上还叠加了表面等离激元的吸收和散射等作用，微纳周期光栅结构可以作为色散媒介对波段进行选择。通过改变光栅结构的周期和占空比等参数，可以控制光的吸收、散射、衍射和偏振等特性，从而实现滤波特性。如图19所示，为根据多层干涉薄膜结构制作而成的滤色片，在这种结构中，根据引起光学滤波的主要物理机制，这种滤波结构可分为两类：基于局域表面等离激元共振的金属/介质/金属(metal insulator metal，MIM)型，以及 基于多光束干涉的法布里珀罗(fabry-perot，FP)型。MIM型滤波结构顶层金属一般具有微纳结构，顶层局域表面等离激元共振和上下两层金属反平行表面电流共同作用，使得MIM结构对特定波段的光具有吸收特性，从而实现滤波功能。FP型滤波结构利用两层半透半反膜构成反射层，中间夹截止层构成谐振器，利用多光束干涉，使干涉相长的光透过或反射。这种模式不需要顶层的纳米结构，一般只考虑各膜层的光学性能和制造工艺。
通过以上本申请实施例提供的滤光阵列可以获取到高分辨率的低光谱图像，以及低分辨率的高光谱图像，对高分辨率的低光谱图像以及低分辨率的高光谱图像进行融合处理，即可以获取到高分辨率的高光谱图像。通过本申请提供的方案不需要多次成像，只需要一次成像就可以获取到高分辨率的高分辨率的高光谱图像。
需要说明的是，对于不同的集成信号处理器(integrated signal processor,ISP)对图像的处理过程可能不相同，下面以一种处理过程为例对如何获取高分辨率的低光谱图像，以及低分辨率的高光谱图像进行说明。应当理解的是，获取高分辨率的低光谱图像和获取低分辨率的高光谱图像的过程相似，下面将高分辨率的低光谱图像和低分辨率的高光谱图像统一简称为图像。
ISP指的是专门用来处理感光件信号并生成最终图像的硬件，通常会作为一个模块集成到片上系统。接下来将要介绍的各种处理都是在ISP上进行的。
光线进入镜头，到达带有本申请实施例提供的滤光阵列的传感器，得到最原始的电信号。对这个信号进行光照度估计,去马赛克处理以及降噪，就会得到raw格式的图片，对它接着进行白平衡处理等，获取图像。其中，光照度估计是为了使信号满足预设的标准(international standardization organization，ISO)，比如该标准可以是相机的感光度的标准，为了达到与设置相对应的，相机会将接收到的信号进行增益。去马赛克处理可以参照上文在介绍拜耳阵列中提到的去马赛克处理进行理解，这里不再重复赘述。降噪的方法由很多，比如噪音往往比较突兀，因此可以使用模糊来减小噪音。但模糊也会影响细节，因此考虑将去掉部分中信号较强的区域补回图像。通过白平衡处理可以修正色彩偏差，比如以第一类型的滤光阵列是RGB通道为例，白平衡也就是矫正RGB相等的点(白点)，可以通过对三个通道分别进行放缩实现矫正。白平衡处理就是针对不同色温条件下，通过调摄像头内部的色彩电路使拍摄出来的影像抵消偏色，更接近人眼的视觉习惯。白平衡也可以简单地理解为在任意色温条件下，摄像头所拍摄的标准白色经过电路的调整，使之成像后仍然为白色。需要说明的是，上述过程仅为示例性说明，在实际应用场景中可以包括更多的步骤。通过上述方法对通过第一类型的滤光阵列获取的数据进行处理，对第二类型的滤光阵列获取的数据进行处理，并对处理后的数据进行融合，就可以获取的高光谱高分辨率的图像。示例性的，下面给出一种对处理后的数据进行融合，获取高光谱高分辨率的图像的方法。
多光谱图像可以视为对待重建的光谱图像进行模糊和下采样，低光谱图像可以视为对待重建的光谱图像进行光谱下采样。因此，光谱图像融合问题也可以看成利用两幅退化的光谱图像重构其退化前的光谱图像。观察到的多光谱图像Yh可以看成待重建图像Z的模糊、下采样和加噪数据。观察到的低光谱图像Ym可以看成待重建图像Z的光谱下采样和加噪数据。因此，将Z与Yh和Ym连接起来的光谱图像退化模型可以表示为：
Ym＝RZ+Nm
Yh＝ZBM+Nh
其中Nm和Nh表示高斯噪声。
其中光谱下采样矩阵R和空间模糊矩阵B为预设的。实际的应用当中，采用光谱分解等方法可以对光谱下采样矩阵和空间模糊矩阵进行估计，即可得到高分辨率高光谱图像。
其中光谱分解因其能将光谱图像分解为两个低维矩阵相乘的形式而被引入到光谱融合中。光谱分解旨在将光谱图像中的每一个像素点分解为一系列的成分光谱特征，称之为端元(endmembers)和丰度系数(abundance)。端元通常假设为图像中纯净的物质，每个像素点的丰度系数表示此像素点中不同物质所占的比例。
待重建的光谱图像Z可以表示为：
Z＝EX
其中，
Figure PCTCN2020104797-appb-000003
表示端元矩阵，等价于光谱信息，ns表示端元的个数或者成为子空间维度。λ h表示多光谱图像光谱波段。
Figure PCTCN2020104797-appb-000004
表示丰度矩阵，等价于空间信息，其中的每一个列向量表示待重建的光谱图
像Z中的一个像素的端元信息。ns的值通常比较小，也就表明E和X处于相对低维
的空间中。则：
Ym＝RZ+Nm≈EmX
Yh＝ZBM+Nh≈EXh
Figure PCTCN2020104797-appb-000005
表示光谱退化端元矩阵，
Figure PCTCN2020104797-appb-000006
空间退化丰度矩阵。很明显有Xh＝XBM和Em＝RE。λ m表示低光谱光谱波段。
待重建的光谱图像Z是表示光谱信息的端元矩阵E和表示空间信息的丰度矩阵X的线性组合。又因为观察到的多光谱图像具有高的空间分辨率，观察到的高光谱图像具有高的光谱分辨率。因此，重构高的空间和高光谱分辨率的光谱图像时采用两个线性优化模型分别提取多光谱图像的丰度矩阵和高光谱图像的端元矩阵。
对于提取多光谱图像Ym的丰度矩阵X，建立如下线性优化模型：
Figure PCTCN2020104797-appb-000007
其中，|| || F表示frobenius范数。相似的，提取高光谱图像Yh的端元矩阵E时，建立如下线性优化模型：
Figure PCTCN2020104797-appb-000008
因此，重构高的高分辨率和高光谱分辨率的光谱图像Z转换为求解上述两个线性优化模型。
针对本申请提供的第一种设计，在本发明中将第一类型滤光片阵列获取的图像作为低光谱图像Ym，将第二类型滤光片阵列获取的图像作为多光谱图像Yh，通过上述方案方法和算法获得高分辨率多光谱图像。
针对本申请提供的第二种设计，在本发明中将第一类型滤光片阵列获取的图像作为低光谱图像Ym，将第二类型滤光片阵列和第二类型滤光片阵列邻近的一组第一类型滤光片阵列共同获取的图像作为多光谱图像Yh，通过上述方案方法和算法获取高分辨率多光谱图像。
本申请还提供一种电子设备，电子设备可以包括镜头，镜头支架和滤光阵列，镜头安装在镜头支架上，镜头支架设置于滤光阵列和镜头之间。滤光阵列为本申请实施例中描述的滤光阵列。本申请提供的电子设备可以是移动终端，智能汽车，监控设备等任何需要安装滤光阵列的设备。
本申请提供的电子设备可以通过图20中的通信设备来实现。图20所示为本申请实施例提供的通信设备的硬件结构示意图。包括：处理器2001和存储器2002。
处理器2001包括但不限于中央处理器(central processing unit，CPU)，网络处理器(network processor，NP)，专用集成电路(application-specific integrated circuit，ASIC)或者可编程逻辑器件(programmable logic device，PLD)中的一个或多个。上述PLD可以是复杂可编程逻辑器件(complex programmable logic device，CPLD)，现场可编程逻辑门阵列(field-programmable gate array，FPGA)，通用阵列逻辑(generic array logic,GAL)或其任意组合。
存储器2002可以是只读存储器(read-only memory，ROM)或可存储静态信息和指令的其他类型的静态存储设备，随机存取存储器(random access memory，RAM)或者可存储信息和指令的其他类型的动态存储设备，也可以是电可擦可编程只读存储器(electrically er服务器able programmable read-only memory，EEPROM)、只读光盘(compact disc read-only memory，CD-ROM)或其他光盘存储、光碟存储(包括压缩光碟、激光碟、光碟、数字通用光碟、蓝光光碟等)、磁盘存储介质或者其他磁存储设备、或者能够用于携带或存储具有指令或数据结构形式的期望的程序代码并能够由计算机存取的任何其他介质，但不限于此。存储器可以是独立存在，通过通信线路与处理器2001相连接。存储器2002也可以和处理器2001集成在一起。如果存储器2002和处理器2001是相互独立的器件，存储器2002和处理器2001相连，例如存储器2002和处理器2001可以通过通信线路通信。
在一个可能的实施方式中，该电子设备可以是智能汽车，该智能汽车可以包括处理器和存储器，处理器和存储器耦合，存储器中存储有与图像处理相关的程序指令，当存储器存储的程序指令被处理器执行时，对第一类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到高分辨率的低光谱图像，对第二类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到低分辨率的高光谱图像，对高分辨率的低光谱图像和低分辨率的高光谱图像进行融合处理，以得到高分辨率的高光谱图像。
在一个可能的实施方式中，该电子设备可以是监控设备，该监控设备可以包括处理器和存储器，处理器和存储器耦合，存储器中存储有与图像处理相关的程序指令，当存储器存储的程序指令被处理器执行时，对第一类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到高分辨率的低光谱图像，对第二类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到低分辨率的高光谱图像，对高分辨率的低光谱图像和低分辨率的高光谱图像进行融合处理，以得到高分辨率的高光 谱图像。
在一个可能的实施方式中，该电子设备可以是移动终端，该移动终端可以包括处理器和存储器，处理器和存储器耦合，存储器中存储有与图像处理相关的程序指令，当存储器存储的程序指令被处理器执行时，对第一类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到高分辨率的低光谱图像，对第二类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到低分辨率的高光谱图像，对高分辨率的低光谱图像和低分辨率的高光谱图像进行融合处理，以得到高分辨率的高光谱图像。
以上所述仅为本申请的可选实施例，并不用以限制本申请，凡在本申请的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本申请的保护范围之内。

Claims (29)

  1. 一种滤光阵列，其特征在于，包括：第一类型的滤光阵列和第二类型的滤光阵列，所述第一类型的滤光阵列涵盖第一波段范围的光谱信息，所述第二类型的滤光阵列涵盖第二波段范围的光谱信息，所述第二波段范围的光谱信息与所述第一波段范围的光谱信息不同，或者所述第二波段范围的光谱信息与所述第一波段范围的光谱信息在预设波段范围内重叠。
  2. 根据权利要求1所述的滤光阵列，其特征在于，所述第一类型的滤光阵列包括三个通道的滤光阵列，所述第一类型的滤光阵列包括的三个通道中的每一个通道分别用于表示所述第一波段范围内的一种光谱信息。
  3. 根据权利要求2所述的滤光阵列，其特征在于，所述三个通道为红色通道R，绿色通道G，蓝色通道B，黄色通道Y以及近红外通道IR中的任意三个通道。
  4. 根据权利要求1至3任一项所述的滤光阵列，其特征在于，所述第二类型的滤光阵列包括的通道数目为大于3的正整数，所述第二类型的滤光阵列包括的一个通道用于表示所述第二波段范围内的一种光谱信息。
  5. 根据权利要求3所述的滤光阵列，其特征在于，所述三个通道为RGB通道，RYB通道，RBGIR通道中的任意一种通道。
  6. 根据权利要求1所述的滤光阵列，其特征在于，所述第一类型的滤光阵列包括四通道的滤光阵列，或者所述第一类型的滤光阵列包括三通道的滤光阵列和四通道的滤光阵列，所述第二类型的滤光阵列的通道数目为大于4的正整数。
  7. 根据权利要求6所述的滤光阵列，其特征在于，所述四个通道为红色通道R，绿色通道G，蓝色通道B，黄色通道Y以及近红外通道IR中的任意四个通道。
  8. 根据权利要求6或7所述的滤光阵列，其特征在于，所述第二类型的滤光阵列的通道数目为不小于2且不大于25的正整数，所述第二类型的滤光阵列包括的一个通道用于表示所述第二波段范围内的一种光谱信息。
  9. 根据权利要求1至8任一项所述的滤光阵列，其特征在于，所述第一类型的滤光阵列和所述第二类型的滤光阵列均为矩阵结构。
  10. 根据权利要求1至9任一项所述的滤光阵列，其特征在于，所述滤光阵列包括多组所述第一类型的滤光阵列和多组所述第二类型的滤光阵列，多组所述第一类型的滤光阵列包围多组所述第二类型的滤光阵列。
  11. 根据权利要求10所述的滤光阵列，其特征在于，越靠近所述滤光阵列的中心，所述第二类型的滤光阵列分布密度越大，所述第一类型的滤光阵列的分布密度越小。
  12. 根据权利要求1至9任一项所述的滤光阵列，其特征在于，所述滤光阵列包括多组所述第一类型的滤光阵列和多组所述第二类型的滤光阵列，多组所述第二类型的滤光阵列包围多组所述第一类型的滤光阵列。
  13. 根据权利要求1至9任一项所述的滤光阵列，其特征在于，所述滤光阵列包括多组所述第一类型的滤光阵列和多组所述第二类型的滤光阵列，多组所述第一类型的滤光阵列和多组所述第二类型的滤光阵列并列设置。
  14. 根据权利要求10至13任一项所述的滤光阵列，其特征在于，在所述滤光阵列中， 所述第二类型的滤光阵列是周期性分布的或者非周期性分布的。
  15. 根据权利要求10至14任一项所述的滤光阵列，其特征在于，多组所述第一类型的滤光阵列的总面积大于多组所述第二类型的滤光阵列的总面积。
  16. 根据权利要求1至15任一项所述的滤光阵列，其特征在于，所述第一类型的滤光阵列包括滤色片，微透镜以及光电转换单元，所述光电转换单元用于将光信号转换为电信号，所述滤色片位于所述微透镜和所述光电转换单元的中间，针对一个所述第一类型的滤光阵列，一个微透镜对应一个滤色片，且对应一个光电转换单元，或者一个微透镜对应一个滤色片，且对应多个光电转换单元，或者一个滤色片对应多个微透镜，且对应多个光电转换单元。
  17. 根据权利要求1至15任一项所述的滤光阵列，其特征在于，所述第二类型的滤光阵列包括滤色片，微透镜以及光电转换单元，所述光电转换单元用于将光信号转换为电信号，所述滤色片位于所述微透镜和所述光电转换单元的中间，针对一个所述第二类型的滤光阵列，一个微透镜对应一个滤色片，且对应一个光电转换单元，或者一个微透镜对应一个滤色片，且对应多个光电转换单元，或者一个滤色片对应多个微透镜，且对应多个光电转换单元。
  18. 一种摄像模组，其特征在于，所述摄像模组包括底座和滤光阵列，所述滤光阵列安装在所述底座上，所述滤光阵列为权利要求1至17任一项所述的滤光阵列。
  19. 根据权利要求18所述的摄像模组，其特征在于，所述摄像模组还包括处理器和存储器，所述处理器和所述存储器耦合，所述存储器中存储有与图像处理相关的程序指令，当所述存储器存储的程序指令被所述处理器执行时，对所述第一类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到第一图像，对所述第二类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到第二图像，对所述第一图像和所述第二图像进行融合处理，以得到高分辨率的高光谱图像。
  20. 一种摄像机，其特征在于，所述摄像机包括壳体和摄像模组，所述摄像模组收容于所述壳体内，所述摄像模组为权利要求18所述的摄像模组。
  21. 根据权利要求20所述的摄像机，其特征在于，所述摄像机还包括处理器和存储器，所述处理器和所述存储器耦合，所述存储器中存储有与图像处理相关的程序指令，当所述存储器存储的程序指令被所述处理器执行时，对所述第一类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到第一图像，对所述第二类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到第二图像，对所述第一图像和所述第二图像进行融合处理，以得到高分辨率的高光谱图像。
  22. 一种移动终端，其特征在于，所述移动终端包括镜头，镜头支架和滤光阵列，所述镜头安装在所述镜头支架上，所述镜头支架设置于所述滤光阵列和所述镜头之间，所述滤光阵列为权利要求1至17任一项所述的滤光阵列。
  23. 根据权利要求22所述的移动终端，其特征在于，所述移动终端还包括处理器和存储器，所述处理器和所述存储器耦合，所述存储器中存储有与图像处理相关的程序指令，当所述存储器存储的程序指令被所述处理器执行时，对所述第一类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到第一图像，对所述第二类型的 滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到第二图像，对所述第一图像和所述第二图像进行融合处理，以得到高分辨率的高光谱图像。
  24. 一种智能汽车，其特征在于，其特征在于，所述智能汽车包括镜头，镜头支架和滤光阵列，所述镜头安装在所述镜头支架上，所述镜头支架设置于所述滤光阵列和所述镜头之间，所述滤光阵列为权利要求1至17任一项所述的滤光阵列。
  25. 根据权利要求24所述的智能汽车，其特征在于，所述智能汽车还包括处理器和存储器，所述处理器和所述存储器耦合，所述存储器中存储有与图像处理相关的程序指令，当所述存储器存储的程序指令被所述处理器执行时，对所述第一类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到第一图像，对所述第二类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到第二图像，对所述第一图像和所述第二图像进行融合处理，以得到高分辨率的高光谱图像。
  26. 一种监控设备，其特征在于，所述监控设备包括镜头，镜头支架和滤光阵列，所述镜头安装在所述镜头支架上，所述镜头支架设置于所述滤光阵列和所述镜头之间，所述滤光阵列为权利要求1至17任一项所述的滤光阵列。
  27. 根据权利要求26所述的监控设备，其特征在于，所述监控设备还包括处理器和存储器，所述处理器和所述存储器耦合，所述存储器中存储有与图像处理相关的程序指令，当所述存储器存储的程序指令被所述处理器执行时，对所述第一类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到第一图像，对所述第二类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到第二图像，对所述第一图像和所述第二图像进行融合处理，以得到高分辨率的高光谱图像。
  28. 一种电子设备，其特征在于，所述电子设备包括镜头，镜头支架和滤光阵列，所述镜头安装在所述镜头支架上，所述镜头支架设置于所述滤光阵列和所述镜头之间，所述滤光阵列为权利要求1至17任一项所述的滤光阵列。
  29. 根据权利要求28所述的电子设备，其特征在于，所述电子设备还包括处理器和存储器，所述处理器和所述存储器耦合，所述存储器中存储有与图像处理相关的程序指令，当所述存储器存储的程序指令被所述处理器执行时，对所述第一类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到第一图像，对所述第二类型的滤光阵列获取的图像进行光照度估计、白平衡处理以及解马赛克处理，以得到第二图像，对所述第一图像和所述第二图像进行融合处理，以得到高分辨率的高光谱图像。
PCT/CN2020/104797 2020-07-27 2020-07-27 一种滤光阵列、移动终端以及设备 WO2022020989A1 (zh)

Priority Applications (4)

Application Number Priority Date Filing Date Title
CN202080012899.2A CN114287127B (zh) 2020-07-27 2020-07-27 一种滤光阵列、移动终端以及设备
EP20946880.0A EP4181509A4 (en) 2020-07-27 2020-07-27 FILTER MATRIX, MOBILE TERMINAL, AND DEVICE
PCT/CN2020/104797 WO2022020989A1 (zh) 2020-07-27 2020-07-27 一种滤光阵列、移动终端以及设备
US18/159,724 US20230170363A1 (en) 2020-07-27 2023-01-26 Optical filter array, mobile terminal, and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/104797 WO2022020989A1 (zh) 2020-07-27 2020-07-27 一种滤光阵列、移动终端以及设备

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/159,724 Continuation US20230170363A1 (en) 2020-07-27 2023-01-26 Optical filter array, mobile terminal, and device

Publications (1)

Publication Number Publication Date
WO2022020989A1 true WO2022020989A1 (zh) 2022-02-03

Family

ID=80037257

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/104797 WO2022020989A1 (zh) 2020-07-27 2020-07-27 一种滤光阵列、移动终端以及设备

Country Status (4)

Country Link
US (1) US20230170363A1 (zh)
EP (1) EP4181509A4 (zh)
CN (1) CN114287127B (zh)
WO (1) WO2022020989A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220052090A1 (en) * 2020-08-17 2022-02-17 Au Optronics Corporation Sensing device
US11698478B2 (en) * 2021-03-10 2023-07-11 Viavi Solutions Inc. Optical device

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103854266A (zh) * 2012-12-05 2014-06-11 三星泰科威株式会社 用于对图像进行处理的方法和设备
CN103971351A (zh) * 2013-02-04 2014-08-06 三星泰科威株式会社 使用多光谱滤光器阵列传感器的图像融合方法和设备
CN105675136A (zh) * 2016-03-22 2016-06-15 深圳先进技术研究院 一种编码孔径光谱成像系统
CN110211044A (zh) * 2019-05-24 2019-09-06 北京理工大学 基于去马赛克算法和主成分分析的多光谱成像方法及装置
CN110660088A (zh) * 2018-06-30 2020-01-07 华为技术有限公司 一种图像处理的方法和设备

Family Cites Families (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1136709C (zh) * 2000-03-21 2004-01-28 华为技术有限公司 Ip查找的一种内存回收方法
EP1213569B1 (de) * 2000-12-08 2006-05-17 Gretag-Macbeth AG Vorrichtung zur bildelementweisen Ausmessung eines flächigen Messobjekts
JP2007317750A (ja) * 2006-05-23 2007-12-06 Matsushita Electric Ind Co Ltd 撮像装置
US8514491B2 (en) * 2009-11-20 2013-08-20 Pelican Imaging Corporation Capturing and processing of images using monolithic camera array with heterogeneous imagers
US9654745B2 (en) * 2010-03-17 2017-05-16 Haishan Zeng Rapid multi-spectral imaging methods and apparatus and applications for cancer detection and localization
US20140320611A1 (en) * 2013-04-29 2014-10-30 nanoLambda Korea Multispectral Multi-Camera Display Unit for Accurate Color, Multispectral, or 3D Images
US9521385B2 (en) * 2014-03-27 2016-12-13 Himax Imaging Limited Image sensor equipped with additional group of selectively transmissive filters for illuminant estimation, and associated illuminant estimation method
US9503623B2 (en) * 2014-06-03 2016-11-22 Applied Minds, Llc Color night vision cameras, systems, and methods thereof
JP2016197794A (ja) * 2015-04-03 2016-11-24 株式会社シグマ 撮像装置
CN106840398B (zh) * 2017-01-12 2018-02-02 南京大学 一种多光谱光场成像方法
JP6410203B1 (ja) * 2017-02-21 2018-10-24 株式会社ナノルクス 固体撮像素子及び撮像装置
CN107195650B (zh) * 2017-06-13 2024-05-03 江苏城讯成联网络科技有限公司 多光谱摄像装置
CN108257107A (zh) * 2018-01-11 2018-07-06 天津大学 基于局部低秩矩阵和光谱分解的图像融合算法
CN109883959B (zh) * 2019-02-26 2021-11-23 江苏大学 一种基于阵列传感器芯片的便携式多光谱成像装置及其应用
WO2020199205A1 (zh) * 2019-04-04 2020-10-08 合刃科技（深圳）有限公司 一种混合型高光谱图像重构的方法及系统
CN109856058B (zh) * 2019-04-10 2021-10-08 河北大学 一种高分辨率实时偏振光谱分析装置及方法
CN110620861B (zh) * 2019-09-24 2021-10-15 Oppo广东移动通信有限公司 图像传感器、相机模组和终端

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103854266A (zh) * 2012-12-05 2014-06-11 三星泰科威株式会社 用于对图像进行处理的方法和设备
CN103971351A (zh) * 2013-02-04 2014-08-06 三星泰科威株式会社 使用多光谱滤光器阵列传感器的图像融合方法和设备
CN105675136A (zh) * 2016-03-22 2016-06-15 深圳先进技术研究院 一种编码孔径光谱成像系统
CN110660088A (zh) * 2018-06-30 2020-01-07 华为技术有限公司 一种图像处理的方法和设备
CN110211044A (zh) * 2019-05-24 2019-09-06 北京理工大学 基于去马赛克算法和主成分分析的多光谱成像方法及装置

Also Published As

Publication number Publication date
EP4181509A4 (en) 2023-08-09
EP4181509A1 (en) 2023-05-17
CN114287127A (zh) 2022-04-05
CN114287127B (zh) 2023-10-31
US20230170363A1 (en) 2023-06-01

Similar Documents

Publication Publication Date Title
WO2021227250A1 (zh) 图像传感器和电子设备
US8462238B2 (en) Camera design for the simultaneous capture of near-infrared and visible images
JP5074106B2 (ja) 固体撮像素子及びカメラ
US8576313B2 (en) Color filters and demosaicing techniques for digital imaging
US20030103150A1 (en) Integrated color pixel ( ICP )
US20230170363A1 (en) Optical filter array, mobile terminal, and device
US11747533B2 (en) Spectral sensor system using optical filter subarrays
TWI579540B (zh) 多點光譜系統
KR20140041679A (ko) 다중 피사계 심도 감광소자, 시스템, 피사계 심도 확장방법 및 광학 화상 시스템
JP2004228662A (ja) 撮像装置
CN103052914B (zh) 三维摄像装置
WO2022037557A1 (zh) 一种图像传感器、信号处理方法以及相关设备
Zhao et al. Hyperspectral imaging with random printed mask
CN108370422A (zh) 利用单个矩阵传感器采集可见和近红外图像的系统和方法
US20180184052A1 (en) Image generation device and imaging device
CN111741239B (zh) 图像传感器和电子设备
CN114830341A (zh) 一种图像传感器、分光滤色器件及图像传感器的制备方法
CN204633909U (zh) 成像器
WO2013099151A1 (ja) 固体撮像素子、撮像装置、および信号処理方法
WO2015133130A1 (ja) 映像撮影装置、信号分離装置および映像撮影方法
US11696043B2 (en) White balance compensation using a spectral sensor system
CN115000107B (zh) 多光谱成像芯片、多光谱成像组件、制备方法及移动终端
CA2996199A1 (en) Color filter sensors
Tominaga CIC@ 20: multispectral imaging
US20230082539A1 (en) Illuminant correction in an imaging system

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20946880

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2020946880

Country of ref document: EP

Effective date: 20230207