WO2016201562A1 - Systèmes et procédés d'imagerie de caméra - Google Patents

Systèmes et procédés d'imagerie de caméra Download PDF

Info

Publication number
WO2016201562A1
WO2016201562A1 PCT/CA2016/050685 CA2016050685W WO2016201562A1 WO 2016201562 A1 WO2016201562 A1 WO 2016201562A1 CA 2016050685 W CA2016050685 W CA 2016050685W WO 2016201562 A1 WO2016201562 A1 WO 2016201562A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
digital image
high resolution
multispectral
format
Prior art date
Application number
PCT/CA2016/050685
Other languages
English (en)
Inventor
Yun Zhang
Original Assignee
University Of New Brunswick
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US14/743,993 external-priority patent/US9930316B2/en
Application filed by University Of New Brunswick filed Critical University Of New Brunswick
Publication of WO2016201562A1 publication Critical patent/WO2016201562A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
    • H04N23/13—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths with multiple sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N2209/00—Details of colour television systems
    • H04N2209/04—Picture signal generators
    • H04N2209/041—Picture signal generators using solid-state devices
    • H04N2209/048—Picture signal generators using solid-state devices having several pick-up sensors

Definitions

  • This invention relates to camera imaging.
  • FIG. 1 is a schematic diagram of a dual video image system according to one or more embodiments of the present invention
  • FIG. 2 is a schematic diagram of consecutive image frames according to one or more embodiments of the present invention.
  • FIG. 3 is a schematic diagram of a two camera system according to one or more embodiments of the present invention.
  • FIG. 4 is a schematic diagram of a two sensor single lens system according to one or more embodiments of the present invention.
  • Fig. 5 is a schematic diagram of a rotating half mirror system according to one or more embodiments of the present invention.
  • FIG. 6 illustrates the general relationship between solar radiation and sensor response with respect to the wavelengths and spectral band widths of Pan and MS spectral bands (prior art);
  • FIG. 7 illustrates a configuration of a 2-chip color camera system and method, according to one embodiment of the invention.
  • FIG. 8 is an illustration of the image data flow and the image data volume in the transmission, storage and display stages according to one embodiment of the invention.
  • FIG. 9 illustrates image transmission, storage and display flow of prior art 1-chip color and 3-chip color cameras
  • FIG. 10 illustrates the configuration of a 2-chip 4-color color camera with one high resolution Pan (H.Pan) sensor and one low resolution 4-color MS (L.MS) sensor, and steps of creating a high resolution color image, according to one embodiment of the invention
  • FIG. 11 is a conceptual illustration of image data flow and image data volume in the transmission, storage and display stages, according to an embodiment of the invention.
  • FIG. 12 illustrates the concept of the image transmission, storage and display flow of a 1- chip 4-color and 4-chip 4-color cameras based in a prior art design
  • FIG. 13 illustrates the configuration of a 2-chip 3-color camera with two high resolution sensors, one high resolution Pan (H.Pan) sensor and one high resolution MS (H.MS) sensor, and steps of creating a high sensitivity and high resolution color image, according to an embodiment of the invention
  • FIG. 14 illustrates the configuration of a 2-chip 4-color camera with two high resolution sensors, one high resolution Pan (H.Pan) sensor and one high resolution 4-color MS (H.MS) sensor, and steps of creating a high resolution color image, according to an embodiment of the invention
  • FIG. 15 illustrates the configuration of a 2-chip color camera with one high resolution Pan (H.Pan) sensor and one low resolution MS (L.MS) sensor, according to the invention.
  • FIG. 16 illustrates data size before and after de-bayermg and fusion, according to an embodiment of the invention
  • FIG. 17 illustrates the configuration of a 2-chip 4-color camera with one high resolution Pan (H.Pan) sensor and one low resolution 4-color MS (L.MS) sensor, according to an embodiment of the invention
  • FIG. 18 illustrates data size before and after de-bayering and fusion, according to an embodiment of the invention
  • FIG. 19 illustrates the configuration of a 2-chip color camera with two high resolution sensors, one high resolution Pan (H.Pan) sensor and one high resolution MS (H.MS) sensor, according to an embodiment of the invention
  • FIG. 20 illustrates the configuration of a 2-chip 4-color camera with two high resolution sensors, one high resolution Pan (H.Pan) sensor and one high resolution MS (H.MS) sensor with a R-G-B-NTR color filter, according to an embodiment of the invention.
  • a video imaging system comprising a low resolution colour digital video camera and a high resolution monochromatic digital video camera operably connected to a digital processing system
  • the system can further comprise an object motion module for detecting objects moving within the fields of view of the cameras, and an object position module for detenrdning the position of an object in the overlapping field of view of the cameras.
  • a method comprising providing an image frame from a low resolution colour digital video camera and a corresponding image frame from high resolution monochromatic digital video camera and fusing the two image frames to obtain a colour image having higher resolution than the image frame from the low resolution colour digital video camera.
  • the method can further comprise providing a three dimensional coordinate system for detenrtining the position of a moving object in the overlapping fields of view of the cameras whereby the two dimensional position of the moving object is determined according its position in the images, whereas the distance from the cameras to the object in the axis perpendicular to the plane of the images is derived from the parallax error between the two image frames to be fused.
  • a camera imaging system comprising a low resolution colour digital sensor chip, a high resolution monochromatic digital sensor chip, a beam sputter, and a lens, wherein the lens gathers incident light towards the beam splitter, and the beam splitter splits the light towards the two sensor chips.
  • the system further comprises a digital processing system which fuses a low resolution colour image from the colour sensor and a high resolution monochromatic image from monochromatic sensor to produce a high resolution colour image.
  • the technology relates to a method to: (1) directly capture data-volume-reduced or compressed color images or video image frames by a camera system compared to those from conventional color camera systems with the same image resolution, and
  • the technology relates to:
  • the raw data volume of a conventional 9 mega pixels (“MP") color camera is 3x9MP for a total of 27MP, with each color band (R, G, B) having 9MP.
  • the data volume would be less.
  • the combined raw data volume of images of a scene acquired by the sensors is only 3MP for the color sensor (IMP for each of the 3 color bands) plus 9MP for the Pan sensor for a total of 12MP.
  • the data volume becomes 3x9MP for a total of 27MP. Therefore, the combined raw data volume of the images before pan-sharpening is equal to the data volume of an equivalent lossless compressed color image.
  • the camera sensor can be ground-based, aerial vehicle based including unmanned aerial vehicle (“UAV”)-based, or satellite-based including small satellite based.
  • UAV unmanned aerial vehicle
  • the technology relates to methods to minimize the data volume for storage and transmission and to maximize image resolution and image quality for display and visualization.
  • the technology in another aspect, relates to a method including providing a low resolution multispectral digital image from a camera imaging system, providing a high resolution monochromatic digital image from the camera imaging system, where the images are of the same scene and have a degree of overlap, generating, at the camera imaging system, a single digital file comprising the images, where the images have a common file format, and are suitable for processing into a fused high resolution multispectral image by a client device, and where the combined data volume of the low resolution multispectral digital image and the monochromatic digital image is less than the data volume of the fused high resolution multispectral image.
  • the method may further include de-bayering the multispectral digital image into individual color image bands.
  • the method may also further include binning a high resolution multispectral image to produce a low resolution multispectral image.
  • the method may also include the low resolution multispectral digital image in generating the single digital file.
  • the low resolution multispectral digital image may include color bands selected from the group consisting of red, green, blue, near infrared and beyond.
  • the digital file may further include a format name which is readable by a digital image fusion software to enable the software to process the images into a fused high resolution multispectral image.
  • the common file format may be selected from the group consisting of a file compression format, TIFF, JPEG, JPEG2000, raw format, and other formats.
  • the multispectral digital image may include a plurality of multispectral digital image frames forming a video
  • the monochroma tic digital image comprises a plurality of monochrome digital image frames forming a video.
  • the common file format may be selected from the group consisting of a video compression format, H.264, MPEG-4, and other formats.
  • the technology in another aspect, relates to a camera imaging system including a multispectral digital sensor, a monochromatic digital sensor, a processor, a memory in communication with the processor, the memory storing a plurality of processing instructions that, when executed, cause the processor to perform operations including receiving a low resolution multispectral digital image from the multispectral digital sensor, receiving a high resolution monochromatic digital image from the monochromatic digital sensor, where the images are of the same scene and have a degree of overlap, formatting the images with a common file format; generating a single digital file comprising the formatted images, where the file can be read by a digital image fusion software at a client device to enable the software to process the images into a fused high resolution multispectral image at the client device, and where the combined data volume of the low resolution multispectral digital image and the monochromatic digital image is less than the data volume of the fused high resolution multispectral image.
  • the operations performed by the processor may further include de-bayering the multispectral digital image into individual color image bands.
  • the method may further include binning a high resolution multispectral image to produce a low resolution multispectral image.
  • the method may also include the low resolution multispectral digital image in generating the single digital file.
  • the low resolution multispectral digital image may further include color bands selected from the group consisting of red, green, blue, near infrared, and beyond.
  • the operations performed by the processor may further include assigning a format name to the single digital file which is readable by the digital image fusion software to indicate to the software that the single file comprises a low resolution multispectral digital image and a high resolution monochromatic digital image which are in a format suitable for processing into a fused high resolution multispectral image by the software at the client device, m the camera imaging system, in the operations performed by the processor, the common file format may be selected from the group consisting of a file compression format, TIFF, JPEG, JPEG2000, raw format and other image formats.
  • the multispectral digital image may include a plurality of multispectral digital image frames forming a video
  • the monochromatic digital image comprises a plurality of monochrome digital image frames forming a video
  • the common file format may be selected from the group consisting of a video compression format, H.264, MPEG-4, and other formats.
  • the technology in another aspect, relates to a method including receiving a single digital file comprising a low resolution multispectral digital image and a high resolution monochromatic digital image, where the images have been acquired by a camera imaging system, have a common file format, and are suitable for processing into a fused high resolution multispectral image, and where the combined data volume of the low resolution multispectral digital image and the monochromatic digital image is less man the data volume of the fused high resolution multispectral image, and processes the images at a client device, using digital image fusion software, into a fused high resolution multispectral image.
  • the single digital file may be received on the client device over a network or by direct connection to the camera imaging system.
  • the method may further include de-bayering the low resolution multispectral digital image into individual color image bands and iriduding the individual color image bands in place of the low resolution multispectral digital image in the processing step.
  • the low resolution multispectral digital image may include color bands selected from the group consisting of red, green, blue, near infrared, and beyond.
  • the digital file may further include a format name which is readable by the digital image fusion software to indicate to the software that the single file comprises a low resolution multispectral digital image and a high resolution monochromatic digital image which are in a format suitable for processing into a fused high resolution multispectral image by the software.
  • the common file format may be selected from the group consisting of a file compression format, TIFF, JPEG, JPEG2000, raw format and other formats.
  • the multispectral digital image may include a plurality of multispectral digital image frames forming a video
  • the monochromatic digital image comprises a plurality of monochrome digital image frames forming a video.
  • the common file format may be selected from the group consisting of a video compression format; H.264 MPEG-4, and other formats.
  • the technology in another aspect, relates to a client device including a processor, a memory in communication with the processor, the memory storing a plurality of processing instructions that, when executed, cause the processor to perform operations including receiving a single digital file comprising a low resolution multispectral digital image and a high resolution monochromatic digital image, where the images have been acquired by a camera imaging system, have a common file format and, are suitable for processing into a fused high resolution multispectral image, and where the combined data volume of the low resolution multispectral digital image and the monochromatic digital image is less than the data volume of the fused high resolution multispectral image, processing the images at a client device / using digital image fusion software, into a fused high resolution multispectral image.
  • the operations performed by the processor may further include de- bayering the multispectral digital image into individual color image bands, binning a high resolution multispectral image to produce a low resolution multispectral image, and / or including the low resolution multispectral digital image in the processing operation.
  • the low resolution multispectral digital image may include color bands selected from the group consisting of red, green, bhie, near infrared and beyond.
  • the device may further include a format name which is readable by the digital image fusion software to indicate to the software that the single hie comprises a low resolution multispectral digital image and a high resolution monochromatic digital image which are in a format suitable for processing into a fused high resolution multispectral image by the software.
  • the common file format may be selected from the group consisting of a file compression format, TIFF, JPEG, JPEG2000, raw format, and other formats.
  • the multispectral digital image may include a plurality of multispectral digital image frames fonning a video
  • the monochromatic digital image comprises a plurality of monochrome digital image frames fonning a video.
  • the common file format may be selected from the group consisting of a video compression format, H.264, MPEG-4, and other formats.
  • the multispectral image may be a color image which has a narrow spectral bandwidth
  • the monochromatic image may be a panchromatic image or black and white image which has a broad spectral bandwidth.
  • the technology relates to a computer readable storage medium storing instructions that; when executed by a processor, causing the processor to perform steps according to the embodiments of the invention described herein.
  • a camera system can capture compressed color images and achieve higher light sensitivity and better image quality compared to conventional 1-chip and 3-chip color cameras is provided.
  • the camera system utilizes human eyes' sensitivity difference to himinance and chrominance, sensors' sensitivity difference to panchromatic (Pan) and multispectral (MS, or color) images, and light energy distribution in visible and infrared spectrum to achieve the goals of color image compression, sensor sensitivity improvement and image quality improvement
  • FIG. 6 illustrates the general relationship between solar radiation and sensor response with respect to the wavelengths and spectral band widths of Pan and MS spectral bands. Because Pan sensors mainly capture luminance information and MS sensors mainly capture chrominance information and also because Pan sensors are much more sensitive than MS sensors due to their much broader spectral bandwidths, these characteristics are utilized in this invention to directly capture compressed color images.
  • the general principle of the system according to an embodiment of the present invention is described in the following sections in more detail
  • a method includes (1) using a panchromatic (Pan) sensor to capture a Pan (black and white) image covering a broad spectral bandwidth of the incoming light and (2) a multispectral (MS) sensor to capture a MS (color) image with a lower spatial resolution to reduce the data volume.
  • Pan panchromatic
  • MS multispectral
  • the two images are from the same scene and overlap with each other.
  • the combination of a high resolution Pan image and a low resolution MS color image forms a compressed high resolution color image, which mimics a lossless compression through the RGB to YUV conversion and the downsampling of the chrominance components in an image compression process.
  • a camera with a Pan and a MS sensor is referred to herein as a 2-chip color camera.
  • the Pan sensor in the 2-chip color camera is a CCD (charge-coupled device) or CMOS (complementary melal-oxide-semiconductor) sensor.
  • the Pan sensor can be used without a light filter in order to capture as much light as possible.
  • the Pan sensor can also be used with a partial light filter to block certain unwanted light, such as near infrared, to capture a broad spectral bandwidth of the mcoming light in visible spectral area.
  • the MS sensor in the 2-chip color camera is a CCD or CMOS sensor with a color filter to allow each pixel to receive only one color, either blue, green, red or near infrared.
  • the near infrared includes all the light in the infrared area that can be sensed by a CCD or CMOS sensor.
  • the MS sensor can also be designed to allow each pixel to receive blue, green or red light respectively to capture a natural color image, hi order to capture special colors for special uses, the color filter can also be designed to allow each pixel to receive blue, green or red, or a color in the spectral wavelength below blue, between blue and green, green and red, red and near infrared, or different near infrared bands, respectively.
  • a high resolution color image can be produced by fusing the high resolution Pan image and the low resolution MS image before displaying or printing or using the high resolution color image, or before the manipulation of the high resolution color image for other image processing or visualization purposes. This fusion process mimics the image decompression process of a conventional image compression and decompression process.
  • the chip sizes of the Pan and MS sensors are the same, and the optical lenses of the two sensors are also the same. This allows for the two sensors to capture images covering the same area with the same dimension.
  • the spatial resolution of the Pan image is higher than that of the MS image.
  • the low resolution MS image is either directly collected by a low resolution MS sensor with a larger pixel size, or generated by binning the pixels of a high resolution MS sensor or binning the color image pixels collected by a high resolution MS sensor.
  • the benefit of capturing a high resolution Pan image and a low resolution MS image is not only the reduction of image data volume, but also the increase of the sensitivity of the 2-chip color camera and the improvement of the color image quality. Because a Pan sensor is much more sensitive than an MS sensor, and the sensitivity of a color sensor can be increased by lowering the resolution of the color image through either enlarging the sensor pixel size or binning the image pixels, a high resolution Pan sensor and a low resolution MS sensor with a similar level of high sensitivity can be found or created. The combination of the two highly sensitive sensors creates a highly sensitive and high resolution color camera.
  • the 2-chip color camera can produce full resolution color images (i.e. every pixel in every color band has information); whereas a conventional 1-chip color camera can only produce "hollow 0 resolution color images (Le. only 1/4 of the pixels in blue and red bands and 1/2 of the pixels in green band have information).
  • an overall system configuration for a dual video imaging system comprises a colour digital video camera 2 having lens 20 and a monochromatic digital video camera 4 having lens 22.
  • the cameras 2 and 4 each generate a digital signal of scene 6, which is then transmitted to digital processing system ("DPS") 1Z
  • the cameras 2 and 4 employ charge-coupled device (“CCD”) sensors or complementary metal-oxide-semiconductor (“CMOS”) sensors.
  • CCD charge-coupled device
  • CMOS complementary metal-oxide-semiconductor
  • Camera 2 is a low resolution colour (“LC”) video camera while camera 4 is a high resolution monochrome (“HM”) video camera.
  • LC low resolution colour
  • HM monochrome
  • Cameras 2 and 4 are capable of providing streaming video signals as part of a security, surveillance or monitoring system. It will be understood, however, that the applications for the cameras 2 and 4 are not limited to such systems.
  • Camera 2 has a field of view defined by light rays 8 while camera 4 has a field of view defined by light rays 10.
  • Colour camera 2 and monochrome camera 4 produce separate streaming video signals which are then supplied to the DPS 12.
  • the cameras 2 and 4 are adjacent and can be housed together in a single camera housing (not shown).
  • the low resolution colour streaming video signals from camera 2 are fused by image fusing module ("FM") 26 in processor 12 with corresponding high resolution monochrome streaming video signals from camera 4 to produce a fused high resolution colour streaming video signal (“HC) 28.
  • FM image fusing module
  • HC fused high resolution colour streaming video signal
  • Fusing the colour and monochrome video signals provides the dual camera system with improved sensitivity capable of acquiring high resolution colour video signals under poor lighting conditions due to the inclusion of the high resolution signal from the monochrome camera and the colour signal from the colour camera.
  • the colour and monochrome video signals are comprised of individual image frames. Corresponding pairs of video image frames from cameras 2 and 4 are isolated and then fused. Various methods for fusing the frame pairs can be used. For example, image fusion methods for fusing a low resolution multJspectral satellite images with high resolution panchromatic satellite images are known in the field of remote sensing and can be adapted to fuse video image frames from cameras 2 and 4. One such fusion method is disclosed in US. Patent No. 7,340,099 (Zhang) which is incorporated herein by reference in its entirety. Other image fusion methods used for satellite imagery include arithmetic based, statistics based, ratio based and wavelet based methods. By substituting colour and monochrome video image frame pairs according to the present invention for multispectral and panchromatic images respectively, prior art image fusing methods can be adapted to fuse video image frames acquired by camera 2 with video image frames acquired by camera 4.
  • a moving object 30 in the scene 6 can be detected by both video cameras 2 and 4 based on finding changes in the consecutive image frames 32 of each video camera. If there is no moving object in the scene 6, the images in the consecutive frames will be the same. If there is a moving object 30 in scene 6, the images in the consecutive frames will be not the same.
  • the changed area between two adjacent frames is the location of the moving object on the image. The changed areas can be found by comparing images in adjacent frames. Suitable conventional methods, techniques and algorithms for comparing consecutive image frames and finding changes in such image frames can be adopted for this system to find moving objects in consecutive images frames from each of the two cameras 2 and 4.
  • the position of objects Ol and 02 in the scene 6 is provided in a three dimensional coordinate system Since cameras 2 and 4 are adjacent, not overlaid with each other, the light rays from lens 20 and lens 22 to any object in the scene 6 are not parallel. The closer the object to the two lenses, the larger the angle between the two light rays from the two lenses to the object For example, the object 02 is closer to the two lenses 20 and 22 than is object Ol. The angle A2 is therefore larger than angle A1.
  • the distance from lenses 20 and 22 to any object in the scene 6 can be calculated according to the base line distance between lenses 20 and 22 and the viewing angle between the two light rays. This distance gives the coordinate along the Z-axis of the three- dimensional coordinate system.
  • the plane of the image frames 32 in FIG. 2 is defined by an X-Y coordinate system which is used to position objects (such as object 30) in two dimensions on the image frames 32
  • the X-Y axis position of an object phis its Z-axis position provides the object's position in three dimensions relative to cameras 2 and 4.
  • the X-Y-Z positions are provided to a position calculating module in the DPS 12 which calculates the position of objects in the scene 6.
  • the position calculating module is programmed with suitable computer algorithms based on prior art photogrammetric or computer vision methods as described above.
  • the dual camera system according to the present invention provides colour video with improved sensitivity compared with a conventional video camera, the detection of moving objects, and the three dimensional position of the objects in the common field of view of the cameras 2 and 4.
  • methods of the present invention can be applied to image frames from two corresponding still cameras.
  • the dual imaging system described above with initial reference to FIG. 1 is modified by replacing the dual cameras and dual lenses with a dual sensor single lens camera.
  • Lens 22 is omitted and a light splitter 40 is added.
  • the light beam splitter 40 splits the incoming light into two directions.
  • Camera 2 is reconfigured with its low resolution digital colour sensor 42 towards one of the split light beams and camera 4 is re-configured with its high resolution monochrome digital sensor 44 towards the other split beam.
  • Cameras 2 and 4 are positioned such that when the light splitter 40 splits the incoming light into two directions, about half of the incident light is directed towards the colour digital sensor 42 and about the other half of the incident light is directed towards the monochromatic digital sensor 44. In this embodiment, the capacity of detecting distance from the camera to a moving object is reduced. Separate streaming video signals from sensors 42 and 44 are then supplied to the DPS 12 in a similar manner to the signals from cameras 2 and 4 in the system described with initial reference to FIG. 1.
  • Low resolution colour streaming video signals from sensor 42 are fused by the FM 26 in processor 12 with corresponding high resolution monochrome streaming video signals from sensor 44 to produce a fused high resolution colour streaming video signal ("HC") 28 using the methods described herein.
  • a rotating half mirror 50 or vibrating mirror can be used in place of the splitter 40 of FIG.4.
  • the half mirror 50 rotates between a first position (P1) where incident light from the lens 20 passes through the empty half of the mirror directly reaching the sensor 44 and a second position (P2) where incident light passing through the lens 20 is directed by the half mirror to the sensor 42.
  • the mirror 50 rotates between the first and second positions sufficiently quickly to obtain a suitable image pair of a scene.
  • splitter 40 or a mirror 50, as long as they can direct incident light from lens 20 towards both sensor 42 and sensor 44 simultaneously.
  • Figure 7 illustrates the configuration of a 2-chip color camera, according to one embodiment of the invention, with one high resolution Pan (H.Pan) sensor and one low resolution MS (L.MS) sensor, and the steps of creating a high resolution color image.
  • the L.MS sensor has a RGB Bayer filter (3-color) to allow each pixel to receive only one of the three primary colors.
  • a de-bayering process is carried out in the camera to interpolate the recorded red, green and blue pixels to produce three low resolution color image bands (De- bayered L.MS).
  • the H.Pan and the De-bayered L.MS images are fused together when there is a need to see or use the high resolution color image (3 x H.MS). If there is no immediate need to see or use the high resolution color image, the H.Pan image and De-bayered L.MS image will not be fused together.
  • the total data volume is 48 MB before the fusion and 108MB after the fusion. If the 48MB is used for storage and transmission purposes and the 108MB is used for final uses of the high resolution color image, a compression ratio of more than 2:1 is achieved.
  • Figure 8 is a conceptual illustration of the image data flow and the image data volume in the transmission, storage and display stages.
  • the H.Pan (36MB) and the three De-bayered low resolution color bands (3 x 4MB), totaling 48MB are used for data storage, in a camera storage, DVD or other storage devices, and for data transmission, through a network such as the Internet or the cloud, or through a direct connection such as through a cable.
  • a network such as the Internet or the cloud
  • a direct connection such as through a cable.
  • the image fusion is used to produce the high resolution color image (3 x 36MB), totalling 108MB, in real time on the display side. More than 2:1 data compression ratio is achieved for storage and transmission.
  • An image compression can be applied to the H.Pan and the De-bayered L.MS image respectively to further reduce the data volume on top of the 2:1 compression ratio.
  • a decompression needs to be applied before the image fusion.
  • a much higher data compression ratio can be achieved for storage and transmission.
  • Figure 9 illustrates the concept of the image transmission, storage and display flow of conventional 1-chip color and 3-chip color cameras. If the raw image data are used, the three high resolution (RGB) color bands (3 x 36MB), totaling 108 MB, are used for viewing on the camera screen; for data storage in a camera storage, DVD or other storage devices; for data transmission through the Internet, cable or the cloud; and for final display and other purposes. No data compression is achieved for the storage and transmission.
  • RGB RGB
  • the data volume of the high resolution color image can be reduced.
  • the data compression ratio is significantly lower than that of the 2-chip color camera described above for keeping the same image quality.
  • Figure 10 illustrates the configuration of a 2-chip 4-color color camera with one high resolution Pan (H.Pan) sensor and one low resolution 4-color MS (L.MS) sensor, and the steps of creating a high resolution color image.
  • the L.MS sensor has a R-G-B-NIR color filter to allow each pixel just to receive either red, green, blue or near infrared (NIR) respectively.
  • a color interpolation similar to de-bayering process is done in the camera to interpolate the recorded red, green, blue and NIR pixels to produce four low resolution color image bands (De-bayered L.MS).
  • the H.Pan and the De-bayered L.MS images are fused together when there is a need to see or use the high resolution 4-color image (4 x H.MS). If there is no immediate need to see or use the high resolution color image, the H.Pan image and De-bayered L.MS image will not be fused together.
  • the total data volume is 52 MB before the fusion and 144 MB after the fusion. If the 52 MB is used for storage and transmission purposes and the 144 MB is used for final uses of the high resolution 4-color image, a compression ratio of more than 2.5:1 is achieved.
  • Figure 11 is a conceptual illustration of the image data flow and the image data volume in the tiansmission, storage and display stages.
  • the RPan (36MB) and the four De-bayered low resolution color bands (4 x 4MB), totaling 52MB are used for data storage, in a camera storage, DVD or other storage devices, and for data transmission, through the Internet, cable or the cloud.
  • the image fusion is used to produce the 4-color high resolution color image (4 x 36MB), totalling 144 MB, in real time on the display side. More than 2.5:1 data compression is achieved for storage and transmission.
  • An image compression can be applied to the H.Pan image and the De-bayered low resolution 4-color image respectively, the total image data volume can be further reduced on top of the 2.5:1 compression ratio.
  • a decompression needs to be applied before the image fusion.
  • a much higher data compression ratio can be achieved for storage and transmission.
  • Figure 12 illustrates the concept of the image transmission, storage and display flow of a 1-chip 4-color and 4-chip 4-color cameras based on the conventional camera design.
  • No 1-chip 4-color camera has been found in the market; whereas a few 4-chip 4-color cameras have been developed for remote sensing purposes.
  • the four high resolution color bands RGB and NIR
  • 4 x 36 MB totaling 144 MB
  • No data compression is achieved for storage and transmission.
  • the data volume of the high resolution color image can be reduced.
  • the data compression ratio can be achieved for storage and transmission is significantly lower than that of the 2-chip 4-color camera described above for keeping the same image quality.
  • Figure 13 illustrates the configuration of a 2-chip 3-color camera with two high resolution sensors, one high resolution Pan (H.Pan) sensor and one high resolution MS (H.MS) sensor, and the steps of creating a high sensitivity and high resolution color image.
  • the H.MS sensor has a RGB Bayer filter to allow each pixel just to receive one of the three primary colors.
  • a pixel binning and de-bayering process is conducted in the camera to lower the resolution of the color bands to produce three low resolution color bands (De- bayered L.MS) and to increase the sensitivity of the color sensor.
  • the de-bayering process can be removed without having a negative impact.
  • the H.Pan and the De-bayered L.MS images are fused together when there is a need to see or use the high resolution color image (3 x H.MS).
  • the fused high resolution color image has much better image quality than that of a 1-chip 3-color high resolution camera, because the 2-chip 3-color camera has a much higher light sensitivity and produces a full resolution color image, not "hollow" resolution.
  • the total data volume is 48 MB before the fusion and 108MB after the fusion. If the 48MB is used for storage and transmission purposes and the 108MB is used for final uses of the high resolution color image, a compression ratio of more than 2:1 is achieved.
  • An image compression can be applied to the H.Pan and the De-bayered low resolution color image respectively to further reduce the data volume on top of the 2:1 compression ratio. A much higher data compression ratio can be achieved for storage and transmission.
  • Figure 14 illustrates the configuration of a 2-chip 4-color camera with two high resolution sensors, one high resolution Pan (H.Pan) sensor and one high resolution 4-color MS (H.MS) sensor, and the steps of creating a high resolution color image.
  • the H.MS sensor has a R-G-B-NIR color filter to allow each pixel just to receive either R, G, B or NIR respectively.
  • a pixel binning and de-bayering process is conducted in the camera to lower the resolution of the color bands to produce four low resolution color bands (De-bayered L.MS) and to increase the sensitivity of the color sensor.
  • the de-bayering process can be removed without having a negative impact
  • the H.Pan and the De-bayered L.MS images are fused together when there is a need to see or use the high resolution color image (4 x H.MS). If there is no immediate need to see or use the high resolution color image, the H.Pan image and De-bayered L.MS image will not be fused together.
  • the total data volume is 52 MB before the fusion and 144 MB after the fusion. If the 52 MB is used for storage and transmission purposes and the 144 MB is used for final uses of the high resolution 4-color image, a compression ratio of more than 2.5:1 is achieved.
  • An image compression can be applied to the H.Pan and the De-bayered low resolution color image to further reduce the data volume on top of the 2.5:1 compression ratio. A much higher data compression ratio can be achieved for storage and transmission.
  • Figure 15 illustrates the configuration of a 2-chip color camera with one high resolution Pan (H.Pan) sensor and one low resolution MS (L.MS) sensor.
  • the L.MS sensor has a RGB Bayer filter ID allow each pixel just to receive one of the three primary colors.
  • a de-bayering process is not done in me camera but on the display side after the transmission and before the image fusion and image display.
  • the H.Pan and the De-bayered L.MS images are fused together when mere is a need to see or use the high resolution color image (3 x H.MS).
  • the total data volume is 40 MB before the de-bayering and fusion and 108MB after the de-bayering and fusion. If the 40 MB is used for storage and transmission purposes and the 108MB is used for final uses of the high resolution color image, a compression ratio of 2.7:1 is achieved.
  • Figure 17 illustrates the configuration of a 2-chip 4-color camera with one high resolution Pan (H.Pan) sensor and one low resolution 4-color MS (L.MS) sensor.
  • the L.MS sensor has a R-G-B-NIR color filter to allow each pixel just to receive either R, G, B or NIR color.
  • a de-bayering process is done on the display side after the transmission and before the image fusion and image display.
  • the H.Pan and the De-bayered L.MS images are fused together when mere is a need to see or use the high resolution color image (4 x H.MS). If there is no immediate need to see or use the high resolution color image, the H.Pan image and De-bayered L.MS image will not be fused together.
  • the total data volume is 40 MB before the de-bayering and fusion and 144 MB after the de-bayering and fusion. If the 40 MB is used for storage and transmission purposes and the 144 MB (4 x 36 MB) is used for final applications of the high resolution color image, a compression ratio of 3.6:1 is achieved.
  • Figure 19 illustrates the configuration of a 2-chip color camera with two high resolution sensors, one high resolution Pan (H.Pan) sensor and one high resolution MS (H.MS) sensor.
  • the H.MS sensor has a RGB Bayer filter.
  • a pixel binning is conducted in the camera to lower the resolution of the color pixels to create low resolution MS (L.MS) to increase the sensitivity of the color sensor.
  • the H.Pan (36 MB) and the L.MS (4 MB), totaling 40 MB, are stored and transmitted to a recipient device.
  • a de-bayering process and fusion process are conducted to produce a high resolution color image (3 x 36 MB), totaling 108 MB; achieving a 2.7:1 compression ratio for the raw color image.
  • Figure 20 illustrates the configuration of a 2-chip 4-color camera with two high resolution sensors, one high resolution Pan (H.Pan) sensor and one high resolution MS (H.MS) sensor with a R-G-B-NIR color filter.
  • a pixel binning is conducted in the camera to lower the resolution of the color pixels to create low resolution MS (L.MS) to increase the sensitivity of the color sensor.
  • the H.Pan (36 MB) and the L.MS (4 MB), totaling 40 MB, are stored and transmitted to a recipient device.
  • a de-bayering process and fusion process are conducted to produce a high resolution color image (4 x 36 MB), totaling $144 MB; achieving a 3.6:1 compression ratio for the raw color image.
  • image depth i.e. 8 bits
  • image depth i.e. 8 bits
  • other sensor configurations are possible.
  • image depths can also be used, such as 6 bits, 11 bits, 12 bits or 16 bits, depending on the need.
  • the resolution difference between the Pan image and MS image is based on the assumption that the linear ratio of pixel sizes between H. Pan and L.MS is 1:3, i.e. 3x3 Pan pixels cover the same area of 1 MS pixel.
  • the resolution difference between H.Pan and L.MS can also be set to a linear pixel size ratio of 1:8, 1:7, 1:6, 1:5, 1:4, 1:2 or 1:1, etc., depending on the lighting condition of the scene and sensitivity difference, between the Pan sensor and the MS sensor.
  • a larger linear pixel size ratio such as 1:6 or 1:5 or 1:4, should be used.
  • 1:2 or 1:1 ratio can be used.
  • a large linear pixel size ratio such as 1:7, 1:6, 1:5, or 1:4 should be used too, to boost the sensitivity of the color sensor.
  • the number of color bands is assumed to be 3 or 4.
  • Three color bands (RGB) are widely used in normal consumer products.
  • Four color bands (R, G, B, NIR) can be useful for professional uses, such as scientific, industry, security, and military uses. More than four color bands can be developed through redesign the color filter.
  • an effective way to utilize the sensor compressed color images for saving storage space and increasing transmission speed is to:
  • the high resolution Pan and low resolution MS images can be further compressed respectively, using a compression technique, such as JPEG, JPEG 2000, or others.
  • a compression technique such as JPEG, JPEG 2000, or others.
  • the Pan and MS images need to be decompressed respectively and then fused together to produce the final high resolution MS image. Otherwise, the Pan and MS images will not be decompressed and fused, to keep a small data volume for storage and transmission.
  • still image compression such as JPEG
  • video image compression such as H.264 or MPEG-4
  • JPEG Joint Photographic Experts Group
  • video image compression such as H.264 or MPEG-4
  • the Pan and MS image streams need to be decompressed respectively and then fused together to produce the final high resolution MS image stream. Otherwise, the Pan and MS image streams will not be decompressed and fused, to keep a small data volume.
  • the image compression, video compression and/ or image fusion software can be pre-installed on the end users' computer or display device.
  • the final high resolution color image or color image stream is produced when it is needed. If the Pan and MS images or image streams are not compressed using either an image compression or video compression technique, the image fusion technique can be directly applied to fuse the Pan and MS images or image streams.
  • the image compression, video compression and/ or image fusion software can also be bundled together with the high resolution Pan and low resolution MS images or image streams.
  • the image compression and/ or video compression can be called on demand to compress the Pan and MS image respectively for storage or transmission purposes, and then called to decompress the Pan and MS image respectively when the final high resolution MS images or image streams are needed. After the decompression, the image fusion will be applied to produce the final high resolution MS image or image stream. If no compression is called to compress the Pan and MS images, the image fusion can be directly applied to produce the final high resolution MS image or image stream.
  • Systems and methods according to the invention can be implemented by software that runs on a camera imaging system or on a client device.
  • the invention also relates to a new dual image format which contains a high resolution Pan image and a low resolution MS image in one digital file, to facilitate image storage and transmission.
  • the Pan image and MS image in the dual image format can be raw images, images in TIFF format or in other formats without any data compression.
  • the Pan and MS images should be in the same image format (e.g. either raw, or TIF or another format) and then be stored in the dual image format
  • a new dual image format contains a compressed high resolution Pan image and a compressed low resolution MS image, to facilitate image storage and transmission.
  • the compressed Pan image and MS image can be in either JPEG, JPEG 2000, or other formats which can compress still images. The same format should be used store the compressed Pan and MS images.
  • a new dual image format can be developed mat contains a high resolution Pan image, a low resolution MS image, and an image fusion software to facilitate image storage, transmission and producing high resolution MS image when it is needed.
  • a new image format can be developed that contains a compressed high resolution Pan image, a compressed low resolution MS image, a decompression software and an image fusion software, for facilitating image storage and transmission and producing high resolution MS image when it is needed.
  • the decompression software is correlated to the compression image format that is used to store the compressed Pan and MS images, such as JPEG, JPEG 2000, or other formats.
  • a new dual image format can be developed that contains a high resolution Pan image, a low resolution MS image, a compression and decompression software and an image fusion software.
  • the compression and decompression software can compress the Pan and MS images in one of the compressed image formats, such as JPEG, JPEG 2000, or other formats, and then decompress the Pan and MS images. If the user does not call the compression function, the dual image format will contain un-compressed Pan and MS images. If the user calls the compression function, the dual image format will contain compressed Pan and MS images.
  • the Pan and MS images will not be decompressed and fused.
  • the fusion or the decompression and fusion can be called at the user end when the dual image format file is downloaded or displayed on a large display.
  • a new dual image format Writer and Reader software tool is provided to allow users to generate a dual image format by writing a Pan and a MS image into the dual image format; and allow users to generate a high resolution color image by reading the Pan and MS images from the dual image format and fusing them together.
  • the dual image Writer and Reader software can be downloaded from the Internet and installed on users' devices or displays.
  • a new dual video image format is provided here contains a high resolution Pan video stream and a low resolution MS video stream, both are synchronized to each other, for facilitating video image storage and transmission.
  • the Pan video and MS video in the dual video image format can be raw images, or in other formats without any data compression.
  • the Pan and MS videos should be in the same image format
  • a new dual video image format can be developed that contains a compressed high resolution Pan video stream and a compressed low resolution MS video stream, both are synchronized to each other, for facilitating image storage and transmission.
  • the compressed Pan and MS video streams can be in a compressed video image format, such as H.264 or MPEG4.
  • Each image frame in the videos can be compressed using a still image compression too, such JPEG, JPEG 2000, or other formats which can compress still images.
  • the same format should be used store the compressed Pan and MS video streams.
  • a new dual video image format can be developed that contains a high resolution Pan video image stream, a low resolution MS video image stream, and an image fusion software to facilitate video storage and transmission and producing high resolution MS video stream when it is needed.
  • a new dual video image format can be developed that contains a compressed high resolution Fan video stream, a compressed low resolution MS video stream, a decompression software and image fusion software, for facilitating video storage and transmission and producing high resolution MS video stream when it is needed.
  • the decompression software is correlated to the compression video format that is used to store the compressed Pan and MS video streams, such as H.264 or MPEG-4, or other formats.
  • a new dual video image format can be developed that contains a high resolution Pan video image stream, a low resolution MS video image stream, a compression and decompression software and an image fusion software.
  • the compression and decompression software can compress the Pan and MS video stream respectively, and then decompress the Pan and MS video streams. If the user does not call the compression function, the dual video format will contain un-compressed Pan and MS video streams. If the user calls the compression function, the dual video format will contain compressed Pan and MS video streams.
  • the Pan and MS video streams will not be decompressed and fused.
  • the fusion or the decompression and fusion can be called at the user end when the dual video format file is downloaded or displayed on a large display.
  • a new dual video format Writer and Reader software tool can also be developed to allow users to generate a dual video format by writing a Pan and a MS video streams into the dual video format; and allow users to generate a high resolution color video stream by reading the Pan and MS video streams from the dual video format and fusing them together.
  • the dual image Writer and Reader software can be downloaded from the Internet and installed on users' devices or displays.
  • a new color image compression method is provided to compress color images captured by a conventional color camera, through the steps of (1) simulating a black and white image from the high resolution color image or multispectral image (MS), (2) lowering the spatial resolution of the MS image bands to achieve the first level of color image compression, and (3) applying further image compression to the simulated high resolution Pan and the generated low resolution MS image bands to achieve a higher image compression ratio.
  • the present invention can be used with cloud computing applications where the direct sensor compression methods can be used to effectively reduce image data volume for storage, transmission and online computation.

Abstract

La présente invention concerne un système et un procédé pour (1) capturer directement, par un système de caméra, des trames d'images vidéo ou des images en couleurs compressées ou ayant un volume de données réduit qui sont comparées à celles provenant de systèmes de caméra en couleurs classiques ayant la même résolution d'image, et (2) récupérer des trames d'images vidéo ou des images en couleurs non compressées pour l'affichage, la visualisation ou pour d'autres traitements d'image/de vidéo du côté utilisateur final.
PCT/CA2016/050685 2015-06-18 2016-06-14 Systèmes et procédés d'imagerie de caméra WO2016201562A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US14/743,993 2015-06-18
US14/743,993 US9930316B2 (en) 2013-08-16 2015-06-18 Camera imaging systems and methods

Publications (1)

Publication Number Publication Date
WO2016201562A1 true WO2016201562A1 (fr) 2016-12-22

Family

ID=57544649

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CA2016/050685 WO2016201562A1 (fr) 2015-06-18 2016-06-14 Systèmes et procédés d'imagerie de caméra

Country Status (1)

Country Link
WO (1) WO2016201562A1 (fr)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6011875A (en) * 1998-04-29 2000-01-04 Eastman Kodak Company Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening
US20040184531A1 (en) * 2003-03-20 2004-09-23 Byeong-Jin Lim Dual video compression method for network camera and network digital video recorder
US20130335599A1 (en) * 2010-10-22 2013-12-19 University Of New Brunswick Camera Imaging Systems and Methods

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6011875A (en) * 1998-04-29 2000-01-04 Eastman Kodak Company Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening
US20040184531A1 (en) * 2003-03-20 2004-09-23 Byeong-Jin Lim Dual video compression method for network camera and network digital video recorder
US20130335599A1 (en) * 2010-10-22 2013-12-19 University Of New Brunswick Camera Imaging Systems and Methods

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
C.POHL & VAN GENDEREN: "Multisensor image fusion in remote sensing: Concepts, methods and applications", INTERNATIONAL JOURNAL OF REMOTE SENSING, vol. 19/5, 1998, pages 823 - 854, XP055138064 *
JOHN PRICE: "Combining Panchromatic and Multispectral Imagery from Dual Resolution Satellite instruments", REMOTE SENSING OF ENVIRONMENT, vol. 21, 1987, pages 119 - 128, XP002588220 *
LUCIEN WALD ET AL.: "Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images", PHOTOGRAMMETRIC ENGINEERING & REMOTE SENSING, vol. 63/6, June 1997 (1997-06-01), pages 691 - 699, XP055337573 *

Similar Documents

Publication Publication Date Title
US10904513B2 (en) Camera image fusion methods
US9936109B2 (en) Method and system for fusing images
US7855786B2 (en) Single camera multi-spectral imager
US11553165B2 (en) Color night vision cameras, systems, and methods thereof
KR101340688B1 (ko) 루미넌스 및 크로미넌스 센서들을 이용한 이미지 캡처
US10390005B2 (en) Generating images from light fields utilizing virtual viewpoints
EP3416369B1 (fr) Procédé et appareil de traitement d'image pour un terminal, et terminal
KR101824290B1 (ko) 고해상도 멀티스펙트럼 이미지 캡처 기법
US20170094141A1 (en) Infrared and visible light dual sensor imaging system
WO2014185064A1 (fr) Méthode et système de traitement d'image
US20070159640A1 (en) Shared color sensors for high-resolution 3-D camera
CN106662749A (zh) 用于全视差光场压缩的预处理器
JP2015521411A (ja) πフィルタ群を用いてパターン化されたカメラモジュール
JP6221682B2 (ja) 画像処理装置、撮像システム、画像処理方法およびプログラム
WO2015133130A1 (fr) Dispositif de capture vidéo, dispositif de séparation de signal et procédé de capture vidéo
JP2000222116A (ja) 表示画像の位置認識方法とその位置認識装置および仮想画像立体合成装置
WO2016201562A1 (fr) Systèmes et procédés d'imagerie de caméra
US9110293B2 (en) Prismatic image replication for obtaining color data from a monochrome detector array
GB2518393A (en) Camera
Chenault et al. Pyxis: enhanced thermal imaging with a division of focal plane polarimeter
Howard et al. Real-time color fusion of E/O sensors with PC-based COTS hardware
JP7114387B2 (ja) 撮像装置、撮像装置の制御方法およびプログラム
WO2021249955A1 (fr) Métadonnées pour décrire un filtre chromatique sur la partie supérieure des capteurs d'appareil de prise de vues

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16810666

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16810666

Country of ref document: EP

Kind code of ref document: A1