US20190349537A1

US20190349537A1 - Apparatus and method for combining images

Info

Publication number: US20190349537A1
Application number: US16/525,001
Authority: US
Inventors: Won-Hee Choe; Jae-Hyun Kwon; Seong-deok Lee
Original assignee: Samsung Electronics Co Ltd
Current assignee: Samsung Electronics Co Ltd
Priority date: 2008-05-19
Filing date: 2019-07-29
Publication date: 2019-11-14
Also published as: US20180013963A1; US8989487B2; US20150172568A1; US10368015B2; US20090285476A1; KR20090120159A; US9736403B2

Abstract

Provided are an image composition apparatus for composing color images with black-and-white images including infrared components, and an image composition method thereof. The image composition method includes generating a first image signal with color information and a second image signal including infrared components without color information, dividing the first image signal into a brightness signal and a color signal, composing the brightness signal of the first image signal with a brightness signal of the second image signal to generate a composed brightness signal, and composing the composed brightness signal with the color signal of the first image signal to generate a color image.

Description

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation application of prior application Ser. No. 15/676,235, filed on Aug. 14, 2017, which is a continuation application of prior application Ser. No. 14/627,498, filed on Feb. 20, 2015, which has issued as U.S. Pat. No. 9,736,403 on Aug. 15, 2017, which is a continuation application of prior application Ser. No. 12/467,321, filed on May 18, 2009, which has issued as U.S. Pat. No. 8,989,487 on Mar. 24, 2015 and claimed the benefit under 35 U.S.C. § 119(a) of Korean Patent Application number 10-2008-0046060, filed on May 19, 2008, the disclosure of which is incorporated by reference herein in its entirety.

BACKGROUND

Field

The following description relates to a technology of combining images, and more particularly, to an apparatus and method for combining color images with black-and-white images including infrared components.

Description of the Related Art

With popularization of digital cameras, interests in digital imaging devices have been growing. A digital imaging device may edit or store captured images as it digitalizes and processes various image information.
In general, a digital imaging device includes a lens, an image sensor, and an image processor. The lens adjusts a focus of light reflected from an object and transmits the light to the image sensor so that the light forms a proper image on the image sensor. The image sensor senses the light incident thereon and generates image signals, that is, electrical signals. The generated image signals are subjected to processing and may be displayed or stored.
Types of image sensors include image pickup tubes and solid image sensors. The solid image sensors include charge coupled devices (CCDs), complementary metal-oxide-semiconductors (CMOSs), and the like.
A CCD sensor includes a circuit in which a plurality of capacitors are connected in pairs. Also, a CCD chip including a plurality of photo diodes generates electrons according to an amount of light incident on each photodiode. Then, by reconfiguring information generated by the photodiodes, image information may be created.
CMOS image sensors may be manufactured at lower costs than CCD image sensors as the CMOS image sensors may be manufactured using a general-purpose semiconductor manufacturing equipment. Therefore, CMOS image sensors have been typically utilized in low-priced digital cameras or slow-frame television cameras. However, CMOS image sensors may be unstable or have poor performance in a low illumination environment, and images captured by a CMOS image sensor may have noises.
While a CMOS image sensor can convert infrared light as well as visible light into image signals, such infrared components are generally removed by an infrared blocking filter in order to easily restore color signals. However, in order to acquire images over wider bands, it is desirable to use the infrared components.

SUMMARY

According to one general aspect, there is provided an image composition apparatus including an image acquiring unit to sense incident light and generate a first image signal with color information and a second image signal including infrared components without color information, an image signal divider to divide the first image signal into a brightness signal and a color signal, a brightness composer to compose the brightness signal of the first image signal with a brightness signal of the second image signal to generate a composed brightness signal, and an image restoring unit to compose the composed brightness signal with the color signal of the first image signal, so as to generate a color image.
The first image signal may include a signal corresponding to a specific region of a visible band of an optical spectrum, and the second image signal may include a signal corresponding to an infrared band of the optical spectrum and a combination of signals corresponding to specific regions of the visible band.
The first image signal may be a color image signal, and the second image signal may be a black-and-white image signal including infrared components.
The apparatus may further include a color space converter to convert a color space of the first image signal.
The apparatus may further include a dynamic bandwidth adjusting unit to equalize dynamic bandwidths of the first image signal and the second image signal.
The apparatus may further include a resolution adjusting unit to equalize resolutions of the first image signal and the second image signal.
The apparatus may further include a domain transformer to transform spatial domains of the first image signal and the second image signal into frequency domains.
According to another aspect, there is provided an image composition method in an image composition apparatus, the method including generating a first image signal with color information and a second image signal including infrared components without color information, dividing the first image signal into a brightness signal and a color signal, composing the brightness signal of the first image signal with a brightness signal of the second image signal to generate a composed brightness signal, and composing the composed brightness signal with the color signal of the first image signal to generate a color image.
The method may further include converting a color space of the first image signal prior to the dividing of the first image signal into the brightness signal and the color signal.
The method may further include equalizing dynamic bandwidths of the first image signal and the second image signal. The equalizing of the dynamic bandwidths of the first and second image signals may comprise compressing the second image signal to match with the dynamic bandwidth of the first image signal.
The method may further include matching a resolution of the first image signal with a resolution of the second image signal. The matching of the resolution may comprise interpolating an image signal with the lower resolution among the first and second image signals with respect to an image signal with the higher resolution among the first and second image signals.
Other features and aspects will be apparent from the following detailed description, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an exemplary image composition apparatus.

FIG. 2 is a diagram illustrating an optical spectrum.

FIG. 3 is a diagram illustrating a configuration of an exemplary image acquiring unit.

FIG. 4 is a block diagram illustrating another exemplary image composition apparatus.

FIGS. 5 through 8 are block diagrams illustrating still another exemplary image composition apparatuses.

FIG. 9 is a flowchart of an exemplary image composition method.

FIG. 10 is a flowchart of another exemplary image composition method.

Throughout the drawings and the detailed description, unless otherwise described, the same drawing reference numerals will be understood to refer to the same elements, features, and structures. The relative size and depiction of these elements may be exaggerated for clarity, illustration, and convenience.

DETAILED DESCRIPTION

The following detailed description is provided to assist the reader in gaining a comprehensive understanding of the methods, apparatuses, and/or systems described herein. Accordingly, various changes, modifications, and equivalents of the systems, apparatuses and/or methods described herein will be suggested to those of ordinary skill in the art. Also, descriptions of well-known functions and constructions may be omitted for increased clarity and conciseness.
FIG. 1 shows an exemplary image composition apparatus.
The image composition apparatus may be applied to an imaging device to detect light reflected from an object and create or store an image. For example, the image composition apparatus may be applied to a digital camera, a hardware system to drive a digital camera, an image processing chip, and the like. Referring to FIG. 1, the image composition apparatus includes an image acquiring unit 101, a color space converter 104, an image signal divider 105, a brightness composer 108, and an image restoring unit 109.
The image acquiring unit 101 may be a CCD or CMOS image sensor to sense light reflected from an object and converting the sensed light into electrical signals. The image acquiring unit 101 senses light incident thereon to generate a predetermined image signal, wherein the image signal may be divided into a first image signal 102 and a second image signal 103.
The first image signal 102 includes color information, and the second image signal 103 includes infrared components without any color information. The first and second image signals 102 and 103 will be described further with reference to an optical spectrum illustrated in FIG. 2.
Referring to FIG. 2, light may be classified into a visible band 111, an infrared band 112 and an ultraviolet band 113 according to its wavelength. The first and second image signals 102 and 103 may mean signals belonging to specific wavelength bands of an optical spectrum. For example, the first image signal 102 includes image signals corresponding to light belonging to specific regions 114 of the visible band of the optical spectrum, and the second image signal 103 includes combinations (for example, 300) of signals belonging to the specific regions 114 of the visible band 111, and image signals corresponding to light belonging to the infrared band 112 of the optical spectrum. In other words, the first image signal 102 is a color image signal with color information, and the second image signal 103 is a black-and-white image signal including infrared components without color information.
Image signals, such as the first and second image signals 102 and 103, having information of different wavelength bands, may be obtained by using, for example, a multi-sensor technology utilizing different optical systems and image sensors or by controlling the filtering function of a color filter array (CFA) without having to change an optical system.
FIG. 3 illustrates a configuration of an exemplary image acquiring unit, wherein the image acquiring unit may be used to generate the first and second image signals 102 and 103 described above.
Referring to FIG. 3, the image acquiring unit includes an image sensor 116, and a filter array 115 disposed on the image sensor 116 and coupled with the image sensor 116. The image sensor 116 senses light incident thereon and converts the light into electrical signals, and the filter array 115 filters light that is to be incident on the image sensor 116 so that light belonging to specific wavelength bands is only incident to the image sensor 116.
The filter array 115 includes color filters 117 to selectively transmit light belonging to specific regions (for example, the regions 114 of FIG. 2) of the visible band of an optical spectrum, and a transparent filter 118 to transmit light over all bands of the optical spectrum. The filter array 115 has no infrared blocking function.
Accordingly, where light reflected from an object passes through the filter array 115, the color filter units 117 transmit light belonging to specific bands of the visible band and infrared light therethrough, and the transparent filter 118 transmits light (including infrared light) over all bands therethrough.
The image sensor 116 disposed below the filter array 115 may have a multi-layer structure with stacked sensor modules. For example, the upper layer 130 of the image sensor 116 includes first light receivers 119 to sense light belonging to the visible band from the light which has passed through the color filters 117, and a second light receiver 120 to sense white light which has passed through the transparent filter 118, and the lower layer 131 of the image sensor 116 includes a third light receiver 121 to sense light belonging to the infrared band.
The image sensor 116 may be formed by a semiconductor manufacturing process, and each light receiver may be a photodiode made of silicon. Here, since infrared light with wavelengths longer than those of visible light is absorbed at a relatively deeper location (that is, the lower layer 131), the multi-layer structure is provided in which the upper layer 130 detects light of the visible band and the lower layer 131 detects light of the infrared band.
In the image acquiring unit, the first light receiver 119 detects light with color information, the second light receiver 120 detects light (for example, white light) without color information, and the third light receiver 121 detects infrared light, respectively. Accordingly, the output signal of the first light receiver 119 is used as the first image signal 102, and the output signals of the second and third light receivers 120 and 121 are used as the second image signal 103.
Referring back to FIG. 1, the color space converter 104 converts a color space of the first image signal 102. As described above, since the first image signal 102 has color information, the first image signal 102 may be represented as an RGB signal in a color space. For example, the color space converter 104 may convert the first image signal 102 represented as an RGB signal into a YCbCr signal, using a color space conversion function. This is only one example and the color spacer converter 104 may convert the RGB signal into another signal, such as HSV, HIS, LUV, and the like.
The first image signal 102 whose color space has been converted by the color space converter 104 is input to the image signal divider 105 (see FIG. 1), and the image signal divider 105 divides the first image signal 102 into a brightness signal and a color signal 107.
Here, the brightness signal 106 corresponds to brightness information for the first image signal 102, and the color signal 107 corresponds to color information for the first image signal 102. For example, if the first image signal 102 represented as an RGB signal is converted into a YCbCr signal, the brightness signal 106 corresponds to the Y signal and the color signal 107 corresponds to the CbCr signal. Also, in a HSV space, V information may be used as the brightness signal 106, and in a HIS space, I information may be used as the brightness signal 106.
The brightness composer 108 composes the brightness signal 106 divided from the first image signal 102 with the brightness signal of the second image signal 103. Since the second image signal 103 has only brightness information without any color information, the second image signal 103 may be composed with the brightness signal 106 divided from the first image signal 102 without any additional processing. For example, if the second image signal 103 is represented in the YCbCr color space, the Y information of the second image signal 103 is composed with the Y information of the first image signal 102 output from the image signal divider 105.
A brightness composition method may be used in which brightness signals are composed using, for example, a lookup table or a conversion function representing the relationship between the brightness information of the first image signal 102 and the brightness information of the second image signal 103. As another example, a brightness composition method may be used which adds the coefficients of brightness signals using Discrete Wavelet Transform (DWT).
The image restoring unit 109 combines the composed brightness signal 110 generated by the brightness composer 108 with the color signal 107 of the first image signal 102 divided by the image signal divider 105, so as to generate a color image. Combining the composed brightness signal 110 with the color signal 107 may be done by the inverse processing of the division processing by the image signal divider 105, and the final color image may be obtained by the inverse processing of the conversion processing by the color space converter 104.
Accordingly, the color image provided by the image composition apparatus is an image created using both visible light signals and black-and-white image signals including infrared components. That is, by using signals over a wide band, recognizable image information may be obtained even in a low illumination environment.
FIG. 4 shows another exemplary image composition apparatus. Here, the image composition apparatus adjusts the dynamic bandwidths of images compared to the image composition apparatus of FIG. 1.
Referring to FIG. 4, the image composition apparatus includes a dynamic bandwidth adjusting unit 122. The second image signal 103 which corresponds to a black-and-white image may include image information with a bandwidth wider than that of the first image signal 102 which corresponds to a color image. In this case, due to the second image signal 103, an unnatural image may be created upon image composition.
The dynamic bandwidth adjusting unit 122, which equalizes the dynamic bandwidths of the first and second images signals 102 and 103, adjusts the dynamic bandwidth of the second image signal 103 to be suitable for an image output apparatus, or compresses the dynamic bandwidth of the second image signal 103 to be equalized to the dynamic bandwidth of the first image signal 102.
FIGS. 5 through 7 show still another exemplary image composition apparatuses, wherein the image composition apparatuses adjust resolution.
Referring to FIGS. 5 through 7, the image composition apparatuses include a resolution adjusting unit 123.
For example, the resolution adjusting unit 123 interpolates an image signal with lower resolution to equalize the resolutions of image signals. This process may be performed after brightness/color division (FIG. 5) or before brightness/color division (FIG. 6). Where the first image signal 102 which corresponds to a color image has a resolution lower than that of the second image signal 103, the resolution adjusting unit 123 equalizes the resolution of the first image signal 102 to the resolution of the second image signal 103. Where the first image signal 102 which corresponds to a color image has a resolution higher than that of the second image signal 103 which corresponds to a black-and-white image, the resolutions of the first and second image signals 102 and 103 may be equalized by interpolating the second image signal 103 with the lower resolution.
FIG. 8 shows still another exemplary image composition apparatus.
Referring to FIG. 8, the image composition apparatus includes a domain transformer 201 and a domain inverse-transformer 202.
The domain transformer 201 transforms the spatial domains of the first and second image signals 102 and 103. For example, the domain transformer 201 transforms the spatial domains of the first and second image signals 102 and 103 into frequency domains. The domain inverse-transformer 202 inverse-transforms the frequency domains into the original spatial domains before generating a final color image.
FIG. 9 shows a flowchart of an exemplary image composition method. The image composition method may be performed by an image composition apparatus described above.
Referring to FIG. 9, a first image signal with color information and a second image signal including infrared components without color information are acquired by sensing incident light in operation S201. Here, the first image signal may be a color image signal, that is, a signal which corresponds to specific regions of the visible band of an optical spectrum, and the second image signal may be a black-and-white image signal including infrared components, that is, a signal which corresponds to a combination of all or specific regions of the visible band, or a signal which corresponds to the infrared band. The first or second image signal may be generated by, for example, the image acquiring unit of FIG. 3.
The color space of the first image signal is converted in operation S202. For example, the first image signal represented as an RGB signal is converted into a YCrCb signal.
In operation S203, the first image signal whose color space has been converted is divided into a brightness signal and a color signal. For example, the brightness signal corresponds to brightness information represented as a Y signal and the color signal corresponds to color information represented as a CbCr signal.
Where the first image signal is divided into the brightness signal and color signal, the brightness signal of the first image signal is composed with the brightness signal of the second image signal to generate a composed brightness signal in operation S204. The brightness composition may be carried out by adding the coefficients of brightness signals using DWT to obtain a composed brightness signal.
In operation S205, the composed brightness signal is composed with the color signal divided in the operation S203, so as to generate a color image. Here, the color image may be obtained by the inverse processing of the division processing performed in the operation S203.
FIG. 10 shows a flowchart of another exemplary image composition method, wherein the image composition method includes an operation of adjusting the resolutions and dynamic bandwidths of images.
Referring to FIG. 10, first and second image signals are acquired in operation S301, and equalizing the resolutions of the first and second image signals is performed in operation S302. The operation of adjusting the resolutions of the first and second image signals may be carried out by interpolating an image signal with the lower resolution among the first and second image signals to equalize the resolutions of the first and second image signals.
In operation S303, the first image signal is divided into a brightness signal and a color signal after adjusting the resolutions of the first and second image signals.
In operation S304, the dynamic bandwidths of the first and second image signals are equalized. The operation of equalizing the bandwidths of the first and second image signals may be carried out by compressing the second image signal with the wider dynamic bandwidth to be matched with the bandwidth of the first image signal or by adaptively matching the dynamic bandwidth of the first image signal with the dynamic bandwidth of the second image signal.
In operation S306, the brightness signal of the first image signal is composed with the brightness signal of the second image signal to generate a composed brightness signal. The composed brightness signal is composed with the color signal of the first image signal to generate a color image in operation S306.
According to examples described above, color images are combined with black-and-white images including infrared components.
According to examples described above, since black-and-white image signals having the wider band as well as color image signals are additionally used to generate images, high-sensitivity images may be obtained in a low illumination environment.
According to examples described above, since infrared signals having no color information may be represented as black-and-white images, appropriate composition of black-and-white images with color images may obtain images with high-sensitivity and wide-bandwidth.
The methods described above may be recorded, stored, or fixed in one or more computer-readable media that includes program instructions to be implemented by a computer to cause a processor to execute or perform the program instructions. The media may also include, alone or in combination with the program instructions, data files, data structures, and the like. Examples of computer-readable media include magnetic media, such as hard disks, floppy disks, and magnetic tape; optical media such as CD ROM disks and DVDs; magneto-optical media, such as optical disks; and hardware devices that are specially configured to store and perform program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory, and the like. Examples of program instructions include machine code, such as produced by a compiler, and files containing higher level code that may be executed by the computer using an interpreter. The described hardware devices may be configured to act as one or more software modules in order to perform the operations and methods described above, or vice versa.
A number of exemplary embodiments have been described above. Nevertheless, it will be understood that various modifications may be made. For example, suitable results may be achieved if the described techniques are performed in a different order and/or if components in a described system, architecture, device, or circuit are combined in a different manner and/or replaced or supplemented by other components or their equivalents. Accordingly, other implementations are within the scope of the following claims.

Claims

What is claimed is:

1. A portable communication device comprising:

a color image sensor to generate a first image signal having a color image signal;

a monochrome image sensor to generate a second image signal having a monochrome image signal; and

one or more processor adapted to:

obtain the first image signal and the second image signal respectively from the color image sensor and the monochrome image sensor, wherein the first image signal and the second image signal are generated substantially at a same time,

generate an image by composing a bright component of the first image signal and the second image signal and adding a color component of the first image signal.