KR20160137289A - Photographing apparatus and method for controlling the same - Google Patents

Abstract

According to an embodiment of the present invention, a filming apparatus includes: first and second camera modules filming the same subject; and a control unit which maps a second image obtained from the second camera module onto a first image obtained from the first camera module and uses the second image mapped onto the first image to synthesize a third image of the subject. Image sensors included in the first camera module have a first pixel structure with rectangular shapes. Image sensors included in the second camera module have a second pixel structure different from the first pixel structure.

Description

TECHNICAL FIELD [0001] The present invention relates to a photographing apparatus and a method of operating the photographing apparatus,

BACKGROUND OF THE INVENTION Field of the Invention [0002] The present invention relates to an image capturing apparatus and an operation method thereof, and more particularly, to an image capturing apparatus and an operation method thereof that improve image resolution using images obtained from a plurality of modules.

The multiple camera system includes two or more camera modules, and can detect a focus on a specific object or generate a stereoscopic image using the images input to each of the camera modules. Recently, a multi-camera system has been used to acquire a plurality of image information to improve the resolution of an image.

Various embodiments provide a photographing apparatus and an operation method thereof that improve image resolution using respective images obtained from a first camera module and a second camera module having different pixel shapes of the image sensor.

As a technical means for achieving the above-mentioned technical object, a first aspect of the present disclosure is a camera module comprising: a first camera module and a second camera module for photographing the same object; A second image obtained from the second camera module is mapped to a first image acquired from the first camera module, and a third image of the subject is synthesized using the first image and the second image mapped Wherein the image sensor included in the first camera module has a first pixel structure in which the pixel shape is a quadrangle and the image sensor included in the second camera module has a second pixel structure different from the first pixel structure, Structure can be provided.

One of the first camera module and the second camera module according to an embodiment is a color camera module and the other is a mono camera module.

The first and second regions among the plurality of pixel structures included in the second pixel structure according to an exemplary embodiment may have at least one of a size and a shape different from each other.

The controller may be configured to determine whether the image obtained from the color camera module based on the parallax information of the first image and the mapped second image is an image obtained from the mono camera module It is possible to provide a photographing apparatus which determines whether to make an acquired image.

The controller determines whether the parallax information is greater than a threshold value. If the parallax information is greater than the threshold value, the controller determines the image to be a reference for the image synthesis as an image acquired from the color camera module And determining an image obtained as a reference in the image synthesis as an image acquired from the mono camera module if the parallax information is not greater than the threshold value.

The parallax information may include at least one of disparity and occlusion information of the first image and the mapped second image.

According to a second aspect of the present disclosure, there is provided an image processing method comprising the steps of: obtaining a first image of a subject photographed from a first camera module and a second image of the subject photographed from a second camera module; Mapping the acquired second image to the acquired first image; And synthesizing a third image of the subject using the first image and the mapped second image, wherein the image sensor included in the first camera module includes a first pixel having a pixel shape of a rectangle, And the image sensor included in the second camera module has a second pixel structure different from the first pixel structure.

One of the first camera module and the second camera module according to an exemplary embodiment may be a color camera module and the other may be a mono camera module.

The first and second regions of the plurality of pixel structures included in the second pixel structure may have at least one of a size and a shape different from each other.

The step of synthesizing the third image of the subject according to an exemplary embodiment may further include a step of acquiring an image serving as a reference upon image synthesis based on the parallax information of the first image and the mapped second image from the color camera module And determining whether to use the image obtained from the mono camera module or the image obtained from the mono camera module.

The step of synthesizing the third image of the subject according to an exemplary embodiment may include determining whether the parallax information is greater than a threshold value and if the parallax information is greater than the threshold value, Determining whether the parallax information is greater than the threshold value, and determining an image obtained as a reference in the image synthesis as an image acquired from the mono camera module if the parallax information is not greater than the threshold value.

The parallax information may include at least one of disparity and occlusion information of the first image and the mapped second image.

The third aspect of the present disclosure can also provide a computer-readable recording medium on which a program for causing a computer to execute the method of the second aspect is recorded.

1 is a diagram illustrating a multiple camera system including a plurality of camera modules.
2 is a block diagram showing the configuration of a photographing apparatus according to an embodiment.
3 is a block diagram showing the configuration of the first camera module of Fig.
4 is a block diagram showing a configuration of a photographing apparatus according to another embodiment.
5 is a diagram referred to explain a first pixel structure according to an embodiment.
FIG. 6 is a diagram referred to explain a second pixel structure according to an embodiment, and FIG. 7 is a diagram referred to explain a second pixel structure according to another embodiment.
8 is a diagram for explaining a method of obtaining a second image mapped by the image mapping unit according to an embodiment.
FIG. 9 is a diagram for explaining a method of composing a third image according to an embodiment.
FIG. 10 and FIG. 11 are flowcharts illustrating an operation method of an image combining unit according to an exemplary embodiment.
FIG. 12 is a diagram for explaining a method of using disparity to obtain parallax information in an image synthesizing unit according to an embodiment. FIG. 13 is a flowchart illustrating a method of obtaining parallax information in an image synthesizing unit according to another embodiment. Lt; RTI ID = 0.0 > occlusion < / RTI >
FIG. 14 is a diagram for explaining a method of composing a third image based on a mono image according to an embodiment, and FIG. 15 illustrates a method of synthesizing a third image based on a color image according to another embodiment. Reference is made to the drawings.
16 is a flowchart showing an operation method of the photographing apparatus according to an embodiment.

The terms used in this specification will be briefly described and the present invention will be described in detail.

While the present invention has been described in connection with what is presently considered to be the most practical and preferred embodiment, it is to be understood that the invention is not limited to the disclosed embodiments. Also, in certain cases, there may be a term selected arbitrarily by the applicant, in which case the meaning thereof will be described in detail in the description of the corresponding invention. Therefore, the term used in the present invention should be defined based on the meaning of the term, not on the name of a simple term, but on the entire contents of the present invention.

When an element is referred to as "including" an element throughout the specification, it is to be understood that the element may include other elements as well, without departing from the spirit or scope of the present invention. Also, the terms "part," " module, "and the like described in the specification mean units for processing at least one function or operation, which may be implemented in hardware or software or a combination of hardware and software .

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

≪ 1 >

1 is a diagram illustrating a multiple camera system including a plurality of camera modules.

The multi-camera system may be a stereoscopic camera that generates stereoscopic images. The multiple camera system may include a first camera module 21 and a second camera module 22. At this time, the first camera module 21 and the second camera module 22 may be spaced apart from each other by a predetermined distance. The multi-camera system can acquire the third image using the first image acquired from the first camera module 21 and the second image acquired from the second camera module 22. [

2 and 3,

2 is a block diagram showing the configuration of a photographing apparatus according to an embodiment.

Referring to FIG. 2, the photographing apparatus 100 may include a first camera module 110, a second camera module 120, and a controller 130. The imaging apparatus 100 according to one embodiment may be implemented with a multiple camera system including a plurality of camera modules, as described with reference to FIG. In addition, the photographing apparatus 100 may be implemented in various forms such as a digital still camera for photographing a still image or a digital video camera for photographing a moving image. Further, the photographing apparatus 100 may include a digital single-lens reflex camera (DSLR), a mirrorless camera, or a smart phone. However, the present invention is not limited thereto, and the photographing apparatus 100 according to one embodiment may include an apparatus including a lens and an image pickup device, and mounting a plurality of camera modules capable of photographing a subject and generating an image. In addition, the controller 130 may include an image mapping unit 140 and an image synthesizer 150 according to an exemplary embodiment of the present invention.

The image sensor included in the first camera module 110 and the image sensor included in the second camera module 120 according to one embodiment may have different pixel structures. For example, the image sensor included in the first camera module 110 has a first pixel structure having a pixel shape of a quadrangle, and the image sensor included in the second camera module 120 has a pixel structure different from the first pixel structure. 2 pixel structure. The pixel structure refers to the shape of the pixels constituting the image sensor of the camera module, and the pixels having different shapes may show a difference in the amount of light to be received at the time of photographing. For example, a pixel having a large pixel size can obtain a high-sensitivity or bright image in comparison with a pixel having a small pixel size under the same condition, and a pixel having a small pixel size can be reduced in sensitivity A dark image can be obtained. Thus, pixel sizes can be used to obtain HDR (High Dynamic Range) images using different pixel structures. In the case of a color camera module, the degree of color reproduction may vary depending on the difference in pixel type.

The first pixel structure according to an embodiment may have a general pixel structure having a pixel structure of a rectangular or square shape and the second pixel structure may have a pixel structure having a polygonal structure having a shape different from the first pixel structure.

In addition, the resolution of the image sensor included in the first camera module 110 and the resolution of the image sensor included in the second camera module 120 may be different from each other. The resolution of the image sensor is determined by the number of pixels included in the image sensor, and the higher the number of pixels included in the image sensor, the higher the resolution of the image sensor. For example, the resolution of the image sensor included in the first camera module 110 may be lower than the resolution of the image sensor included in the second camera module 120, The resolution of the sensor may be higher than the resolution of the image sensor included in the first camera module 110.

The first camera module according to an exemplary embodiment may be a color camera module, and the second camera module may be a mono camera module. The first camera module according to another embodiment may be a mono camera module, and the second camera module may be a color camera module. For the same image sensor, the color image has an effect of obtaining color information in comparison with the mono image, and the mono image absorbs more light amount compared with the color image, so that the improved detail information can be obtained. Therefore, when the color camera module and the mono camera module are used simultaneously, the effects on the color image and the mono image can be obtained at the same time.

The control unit 130 can control the entire operation of the photographing apparatus 100. The control unit 130 provides control signals for operation of each component to each component included in the photographing apparatus 100. [

The control unit 130 processes the input video signal and can control the respective components according to the input video signal or the external input signal. The controller 130 may correspond to one or a plurality of processors. A processor may be implemented as an array of a plurality of logic gates, or may be implemented as a combination of a general purpose microprocessor and a memory in which a program executable in the microprocessor is stored. It will be appreciated by those skilled in the art that the present invention may be implemented in other forms of hardware.

The control unit 130 executes a stored program or generates a control signal for controlling autofocusing, zooming, focus change, automatic exposure correction, and the like by using a separate module, The lens driving unit, and the image sensor control unit included in the camera module 120 to control the operation of components included in the photographing apparatus 100 such as a shutter and a strobe.

The control unit 130 is connected to an external monitor to process a video signal inputted from the image signal processing unit included in the first camera module 110 or the second camera module 120, And transmits the processed image data to display the corresponding image on the external monitor.

Meanwhile, the image mapping unit 140 may map a second image acquired from the second camera module 120 to a first image acquired from the first camera module 110. For example, the image mapping unit 140 may map each pixel of the first image using pixels adjacent to the pixel position corresponding to the second image.

Meanwhile, the image combining unit 150 may combine the third image of the subject using the parallax information between the first image and the second image mapped. The image synthesizer 150 may acquire an image based on the parallax information of the second image mapped with the first image from the color camera module, You can decide if you want to make one image.

The image synthesizer 150 determines whether the parallax information is greater than the threshold value. If the parallax information is greater than the threshold value, the image synthesizer 150 determines the image as a reference image in the image synthesis as an image acquired from the color camera module , And if the parallax information is not greater than the threshold value, the reference image can be determined as an image obtained from the mono camera module.

In addition, the image combining unit 150 may synthesize a third image of a subject using at least one of disparity and occlusion information of the second image mapped to the first image.

According to one embodiment, a third image can be synthesized based on a color image or a mono image reference. In a case where a third image is synthesized based on a color image reference, since color information can be maintained as it is, color reproducibility And can also obtain improved detail information of the mono image. On the other hand, in the case of synthesizing the third image based on the mono image, since the detail information of the mono image can be maintained as it is, the resolution and SNR (Signal to Noise Ratio) of the mono image are ensured and the color information of the color image is used There is an effect that can be.

3 is a block diagram showing the configuration of the first camera module of Fig.

3, the first camera module 110 includes a lens 111, a lens driver 112, a diaphragm 113, a diaphragm driver 115, an image sensor 116, an image sensor controller 117, A signal processing unit 118, and an image signal processing unit 119.

The lens 111 may include a plurality of lenses and a plurality of lenses. The position of the lens 111 can be adjusted by the lens driving unit 112. [ The lens driving unit 112 adjusts the position of the lens 111 according to a control signal provided from the control unit 130. The lens driving unit 112 adjusts the position of the lens 111 to adjust the focal length, and performs operations of auto focusing, zoom change, and focus change.

The diaphragm 113 is controlled by the diaphragm driving unit 115 to adjust the amount of light incident on the image sensor 116.

The optical signal transmitted through the lens 111 and the diaphragm 113 arrives at the light receiving surface of the image sensor 116 and forms an image of the subject. The image sensor 116 may be a CCD (Charge Coupled Device) image sensor or a CIS (Complementary Metal Oxide Semiconductor Image Sensor) for converting an optical signal into an electric signal. The sensitivity of the image sensor 116 may be adjusted by the image sensor control unit 117. FIG. The image sensor control unit 117 can control the image sensor 116 according to a control signal automatically generated by a video signal input in real time or a control signal manually input by a user's operation.

The analog signal processing unit 118 performs noise reduction processing, gain adjustment, waveform shaping, analog-to-digital conversion processing, and the like on the analog signal supplied from the image sensor 116.

The image signal processing unit 119 is a signal processing unit for processing a special function for the image data signal processed by the analog signal processing unit 118. For example, it is possible to reduce noise to input image data, and to reduce noise, such as gamma correction, color filter array interpolation, color matrix, color correction, color enhancement color enhancement) It is possible to perform image signal processing for improving image quality such as white balance adjustment, smoothing of brightness and color shading, and for providing special effects. The image signal processing unit 119 may compress the input image data to generate an image file, or may restore the image data from the image file. The compression format of the image may be reversible or irreversible. As an example of a proper format, the still image can be converted into a JPEG (Joint Photographic Experts Group) format or a JPEG 2000 format. In addition, when recording a moving image, a moving image file can be generated by compressing a plurality of frames according to the MPEG (Moving Picture Experts Group) standard. The image file can be created according to the Exif (Exchangeable image file format) standard, for example.

The image signal processing unit 119 can generate a moving image file from the imaging signal generated by the image sensor 116. [ The image pickup signal may be a signal generated by the image sensor 116 and processed by the analog signal processing unit 118. The image signal processing unit 119 generates frames to be included in the moving picture file from the image pickup signal and outputs the frames to a standard such as Moving Picture Experts Group 4 (MPEG4), H.264 / AVC, WMV And the moving image file is compressed using the compressed moving image file. Video files can be created in various formats such as mpg, mp4, 3gpp, avi, asf, and mov. The image signal processing unit 119 may output the generated first image to the control unit 130.

In addition, the image signal processing unit 119 can perform blurring processing, color processing, blur processing, edge emphasis processing, image analysis processing, image recognition processing, image effect processing, and the like on the input image data. The face recognizing process, the scene recognizing process, and the like can be performed by the image recognizing process. The image signal processing unit 119 can perform image signal processing for displaying image data on a display unit. For example, brightness level adjustment, color correction, contrast adjustment, contour enhancement adjustment, screen division processing, character image generation, and image synthesis processing can be performed.

The second camera module 120 may include a lens, a lens driver, a diaphragm, a diaphragm driver, an image sensor, an image sensor controller, an analog signal processor, and an image signal processor, in the same manner as the first camera module 110. Since the components are already described in the first camera module 110, they will not be described in duplicate.

<Fig. 4>

4 is a block diagram showing the configuration of the image pickup apparatus 200 according to another embodiment.

4, the photographing apparatus 200 includes a first camera module 210, a second camera module 220, a controller 230, a memory 240, a storage / read controller 250, a memory card 242 A program storage unit 260, a display unit 270, a display driving unit 272, an operation unit 280, and a communication unit 290.

The first camera module 210, the second camera module 220 and the controller 230 of FIG. 4 are connected to the first camera module 110, the second camera module 120, and the controller 130, respectively, The same description will be omitted and the remaining components will be described.

Referring to FIG. 4, the storage / read controller 250 may store image data output from the first camera module 210 or the second camera module 220 in the memory card 242. At this time, the storage / read control unit 250 may store the image data according to a signal from the user or automatically. The storage / read control unit 250 reads data relating to an image from the image file stored in the memory card 242 and inputs the read data to the display driver 272 through the memory 240 or another path, 270 to display an image.

The memory card 242 may be detachable or permanently attached to the imaging device 200. For example, the memory card 242 may be a flash memory card such as an SD (Secure Digital) card.

The image signal processed by the first camera module 210 or the second camera module 220 may be input to the control unit 230 via the memory 240 or may be input to the control unit 230 without passing through the memory 240. [ As shown in FIG. The memory 240 operates as a main memory of the photographing apparatus 200 and can temporarily store information necessary for the photographing apparatus 200 to operate. The program storage unit 260 may store programs such as an operating system and an application system for driving the photographing apparatus 200. [

The photographing apparatus 200 may include a display unit 270 to display an operation state of the photographing apparatus 200 or image information photographed by the photographing apparatus 200. [ The first camera module 210 and the second camera module 220 may perform display image signal processing to display photographed image information on the display unit 270. For example, the first camera module 210 and the second camera module 220 generate luminance level adjustment, color correction, contrast adjustment, outline enhancement adjustment, screen division processing, character image generation, Synthesis processing, and the like.

The display unit 270 can provide visual information to the user. In order to provide visual information, the display unit 270 may include, for example, a liquid crystal display panel (LCD), an organic light emitting display panel, or the like. Also, the display unit 270 may be a touch screen capable of recognizing a touch input.

The display driver 272 provides a driving signal to the display unit 270.

The operation unit 280 is a place where the user can input the control signal. The operation unit 280 includes a shutter-release button for inputting a shutter-release signal for exposing the image sensor to light for a predetermined period of time to take a picture, a power button for inputting a control signal for controlling on / off of the power source, A zoom button, a mode selection button, and other shooting setting value adjustment buttons for widening the angle of view according to the zoom angle, or narrowing the angle of view. The operation unit 280 may be implemented in any form in which a user can input a control signal, such as a button, a keyboard, a touch pad, a touch screen, a remote controller, or the like.

The communication unit 290 may include a network interface card (NIC), a modem, or the like, and may function to allow the photographing apparatus 200 to communicate with an external device via a network in a wired / wireless manner have.

Meanwhile, the block diagrams of the photographing apparatuses 100 and 200 shown in Figs. 2, 3 and 4 are block diagrams for an embodiment. Each component of the block diagram can be integrated, added, or omitted according to the specifications of the photographing apparatuses 100 and 200 actually implemented. That is, two or more constituent elements may be combined into one constituent element, or one constituent element may be constituted by two or more constituent elements, if necessary. In addition, the functions performed in each block are intended to illustrate embodiments, and the specific operation or apparatus does not limit the scope of the present invention.

5, 6 and 7,

FIG. 5 is a diagram referred to explain a first pixel structure according to an embodiment, FIG. 6 is a diagram referred to explain a second pixel structure according to an embodiment, and FIG. Lt; RTI ID = 0.0 &gt; a &lt; / RTI &gt; second pixel structure.

The first pixel structure according to an embodiment may represent a rectangular or square pixel structure of an image sensor of a general camera module and the second pixel structure may represent a pixel structure having a polygonal structure different from the first pixel structure .

5 is a diagram referred to explain a first pixel structure according to an embodiment. Referring to FIG. 5, all the pixels 81 and 82 constituting the first pixel structure shown in FIG. 5 are all rectangular in shape and have the same size.

6 is a diagram referred to explain a second pixel structure according to an embodiment. 6 (a) is a view showing the second pixel structure of the image sensor obtained by rotating the image sensor of the general camera module of FIG. 5 by 45 degrees, and FIG. 6 (b) is a view showing an image sensor of HoneyComb type Lt; / RTI &gt; illustrates a second pixel structure. 6A and 6B are different from the first pixel structure of FIG. 5, which is an embodiment of an image sensor of a general camera module, respectively. However, at least two arbitrary pixels 31 and 32 of FIG. 6 (a) and at least two arbitrary pixels 36 and 37 of FIG. 6 (b) are the same in size and shape.

7 is a diagram referred to explain a second pixel structure according to another embodiment. Referring to FIG. 7, the image sensor of the camera module may have a second pixel structure in which at least two arbitrary pixels have different sizes and shapes. For example, the second pixel structure of Fig. 7 (a) has a hexagonal pixel 41 and a triangular pixel 42, and the second pixel structure of Fig. 7 (b) 7 (c) has a pixel 51 of an octagonal shape and a pixel 52 of a rectangular shape or a 45-degree rotated shape, and the pixel structure of FIG. 7 (d) Has a pixel 56 in the shape of a dodecagon and a pixel 57 in the form of a triangle.

8,

FIG. 8 is a diagram for explaining a method of obtaining a second image mapped by the image mapping unit 140 according to an embodiment.

The image mapping unit 140 receives the image obtained from the first camera module 110 having the first pixel structure and the image obtained from the second camera module 120 having the second pixel structure, for example. In order to obtain a third image having improved image quality by using the first image obtained in the first pixel structure and the second image obtained in the second pixel structure, the image mapping unit 140 first obtains the first image and the second image, It is necessary to map between images, pixel size, shape, resolution, and the like. Specifically, the image mapping unit 140 must obtain the pixel value of the second image at a position corresponding to each pixel of the first image obtained in the first pixel structure.

8 (a) is a diagram referred to explain a first pixel structure, and FIG. 8 (b) is a diagram referred to explain a method of obtaining a second image mapped from a second pixel structure. Referring to Fig. 8 (a), all pixels constituting the first pixel structure shown in Fig. 8 (a) are all rectangular in shape and have the same size. According to one embodiment, each pixel value of the mapped second image can be obtained using pixels adjacent to the position corresponding to the center of each pixel of the first pixel structure.

For example, referring to FIG. 8 (b), the position in the second pixel structure corresponding to the pixel center 61 on (2,2) of the first pixel structure can be known. According to one embodiment, the value obtained by linear combination of the pixel in the second pixel structure in which the pixel center 61 is located and its adjacent pixels 72 to 78 is divided into pixels on the (2, 2) (2, 2) of the second pixel structure corresponding to the value of the second pixel structure.

 According to one embodiment, when linearly combining pixels and adjacent pixels in the second pixel structure in which the pixel center is located, and adjacent pixels, the distance and angle between the centers of the respective pixels, the pixel And whether the sizes and shapes of adjacent pixels are the same or not. According to another embodiment, a value obtained by nonlinearly combining a pixel and a neighboring pixel in a second pixel structure in which the pixel center is located is defined as a value of ((2, 2) 2, 2).

9,

FIG. 9 is a diagram for explaining a method of synthesizing a third image in the image combining unit 150 according to an embodiment.

9, the image synthesis unit 150 receives Y1Cb1Cr1, which is information from the first image 310, and Y2Cb2Cr2, which is information from the mapped second image 320, from the image mapping unit 140. FIG. The first image 310 is an image obtained from the first camera module and the mapped second image 320 is an image obtained by mapping the second image acquired from the second camera module to the first image 310. According to an exemplary embodiment, R, G, and B information may be transmitted to the image combining unit 150 instead of Y, Cb, and Cr information, but the present invention is not limited thereto.

According to one embodiment, the first camera module may be a color camera module and the second camera module may be a mono camera module. In this case, the first image 310 obtained from the color camera module is a color image, the second image 320 obtained from the mono camera module is a mono image, and the Cb, Cr Information can be ignored. According to another embodiment, the first camera module may be a mono camera module and the second camera module may be a color camera module. In this case, the first image 310 acquired from the mono camera module is a mono image, the mapped second image 320 acquired from the color camera module is a mono image, and the Cb and Cr information of the first image 310 is Can be ignored.

The image synthesis unit 150 acquires Y3Cb3Cr3 information of the third image 330 using the first image 310 and the mapped second image 320. [ According to one embodiment, the image synthesis unit 150 acquires Y3Cb3Cr3 information of the third image 330 using the parallax information of the same object between the first image 310 and the mapped second image 320 can do. The parallax information means a viewpoint difference between a left view image and a right view image for the same object. The parallax information for the same object between the first image 310 and the mapped second image 320 according to an exemplary embodiment may include at least one of disparity, occlusion, and monocular cues information It can be one.

According to an embodiment, when synthesizing a third image by combining a first image, which is a color image, and a second image, which is a mono image, the image synthesizing unit 150 may use the parallax information as a color image , And a mono image.

When the reference image is a color image, the color information is maintained, so that the color reproducibility of the image is guaranteed, but the detail information of the mono image may be lost in each area. When the reference image is a mono image, the SNR improvement at the mono level is guaranteed, but since the color information of the mono viewpoint must be newly generated, it is difficult to reproduce the color of the occlusion region. As described above, there are advantages and disadvantages both in terms of color image and in mono image.

Accordingly, the image combining unit 150 may use a two-track combining technique to obtain the advantages of the image combining based on the color image and the advantages of the image combining based on the mono image. have. That is, the image synthesizer 150 can determine which of the two methods is to be performed using the parallax information, without determining in advance whether the reference is based on the color image or the mono image. 10 and 11, an operation of the image synthesizing unit 150 for determining a reference image for synthesizing a color image and a mono image will be described.

10 and 11,

FIG. 10 and FIG. 11 are flowcharts illustrating an operation method of an image combining unit according to an exemplary embodiment.

Referring to FIG. 10, in step S410, the parallax information of the first image and the mapped second image is obtained. The parallax information may be at least one of disparity, occlusion, and monocular cues information.

According to an embodiment, the first camera module may be a color camera module and the second camera module may be a mono camera module.

In step S420, whether to compose the color image reference or the mono image reference is determined by using the time difference information. For the same image sensor, the color image has an effect of obtaining color information in comparison with the mono image, and the mono image absorbs more light amount compared with the color image, so that the improved detail information can be obtained.

In the case of synthesizing the third image based on the color image, since the color information can be maintained as it is, the color reproducibility in the color image is ensured and the improved detail information of the mono image can be obtained. On the other hand, in the case of synthesizing the third image based on the mono image, since the detail information of the mono image can be maintained as it is, the resolution and SNR (Signal to Noise Ratio) of the mono image are ensured and the color information of the color image is used There is an effect that can be.

In step S430, the third image is acquired based on the image determined in step S420.

11 is a flowchart illustrating a specific operation method of the image combining unit according to the magnitude of the parallax information value according to an exemplary embodiment.

In step S510, the parallax information of the first image and the mapped second image is obtained.

When the parallax information is greater than the predetermined threshold value in step S520, the parallax information is synthesized on the basis of the color image in step S530. If the parallax information is smaller than the predetermined threshold value, If it is smaller than the threshold value, the third image is synthesized on the basis of the mono image in step S540. Since the color information can be obtained only in the color image and the detail information can be obtained in both the color image and the mono image, when the parallax information is larger than the predetermined threshold value, the third image Can be synthesized.

In step S550, the third image is acquired based on the image determined in steps S530 and S540.

12 and 13,

FIG. 12 is a diagram for explaining a method of using disparity to obtain parallax information in an image synthesizing unit according to an embodiment. FIG. 13 is a flowchart illustrating a method of obtaining parallax information in an image synthesizing unit according to another embodiment. Lt; RTI ID = 0.0 &gt; occlusion &lt; / RTI &gt;

Referring to FIG. 12, the first image 610 is an image obtained from the first camera module, and the mapped second image 620 is an image obtained by mapping a second image acquired from the second camera module to the first image 620 . Even if the first camera module and the second camera module have the same optical characteristics, the first image 610 and the object in the mapped second image 620 due to the positional difference between the first camera module and the second camera module, The position of the light source is changed. According to one embodiment, when xl is the position of the subject 630 in the left view image and xr is the position of the same subject 630 in the right view image, the parallax information is obtained by obtaining the difference between the two values, Can be obtained.

Referring to FIG. 13, the first image 710 is an image obtained from the first camera module, and the mapped second image 720 is an image obtained by mapping a second image acquired from the second camera module to the first image . The multiple camera system may include a first camera module and a second camera module. In this case, the first camera module and the second camera module may be spaced apart from each other by a predetermined distance. When the first camera module and the second camera module capture the same subjects A and B, unlike the second image 720 acquired from the second camera module, A zone region may occur. AL is the position of the subject A in the left view image, aR is the position of the same subject A in the right view image, bL is the position of the subject B in the left view image, and bR is the position of the subject B in the right view image , The parallax information can be obtained by obtaining the difference between aL and aR, bL and bR, respectively, and adding weight values.

14 and 15,

FIG. 14 is a diagram for explaining a method of composing a third image based on a mono image according to an embodiment, and FIG. 15 illustrates a method of synthesizing a third image based on a color image according to another embodiment. Reference is made to the drawings.

Referring to FIG. 14, when a viewpoint difference between a first image and a mapped second image is equal to or less than a predetermined threshold value, a third image may be synthesized based on an image obtained from the mono camera module. In this case, the Y value of the third image is obtained by using the Y value obtained from the mono camera module, and the Cb and Cr values are obtained by compensating the Cb and Cr values obtained from the color camera module for the viewpoint difference between the mono camera module and the color camera module Value can be used.

Referring to FIG. 15, according to another embodiment, when the viewpoint difference between the first image and the mapped second image is equal to or greater than a predetermined threshold value, the third image can be synthesized based on the image obtained from the color camera module. In this case, the Y value of the third image is obtained by compensating the Y value obtained from the mono camera module for the viewpoint difference between the mono camera module and the color camera module, and the Cr and Cr values are obtained from the Cb and Cr values obtained from the color camera module Can be used.

<Fig. 16>

16 is a flowchart showing an operation method of the photographing apparatus according to an embodiment.

In step S610, a first image of the subject is captured from the first camera module.

In step S620, a second image obtained by capturing the same subject from the second camera module is acquired.

The image sensor included in the first camera module may have a first pixel structure in which the pixel shape is only composed of a rectangle and the image sensor included in the second camera module may have a second pixel structure different from the first pixel structure .

The second image acquired from the second camera module may be mapped to the first image acquired from the first camera module in step S630. The mapping process has already been described with reference to FIG. 5, and will not be repeated.

In step S640, the third image may be synthesized using the second image mapped to the first image. Since the process of synthesizing has already been described with reference to FIG. 6, it will not be repeated.

The image display method according to one embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like.

While the present invention has been particularly shown and described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, .

110: first camera module
120: second camera module
130:
140: Image mapping unit
150:

Claims (13)

A first camera module and a second camera module for photographing the same subject;
A second image obtained from the second camera module is mapped to a first image acquired from the first camera module, and a third image of the subject is synthesized using the first image and the second image mapped And a control unit,
Wherein the image sensor included in the first camera module has a first pixel structure having a pixel shape of a quadrangle and the image sensor included in the second camera module has a second pixel structure different from the first pixel structure, Device. The method according to claim 1,
Wherein one of the first camera module and the second camera module is a color camera module and the other is a mono camera module. The method according to claim 1,
Wherein at least one of a size and a shape of the first area and the second area of the plurality of pixel structures included in the second pixel structure is different. 3. The method of claim 2,
Wherein,
Determining whether to use an image obtained from the color camera module or an image obtained from the mono camera module based on parallax information of the first image and the mapped second image, Shooting device. 5. The method of claim 4,
Wherein,
Determines whether the parallax information is larger than a threshold value,
If the parallax information is greater than the threshold value, deciding the image as a reference for the image synthesis as an image acquired from the color camera module,
And determines an image as a reference for the image synthesis as an image acquired from the mono camera module if the parallax information is not greater than the threshold value. 5. The method of claim 4,
Wherein the parallax information includes at least one of disparity and occlusion information of the first image and the mapped second image. Obtaining a first image of a subject photographed from a first camera module and a second image of the subject photographed from a second camera module;
Mapping the acquired second image to the acquired first image; And
And synthesizing a third image of the subject using the first image and the mapped second image,
Wherein the image sensor included in the first camera module has a first pixel structure having a pixel shape of a quadrangle and the image sensor included in the second camera module has a second pixel structure different from the first pixel structure, Method of operation of the device. 8. The method of claim 7,
Wherein one of the first camera module and the second camera module is a color camera module and the other is a mono camera module. 8. The method of claim 7,
Wherein at least one of a size and a shape of the first area and the second area of the plurality of pixel structures included in the second pixel structure is different. 9. The method of claim 8,
Wherein synthesizing the third image of the subject comprises:
Determining whether to use an image obtained from the color camera module or an image obtained from the mono camera module based on the parallax information of the first image and the mapped second image, The method comprising the steps of: 11. The method of claim 10,
Wherein synthesizing the third image of the subject comprises:
Determines whether the parallax information is larger than a threshold value,
If the parallax information is greater than the threshold value, decides an image as a reference for the image synthesis as an image acquired from the color camera module,
Further comprising the step of, if the parallax information is not greater than the threshold value, determining an image as a reference for the image synthesis as an image acquired from the mono camera module. 11. The method of claim 10,
Wherein the parallax information includes at least one of disparity and occlusion information of the first image and the mapped second image. A computer-readable recording medium storing a program for causing a computer to execute the method of claim 7.

Images