CN112889264A

CN112889264A - Method, apparatus, program and recording medium for processing image data

Info

Publication number: CN112889264A
Application number: CN201880098710.9A
Authority: CN
Inventors: 成温滝澤; 星輝; 伊波康
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2018-10-15
Filing date: 2018-10-15
Publication date: 2021-06-01
Also published as: JP7334880B2; WO2020077513A1; JP2022504849A

Abstract

The invention provides a method and apparatus for processing high quality image data. The method for processing image data includes: acquiring first image data captured in a first mode in which a first resolution is set; switching the first mode to a second mode in which a second resolution is set in response to a photographing instruction, the second resolution being different from the first resolution; acquiring second image data captured in the second mode; and applying a first mode-based fusion to merge the second image data with the first image data as a reference image.

Description

Method, apparatus, program and recording medium for processing image data

Technical Field

The present invention relates to a technique related to image processing, and more particularly, to a method and apparatus for processing image data, such as image data captured by a camera.

Background

As is well known, cameras are installed in personal digital assistants such as smartphones for photographing objects. In general, a camera includes an image sensor, and processes image data from the image sensor to display an object.

Disclosure of Invention

Embodiments of the present invention provide a method, apparatus, program, and recording medium to output a high quality image. In order to achieve the foregoing objective, the following technical solutions are used in the embodiments.

A first aspect of the embodiments provides the following method.

A method for processing image data (e.g., image data captured by an imaging unit), comprising:

acquiring first image data captured in a first mode in which a first resolution is set;

switching the first mode to a second mode in response to a photographing instruction (e.g., a photographing instruction issued by a user, a timer, a program, or the like), the second mode having a second resolution set, the second resolution being different from the first resolution;

acquiring second image data captured in the second mode; and

applying a first mode-based fusion to merge the second image data with the first image data used as reference image data.

It may be noted that the first mode image data (i.e., the first image data captured in the first mode) becomes the reference image data in the fusion based on the first mode.

After combining the first mode image data (i.e., the first image data captured in the first mode) and the second mode image data (i.e., the second image data captured in the second mode), the fused image data is output.

Note that in addition to the first resolution, the normal mode or information of the normal mode is included in the first mode, please refer to the detailed description, for example, paragraph [0107 ].

It is noted that the imaging unit 20 is described in detail in the detailed description, for example in paragraph [0042 ].

Note that by using the fusion based on the first mode, the image data of the second mode is fused with the image data of the first mode. For example, the first mode-based fusion may correspond to blocks (61 to 67) in fig. 4.

According to the first aspect, high-quality image data can be output. Specifically, image data between the first mode and the second mode is fused. These modes have different resolutions, so that images with different resolutions and sensitivities can be acquired in these modes. Therefore, the fused image data can be output to obtain high-quality image data.

In a first possible manner of the first aspect, the method may further include: selecting the first image data and selecting the second image data. This may enable fusion of the first and second mode image data.

According to the first aspect, in a second possible manner of the first aspect, the selected first image data may be an image with no or minimal moving objects and the selected second image data may be an image with no or minimal moving objects and a highest sharpening degree. This may make the fused image data the sharpest image.

It will be appreciated that the most sharpened image with the least moving object may be selected or determined from the set of image data captured in the first mode; it will be appreciated that the most sharpened image with the least moving object may be selected or determined from the set of image data captured in the second mode.

In a third possible implementation form of the first aspect, according to any one of the first and second possible implementations, the method may further include: determining whether to apply the first mode-based fusion based on imaging conditions;

wherein the step of switching the first mode to the second mode comprises:

switching the first mode to a second mode when it is determined that the fusion based on the first mode is to be applied and the photographing instruction is received. In one example, equations (1) through (7) in paragraph [0070] correspond to imaging conditions. This can ensure that the fused image data is high-quality image data.

According to the third possible approach, in a fourth possible approach of the first aspect, the step of determining whether to apply the first mode-based fusion based on imaging conditions comprises:

determining whether to apply the first mode-based fusion based on a change between consecutive frames captured in the first mode before the photographing instruction is received. This can ensure that the fused image data is high-quality image data.

According to the fourth possible approach, in a fifth possible approach of the first aspect, the change between the consecutive frames may include all or any of the changes related to: exposure, focus position, block mean, block deviation, and angular velocity of each axis. This can ensure that the fused image data is high-quality image data.

According to the first aspect, the first possible manner to the fifth possible manner, in a sixth possible manner of the first aspect, the first image data and the second image data may be set to have the same angle of view and the same focus position. This can ensure that the fused image data is high-quality image data.

In a seventh possible implementation form of the method according to the first aspect as such or according to any of the first to sixth possible implementation forms of the first aspect, the first mode-based fusion may be combined with a shutter lag zero latency, wherein there is no time lag from the reception of the shooting instruction to the completion of the shooting using the image captured before the reception of the shooting instruction. This can ensure that the fused image data is high-quality image data.

In an eighth possible manner of the first aspect, according to any one of the first to seventh possible manners of the first aspect, the method may further include:

performing camera shake compensation between the first mode and the second mode, and in particular, between frames captured in the first mode and the second mode, wherein the frames may include a first frame (including the first image data) captured in the first mode and a second frame (including the second image data) captured in the second mode;

wherein the first image data and the second image data after the camera shake compensation are used in the applying the first mode-based fusion.

It will be appreciated that the first image data may be used as reference image data, and therefore the second image data is subject to camera shake relative to the first image data, so camera shake compensation is applied to the second image data. This can ensure that the fused image data is high-quality image data.

According to the first aspect as well as any one of the first to eighth possible manners, in a ninth possible manner of the first aspect,

the first mode may be a high resolution mode and the second mode may be a normal mode; or

The first mode may be a normal mode and the second mode may be a high resolution mode. This can ensure that the fused image data is high-quality image data.

According to the first aspect, any one of the first to ninth possible manners, in a tenth possible manner of the first aspect, the first image data and the second image data may be acquired from a same image sensor in the imaging unit. This can ensure that the fused image data is high-quality image data.

In an eleventh possible manner of the first aspect, according to any one of the first to tenth possible manners of the first aspect, the method may further include:

merging a plurality of pieces of image data in units of image data captured in the first mode and the second mode, thereby performing a noise reduction process;

wherein the first image data and the second image data after the noise reduction process may be used in the applying the first mode-based fusion. This can ensure that the fused image data is high-quality image data.

A second aspect of the embodiments provides the following method.

A method for processing image data, such as image data captured by an imaging unit, comprising:

acquiring a video sequence including a plurality of pieces of image data captured in different modes in which different resolutions are set;

determining whether to apply different mode-based fusion based on changes between successive image frames of the video sequence; in particular, determining whether to apply different mode-based fusion to merge the image data captured in different modes based on changes between image frames of the video sequence; and

applying the mode-based fusion to the image data of the video sequence based on a result of the determining; for example, when it is determined that the mode-based fusion is to be applied to merge the image data captured in the different modes, the mode-based fusion is applied to the image data of the video sequence.

According to the second aspect, high-quality image data can be continuously output. Specifically, image data between the first mode and the second mode is fused. These modes have different resolutions, so that images with different resolutions and sensitivities can be acquired in these modes. Therefore, the fused image data can be output to obtain high-quality image data.

A third aspect of the embodiments provides the following apparatus.

An apparatus for processing image data, such as image data captured by an imaging unit, comprising:

a first acquisition unit configured to acquire a plurality of pieces of first image data captured in a first mode having a first resolution;

a switching unit configured to switch the first mode to a second mode in which a second resolution is set in response to a photographing instruction (for example, a photographing instruction issued by a user, a timer, a program, or the like), the second resolution being different from the first resolution;

a second acquisition unit configured to acquire second image data captured in the second mode, specifically, image data captured at the second resolution of the second mode; and

a fusion unit for applying fusion based on a first mode to merge the second image data with the first image data as reference image data.

According to the third aspect, high-quality image data can be output. Specifically, image data between the first mode and the second mode is fused. These modes have different resolutions, so that images with different resolutions and sensitivities can be acquired in these modes. Therefore, the fused image data can be output to obtain high-quality image data.

In a first possible manner of the third aspect, the fusion unit may be configured to select the first image data and the second image data. This may enable fusion of the first and second mode image data.

According to the first possible manner of the third aspect, in a second possible manner of the third aspect, the selected first image data may be an image without a moving object or with the least moving object and with the highest degree of sharpening, and the selected second image data may be an image without a moving object or with the least moving object and with the highest degree of sharpening. This may make the fused image data the sharpest image.

According to the third aspect as well as any one of the first and second possible approaches, in a third possible approach of the third aspect, the apparatus may further include:

a determination unit for determining whether to apply the first mode-based fusion based on an imaging condition,

wherein the switching unit may be configured to switch the first mode to the second mode when it is determined that the fusion based on the first mode is to be applied and the photographing instruction is received. This can ensure that the fused image data is high-quality image data.

In a fourth possible manner of the third aspect, the determination unit may be dedicated to determining whether to apply the first mode-based fusion based on a change between consecutive frames captured in the first mode before the shooting instruction is received. This can ensure that the fused image data is high-quality image data.

According to the fourth possible approach of the third aspect, in a fifth possible approach of the third aspect, the change between the consecutive frames may include all or any of the changes related to: exposure, focus position, block mean, block deviation, and angular velocity of each axis. This can ensure that the fused image data is high-quality image data.

According to any one of the first to fifth possible manners of the third aspect, in a sixth possible manner of the third aspect, the first image data and the second image data may be set to have the same angle of view and the same focus position. This can ensure that the fused image data is high-quality image data.

According to any one of the first to sixth possible approaches of the third aspect, in a seventh possible approach of the third aspect, the first mode-based fusion may be combined with a shutter lag zero latency, wherein there is no latency from receiving the shooting instruction to completing the shooting using an image captured before receiving the shooting instruction. This can ensure that the fused image data is high-quality image data.

According to any one of the first to seventh possible modes of the third aspect, in an eighth possible mode of the third aspect,

the fusion unit may comprise an alignment unit for performing camera shake compensation between the first mode and the second mode, in particular between frames captured in the first mode and the second mode, wherein the frames comprise: a first frame captured in the first mode and a second frame captured in the second mode; and

the fusion unit may be configured to apply the fusion to:

the first image data and the second image data after the camera shake compensation to merge the second image data after the camera shake compensation with the first image data serving as reference image data. Specifically, each piece of second image data has camera shake between different modes.

This can ensure that the fused image data is high-quality image data.

In a ninth possible implementation form of the third aspect, the first mode may be a high resolution mode and the second capture mode may be a normal mode, according to any one of the first to eighth possible implementation forms of the third aspect; or

The first mode may be a normal mode and the second capture mode may be a high resolution mode. This can ensure that the fused image data is high-quality image data.

According to any one of the first to ninth possible modes of the third aspect, in a tenth possible mode of the third aspect, the first image data and the second image data may be acquired from the same image sensor. This can ensure that the fused image data is high-quality image data.

In an eleventh possible manner of the third aspect, according to any one of the first to tenth possible manners of the third aspect, the apparatus may further include a noise reduction unit configured to merge a plurality of pieces of image data in units of image data captured in the first mode and the second mode, thereby performing a noise reduction process,

wherein the fusion unit may be configured to apply the fusion to the first image data and the second image data after the noise reduction process.

A fourth aspect of the embodiments provides the following apparatus.

An apparatus for processing image data captured by an imaging unit, comprising:

an acquisition unit configured to acquire a video sequence including a plurality of pieces of image data captured in different modes in which different resolutions are set;

a determining unit for determining whether to apply a different mode based fusion based on a change between successive frames of the video sequence; and

a fusion unit for applying the different mode-based fusion to the image data of the video sequence based on a result of the determination.

According to the fourth aspect, high-quality image data can be continuously output. Specifically, image data between the first mode and the second mode is fused. These modes have different resolutions, so that images with different resolutions and sensitivities can be acquired in these modes. Therefore, the fused image data can be output to obtain high-quality image data.

A fifth aspect of the embodiments provides a computer-readable storage medium recording a program for causing a computer to execute the method according to any one of the first and second aspects of the embodiments and the first through tenth possible manners of the first aspect.

A sixth aspect of the embodiments provides a computer program for causing a computer to perform the method according to any one of the first and second aspects of the embodiments and the first to tenth possible ways of the first aspect.

According to a seventh aspect, the invention relates to a device for processing image data, comprising a camera module, one or more processors, a memory. The memory stores instructions that cause the one or more processors to perform the method according to the first or second aspect.

The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims.

Drawings

In order to more clearly describe the technical solutions in the embodiments, the drawings required for describing the current embodiment are briefly described below. It is clear that the drawings in the following description depict only some possible embodiments and that a person skilled in the art can derive from them, without inventive effort, still other drawings, in which:

fig. 1 is a schematic diagram of a configuration example of an image processing apparatus according to a first embodiment;

fig. 2 is a diagram of the correlation between the physical pixel number of the image sensor and the pixel number of the image data in each mode in the image processing apparatus according to the first embodiment;

fig. 3 is a timing chart of applying fusion based on a high resolution mode in the image processing apparatus according to the first embodiment;

fig. 4 is a diagram of a detailed configuration example of an image processing apparatus in a high resolution mode according to the first embodiment;

fig. 5 is a timing chart of applying normal mode-based fusion in the image processing apparatus according to the first embodiment;

fig. 6 is a diagram of a detailed configuration example of an image processing apparatus in a normal mode according to the first embodiment;

fig. 7 is a diagram of a functional configuration example of an image processing apparatus according to the first embodiment;

fig. 8 is a flowchart of an example of an entire imaging process of the image processing apparatus according to the first embodiment;

fig. 9A is a diagram of a detailed configuration example of an image processing apparatus according to a second embodiment in a high resolution mode;

fig. 9B is a diagram of a detailed configuration example of an image processing apparatus according to the second embodiment in the normal mode;

fig. 10 is a diagram for describing an outline of fusion carried out by an image processing apparatus according to a second embodiment;

FIG. 11 is a diagram for describing aspects of noise reduction;

fig. 12 is a diagram of a functional configuration example of an image processing apparatus according to a third embodiment;

fig. 13 is a diagram for describing an example of a video sequence implemented by an image processing apparatus according to a third embodiment;

fig. 14 is a flowchart of an example of an imaging process of an image processing apparatus according to a third embodiment;

fig. 15 is a diagram for describing an example of a video sequence when mode switching is not performed.

Detailed Description

The technical solutions of the embodiments of the present invention will be described clearly below with reference to the accompanying drawings of the embodiments of the present invention. It is to be understood that the described embodiments are only a part, and not all, of the embodiments of the invention. It should be noted that all other embodiments obtained by persons skilled in the art based on the embodiments of the present invention without any inventive step belong to the protection scope of the embodiments of the present invention.

[ first embodiment ]

The image processing apparatus 10 in the present exemplary embodiment will be explained below. The image processing apparatus 10 is for processing image data captured by a camera.

[ configuration of image processing apparatus 10]

Fig. 1 is a schematic diagram showing an example of the hardware configuration of an image processing apparatus 10 according to the first embodiment.

As shown in fig. 1, the image processing apparatus 10 includes a processing unit 11, a memory 12, an input device 13, a display device 14, a camera module (imaging unit) 20, and a gyroscope 30. The camera module 20 includes an image sensor 21. In the present embodiment, the image processing apparatus 10 includes, for example, a tablet computer, a tablet phone or a mobile phone, a personal digital assistant, a personal computer or a robot.

The processing unit 11 is connected to the respective components through a bus to perform a process of transmitting control signals and data. The processing unit 11 may also execute various programs, arithmetic processing, timing control, and the like to realize the overall operation of the image processing apparatus 10. The program may be stored in a computer-readable storage medium, such as a Digital Versatile Disk (DVD) -Read Only Memory (ROM), a Compact Disk (CD) -Read Only Memory (ROM). The program may also be stored in a separate readable computer storage medium, such as a secure digital card (SD card). The Processing Unit 11 is a Processing device such as a Central Processing Unit (CPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), or a Field Programmable Gate Array (FPGA). The Memory 12 is a Read Only Memory (ROM), a Random Access Memory (RAM), a Hard Disk Drive (HDD), a Solid State Drive (SSD), or a flash Memory.

The memory 12 stores therein a program for an operating system and various data necessary when the image processing apparatus 10 performs general operation control. A storage area for temporarily storing data and programs is provided in the memory 12 to hold programs and data, and other data required for processing in the apparatus 10.

The input device 13 includes operation buttons, a touch panel, an input pen, and a sensor.

The display device 14 may be a flat panel display such as a liquid crystal display or an Electro-Luminescence (EL) display.

The image sensor 21 included in the camera module 20 acquires image data of an object. As will be described later, the image sensor 21 in the image processing apparatus 10 of the present embodiment has a high resolution mode 91 and a normal mode 92. In the high resolution mode 91 having a specific resolution, high resolution image data is acquired. In the normal mode 92, the resolution is different from that of the high resolution mode 91, and although the resolution of the image data is lower than that of the image data in the high resolution mode, the sensitivity and the Signal to Noise Ratio (SNR) of the image data are higher than those of the image data in the high resolution mode. The modes are not limited to the two modes of mode 91 and mode 92, and a High Dynamic range Rendering (HDR) mode may be used.

In the present embodiment, the image sensor 21 may be configured by, for example, a Complementary Metal Oxide Semiconductor (CMOS), but may be configured by other configurations as long as image data of an object can be acquired at different resolutions. The camera module 20 may be installed outside the image processing apparatus 10.

The gyroscope 30 is used to detect an angular velocity to detect a shake of the camera (i.e., the camera module 20).

Fig. 2 is a diagram showing the correlation between the physical pixel number of the image sensor 21 and the pixel number of the image data in each mode.

The number of pixels of the image data in the high resolution mode 91 in fig. 2(B) is the same as the number of pixels of the image sensor 21 in fig. 2 (a). In this case, one pixel in fig. 2(B) is output as one pixel. The pixel array shown in fig. 2(B) shows one example of a bayer array.

However, the number of pixels of the image data in the normal mode 92 in fig. 2(C) is smaller than that of the image sensor 21 in fig. 2 (a). For example, 2 × 2 pixels in fig. 2(C) are output as one pixel.

The image data of the image sensor 21 is periodically output to the processing unit 11 and the memory 12 in the form of frames.

Image processing summary in high resolution mode

An outline of image processing in the high resolution mode 91 implemented by the image processing apparatus 10 is described next with reference to fig. 1 to 3. Fig. 3 is a timing diagram illustrating application of high resolution mode-based fusion.

As shown in fig. 3, in this image processing apparatus 10k, the image sensor 21 sequentially acquires a plurality of pieces of image data H (n-x) to H (n) (where n and x are positive integers) corresponding to the high resolution mode 91.

As shown in fig. 3, for example, an instruction to capture image data is received at t1, after which the processing unit 11 switches the high resolution mode 91 to the normal mode 92 at t 2. Possible examples of the shooting instruction include, for example, performing a touch operation on a shutter button displayed on the display device 14.

The plurality of pieces of image data N (N +1) to N (N +5) in the normal mode 92 are sequentially acquired from the image sensor 21.

In fig. 3, the duration from T0 to T1 is a period T of shutter lag zero delay shooting during which the image data H (n-x) to H (n) in the above-described high resolution mode 91 can be preview-displayed on the display device 14. The shutter time lag zero time lag means that there is no time lag from the reception of the photographing instruction to the completion of the photographing of the subject.

Then, the fusion based on the high resolution mode is applied to the image data N (N +1) in the normal mode 92, so that the image data N (N +1) and the image data H (N-x) indicated by the symbol P are fused. Therefore, the fused image data is output as the image data required by the photographing instruction.

In the above fusion, based on the pixel characteristics (luminance values, etc.) in the two pieces of image data N (N +1), H (N-x), the image processing apparatus 10 selects a pixel in an appropriate one of the two pieces of image data, and outputs the image data.

For example, when the pixel feature reflects an edge, a pixel in the image data H (n-x) in the high resolution mode 91 is selected. On the other hand, when the pixel characteristics reflect the background with less variation, the pixel in the image data N (N +1) in the normal mode 92 is selected. Therefore, the image data to be output contains pixels in an appropriate pattern according to the pixel characteristics. Thereby, high-quality image data can be output.

The application relationship of the fusion between the patterns 91 and 92 is not limited to that illustrated in fig. 3. For the image data between the modes 91 and 92 in the fusion process, high-quality image data can be output by using image data that is less affected by the shake (for example, image data having a smaller difference in equations (3) and (4) given below). Thus, for example, image data in the high resolution mode 91 and the image data N (N +1) may be fused instead of the image data H (N-x) and the image data N (N + 1). Alternatively, any one of the image data H (N-x) to H (N) in the high resolution mode 91 and any one of the image data N (N +1) to N (N +5) may be fused.

Detailed configuration example of image processing apparatus in high resolution mode

Fig. 4 is a diagram showing a detailed configuration example of the image processing apparatus 10 in the high resolution mode.

Referring to fig. 4, the image processing apparatus 10 includes a memory 12, a camera module 20, a gyroscope 30, a first processing unit 50, and a second processing unit 60.

First, the first processing unit 50 related to switching between the mode 91 and the mode 92 is described. The first processing unit 50 includes a 3A unit 51, a moving object detector 52, a camera shake detector 53, and a capture mode controller 54. 3A refers to Automatic Exposure (AE), Auto Focus (AF), and Automatic White Balance (AWB) of preprocessing.

The image data output from the image sensor 21 in the camera module 20 is sequentially output to the 3A unit 51 and the moving object detector 52 frame by frame.

As will be described later, the 3A unit 51 calculates an exposure difference and a focus position difference between two consecutive frames. The moving object detector 52 calculates the average value and the deviation value of the luminance on a block-by-block basis (8 × 8 pixels, etc.), for example, to detect the motion of the object.

The respective velocities of 3 axes (x, y, z) between two consecutive frames are input from the gyroscope 30 into the camera shake detector 53.

The values of the respective units 51 to 53 are sequentially input to the capture mode controller 54 to be stored therein. The capture mode controller 54 then determines whether the first mode based fusion can be applied. The conditions given by the following equations (1) to (7) are used in this determination.

Exposure difference is less than or equal to threshold A (1)

Difference in focusing position is less than or equal to threshold B (2)

Mean value difference is less than or equal to threshold C (3)

Deviation difference is less than or equal to threshold D (4)

Acceleration difference of x-axis is less than or equal to threshold E (5)

Acceleration difference of y-axis is less than or equal to threshold F (6)

z-axis acceleration difference ≤ threshold G (7)

The difference value of equations (1) and (2) corresponds to the value calculated by the 3A unit 51. The difference values of equations (3) and (4) correspond to the values calculated by the moving object detector 52. The difference values of equations (5) to (7) represent values acquired from the camera shake detector 53.

Each disparity value of equations (1) to (7) shows a change between successive frames when image data in the high resolution mode 91 (image data H (n-x) to H (n) in fig. 3) is captured.

Referring to fig. 3, when all the conditions in equations (1) to (7) are satisfied, the capture mode controller 54 determines to apply the above-mentioned high resolution mode-based fusion. The conditions for determination are not limited to the conditions shown in equations (1) to (7), and equations (1) to (7) may be combined arbitrarily.

Satisfying the conditions in equations (1) to (7) means that the change of the image data (scene, brightness, motion of the object) is small. For example, when the image data is a scene of a landscape or the like, the image data may be one in which the above-described variation is small (for example, each of the disparity values in equations (1) to (7) is small).

However, when any one of the conditions of equations (1) to (7) is not satisfied, it indicates that the change in the image data is large. For example, when the image data is a scene in which an object in the image data moves, the image data may be image data in which the above-described variation is large (for example, each difference value in equations (1) to (7) is large). In this case, whether or not fusion is applied is not determined, and image data from the image sensor 21 may be output in which fusion described later is not applied.

When receiving a photographing instruction (time t1 in fig. 3), the capture mode controller 54 switches the high resolution mode 91 to the normal mode 92 (time t2 in fig. 3) to acquire image data (image data N (N +1) to N (N +5) in fig. 3) in the normal mode 92. In this case, the capture mode controller 54 communicates with the camera module 20 to perform exposure control, focus control, mode configuration, and the like, following a communication protocol such as a Mobile Industry Processor Interface (IMPI) and an Inter-Integrated Circuit (I2C).

In the capture mode controller 54, each piece of image data between the modes 91 and 92 is set to have the same angle of view and focus position.

The second processing unit 60 is described next. The second processing unit 60 includes

image processing units

61 and 63, resizing

units

62 and 64, a motion calculation unit 65, an image alignment unit 66, and a fusion unit 67.

In the present embodiment, the image processing unit 61 and the resizing unit 62 are supplied with image data in the high resolution mode 91. The image processing unit 61 converts the raw image data from the image sensor 21 into Red, Green and Blue (RGB) image data. In this case, the image data is read out from the memory 12. The image data may be converted into image data having, for example, a luminance signal (Y), a differential signal (U) of a blue component, and a differential signal (V) of a red component.

The resizing unit 62 is configured to reduce the image data in the high resolution mode 91 such that the size (the number of pixels) of the image data in the high resolution mode 91 is the same as the size (the number of pixels) of the image data in the normal mode 92. This is because the number of pixels output by the image sensor 21 differs between the modes 91 and 92, as shown in fig. 2.

The image sensor 21 supplies the image data in the normal mode 92 to the image processing unit 63 and the resizing unit 64. The image processing unit 63 converts the raw image data from the image sensor 21 into YUV or RGB image data. The resizing unit 64 is configured to enlarge the image data in the normal mode 92 such that, in the high resolution mode 91, the size of the image data in the normal mode 92 is the same as the size of the image data in the high resolution mode 91.

The resizing

units

62 and 64 supply the image data to the motion calculation unit 65, and then the motion calculation unit 65 calculates motion compensation data to compensate for camera shake in the image data between the modes 91 and 92. The motion calculation unit 65 performs alignment so that the image data in the normal mode 92 can be aligned with the image data in the high resolution mode 91 according to the calculation result in the motion calculation unit 65. In the example of fig. 3, image data N (N +1) is aligned with image data H (N-x) selected as a reference frame.

The image data from the resizing unit 62 and the image data from the image alignment unit 66 are supplied to the fusion unit 67. The two pieces of image data are fused and output by applying fusion based on a high resolution mode. In the example of fig. 3, image data H (N-x) indicated by symbol p is selected as a reference frame, and is fused with N (N +1) and output. When the image data after the fusion is a landscape scene including a building, for example, in fig. 3, pixels of the image data H (N-x) may be used to represent an edge of a boundary portion between the building and the background (where the above-described variation is large), and pixels of the image data N (N +1) may be used to the background (where the above-described variation is small). Thereby, a landscape scene with high image quality can be captured.

If no fusion is applied, the image data from the image sensor 21 may be output through the image processing unit 61.

Image processing summary in normal mode

The outline of image processing in the normal mode 92 is described next with reference to fig. 1, 2, and 5.

FIG. 5 is a timing diagram similar to FIG. 3 showing application of normal mode based fusion. In the example of fig. 5, image data N (N-x) to N (N) (N and x are positive integers) in the normal mode 92 are sequentially acquired. Then, upon receiving the photographing instruction at t1, the processing unit 11 switches the normal mode 92 to the high resolution mode 91 at time t2 to sequentially acquire image data H (n +1) to H (n +5) in the high resolution mode 91.

In the example of fig. 5, image data N (N-x) indicated by symbol P is selected as a reference frame and fused with H (N + 1). This fusion is performed in a similar manner to the fusion shown in fig. 3. In other words, based on the pixel characteristics (luminance values and the like) in the image data N (N +1) and H (N-x), the image processing apparatus 10 selects a pixel in an appropriate piece of image data of the two pieces of image data to set image data to be output. For example, when the pixel feature reflects an edge, a pixel in the image data H (n-x) in the high resolution mode 91 is selected. On the other hand, when the pixel characteristics reflect the background with less variation, the pixel in the image data N (N +1) in the normal mode 92 is selected. Therefore, the image data to be output contains pixels in an appropriate pattern according to the pixel characteristics. Thereby outputting high-quality image data.

In fig. 5, similarly to the one shown in fig. 3, the duration from T0 to T1 is a shutter lag zero delay shooting period T during which the image data N (N-x) to N (N) in the above-described normal mode 92 can be preview-displayed on the display device 14.

The application relationship of the fusion between the patterns 91 and 92 is not limited to that illustrated in fig. 5, and various aspects may be implemented. Any one of the image data N (N-x) to N (N) in the normal mode 92 and any one of the image data H (N +1) to H (N +5) in the high resolution mode 92 may be fused.

[ detailed configuration example of image processing apparatus in Normal mode ]

Fig. 6 shows a detailed configuration example of the image processing apparatus 10 in the normal mode 92. Since the difference between the configuration of fig. 6 and the configuration of fig. 4 is that the normal mode 92 is switched to the high resolution mode 91, this point is described below with emphasis.

Referring to fig. 6, when all the conditions in equations (1) to (7) are satisfied, the capture mode controller 54 of the first processing unit 50 determines to apply the normal mode-based fusion described above with reference to fig. 5.

Upon receiving the photographing instruction (time t1 in fig. 5), the capture mode controller 54 switches the normal mode 92 to the high resolution mode 91 (time t2 in fig. 5), and acquires image data in the high resolution mode 91 (H (n +1) to H (n +5) in fig. 5).

The image alignment unit 66A in the second processing unit 60 aligns the image data in the high resolution mode 91 with the image data in the normal mode 92 according to the calculation result in the motion calculation unit 65. The motion calculation unit 65 calculates the motion of the image data in the high resolution mode 91 in conjunction with the image data in the normal mode 92. In the example of fig. 5, the image data H (N +1) is aligned with the image data N (N-x) selected as the reference frame.

The image data from the resizing unit 64 and the image data from the image alignment unit 66A are supplied to the fusion unit 67. In the fusion unit 67, the two pieces of image data are fused and output by applying fusion based on the high resolution mode. In the example of fig. 5, image data N (N-x) and H (N +1) are fused and output. Therefore, as in the case of fig. 3, a landscape scene with high image quality can be captured.

If no fusion is applied, the image data from the image sensor 21 may be output through the image processing unit 63.

Functional configuration of image processing apparatus 10

Fig. 7 is a diagram showing a functional configuration of the image processing apparatus 10. The functional configuration is described below in conjunction with the figure. As shown in fig. 7, the image processing apparatus 10 includes a first acquisition unit 101, a determination unit 102, a switching unit 103, a second acquisition unit 104, a fusion unit 105, and a display control unit 106. In an exemplary embodiment, these components are implemented by the processing unit 11 and the memory 12 of fig. 1, and are configured as follows.

The first acquisition unit 101 is used to acquire a plurality of pieces of first image data captured in a first mode (high resolution mode 91 or normal mode 92) having a preset resolution. The first acquisition unit 101 corresponds to the memory 12 in fig. 4 and 6 and the 3A unit 51 and the detector 52 in the first processing unit 50.

The determination unit 102 is configured to determine whether or not to apply mode-based fusion based on imaging conditions (equations (1) to (7) described above in the present embodiment) when image data is captured in the first mode in response to a shooting instruction. The determination unit 102 corresponds to the detector 53 and the acquisition mode controller 54 in the first processing unit 50 in fig. 4 and 6.

The switching unit 103 is configured to switch the first mode to the second mode (the normal mode 92 or the high resolution mode 91) upon receiving a shooting instruction. The switching unit 103 corresponds to the acquisition mode controller 54 in fig. 4 and 6.

The second acquisition unit 104 is configured to acquire second image data captured at a resolution of the second mode. The second acquisition unit 104 corresponds to the memory 12 in fig. 4 and 6.

The fusion unit 105 applies fusion based on the first mode to the second image data for output of the second image data to use the first image data. The fusion unit 105 corresponds to the second processing unit 60 in fig. 4 and 6.

The alignment unit 1051 in the fusion unit 105 is used to align the motion of the image data between the first mode and the second mode. The alignment unit 1051 corresponds to the motion calculation unit 65 and the

image alignment units

66, 66A in the second processing unit 60 in fig. 4 and 6.

The display control unit 106 is configured to preview-display the first image data before receiving the photographing instruction.

Reference is made to the aforementioned components as necessary in the description of the operation of the image processing apparatus 10.

[ operation of the image processing apparatus 10]

General image processing of the image processing apparatus 10 is described below with reference to fig. 1 to 8. In the image processing apparatus 10 of the present embodiment, the processing unit 11 can execute various processes to be described later according to a program.

Fig. 8 is a flowchart showing an example of general image processing by the image processing apparatus 10.

In fig. 8, the processing unit 11 sequentially acquires image data in the first mode from the image sensor 21 (step S101). In the example of fig. 3, image data H (n-x) to H (n) in the high resolution mode 91 are acquired as the first mode. In the example of fig. 5, image data N (N-x) to N (N) in the normal mode 92 are acquired as the second mode. These image data are sequentially stored in the memory 12.

In this step, the processing unit 11 cooperates with the memory 12 and the camera module 20 to be implemented as the first acquisition unit 101.

When the image data in step S101 has been captured, in response to a shooting instruction (for example, a shutter lag zero-delay shooting instruction), the processing unit 11 determines whether or not to apply fusion based on the first mode based on the imaging conditions (equations (1) to (7)) (step S102). In this case, the processing unit 11 determines whether the conditions of equations (1) to (7) are satisfied. When these conditions are satisfied, it is determined that the fusion based on the first mode is applied, and when these conditions are not satisfied, it is not determined that the fusion based on the first mode is applied.

In step S102, the processing unit 11 cooperates with the gyroscope 30 to be implemented as the determination unit 102.

When receiving the photographing instruction (time t1 in fig. 3 and 5), the processing unit 11 switches the first mode to the second mode (step S103). In the example of fig. 3, at t2, the high resolution mode 91 is switched to the normal mode 92. In the example of fig. 5, at t2, the normal mode 92 is switched to the high resolution mode 91.

In step S103, the processing unit 11 is implemented as the switching unit 103.

The processing unit 11 acquires image data captured by the image sensor 21 at the resolution of the second mode (step S104). In the example of fig. 3, image data N (N +1) to N (N +5) in the normal mode 92 are acquired as the second mode. In the example of fig. 5, image data N (N +1) to N (N +5) in the normal mode 92 is acquired as the second mode. These image data are sequentially stored in the memory 12.

In step S104, the processing unit 11 cooperates with the memory 12 to be implemented as the second acquisition unit 104.

For the output of the image data acquired in step S104, the processing unit 11 applies fusion based on the first mode to the image data to use the first image data acquired in step S101 (step S105). In this case, the processing unit 11 may select a pixel in the image data in the appropriate mode among the two pieces of image data based on the pixel feature in the target image data, and output the image data according to the shooting instruction. In the example of fig. 3, the processing unit 11 may select image data H (N-x) of the image data H (N-x) to H (N) in the high resolution mode 91 and image data N (N +1) of the image data N (N) to N (N +5) in the normal mode 92, and further fuse the image data H (N-x) in the high resolution mode 91 with the image data N (N +1) in the normal mode 92 to output fused image data. In the example of fig. 5, image data N (N-x) in the normal mode 92 and image data H (N +1) in the high resolution mode 91 are fused to be output. The selected image data for each mode may be the most sharpened image with no or minimal moving objects.

Referring to the flowchart in fig. 8, the image data acquired in step S101 is different in size from the image data acquired in step S104. In step S105, the processing unit 11 adjusts the size of the target image data (image data H (n-x) in fig. 3) or the target image data (image data H (n +1) in fig. 5), and aligns the two pieces of image data to compensate for camera shake.

In step S105, the processing unit 11 is implemented as the fusion unit 105.

In fig. 8, before receiving the photographing instruction in step S103, the processing unit 11 (display control unit 106) may preview-display the image data acquired in step S101 on the display device 14. Therefore, the image data acquired in step S101 may be visible, so that the shutter can be pressed at an appropriate time.

As described above, the image processing apparatus 10 of the present embodiment is used to fuse image data in different modes 91 and 92 with respect to the output of the image data. In the high resolution mode 91, high resolution image data may be captured. On the other hand, in the normal mode 92, although the resolution is lower than that in the high resolution mode 91, high-sensitivity, high-SNR image data can be captured. Therefore, the image data fused in step S105 is composed of the image data captured in the appropriate mode, and thus may have higher image quality.

Image data in each of the modes 91 and 92 is acquired from one image sensor 21. This reduces the number of required image sensors, so that the image processing apparatus 10 can be downsized.

[ second embodiment ]

An image processing apparatus 10A according to a second embodiment is described with reference to fig. 9A and 9B. Fig. 9A is a diagram for describing an outline of the fusion based on the high resolution mode in the image processing apparatus 10A. Fig. 9B is a diagram for describing an outline of normal mode-based fusion in the image processing apparatus 10A. Symbols and the like used in the description of the first embodiment are also used in the following description of the second embodiment unless otherwise specified.

In fig. 9A and 9B, the image processing apparatus 10A further includes

noise reduction units

68 and 69, unlike the image processing apparatus 10 of the first embodiment (fig. 4 and 6). Although the

noise reduction units

68 and 69 shown in fig. 9A and 9B are shown in solid lines and broken lines, the

units

68 and 69 may be provided at positions indicated by solid lines or broken lines.

The noise reduction unit 68 is configured to perform a noise reduction process on each piece of image data in the high resolution mode 91 in units of a plurality of frames. The noise reduction unit 69 is configured to perform a noise reduction process on each piece of image data in the normal mode 92 in units of a plurality of frames.

The noise reduction process is described in detail with reference to fig. 9A, 9B, and 10.

Fig. 10 illustrates an aspect of noise reduction by the

noise reduction units

68 and 69. In fig. 10, four consecutive frames f0 to f3 correspond to image data in the normal mode 92, for example, and four consecutive frames f4 to f7 correspond to image data in the high resolution mode 91, for example.

In fig. 10, in the noise reduction unit 68, four consecutive frames F4 to F7 are merged into one frame F0 in the course of the MFNR 201. This provides the noise reduction frame F0 in the high resolution mode 91. The number of samples of the frame is not limited to four illustrated in fig. 10, but may be changed.

In the process of digital binning 202, in order to adjust the size between the modes 91 and 92, four frames f4 through f7 are changed to frames f41 through f44 so as to be aligned with the sizes of frames f0 through f3 in the normal mode 92.

Although the frames f41 to f44 are obtained by changing the sizes of the frames f4 to f7, changing the sizes can reduce noise. This is because the generation of noise components may be affected by the resolution.

Fig. 11 shows an example of a process of noise-reducing image data containing

noise components

82a, 82b, 82c in a high-resolution mode 91, in which the mode of the image data is converted into a normal mode 92 to eliminate the

noise components

82a, 82b, 82 c. In the luminance variation 81 of the image data in the high resolution mode 91 in fig. 11(a), noise components 82a to 82c are contained. Since the generation of these noise components may be affected by the size of the resolution, reducing the resolution may reduce the noise components. In the luminance variation 83 of the image data in the normal mode 92 in fig. 11(B), the noise components 82a to 82c can be eliminated. The noise reduction frames f41 to f44 in fig. 10 are obtained in this manner.

In fig. 10, in the noise reduction unit 69, eight frames F0 to F4, F41 to F44 are combined into one frame F1 in the process of the MFNR 203. This provides a noise reduced frame F1 in normal mode 92.

In the process of the fusion 204 performed by the fusion unit 67, as in the fusion according to the first embodiment, two frames of F0 and F1 are also fused to output a single frame F2. Therefore, high-quality image data can be output.

[ third embodiment ]

An image processing apparatus 10B according to the third embodiment is described. The configuration of the image processing apparatus 10B is substantially the same as that shown in fig. 1, 4, and 6. Symbols and the like used in the description of the first embodiment are also used in the following description of the third embodiment unless otherwise specified.

The image processing apparatus 10B processes a video sequence including a plurality of pieces of image data. This can be achieved, for example, as follows: the processing unit 11 acquires image data in different modes 91 and 92, and fuses pieces of image data based on inter-frame changes in the image data.

Fig. 12 shows a functional configuration of the image processing apparatus 10B.

Referring to fig. 12, the image processing apparatus 10B includes an acquisition unit 301, a determination unit 302, and a fusion unit 303. The acquisition unit 301 corresponds to the first processing unit 50 and the memory 12 in fig. 4 and 6. The determination unit 302 corresponds to the capture mode controller 54 in fig. 4 and 6. The merging unit 303 corresponds to the second processing unit 60 in fig. 4 and 6. These components are referenced as needed in the operational description given later with reference to fig. 13.

Fig. 13 shows an aspect of fusion of image data in a video sequence in the image processing apparatus 10B. At the input in fig. 13(a), for example,

image data

600, 602, 604, 606 in the high resolution mode 91 and

image data

601, 603, 605, 607 in the high resolution mode 91 are alternately given. In each of the Motion Compensation (MC) 501 to 507, fusion is performed for two pieces of image data in different modes, and inter-mode Motion is compensated therein. The process of each MC is similar to that in each of the units 65 to 67 in fig. 4, or similar to that in each of the

units

65, 66A, and 67 in fig. 6.

At the output of fig. 13(B), fused image data 700 to 707 are output. This outputs a video sequence with high image quality.

Fig. 14 is a flowchart showing an example of general image processing of the image processing apparatus 10B.

Referring to fig. 14, the processing unit 11 acquires a video sequence containing a plurality of pieces of image data (image data 600 to 607 in fig. 13 (a)) captured in different modes 91 and 92 (step S201). Switching between the different modes 91 and 92 is performed on a frame-by-frame basis. In the present embodiment, this switching is also performed by the capture mode controller 54 (fig. 4 and 6).

In step S201, the processing unit 11 cooperates with the memory 12 and the camera module 20 to be implemented as the acquisition unit 301.

The processing unit 11 determines whether or not fusion based on different modes should be applied based on the change between frames of the continuous image data acquired in step S201 (step S202). In the present embodiment, as in the first embodiment, this determination is also performed by the fusion unit 67 in fig. 4 and 6. In other words, whether to apply fusion is determined based on pixel characteristics in the data image.

In step S202, the processing unit 11 is implemented as the determination unit 302.

Based on the determination result in step S202, the processing unit 11 applies mode-based fusion to the output of the image data acquired in step S201 to use the image data in the corresponding mode (image data 700 to 707 in fig. 13 (B)) (step S203). This outputs a video sequence with high image quality.

Next, a modification of the third embodiment is described.

Fig. 15 illustrates a modification of the aspect in fig. 13, which is one aspect of fusion between modes when image data in the normal mode 92 is continuously input without switching between the modes 91 and 92. One example of the case where switching between the modes 91 and 92 is not performed is the case where the processing unit 11 determines that the foregoing equations (1) to (7) are not satisfied. In this case, for example, image data in the normal mode 92, such as image data 612 to 614, 616 to 617, may be continuously input at the input of fig. 15 (a). Therefore, as shown in fig. 15, in addition to the

image data

612 or 613 in the normal mode 92, the image data 611 in the high resolution mode 91 immediately before the image data 612 is given at the MC 503 or the MC 504, so that the fusion is performed while compensating for the motion between the different modes. At MC 507, image data 615 in the immediately preceding high resolution mode 91 is given in addition to image data 617 in normal mode 92, so that the fusion is performed with compensation for motion between the different modes. The processes of these MCs are similar to those in the respective cells 65 to 67 in fig. 4 and those in the

respective cells

65, 66A and 67 in fig. 6. This reduces the power consumption of the image processing apparatus 10B.

The apparatus-related embodiment and the method-related embodiment are based on the same concept, and therefore the technical advantages brought by the apparatus-related embodiment are the same as those brought by the method-related embodiment. For the specific principles, reference is made to the description of the embodiments related to the apparatus, which is not repeated here.

Although embodiments of the present invention have been mainly described based on image processing, it should be noted that embodiments of the apparatus described herein and other embodiments may also be used for image processing. In general, where picture processing encoding is limited to a single picture 17, only inter prediction units 244 (encoders) and 344 (decoders) may not be available. All other functions of the device (also referred to as tools or techniques) may equally be used for image processing.

The embodiments described herein, e.g., of the apparatus, and functions, may be implemented by hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on a computer-readable medium or transmitted as one or more instructions or code over a communication medium and executed by a hardware-based processing unit. Computer readable media may include tangible media such as computer readable storage media, corresponding data storage media, or corresponding communication media including any medium that facilitates transfer of a computer program from one place to another in accordance with a communication protocol or the like. In this manner, the computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium or (2) a communication medium such as a signal or carrier wave. The data storage medium may be any available medium: the media may be accessed by one or more computers or one or more processors to retrieve instructions, code and/or data structures for implementing the techniques described herein. The computer program product may include a computer-readable medium.

By way of example, and not limitation, such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Further, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website site, server, or other remote source over a coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), or wirelessly, e.g., infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies, e.g., infrared, radio, and microwave, are also included in the defined computer-readable medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but are instead directed to non-transitory tangible storage media. Disk and disc (disc) include Compact Disc (CD), laser disc, optical disc, Digital Versatile Disc (DVD), floppy disk and blu-ray disc where disks usually reproduce data magnetically, while discs usually reproduce data optically with lasers. Combinations of the above may also be included within the scope of computer-readable media.

The instructions may be executed by one or more processors, such as one or more Digital Signal Processors (DSPs), general purpose microprocessors, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term "processor" as used herein may refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. Further, in some aspects, the functionality described herein may be provided in dedicated hardware and/or software modules for encoding and decoding, or included in a combined codec. In addition, these techniques may be fully implemented in one or more circuits or logic elements.

The techniques of this disclosure may be implemented in various devices or apparatuses, including a wireless handset, an Integrated Circuit (IC), or a set of ICs (e.g., a chipset). Various components, modules, or units are described herein to emphasize functional aspects of devices for performing the disclosed techniques, but they do not necessarily need to be implemented by different hardware units. As mentioned above, the units may be combined in a codec hardware unit, or provided by a set of interoperating hardware units comprising one or more processors as described above, together with suitable software and/or firmware.

The disclosure set forth above is illustrative of embodiments only and is not intended to limit the scope of the present invention. It will be understood by those skilled in the art that the above-described embodiments and all or part of the processes for implementing the examples equivalently modified embodiments within the scope of the claims of the invention also belong to the scope of the invention.

Claims

1. A method for processing image data, comprising:

switching the first mode to a second mode in which a second resolution is set in response to a photographing instruction, the second resolution being different from the first resolution;

acquiring second image data captured in the second mode; and

applying a first mode-based fusion to merge the second image data with the first image data as reference image data.

2. The method of claim 1, further comprising:

selecting the first image data and selecting the second image data.

3. The method of claim 2, wherein the selected first image data is the most sharpened image with no or minimal moving objects, and wherein the selected second image data is the most sharpened image with no or minimal moving objects.

4. The method according to any one of claims 1 to 3, further comprising:

determining whether to apply the first mode-based fusion based on imaging conditions;

wherein the step of switching the first mode to the second mode comprises:

switching the first mode to a second mode when it is determined that the fusion based on the first mode is to be applied and the photographing instruction is received.

5. The method of claim 4, wherein the step of determining whether to apply the first mode-based fusion based on imaging conditions comprises:

determining whether to apply the first mode-based fusion based on a change between consecutive frames captured in the first mode before the photographing instruction is received.

6. The method of claim 5, wherein the changes between the consecutive frames comprise all or any changes related to: exposure, focus position, block mean, block deviation, and angular velocity of each axis.

7. The method according to any one of claims 1 to 6, wherein the first image data and the second image data are set to have the same angle of view and the same focus position.

8. The method of any of claims 1 to 7, wherein the first mode-based fusion is combined with a shutter time lag zero time delay, wherein there is no time lag from the receipt of the capture instruction to the completion of the capture using an image captured prior to the receipt of the capture instruction.

9. The method of any one of claims 1 to 8, further comprising:

performing camera shake compensation between frames captured in the first mode and the second mode, wherein the frames include: a first frame captured in the first mode and a second frame captured in the second mode;

10. The method according to any one of claims 1 to 9, characterized in that:

the first mode is a high resolution mode and the second mode is a normal mode; or

The first mode is a normal mode and the second mode is a high resolution mode.

11. The method according to any one of claims 1 to 10, characterized in that the first image data and the second image data are acquired from one image sensor in the imaging unit.

12. The method according to any one of claims 1 to 11, further comprising:

executing a noise reduction process by merging a plurality of pieces of image data in units of image data captured in the first mode and the second mode;

wherein the first image data and the second image data after the noise reduction process are used in the process of applying the first mode-based fusion.

13. A method for processing image data, comprising:

determining, based on a change between image frames of the video sequence, whether to apply a different mode-based fusion to merge the image data captured in the different modes; and

applying the mode-based fusion to the image data of the video sequence based on a result of the determining.

14. An apparatus for processing image data, comprising:

a switching unit configured to switch the first mode to a second mode in which a second resolution is set in response to a photographing instruction, the second resolution being different from the first resolution;

a second acquisition unit configured to acquire second image data captured in the second mode; and

a fusion unit for applying fusion based on a first mode to merge the second image data with the first image data serving as reference image data.

15. The apparatus of claim 14, wherein the fusion unit is further configured to select the first image data and the second image data.

16. The apparatus according to claim 14 or 15, wherein the selected first image data is the most sharpened image with no or minimal moving objects, and the selected second image data is the most sharpened image with no or minimal moving objects.

17. The apparatus of any one of claims 14 to 16, further comprising:

wherein the switching unit is configured to switch the first mode to a second mode when it is determined that the fusion based on the first mode is to be applied and the photographing instruction is received.

18. The apparatus according to claim 17, wherein the determining unit is dedicated to determining whether to apply the first mode-based fusion based on a change between consecutive frames captured in the first mode before the photographing instruction is received.

19. The apparatus of claim 18, wherein the change between the frames comprises all or any of the changes related to: exposure, focus position, block mean, block deviation, and angular velocity of each axis.

20. The apparatus according to any one of claims 14 to 19, wherein the first image data and the second image data are set to have the same angle of view and the same focus position.

21. The apparatus of any of claims 14 to 20, wherein the first mode based fusion is combined with a shutter time lag zero time delay, wherein there is no time lag from receiving the capture instruction to completing the capture using an image captured prior to receiving the capture instruction.

22. The apparatus according to any one of claims 14 to 21, wherein:

the fusion unit includes an alignment unit for performing camera shake compensation between frames captured in the first mode and the second mode, wherein the frames include: a first frame captured in the first mode and a second frame captured in the second mode; and

the fusion unit is configured to apply the fusion to the first image data and the second image data after the camera shake compensation to merge the second image data after the camera shake compensation with the first image data serving as reference image data.

23. The apparatus of any of claims 14 to 22, wherein the first mode is a high resolution mode and the second capture mode is a normal mode; or

The first mode is a normal mode and the second capture mode is a high resolution mode.

24. The apparatus of any of claims 14 to 23, wherein the first image data and the second image data are acquired from the same image sensor.

25. The apparatus of any one of claims 14 to 24, further comprising:

a noise reduction unit configured to perform a noise reduction process by merging a plurality of pieces of image data in units of image data captured in the first mode and the second mode,

wherein the fusion unit is configured to apply the fusion to the first image data and the second image data after the noise reduction process.

26. An apparatus for processing image data captured by an imaging unit, comprising:

a determining unit for determining whether to apply different mode based fusion based on a change between image frames of the video sequence; and

27. An apparatus, comprising:

a camera module;

a non-transitory memory including instructions; and

one or more processors in communication with the camera module and the memory, wherein the one or more processors execute the instructions to:

acquiring second image data captured in the second mode; and

28. The device of claim 27, wherein the one or more processors are further configured to select the first image data and to select the second image data.

29. The device of claim 28, wherein the selected first image data is a most sharpened image with no or minimal moving objects, and wherein the selected second image data is a most sharpened image with no or minimal moving objects.

30. The device according to any one of claims 27 to 29, wherein the one or more processors are further configured to determine whether to apply the first mode-based fusion based on imaging conditions; and

31. The device of claim 30, wherein the one or more processors are configured to:

32. The apparatus of claim 31, wherein the change between the consecutive frames comprises all or any of changes related to: exposure, focus position, block mean, block deviation, and angular velocity of each axis.

33. The apparatus according to any one of claims 27 to 32, wherein the first image data and the second image data are arranged to have the same angle of view and the same focus position.

34. The apparatus of any of claims 27 to 33, wherein the first mode based fusion is combined with a shutter time lag zero time lag, wherein there is no time lag from the receipt of the capture instruction to the completion of the capture using an image captured prior to the receipt of the capture instruction.

35. The device of any one of claims 27 to 34, wherein the one or more processors are further configured to:

36. The apparatus of any one of claims 27 to 35, wherein:

The first mode is a normal mode and the second mode is a high resolution mode.

37. The apparatus according to any one of claims 27 to 36, characterized in that the first image data and the second image data are acquired from one image sensor in the imaging unit.

38. The device of any one of claims 27 to 37, wherein said one or more processors are further configured to: executing a noise reduction process by merging a plurality of pieces of image data in units of image data captured in the first mode and the second mode;

39. The apparatus of any of claims 27 to 38, wherein the camera module comprises an image sensor.

40. A computer-readable storage medium characterized by recording a program for causing a computer to execute the method according to any one of claims 1 to 13.

41. A computer program for causing a computer to perform the method according to any one of claims 1 to 13.