JP7334880B2 - METHOD, APPARATUS, PROGRAM, AND RECORDING MEDIUM FOR PROCESSING IMAGE DATA - Google Patents

Description

TECHNICAL FIELD This disclosure relates to technology related to image processing, and more particularly to methods and apparatus for processing image data, such as image data captured by a camera.

It is known that a personal information terminal such as a smart phone is equipped with a camera for photographing a subject. Generally, a camera has an image sensor and processes image data from the image sensor to display an object.

Embodiments of the present disclosure provide methods, devices, programs, and recording media for outputting high image quality images. To achieve the aforementioned objectives, the following technical solutions are used in the embodiments.

A first aspect of the embodiments provides the following method.

acquiring first image data captured in a first mode, wherein the first mode is set to a first resolution;
A step of switching the first mode to the second mode according to a photographing command (a photographing command by a user, a timer, a program, etc.), wherein the second resolution is set in the second mode, and the second resolution is set in the second mode. a resolution different from the first resolution;
obtaining second image data captured in the second mode;
applying a first mode-based fusion to fuse the second image data with the first image data used as reference image data;
A method of processing image data, such as image data captured by an imaging unit, comprising:
It may be noted that the first mode image data (ie, the first image data captured in the first mode) becomes the reference image data in the first mode based fusion.
After fusing the first mode image data (i.e., the first image data captured in the first mode) and the second mode image data (i.e., the second image data captured in the second mode), Image data is output.
Note that except for the first resolution, the normal mode or normal mode information is also included in the first mode. For example, see the discussion in paragraph [0107].
Note that the imaging unit 20 is described in detail in the description, such as paragraph [0042].
Note that by using first mode-based fusion, the second mode image data is fused with the first mode image data. For example, the first mode-based fusion may correspond to blocks (61-67) of FIG.

According to the first aspect, high image quality image data can be output. In particular, image data between the first and second modes are fused. The modes have different resolutions and images of different resolutions and sensitivities can be acquired in these modes. Therefore, the fused image data can be output as high quality image data.

In a first possible manner of the first aspect, the method may further comprise selecting first image data and selecting second image data. This may allow the first and second image data to be fused.

In a second possible manner of the first aspect according to the first aspect, the first image data selected is the sharpest image with no moving objects or with the fewest moving objects. The second image data selected may be the sharpest image with no moving objects or with the fewest moving objects. This may allow the fusion image data to be the sharpest image.
It can be appreciated that the sharpest image with the fewest moving objects can be selected or determined from the set of image data captured in the first mode. It can be appreciated that the sharpest image with the fewest moving objects can be selected or determined from the set of image data captured in the second mode.

In a third possible manner of the first aspect, according to any one of the first aspect, the first and second possible manners, the method comprises, based on imaging conditions, the first mode-based may further comprise determining whether fusion is applied;
switching the first mode to a second mode comprises switching the first mode to a second mode when it is determined that the first mode-based fusion is to be applied and the capture command is received. include. In one example, equations (1) to (7) in paragraph [0070] correspond to imaging conditions. This may ensure that the fused image data is high image quality image data.

A fourth possible scheme of the first aspect, according to a third possible scheme, wherein determining, based on said imaging conditions, whether said first mode-based fusion is to be applied comprises: determining whether the first mode-based blending is applied based on changes between successive frames captured in the first mode until a capture command is received. This may ensure that the fused image data is high image quality image data.

A fifth possible scheme of the first aspect, according to a fourth possible scheme, wherein said changes between said successive frames are changes in exposure, focus position, block mean, block deviation, and angular velocity of each axis. may include all or any of This may ensure that the fused image data is high image quality image data.

A sixth possible manner of the first aspect, according to the first aspect and any one of the first to fifth possible manners, wherein said first image data and said second image data comprise: They may be set to have the same viewing angle and the same focus position. This may ensure that the fused image data is high image quality image data.

A seventh possible manner of the first aspect, according to the first aspect and any one of the first through sixth possible manners, wherein said first mode-based fusion captures by said shooting command Using the captured image may be combined with zero shutter lag, where there is no time lag from the command to shoot to the completion of shooting. This may ensure that the fused image data is high image quality image data.

In an eighth possible manner of the first aspect, according to the first aspect and any one of the first to seventh possible manners, the method comprises:
performing camera shake compensation between said first mode and said second mode, in particular performing camera shake compensation between frames captured in said first mode and said second mode; the frames may include a first frame captured in a first mode (including the first image data) and a second frame captured in a second mode (including the second image data);
The second image data after the camera shake compensation and the first image data are used in applying the first mode-based fusion. The first image data may be used as reference image data, so the second image data has camera shake relative to the first image data, so camera shake compensation is applied to the second image data. It can be understood that This may ensure that the fused image data is high image quality image data.

A ninth possible manner of the first aspect, according to the first aspect and any one of the first to eighth possible manners, wherein said first mode may be a high resolution mode; 2 mode may be normal mode, or
The first mode may be the normal mode, and the second mode may be the high resolution mode. This may ensure that the fused image data is high image quality image data.

A tenth possible manner of the first aspect, according to the first aspect and any one of the first to ninth possible manners, wherein said first image data and said second image data are: It may be obtained from the same image sensor in the imaging unit. This may ensure that the fused image data is high image quality image data.

In an eleventh possible manner of the first aspect, according to the first aspect and any one of the first to tenth possible manners, the method comprises capturing in the first mode and in the second mode performing a noise reduction process by fusing a plurality of image data on a unit of image data obtained;
The first image data and the second image data after the noise reduction process may be used in applying the first mode-based fusion. This may ensure that the fused image data is high image quality image data.

A second aspect of the embodiments provides the following method.

acquiring a video sequence comprising a plurality of image data captured in different modes, wherein different resolutions are set for the different modes;
determining whether different mode-based blending is to be applied based on changes between successive image frames of a video sequence, in particular based on changes between image frames of a video sequence captured in said different modes; determining whether different mode-based fusions are applied to fuse the image data;
applying the mode-based fusion to the image data of the video sequence based on the result of the determination, e.g., applying the mode-based fusion to fuse image data captured in different modes. applying the mode-based blending to the image data of the video sequence if it is determined that the blending will be performed;
A method of processing image data, such as image data captured by an imaging unit, comprising:

According to the second aspect, high image quality image data can be output continuously. In particular, image data between the first and second modes are fused. The modes have different resolutions and images of different resolutions and sensitivities can be acquired in these modes. Therefore, the fused image data can be output as high quality image data.

A third aspect of the embodiments provides the following apparatus.

a first acquisition unit configured to acquire a plurality of first image data captured in a first mode having a first resolution;
A switching unit configured to switch the first mode to a second mode according to a photographing command (such as a photographing command by a user, a timer, a program, etc.), wherein the second resolution is in the second mode. a switching unit configured, wherein said second resolution is different than said first resolution;
a second acquisition unit configured to acquire second image data captured in said second mode, in particular captured at a second resolution of said second mode;
a fusion unit configured to apply a first mode-based fusion to fuse the second image data with the first image data being reference image data;
An apparatus for processing image data, such as image data captured by an imaging unit, comprising: According to a third aspect, high image quality image data can be output. In particular, image data between the first and second modes are fused. The modes have different resolutions and images of different resolutions and sensitivities can be acquired in these modes. Therefore, the fused image data can be output as high quality image data.

In a first possible manner of the third aspect, the fusion unit may be arranged to select between the first image data and the second image data. This may allow the first and second image data to be fused.

In a second possible manner of the third aspect, according to a first possible manner of the third aspect, the selected first image data has no moving objects or the fewest number of moving objects. The second image data selected may be the sharpest image having no moving objects or the sharpest image having the fewest moving objects. This may allow the fused image data to be the sharpest image.

In a third possible manner of the third aspect, according to any one of the third manner of the third aspect, the first and second possible manners, the apparatus, based on imaging conditions, may further comprise a determining unit configured to determine whether said first mode-based fusion is applied;
The switching unit may be configured to switch the first mode to a second mode when it is determined that the first mode-based fusion is to be applied and the shooting command is received. This may ensure that the fused image data is high image quality image data.

In a fourth possible way of the third aspect, the decision unit specifically determines the first mode based on the changes between consecutive frames captured in the first mode until the shooting command is received. It may be configured to determine whether base fusion is applied. This may ensure that the fused image data is high image quality image data.

A fifth possible scheme of the third aspect, according to a fourth possible scheme of the third aspect, wherein said change between said successive frames comprises exposure, focus position, block mean, block deviation, and each It may include all or any of the changes in the angular velocity of the shaft. This may ensure that the fused image data is high image quality image data.

A sixth possible manner of the third aspect, according to any one of the first to fifth possible manners of the third aspect, wherein said first image data and said second image data are: They may be set to have the same viewing angle and the same focus position. This may ensure that the fused image data is high image quality image data.

A seventh possible manner of the third aspect according to any one of the first to sixth possible manners of the third aspect, wherein said first mode-based fusion captures by said command to shoot. Using the captured image may be combined with zero shutter lag, where there is no time lag from the command to shoot to the completion of shooting. This may ensure that the fused image data is high image quality image data.

An eighth possible manner of the third aspect, according to any one of the first to seventh possible manners of the third aspect, wherein the fusion unit is configured between said first mode and said second mode an alignment unit configured to perform camera shake compensation, in particular camera shake compensation between frames captured in said first mode and said second mode, wherein said frames are in said first mode an alignment unit comprising a captured first frame and a second frame captured in said second mode;
The fusion unit applies fusion to the camera-shake compensated second image data and the first image data, and the camera-shake compensated second image data is used as reference image data. It may be configured to merge with one image data. In particular, each second image data has camera shake between different modes. This may ensure that the fused image data is high image quality image data.

A ninth possible manner of the third aspect according to any one of the first to eighth possible manners of the third aspect, wherein said first mode may be a high resolution mode; The two acquisition modes may be normal mode, or the first mode may be normal mode and the second acquisition mode may be high resolution mode. This may ensure that the fused image data is high image quality image data.

A tenth possible manner of the third aspect, according to any one of the first to ninth possible manners of the third aspect, wherein said first image data and said second image data are: may be obtained from the same image sensor. This may ensure that the fused image data is high image quality image data.

The eleventh possible manner of the third aspect, according to any one of the first to tenth possible manners of the third aspect, wherein the device acquires in the first mode and in the second mode a reduction unit configured to perform a noise reduction process by fusing a plurality of image data in units of image data obtained;
The fusion unit may be arranged to apply fusion to the first image data and the second image data after noise reduction processing.

A fourth aspect of the embodiment provides the following apparatus.

an acquisition unit configured to acquire a video sequence comprising multiple image data captured in different modes, wherein different resolutions are set for the different modes;
A decision unit configured to decide, based on changes between successive frames of a video sequence, whether different mode-based fusion is applied to fuse said image data captured in said different modes. and,
a fusion unit configured to apply different mode-based fusions to the image data of the video sequence based on a result of the determination;
An apparatus for processing image data captured by an imaging unit, comprising:

According to the fourth aspect, high image quality image data can be output continuously. Specifically, the image data between the first and second modes are fused. The modes have different resolutions and images of different resolutions and sensitivities can be acquired in these modes. Therefore, the fused image data can be output as high quality image data.

A fifth aspect of the embodiment provides a program for causing a computer to perform the method according to any one of the first and second aspects of the embodiment and the first to tenth possible schemes of the first aspect. A recorded computer readable storage medium is provided.

A sixth aspect of the embodiment is a computer that causes the computer to perform the method according to any one of the first and second aspects of the embodiment and the first through tenth possible schemes of the first aspect. Offer a program. According to a seventh aspect, the invention relates to a device for processing image data, comprising a camera module, one or more processors and a memory. The memory stores instructions that cause one or more processors to perform the method according to the first or second aspect. The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will become apparent from the specification, drawings, and claims.

To describe the technical solutions in the embodiments more clearly, the following briefly describes the accompanying drawings required to describe the embodiments. It is obvious that the accompanying drawings in the following description merely show some of the possible embodiments and that those skilled in the art can derive other drawings from these accompanying drawings without creative efforts. be.
1 is a schematic diagram showing an example of the configuration of an image processing apparatus according to a first embodiment; FIG. 4 is a diagram showing the correlation between the number of physical pixels of an image sensor and the number of pixels of image data in each mode in the image processing apparatus according to the first embodiment; FIG. 4 is a timing chart showing application of high-resolution mode-based fusion in the image processing apparatus according to the first embodiment; 3 is a diagram showing a detailed configuration example of the image processing apparatus according to the first embodiment in high resolution mode; FIG. 4 is a timing chart showing application of normal mode-based fusion in the image processing apparatus according to the first embodiment; 3 is a diagram showing a detailed configuration example of the image processing apparatus according to the first embodiment in normal mode; FIG. 1 is a diagram illustrating an example of a functional configuration of an image processing apparatus according to a first embodiment; FIG. 4 is a flow chart showing an example of all image processes of the image processing apparatus according to the first embodiment; FIG. 11 is a diagram showing a detailed configuration example of the image processing apparatus according to the second embodiment in high resolution mode; FIG. 10 is a diagram showing a detailed configuration example of the image processing apparatus according to the second embodiment in normal mode; FIG. 10 is a diagram illustrating an outline of fusion implemented by an image processing apparatus according to a second embodiment; FIG. It is a figure explaining the aspect by which the noise was reduced. FIG. 13 is a diagram illustrating an example of the functional configuration of an image processing apparatus according to a third embodiment; FIG. FIG. 11 is a diagram illustrating an example of a video sequence realized by an image processing device according to a third embodiment; FIG. 10 is a flow chart showing an example of an imaging process of an image processing apparatus according to the third embodiment; FIG. 10 is a diagram illustrating an example of a video sequence when mode switching is not performed;

In the following, the technical solutions of the embodiments of the present disclosure are clearly described with reference to the accompanying drawings of the embodiments of the present disclosure. Apparently, the described embodiments are merely a part rather than all of the embodiments of the present disclosure. It should be noted that all other embodiments that can be obtained by a person skilled in the art based on the embodiments of the present disclosure without creative efforts fall within the protection scope of the embodiments of the present disclosure.
[First embodiment]

A description of the image processing device 10 in this exemplary embodiment is provided below. The image processing device 10 is configured to process image data captured by the camera.
[Configuration of image processing apparatus 10]

FIG. 1 is a schematic diagram showing a hardware configuration example of an image processing apparatus 10 according to the first embodiment.

As shown in FIG. 1, the image processing apparatus 10 includes a processing unit 11, a memory 12, an input device 13, a display device 14, a camera module (imaging unit) 20, and a gyroscope 30. The camera module 20 has an image sensor 21 . In this embodiment, the image processing apparatus 10 includes, for example, a tablet computer, phablet or mobile phone, mobile phone, personal information terminal, personal computer, robot, or the like.

A processing unit 11 is connected to the individual components by buses to perform the process of moving control signals and data. The processing unit 11 may execute various programs, arithmetic operations, timing control, etc. to implement the overall operation of the image processing device 10 . The program may be stored on a computer readable storage medium such as DVD (Digital Versatile Disc)-ROM (Read Only Memory), CD (Compact Disc)-ROM (Read Only Memory). The program may be stored on a removable and readable computer storage medium such as a secure digital card (SD card). The processing unit 11 is a processing device such as a CPU (Central Processing Unit), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), or FPGA (Field Programmable Gate Array). The memory 12 is a storage device such as ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), or flash memory.

Programs for the operating system and various data required for general operational control of the image processing apparatus 10 are stored in the memory 12 . A memory area for temporarily storing data and programs is provided in memory 12 for holding programs and data, as well as other data necessary for processing in device 10 .

The input device 13 includes operation buttons, a touch panel, an input pen, and sensors.

The display device 14 may be a flat display such as a liquid crystal display or an EL (electroluminescence) display.

An image sensor 21 included in the camera module 20 acquires image data of an object. As will be described later, the image sensor 21 in the image processing apparatus 10 of this embodiment has a high resolution mode 91 and a normal mode 92. FIG. In high resolution mode 91, which uses constant resolution, high resolution image data is acquired. In normal mode 92, which uses a resolution different from that of high resolution mode 91, the resolution of the image data is lower than that in high resolution mode, but the sensitivity and SNR (signal-to-noise ratio) of the image data are higher than those in high resolution mode. higher than the sensitivity and SNR (signal-to-noise ratio) of the image data at The modes are not limited to these two modes 91 and 92, HDR (High Dynamic Range Rendering) mode may also be applied.

In this embodiment, the image sensor 21 may, for example, have a CMOS (Complementary Metal Oxide Semiconductor) configuration, but may have other configurations as long as image data of the object can be acquired at different resolutions. good. The camera module 20 may be mounted outside the image processing device 10 .

The gyroscope 30 is configured to detect angular velocity to detect shake of the camera (ie camera module 20).

FIG. 2 is a diagram showing the correlation between the number of physical pixels of the image sensor 21 and the number of pixels of image data in each mode.

The number of pixels of the image data in the high resolution mode 91 of FIG. 2B is the same as the number of pixels of the image sensor 21 of FIG. 2A. In this case, one pixel in (B) of FIG. 2 is output as one pixel. The array of pixels shown in FIG. 2B is an example of a Bayer array.

On the other hand, the number of pixels of the image data in the normal mode 92 of FIG. 2C is smaller than the number of pixels of the image sensor 21 of FIG. 2A. For example, the 2×2 pixels shown in (C) of FIG. 2 are output as one pixel.

Image data from image sensor 21 is periodically output to processing unit 11 and memory 12 in the form of frames.
[Outline of image processing in high resolution mode]

Next, an overview of image processing in the high resolution mode 91 implemented by the image processing apparatus 10 will be described with reference to FIGS. 1 to 3. FIG. FIG. 3 is a timing diagram illustrating the application of high resolution mode-based fusion.

As shown in FIG. 3, in the image processing apparatus 10, the image sensor 21 receives a plurality of image data H(n−x) to H(n) (where n and x are positive) corresponding to the high resolution mode 91. ) are obtained sequentially.

As shown in FIG. 3, for example, at t1, an instruction to capture image data is received, after which processing unit 11 switches high resolution mode 91 to normal mode 92 at t2. A possible example of a shooting command is, for example, performing a touch operation on a shutter button displayed on the display device 14 .

A plurality of image data N(n+1) to N(n+5) in the normal mode 92 are sequentially acquired from the image sensor 21 .

In FIG. 3, the duration of t0 to t1 is the zero shutter lag imaging period T, during which the image data H(n−x) to H(n) described above in the high resolution mode 91 are captured by the display device. 14 may be displayed in preview. Zero shutter lag means that there is no time lag from the shooting command to the completion of shooting the object.

High-resolution mode-based fusion is then applied to image data N(n+1) in normal mode 92 to fuse image data N(n+1) with image data H(n−x), denoted by symbol P. . Therefore, the fused image data is output as the image data for which the photographing command has been received.

In the above fusion, based on the characteristics (brightness value, etc.) of pixels in the two image data N(n+1) and H(n−x), the image processing device 10 selects appropriate one of the two image data. Select pixels and output image data.

For example, a pixel in the image data H(nx) in high resolution mode 91 is selected if the pixel feature reflects an edge. On the other hand, pixels in the image data N(n+1) in normal mode 92 are selected if the characteristics of the pixels reflect a background with relatively little variation. As a result, the output image data contains pixels in the appropriate mode according to the pixel's characteristics. Therefore, high image quality image data can be output.

The relationship of application of fusion between modes 91 and 92 is not limited to that illustrated in FIG. High image quality image data uses image data with less shake effect (e.g., image data with less difference in equations (3) and (4) given below) for image data between modes 91 and 92 in fusion. may be output by Therefore, for example, the image data that is not the image data H(n−x) in the high resolution mode 91 and the image data N(n+1) may be fused. Alternatively, any one of the image data H(n−x) to H(n) in the high resolution mode 91 and any one of the image data N(n+1) to N(n+5) may be fused. .
[Detailed Configuration Example of Image Processing Apparatus in High Resolution Mode]

FIG. 4 is a diagram showing a detailed configuration example of the image processing apparatus 10 in the high resolution mode.

Referring to FIG. 4, the image processing device 10 comprises a memory 12, a camera module 20, a gyroscope 30, a first processing unit 50 and a second processing unit 60. FIG.

First, the components in the first processing unit 50 related to switching between modes 91 and 92 will be described. The first processing unit 50 comprises a 3A unit 51 , a moving object detector 52 , a camera shake detector 53 and a capture mode controller 54 . 3A refers to auto exposure (AE), auto focus (AF), and auto white balance (AWB) as preprocessing.

Image data output from the image sensor 21 in the camera module 20 is sequentially output to the 3A unit 51 and the moving object detector 52 frame by frame.

As will be explained later, the 3A unit 51 calculates the exposure difference and the focus position difference between two consecutive frames. The moving object detector 52, for example, calculates the mean and deviation values of brightness for each block (e.g., 8x8 pixels) to detect object motion.

The individual velocities in the three axes (x, y, z) between two consecutive frames are input from gyroscope 30 to camera shake detector 53 .

The values of the individual units 51-53 are sequentially input to the acquisition mode controller 54 and stored. Acquisition mode controller 54 then determines whether the first mode-based fusion can be applied. The conditions given by equations (1)-(7) below are used in this determination.

Exposure difference ≤ threshold A (1)
In-focus position difference ≤ threshold B (2)
Difference in average value ≤ threshold C (3)
Deviation value difference ≤ threshold value D (4)
x-axis acceleration difference≦threshold E (5)
y-axis acceleration difference≦threshold F (6)
z-axis acceleration difference≦threshold G (7)

The difference in equations (1) and (2) corresponds to the value calculated by the 3A unit 51. The difference in equations (3) and (4) corresponds to the value calculated by moving object detector 52 . The differences in equations (5)-(7) represent the values obtained from the camera shake detector 53.

Each of the differences in equations (1) to (7) is the change between successive frames when the image data (image data H(n−x) to H(n) in FIG. 3) were captured in the high resolution mode 91. indicates

Capture mode controller 54 determines to apply high resolution mode-based fusion as described above with reference to FIG. 3 when all conditions in equations (1)-(7) are met. Conditions for determination are not limited to those shown in formulas (1) to (7), and formulas (1) to (7) can be combined arbitrarily.

Satisfying the conditions in equations (1)-(7) means that the image data exhibits relatively little variation (scene, brightness, object motion). For example, if the image data belongs to a landscape scene or the like, the image data may be image data with small variation (eg, small individual differences in equations (1) to (7)) as described above.

On the other hand, if any one of the conditions in equations (1)-(7) is not met, this means that the image data exhibits large fluctuations. For example, if the image data belongs to a scene in which an object in the image data moves, the image data is image data with large fluctuations (eg, large individual differences in equations (1) to (7)) as described above. good. In this case, the application of fusion is not determined and the image data from the image sensor 21 may be output without applying the fusion described below.

When the photographing command (time t1 in FIG. 3) is received, the capture mode controller 54 switches the high-resolution mode 91 to the normal mode 92 (at the timing t2 in FIG. 3) to obtain the image data in the normal mode 92 (time t1 in FIG. 3). 3 image data N(n+1) to N(n+5)) are obtained. In this case, the capture mode controller 54 controls the camera module 20 in accordance with communication protocols such as MIPI (mobile industrial processor interface) and I2C (inter-integrated circuit communication) to perform exposure control, focus control, mode configuration, etc. communicate with

In the capture mode controller 54, each image data between modes 91 and 92 is set to have the same viewing angle and focus position.

Next, the second processing unit 60 will be described. The second processing unit 60 comprises image processing units 61 , 63 , resizing units 62 , 64 , motion calculation unit 65 , image alignment unit 66 and fusion unit 67 .

In this embodiment, image data in high resolution mode 91 are provided to image processing unit 61 and resizing unit 62 . Image processing unit 61 converts the raw image data from image sensor 21 into RGB (red, green, and blue) image data. In this case, the image data is read from memory 12 . The image data may be converted into image data comprising, for example, a luminance signal (Y), a blue component differential signal (U), and a red component differential signal (V).

The resizing unit 62 resizes the image data in the high resolution mode 91 so that the size (number of pixels) of the image data in the high resolution mode 91 is the same as the size (number of pixels) of the image data in the normal mode 92 in the normal mode 92 . is configured to downscale the This is because the number of pixels in the output of the image sensor 21 differs between modes 91 and 92, as shown in FIG.

Image sensor 21 provides image data in normal mode 92 to image processing unit 63 and resizing unit 64 . Image processing unit 63 converts the raw image data from image sensor 21 into YUV or RGB image data. The resizing unit 64 is configured to upscale the image data in the normal mode 92 in the high resolution mode 91 such that the size of the image data in the normal mode 92 is the same as the size of the image data in the high resolution mode 91 . ing.

Resize units 62 and 64 provide image data to motion calculation unit 65 which also calculates motion compensation data to compensate for camera shake in the image data between modes 91 and 92 . Motion calculation unit 65 performs alignment, which allows the image data in normal mode 92 to be aligned with the image data in high resolution mode 91 according to the results of the calculations in motion calculation unit 65 . In the example of FIG. 3, image data N(n+1) is aligned with image data H(n−x) selected as a reference frame.

Image data from resize unit 62 and image data from image alignment unit 66 are provided to fusion unit 67 . The two image data are fused and output by applying high resolution mode-based fusion. In the example of FIG. 3, image data H(n−x) indicated by symbol P is selected as a reference frame, merged with N(n+1), and output. For example, in FIG. 3, if the image data after fusion is a landscape scene including a building, the pixels of the image data H(nx) are the edge (here, the above-mentioned fluctuation are used for the background (where the variation is small). Therefore, high image quality scenic scenes can be captured.

If no fusion is applied, the image data from image sensor 21 may be output via image processing unit 61 .
[Outline of image processing in normal mode]

Next, an outline of image processing in the normal mode 92 will be described with reference to FIGS. 1, 2 and 5. FIG.

FIG. 5 is a timing diagram similar to that of FIG. 3 showing the application of normal mode-based fusion. In the example of FIG. 5, image data N(n−x) to N(n) (where n and x are positive integers) in normal mode 92 are sequentially obtained. After that, after the photographing command at t1, at timing t2, the processing unit 11 switches the normal mode 92 to the high resolution mode 91, and sequentially processes the image data H(n+1) to H(n+5) in the high resolution mode 91. get.

In the example of FIG. 5, image data N(n−x), denoted by symbol P, is selected as a reference frame and fused with H(n+1). This fusion is performed in a manner similar to that of the fusion shown in FIG. That is, the image processing device 10 selects a pixel in the appropriate one of the two image data based on the pixel characteristics (brightness value, etc.) in the image data N(n+1) and H(n−x). , to set the output image data. For example, a pixel in the image data H(nx) in high resolution mode 91 is selected if the pixel feature reflects an edge. On the other hand, pixels in the image data N(n+1) in normal mode 92 are selected if the characteristics of the pixels reflect a background with relatively little variation. As a result, the output image data contains pixels in the appropriate mode according to the pixel's characteristics. Therefore, high image quality image data is output.

3, in FIG. 5, the duration of t0 to t1 is the zero shutter lag imaging period T, during which the above-mentioned image data N(n−x) to N(n) may be displayed in preview on display device 14 .

The relationship of application of fusion between modes 91 and 92 is not limited to that illustrated in FIG. 5, and various aspects may be implemented. Any one of the image data N(n−x) to N(n) in the normal mode 92 and any one of the image data H(n+1) to H(n+5) in the high resolution mode 92 may be fused.
[Detailed Configuration Example of Image Processing Apparatus in Normal Mode]

FIG. 6 shows a detailed configuration example of the image processing apparatus 10 in the normal mode 92. As shown in FIG. The configuration of FIG. 6 differs from the configuration of FIG. 4 in that the normal mode 92 is switched to the high resolution mode 91, so the following description focuses on this point.

Referring to FIG. 6, acquisition mode controller 54 of first processing unit 50 performs normal mode-based fusion as described above with reference to FIG. Decide on application.

Upon receiving a photographing command (time t1 in FIG. 5), the capture mode controller 54 switches the normal mode 92 to the high resolution mode 91 (time t2 in FIG. 5), and the image data in the high resolution mode 91 (H in FIG. 5). (n+1) to H(n+5)) are obtained.

The image alignment unit 66A in the second processing unit 60 aligns the image data in the high resolution mode 91 with the image data in the normal mode 92 according to the result of the calculation in the motion calculation unit 65. FIG. Motion calculation unit 65 calculates the motion of the image data in high resolution mode 91 relative to the image data in normal mode 92 . In the example of FIG. 5, image data H(n+1) is aligned with image data N(n−x) selected as the reference frame.

Image data from resize unit 64 and image data from image alignment unit 66 A are provided to fusion unit 67 . In fusion unit 67, the two image data are fused and output by application of high resolution mode-based fusion. In the example of FIG. 5, the image data N(n−x) and H(n+1) are fused and output. Therefore, as in the case of FIG. 3, a high image quality landscape scene can be captured.

If no fusion is applied, the image data from image sensor 21 may be output via image processing unit 63 .
[Functional Configuration of Image Processing Apparatus 10]

FIG. 7 is a diagram showing the functional configuration of the image processing apparatus 10. As shown in FIG. This functional configuration will be described below with reference to this figure. As shown in FIG. 7, the image processing apparatus 10 includes a first acquisition unit 101, a determination unit 102, a switching unit 103, a second acquisition unit 104, a fusion unit 105, and a display control unit 106. Prepare. In an exemplary implementation, these components are realized by processing unit 11 and memory 12 of FIG. 1 and are organized as follows.

The first acquisition unit 101 is configured to acquire a plurality of first image data captured in a first mode (high resolution mode 91 or normal mode 92) having a preset resolution. The first acquisition unit 101 corresponds to the memory 12 and the 3A unit 51 and detector 52 in the first processing unit 50 in FIGS.

The decision unit 102 applies mode-based fusion based on the imaging conditions (equations (1)-(7) above in this embodiment) if the image data was captured in the first mode in response to the capture command. is configured to determine The decision unit 102 corresponds to the detector 53 and acquisition mode controller 54 in the first processing unit 50 in FIGS.

The switching unit 103 is configured to switch the first mode to the second mode (normal mode 92 or high resolution mode 91) when a photographing command is received. Switching unit 103 corresponds to acquisition mode controller 54 in FIGS.

The second acquisition unit 104 is configured to acquire second image data captured at the second mode of resolution. A second acquisition unit 104 corresponds to the memory 12 in FIGS.

Fusion unit 105 applies the first mode-based fusion to the second image data to output the second image data and uses the first image data. The fusion unit 105 corresponds to the second processing unit 60 in FIGS. 4 and 6. FIG.

Alignment unit 1051 in fusion unit 105 is configured to align the motion of the image data between the first mode and the second mode. Alignment unit 1051 corresponds to motion calculation unit 65 and image alignment units 66, 66A in second processing unit 60 in FIGS.

The display control unit 106 is configured to display a preview of the first image data before issuing the command to shoot.

The aforementioned components are referred to as necessary in describing the operation of image processing apparatus 10 .
[Operation of image processing apparatus 10]

General image processing of the image processing apparatus 10 will be described below with reference to FIGS. 1 to 8. FIG. In the image processing apparatus 10 of this embodiment, the processing unit 11 can execute various processes described later according to the program.

FIG. 8 is a flow chart showing an example of general image processing of the image processing apparatus 10. As shown in FIG.

In FIG. 8, the processing unit 11 sequentially acquires image data in the first mode from the image sensor 21 (step S101). In the example of FIG. 3, the image data H(n−x) to H(n) in the high resolution mode 91, which is the first mode, are acquired. In the example of FIG. 5, the image data N(n−x) to N(n) in the normal mode 92, which is the second mode, are acquired. These image data are sequentially stored in the memory 12 .

At this stage the processing unit 11 cooperates with the memory 12 and the camera module 20 to be implemented as a first acquisition unit 101 .

When the image data in step S101 is captured, the processing unit 11 performs first mode-based imaging according to a shooting command (for example, a zero shutter lag shooting command) based on the imaging conditions (formulas (1) to (7)). The application of fusion is determined (step S102). In this case, processing unit 11 determines whether the conditions of equations (1)-(7) are met. If the condition is met, it is decided to apply the first mode-based fusion, and if the condition is not met, it is not decided to apply the first mode-based fusion.

At step S 102 , the processing unit 11 cooperates with the gyroscope 30 as implemented as a decision unit 102 .

When the shooting command is received (timing t1 in FIGS. 3 and 5), the processing unit 11 switches the first mode to the second mode (step S103). In the example of FIG. 3, the high resolution mode 91 is switched to the normal mode 92 at t2. In the example of FIG. 5, the normal mode 92 is switched to the high resolution mode 91 at t2.

At step S<b>103 , the processing unit 11 is implemented as a switching unit 103 .

The processing unit 11 obtains image data captured at the second mode resolution from the image sensor 21 (step S104). In the example of FIG. 3, the image data N(n+1) to N(n+5) in the normal mode 92, which is the second mode, are acquired. In the example of FIG. 5, the image data N(n+1) to N(n+5) in the normal mode 92, which is the second mode, are acquired. These image data are sequentially stored in the memory 12 .

At step S 104 , the processing unit 11 cooperates with the memory 12 so as to be implemented as a second acquisition unit 104 .

On outputting the image data acquired in step S104, processing unit 11 applies a first mode-based fusion to the first image data acquired in step S101 ( step S105). In this case, the processing unit 11 selects the pixels in the image data in the appropriate mode among the two image data according to the characteristics of the pixels in the target image data, and outputs the image data for which the shooting command is received. you can In the example of FIG. 3, the processing unit 11 selects the image data H(n−x) from the image data H(n−x) to H(n) in the high resolution mode 91, and selects the image data H(n−x) in the normal mode 92. Select image data N(n+1) from among N(n) to N(n+5), and further fuse image data H(n−x) in high resolution mode 91 and image data N(n+1) in normal mode 92 and output the fusion image data. In the example of FIG. 5, the image data N(n−x) in the normal mode 92 and the image data H(n+1) in the high resolution mode 91 are fused and output. The selected image data for each of the modes may be the sharpest image with no moving objects or the fewest moving objects.

Referring to the flowchart of FIG. 8, the image data acquired in step S101 is different in size from the image data acquired in step S104. In step S105, the processing unit 11 resizes the target image data (image data H(n−x) in FIG. 3) or the target image data (image data H(n+1) in FIG. 5) to compensate for camera shake. The two image data are aligned as much as possible.

At step S<b>105 , the processing unit 11 is implemented as a fusion unit 105 .

In FIG. 8, the processing unit 11 (display control unit 106) may display a preview of the image data acquired in step S101 on the display device 14 before commanding the photographing in step S103. Therefore, the image data acquired in step S101 may be visible, allowing the shutter to be pressed at the appropriate time.

As explained above, the image processing apparatus 10 of this embodiment is configured to fuse image data in different modes 91 and 92 with respect to outputting image data. High resolution mode 91 allows high resolution image data to be captured. On the other hand, the normal mode 92 has a lower resolution than the high resolution mode 91, but can capture high sensitivity and high SNR image data. As a result, the image data fused in step S105 may consist of image data captured in the appropriate mode and thus have higher image quality.

Image data in individual modes 91 and 92 are acquired from one image sensor 21 . As a result, the number of required image sensors is reduced, and the image processing apparatus 10 can be made smaller.
[Second embodiment]

An image processing apparatus 10A according to the second embodiment will be described with reference to FIGS. 9A and 9B. FIG. 9A is a diagram illustrating an overview of high-resolution mode-based fusion in the image processing apparatus 10A. FIG. 9B is a diagram illustrating an overview of normal mode-based fusion in the image processing apparatus 10A. Symbols and the like used in the description of the first embodiment are also used in the following description of the second embodiment, unless otherwise specified.

9A and 9B, the image processing device 10A further comprises noise reduction units 68 and 69 unlike the image processing device 10 of the first embodiment (FIGS. 4 and 6). Although the noise reduction units 68 and 69 shown in Figures 9A and 9B are shown in solid and dashed lines, the units 68 and 69 may be provided in the locations shown in solid or dashed lines.

The noise reduction unit 68 is configured to perform noise reduction processing on each image data in the high resolution mode 91 in units of multiple frames. The noise reduction unit 69 is configured to perform noise reduction processing on each image data in the normal mode 92 in units of multiple frames.

The noise reduction process is described in detail with reference to FIGS. 9A, 9B and 10. FIG.

FIG. 10 illustrates how noise is reduced by noise reduction units 68 and 69 . In FIG. 10, four consecutive frames f0-f3 correspond to image data in normal mode 92, for example, and four consecutive frames f4-f7 correspond to image data in high resolution mode 91, for example.

10, in noise reduction unit 68, four consecutive frames f4-f7 are combined into one frame F0 in the MFNR 201 process. This provides a noise-canceled frame F0 in high resolution mode 91 . The number of samples in a frame is not limited to the four illustrated in FIG. 10 and may vary.

In the process of digital binning 202, the four frames f4-f7 are changed to frames f41-f44 to adjust the size between modes 91 and 92 to match the size of frames f0-f3 in normal mode 92. .

Frames f41-f44 are obtained by resizing frames f4-F7, which can reduce noise. This is because the generation of noise components can be affected by resolution.

FIG. 11 illustrates an example process for reducing noise in image data in high resolution mode 91 that includes noise components 82a, 82b, 82c. Here, the mode of the image data is converted to normal mode 92 in order to cancel the noise components 82a, 82b, 82c. The luminance change 81 in the image data in the high resolution mode 91 of FIG. 11(A) contains noise components 82a to 82c. Since the generation of these noise components can be affected by the magnitude of resolution, lowering the resolution can reduce the noise components. The noise components 82a-82c can be canceled out in the luminance change 83 in the image data in the normal mode 92 of FIG. 11(B). The noise-reduced frames f41-f44 in FIG. 10 are thus obtained.

In FIG. 10, in the noise reduction unit 69, eight frames f0-f4, f41-f44 are combined into one frame F1 in the MFNR 203 process. This provides a noise-cancelled frame F1 in normal mode 92 .

Similar to the fusion according to the first embodiment, in the process used at 204 by fusion unit 67, two frames F0 and F1 are also fused to output a single frame F2. Therefore, high image quality image data can be output.
[Third embodiment]

An image processing apparatus 10B according to the third embodiment will be described. The configuration of the image processing apparatus 10B is substantially the same as the configurations shown in FIGS. 1, 4 and 6. FIG. Symbols and the like used in the description of the first embodiment are also used in the following description of the second embodiment, unless otherwise specified.

The image processing device 10B handles a video sequence including multiple image data. This can be implemented, for example, by having the processing unit 11 acquire image data in different modes 91 and 92 and fuse the multiple image data based on frame-to-frame variations in the image data.

FIG. 12 shows the functional configuration of the image processing device 10b.

Referring to FIG. 12, the image processing device 10B comprises an acquisition unit 301, a determination unit 302, and a fusion unit 303. As shown in FIG. Acquisition unit 301 corresponds to first processing unit 50 and memory 12 in FIGS. Decision unit 302 corresponds to acquisition mode controller 54 of FIGS. The fusion unit 303 corresponds to the second processing unit 60 of FIGS. 4 and 6. FIG. These components are referred to as necessary in the description of operation given later with reference to FIG.

FIG. 13 shows how the image data in the video sequence are fused in the image processing device 10B. For example, in the input of FIG. 13A, the image data 600, 602, 604, 606 in the high resolution mode 91 and the image data 601, 603, 605, 607 in the high resolution mode 91 are alternately supplied. In each of MC (Motion Compensation) 501-507, fusion is performed on two image data in different modes while motion between modes is compensated. The process of each MC is similar to the process in each of units 65-67 of FIG. 4 or each of units 65, 66A and 67 of FIG.

In the output of FIG. 13B, fusion image data 700 to 707 are output. The result is a high picture quality video sequence output.

FIG. 14 is a flow chart showing an example of general image processing of the image processing apparatus 10B.

Referring to FIG. 14, processing unit 11 acquires a video sequence including a plurality of image data (image data 600-607 in FIG. 13A) captured in different modes 91 and 92 (step S201). Switching between different modes 91 and 92 is performed every frame. This switching is also performed by acquisition mode controller 54 (FIGS. 4 and 6) in this embodiment.

At step S<b>201 , the processing unit 11 cooperates with the memory 12 and the camera module 20 as implemented as an acquisition unit 301 .

Processing unit 11 determines whether different mode-based blending should be applied based on the frame-to-frame variation of the sequential image data obtained in step S201 (step S202). In this embodiment, this determination is also performed by the fusion unit 67 of FIGS. 4 and 6 as in the first embodiment. That is, the application of fusion is determined based on pixel characteristics within the image data.

At step S<b>202 , the processing unit 11 is implemented as a decision unit 302 .

Based on the result of the determination in step S202, the processing unit 11 converts the image data acquired in step S201 to use the image data in the corresponding mode (image data 700-707 in (B) of FIG. 13). (step S203). The result is a high picture quality video sequence output.

Next, a modification of the third embodiment will be described.

FIG. 15 illustrates a modification of the embodiment of FIG. 13 that blends between modes when image data in normal mode 92 is continuously input without switching between modes 91 and 92. It is an aspect of An example of when switching between modes 91 and 92 is not performed is when processing unit 11 determines that equations (1)-(7) above are not satisfied. In this case, for example, in the input of FIG. 15A, the image data in the normal mode 92 such as the image data 612-614, 616-617 can be input continuously. As a result, as shown in FIG. 15, in addition to image data 612 or 613 in normal mode 92, image data 611 in high resolution mode 91 immediately preceding image data 612 is also provided in MC 503 or 504, thereby allowing different modes Fusion is performed while motion between is compensated. In MC 507, in addition to the image data 617 in normal mode 92, the previous image data 615 in high resolution mode 91 is also provided to perform the blending while compensating for motion between different modes. The processes of these MCs are similar to the processes in individual units 65-67 of FIG. 4 and individual units 65, 66A and 67 of FIG. As a result, the power consumption of the image processing device 10B is reduced.

Apparatus-related embodiments and method-related embodiments are based on the same idea, and the technical advantages provided by the apparatus-related embodiments are the same as the technical advantages provided by the method-related embodiments. For specific principles, please refer to the description of the device-related embodiments, and the details are not described here. While embodiments of the present invention have been described primarily based on image processing, it should be noted that the apparatus embodiments and other embodiments described herein may be configured for image processing. Please note. In general, if picture processing coding is limited to a single picture 17, only inter prediction units 244 (encoder) and 344 (decoder) may not be available. All other functions of the device (also called tools or techniques) can equally be used for image processing. For example, embodiments of the apparatus and functionality described herein may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or code on a computer-readable medium or transmitted over a communication medium and executed by a hardware-based processing unit. Computer-readable storage medium corresponds to a tangible medium such as a data storage medium or any medium that facilitates transfer of a computer program from one place to another, for example, according to a communication protocol. It may include a communication medium comprising: As such, computer-readable media generally may correspond to (1) tangible computer-readable storage media that is non-volatile or (2) a communication medium such as a signal or carrier wave. Data storage media can be any accessible by one or more computers or one or more processors for obtaining instructions, code and/or data structures for implementation of the techniques described in this disclosure. It can be any available medium. A computer program product may include a computer-readable medium. By way of example, and not limitation, such computer readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory or It can include any other computer-accessible medium that can be used to store desired program code in the form of data structures. Also, any connection is properly termed a computer-readable medium. For example, instructions may be transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair Digital Subscriber Line (DSL), or wireless technologies such as infrared, radio, and microwave. coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media refer to non-volatile, tangible storage media that do not include connections, carrier waves, signals, or other transitory media. As used herein, disk and disc includes compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy discs, and Blu-ray discs, and is commonly referred to as a disk is for reproducing data magnetically, and a disc is for optically reproducing data using a laser. Combinations of the above should also be included within the scope of computer-readable media. Instructions are implemented in one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. It may be executed by one or more processors. Accordingly, the term "processor" as used herein may refer to any of the structures described above, or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects the functionality described herein is provided in dedicated hardware and/or software modules configured for encoding and decoding, or combined codec may be incorporated into Also, the techniques can be fully implemented in one or more circuits or logic elements. The techniques of this disclosure may be implemented in a wide variety of devices or apparatus, including wireless handsets, integrated circuits (ICs) or sets of ICs (eg, chipsets). Although various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, implementation by different hardware units may be not necessarily required. Rather, the various units are combined into a codec hardware unit, in conjunction with suitable software and/or firmware, as described above, or interoperable, including one or more processors, as described above. It may be provided by a collection of hardware units.

The disclosures given above are only example embodiments and are not intended to limit the protection scope of the present invention. A person skilled in the art will understand that all or part of the processes implementing the above-described embodiments and equivalent modifications made within the scope of the claims of the present invention are also included within the scope of the present invention. will understand.
(Item 1) acquiring first image data captured in a first mode, wherein a first resolution is set in the first mode;
switching the first mode to a second mode in response to a photographing command, wherein a second resolution is set in the second mode, the second resolution being different from the first resolution; and,
obtaining second image data captured in the second mode;
applying a first mode-based fusion to fuse the second image data with the first image data, which is reference image data;
A method of processing image data, comprising:
(Item 2) The method according to item 1, wherein the method further includes selecting the first image data and selecting the second image data.
(Item 3) The first image data to be selected is the sharpest image having no moving objects or the fewest number of moving objects, and the second image data to be selected has moving objects. 3. The method of item 2, which is the sharpest image with no or the fewest moving objects.
Item 4: the method further includes determining whether the first mode-based fusion is to be applied based on imaging conditions;
switching the first mode to a second mode comprises switching the first mode to a second mode when it is determined that the first mode-based fusion is to be applied and the capture command is received. The method of any of items 1-3, comprising:
(Item 5) Determining whether the first mode-based fusion is to be applied based on the imaging conditions includes: 5. The method of item 4, comprising determining whether the first mode-based fusion is applied based on the change.
6. The method of claim 5, wherein the changes between successive frames include all or any of changes in exposure, focus position, block mean, block deviation, and angular velocity of each axis.
(Item 7) The method according to any one of items 1 to 6, wherein the first image data and the second image data are set to have the same viewing angle and the same focus position.
(Item 8) The first mode-based fusion is combined with zero shutter lag from the capture command to completion of capture using the image captured until the capture command, items 1-7. A method according to any one of
(Item 9) performing camera shake compensation between frames captured in the first mode and the second mode, wherein the frames are the first frame captured in the first mode and the first frame captured in the first mode; a second frame captured in two modes;
any one of items 1 to 8, wherein in applying the first mode-based fusion, the second image data after the camera shake compensation and the first image data are used. described method.
(Item 10)
the first mode is a high resolution mode and the second mode is a normal mode; or
A method according to any one of items 1 to 9, wherein said first mode is normal mode and said second mode is high resolution mode.
(Item 11)
11. The method according to any one of items 1 to 10, wherein said first image data and said second image data are obtained from one image sensor in said imaging unit.
(Item 12)
The method further comprises performing noise reduction processing by fusing a plurality of image data in units of image data captured in the first mode and the second mode;
12. The method of any one of items 1-11, wherein the step of applying the first mode-based fusion uses the first image data and the second image data after the noise reduction process. Method.
(Item 13)
acquiring a video sequence comprising a plurality of image data captured in different modes, wherein different resolutions are set for the different modes;
determining whether different mode-based fusion is applied to fuse the image data captured in the different modes based on changes between image frames of a video sequence;
applying the mode-based blending to the image data of the video sequence based on the results of the determination;
A method of processing image data, comprising:
(Item 14)
a first acquisition unit configured to acquire a plurality of first image data captured in a first mode having a first resolution;
A switching unit configured to switch the first mode to a second mode in response to a photographing command, wherein a second resolution is set in the second mode, and the second resolution is the second resolution. a switching unit different from one resolution;
a second acquisition unit configured to acquire second image data captured in the second mode;
a fusion unit configured to apply a first mode-based fusion to fuse the second image data with the first image data used as reference image data;
An apparatus for processing image data, comprising:
(Item 15)
15. Apparatus according to item 14, wherein the fusion unit is further configured to select between the first image data and the second image data.
(Item 16)
The selected first image data is the sharpest image without or with the fewest moving objects, and the second image data is selected with no or the fewest moving objects. 16. Apparatus according to item 14 or 15, wherein the sharpest image with a moving object of .
(Item 17)
further comprising a determining unit configured to determine, based on imaging conditions, whether said first mode-based fusion is to be applied;
wherein the switching unit is configured to switch the first mode to a second mode when it is determined that the first mode-based fusion is to be applied and the shooting command is received, items 14- 17. The apparatus according to any one of 16.
(Item 18)
The determining unit determines whether said first mode-based fusion is applied, specifically based on changes between consecutive frames captured in a first mode until said shooting command is received. 18. Apparatus according to item 17, configured to:
(Item 19)
19. Apparatus according to item 18, wherein the changes between the frames include changes in exposure, focus position, block mean, block deviation, and/or angular velocity of each axis.
(Item 20)
20. The apparatus of any one of items 14-19, wherein the first image data and the second image data are set to have the same viewing angle and the same focus position.
(Item 21)
21. Any one of items 14-20, wherein the first mode-based fusion is combined with zero shutter lag from the capture command to completion of capture using the image captured up to the capture command. The apparatus described in 1.
(Item 22)
The fusion unit is an alignment unit configured to perform camera shake compensation between frames captured in the first mode and the second mode, wherein the frames are captured in the first mode. an alignment unit comprising one frame and a second frame captured in said second mode;
The fusion unit applies fusion to the camera-shake compensated second image data and the first image data, and the camera-shake compensated second image data is used as reference image data. 22. Apparatus according to any one of items 14 to 21, wherein the apparatus is configured to merge with 1 image data.
(Item 23)
wherein said first mode is a high resolution mode and said second capture mode is a normal mode; or said first mode is a normal mode and said second capture mode is a high resolution mode; 23. Apparatus according to any one of 14-22.
(Item 24)
24. The apparatus of any one of items 14-23, wherein the first image data and the second image data are obtained from the same image sensor.
(Item 25)
further comprising a reduction unit configured to perform noise reduction processing by fusing a plurality of image data in units of image data captured in the first mode and the second mode;
25. Apparatus according to any one of items 14 to 24, wherein the fusion unit is arranged to apply fusion to the first image data and the second image data after noise reduction processing.
(Item 26)
an acquisition unit configured to acquire a video sequence comprising multiple image data captured in different modes, wherein different resolutions are set for the different modes;
a determining unit configured to determine whether different mode-based blending is to be applied based on changes between image frames of a video sequence;
a fusion unit configured to apply different mode-based fusions to the image data of the video sequence based on a result of the determination;
An apparatus for processing image data captured by an imaging unit, comprising:
(Item 27)
a camera module;
non-volatile memory storage containing instructions;
one or more processors in communication with the camera module and the memory;
wherein the one or more processors are:
acquiring first image data captured in a first mode, wherein the first mode is set to a first resolution;
switching the first mode to a second mode in response to a photographing command, wherein a second resolution is set in the second mode, the second resolution being different from the first resolution; and,
obtaining second image data captured in the second mode;
applying a first mode-based fusion to fuse the second image data with the first image data, which is reference image data;
A device that executes the above instructions to cause the
(Item 28)
28. The device of item 27, wherein the one or more processors are further configured to select the first image data and select the second image data.
(Item 29)
The selected first image data is the sharpest image without or with the fewest moving objects, and the second image data is selected with no or the fewest moving objects. 29. The device of item 28, wherein the sharpest image with a moving object of .
(Item 30)
one or more processors determine whether the first mode-based fusion is to be applied based on imaging conditions;
Any of items 27-29, further configured to switch the first mode to a second mode when it is determined that the first mode-based fusion is to be applied and the shooting command is received. devices described in .
(Item 31)
The one or more processors determine whether the first mode-based blending is applied based on changes between successive frames captured in the first mode until the capture command is received. 31. The device of item 30, configured to:
(Item 32)
32. The device of item 31, wherein the changes between the successive frames include changes in exposure, focus position, block mean, block deviation, and/or angular velocity of each axis.
(Item 33)
33. The device of any one of items 27-32, wherein the first image data and the second image data are set to have the same viewing angle and the same focus position.
(Item 34)
34. Any one of items 27-33, wherein the first mode-based fusion is combined with zero shutter lag from the capture command to completion of capture using the image captured until the capture command. device described in one.
(Item 35)
The one or more processors are further configured to perform camera shake compensation between frames captured in the first mode and the second mode, wherein the frames are captured in the first mode. 1 frame and a second frame captured in the second mode,
any one of items 27 to 34, wherein in applying the first mode-based fusion, the second image data after the camera shake compensation and the first image data are used. Devices listed.
(Item 36)
the first mode is a high resolution mode and the second mode is a normal mode; or
36. The device according to any one of items 27-35, wherein said first mode is normal mode and said second mode is high resolution mode.
(Item 37)
37. The device according to any one of items 27 to 36, wherein said first image data and said second image data are obtained from one image sensor in said imaging unit.
(Item 38)
The one or more processors are further configured to perform noise reduction processing by fusing a plurality of image data in units of image data captured in the first mode and the second mode;
38. The method of any one of items 27 to 37, wherein applying the first mode-based fusion uses the first image data and the second image data after the noise reduction process. device.
(Item 39)
39. The device according to any one of items 27-38, wherein said camera module comprises an image sensor.
(Item 40)
A computer-readable storage medium recording a program for causing a computer to execute the method according to any one of items 1 to 13.
(Item 41)
A computer program causing a computer to perform the method of any one of items 1-13.

Claims (14)

acquiring first image data captured in a first mode, wherein the first mode is set to a first resolution;
switching the first mode to a second mode in response to a photographing command, wherein a second resolution is set in the second mode, the second resolution being different from the first resolution; and,
obtaining second image data captured in the second mode;
applying a first mode-based fusion to fuse the second image data with the first image data, which is reference image data;
determining whether the first mode-based fusion is to be applied based on imaging conditions;
including
switching the first mode to a second mode comprises switching the first mode to a second mode when it is determined that the first mode-based fusion is to be applied and the shooting command is received. methods of processing image data, including; 2. The method of claim 1, said method further comprising selecting said first image data and selecting said second image data. The selected first image data is the sharpest image having no or the fewest moving objects, and the selected second image data is the none or the fewest moving objects. 3. The method of claim 2, wherein the sharpest image with a moving object of . Determining whether the first mode-based fusion is to be applied based on the imaging conditions is based on changes between successive frames captured in the first mode until the capture command is received. 4. A method according to any one of claims 1 to 3, comprising determining whether said first mode-based fusion is to be applied. 5. The method of claim 4, wherein the changes between successive frames include changes in exposure, focus position, block mean, block deviation, and/or angular velocity of each axis. A method according to any one of claims 1 to 5, wherein said first image data and said second image data are set to have the same viewing angle and the same focus position. Claim 1- wherein said first mode-based fusion is combined with zero shutter lag with no time lag from said capture command to completion of capture using said first image data captured until said capture command. 7. The method of any one of 6. performing camera shake compensation between frames captured in said first mode and said second mode, said frames being a first frame captured in said first mode and a frame captured in said second mode; a second frame;
The second image data after the camera shake compensation and the first image data are used in applying the first mode-based fusion. The method described in . the first mode is a high resolution mode and the second mode is a normal mode; or
A method according to any one of the preceding claims, wherein said first mode is normal mode and said second mode is high resolution mode. A method according to any one of the preceding claims, wherein said first image data and said second image data are obtained from one image sensor in an imaging unit. The method further comprises performing noise reduction processing by fusing a plurality of image data in units of image data captured in the first mode and the second mode;
The first image data and the second image data after the noise reduction process are used in the step of applying the first mode-based blending according to any one of claims 1 to 10. the method of. a camera module;
non-volatile memory storage containing instructions;
one or more processors in communication with the camera module and the non-volatile memory storage for executing the instructions to perform the method of any one of claims 1-11 a processor of
A device comprising: A computer-readable storage medium recording a program for causing a computer to execute the method according to any one of claims 1 to 11 . A computer program for causing a computer to carry out the method according to any one of claims 1-11 .

Images