JP2023160574A - Image processing device, control method thereof, and program - Google Patents

Abstract

To provide an image processing device, a control method thereof, and a program for making differences between small areas inconspicuous while suppressing an amount of calculation.SOLUTION: An imaging device includes a control unit, a ROM, a RAM, an optical system, an imaging unit, an A/D conversion unit, an image processing unit, a recording unit, a communication unit, a display unit, and an instruction input unit. The image processing unit includes: a first small-area extraction unit that extracts a plurality of first small-area images at a first position obtained by dividing a first input image according to a predetermined size at the first position, and a plurality of first small-area images at a second position obtained by dividing the first input image according to the predetermined size at the second position that is shifted by half the predetermined size in two mutually perpendicular directions from the first position; and a two-position small-area blending unit that combines a plurality of second small-area images at the first position and a plurality of second small-area images at the second position based on a combination rate that is set to become smaller as approaching vertices for the plurality of second small-area images in each of the first position and the second position.SELECTED DRAWING: Figure 2

Description

The present invention relates to an image processing apparatus, a control method thereof, and a program.

In order to obtain images with less noise using a digital camera, it is effective to ensure sufficient exposure time. However, when the exposure time is increased, the image becomes blurred due to camera movement due to camera shake or movement of the subject. An electronic blur correction method has been proposed as a method for dealing with such "shake." In the electronic blur correction method, images are captured multiple times in succession using short exposure times with little "shake." Next, alignment processing is performed so that movement between the plurality of images obtained is canceled, and then compositing processing is performed. This makes it possible to obtain an image with no "blur" and low noise.

Further, a method for responding to local movement of a subject is disclosed in Patent Document 1 and the like. In the method disclosed in Patent Document 1, an image to be combined is divided into a plurality of small regions, and alignment processing is performed for each of the divided small regions. Furthermore, by adaptively changing the compositing rate for each small region, it is possible to suppress the subject from becoming multiplexed due to compositing. Furthermore, by adaptively changing the strength of noise suppression by the spatial filter for each small region, differences in noise level between small regions due to differences in synthesis rate are suppressed. However, when processing each small area, the continuity of the subject may be interrupted due to the positioning of each small area, and differences in noise level and resolution may become noticeable due to adaptive processing for each small area. . Therefore, in Patent Document 2, small regions are extracted with parts of the small regions overlapping, and blend processing is performed on the overlapping portions of the small regions at a later stage of processing. This disclosed a method for making differences in resolution between small areas less noticeable.

Japanese Patent Application Publication No. 2009-164857 Japanese Patent Application Publication No. 2005-115548

However, in the method of overlapping each small area by half the size of the small area in the horizontal direction and the vertical direction, the number of small areas increases with respect to the image used for extracting the small area. As a result, there is a problem in that the amount of calculations performed for each small area increases. Furthermore, in the method of overlapping the small regions by a small width in the horizontal and vertical directions, although an increase in the amount of calculation can be suppressed, a region with no overlap occurs in the center of the small region. As a result, there has been a problem in that differences between small areas tend to be noticeable at the center of the small areas.

SUMMARY OF THE INVENTION An object of the present invention is to provide an image processing device, a control method thereof, and a program that make differences between small regions less noticeable while suppressing the amount of calculation.

In order to achieve the above object, the image processing device of the present invention includes a first small region extraction unit that extracts a plurality of first small region images whose regions partially overlap each other from a first input image;
a first blending unit that performs a blending process on an overlapping region of a second small area image that is at least one of the first small area image or a small area image obtained based on the first small area image; and,
The first small region extracting unit extracts the plurality of first small region images at the first position obtained by dividing the first input image into predetermined sizes at a first position, and the first small region image at the first position. A plurality of the first small portions at the second position obtained by dividing the input image into the predetermined size at a second position shifted by half the predetermined size in two directions perpendicular to each other from the first position. Extract the region image and
The first blending portion is set to become smaller as the plurality of second small area images approach the apex at each of the first position and the second position. The plurality of second small area images at the first position and the plurality of second small area images at the second position are combined based on a combination rate.

According to the present invention, it is possible to provide an image processing device, a control method thereof, and a program that make differences between small regions inconspicuous while suppressing an increase in the amount of calculation.

FIG. 1 is a configuration diagram of an imaging device according to a first embodiment of the present invention. FIG. 2 is a functional configuration diagram of an image processing section according to the first embodiment of the present invention. FIG. 3 is an explanatory diagram of small areas at first and second positions in the first embodiment of the present invention. FIG. 3 is an explanatory diagram of weights according to the first embodiment of the present invention. FIG. 3 is an explanatory diagram of weights according to the first embodiment of the present invention. FIG. 3 is an explanatory diagram of weights according to the first embodiment of the present invention. FIG. 3 is an explanatory diagram of weights according to the first embodiment of the present invention. FIG. 3 is a functional configuration diagram of an image processing section according to a second embodiment of the present invention. FIG. 7 is an explanatory diagram of small areas at third and fourth positions in the second embodiment of the present invention. FIG. 7 is an explanatory diagram of weights according to the second embodiment of the present invention. FIG. 7 is an explanatory diagram of weights according to the second embodiment of the present invention. FIG. 7 is an explanatory diagram of a user operation interface according to a second embodiment of the present invention. It is a flow chart explaining processing of auto mode of a 2nd embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the configurations described in the embodiments below are merely examples, and the scope of the present invention is not limited by the configurations described in the embodiments.

Note that in this specification, a "small area image" is also simply referred to as a "small area." In addition, in this specification, a "position (first position, etc.)" indicates a "deviation amount" from a reference point (reference position), and a "reference point" is, for example, a frame image to be divided into small regions. This is the upper left point. Furthermore, "a small area (small area image) at a certain position (such as a first position)" indicates that the upper left point of this small area (small area image) is the certain position. Also, "dividing the input image into a predetermined size at a certain position (first position, etc.)" means, for example, dividing a "small area image" with the "certain position" as the upper left point into multiple rectangular blocks. It is divided into two parts. Furthermore, each of these square blocks will also be referred to as a "small area (image)". Although each block will be described as being divided into "rectangular shapes", the present invention includes dividing into "quadrangular shapes". In other words, each block is divided into "squares", which is a broad concept including squares, rectangles, and the like. First, a first embodiment of the present invention will be described.

<First embodiment>
<Basic configuration>
FIG. 1 is a diagram showing an example of the basic configuration of a first embodiment of the present invention. The configuration of the first embodiment of the present invention will be described below with reference to FIG. Note that although the imaging device 100 is employed as an application example of the image processing device of the present invention, the application example of the present invention is not limited to this. For example, the image processing of the present invention may be executed by a personal computer (PC), an application server, or the like.

The imaging device 100 includes 101 control section 101, ROM 102, RAM 103, optical system 104, imaging section 105, A/D conversion section 106, image processing section 107, recording section 108, communication section 109, display section 110, and instruction input section 111. has. Each component is connected to a bus 112 and configured to be able to send and receive required information to and from each other.

The control unit 101 reads a control program for controlling each component included in the imaging apparatus 100 from the ROM 102, loads the read program into the RAM 103, and executes it. Thereby, the control unit 101 controls the operation of each component included in the imaging device 100. The control unit 101 is composed of a processor such as a CPU and a DSP. The ROM 102 is a nonvolatile memory that can be electrically erased and recorded, and nonvolatilely stores operation programs for each component included in the imaging device 100 as well as parameters necessary for the operation of each component. The RAM 103 is a rewritable volatile memory, and temporarily stores data generated by the development of programs executed by the control unit 101 and the like, the operations of each component included in the imaging apparatus 100, and the like.

The optical system 104 is composed of a lens group including a zoom lens, a focus lens, etc., and forms a subject image on the imaging surface of the imaging unit 105. The imaging unit 105 is an imaging device such as a CCD sensor or a CMOS sensor, and photoelectrically converts the optical image formed on the imaging surface by the optical system 104 to obtain an analog image signal. The obtained analog image signal is output to the A/D converter 106. The A/D converter 106 converts the input analog image signal into digital image data. Digital image data output from the A/D converter 106 is temporarily stored in the RAM 103.

The image processing unit 107 performs various image processing on the image data stored in the RAM 103. For example, it develops digital image data and performs various image processing such as demosaicing processing, white balance correction processing, and gamma processing for displaying and recording. It also performs various image processing to improve the image quality of digital image data, such as spatial filtering and noise suppression processing by combining multiple pieces of image data. The recording unit 108 records data including digital image data on a built-in recording medium. The communication unit 109 wirelessly connects to an external device and communicates image data.

The display unit 110 is realized by a display device such as an LCD, and is a display device that displays images stored in the RAM 103 and images recorded in the recording unit 108. The display unit 110 also displays an operational user interface for receiving instructions from the user. The instruction input unit 111 is an input interface that includes various operating members such as a touch panel and a shutter button, and accepts input of instructions from the user.

<Configuration of characteristic parts>
FIG. 2 is an example of the functional configuration of the image processing unit 107, which focuses on the process of synthesizing image signals of a plurality of frames, among the various types of image processing performed by the image processing unit 107 in FIG. Hereinafter, with reference to FIG. 2, an example of a functional configuration for executing image signal synthesis processing in the first embodiment will be described.

The image processing section 107 includes a small region extraction section 201 , a similar small region extraction section 202 , a composition section 203 , a noise suppression section 204 , and a two-position small region blending section 205 . The image processing unit 107 in FIG. 2 receives an input image signal "SIG_B", an input image signal "SIG_R", and two-position small area coordinate data, and outputs an output image signal "SIG_O" which is a combination of "SIG_B" and "SIG_R". Output. In the first embodiment, a total of two frames of images "SIG_B" and "SIG_R" will be synthesized. Furthermore, it is assumed that "SIG_B" and "SIG_R" are image signals obtained by continuous shooting, and that the subjects are similar.

<Small area extraction unit 201>
The small area extraction unit 201 extracts the small area image signal "SIG_B_P" from the input image signal "SIG_B" with reference to the two-position small area coordinate data. Here, with reference to FIG. 3, the two-position small area coordinate data and the specific operation of the small area extraction unit 201 will be described.

FIG. 3 shows some of the plurality of small regions extracted from the input image signal “SIG_B” by the small region extraction unit 201. A symbol 301 represented by a solid line indicates a "small area at the first position." Further, a symbol 302 represented by a broken line indicates a "small area at the second position." A plurality of small regions 301 and a plurality of small regions 302 are arranged horizontally and vertically. That is, a plurality of small regions 301, which are each of four blocks, are extracted from the input image so that they are lined up in the horizontal and vertical directions. Similarly, a plurality of small regions 302, which are each of four blocks, are extracted from the input image so as to be lined up in the horizontal and vertical directions. The points "." in FIG. 3 indicate these points.

That is, from the input image signal "SIG_B", a large number of "small areas 301" and "small areas 302" shown in FIG. 3 are extracted in the horizontal and vertical directions. For convenience of explanation, FIG. 3 shows one each of a large number of extracted small areas 301 and 302. For convenience of explanation, FIG. 3 shows a small area 301 at a first position and a small area 302 at a second position.

In this way, the small area 301 at the first position and the small area 302 at the second position are extracted in a state in which the input image signal "SIG_B" is divided into rectangular blocks of a predetermined size at each position. Here, the rectangle with the predetermined size is, for example, a rectangle with a size of "16 pixels" in the horizontal direction and "16 pixels" in the vertical direction. Furthermore, the small area 301 at the first position and the small area 302 at the second position are extracted with their block division positions shifted from each other by half the predetermined size in the horizontal and vertical directions. For example, if one of the four small areas (blocks) has a rectangular size of "16 pixels" in the horizontal direction and "16 pixels" in the vertical direction, the rectangular size is "8 pixels" in the horizontal direction and "8 pixels" in the vertical direction. The block division position is extracted with a shift of 1 minute. In other words, the small region extracting unit 201 extracts a small region (for one block) at a first position of "256 pixels" and a second small region of "256 pixels" based on an input image signal "SIG_B" consisting of a plurality of pixels. The small area (one block) is shifted by 8 pixels in each of the horizontal and vertical directions. Thus, each of the small area 301 at the first position and the small area 302 at the second position is divided into four small areas.

As a result, each pixel of the input image signal "SIG_B" is included in one of the plurality of small regions 301 at the first position and one of the plurality of small regions 302 at the second position. It is possible to extract small areas in the state. Furthermore, the small area is arranged so that the vertices shared by the four blocks of the small area 301 at the first position overlap with the center of any one of the plurality of small areas 302 (blocks) at the second position. (block) can be extracted. In FIG. 3, the vertex (the center of each of the four blocks) shared by the four small areas (blocks) at the first position is one of the four small areas (blocks) at the second position. An example of overlap with the center of a small area (one block) in the upper left of is shown.

Here, the two-position small area coordinate data is data indicating the upper left coordinates of the small area 301 at the first position and the small area 302 at the second position. The two-position small area coordinate data is sequentially calculated for each small area by, for example, a clock circuit so as to have the above-mentioned coordinates, and is sequentially supplied to the small area extraction unit 201. The small region extracting unit 201 sequentially extracts small regions of the above-described predetermined size, with the coordinates indicated in the two-position small region coordinate data as the upper left of the small region. Thereby, the small area extracting unit 201 can extract the small area 301 at the first position and the small area 302 at the second position with the positional relationship shown in FIG. Furthermore, the small area extraction unit 201 extracts small areas that are not divided into four blocks as described above from each of the small area 301 at the first position and the small area 302 at the second position. The above is a description of the two-position small area coordinate data and the specific operation of the small area extraction unit 201. Hereinafter, the explanation will return to FIG. 2.

<Similar small region extraction unit 202>
The similar small region extraction unit 202 extracts a small region image similar to the small region image signal "SIG_B_P" from the input image signal "SIG_R" based on the small region image signal "SIG_B_P" and "two-position small region coordinate data". Extract the signal "SIG_R_P". The specific operation is as follows. The similar small region extracting unit 202 first determines the search range of the "similar small region" in the input image signal "SIG_R" using the coordinates indicated by the "two-position small region coordinate data" as a reference. Here, "similar small area" means, for example, a small area where the difference between the number of vertical and horizontal pixels of a certain small area and the number of vertical and horizontal images of another small area is within a predetermined number of pixels. Refers to an area. Alternatively, for example, small regions whose vertical and horizontal pixel number ratios fall within a predetermined ratio are similar small regions. Next, the similar small region extracting unit 202 extracts a small region of the aforementioned predetermined size from the input image signal “SIG_R” for each coordinate in the search range, and extracts a small region from the input image signal “SIG_R” with the small region image signal “SIG_R” output from the small region extracting unit 201. SIG_B_P".

Next, the similar small region extracting unit 202 determines the small region of the input image signal "SIG_R" at the coordinate with the minimum difference among the coordinates in the search range as the small region image signal "SIG_R_P" and outputs it. . Thereby, the similar small region extracting unit 202 can extract a small region image signal “SIG_R_P” similar to “SIG_B_P” output from the small region extracting unit 201 from the input image signal “SIG_R”. That is, the similar small region extraction unit 202 extracts a small region image similar to the small region image from the second input image (for example, the second frame image) based on the small region image extracted by the small region extraction unit 201. . On the other hand, by extracting the small region image signal "SIG_R_P" from a position different from the original coordinates in the input image signal "SIG_R", the continuity of the subject may be interrupted between the small region image signals "SIG_R_P".

<Composition section 203>
The combining unit 203 combines the small area image signal “SIG_B_P” outputted from the small area extracting unit 201 and the small area image signal “SIG_R_P” outputted from the similar small area extracting unit 202 based on the combining rate. Then, the combining unit 203 outputs the combined small area image signal "SIG_M_P" and the combining rate "SIG_M_RATIO". It is assumed that the combining unit 203 receives a small area image signal "SIG_R_P" corresponding to each of the small area image signals "SIG_B_P" at each coordinate indicated by the two-position small area coordinate data. For example, under the control of the delay circuit, it is possible to input the small region image signal "SIG_R_P" output from the similar small region extracting section 202, which corresponds to the small region image signal "SIG_B_P" output from the small region extracting section 201. can.

The specific operation of the synthesis performed by the synthesis unit 203 will be explained. First, the combining unit 203 calculates the combining rate "SIG_M_RATIO" based on the difference between the small area image signal "SIG_B_P" and the small area image signal "SIG_R_P". Specifically, the synthesis unit 203 calculates the sum of absolute values (SAD) of the differences between each pixel of the small area image signal "SIG_B_P" and the small area image signal "SIG_R_P" as a difference value.

Next, the synthesis unit 203 refers to a table (not shown) showing the relationship between the difference value and the synthesis rate, and sets the synthesis rate so that the smaller the difference value is, the higher the synthesis rate is, and the larger the difference value is, the lower the synthesis rate is. Set “SIG_M_RATIO”. Next, the synthesizing unit 203 performs weighted synthesis of the small area image signal "SIG_B_P" and the small area image signal "SIG_R_P" based on the synthesis rate "SIG_M_RATIO" to generate the small area image signal "SIG_M_P". Specifically, when the synthesis rate “SIG_M_RATIO” is the maximum, the synthesis unit 203 synthesizes the small area image signal “SIG_B_P” and the small area image signal “SIG_R_P” so that the ratio is “50 (%)” each. do.

In addition, when the combination rate "SIG_M_RATIO" is the minimum, the ratio of the small area image signal "SIG_B_P" is "100 (%)" and the ratio of the small area image signal "SIG_R_P" is "0 (%)". do. Further, when the combination ratio "SIG_M_RATIO" is a value between the above-mentioned values, the small-area image signal "SIG_B_P" and the small-area image signal "SIG_R_P" are synthesized according to a ratio interpolated from the above-mentioned ratio. Due to the above-described operation of the compositing unit 203, compositing is suppressed when there is a large difference between the subjects of the small area images, so it is possible to suppress the subject from being in a multiple image state by the small area image signal "SIG_M_P". can. On the other hand, by adaptively changing the synthesis rate and synthesizing, a difference occurs in the degree of reduction of synthesis noise between the small area image signals "SIG_M_P".

<Noise suppression unit 204>
The noise suppression unit 204 performs noise suppression processing using a spatial filter on the small area image signal “SIG_M_P” output from the synthesis unit 203 with the intensity adjusted according to the synthesis rate “SIG_M_RATIO” in the synthesis unit 203. As a result, the noise suppression unit 204 outputs a small area image signal "SIG_S_P" in which noise is suppressed. Specifically, the noise suppression unit 204 adaptively suppresses the difference in the standard deviation of noise between the small area image signals “SIG_M_P” generated by the synthesis processing using the adaptive synthesis rate in the synthesis unit 203. Noise suppression is performed with sufficient strength. In order to perform this operation, the noise suppression unit 204 first applies a spatial filter with a predetermined strength to the small area image signal "SIG_M_P". For example, a low-pass filter, a bilateral filter, or the like is used as the spatial filter, but any type of spatial filter may be used as long as it suppresses high frequency components due to noise.

Next, the noise suppression unit 204 converts the small area image signal “SIG_M_P” before applying the spatial filter to the signal after applying the spatial filter to the synthesis rate “SIG_S_RATIO” calculated from the synthesis rate “SIG_M_RATIO”. Perform weighted addition synthesis based on the Note that here, the noise suppression unit 204 calculates the combination rate "SIG_S_RATIO" with reference to a table showing the correspondence between the combination rate "SIG_M_RATIO" and the combination rate "SIG_S_RATIO". In addition, this table is based on empirical rules so that the standard deviation of noise in "SIG_S_P" which is the output signal of the noise suppression unit 204 is the same regardless of the standard deviation of noise of "SIG_M_P" which is the input signal. Adjusted and set.

Specifically, when the synthesis rate "SIG_M_RATIO" is large, the synthesis rate "SIG_S_P" is set to be large, and when the synthesis rate "SIG_M_RATIO" is small, the synthesis rate "SIG_S_P" is set to be small. Furthermore, in the synthesis based on the synthesis rate "SIG_S_RATIO", the noise suppression unit 204 performs synthesis as follows when the synthesis rate "SIG_S_RATIO" is the maximum. That is, the signals are synthesized so that the ratio of the signal after applying the spatial filter is 0 (%), and the ratio of the small area image signal "SIG_M_P" is 100 (%). Furthermore, when the synthesis rate "SIG_S_RATIO" is the minimum, the signals are synthesized so that the signal after applying the spatial filter has a ratio of 100 (%) and the small area image signal "SIG_M_P" has a ratio of 0 (%). Further, when the synthesis rate "SIG_S_RATIO" is a value between the above-mentioned values, the signal after applying the spatial filter and the small area image signal "SIG_M_P" are synthesized according to the ratio interpolated from the above-mentioned ratio.

As explained above, the noise suppression unit 204 calculates the difference in the standard deviation of noise between the small area image signals “SIG_S_P” which are the outputs thereof, and the difference in the standard deviation of noise between the small area image signals “SIG_M_P” which is the output from the combining unit 203. It can be made smaller than the difference. On the other hand, by adaptively changing the strength of noise suppression by the spatial filter, a difference in resolution occurs between the small area image signals "SIG_S_P" that are the output signals of the noise suppression unit 204.

<Two-position small area blending unit 205>
The two-position small area blending unit 205 (blending unit) performs weighted addition of small areas on the small area image signal “SIG_S_P” output from the noise suppression unit 204 based on the two-position small area coordinate data. SIG_O" is output. Specifically, first, the two-position small area blending unit 205 multiplies the small area image signal "SIG_S_P" by a weight for performing blending to generate the small area image signal "SIG_W_P". Next, the two-position small area blending unit 205 applies the small-area image signal to an area whose upper left is the coordinate indicated by the two-position small area coordinate data (coordinate data of small areas at two positions) in the output signal "SIG_O". "SIG_W_P" is added and the output signal "SIG_O" is updated and output.

By repeating this process, the small area image signal "SIG_S_P" at the first position and the small area image signal "SIG_S_P" at the second position are weighted and added in the positional relationship between the two positions explained with reference to FIG. will be done. It is assumed that the output signal "SIG_O" has a pixel value of "0" in the entire region in the initial state. Further, it is assumed that the two-position small-area blending unit 205 receives a small-area image signal “SIG_S_P” corresponding to each of the small-area image signals “SIG_B_P” at each coordinate indicated by the two-position small-area coordinate data. Thereby, the small area extraction unit 201 can synthesize the input image signal "SIG_B" so that the small area image signal "SIG_W_P" is placed in the same area as the area where the small area image signal "SIG_B_P" is extracted.

Note that the two-position small area blending unit 205 can also be configured to perform blending processing on overlapping areas of small areas obtained based on the small area image signal “SIG_B_P” that is the output of the small area extraction unit 201. Further, the two-position small region blending unit 205 inputs and blends both the small region image signal “SIG_B_P” which is the output of the small region extraction unit 201 and the small region obtained based on the small region image signal “SIG_B_P”. It is also possible to have a configuration in which the information is processed. In other words, the overlapping areas of the "second small area image" which is at least one of the small area image (the small area 301 at the first position) or the small area image obtained based on this small area image are blended. Process.

<Generation of small area image signal “SIG_W_P” by two-position small area blending unit 205>
Here, with reference to FIGS. 4, 5, 6, and 7, two-position small area blending unit 205 multiplies the small area image signal "SIG_S_P" to generate the small area image signal "SIG_W_P""Weight" will be explained.

FIG. 4 is an explanatory diagram of weights multiplied by the small area image signal "SIG_S_P" by the two-position small area blending unit 205. The entire rectangle in FIG. 4 represents one "small area 301 at the first position" and one "small area 302 at the second position" shown in FIG. 3. Symbols 401a, 401b, 401c, and 401d are the vertices of the small regions, respectively. Further, reference numeral 402 is the center of the small area. At this time, the weight multiplied by the two-position small area blending unit 205 is the largest at the center 402. Further, as the spatial distance from the center 402 to each vertex 401a, 401b, 401c, and 401d approaches, the weight value becomes smaller. A specific example of weight values will be described with reference to FIG.

<Specific examples of weight values>
FIG. 5 is an extracted diagram of the upper left region among the four regions divided by the broken line in FIG. At this time, the two-position small area blending unit 205 calculates the urban area distance (also known as "Manhattan distance" or "L1 distance") to the vertex 401a when the horizontal direction is the first coordinate axis and the vertical direction is the second coordinate axis. Different weight values are used for multiplication depending on . For example, the center 402 is the coordinate that has the farthest urban distance to the vertex 401a in the upper left part of FIG. For the purpose of explanation, the city distance from the center 402 to the vertex 401a is assumed to be "100 (%)". At this time, the weight value at the center 402 is, for example, "100 (%)". Further, the vertex 401a is a coordinate whose urban distance to the vertex 401a is "0 (%)", and the weight value is, for example, "0 (%)".

A broken line 501 indicates a line formed by connecting the midpoints of adjacent sides (two sides having the vertex 401a as one end) in the small area shown in FIG. In the set of coordinates indicated by the broken line 501, the urban distance to the vertex 401a is "50(%)". The weight value at the broken line 501 is, for example, "50 (%)", which is a value obtained by linearly interpolating the weight value at the center 402 and the weight value at the vertex 401a. Similarly, in the set of coordinates where the city distance is "75 (%)", which is indicated by the broken line 502, the weight value is "75 (%)". Further, in the set of coordinates where the city distance is "25 (%)", which is indicated by the broken line 503, the weight value is "25 (%)". Similarly, for other coordinates, the two-position small area blending unit 205 uses different weight values depending on the urban distance to the vertex 401a. In addition, in the above description, "weight according to the urban distance to the apex of the small area" can be rephrased as "weight according to the urban area distance to the center of the small area". The above is the explanation of FIG. 5.

Further, the two-position small area blending unit 205 can set weights in the same manner in areas other than the area shown in FIG. 5 among the four areas divided by the broken line in FIG. 4. At this time, weights are set according to the urban distance to the vertex 401b in the upper right region, the vertex 401c in the lower left region, and the vertex 401d in the lower right region. For example, FIG. 6 shows an example of weight values multiplied by the two-position small area blending unit 205 for each coordinate of a small area of "16 pixels" in the horizontal direction and "16 pixels" in the vertical direction. be. In addition, in FIG. 6, the value corresponding to "100 (%)" of the weight value in the above description is set to "16". As shown in FIG. 6, it can be seen that the weight value is large near the center 402 and becomes smaller toward the vertices 401a, 401b, 401c, and 401d.

Thereby, the two-position small area blending unit 205 allows the intensity of the components of the small area image signal "SIG_S_P" at the first position and the small area image signal "SIG_S_P" at the second position to change spatially smoothly. The output signal “SIG_O” can be generated. Therefore, the difference between the small area image signals caused by the processing of the similar small area extracting unit 202, the combining unit 203, the noise suppressing unit 204, etc. can be made less noticeable by the output signal “SIG_O”. In addition, the two-position small area blending unit 205 preferably multiplies each of the small area image signals "SIG_S_P" by the same weight. This makes it possible to suppress unevenness in the output signal "SIG_O" in the weighted addition process of the two-position small area blending unit 205. In this case, the two-position small area blending unit 205 multiplies each of the small area image signals "SIG_S_P" by the weight values shown in FIG. 6, for example.

In addition, the two-position small area blending unit 205 weights the small area image signal "SIG_S_P" so that as a result of the weighted addition of the small area image signal "SIG_S_P", the total value of the weight values becomes 100 (%). It is preferable to multiply . In this case, the weight value used by the two-position small area blending unit 205 is n (%) (100≧n>50) at the center of the small area (four blocks). Then, the vertices of the small area (four blocks) may be set to "100-n (%)", and the coordinates therebetween may be values linearly interpolated according to the urban distance to the vertices. In addition, in the weighted addition of the small area image signal "SIG_S_P" by the two-position small area blending unit 205, if the weighting method is such that the total value of the weight of each coordinate of the small area is 100 (%), Any weighting may be used.

<Examples of other weights>
Next, examples of other weights will be explained with reference to FIG. The horizontal axis in FIG. 7 indicates each coordinate position on the straight line connecting the vertex 401a and the center 402 in FIG. 5, and the vertical axis indicates the weight value. A solid line 701 shows an example in which the weight value between the vertex 401a and the center 402 is a value obtained by linear interpolation using the method described above. As shown in FIG. 7, it can be seen that the weight increases linearly from the vertex 401a to the center 402. Here, for the sake of explanation, the weight value at the coordinate position where the city distance to the vertex 401a is "d (%)" (0≦d≦100) is expressed as "m (%): (0≦m≦100)". do. At this time, if the weight satisfies the condition that the weight value at the coordinate position where the distance to the vertex 401a is "100-d (%)" is "100-m (%)" at all coordinate positions, the solid line 701 Other weights may be used.

For example, the broken line 702 is a modified example in which the weight value between the vertex 401a and the center 402 is changed in a curved manner so as to satisfy the above-mentioned conditions. In the case of weights that satisfy the above-mentioned conditions, the two-position small area blending unit 205 adds weights to each of the small area image signals "SIG_S_P" with the same weight, so that the total value of the weight values becomes 100 (%). The output signal "SIG_O" can be generated as follows. Note that under the above conditions, no matter what the weight is, it is necessary to set the weight value to ``50 (%)'' on the straight line (501) connecting the midpoints of adjacent sides of the small area shown in Figure 5. There is. The above is an explanation of the weight that the two-position small area blending unit 205 multiplies the small area image signal "SIG_S_P". Hereinafter, the explanation will return to FIG. 2.

The two-position small area blending unit 205 calculates the weight described above based on each coordinate value of the small area image signal "SIG_S_P". Alternatively, a configuration may be adopted in which the control unit 101 gives the aforementioned weights stored in advance in the ROM 102 and RAM 103 to the two-position small area blending unit 205. The above is the explanation of the two-position small area blending unit 205.

The image processing unit that performs the image signal synthesis process shown in FIG. By performing the processing, the final output signal "SIG_O" can be generated. Note that "all two-position small area coordinate data" is, for example, two-position small area coordinate data that indicates all coordinates for extracting a small area from the entire area of the input image signal "SIG_B" or the input image signal "SIG_R". It is. The above is a description of the image processing section that performs the image signal synthesis process in FIG. 2.

Note that FIG. 2 is an example of an embodiment to which the present invention is applied, and as long as the structure allows the operations of the small region extracting section 201 and the two-position small region blending section 205, the structure can be as shown in FIG. Not limited to. For example, the combining section 203 and the noise suppressing section 204 may not be provided. In this case, the small area image signal “SIG_R_P” output from the similar small area extracting unit 202 is input to the two-position small area blending unit 205 instead of the small area image signal “SIG_S_P”. Even in this case, the present invention can prevent a break in the continuity of the subject occurring between the small area image signals "SIG_R_P" from becoming noticeable in the output image signal "SIG_O".

Further, for example, the similar small area extraction unit 202 may not be provided. In this case, for example, the small area image signal "SIG_R_P" is extracted from the input image signal "SIG_R" by the same operation as the small area extraction unit 201. Even in this case, the present invention allows the difference in the degree of noise reduction occurring between the small area image signals "SIG_M_P" and the difference in resolution occurring between the small area image signals "SIG_S_P" to be ” can be suppressed from becoming conspicuous.

Further, in the first embodiment, an example was shown in which two frames of the input image signal "SIG_B" and the input image signal "SIG_R" are combined, but the present invention is applicable regardless of the number of frames to be combined. For example, when composing three or more frames, do as follows. That is, the similar small region extracting unit 202 extracts a plurality of small region image signals “SIG_R_P” from a plurality of input image signals “SIG_R”, and the combining unit 203 extracts a plurality of small region image signals “SIG_B_P” and a plurality of small region image signals A configuration may be adopted in which "SIG_R_P" is synthesized.

The following configuration is obtained from the above explanation. That is, it includes a small area extraction unit 201 (first small area extraction unit) that extracts a plurality of small area images (first small area image 301, etc.) whose areas partially overlap each other from an input image. Further, the overlapping region of the second small area image (described above) which is at least one of the small area image (the first small area image) or the small area image obtained based on the small area image is blended. A two-position small area blending section 205 is provided. The small area extraction unit 201 extracts first small area images at a plurality of first positions obtained by dividing an input image into predetermined sizes at a first position. Next, first small area images at a plurality of second positions are obtained by dividing the image into predetermined sizes at second positions shifted by half a predetermined size in two directions perpendicular to each other from the first position. Extract. The two-position small area blending unit 205 also reduces the size of each of the plurality of second small area images (301, 302) at each of the first position and the second position as it approaches its apex. The next synthesis is performed based on the synthesis rate set so that. That is, the plurality of second small area images (301) at the first position and the plurality of second small area images (302) at the second position are combined.

Furthermore, the two-position small area blending section 205 (first blending section) can also be synthesized in the following manner. The two-position small area blending unit 205 (first blending unit) combines the plurality of second small area images (301) at the first position and the plurality of second small area images at the second position based on the above-mentioned synthesis rate. It is possible to adopt a configuration in which the small area images (302) of 2 are combined by weighted addition. Furthermore, the next synthesis is performed for each of the plurality of second small area images (301, 302) based on a synthesis rate that is set to vary depending on the urban area distance from a predetermined position in the image to the vertex. That is, the plurality of second small area images (301) at the first position and the plurality of second small area images (302) at the second position are combined. In addition, the plurality of second small area images (301, ) and a plurality of second small area images (302) at the second position may be combined. The plurality of second small area images at the first position are calculated based on the synthesis rate such that the synthesis rate is "50(%)" in the straight line connecting the midpoints of adjacent sides of each of the plurality of second small area images. It is also possible to adopt a configuration in which the regional image (301) and a plurality of second small region images (302) at the second position are combined.

Further, noise adjustment is performed on the plurality of second small area images (301, 302) using mutually different (noise suppression) strengths between two or more second small area images (between 301 and 302). A configuration including an "adaptive adjustment section" may also be used. Further, the plurality of second small area images are generated by a process including adjustment processing by the adaptive adjustment unit. This "adaptive adjustment section" corresponds to the above-mentioned synthesis section 203 and noise suppression section 204. Moreover, the "strength" here refers to "image processing strength" and "noise suppression strength." The “adaptive adjustment unit” changes the intensity of image processing by changing the “synthesis rate SIG_M_RATIO” output from the synthesis unit 203 and the “synthesis rate SIG_S_RATIO” input to the noise suppression unit 204.

Further, the adaptive adjustment unit further adjusts the plurality of first small region images (301, 302) and the plurality of third small region images extracted from the third input image based on the inter-image synthesis rate. It is also possible to adopt a configuration in which a process including a compositing process to obtain a plurality of fourth small area images is executed. At this time, the inter-image synthesis rate is set to be different between the two or more first small area images (301, 302). Here, the third input image is, for example, the image signal "SIG_R" in FIG. 2. Further, the adaptive adjustment unit further executes noise suppression processing for suppressing noise based on a predetermined suppression rate on the plurality of fourth small area images, and the predetermined suppression rate is a fourth small area image of 2 or more. The regional images can have different configurations.

<Second embodiment>
In Embodiment 2, the configuration of Embodiment 1 is modified, and an example is shown in which the two-position small area extraction and blending process and the four-position small area extraction and blending process described in Embodiment 1 are used. In other words, the second embodiment is characterized in that the second position and the fourth position are used selectively. Note that unless otherwise specified, components having the same reference numerals as in the first embodiment perform the same operations and processing as in the first embodiment. Further, the basic configuration example of the second embodiment is the same as that of the first embodiment, and the description thereof will be omitted.

<Characteristic configuration of the second embodiment>
FIG. 8 shows, among various types of image processing performed by the image processing unit 107 of the second embodiment, four-position processing that is provided separately from the synthesis processing of multiple frames of image signals using small regions at two positions described in FIG. 2. This is an example of a functional configuration for performing compositing processing using a small area. Hereinafter, with reference to FIG. 8, an example of a functional configuration of a compositing process using small regions at four positions will be described. Note that, hereinafter, "combining processing of image signals of multiple frames using small regions at two positions" will be simply referred to as "two-position combining processing." Furthermore, "combining processing of image signals of multiple frames using small regions at four positions" is simply referred to as "four-position combining processing."

The image processing section 107 shown in FIG. 8 includes a small region extraction section 201, a similar small region extraction section 202, a composition section 203, a noise suppression section 204, and a four-position small region blending section 801. The image processing unit 107 inputs the input image signal "SIG_B", the input image signal "SIG_R", and the four-position small area coordinate data, and outputs an output image signal "SIG_O" which is a combination of "SIG_B" and "SIG_R". Note that in the second embodiment, for the sake of explanation, it is assumed that two frame images "SIG_B" and "SIG_R" are combined. As can be seen by comparing with FIG. 2, in the image signal synthesis process of FIG. Input the area coordinate data. The four-position small area coordinate data will be explained with reference to FIG.

FIG. 9 shows some of the plurality of small regions extracted from the input image signal “SIG_B” by the small region extraction unit 201 in the second embodiment. In FIG. 9A, each of the four blocks (small areas) indicated by solid lines indicates the small area 301 at the first position in FIG. Furthermore, four blocks (small areas) indicated by dotted lines indicate the small area 901 at the third position. Furthermore, in FIG. 9B, each of the four blocks (small areas) indicated by dotted lines indicates the small area 902 at the fourth position. That is, in FIG. 9(a), the reference numeral 301 represented by a solid line indicates a small area at the first position, as in FIG. A plurality of small areas 901 and 902 are arranged in parallel in the horizontal and vertical directions.

That is, a plurality of small regions 901 are extracted from the input image so as to be lined up in the horizontal and vertical directions. Similarly, a plurality of small regions 902 are extracted from the input image so as to be lined up in the horizontal and vertical directions. In FIG. 9, four small areas such as small area 901 and four small areas such as small area 902 are shown among the plurality of small areas, and the dots "・" in FIG. Indicates that the region has been omitted from the diagram. In this way, a plurality of these small areas 901 and 902 are lined up horizontally and vertically, respectively. As described above, a plurality of small regions 301 are extracted, but for convenience of explanation, only four small regions 301 at the first position are shown in FIG.

A plurality of small regions 301 and a plurality of small regions 302 are arranged horizontally and vertically. That is, a plurality of small areas 301 are extracted from the input image so that they are lined up in the horizontal and vertical directions. Similarly, a plurality of small regions 302 are extracted from the input image so as to be lined up in the horizontal and vertical directions. The three dots "." in FIG. 3 indicate that small areas other than the illustrated small areas are omitted from the diagram.

Furthermore, the small area 901 at the third position and the small area 902 at the fourth position are generated by dividing the input image signal "SIG_B" into rectangular blocks of a predetermined size at the respective positions. The rectangular block of the predetermined size has the same size as the small area 301 at the first position and the small area 302 at the second position shown in FIG.

Further, the small area 301 at the first position and the small area 901 at the third position are in a state where the block division positions are shifted from each other in the horizontal direction by half of the above-described predetermined size. For example, if the small area is a rectangle with a size of "16 pixels" in the horizontal direction and "16 pixels" in the vertical direction, the position of the block division is shifted by "8 pixels" in the horizontal direction. Further, the small area 301 at the first position and the small area 902 at the fourth position are in a state in which the block division positions are shifted from each other by half of the predetermined size in the vertical direction. For example, if the small area is a rectangle with a size of "16 pixels" in the horizontal direction and "16 pixels" in the vertical direction, the position of the block division is shifted by "8 pixels" in the vertical direction.

Here, the four-position small area coordinate data includes the small area 301 at the first position, the small area 302 at the second position, the small area 901 at the third position, and the fourth position shown in FIGS. This is data indicating the upper left coordinates of each small area 902 at the position. The four-position small area coordinate data (coordinate data of the small area at four positions) is sequentially calculated for each small area by, for example, a clock circuit so as to have the above-mentioned coordinates. The above is the explanation of the four-position small area coordinate data. Hereinafter, the explanation will return to FIG. 8.

In this way, the small area extraction unit 201 can extract small areas for each of the small areas (301, 302, 901, 902) at each of the first to fourth positions. Further, the similar small region extraction unit 202 searches for a similar small region in the input image signal “SIG_R” using the coordinates indicated by the four-position small region coordinate data as a reference instead of the two-position small region coordinate data of the first embodiment. Determine the range.

The four-position small region blending unit 801 performs weighted addition of small regions on the small region image signal "SIG_S_P" based on the four-position small region coordinate data, and outputs "SIG_O". Specifically, first, the four-position small area blending unit 801 multiplies the small area image signal "SIG_S_P" by a weight for performing blending to generate the small area image signal "SIG_W_P." Next, the four-position small-area blending unit 801 adds the small-area image signal "SIG_W_P" to the area whose coordinates indicated by the four-position small-area coordinate data in the output signal "SIG_O" are in the upper left corner, and adds the small-area image signal "SIG_W_P" to the output signal "SIG_O". ” is updated and output.

By repeating this process, the small area image signal SIG_S_P at the first position, second position, third position, and fourth position is obtained in the positional relationship of the four positions explained with reference to FIGS. 3 and 9. It will be added weighted. Note that the output signal "SIG_O" has a pixel value of "0" in the entire region in the initial state. Further, the four-position small-area blending unit 801 receives a small-area image signal “SIG_S_P” corresponding to each of the small-area image signals “SIG_B_P” at each coordinate indicated by the four-position small-area coordinate data. Thereby, the small area extraction unit 201 can synthesize the small area image signal "SIG_W_P" so as to place it in the same area as the area where the small area image signal "SIG_B_P" is extracted from the small area image signal "SIG_B". Here, with reference to FIGS. 10 and 11, the weight multiplied by the small area image signal "SIG_S_P" in order for the four-position small area blending unit 801 to generate the small area image signal "SIG_W_P" will be described.

<“Weight” of the second embodiment>
<Horizontal weight>
FIG. 10 is an explanatory diagram of weights multiplied by the small area image signal "SIG_S_P" by the four-position small area blending unit 801. The entire rectangle shown in FIG. 10 represents one small area (301, 302, 901, 902). The four-position small area blending unit 801 multiplies the small area image signal "SIG_S_P" by weights that change in the horizontal and vertical directions. First, the weight in the horizontal direction will be explained. Each of reference numeral 1001a and reference numeral 1001b is a side extending in the vertical direction of the small area. Further, reference numeral 1002 is a center line for dividing the small area in the vertical direction. At this time, the weight multiplied by the four-position small area blending unit 801 is the largest at the center line 1002 in the horizontal direction. Further, as the spatial distance from the center line 1002 to the sides 1001a and 1001b becomes longer, the value of the weight in the horizontal direction becomes smaller. Specifically, the four-position small area blending unit 801 uses different weight values for the multiplication depending on the Euclidean distance to the nearest side 1001a or 1001b. For example, the center line 1002 is a set of coordinates having the farthest Euclidean distance from sides 1001a and 1001b in the area shown in FIG.

Here, for the sake of explanation, the Euclidean distance from the center line 1002 to the sides 1001a and 1001b is assumed to be "100 (%)". At this time, the weight value for the horizontal direction at the center line 1002 is, for example, "100 (%)". Furthermore, the sides 1001a and 1001b are a set of coordinates whose Euclidean distance to the nearest side 1001a or 1001b is "0 (%)", and the weight value in the horizontal direction is, for example, "0 (%)". It is. A broken line 1003a and a broken line 1003b extending in the vertical direction indicate a set of coordinates whose Euclidean distance to the nearest side 1001a or 1001b is "50(%)". The value of the weight in the horizontal direction on the broken lines 1003a and 1003b is, for example, "50 (%)", which is a value obtained by linearly interpolating the weight value on the center line 1002 and the weight value on the sides 1001a and 1001b. Similarly, for other coordinates, the four-position small area blending unit 801 uses values of weights in the horizontal direction that are varied depending on the Euclidean distance to the nearest side 1001a or 1001b. FIG. 11A shows an example of the weight value in the horizontal direction at each coordinate of a small region of "16 pixels" in the horizontal direction and "16 pixels" in the vertical direction. Note that in FIG. 11A, the value corresponding to the weight value "100 (%)" in the above description is "9". The above is an explanation of the weight in the horizontal direction.

<Vertical weight>
Next, weights in the vertical direction will be explained. Reference numerals 1004a and 1004b in FIG. 10 are sides extending in the horizontal direction of the small area. Further, reference numeral 1005 is a center line for dividing the small area in the horizontal direction. At this time, the weight multiplied by the four-position small area blending unit 801 is the largest at the center line 1005 in the vertical direction. Further, as the spatial distance from the center line 1005 to the sides 1004a and 1004b becomes greater, the value of the weight in the vertical direction becomes smaller. Specifically, the four-position small area blending unit 801 uses different weight values for multiplication depending on the Euclidean distance to the closer of the sides 1004a or 1004b. For example, the center line 1005 is a set of coordinates having the farthest Euclidean distance to sides 1004a and 1004b in the area shown in FIG. For the sake of explanation, here, the Euclidean distance from the center line 1005 to the sides 1004a and 1004b is assumed to be "100 (%)". At this time, the value of the weight in the vertical direction at the center line 1005 is, for example, "100 (%)".

Furthermore, the sides 1004a and 1004b are a set of coordinates whose Euclidean distance to the nearest side 1004a or 1004b is "0 (%)", and the weight value in the vertical direction is, for example, "0 (%)". It is. Further, the broken line 1006a and the broken line 1006b indicate a set of coordinates whose Euclidean distance to the nearest side 1004a or 1004b is "50 (%)". The value of the weight in the vertical direction on the broken line 1006a and the broken line 1006b is, for example, "50 (%)", which is a value obtained by linearly interpolating the weight value on the center line 1005 and the weight value on the sides 1004a and 1004b. Similarly, for other coordinates, the four-position small area blending unit 801 uses weight values in the vertical direction that are varied depending on the Euclidean distance to the closer of the sides 1004a or 1004b. FIG. 11B shows an example of weight values in the vertical direction at each coordinate of a small region of "16 pixels" in the horizontal direction and "16 pixels" in the vertical direction. Note that in FIG. 11(b), the value corresponding to "100 (%)" of the weight value in the above description is "9". The above is an explanation of the weight in the vertical direction.

The four-position small region blending unit 801 multiplies the small region image signal "SIG_S_P" by a weight that is a combination of the "horizontal direction weight" and "vertical direction weight" described above to generate a small region image signal "SIG_W_P". do. The weight that is a combination of the "weight in the horizontal direction" and the "weight in the vertical direction" is, for example, the weight obtained by multiplying the "weight in the horizontal direction" and the "weight in the vertical direction" at each coordinate. FIG. 11(c) shows the "weight in the horizontal direction" and "weight in the vertical direction" at each coordinate of a small area (four blocks) of "16 pixels" in the horizontal direction and "16 pixels" in the vertical direction. An example of combined weights is shown. The weights shown in FIG. 11(c) are obtained by multiplying the "horizontal weight" shown in FIG. 11(a) and the "vertical weight" shown in FIG. 11(b) at each coordinate. It is. Note that in FIG. 11(c), the value corresponding to the weight value "100 (%)" in the above description is "81". The four-position small area blending unit 801 also generates a similar small area image signal ``SIG_W_P'' by sequentially multiplying the small area image signal ``SIG_S_P'' by the ``weight for the horizontal direction'' and the ``weight for the vertical direction.'' can be generated.

As a result, the four-position small-area blending unit 801 spatially smoothly transitions the intensity of the component between the small-area image signal "SIG_S_P" at the first position to the fourth position included in the output signal "SIG_O". The output signal "SIG_O" can be generated as follows. In particular, compared to the two-position small-area blending unit 205, the four-position small-area blending unit 801 spatially increases the component intensity between the small-area image signals “SIG_S_P” near the midpoint of the side of the small-area image signal “SIG_S_P”. The output signal "SIG_O" can be generated so as to be smooth. That is, in the four-position synthesis process shown in FIG. 8, it is easier to generate the output signal "SIG_O" with higher image quality than in the two-position synthesis process described in FIG. On the other hand, in the four-position synthesis process shown in FIG. 8, the amount of calculation increases compared to the two-position synthesis process shown in FIG. Specifically, when the number of small area positions to be processed doubles, the amount of calculation also doubles. This increases the processing time and power consumption due to the compositing process. The above is an explanation of the four-position synthesis process using the functional configuration shown in FIG.

Although FIG. 8 shows an example in which two frames of the input image signal "SIG_B" and the input image signal "SIG_R" are combined, the four-position combining process is applicable regardless of the number of frames. For example, when compositing three or more frames, do the following. That is, the similar small region extraction unit 202 extracts a plurality of small region image signals "SIG_R_P" from a plurality of input image signals "SIG_R". Then, the composition unit 203 in FIG. 8 can be configured to compose the small area image signal "SIG_B_P" and the plurality of small area image signals "SIG_R_P". Thereby, even when compositing three or more frames, small regions at each of the first to fourth positions can be composited.

<Switching between two-position compositing processing and four-position compositing processing>
Next, the operation of switching between the two-position synthesis process and the four-position synthesis process performed by the control unit 101 in FIG. 1 will be described. As described above, in four-position compositing processing, image quality tends to be relatively high compared to two-position compositing processing. On the other hand, the relatively large amount of calculation increases processing time and power consumption. In other words, in two-position compositing processing, the image quality tends to be relatively lower than in four-position compositing processing, but processing time and power consumption can be suppressed because the amount of calculation is relatively small. As described above, both have advantages and disadvantages, so it is desirable that the two-position synthesis process and the four-position synthesis process be switchable. Hereinafter, a specific synthesis processing switching operation of the control unit 101 will be described.

The control unit 101 switches between the two-position synthesis processing mode and the four-position synthesis processing mode based on a user's instruction via the instruction input unit 111. With reference to FIG. 12, reception of user instructions will be described. FIG. 12 is an example of a user operation interface displayed on the display unit 110 when the control unit 101 receives a user's instruction. The control unit 101 can receive a user's instruction to select one of “speed priority mode,” “image quality priority mode,” and “auto mode” via the instruction input unit 111. If the selection of the "speed priority mode" is accepted, the control unit 101 sets the image processing unit 107 to perform the two-position compositing process shown in FIG. 2 during the compositing process. Further, when the selection of the "image quality priority mode" is accepted, the control unit 101 sets the image processing unit 107 to perform the four-position compositing process shown in FIG. 8 during the compositing process. Further, when receiving the selection of "auto mode", the control unit 101 sequentially switches between the two-position synthesis process shown in FIG. 2 and the four-position synthesis process shown in FIG. 8 according to predetermined conditions.

Thereby, the control unit 101 can switch the synthesis processing mode based on the user's instructions. Note that the user operation interface shown in FIG. 12 is an example. Any method may be used as long as the control unit 101 can accept a user's instruction for switching between two-position synthesis processing and four-position synthesis processing. Note that the compositing process based on the compositing process mode may be executed at any timing; for example, the control unit 101 may perform the compositing process at the timing at which a user's instruction to perform the compositing process is received or at the timing at which continuous shooting images are acquired. You may also perform the compositing process using

<“Auto mode”>
Next, with reference to FIG. 13, an operation in which the control unit 101 switches between two-position compositing processing and four-position compositing processing according to predetermined conditions when in the "auto mode" will be described using FIG. 13. In the "auto mode", the control unit 101 performs the two-position combining process shown in FIG. 2 and the four-position combining process shown in FIG. 8 in the flow shown in FIG. Select which one to perform.

First, in S1301, the control unit 101 determines whether the number of frames of input image signals used for synthesis is equal to or greater than a threshold value. Note that the number of frames of input image signals used for synthesis is determined in advance by the control unit 101 based on the reception of user instructions and the number of frames of continuously captured images. If the number of frames of the input image signal used for synthesis is equal to or greater than the threshold value (yes), the process moves to S1303.

On the other hand, if the number of frames of the input image signal used for synthesis is not equal to or greater than the threshold (no), the process moves to S1302. Next, in S1302, the control unit 101 determines whether the number of pixels of the input image signal used for synthesis is equal to or greater than a threshold value. Note that the number of pixels of the input image signal used for synthesis is determined in advance by the control unit 101 based on the reception of a user's instruction and the number of pixels of continuously captured images. If it is determined that the number of pixels of the input image signal used for synthesis is equal to or greater than the threshold value (yes), the process moves to S1303. On the other hand, if it is determined that the number of pixels of the input image signal used for synthesis is not equal to or greater than the threshold value (no), the process moves to S1304.

Next, when the process moves to S1303, in S1303, the control unit 101 sets the image processing unit 107 to perform the compositing process by the two-position compositing process shown in FIG. When the process in S1303 is completed, the control unit 101 ends the flow shown in FIG. 13. Further, when the process moves to S1304, in S1304, the control unit 101 sets the image processing unit 107 to perform the compositing process using the four-position compositing process shown in FIG. When the process in S1304 is completed, the control unit 101 ends the flow of FIG. 13. The above is the operation flow of the control unit 101 that switches between the two-position synthesis process and the four-position synthesis process according to predetermined conditions when the "auto mode" is selected.

As a result, under conditions where the amount of calculation increases, such as when there are many frames of input image signals used for synthesis or when the number of pixels is large, the control unit 101 selects two-position synthesis processing with priority given to suppressing the amount of calculation. be able to. Further, if the condition does not increase the amount of calculation, the control unit 101 can select the four-position synthesis process with priority given to the quality of the image signal of the synthesis result. Note that the processing flow shown in FIG. 13 is an example, and the control unit 101 may select the combining process using any method as long as the combining process mode is selected based on conditions related to the amount of calculation. Further, the control unit 101 may make the determination based on a combination of conditions, instead of determining the selection of the synthesis process based on individual conditions related to the amount of calculation as in S1301 and S1302. For example, the composition process may be selected based on the total number of pixels obtained by multiplying the number of frames of input image signals used for composition by the number of pixels. The condition may be the number of frames or the number of pixels of at least one of the input image, the second input image, and the third input image. Alternatively, the apparatus may be configured to automatically determine the conditions and execute the "auto mode" without the user selecting the "auto mode".

Furthermore, the following configuration may be adopted based on the second embodiment. That is, the small area extraction unit 201 (see FIG. 8: second small area extraction unit) extracts a plurality of first small area images (301) at a first position and a plurality of first small area images (301) at a second position. A small area image (302) is extracted. Further, the input image (first input image) is divided into the predetermined size obtained by dividing the input image (first input image) at a third position shifted by half the predetermined size in the first direction of two directions perpendicular to each other from the first position. A plurality of first small area images (901) at position No. 3 are extracted. Furthermore, a plurality of first images at a fourth position obtained by dividing the input image into the predetermined size at a fourth position shifted by half the predetermined size in the second direction of the two directions from the first position. A small area image (902) is extracted.

In addition, the four-position small area blending unit 801 (second blending unit) performs each of the plurality of second small area images (301, 302, 901, 902) at each of the first to fourth positions. Composition is performed based on a composition rate that is set to decrease as the distance approaches that side. That is, a plurality of second small area images (301) at a first position, a plurality of second small area images (302) at a second position, and a plurality of second small area images (302) at a third position. 901) and a plurality of second small area images (902) at the fourth position may be combined.

<Modified example>
Note that, although the present invention is used, for example, to obtain a composite image that is not affected by a subject in camera shake processing through image composition, the application example of the present invention is not limited to this. In addition, it is also possible to perform image smoothing processing between small regions using a smoothing kernel (smoothing spatial filter) on the overlapped portions that have been blended, thereby making the steps between small regions even less noticeable. . Further, it is also possible to have a configuration including a plurality of similar small region extracting units 202 that input input images of the third frame or higher. Further, the present invention is applicable to an imaging device (such as a camera), and is also applicable to a captured image captured by a mobile terminal including an imaging unit. Examples of mobile terminals include smartphones, tablet terminals, and portable notebook computers. Note that the communication standard of the mobile terminal may be of any type.

<Additional notes>
The disclosure of this embodiment includes the following configurations.
(Configuration 1)
a first small region extraction unit that extracts a plurality of first small region images whose regions partially overlap each other from the first input image;
a first blending unit that performs a blending process on an overlapping region of a second small area image that is at least one of the first small area image or a small area image obtained based on the first small area image; and,
The first small area extraction unit is
a plurality of first small area images at the first position obtained by dividing the first input image into predetermined sizes at a first position; extracting a plurality of first small area images at the second position obtained by dividing into the predetermined size at a second position shifted by half the predetermined size in two directions perpendicular to each other;
The first blending section is
Based on a combination rate that is set to decrease as the plurality of second small area images approach the apex of the plurality of second small area images at each of the first position and the second position, the first An image processing device characterized in that the plurality of second small area images at the position and the plurality of second small area images at the second position are combined.
(Configuration 2)
The first blending section is
Based on the synthesis rate, the plurality of second small area images at the first position and the plurality of second small area images at the second position are combined by weighted addition. The image processing device according to configuration 1.
(Configuration 3)
The first blending section is
The plurality of second small area images at the first position are based on a synthesis rate that is set to vary depending on the distance from a predetermined position in the image to the vertex for the plurality of second small area images. The image processing device according to any one of Structures 1 to 2, wherein a region image and the plurality of second small region images at the second position are combined.
(Configuration 4)
The image processing device according to configuration 3, wherein the distance is an urban distance.
(Configuration 5)
The first blending section is
The plurality of second small area images at the first position and the second small area images at the second position such that the total value of the synthesis rate for the plurality of second small area images is 100 (%). The image processing device according to any one of Configurations 1 to 2, wherein the plurality of second small area images are combined.
(Configuration 6)
The first blending section is
The plurality of second small area images at the first position are based on the synthesis rate such that the synthesis rate is 50 (%) in a straight line connecting the midpoints of adjacent sides of the plurality of second small area images. The image processing device according to any one of Configurations 1 and 2, wherein the second small area image and the plurality of second small area images at the second position are combined.
(Configuration 7)
The apparatus further includes similar small area extraction means for extracting a small area image similar to the first small area image from a second input image based on the first small area image, and the second small area image is based on the first small area image. The image processing device according to any one of configurations 1 to 6, wherein the image processing device is extracted by a similar small region extraction unit.
(Configuration 8)
comprising an adjustment unit that performs noise adjustment on the plurality of second small area images using noise suppression strengths that are different from each other between two or more of the second small area images;
The image processing device according to any one of configurations 1 to 7, wherein the plurality of second small area images are generated by a process including noise adjustment processing by the adjustment unit.
(Configuration 9)
The adjustment section further includes:
compositing processing to obtain a plurality of fourth small area images by composing the plurality of first small area images and a third small area image extracted from a third input image based on an inter-image synthesis rate; Execute processing including
9. The image processing device according to configuration 8, wherein the inter-image synthesis rate is different between two or more images of the plurality of first small area images and the third small area image.
(Configuration 10)
The adjustment section further includes:
performing noise suppression processing on the plurality of fourth small region images based on a predetermined suppression rate;
10. The image processing device according to configuration 9, wherein the predetermined suppression rate is different between two or more of the fourth small area images. This noise suppression is performed using, for example, a spatial filter.
(Configuration 11)
a second small area extraction unit that extracts a plurality of first small area images whose areas partially overlap each other from the first input image;
further comprising a second blending unit that performs blending processing on an overlapping region of the second small region image,
The second small area extraction unit
The plurality of first small area images at the first position, the plurality of first small area images at the second position, and the first input image are arranged perpendicularly to each other from the first position. the plurality of first small area images at the third position obtained by dividing into the predetermined size at a third position shifted by half the predetermined size in the first direction of the two directions; A plurality of input images at the fourth position obtained by dividing one input image into the predetermined size at a fourth position shifted by half the predetermined size in the second direction of the two directions from the first position. extracting the first small area image;
The second blending section is
The plurality of second small area images at each of the first position, the second position, the third position, and the fourth position become smaller as they approach the side thereof. Based on the set synthesis rate, the plurality of second small area images in the first position, the plurality of second small area images in the second position, and the plurality of second small area images in the third position are The image processing device according to any one of Configurations 1 to 10, wherein the second small area image and the plurality of second small area images at the fourth position are combined.
(Configuration 12)
a selection unit that allows a user to select which of the first small area extraction unit and the first blending unit, or the second small area extraction unit and the second blending unit to execute the process; The image processing device according to configuration 11, further comprising:
(Configuration 13)
Automatically determining which of the first small region extracting section and the first blending section or the second small region extracting section and the second blending section executes the process based on predetermined conditions. 12. The image processing apparatus according to configuration 11, further comprising a second selection section that allows a user to select an automatic switching mode to be switched.
(Configuration 14)
The image according to configuration 13, wherein the predetermined condition is a calculation amount of a series of processes including processing by the first small area extraction unit or processing by the second small area extraction unit. Processing equipment.
(Configuration 15)
14. The image processing apparatus according to configuration 13, wherein the predetermined condition is the number of frames or the number of pixels of the input image.
(Configuration 16)
Automatic switching mode is
When the number of frames of the input image is a predetermined number or more, the first small region extraction section and the first blending section are selected, and when the number of frames is less than the predetermined number, the second small region extraction is performed. 14. The image processing apparatus according to configuration 13, wherein the image processing apparatus is in a mode of automatically selecting the second blending section and the second blending section.
(Configuration 17)
The automatic switching mode is
When the number of pixels of the input image is greater than or equal to a predetermined number, the first small region extraction section and the second blending section are selected, and when the number of pixels is less than the predetermined number, the second small region extraction section and the second blending section are selected. 14. The image processing apparatus according to configuration 13, wherein the image processing apparatus is in a mode in which the second blending section is automatically selected.
(Configuration 18)
The apparatus is configured to automatically select which of the first small region extraction section and the first blending section, or the second small region extraction section and the second blending section, to execute the process. The image processing device according to configuration 11, characterized in that:

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications and changes can be made within the scope of the gist thereof. For example, the present invention provides a program that implements one or more of the functions of the embodiments described above to a system or device via a network or a recording medium, and a processor of a computer in the system or device reads the program. It can also be realized by executing processing. The present invention can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

100 Imaging device 101 Control unit 102 ROM
103 RAM
104 Optical system 105 Imaging unit 106 A/D conversion unit 107 Image processing unit 108 Recording unit 109 Communication unit 110 Display unit 111 Instruction input unit 112 Bus 201 Small area extraction unit 202 Similar small area extraction unit 203 Combining unit 204 Noise suppression unit 205 Two-position small area blending section 801 Four-position small area blending section

Claims (20)

a first small region extraction unit that extracts a plurality of first small region images whose regions partially overlap each other from the first input image;
a first blending unit that performs a blending process on an overlapping region of a second small area image that is at least one of the first small area image or a small area image obtained based on the first small area image; and,
The first small area extraction unit is
a plurality of first small area images at the first position obtained by dividing the first input image into predetermined sizes at a first position; extracting a plurality of first small area images at the second position obtained by dividing into the predetermined size at a second position shifted by half the predetermined size in two directions perpendicular to each other;
The first blending section is
Based on a combination rate that is set to decrease as the plurality of second small area images approach the apex of the plurality of second small area images at each of the first position and the second position, the first An image processing apparatus characterized in that the plurality of second small area images at the position and the plurality of second small area images at the second position are combined. The first blending section is
Based on the synthesis rate, the plurality of second small area images at the first position and the plurality of second small area images at the second position are combined by weighted addition. The image processing device according to claim 1. The first blending section is
The plurality of second small area images at the first position are based on a synthesis rate that is set to vary depending on the distance from a predetermined position in the image to the vertex for the plurality of second small area images. The image processing device according to any one of claims 1 to 2, wherein a region image and the plurality of second small region images at the second position are combined. 4. The image processing apparatus according to claim 3, wherein the distance is an urban distance. The first blending section is
The plurality of second small area images at the first position and the second small area images at the second position such that the total value of the synthesis rate for the plurality of second small area images is 100 (%). The image processing apparatus according to claim 1 , wherein the plurality of second small area images are combined. The first blending section is
The plurality of second small area images at the first position are based on the synthesis rate such that the synthesis rate is 50 (%) in a straight line connecting the midpoints of adjacent sides of the plurality of second small area images. 3. The image processing apparatus according to claim 1, wherein the second small area image and the plurality of second small area images at the second position are combined. further comprising a similar small region extraction unit that extracts the second small region image similar to the first small region image from a second input image based on the first small region image,
3. The image processing device according to claim 1, wherein the second small region image is extracted by the similar small region extracting section. comprising an adjustment unit that performs noise adjustment on the plurality of second small area images using noise suppression strengths that are different from each other between two or more of the second small area images;
The image processing apparatus according to any one of claims 1 to 2, wherein the plurality of second small area images are generated by a process including noise adjustment processing by the adjustment unit. The adjustment section further includes:
compositing processing to obtain a plurality of fourth small area images by composing the plurality of first small area images and a third small area image extracted from a third input image based on an inter-image synthesis rate; Execute processing including
9. The inter-image synthesis rate is different between two or more images of the plurality of first small area images and the third small area image. Image processing device. The adjustment section further includes:
performing noise suppression processing on the plurality of fourth small region images based on a predetermined suppression rate;
The image processing apparatus according to claim 9, wherein the predetermined suppression rate is different between two or more of the fourth small area images. a second small area extraction unit that extracts a plurality of first small area images whose areas partially overlap each other from the first input image;
further comprising a second blending unit that performs blending processing on an overlapping region of the second small region image,
The second small area extraction unit
The plurality of first small area images at the first position, the plurality of first small area images at the second position, and the first input image are arranged perpendicularly to each other from the first position. the plurality of first small area images at the third position obtained by dividing into the predetermined size at a third position shifted by half the predetermined size in the first direction of the two directions; A plurality of input images at the fourth position obtained by dividing one input image into the predetermined size at a fourth position shifted by half the predetermined size in the second direction of the two directions from the first position. extracting the first small area image;
The second blending section is
The plurality of second small area images at each of the first position, the second position, the third position, and the fourth position become smaller as they approach the side thereof. Based on the set synthesis rate, the plurality of second small area images in the first position, the plurality of second small area images in the second position, and the plurality of second small area images in the third position are The image processing apparatus according to claim 1 , wherein the second small area image and the plurality of second small area images at the fourth position are combined. a selection unit that allows a user to select which of the first small area extraction unit and the first blending unit, or the second small area extraction unit and the second blending unit to execute the process; The image processing device according to claim 11, further comprising: Automatically determining which of the first small region extracting section and the first blending section or the second small region extracting section and the second blending section executes the process based on predetermined conditions. The image processing apparatus according to claim 11, further comprising a second selection section that allows a user to select an automatic switching mode to be switched. 14. The predetermined condition is the amount of calculation for a series of processes including processing by the first small region extracting section or processing by the second small region extracting section. Image processing device. The image processing apparatus according to claim 13, wherein the predetermined condition is the number of frames or the number of pixels of the input image. Automatic switching mode is
When the number of frames of the input image is a predetermined number or more, the first small region extraction section and the first blending section are selected, and when the number of frames is less than the predetermined number, the second small region is selected. 14. The image processing apparatus according to claim 13, wherein the image processing apparatus is in a mode in which the extraction section and the second blending section are automatically selected. The automatic switching mode is
When the number of pixels of the input image is greater than or equal to a predetermined number, the first small region extraction section and the second blending section are selected, and when the number of pixels is less than the predetermined number, the second small region extraction section and the second blending section are selected. 14. The image processing apparatus according to claim 13, wherein the image processing apparatus is in a mode in which the second blending section is automatically selected. The apparatus is configured to automatically select which of the first small region extraction section and the first blending section, or the second small region extraction section and the second blending section, to execute the process. The image processing device according to claim 11. A method for controlling an image processing device, the method comprising:
a small region extraction step of extracting a plurality of first small region images whose regions partially overlap each other from the input image;
a blending step of performing a blending process on an overlapping region of a second small region image that is at least one of the first small region image or a small region image obtained based on the first small region image; have,
The small region extraction step includes:
a plurality of first small area images at the first position obtained by dividing the input image into predetermined sizes at a first position; a step of extracting a plurality of first small area images at the second position obtained by dividing into the predetermined size at a second position shifted by half the predetermined size;
The blending step includes:
Based on a combination rate that is set to decrease as the number approaches the apex of the plurality of second small area images at each of the first position and the second position, A method for controlling an image processing apparatus, characterized in that the method is a step of composing the plurality of second small area images at a first position and the plurality of second small area images at the second position. . A program that causes a computer to execute a control method for an image processing device,
The control method includes:
a small region extraction step of extracting a plurality of first small region images whose regions partially overlap each other from the input image;
a blending step of performing a blending process on an overlapping region of a second small region image that is at least one of the first small region image or a small region image obtained based on the first small region image; have,
The small region extraction step includes:
a plurality of first small area images at the first position obtained by dividing the input image into predetermined sizes at a first position; a step of extracting a plurality of first small area images at the second position obtained by dividing into the predetermined size at a second position shifted by half the predetermined size;
The blending step includes:
Based on a combination rate that is set to decrease as the number approaches the apex of the plurality of second small area images at each of the first position and the second position, A program characterized in that the program is a step of composing the plurality of second small area images at a first position and the plurality of second small area images at the second position.

Images