JP2016167774A - Image signal processor, image signal processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image signal processor capable of achieving filter processing coping with a change in a field angle while suppressing a scale of a filter processing circuit.SOLUTION: The image signal processor includes: a filter processing section 4 for performing filter processing by controlling a filter coefficient for every nine frames in an order of photographing frame images for frame images input in the order of photographing; a synthesis section 8 for synthesizing every nine frames by sequentially shifting frame images after the filter processing by the filter processing section 4 by one frame; and a synthesis ratio generating section 7 for controlling synthesis of frame images by the synthesis section 8 on the basis of a field angle when a frame image is photographed.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image signal processing apparatus, an image signal processing method, and a program for processing an image signal and the like.

  Conventionally, there is a photographing method in which only the background is blurred and the subject is more conspicuous by photographing with a shallow depth of field. The depth of field can be reduced by, for example, decreasing the F value, increasing the focal length, and reducing the shooting distance. However, a lens having a large aperture or an optical system having a long focal length is expensive, and the optical system itself. The size of the will also increase. For this reason, many conventional imaging devices mounted on inexpensive and compact compact digital cameras, portable information terminals, etc., do not easily reduce the depth of field.

  For this reason, some compact digital cameras and imaging devices such as portable information terminals employ a technique that obtains a blurring effect equivalent to a captured image with a shallow depth of field by digital filter processing. For example, Patent Document 1 discloses a technique for generating a blurred image with a large blur amount by a small size filter process by applying a small size filter process to a reduced image and then enlarging the reduced image to the original image size. ing.

JP 2004-102904 A

  However, with the technique disclosed in Patent Document 1, since the filter size is small, it is difficult to realize a beautiful blurred image such as an image shot with a shallow depth of field. For example, when it is desired to obtain an image in which the point light source is greatly rounded and blurred, the high-frequency component is lost due to the reduction of the image in the small size filter processing disclosed in Patent Document 1, so that the details of the shape of the round blur are generated. I can't express enough. Although it is conceivable to perform filter processing without reducing the image, in order to realize a blurred image with a large blur amount, such as an image taken with a shallow depth of field, for example, about 50 taps. A multi-tap two-dimensional filter is required. Such a multi-tap two-dimensional filter has a very large circuit scale.

  On the other hand, recent compact digital cameras and imaging devices of portable information terminals include a zoom optical system that can realize a certain zoom magnification. For this reason, for example, when the angle of view is changed by the zoom optical system, it is desired to realize a blurred image corresponding to the angle of view. However, it is difficult to generate a blurred image corresponding to a change in the angle of view by digital filter processing.

  The present invention has been made in view of such problems, and an image signal processing device, an image signal processing method, and a program for realizing filter processing corresponding to a change in angle of view while suppressing the scale of a filter are provided. The purpose is to provide.

  The image signal processing apparatus according to the present invention includes a filter unit that performs a filter process for controlling a filter coefficient in the imaging order of the frame image for each predetermined number of frames with respect to a frame image input in the imaging order, and a filter process performed by the filter unit A synthesizing unit that synthesizes the subsequent frame image for each predetermined number of frames while sequentially shifting the frame image by one frame, and the synthesis of the frame image by the synthesizing unit based on an angle of view when the frame image is captured. And a synthesis control means for controlling processing.

  According to the present invention, it is possible to realize the filter processing corresponding to the change in the angle of view while suppressing the scale of the filter.

It is a figure which shows schematic structure of the imaging device of 1st Embodiment. It is a figure explaining the tap number and filter coefficient of a division | segmentation filter. It is a figure explaining the reference range of a horizontal and vertical 9 tap spatial filter. It is a flowchart used for outline | summary description of a division | segmentation filter process. It is a figure explaining a division | segmentation filter process and a synthetic | combination (addition) process. It is a figure explaining the division | segmentation filter process with respect to each flame | frame of a moving image. It is a figure explaining the change of the magnitude | size of the image of each flame | frame by a view angle change. It is a figure explaining the relationship between a composition ratio and zoom magnification (view angle). It is a flowchart of the filter process and the synthesis process of the first embodiment. It is a figure explaining the example where a reference range is insufficient by a view angle change. It is a figure which shows schematic structure of the imaging device of 2nd Embodiment. It is a figure explaining operation | movement of the image expansion part of 2nd Embodiment. It is a figure which shows schematic structure of the imaging device of 3rd Embodiment. It is a figure explaining operation | movement of the reference | standard frame selection part of 3rd Embodiment. It is a figure which shows schematic structure of the imaging device of 4th Embodiment. It is a figure explaining the relationship between the shift amount of a reading position, and magnification (view angle). It is a flowchart of the filter process and synthesis process of the fourth embodiment. It is a figure which shows schematic structure of the imaging device of 5th Embodiment. It is a figure which shows schematic structure of the imaging device of 6th Embodiment.

<First Embodiment>
FIG. 1 is a diagram illustrating a schematic configuration of an imaging apparatus according to a first embodiment which is an example of an image signal processing apparatus. In FIG. 1, a system control unit 11 controls the entire imaging apparatus of the present embodiment. Although not shown in FIG. 1, the imaging apparatus also includes various operation units provided in a general camera such as a shutter button.

  The optical system 1 is configured to include a zoom lens, a focus lens, a diaphragm, and the like, and a subject image set to an appropriate brightness by driving the diaphragm and the lens by a control signal from the system control unit 11. Is imaged on the image sensor 2. In the present embodiment, the system control unit 11 can set the angle of view (optical zoom magnification) by controlling the zoom lens, and corresponds to the angle of view to the composition ratio generation unit 7 described later. Optical zoom information can be output. The image pickup device 2 is driven by a drive pulse controlled by the system control unit 11, converts the subject image into an electric signal by photoelectric conversion, and outputs it as an image pickup signal. The development processing unit 3 performs a development process for converting an image pickup signal from the image sensor 2 into a luminance and color difference signal, and outputs the image signal of the luminance and color difference after the development process to the filter processing unit 4. The development processing unit 3 also outputs an image signal to the memory 6. Therefore, the memory 6 holds the image signal from the development processing unit 3 in addition to the image signal after the filter processing by the filter processing unit 4.

  The filter processing unit 4 performs filter processing on the image signal from the development processing unit 3 using the filter coefficient supplied from the filter coefficient control unit 5. The filter coefficient control unit 5 outputs the filter coefficient used in the filter process to the filter processing unit 4 under the control of the system control unit 11.

  Details of the filter processing by the filter processing unit 4 will be described later. The image signal filtered by the filter processing unit 4 is held in the memory 6, read out, and output to the synthesis unit 8.

  The composition unit 8 controls the pixel position read from the memory 6 as described later under the control of the system control unit 11, and composes the image by adding the pixels read from the memory 6. The image formed by the synthesis by this addition becomes a blurred image by the filter processing in the filter processing unit 4.

  The main subject determination unit 9 detects an image portion of the main subject (hereinafter referred to as a main subject image) from the image signal output from the development processing unit 3 and held in the memory 6. Various methods are known for detecting the main subject image from the image signal, and any of them may be used in the present embodiment. The main subject determination unit 9 determines, for example, a subject image portion that is close to the center of the image in the image signal and has a large size as the main subject image, and determines the size of the main subject image and the position of the image signal in the image. The information is output to the background subject composition unit 10.

  Based on the size and position information of the main subject image detected by the main subject determination unit 9, the background subject synthesis unit 10 and the image output from the development processing unit 3 and held in the memory 6 and the blurred image from the synthesis unit 8 And synthesize. Specifically, the background subject composition unit 10 extracts the main subject image from the image signal read from the memory 6 based on the size and position information of the main subject image. Then, the background subject composition unit 10 composes the extracted main subject image by overwriting the same position of the image supplied from the composition unit 8, that is, the position corresponding to the position information of the main subject image.

  Here, the image from the synthesizing unit 8 is an image obtained by performing blur processing on the entire image. On the other hand, the input image to the background subject composition unit 10 is an output image of the development processing unit 3 and is a sharp image signal that has not been subjected to the blurring process. Then, the background subject composition unit 10 extracts a subject image from the image and composes it by overwriting the blurred image from the composition unit 8. As a result, the image output from the background subject synthesizing unit 10 is a sharp and conspicuous image while the background is blurred.

  The image signal after the background subject image combining process by the background subject synthesizing unit 10 is sent to, for example, a recording unit (not shown), recorded on a recording medium represented by an SD card, for example, and to a display unit (not shown), for example. Sent and displayed on the screen.

  Next, a basic process flow in the blur image generation process performed by the filter processing unit 4, the memory 6, and the synthesis unit 8 will be described. In general, a filter processing circuit for generating a blurred image can be realized with a relatively small circuit scale when the blur amount is small, but the circuit scale becomes large when the blur amount is large. In this embodiment, in order to form a blurred image with a large blurring amount without increasing the circuit scale, the filter convolution operation is divided (hereinafter referred to as divided filter processing).

  The imaging apparatus according to the present embodiment is held in the memory 6 as an example of a filter synthesis unit, a filter processing unit 4 that performs division filter processing, a memory 6 that holds a plurality of images after filtering by the filter processing unit 4, and the memory 6. And a combining unit 8 that combines the images by pixel addition. With this configuration, the imaging apparatus according to the present embodiment can generate a blurred image having a large amount of blur equivalent to a large filter having a large number of taps with a small circuit scale filter having a small number of taps. Yes.

  Here, as an example, the division filter processing in the case of a spatial filter having a plurality of taps in the horizontal and vertical directions (for example, 9 taps in the horizontal and vertical directions) will be described with reference to FIGS. The number of taps of the filter processing unit 4 is an example, and is not limited to 9 taps.

  FIG. 2 is a diagram illustrating an example of filter coefficients of a spatial filter having 9 taps in the horizontal direction and 9 taps in the vertical direction. K0 to k8 in FIG. 2 indicate filter coefficients, and indicate the filter coefficients at respective positions when the horizontal direction and the vertical direction of the spatial filter are separated into three taps. FIG. 3 shows a spatial filter reference range of 9 taps in the horizontal direction and 9 taps in the vertical direction at a certain pixel position (i, j) whose (x, y) coordinates are represented by (i, j) in the image. FIG. The filtering process is performed using an arithmetic expression as shown in, for example, Expression (1).

  Here, “x” and “y” in the equation (1) and each equation described later are the coordinates of each pixel of the image shown in FIG. 3, “i” is the horizontal address (x coordinate) of the pixel of interest in the image, “J” represents the vertical address (y coordinate) of the pixel of interest in the image, “a” represents the filter coefficient, and “I” represents the pixel value. Then, “p” in Expression (1) is an image obtained as a result of filtering the target pixel position (i, j) as shown in FIG.

  Next, with reference to the flowchart of FIG. 4, a basic flow of the division filter processing in which the convolution operation is divided into 3 taps in the horizontal direction and 3 taps in the vertical direction, and the synthesis processing by pixel addition of the image after the filter processing is performed. explain. Although details will be described later, the imaging apparatus of the present embodiment performs processing slightly different from the flowchart of FIG. 4 when performing filtering processing on a moving image. The flow chart of FIG. 4 exemplifies the case where one image is subjected to the division filter process in order to explain the basic flow of the division filter process and the synthesis process. In the description of FIG. 4, the system control unit 11 controls the filter processing unit 4 based on the filter coefficient output control from the filter coefficient control unit 5, controls writing and reading of the memory 6, and combines by pixel addition in the combining unit 8. The example which decided to perform control is given.

  In the flowchart of FIG. 4, the system control unit 11 first sets an initial value “0” to a variable n as a process of step S <b> 100. The variable n corresponds to the number of divisions of the division filter. In the case of the present embodiment, the convolution operation is divided into 3 taps in the horizontal direction and 3 taps in the vertical direction, and the number of filter divisions is 9, so that the variable n is a value from “0” to “8”. Become. After step S100, the system control unit 11 advances the process to step S101.

  In step S <b> 101, the system control unit 11 inputs the image signal from the development processing unit 3 to the filter processing unit 4. After step S101, the system control unit 11 advances the process to step S102.

  In step S102, the system control unit 11 controls the filter coefficient control unit 5 to cause the filter processing unit 4 to output the filter coefficient. The filter processing unit 4 at this time generates a first image p0 after the filter processing by performing a filter operation as shown in, for example, Expression (2). Here, the calculation of the expression (2) is performed on the entire image of the image signal. Therefore, the first image p0 is filtered using the filter coefficients k0 on the upper left in FIG. This is a multiplied image signal.

  After step S102, the system control unit 11 advances the process to step S103. In step S103, the system control unit 11 holds the signal of the first image p0 in the memory 6. After step S103, the system control unit 11 advances the process to step S104.

  In step S <b> 104, the system control unit 11 determines whether the filter calculation for all the division numbers has been completed. Specifically, in this embodiment, since the variable n can take a value of “0” to “8”, the system control unit 11 calculates the total number of divisions when the variable n is “8”. Is determined to be complete. In step S104, the system control unit 11 advances the process to step S105 when it is determined that the filter calculation for all the division numbers has not been completed, and advances the process to step S106 when it is determined that the calculation has been completed. .

  In step S <b> 105, the system control unit 11 sets “n + 1” to the variable n in order to move to the filter operation at the next division position. Since the variable n at this time is the initial value “0” set in step S100, the variable n is set to “1” in step S105. After step S105, the system control unit 11 returns the process to step S101.

  Returning to step S <b> 101, the system control unit 11 inputs the image signal from the development processing unit 3 to the filter processing unit 4. Here, to simplify the description, it is assumed that the image input to the filter processing unit 4 at this time is the same image as the previous input image. When the processing proceeds to the next step S102, the system control unit 11 controls the filter coefficient control unit 5 to cause the filter processing unit 4 to output the filter coefficient. At this time, the filter processing unit 4 generates a second image p1 by performing a filter operation as shown in, for example, Expression (3). The calculation of Expression (3) is performed on the entire image of the image signal, and therefore the second image p1 is obtained by filtering the entire image using each filter coefficient k1 on the center in FIG. This is an image signal.

  When the processing proceeds to the next step S103, the system control unit 11 causes the memory 6 to hold the signal of the second image p1. In the next step S104, the system control unit 11 determines whether or not the filter calculation for all the division numbers has been completed. In step S104, the system control unit 11 advances the process to step S105 when it is determined that the filter calculation for all the division numbers has not been completed, and advances the process to step S106 when it is determined that the calculation has been completed. . At this time, the system control unit 11 advances the process to step S105, sets the variable n to “2”, and returns the process to step S101.

  Similarly, the system control unit 11 repeats the process from step S101 to S104, and further returns from step S104 to step S105 and then returns to step S101 until it is determined in step S104 that the filter operation for all the division numbers has been completed. In the present embodiment, since the number of filter divisions is 9 as described above, the variable “n” is incremented from “0” to “8”. After the variable n is set to “2” in step S105, when the above-described processing is performed and the variable n is sequentially incremented, in step S102, the operations shown in the equations (4) to (10) are filtered. Part 4 will be performed. By the filter calculation shown in the equations (4) to (10), the filter processing unit 4 outputs the signals of the third image p2 to the ninth image p8. In step S103, the signals of the third image p2 to the ninth image p8 are held in the memory 6.

  Then, when it is determined in step S104 that the calculation of the total number of divisions has been completed and the process proceeds to step S106, the system control unit 11 controls the synthesis unit 8 to calculate and store in the memory 6 so far. The signals of the first image p0 to the ninth image p8 are read out. At this time, the synthesis unit 8 controls the pixel position of each image read from the memory 6 under the control of the system control unit 11, and reads each pixel at a predetermined position (a reference position described later) from the memory 6. After step S106, the system control unit 11 advances the process to step S107.

  In step S107, the system control unit 11 controls the combining unit 8 to read out pixels from the first image p0 to the ninth image p8 from the memory 6 and combine them by addition. That is, the synthesis unit 8 at this time reads out the pixels of the first image p0 to the ninth image p8 from the memory 6 under the control of the system control unit 11, and adds the read pixels. To compose the image. The image signal formed by the synthesis by the pixel addition in the synthesis unit 8 becomes a signal of the blurred image after the blurring process by the division filter process. The signal of the blurred image after the filter processing is sent to the background subject composition unit 10 described later.

  With reference to FIG. 5A to FIG. 5J, a description will be given of a synthesis operation by pixel division of the first image p <b> 0 to the ninth image p <b> 8 performed by the division filter processing by the filter processing unit 4 and the synthesis unit 8. . 5A to 5J, the pixel at the position (i, j) is the target pixel, and the region 1000 is referred to by the spatial filter of 9 taps in the horizontal direction and 9 taps in the vertical direction with respect to the target pixel. The range is shown. The position of the center pixel in the area 1001 in FIG. 5A indicates the reference position on the image with respect to the target pixel. Similarly, the positions of the central pixels in the regions 1002 to 1009 in FIGS. 5B to 5I indicate the reference positions on the image with respect to the target pixel. FIG. 5A shows the reference range of the target pixel and the spatial filter in the first image p0 and the reference position on the image with respect to the target pixel. Similarly, FIG. 5B shows the second image p1, FIG. 5C shows the third image p2, FIG. 5D shows the fourth image p3, and FIG. 5E shows the fifth image p4. It is a figure about. FIG. 5 (f) shows the sixth image p5, FIG. 5 (g) shows the seventh image p6, FIG. 5 (h) shows the eighth image p7, and FIG. 5 (i) shows the ninth image p8. It is. FIG. 5J illustrates, for example, the fifth image p4, and the regions 1001 to 1009 in the other first to fourth, sixth to ninth images p0 to p3 and p4 to p8 are shown as fifth. It is the figure arranged on the image p4. As shown in FIG. 5J, the sizes of the ranges referred to in the filter processing in the first to ninth images p0 to p8 are the same.

  Here, in order to obtain an effect equivalent to a spatial filter of 9 taps in the horizontal and vertical directions, the position (i-3, j-3) from the first image p0 with respect to the target pixel position (i, j). A pixel with a spatial filter of 3 taps in the horizontal and vertical directions centered on is required. Further, the position (i, j-3) from the second image p1, the position (i + 3, j-3) from the third image p2, and the position (i-3, j) from the fourth image p3. Pixels with a 3-tap spatial filter centered in the horizontal and vertical directions, respectively, are required. Further, the position is (i, j) from the fifth image p4, the position (i + 3, j) from the sixth image p5, and the position (i-3, j + 3) from the seventh image p6. Pixels with a spatial filter with 3 taps in the horizontal and vertical directions are required. Similarly, from the eighth image p7, a pixel having been subjected to a spatial filter of 3 taps in the horizontal and vertical directions centered at the position (i, j + 3) and from the ninth image p8 at the position (i + 3, j + 3), respectively. Necessary. The synthesizing unit 8 acquires and adds each pixel value obtained by applying a spatial filter of three taps in the horizontal and vertical directions around the position corresponding to the target pixel position in the first image p0 to the ninth image p8. The synthesis process is performed.

  As described above, the filter processing unit 4 divides the convolution operation into 3 taps in the horizontal and vertical directions and performs filter processing to generate the first to ninth images p0 to p8. In the first to ninth images p0 to p8, pixel values corresponding to the target pixel are added. As a result, although the circuit scale is small, the same image signal as when a spatial filter with 9 taps in the horizontal direction and 9 taps in the vertical direction is applied can be obtained.

  2 to 5, a single image is subjected to the filter processing corresponding to the number of divisions while changing the filter coefficient, and the first to ninth after the filter processing corresponding to the number of divisions. The outline of the division filter is described by giving an example of adding and synthesizing the images. In this way, when the number of times of filter processing corresponding to the number of divisions is performed on one image, the processing time until the final blurred image is obtained becomes long. Therefore, for example, when dividing filter processing is applied to a moving image, the problem of cost and processing time must be solved. In other words, in order to perform the filter processing corresponding to the number of divisions on one frame image of a moving image, an expensive arithmetic device capable of high-speed processing is required, while on the other hand, an inexpensive arithmetic operation with a low processing speed is required. In the apparatus, there is a possibility that the processing does not end within one frame time.

  Here, as an example of the division filter processing for the moving image, as shown in FIG. 6, one coefficient of the division filter processing is used for each frame in which the moving image is captured and input in the imaging order. Let's consider applying filter processing in order. In FIG. 6, input images N-4, N-3,..., N + 5, N + 6 are frame images that are output from the development processing unit 3 and input to the filter processing unit 4 in the order in which moving images are captured. Each of these images is arranged in time order. The filter coefficients k0, k1,..., K8 are the coefficients of the division filter in FIG. The frame image p0a is an image after the filter processing with the filter coefficient k0 is performed on the frame image of the input image N-4. Similarly, the frame images p1a,..., P7a, p8a are respectively corresponding to the corresponding filter coefficients k1,. It is an image after each filter process by k7 and k8 is made. When the filter process using the filter coefficient k8 is performed, the subsequent frame is subjected to the filter process using the filter coefficients k0 to k8 as described above. In the example of FIG. 6, the input image N + 5 becomes a frame image p9a by being subjected to the filtering process by the filter coefficient k0, and the input image N + 6 becomes a frame image p10a by being subjected to the filtering process by the filter coefficient k1.

  Then, a frame image of a predetermined number of frames (9 frames) corresponding to the number of divisions is synthesized by pixel addition as described above in the synthesis unit 8 while being sequentially shifted by one frame. When the synthesizing unit 8 finishes synthesizing the nine frames of the frame images p0a to p8a, the next nine frames of the frame images p1a to p9a are shifted, and then the frame images p2a to p10a are sequentially shifted one frame at a time. While processing. Of the nine frames of images, if a frame that is temporally reference at the time of combining is called a reference frame, and other frames are called combining frames, the combining unit 8 applies the pixel of interest of the reference frame image to the target pixel. Then, the pixels obtained from the composite side frame image are added. As an example, the reference frame can be a temporally central frame among the nine frames arranged in time order. In the case of the example in FIG. 6, the frame image of the input image N is the reference frame image. As shown in FIG. 6, by performing a filter process using one coefficient of the division filter process for each frame of the moving image, even when using an inexpensive arithmetic device having a low processing speed, It is considered that a blurred image of a moving image can be generated.

  By the way, for example, when moving image capturing is performed while changing the angle of view by the zoom optical system, and frame images captured at different angles of view are input to the filter processing unit 4, blurring may be insufficient. On the contrary, it may become too blurry. That is, when the filter processing as illustrated in FIG. 6 is performed using frame images captured at different angles of view, for example, the reference frame image has a wider or narrow reference range in which the filtering processing of the synthesis-side frame image is performed. It is conceivable that the filtering process will be performed. Further, when the angle of view of the frame image is different, the position of the center of gravity of the convolution calculation may be shifted. In these cases, there is a possibility that a blurred image with insufficient blurring is generated, or conversely, a blurred image with excessive blurring is generated.

  Hereinafter, with reference to FIG. 7, using each frame image of the moving image captured while changing the angle of view by the zoom optical system, the reference range when the filtering process and the combining process as shown in FIG. 6 are performed, A shift in the center of gravity position of the convolution calculation will be described.

  FIG. 7A to FIG. 7I show examples of each frame of a moving image captured while zooming in while gradually narrowing the angle of view (gradually increasing the zoom magnification). 7A to 7I, the pixel at the position (i, j) is the target pixel, and the region 2000 is a space of 9 taps in the horizontal direction and 9 taps in the vertical direction with respect to the target pixel on each image. The range to which the filter refers is shown. The position of each central pixel in the areas 2001 to 2009 indicates the reference position on the image with respect to the target pixel. FIG. 7A is a diagram illustrating the reference range of the target pixel and the spatial filter in the first frame image p0a and the reference position on the image with respect to the target pixel. Similarly, FIG. 7B shows the second frame image p1a, FIG. 7C shows the third frame image p2a, FIG. 7D shows the fourth frame image p3a, and FIG. FIG. 5 is a diagram for a frame image p4a of FIG. Fig. 7 (f) shows the sixth frame image p5a, Fig. 7 (g) shows the seventh frame image p6a, Fig. 7 (h) shows the eighth frame image p7a, and Fig. 7 (i) shows the ninth frame image. It is a figure about p8a. FIG. 7 (j) shows regions 2001 to 2009 when the fifth frame image p4a is a reference frame image and the other composite side frame images p0a to p3a and p5a to p8a are matched to the angle of view of the reference frame image. It is the figure arranged on the reference frame image.

  As shown in FIG. 7J, when the angle of view of each frame image is different, the size of each region 2001 to 2009 of the compositing side frame image with respect to the reference frame image varies depending on the angle of view (zoom magnification). I understand that. In other words, when the reference frame and the compositing side frame have the same angle of view, the size and position of each area are as shown in the areas 1001 to 1009 in FIG. If they are different, areas 2001 to 2009 in FIG. That is, in such a case as shown in FIG. 7J, the degree of blurring changes.

  For this reason, in the imaging apparatus of the present embodiment, the composition ratio generation unit 7 which is an example of the composition control unit generates a composition ratio according to the angle of view (zoom magnification), and according to the composition ratio. The composition process in the composition unit 8 is controlled. Thereby, the imaging apparatus of this embodiment reduces the fluctuation | variation of the blurring condition when a field angle changes by the magnification change of an optical zoom at the time of imaging of a moving image.

  Hereinafter, the control operation of the synthesis process of the synthesis unit 8 by the synthesis ratio generation unit 7 will be described. For example, when the angle of view of the compositing side frame image is narrower than the base frame image (the zoom magnification is high), the reference range is narrowed as in the region 2009 in FIG. Will decrease. On the other hand, when the angle of view of the compositing side frame is wider than the base frame (the zoom magnification is low), the reference range becomes wide as in the region 2001 in FIG. Become.

  Thus, in the present embodiment, the system control unit 11 generates optical zoom information corresponding to the angle of view of each frame image (the zoom magnification set for the zoom lens) when the compositing unit 8 performs the compositing process. Send to part 7.

  The composition ratio generation unit 7 refers to the optical zoom information in each frame. If the angle of view of the composition side frame is narrower than the base frame (the zoom magnification is high), as illustrated in FIG. The composition ratio information that increases the weight for the pixels of the side frame is sent to the composition unit 8. As a result, the composition unit 8 performs composition in which the weights of the pixels of the composition side frame image are increased when the composition side frame image is composed with the reference frame image. That is, in this case, the ratio of the pixels of the combined frame image to the reference frame image is increased in the combined frame image, and the blur amount increases.

  On the other hand, when the angle of view of the compositing side frame is wider (low zoom magnification) than the reference frame, the compositing ratio generation unit 7 assigns weights to the pixels of the compositing side frame image as shown in FIG. The composition ratio information to be lightened is sent to the composition unit 8. As a result, when the combining unit 8 combines the combining-side frame image with the reference frame image, combining is performed with lighter weights for the pixels of the combining-side frame image. That is, in this case, the ratio of the pixels of the reference frame image to the combined frame image is increased in the combined frame image, and excessive blurring can be prevented.

  As described above, in the imaging apparatus according to the present embodiment, when the angle of view of each frame is different due to the change in magnification of the optical zoom, the composition ratio (weight) is changed, so that the degree of blurring can be balanced. It is possible to reduce fluctuations in the degree of blurring.

  Next, FIG. 9 is a flowchart when the filter processing unit 4 performs the filter processing as shown in FIG. 6 on each frame image of the moving image, and the combining unit 8 performs the combining process according to the combining ratio. Indicates. The flowchart of FIG. 9 is executed by the system control unit 11 performing control of the filter processing unit 4, control of the memory 6, control of synthesis ratio generation in the synthesis ratio generation unit 7, and synthesis control of the synthesis unit 8.

  In FIG. 9, the system control unit 11 first sets an initial value “0” to a variable n as a process of step S <b> 200. The variable n corresponds to the number of divisions of the division filter, which in other words corresponds to the predetermined number of frames described in FIG. In the case of FIG. 9, the number of filter divisions is 9, and 9 frames of images are synthesized, so the variable n takes values from “0” to “8”. After step S200, the system control unit 11 advances the process to step S201.

  In step S <b> 201, the system control unit 11 inputs a frame image signal of the first frame, for example, from the development processing unit 3 to the filter processing unit 4. After step S201, the system control unit 11 advances the process to step S202.

  In step S202, the system control unit 11 controls the filter coefficient control unit 5 to output the filter coefficient to the filter processing unit 4, performs the filter operation of the above-described equation (2), and converts the filter signal into the first frame image signal. Filter processing is performed to generate a first frame image p0a. In this case, the calculation of Expression (2) is performed on the entire frame image. The first frame image p0a is an image obtained by filtering the entire frame image using the filter coefficient k0 in FIG.

  After step S202, the system control unit 11 advances the process to step S203. In step S203, the system control unit 11 holds the signal of the first frame image p0a in the memory 6. After step S203, the system control unit 11 advances the process to step S204.

  In step S <b> 204, the system control unit 11 determines whether or not a predetermined number of filter operations corresponding to the total number of divisions have been completed. Specifically, in this embodiment, the filter operation for 9 frames corresponding to 9 divisions is performed, and the variable n can take a value of “0” to “8”. Is determined to be complete when the value is “8”. In step S204, the system control unit 11 advances the process to step S205 when determining that the filter operation of each frame image corresponding to the total number of divisions is not completed, and when determining that the filter operation is completed. The process proceeds to step S206.

  In step S <b> 205, the system control unit 11 sets “n + 1” to the variable n in order to move to the filter operation of the next second frame. Since the variable n at this time is the initial value “0” set in step S200, the variable n is set to “1” in step S205. After step S205, the system control unit 11 returns the process to step S201.

  Returning to step S <b> 201, the system control unit 11 inputs the image signal of the second frame output from the development processing unit 3 to the filter processing unit 4. When the processing proceeds to the next step S202, the system control unit 11 controls the filter coefficient control unit 5 to cause the filter processing unit 4 to output the filter coefficient. The filter processing unit 4 at this time performs a filter operation as shown in the above-described equation (3) to generate the second frame image p1a. In this case as well, the calculation of Equation (3) is performed on the entire frame image, and therefore the second frame image p1a is filtered on the entire frame image using the filter coefficient k1 of FIG. It becomes an image.

  When the processing proceeds to the next step S203, the system control unit 11 causes the memory 6 to hold the signal of the second frame image p1a. In the next step S204, the system control unit 11 determines whether or not the filter calculation of each frame image corresponding to the total number of divisions has been completed. In step S204, the system control unit 11 proceeds to step S205 when determining that the filter operation of the frame image corresponding to the total number of divisions has not been completed, and performs processing when determining that it has been completed. Advances to step S206. At this time, the system control unit 11 advances the process to step S205, sets the variable n to “2”, and returns the process to step S201.

  Similarly, the system control unit 11 determines that the process of returning from step S201 to S204, step S204 to step S205, and returning to step S201 has been completed in step S204 for each frame image corresponding to the total number of divisions. Repeat until In this embodiment, since the number of filter divisions is 9 and the number of frames is 9 as described above, the variable “n” is incremented from “0” to “8”. If the variable n is sequentially incremented after the variable n is set to “2” in step S205, the frame images from the third frame to the ninth frame are sequentially displayed in step S201. Will be entered. In step S202, the filter processing unit 4 sequentially performs the operations shown in the equations (4) to (10) described above in the order of the third to ninth frames. The filter processing unit 4 outputs the signals of the third frame image p2a to the ninth frame image p8a by the filter operations shown in the equations (4) to (10). In step S203, the signals of the third frame image p2a to the ninth frame image p8a are held in the memory 6.

  Then, when it is determined in step S204 that the calculation of each frame image corresponding to the total number of divisions has been completed and the process proceeds to step S206, the system control unit 11 instructs the synthesis ratio generation unit 7 to perform the optical in each frame. Send zoom information. As a result, the composition ratio generation unit 7 generates a composition ratio as described above, and outputs information on the composition ratio to the composition unit 8. After step S206, the system control unit 11 advances the process to step S207.

  In step S207, the system control unit 11 controls the combining unit 8 to read out the images of the reference frame and the combining side frame based on the combining ratio generated by the combining ratio generating unit 7. At this time, the synthesis unit 8 controls the pixel position of each image read from the memory 6 under the control of the system control unit 11, and reads each pixel at the reference position from the memory 6. Further, the system control unit 11 calculates motion information such as a motion vector from each frame image of the first frame to the ninth frame held in the memory 6, and based on the motion information, the synthesis unit 8 stores the memory information. 6 corrects the pixel position when the pixel is read out. That is, each frame image from the first frame to the ninth frame is a frame image of a moving image, and the position of the subject image or the like in each frame image changes due to the movement of the subject or the imaging device at the time of shooting. there is a possibility. For this reason, the system control unit 11 corrects the pixel position when the synthesizing unit 8 reads out the pixels from the memory 6 based on the motion information, so that the subject image or the like in each frame image due to the movement of the imaging device is corrected. Compensate for movement. After step S207, the system control unit 11 advances the process to step S208.

  In step S208, the system control unit 11 controls the synthesizing unit 8 to add and synthesize each pixel read from the first frame image p0a to the ninth frame image p8a. That is, the synthesis unit 8 at this time synthesizes each pixel of each frame image read from the memory 6 by addition under the control of the system control unit 11. The signal of the frame image formed by the synthesis process by the pixel addition in the synthesis unit 8 becomes the signal of the blurred image by the division filter process for the moving image. The blurred image after the filtering process is sent to the background subject combining unit 10 described above, and the main subject image extracted based on the main subject information from the main subject determining unit 9 is combined as described above. Become.

  After step S208, the system control unit 11 advances the process to step S209. In step S209, the system control unit 11 determines whether or not to end the blurring process on the moving image in accordance with an instruction from the user or the like, for example. If the system control unit 11 determines in step S209 to end the blurring process, the system control unit 11 ends the process of the flowchart of FIG.

  On the other hand, when determining in step S209 that the blurring process is not to be ended, the system control unit 11 returns the process to step S201. That is, for example, after the processing of the first to ninth frames is completed as described above, the system control unit 11 converts the image signal of the tenth frame output from the development processing unit 3 in step S201 to the filter processing unit 4. To input. In the next step S202, the system control unit 11 causes the filter processing unit 4 to filter the image signal of the tenth frame to generate the tenth frame image p9a. In the next step S203, the tenth frame image p9a. Is held in the memory 6. The memory 6 at this time holds nine frames of the second frame image p1a to the tenth frame image p9a. Next, in step S204, the system control unit 11 determines that the calculation for nine frames has been completed, proceeds to step S206, and in step S206, the composition ratio generation unit 7 performs composition based on the optical zoom information. The ratio is calculated and sent to the combining unit 8. Then, the system control unit 11 reads the second frame image p1a to the tenth frame image p9a from the memory 6 in step S207, and performs synthesis by addition of the synthesis unit 8 in step S208. The same applies hereinafter, and the filter processing in the filter processing unit 4 and the synthesis processing by addition in the synthesis unit 8 are performed while sequentially shifting one frame at a time until it is determined in step S209 that the blurring processing is finished.

<Second Embodiment>
Next, an imaging apparatus according to the second embodiment will be described with reference to FIGS. In the second embodiment, the same constituent elements as those described in the first embodiment are denoted by the same instruction numbers as those in the first embodiment, and redundant description is omitted.

  The imaging apparatus according to the second embodiment has no sense of incongruity because a composite image with a sense of incongruity may be generated when the angle of view of the compositing side frame is narrower than the reference frame (the zoom magnification is high). It is an example of the image signal processing apparatus which enabled the production | generation of the synthesized image. That is, for example, when the angle of view of the compositing side frame is narrower than that of the base frame, there is a possibility that the reference area in the compositing side frame is insufficient due to the narrower angle of view than the base frame. In this way, if the synthesis process is performed with a shortage of reference areas, there are no pixels in the lacked reference area, so that the image after the synthesis process becomes an uncomfortable image. For this reason, the imaging apparatus according to the second embodiment can generate a composite image without a sense of incongruity by supplementing a reference area that is insufficient from the base frame.

  A specific example in which the reference area is insufficient in the composition side frame when the angle of view of the composition side frame is narrower (the zoom magnification is high) with respect to the base frame will be described with reference to FIG. FIG. 10A is the same image as the fifth frame image p4a of FIG. 7 described above, and shows the reference range 2000 of the spatial filter and the area 2005 for the target pixel when the target pixel position is (k, j). FIG. It is assumed that the fifth frame image p4a is a reference frame image. On the other hand, FIG. 10B is the same image as the ninth frame image p8a of FIG. 7I, and the spatial filter reference range 2000 and the target pixel when the target pixel position is (k, j). It is a figure showing the area | region 2009. FIG. Assume that the ninth frame image p8a is an image of the compositing side frame. An area 3000 in FIG. 10A indicates an area corresponding to the angle of view of the ninth frame image p8a. The pixel-of-interest position (k, j) in FIG. 10A corresponds to the position (m, j) in FIG. 10B, and the region 2009 is set based on the position (m, j). Therefore, the reference position in the area 2009 of the ninth frame image p8a is outside the image. That is, as can be seen from FIGS. 10A and 10B, when the angle of view of the compositing side frame (the ninth frame image p8a) is narrower than the reference frame (the fifth frame image p4a). In addition, the reference area of the synthesis side frame is insufficient.

  FIG. 11 shows a schematic configuration of an imaging apparatus according to the second embodiment that makes it possible to generate a composite image without a sense of incongruity by supplementing a deficient reference area with a reference frame. Hereinafter, components different from the first embodiment will be described. The imaging apparatus according to the second embodiment illustrated in FIG. 11 includes an enlargement / synthesis unit 12 included in the synthesis control unit.

  The enlargement / synthesis unit 12 reads out and acquires the image signals of the reference frame and the synthesis side frame from the memory 6. Further, the enlargement / combination unit 12 compares the angle of view (zoom magnification) of the reference frame and the composition side frame based on the optical zoom information from the system control unit 11, and the angle of view of the composition side frame relative to the reference frame. Determine whether is narrow. If the enlargement / combination unit 12 determines that the angle of view of the compositing side frame is narrower than that of the reference frame, the enlargement / combination unit 12 determines that the reference position of the compositing side frame may be outside the image, and converts the reference frame image into the compositing side frame image. Enlarge to match the angle of view (zoom magnification). Specifically, when the enlargement / combination unit 12 determines that the angle of view of the compositing side frame is narrower than the reference frame, a difference (viewing angle) between the angle of view of the reference frame and the angle of view of the compositing side frame is determined. Difference). Then, the enlargement / combination unit 12 enlarges the reference frame image by a magnification corresponding to the field angle difference. Next, the enlargement / synthesizing unit 12 extracts only the image portion corresponding to the region outside the combining-side frame image from the reference frame image enlarged in accordance with the angle of view of the combining-side frame, and combines the extracted image portions. Composite around the side frame image. As a result, the angle of view in the combining-side frame image after the image portion extracted from the enlarged reference frame image is synthesized around the same is the same as the angle of view in the reference frame image before enlargement.

  FIG. 12 is a diagram illustrating an example of the synthesis-side frame image p8b after the image portion extracted from the enlarged reference frame image as described above is synthesized around the synthesis-side frame image in the enlargement / synthesis unit 12. is there. In FIG. 12, an area 3001 corresponds to the original ninth frame image p8a, and an area 3002 is obtained by changing the fifth frame image p4a, which is a reference frame image, from the angle of view difference (magnification) to the ninth frame image p8a. It corresponds to the image enlarged to match (difference). In the example of FIG. 12, the image portion outside the region 3001 is an image portion extracted from the enlarged reference frame image, and is an image portion combined around the combining-side frame image (p8a). In the combined frame image p8b after combining as shown in FIG. 12, a pixel is present at the reference position of the region 2009.

  The enlargement / synthesis unit 12 outputs the image signal of the synthesis side frame (image p8b) in which the reference region is supplemented from the enlarged standard frame image as described above to the synthesis unit 8. As a result, the composition unit 8 can perform image composition using the composition side frame (image p8b) where the reference position exists. Therefore, according to the imaging apparatus of the second embodiment, a composite image without a sense of incongruity is generated.

<Third Embodiment>
Next, an image pickup apparatus according to the third embodiment will be described with reference to FIGS. 13 and 14. In the third embodiment, the same constituent elements as those described in the first embodiment are denoted by the same instruction numbers as those in the first embodiment, and redundant description is omitted.

  The imaging apparatus according to the third embodiment is an example of an image signal processing apparatus that can generate a composite image without a sense of incongruity when the angle of view of the synthesis side frame is narrower than the reference frame. In the third embodiment, when the angle of view of the compositing side frame is narrower than the reference frame, the reference frame itself is referred to by the compositing side frame by switching to a frame image having a narrow angle of view (high zoom magnification). Avoid running out of space.

  In FIG. 13, the base frame is dynamically switched to a frame image with a narrow angle of view (high zoom magnification), so that a composite image without a sense of incongruity is generated without causing a shortage of a reference area in the compositing side frame. The schematic structure of the imaging device of 3rd Embodiment which enables is shown. Hereinafter, components different from the first embodiment will be described. The imaging apparatus according to the third embodiment illustrated in FIG. 13 includes a reference frame selection unit 13 included in the synthesis control unit.

  The reference frame selection unit 13 compares the angle of view of the reference frame and the composition side frame based on the optical zoom information from the system control unit 11, and the angle of view of the composition side frame is narrower than the reference frame (zoom magnification). Is high). If the reference frame selection unit 13 determines that the angle of view of the compositing side frame is narrower than that of the base frame, the reference frame selecting unit 13 determines that the reference position of the compositing side frame may be outside the image, and has the narrowest angle of view ( Select the frame with the highest zoom magnification. Specifically, when it is determined that the angle of view of the compositing side frame is narrower than that of the reference frame, the enlargement / combination unit 12 selects the frame with the narrowest angle of view among the nine frames described above based on the optical zoom information. To do. Then, the reference frame selection unit 13 sets the selected frame as a reference frame, and notifies the synthesis unit 8 of which of the nine frames is the reference frame.

  An example in which the reference frame selection unit 13 dynamically switches the reference frame will be described with reference to FIG. FIGS. 14A to 14K show examples of frame images p1c to p11c in time order, respectively. Also, the frame images p1c to p6c in FIGS. 14A to 14F show an example of each frame image when zooming in is performed by gradually narrowing the angle of view (gradually increasing the zoom magnification). . On the other hand, the frame images p7c to p11c in FIGS. 14G to 14K show an example of each frame image when zooming out is performed by gradually widening the angle of view (gradually lowering the zoom magnification). . 14A to 14K, a region 4000 indicates a range referred to by a spatial filter of 9 taps in the horizontal and vertical directions with respect to the pixel of interest on each image. The positions of the central pixels in the areas 4001 to 4011 indicate reference positions on the image with respect to the target pixel. FIG. 14A is a diagram illustrating a reference range of the target pixel and the spatial filter in the frame image p0c and a reference position on the image with respect to the target pixel. Similarly, FIGS. 14B to 14K are diagrams of the frame images p1c to p11c.

  In the synthesis process in the synthesis unit 8, it is only necessary to have nine frame images. First, an example in which the nine frame images p <b> 1 c to p <b> 9 c in FIG. 14A to FIG. I will give you. In these frame images p1c to p9c in FIGS. 14A to 14I, the frame image p1c in FIG. 14A has the narrowest angle of view. Therefore, in this case, the reference frame selection unit 13 selects the frame image p1c as a reference frame.

  When synthesizing the next frame image, the synthesizing unit 8 performs a synthesizing process using the nine frame images p2c to p10c of FIGS. 14B to 14J. In the frame images p2c to p10c of FIGS. 14B to 14J, the narrowest angle of view is the frame image p2c of FIG. 14B and the frame image p10c of FIG. 14J. As described above, when there are a plurality of frame images having the narrowest angle of view among the nine frame images, the reference frame selection unit 13 considers the continuity of time in the moving image, and temporally advances the time. The frame image p2c in FIG. 14B, which is a frame image, is selected.

  Furthermore, when synthesizing the next frame image, the synthesizing unit 8 performs a synthesizing process using the nine frame images p3c to p11c of FIGS. 14 (c) to 14 (k). In the frame images p3c to p11c in FIGS. 14C to 14K, the frame image p11c in FIG. 14K has the narrowest angle of view. Therefore, the reference frame selection unit 13 selects this frame image p11c as a reference frame.

  In the case of the frame images p1c to p11c in FIGS. 14A to 14K described above, the frame image p1c in FIG. 14A is first selected as the reference frame, and then FIG. ) Frame image p2c, and then the frame image p11c of FIG. 14 (k) is selected. Accordingly, none of the frame images p3c to p10c of FIGS. 14C to 14J temporally prior to the frame image p11c of FIG. 14K is selected as the reference frame. That is, since the frame images p3c to p10c in FIGS. 14C to 14J are used as reference frames and the synthesized image is lost, the continuity in the time direction is lost. However, in the case of the present embodiment, since there is no shortage of the reference area in the compositing side frame, it is possible to obtain a composite image that does not actually feel strange.

  As described above, according to the third embodiment, when the angle of view of the compositing frame is narrower than the reference frame, the reference frame is dynamically switched to a frame image with a narrow angle of view (high zoom magnification). Thus, it is possible to generate a composite image without a sense of incongruity.

<Fourth Embodiment>
Next, an imaging device according to a fourth embodiment will be described with reference to FIGS. In the fourth embodiment, the same constituent elements as those described in the first embodiment are denoted by the same instruction numbers as those in the first embodiment, and redundant description is omitted.

  FIG. 15 shows a schematic configuration of an imaging apparatus according to the fourth embodiment. Hereinafter, components different from the first embodiment will be described. An imaging apparatus according to the fourth embodiment illustrated in FIG. 15 includes a read control unit 14 that is an example of a synthesis control unit, instead of the synthesis ratio generation unit 7 of the imaging apparatus according to the first embodiment illustrated in FIG. ing.

  The read control unit 14 is supplied with optical zoom information from the system control unit 11, and thus can determine the angle of view (zoom magnification) of the compositing side frame with respect to the reference frame. Then, the read control unit 14 controls the read position by the combining unit 8 based on the angle of view (zoom magnification) of the combining side frame with respect to the reference frame. That is, the reading control unit 14 controls the reading position of the memory 6 by the combining unit 8 so that the center (reference position) of the reference region overlaps the reference position when the base frame and the combining side frame have the same angle of view. To do.

  Read position control by the read control unit 14 will be described with reference to FIGS. 7A to 7J and FIGS. 16A and 16B. FIG. 16A is a diagram showing the relationship between the angle of view (zoom magnification) of the compositing frame with respect to the reference frame and the shift amount when the read control unit 14 performs control to shift the read position. Note that the reading position shift amount is a reading obtained by the reading control unit 14 in accordance with the angle of view with respect to the reading position corresponding to each reference position in each of the regions 2001 to 2009 in FIG. It corresponds to the difference from the position. In other words, the read control unit 14 performs read position control such that the read position corresponding to each reference position in the areas 2001 to 2009 in FIG. 7J is shifted by a shift amount corresponding to the angle of view. FIG. 16B shows the regions 2001r to 2009r when the fifth frame image p4a is a reference frame and the other frame images p0a to p3a and p5a to p8a are matched to the angle of view of the reference frame. It is the figure arranged on the image. That is, FIG. 16B shows a reference frame image (p4a) and other composite side frame images (p0a to p3a, p5a to p8a) when the reading control unit 14 performs reading control according to the angle of view. The positional relationship between the respective regions 2001r to 2009r is shown.

  Here, as described with reference to FIG. 7J described above, the size of the range referred to in the filtering process in each frame image varies depending on the angle of view (zoom magnification). For example, when the reference frame is the fifth frame image p4a and the compositing side frame is the ninth frame image p8a, the angle of view of the compositing side frame is narrower than the reference frame, so the region 2009 in FIG. In this way, the reference area becomes narrower and the blurring amount becomes smaller. On the other hand, for example, when the reference frame is the ninth frame image p8a and the compositing side frame is the first frame image p1a, the angle of view of the compositing side frame is wide, so reference is made as in the area 2001 in FIG. The area becomes wider and too blurry.

  For this reason, in the fourth embodiment, the read control unit 14 causes the center (reference position) of the reference region to overlap the reference position when the base frame and the compositing side frame have the same angle of view. The reading position by the synthesis unit 8 is controlled. Specifically, when the reference frame is the fifth frame image p4a and the compositing side frame is the ninth frame image p8a, the reading control unit 14 overlaps the area 1009 of FIG. The center of the area 2009r in FIG. On the other hand, when the base frame is the ninth frame image p8a and the composition side frame is the fifth frame image p4a, the read control unit 14 determines the reference region 1001 and the center position (reference position) in FIG. The center of the region 2001r in FIG.

  16 (b) and FIG. 7 (j) will be described in comparison with FIG. 5 (i). The central positions of the respective regions 2001r to 2009r in FIG. 7 (j) are the respective regions 1001 in FIG. 5 (j). It is away from the central position of ~ 1009. On the other hand, the center positions of the respective regions 2001r to 2009r in FIG. 16B coincide with the center positions of the respective regions 1001 to 1009 in FIG. That is, when readout control is performed such that the centers of the regions 2001r to 2009r as shown in FIG. 16B are the reference positions, the image after the synthesis processing by the synthesis unit 8 is the same as that in the case of FIG. The image has a better balance of blurring than the image after the composition processing. As described above, according to the imaging apparatus of the fourth embodiment, when the magnification of the optical zoom is changed and the angle of view is different in capturing a moving image, the readout position control is performed, so that the balance of the degree of blur is adjusted. It is possible to reduce fluctuations in the degree of blurring.

  Next, FIG. 17 shows a flowchart from filter processing to synthesis processing in the fourth embodiment. The flowchart of FIG. 17 is executed by the system control unit 11 performing control of the filter processing unit 4, control of the memory 6, read position control in the read control unit 14, and synthesis control of the synthesis unit 8. In FIG. 17, the same processing as that in each step described in FIG. 9 is given the same instruction number, and redundant description is omitted.

  In FIG. 17, processing different from the flowchart of FIG. 9 described above is performed in step S306. In the flowchart of FIG. 17, when determining in step S204 that the calculation for the predetermined number of frames corresponding to the total number of divisions has been completed, the system control unit 11 advances the processing to step S306. In step S <b> 306, the system control unit 11 sends the optical zoom information in each frame to the read control unit 14. Thereby, the read control unit 14 generates the shift amount of the read position as described in FIG. 16A based on the angle of view (zoom magnification) of the reference frame and the compositing side frame, and the shift amount. Is output to the synthesis unit 8. After step S306, the system control unit 11 advances the process to step S207. Therefore, in step S207, the combining unit 8 reads out the pixel at the reading position in consideration of the shift amount obtained by the reading control unit 14 during the combining process.

<Fifth Embodiment>
Next, FIG. 18 illustrates a schematic configuration of an imaging apparatus according to the fifth embodiment. In the fifth embodiment, the same components as those described in the first, second, and fourth embodiments are assigned the same instruction numbers as those in the first, second, and fourth embodiments, respectively. A duplicate description is omitted.

  In the fifth embodiment, as in the second embodiment described above, when the angle of view of the compositing side frame is narrower than that of the base frame, the reference area that is insufficient is compensated for by the base frame, thereby making it uncomfortable. This is an example in which it is possible to generate a non-composite image. That is, the imaging apparatus of the fifth embodiment shown in FIG. 18 includes the same enlargement / synthesizing unit 12 as described in the second embodiment shown in FIG.

  Also in the imaging apparatus of the fifth embodiment, as in the case of the second embodiment described above, it is possible to generate a composite image without a sense of incongruity by supplementing the reference region that is lacking with the reference frame.

<Sixth Embodiment>
FIG. 19 shows a schematic configuration of an imaging apparatus according to the sixth embodiment. In the sixth embodiment, the same constituent elements as those described in the first, third, and fourth embodiments are denoted by the same instruction numbers as those in the first, third, and fourth embodiments, respectively. The explanations made are omitted.

  In the sixth embodiment, as in the third embodiment described above, when the angle of view of the combining side frame is narrower than that of the reference frame, the reference frame is dynamically switched, so that the reference region is changed in the combining side frame. This is an example of preventing the shortage from occurring. That is, the imaging apparatus of the sixth embodiment shown in FIG. 19 includes the same reference frame selection unit 13 as that described in the third embodiment shown in FIG.

  Also in the imaging device of the sixth embodiment, as in the case of the third embodiment described above, when the angle of view of the synthesis side frame is narrower than the reference frame, the reference frame is dynamically switched, so It is possible to generate a composite image that does not exist.

<Other embodiments>
In each of the above-described embodiments, the filtering process and the synthesizing process for the frame image of the moving image are given as an example. However, the present invention is not limited to the moving image and the filtering process and the synthesizing process for each captured image by, for example, high-speed continuous shooting. Is also applicable.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

  DESCRIPTION OF SYMBOLS 1 Optical system, 2 Image pick-up element, 3 Development process part, 4 Filter process part, 5 Filter coefficient control part, 6 Memory, 7 Composition ratio production | generation part, 8 Composition part, 9 Main subject discrimination part, 10 Background subject composition part, 11 System control unit, 12 enlargement / synthesis unit, 13 reference frame selection unit, 14 readout control unit

Claims (10)

Filter means for performing filter processing for controlling the filter coefficient in order of capturing the frame images for each predetermined number of frames with respect to the frame images input in the order of capturing;
A synthesizing unit for synthesizing the frame image after the filtering process by the filtering unit for each of the predetermined number of frames while sequentially shifting by one frame;
An image signal processing apparatus comprising: a synthesis control unit that controls the synthesis process of the frame image by the synthesis unit based on an angle of view when the frame image is captured. The predetermined number of frames corresponds to the number of divisions obtained by dividing the spatial filter having a plurality of taps in the horizontal and vertical directions.
The filter means performs the filter processing using filter coefficients for each of the divided filters obtained by dividing the spatial filter into the plurality, in the order of the frame images of the predetermined number of frames,
The image signal processing apparatus according to claim 1, wherein the synthesizing unit performs a synthesizing process of adding pixels of each frame image of the predetermined number of frames after the filtering process. The synthesizing unit is configured such that, out of the frame images of the predetermined number of frames after the filtering process, a pixel of a reference frame image serving as a reference at the time of the synthesizing process and a synthesizing frame serving as a synthesizing side with the reference frame image Performing the synthesis process of adding a pixel at a predetermined position of the image;
The combining control unit weights the pixels of the combining-side frame image to be added to the pixels of the reference frame image based on an angle of view when the frame image is captured, and The image signal processing apparatus according to claim 1, wherein a composition ratio between the frame image and the composition side frame image is controlled.   The synthesizing control unit increases the weight of the pixel of the synthesizing frame image as the angle of view of the synthesizing frame image with respect to the reference frame image becomes narrower, and synthesizes the synthesizing frame image with the reference frame image. The image signal processing apparatus according to claim 3, wherein the ratio is increased. The synthesizing unit is configured such that, out of the frame images of the predetermined number of frames after the filtering process, a pixel of a reference frame image serving as a reference at the time of the synthesizing process and a synthesizing frame serving as a synthesizing side with the reference frame image Performing the synthesis process of adding a pixel at a predetermined position of the image;
The synthesis control unit is configured to determine the predetermined position with respect to the pixel of the synthesis side frame image to be added to the pixel of the reference frame image by the synthesis unit based on an angle of view when the frame image is captured. The image signal processing device according to claim 1, wherein the image signal processing device is controlled.   When the angle of view of the composition side frame image is different from the angle of view of the reference frame image, the composition control unit is configured to add the predetermined frame to the pixel of the composition side frame image to be added to the pixel of the reference frame image. The image signal processing apparatus according to claim 5, wherein the position is controlled so as to match a predetermined position when the reference frame and the synthesis side frame have the same angle of view. When the angle of view of the compositing side frame image is narrower than the reference frame image, the compositing control means enlarges the reference frame image according to the angle of view difference between the reference frame image and the compositing side frame image, Obtaining an image portion corresponding to the angle-of-view difference from the enlarged reference frame image, and synthesizing the acquired image portion around the synthesis side frame image;
7. The composition processing according to claim 3, wherein the composition unit performs the composition processing of adding the pixel of the composition side frame image in which the image portion is composed around and the pixel of the reference frame image. The image signal processing device according to claim 1.   The image signal according to any one of claims 3 to 6, wherein the synthesis control unit sets a frame image having a narrowest angle of view among the frame images of the predetermined number of frames as the reference frame image. Processing equipment. An image signal processing method executed by an image signal processing apparatus that performs a filtering process on frame images input in the order of imaging,
A filter step for performing a filter process for controlling a filter coefficient in the order of capturing the frame images for each predetermined number of frames with respect to the frame images input in the order of capturing;
A synthesizing step for synthesizing the frame image after the filtering process by the filtering step every predetermined number of frames while sequentially shifting by one frame;
An image signal processing method comprising: a synthesis control step for controlling the synthesis processing of the frame image by the synthesis step based on an angle of view when the frame image is captured. Computer
Filter means for performing a filtering process for controlling a filter coefficient in the order of capturing the frame images for each predetermined number of frames with respect to the frame images input in the order of capturing;
A synthesizing unit for synthesizing the frame image after the filtering process by the filtering unit for each of the predetermined number of frames while sequentially shifting by one frame;
A program that functions as a synthesis control unit that controls the synthesis process of the frame image by the synthesis unit based on an angle of view when the frame image is captured.

Images