JP7028958B2 - Image processing device, image pickup device, and image processing method - Google Patents

Description

The present invention relates to an image processing device, an image pickup device, and an image processing method, and more particularly to an image processing device, an image pickup device, and an image processing method for obtaining the effect of interexposure zoom photography.

In photography, a technique called "zoom photography between exposures" is known. According to this "zoom shooting between exposures", it is possible to obtain an artistic and fantastic work in which the size, trajectory, etc. of the subject are changed by performing a zoom operation during exposure, but the image intended by the photographer. It is necessary to acquire advanced photography techniques in order to obtain this. Therefore, a technique for assisting zoom photography between exposures has been developed. For example, Patent Document 1 describes setting a range of change in focal length via a support image for setting shooting conditions.

Japanese Unexamined Patent Publication No. 2010-200243

In zoom shooting between exposures, if you want to express the trajectory of the subject with clear lines, or if you want to express the trajectory of the subject with lines that gradually blur, it is complicated and difficult to perform the focus operation at the same time as the zoom operation. High operation is required. Also, if you want to move the main subject to a location that is offset from the center of the image, you need to perform complicated and difficult operations such as zooming and changing the direction of the camera (pan and / or tilt) at the same time. be. However, in the technique described in Patent Document 1 described above, even if the change range (zoom range) of the focal length can be easily set, a sufficient effect can be obtained when it is necessary to perform another operation at the same time as the zoom operation. do not have. With such a technique, it is difficult to obtain the effect intended by the photographer (for example, an artistic effect or a fantastic effect).

As described above, it has been difficult to obtain the effect intended by the photographer in the inter-exposure zoom shooting with the conventional technique.

The present invention has been made in view of such circumstances, and an object of the present invention is to provide an image processing device, an image pickup device, and an image processing method capable of easily obtaining an effect intended by a photographer in interexposure zoom photography. And.

In order to achieve the above-mentioned object, the image processing apparatus according to the first aspect of the present invention includes a first image acquisition unit for acquiring a first image, and a zoom for setting a zoom center and a zoom range for the first image. A setting unit, a second image acquisition unit that acquires a second image corresponding to the zoom center and the zoom range, and an intermediate image that has a narrower shooting angle than the first image and a wider shooting angle than the second image are designated. An intermediate image acquisition unit that acquires only the number of images, and an image processing unit that performs at least one process of moving, enlarging or reducing, rotating, and changing the degree of blurring of the first image, the second image, or the intermediate image. A first image, a second image, or an intermediate image that has been processed is weighted and added to generate a composite image.

According to the first aspect, the image processing device weights and adds the first image, the second image, and the intermediate image having different shooting angle of view (corresponding to the zoom magnification), so that the effect of zoom shooting between exposures can be obtained. .. In addition, since at least one of movement, enlargement, rotation, and change of the degree of blur is performed in the generation of the composite image, the effect desired by the user (for example, the trajectory of the subject is clear) without performing complicated and difficult operations. It can be easily obtained, such as being represented by a line, being represented by a line in which the trajectory of the subject is gradually blurred, moving the main subject to a position shifted from the center of the image, and the like). It should be noted that the image processing device can set the weight in the addition of images according to the time for maintaining the shooting angle of view (corresponding to the zoom magnification) in, for example, zoom shooting between exposures. If you want to emphasize the appearance of the subject at a specific shooting angle of view, you can increase the weight of the image at that shooting angle of view. Further, in the generation of the composite image, an image process for changing the color and / or brightness of the image selected from the first image, the second image, and the intermediate image may be performed. As a result, it is possible to obtain an image having a higher artistic effect and a fantastic effect.

In the first aspect, the first image may be captured and acquired by a photographing optical system, or an already captured image may be acquired via a network or a recording medium. Further, the second image can be acquired from the first image by electronic zoom processing, and the intermediate image can be captured and acquired by the photographing optical system. Any integer of zero or more can be specified as the number of shots of the intermediate image. The number of shots of the intermediate image may be set to zero and all may be processed by the second image (electronic zoom), or the number of intermediate images may be increased.

The second image acquisition unit can acquire a second image having a narrower angle of view than the first image.

In the first aspect, the image processing apparatus according to the second aspect acquires a second image by cutting out a portion of the first image corresponding to the zoom center and the zoom range. The second aspect defines an aspect of acquiring a second image by an electronic zoom.

In the first or second aspect of the image processing apparatus according to the third aspect, the intermediate image acquisition unit acquires a plurality of images having different shooting angles of view as a specified number of intermediate images, and the image processing unit has a plurality of images. Process each of the images. In the third aspect, the content and degree of the processing performed by the image processing unit may be the same regardless of the image, or may be changed for each of the plurality of images.

The image processing device according to the fourth aspect further includes an information display unit for displaying information indicating the shooting angle of view of the intermediate image on the display device in any one of the first to third aspects, and is an intermediate image acquisition unit. Acquires the image taken corresponding to the information as an intermediate image. In the fourth aspect, the information display unit can display information indicating, for example, the zoom center and the zoom range by an image, characters, numbers, symbols, figures, etc., and the user can easily refer to this information to display an intermediate image. You can shoot.

In the fourth aspect of the image processing apparatus according to the fifth aspect, the zoom setting unit moves the zoom center according to the deviation between the zoom center indicated by the information and the center of the image captured in response to the information. Let me. When the user shoots an intermediate image with reference to information such as the fourth aspect, there is a possibility that the image expected to be shot in calculation and the actually shot image are different from each other. Therefore, by moving the zoom center by the zoom setting unit as in the fifth aspect, it is possible to obtain a composite image in which the trajectory of the subject changes smoothly. For example, the zoom setting unit can calculate the correlation value between the expected image and the actually captured image, and move the zoom center so that the correlation value is the highest.

In any one of the first to fifth aspects of the image processing apparatus according to the sixth aspect, the image processing unit moves, enlarges, reduces, or rotates the intermediate image and / or the second image by affine transformation. The sixth aspect defines an example of a specific method of movement, enlargement or reduction, and rotation.

In any one of the first to sixth aspects of the image processing apparatus according to the seventh aspect, the zoom setting unit sets the zoom center and the zoom range based on the user's instruction input. According to the seventh aspect, the user can set a desired zoom center and zoom range.

In any one of the first to seventh aspects of the image processing apparatus according to the eighth aspect, the image processing unit changes the degree of blur based on the user's instruction input. According to the eighth aspect, the user can set a desired degree of blurring.

In any one of the first to eighth aspects of the image processing apparatus according to the ninth aspect, the zoom setting unit sets the shooting angle of view of the intermediate image according to the distance between the zoom center and the center of the first image. do. When the center of the zoom is far from the center of the first image, the locus of points away from the center of the zoom changes discontinuously with a change in the angle of view (for example, a constant change in the ratio), and a smooth locus is obtained. It may not be possible. Such discontinuous changes are particularly noticeable at the point farthest from the center of the zoom (eg, at any of the four corners of the image). Therefore, in the ninth aspect, the zoom setting unit sets the shooting angle of view of the intermediate image according to the distance between the zoom center and the center of the first image, whereby the locus of points away from the zoom center is continuous. Can be changed to. For example, when the zoom setting unit changes the shooting angle of view of an intermediate image from a wide state to a narrow state, the zoom setting unit makes the change in the angle of view between images slower in the state where the angle of view is wide than in the state where the angle of view is narrow. Further, the number of intermediate images in a state where the angle of view is wide may be increased from the number of intermediate images in a state where the angle of view is narrow.

The image processing device according to the tenth aspect includes a display control unit for displaying a composite image on the display device in any one of the first to ninth aspects. According to the tenth aspect, the user can visually recognize the composite image (the image having the effect of zooming between exposures). The display control unit may display the first image, the second image, and the intermediate image on the display device in addition to the composite image.

In order to achieve the above-mentioned object, the image pickup apparatus according to the eleventh aspect of the present invention relates to an image pickup unit that captures a subject image with a photographing lens including a zoom lens, and one of the first to tenth aspects. An image pickup device including an image processing device, in which the zoom setting unit controls the zoom lens according to the zoom center and the zoom range, and the intermediate image acquisition unit obtains an intermediate image corresponding to the zoom center and the zoom range by the image pickup unit. get. According to the eleventh aspect, the user can capture an image optically zoomed by a zoom lens and use it as an intermediate image.

In order to achieve the above-mentioned object, the image processing method according to the twelfth aspect of the present invention includes a first image acquisition step for acquiring a first image and a zoom for setting a zoom center and a zoom range for the first image. A setting step, a second image acquisition step of acquiring a second image corresponding to the zoom center and the zoom range, and an intermediate image having a narrower shooting angle than the first image and a wider shooting angle than the second image are specified. An intermediate image acquisition step of acquiring the number of images, and an image processing step of performing at least one of moving, enlarging or reducing, rotating, and changing the degree of blurring of the first image, the second image, or the intermediate image. It has an image synthesizing step of weighting and adding the processed first image, second image, or intermediate image to generate one image. According to the twelfth aspect, the effect intended by the photographer can be easily obtained in the inter-exposure zoom photography as in the first aspect. In the second image acquisition step, it is possible to acquire a second image having a narrower shooting angle of view than the first image.

The image processing method according to the twelfth aspect may further have the same configuration as the second to tenth aspects. In addition, a program for causing an image processing device and / or an image pickup device to execute the image processing method of these embodiments, and a non-temporary recording medium on which a computer-readable code of such a program is recorded may also be mentioned as aspects of the present invention. can.

As described above, according to the image processing device, the image pickup device, and the image processing method of the present invention, it is possible to easily obtain the effect intended by the photographer in the inter-exposure zoom shooting.

FIG. 1 is a diagram showing a configuration of an image pickup apparatus according to a first embodiment. FIG. 2 is a diagram showing a functional configuration of an image processing unit. FIG. 3 is a flowchart showing the procedure of image processing. FIG. 4 is a flowchart showing the procedure of image processing (continuation of FIG. 3). FIG. 5 is a diagram showing an example of a wide-angle image. FIG. 6 is a diagram showing how the zoom center is set. FIG. 7 is a diagram showing a state of setting the zoom angle of view. FIG. 8 is a diagram showing a state of focus position setting. FIG. 9 is a diagram showing a state of setting the number of shots of the intermediate image. FIG. 10 is a diagram showing an actual shooting state (first image). FIG. 11 is a diagram showing an example (first image) of an image actually taken. FIG. 12 is a diagram showing an actual shooting state (second image). FIG. 13 is a diagram showing an example (second image) of an image actually taken. FIG. 14 is a diagram showing a deviation between an intermediate image expected to be photographed and an actual intermediate image. FIG. 15 is a diagram showing a state of correction of deviation. FIG. 16 is a diagram showing a state of adjusting the scaling ratio. FIG. 17 is a diagram showing an example of image composition by simulation. FIG. 18 is a diagram showing another example of image composition by simulation. FIG. 19 is a diagram showing an example of an output image. FIG. 20 is a diagram showing an example of an image synthesized by simulation. FIG. 21 is a diagram showing another example of the image synthesized by the simulation. FIG. 22 is a diagram showing still another example of the image synthesized by the simulation. FIG. 23 is a diagram showing an example of a composite image of a general subject.

Hereinafter, embodiments for implementing the image processing device, the image pickup device, and the image processing method according to the present invention will be described in detail with reference to the accompanying drawings.

<First Embodiment>
<Overall configuration of the image pickup device>
FIG. 1 is a diagram showing a configuration of an image pickup device 10 (image pickup device, image processing device) according to the first embodiment. The image pickup device 10 is composed of an interchangeable lens 100 (shooting lens, image pickup unit) and an image pickup device main body 200 (image processing device), and a subject image (optical image) is connected to the image pickup element 210 by a photography lens including a zoom lens 110 described later. Make an image. The interchangeable lens 100 and the image pickup apparatus main body 200 can be attached to and detached from each other via a mount (not shown).

<Construction of interchangeable lens>
The interchangeable lens 100 includes a zoom lens 110 (zoom lens, photographing lens), a focus lens 120 (photographing lens), an aperture 130, and a lens driving unit 140. The lens drive unit 140 drives the zoom lens 110 and the focus lens 120 forward and backward in response to a command from the image processing device 240 (lens drive control unit 240H in FIG. 2) to perform zoom (optical zoom) adjustment and focus adjustment. In addition to performing zoom adjustment and focus adjustment in response to commands from the image processing device 240, zoom adjustment and focus adjustment may be performed in response to zoom operations and focus operations (zoom ring not shown, rotation of focus ring, etc.) performed by the user. good. Further, the lens driving unit 140 controls the aperture 130 in response to a command from the image processing device 240 to adjust the exposure. On the other hand, information such as the positions of the zoom lens 110 and the focus lens 120 and the degree of opening of the aperture 130 is input to the image processing device 240. The interchangeable lens 100 has an optical axis L.

<Structure of the main body of the image pickup device>
The image sensor main body 200 includes an image sensor 210 (imaging unit), an AFE 220 (AFE: Analog Front End, imaging unit), an A / D converter 230 (A / D: Analog to Digital, imaging unit), and an image processing device 240. To prepare for. The image pickup element 210 includes a light receiving surface in which a large number of light receiving elements are arranged in a matrix. Then, the subject light transmitted through the zoom lens 110, the focus lens 120, and the aperture 130 is imaged on the light receiving surface of the image pickup element 210, and is converted into an electric signal by each light receiving element. An R (red), G (green), or B (blue) color filter is provided on the light receiving surface of the image sensor 210, and a color image of the subject can be acquired based on the signals of each color. As the image pickup device 210, various photoelectric conversion elements such as CMOS (Complementary Metal-Oxide Semiconductor) and CCD (Charge-Coupled Device) can be used. The AFE 220 performs noise reduction, amplification, and the like of the analog image signal output from the image sensor 210, and the A / D converter 230 converts the captured analog image signal into a digital image signal having a gradation width.

<Configuration of image processing device>
FIG. 2 is a diagram showing a functional configuration of the image processing device 240 (image processing device). The image processing device 240 includes a first image acquisition unit 240A (first image acquisition unit), a zoom setting unit 240B (zoom setting unit), a second image acquisition unit 240C (second image acquisition unit), and an intermediate image acquisition unit 240D ( Intermediate image acquisition unit), image processing unit 240E (image processing unit), information display unit 240F (information display unit), display control unit 240G (display control unit), lens drive control unit 240H (lens drive control unit), and image. A compositing unit 240I (image compositing unit) is provided, and processing such as generation of a composite image is performed based on a digital image signal input from the A / D converter 230. Details of the processing by the image processing apparatus 240 will be described later.

The function of the image processing device 240 can be realized by using various processors. The various processors include, for example, a CPU (Central Processing Unit), which is a general-purpose processor that executes software (program) to realize various functions. In addition, the various processors described above include programmable logic devices such as GPU (Graphics Processing Unit) and FPGA (Field Programmable Gate Array), which are specialized processors for image processing, whose circuit configuration can be changed after manufacturing. (Programmable Logic Device: PLD) is also included. Further, the above-mentioned various processors also include a dedicated electric circuit, which is a processor having a circuit configuration specially designed for executing a specific process such as an ASIC (Application Specific Integrated Circuit).

The function of each part may be realized by one processor, or may be realized by a plurality of processors of the same type or different types (for example, a plurality of FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Further, a plurality of functions may be realized by one processor. As an example of configuring a plurality of functions with one processor, first, one processor is configured by a combination of one or more CPUs and software, as represented by a computer such as an image processing unit main body and a server. , There is a form in which this processor is realized as a plurality of functions. Secondly, as typified by System On Chip (SoC), there is a form of using a processor that realizes the functions of the entire system with one IC (Integrated Circuit) chip. As described above, various functions are configured by using one or more of the above-mentioned various processors as a hardware structure. Further, the hardware-like structure of these various processors is, more specifically, an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined.

When the above-mentioned processor or electric circuit executes software (program), the processor (computer) readable code of the software to be executed is stored in a non-temporary recording medium such as ROM (Read Only Memory), and the processor Refers to the software. The software stored in the non-temporary recording medium includes a program for executing image input, zoom processing, compositing processing, and the like. The code may be recorded on a non-temporary recording medium such as various optical magnetic recording devices or semiconductor memories instead of the ROM. For example, RAM (Random Access Memory) is used as a temporary storage area during processing using software, and for example, data stored in an EEPROM (Electronically Erasable and Programmable Read Only Memory) (not shown) can also be referred to. can.

The image processing apparatus 240 includes a ROM 242 (ROM: Read Only Memory, non-temporary recording medium) in addition to the above-mentioned parts. A computer-readable code of a program (including a program for executing the image processing method according to the present invention) necessary for image input, zoom processing, compositing processing, etc. is recorded in the ROM 242.

<Operation unit>
The operation unit 250 has a release button (not shown), an operation button (for example, a cross button, a quick button, an OK button, etc.), a dial, a switch, and the like, and the user sets shooting conditions, shoots an instruction, and synthesizes the image via the operation unit 250. Various operations such as image generation instruction, image processing content and / or degree setting can be performed. The monitor 270 may be configured as a touch panel and used as the operation unit 250.

<Memory>
The storage unit 260 is composed of various optical magnetic recording media, a non-temporary recording medium such as a semiconductor memory, and a control circuit thereof, and stores captured images (first image, second image), intermediate images, composite images, and the like. As the recording medium, a type that can be attached to and detached from the image pickup apparatus main body 200 can be used.

<Monitor (display device)>
The monitor 270 (display device) is composed of, for example, a liquid crystal display panel, and can display a captured image (first image, second image), an intermediate image, a composite image, shooting auxiliary information, and the like. The monitor 270 can be arranged on the back side, the top surface side, or the like of the image pickup apparatus main body 200.

<Image processing procedure>
The image processing in the image pickup apparatus 10 having the above-described configuration will be described. 3 and 4 are flowcharts showing the procedure of image processing according to the first embodiment.

<Acquisition of wide-angle image>
In step S100, the image processing device 240 (first image acquisition unit 240A) acquires a wide-angle image (first image) (first image acquisition step). The first image acquisition unit 240A may acquire an image taken by the user with the image pickup apparatus 10, or may acquire an image stored in the storage unit 260 in advance. Further, the image may be acquired via a recording medium or a network. An example of a wide-angle image (wide-angle image I0) is shown in FIG.

<Setting the zoom center and zoom range for the telephoto image>
The image processing device 240 (zoom setting unit 240B) sets the shooting angle of view (zoom center and zoom range) of the telephoto image (second image) with respect to the acquired wide-angle image (step S110: zoom setting step). As the shooting angle of view of the telephoto image, a shooting angle of view narrower than that of the wide-angle image is set. The image processing device 240 can make this setting based on the user's operation via the operation unit 250 and the monitor 270 as in the following example.

6 and 7 are diagrams showing how the zoom center is set for a wide-angle image. As shown in the part (a) of FIG. 6, the zoom setting unit 240B has a message M1 prompting the setting of the zoom center (“Please set the center” in the example of FIG. 6) and a cross-shaped mark indicating the zoom center. C1 is superimposed on the wide-angle image I0 and displayed on the monitor 270. The user can move the mark C1 to a desired position as shown in the portions (b) and (c) of FIG. 6 by operating a button (for example, a cross-shaped button) (not shown) of the operation unit 250. can. In the example of FIG. 6, the center of the zoom is once moved to the left and then moved downward to be in the state shown in the portion (c). The arrows in the portions (b) and (c) of FIG. 6 indicate the moving direction and the moving distance of the mark C1 from the center of the original wide-angle image (the state of the portion (a) of FIG. 6). Here, it will be described below assuming that the zoom center is fixed at the position shown in the portion (c) of FIG. 6 (it can be determined by operating the OK button (not shown)).

As shown in the portion (a) of FIG. 7, the zoom setting unit 240B displays a message M2 prompting the setting of the zoom range (“Please set the angle of view” in the example of FIG. 7) and a frame F1 indicating the zoom range. It is superimposed on the wide-angle image I0 and displayed on the monitor 270. The frame F1 can be a rectangle centered on the position of the mark C1 set in the above procedure. By operating a button (for example, an up / down button (not shown)) of the operation unit 250, the user makes the frame F1 into a desired size as shown in the portions (b) and (c) of FIG. Can be changed. Here, it is assumed that the zoom center is fixed at the position shown in the portion (c) of FIG. 7 (it can be determined by operating the OK button (not shown)).

The zoom setting unit 240B sets the zoom center and the zoom range determined by the above operation as the zoom center and the zoom range of the telephoto image (second image) (step S110).

<Focus position setting>
The image processing unit 240E sets the focus position for the telephoto image and the intermediate image (step S120: focus position setting step). The image processing unit 240E can make this setting based on the user's operation via the operation unit 250 and the monitor 270 as in the following example. The "focus position" means "the difference in the degree of blur between the image that is the reference for the blurring process (for example, a wide-angle image) and the image of interest". Specifically, the image processing unit 240E has, for example, a message M3 prompting the setting of the focus position (“focus position setting” in the example of FIG. 8) and messages M3A, M3B, M3C indicating the focus position (“degree of blur:”, respectively: "Weak", "Bokeh degree: Strong", and "Match the degree of blur to a wide-angle image") are displayed on the monitor 270. On the other hand, by operating a button (for example, an up / down button (not shown)) of the operation unit 250, the user operates the parts (a), (b), and (c) of FIG. You can select the focus position. The image processing unit 240E sets the focus position determined by the above operation as the focus position of the telephoto image and the intermediate image (step S120).

<Setting the number of shots>
The second image acquisition unit 240C and the intermediate image acquisition unit 240D set the number of shots N (the number of shots actually taken by the image pickup apparatus 10) (step S130: the number of shots setting step). This setting can be performed based on the user's operation via the operation unit 250 and the monitor 270 as in the following example. Specifically, the second image acquisition unit 240C and the intermediate image acquisition unit 240D indicate, for example, a message M4 prompting the setting of the number of shots (in the example of FIG. 9, "Please set the number of shots") and the number of shots. The message M4A is displayed on the monitor 270. On the other hand, the user selects the number of shots as shown in the part (a), the part (b), and the part (c) of FIG. 9 by operating a button, a dial, etc. (not shown) of the operation unit 250. be able to. The second image acquisition unit 240C and the intermediate image acquisition unit 240D set the number of shots N based on the number specified by the above operation (step S130). When the number of shots N = 0, no shooting is performed and a telephoto image and an intermediate image are generated by electronic zoom. On the other hand, when the number of shots N ≧ 1, N images (from the 0th to the (N-1) th) are actually taken, and the image with the narrowest angle of view is taken as a telephoto image. (N-1) images excluding the above can be used as an intermediate image.

<Shooting>
The second image acquisition unit 240C and the intermediate image acquisition unit 240D reset the value of i (counter) to zero (step S140) as an initial setting, proceed to step S150, and determine whether or not the number of shots N is 0. .. When the number of shots N is zero, YES is set in step S150, the process proceeds to step S220, and a second image and an intermediate image are generated from the first image by electronic zoom processing. On the other hand, when the number of shots N is not zero (NO in step S150), the process proceeds to step S160, and the second image acquisition unit 240C and the intermediate image acquisition unit 240D display the imaging guide on the monitor 270. Specifically, for example, a message M5 prompting for shooting (in the example of FIG. 10, "Please shoot the first image") is displayed. The second image acquisition unit 240C and the intermediate image acquisition unit 240D further include an image I1A (a live view image, a preview image, etc. can be used) corresponding to the current shooting direction, zoom center, and zoom range, and a reference image. I1 (information indicating the shooting angle of view of the telephoto image and the intermediate image) is displayed on the monitor 270 (see the portion (a) in FIG. 10). In the example shown in the portion (a) of FIG. 10, the reference image I1 is an image that is expected or expected to be photographed, and for example, a plurality of images having different zoom magnifications (shooting angles of view) have a desired effect (for example, of a subject). This is the first image when the locus changes continuously or smoothly). When shooting a plurality of images having different zoom magnifications, it is possible to display a reference image (for example, the reference image I2 in FIG. 12) set according to each zoom magnification at each shooting. preferable.

The user can easily capture an intermediate image and a telephoto image for achieving the intended effect by matching the image I1A with the reference image I1 by changing the zoom center, the zoom range, and the like. The portion (b) in FIG. 10 shows a state in which the image corresponding to the current shooting direction, the zoom center, and the zoom range is the image I1B. In this example, the zoom center and the zoom range are closer to the reference image I1 than the image I1A by the user's operation. The second image acquisition unit 240C and the intermediate image acquisition unit 240D compare the current image (images I1A and I1B) with the reference image I1 to calculate differences in the zoom center, zoom range, etc., and make a difference. If it is larger than the threshold value, a warning message ("zoom to the telephoto side", "pan to the right", etc.) may be output (display, voice, etc.).

The second image acquisition unit 240C and the intermediate image acquisition unit 240D determine whether or not an instruction to start shooting has been input (step S170). This determination can be made, for example, depending on whether or not the user presses down the release button (not shown). The second image acquisition unit 240C and the intermediate image acquisition unit 240D wait until an instruction to start shooting is input (until YES is obtained in step S170).

The second image acquisition unit 240C, the intermediate image acquisition unit 240D, and the image processing unit 240E correct the focus position of the image to be captured (step S180). In step S180, the degree of blurring of the image to be captured is changed by focus control, and the difference in the degree of blurring from the image (for example, a wide-angle image) that is the reference of the blurring process is set in step S120 at the focus position (see FIG. 8). (In step S190, which will be described later, a blurred image is taken according to the focus position corrected in step S180). Focus control can be performed by the lens drive control unit 240H and the lens drive unit 140 driving the focus lens 120 forward and backward in the direction of the optical axis L (see FIG. 1). By such focus control (correction of focus position), for example, it is possible to express effects such as "expressing the locus of an image with clear lines" and "the locus of an image gradually blurs with zooming". ..

The second image acquisition unit 240C and the intermediate image acquisition unit 240D acquire (acquire) a telephoto image and / or an intermediate image via the interchangeable lens 100, the image pickup element 210, and the like (step S190: second image acquisition step, intermediate image). Acquisition process). FIG. 11 is a diagram showing an image I1R which is an example of an actually captured image (first image).

The second image acquisition unit 240C and the intermediate image acquisition unit 240D determine whether or not the number of captured images is N or more (step S210), and repeat the processes of steps S160 to S200 until the determination result becomes YES. ..

FIG. 12 is a diagram showing a state of the second shooting. The part (a) in FIG. 12 corresponds to the message M6 prompting the shooting by the second image acquisition unit 240C and the intermediate image acquisition unit 240D, the current zoom center, and the zoom range, as in the case of the first image shooting described above. The image I2A (for example, a live view image and a preview image) and the reference image I2 (information indicating the shooting angle of view of the telescopic image and the intermediate image) are displayed on the monitor 270. The user can easily capture an intermediate image for achieving the intended effect by matching the image I2A with the reference image I2 by changing the zoom center and the zoom range. The portion (b) in FIG. 12 shows a state in which the image corresponding to the current zoom center and zoom range is the image I2B. FIG. 13 is a diagram showing an image I2R which is an example of an actually captured image (second image).

<Image generation by electronic zoom>
Depending on the effect that the photographer wants to obtain (for example, "I want to express the trajectory of the subject smoothly"), it may be preferable to use many images in the generation of the composite image. However, it takes time and effort to perform many actual shootings. Therefore, in the first embodiment, when the number of captured images is smaller than the number of images required to obtain the desired effect, the intermediate image acquisition unit 240D electronically outputs intermediate images having different zoom centers and / or zoom ranges. It can be generated by zooming (step S220: intermediate image acquisition step). For example, the intermediate image acquisition unit 240D may generate (acquire) an intermediate image of the required zoom center and zoom range by cutting out a portion of the wide-angle image (first image) corresponding to the zoom center and the zoom range. can. Then, the intermediate image acquisition unit 240D can generate a required number of images by repeating such electronic zooming. The intermediate image acquisition unit 240D does not zoom the wide-angle image, but cuts out the portion corresponding to the zoom center and the zoom range of the intermediate image having a wide angle of view, so that the intermediate image of the required zoom center and zoom range is obtained. May be generated (acquired).

As described above, by generating the insufficient number of intermediate images by the electronic zoom, the number of images required to obtain the desired effect can be obtained quickly and easily. In addition, it may be difficult to take a large number of images for a moving subject, but in this case as well, the number of images required to obtain the desired effect can be quickly and easily obtained by using the electronic zoom. be able to. When the number of shots N = 0, the second image acquisition unit 240C and the intermediate image acquisition unit 240D generate all the telephoto image and the intermediate image by electronic zoom.

<Image processing for each image>
The image processing unit 240E performs at least one process of moving, enlarging or reducing, rotating, and changing the degree of blur on the first image, the second image, and the intermediate image (step S230: image processing step). This image processing is a process for producing a desired effect of the composite image. The image processing unit 240E can set the content and degree of processing according to the user's operation (for example, the content and degree of "desired effect") via the operation unit 250. For example, when enlarging or reducing an image, techniques such as bilinear interpolation method and bicubic interpolation method can be used. However, especially when the center position is set to deviate from the center of the wide-angle image, the farther the point is from the center of scaling, the larger the amount of movement and the more noticeable the discontinuity during image composition, so the point farthest from the center of scaling looks continuous. Processing is performed at the same scaling ratio (described later). Further, when changing the degree of blur (change by image processing after shooting), a technique such as a Gaussian filter can be used. Further, if the image expected to be photographed and the image actually photographed are different from each other, the scaling center may be corrected (described later).

At the time of image processing in step S230, the image processing unit 240E may display a combination of processing contents and degrees on the monitor 270 and perform processing according to the combination selected by the user. The image to be processed can be selected from a wide-angle image (first image), a telephoto image (second image), and an intermediate image according to the content and degree of the effect to be played. In addition to moving, enlarging or reducing, rotating, and changing the degree of blur, the color and / or brightness of the image may be changed.

<Example of processing in the image processing process>
<Correction of scaling center>
In the first embodiment, the shooting guide (display of a message, reference image, etc.) makes it easy to match the image expected to be shot with the image actually shot, but still, shooting is expected. There is a possibility that the image will be different from the image actually taken. For example, the image expected to be captured is the image I3 shown in the portion (a) of FIG. 14, and the actually captured image (intermediate image or telephoto image) is the image I3R shown in the portion (b) of FIG. If so, the shooting range of the image I3 and the image I3R are different from each other. In this case, the image processing unit 240E superimposes the images I3 and I3R and calculates the correlation while shifting the images I3R little by little, and adjusts the direction and amount of the shift so that the correlation is the highest. (Zoom center) is corrected (step S230: image processing step). FIG. 15 is a diagram showing an example of correction of the scaling center. The parts (a), (b), and (c) of FIG. 15 are "a state in which images I3 and I3R are superimposed as they are", "a state in which images I3 and I3R are shifted and overlapped", and "images I3 and I3", respectively. A state in which I3R is shifted more than the state of the portion (b) and superposed (an example of a state in which the correlation is highest) ”is shown.

The zoom setting unit 240B and the image processing unit 240E perform relative enlargement / reduction / rotation between the image expected to be taken and the image actually taken, in addition to the correction of the enlargement / reduction center by movement. The deviation may be corrected. Image processing such as enlargement may be performed every time an image is taken, instead of being performed after all the images have been taken. It should be noted that such movement, enlargement or reduction, and rotation of the image can be performed by affine transformation of a wide-angle image (first image), an intermediate image, or a telephoto image (second image).

<Adjustment of scaling ratio>
When enlarging or reducing (zooming, changing the angle of view) an image to give the photographer's intended effect, if the image is enlarged or reduced at a fixed ratio, the farther the image is from the center of enlargement (zoom center), the more the amount of movement is. Is large, and the discontinuity of the locus is easily noticeable in the composite image. For example, the portion (a) in FIG. 16 is a diagram showing a state in which the zoom ratio of an image is kept constant and enlarged. Specifically, 125% of the zoom range Z1 is defined as the zoom range Z2, 125% of the zoom range Z2 is defined as the zoom range Z3, and 125% of the zoom range Z3 is defined as the zoom range Z4. When these zoom ranges are superimposed, the amount of movement between the zoom ranges of the upper right point farthest from the center of scaling is the amount of movement ΔZ3 (the amount of movement between the zoom ranges Z4 and Z3)> the amount of movement ΔZ2 (zoom range Z3). , Movement amount between Z2)> movement amount ΔZ1 (movement amount between zoom range Z2 and Z1), and the interval increases from the inside to the outside (that is, from the zoom range Z1 to the zoom range Z4). The appearance of spreading is conspicuous.

From this point of view, in the first embodiment, the zoom setting unit 240B and the image processing unit 240E adjust the scaling ratio (zoom magnification, shooting angle of view) to the point farthest from the scaling center (in the example of FIG. 16). Make it appear continuous (change at regular intervals) even at the upper right point of the zoom range (step S230: image processing step). For example, as shown in the portion (b) of FIG. 16, the movement interval of the upper right point is made uniform in the movement range ΔZ4. Such image processing can be performed by the zoom setting unit 240B and the image processing unit 240E setting the shooting angle of view of the intermediate image according to the distance between the zoom center and the center of the wide-angle image. Specifically, the above-mentioned image processing can be realized by the zoom setting unit 240B and the image processing unit 240E gradually changing the scaling ratio outside the zoom range (in a state where the angle of view is wide) (described later). It should be noted that such processing may be performed at the time of shooting (for example, steps S110, S160, etc.) to shoot an image in which the scaling center is adjusted.

<Processing of scaling ratio adjustment>
The process of adjusting the scaling factor described above will be described in detail. As shown in FIG. 17, the width of the image having a wider angle of view is W pixels, and the height is H pixels. It is assumed that the image having a narrow angle of view has an angle of view corresponding to (W−ΔW) pixels in width and (H—ΔH) pixels in the image having a wide angle of view. The zoom setting unit 240B and the image processing unit 240E calculate the scaling ratio based on the smaller of W and H (W> H in the example of FIG. 17). The maximum distance between the scaling center point and the point farthest from the scaling center point is when the scaling center points are at the four corners of the image, so continuity may be maintained even in this case. In this case, if the width due to enlargement becomes larger than one pixel, discontinuity occurs. Therefore, the zoom setting unit 240B and the image processing unit 240E have the number of steps of ΔW so that the width due to enlargement increases by one pixel. Enlarge processing is performed. That is, the enlargement ratio M_j at each step count j (j = 1, 2, ..., ΔW) is M_1 = (W−ΔW + 1) ÷ (W−ΔW).
M_1 = (W-ΔW + 2) ÷ (W-ΔW + 1)
…
M_ΔW = W ÷ (W-1)
And.

<Effect of scaling ratio adjustment>
FIG. 18 is a diagram for explaining the effect of the above-mentioned scaling ratio adjustment. The portion (a) in FIG. 18 is a specific example of an image in which the locus cannot be continuously seen as in the portion (a) in FIG. For example, the discontinuity caused by the region 301 of the bird's feather portion appears in the region 302. On the other hand, the part (b) in FIG. 18 is a specific example of an image in which the locus can be seen continuously by setting the enlargement ratio by the above-mentioned method as in the part (b) in FIG. .. As described above, according to the first embodiment, it is possible to easily obtain the effect intended by the photographer in the inter-exposure zoom image (in this case, the trajectory of the subject is continuously and smoothly changed).

<Generation of composite image by weighted average>
The image composition unit 240I generates a composite image by weighting and adding the images (wide-angle image, telephoto image, or intermediate image) processed up to step S230 (step S240: image composition step). The weight for the image is the degree of emphasis of the image at the start, middle, and end of the zoom (blurring from the start to the end of the zoom, or conversely, the blurred state at the start of the zoom becomes clearer toward the end of the zoom. , Etc.) can be taken into consideration when setting. By such weighting addition, it is possible to generate an image that has the effect of zoom shooting between exposures.

<Display of composite image>
The display control unit 240G causes the monitor 270 to display the composite image generated in step S240 (step S250: image display step). In addition to displaying the composite image, it may be stored in the storage unit 260. Further, the image synthesizing unit 240I sequentially synthesizes a wide-angle image, an intermediate image, and a telephoto image, and continuously changes from a state in which the number of added images is small to a state in which all the images are added, like a moving image (or like an animation). By displaying the image, the effect such as "the subject pops out" or "is sucked in" may be emphasized.

<Example 1 of composite image>
FIG. 19 is a diagram showing an example of a composite image (an image obtained by zooming and synthesizing an actually captured image). In the image I4 shown in FIG. 19, the subject flows radially from the zoom center I4C, and it is possible to give the impression that the subject pops out (or is sucked in).

<Example 2 of composite image>
FIG. 20 is a diagram showing another example of a composite image (composite image by simulation). In the example of FIG. 20, 41 images (40 intermediate images, 1 telephoto image) obtained by enlarging and unsharp (enhancing) the original image I5A (wide-angle image) shown in the portion (a) of FIG. 20. Sheets) are weighted and averaged to generate the composite image I5B (composite image) shown in the portion (b) of the figure. In the composite image I5B, the effect that the subject I5X in the original image I5A is zoomed (enlarged) in a clear state can be obtained.

<Example 3 of composite image>
FIG. 21 is a diagram showing another example of a composite image (composite image by simulation). In the example of FIG. 21, the original image I6A (wide-angle image; same as the original image I5A shown in the part (a) of FIG. 20) shown in the portion (a) of FIG. 21 is enlarged by the affine transformation and the degree of blurring by the Gaussian filter. The 41 images (40 intermediate images and 1 telescopic image) subjected to the change processing of the above are weighted and averaged to generate the composite image I6B (composite image) shown in the portion (b) of the figure. In the composite image I6B, the effect that the subject I6X in the original image I6A is blurred as the zoom (enlargement) is obtained can be obtained.

<Example 4 of composite image>
FIG. 22 is a diagram showing still another example of the composite image (composite image by simulation). In the example of FIG. 22, 41 images obtained by enlarging and rotating the original image I7A (wide-angle image) shown in the portion (a) of FIG. 22 by affine transformation and changing (blurring) the degree of blurring by the Gaussian filter. (40 intermediate images and 1 telescopic image) are weighted and averaged to generate the composite image I7B (composite image) shown in the portion (b) of the figure. In the composite image I7B, the effect that the subject I7X in the original image I7A is blurred while being zoomed (enlarged) and rotated can be obtained.

<Example 5 of composite image>
FIG. 23 is a diagram showing still another example of the composite image (composite image by an actual image). In the example of FIG. 23, 41 images obtained by subjecting the original image I8A (wide-angle image) shown in the portion (a) of FIG. 23 to enlargement by affine transformation and change (blurring) of the degree of blurring by the Gaussian filter ( 40 intermediate images and 1 telescopic image) are weighted and averaged to generate the composite image I8B (composite image) shown in the portion (b) of the figure. In the composite image I8B, the effect that the subject I8X in the original image I8A is blurred while zooming (enlarged) can be obtained.

<Effect of the first embodiment>
As described above, according to the first embodiment, the first image, the second image, and the intermediate image having different shooting angles of view are weighted and added, so that the effect of zoom shooting between exposures can be obtained. Further, since at least one of movement, enlargement, rotation, and change of the degree of blurring is performed in the generation of the composite image, the effect desired by the user can be easily obtained without performing complicated and difficult operations.

<Others>
Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention. For example, the image pickup apparatus in the first embodiment can be realized by a digital camera, a smartphone, a tablet terminal, or the like. Further, instead of providing an image pickup unit as in the image pickup apparatus 10, a mode may be adopted in which an image captured separately is acquired via a recording medium, a network, or the like and processed by a computer.

10 Imaging device 100 Interchangeable lens 110 Zoom lens 120 Focus lens 130 Aperture 140 Lens drive unit 200 Imaging device main unit 210 Imaging element 220 AFE
230 A / D converter 240 Image processing device 240A First image acquisition unit 240B Zoom setting unit 240C Second image acquisition unit 240D Intermediate image acquisition unit 240E Image processing unit 240F Information display unit 240G Display control unit 240H Lens drive control unit 240I Image Composite unit 242 ROM
250 Operation unit 260 Storage unit 270 Monitor 301 Area 302 Area C1 Mark F1 Frame I0 Wide angle image I1 Reference image I1A Image I1B Image I1R Image I2 Reference image I2A Image I2B Image I2R Image I3 Image I3R Image I4 Image I4C Zoom center I5A Original Image I5B Composite image I5X Subject I6A Original image I6B Composite image I6X Subject I7A Original image I7B Composite image I7X Subject I8A Original image I8B Composite image I8X Subject L Optical axis M1 Message M2 Message M3 Message M3A Message M4 Message M4A Message M5 Message M6 Message N Number of images taken S100 to S250 Each step of the image processing method Z1 Zoom range Z2 Zoom range Z3 Zoom range Z4 Zoom range ΔZ1 Movement amount ΔZ2 Movement amount ΔZ3 Movement amount ΔZ4 Movement range

Claims (11)

The first image acquisition unit that acquires the first image,
A zoom setting unit that sets the zoom center and zoom range for the first image, and
A second image acquisition unit that acquires a second image corresponding to the zoom center and the zoom range, and
An intermediate image acquisition unit that acquires a specified number of intermediate images having a narrower shooting angle of view than the first image and a wider shooting angle of view than the second image.
An image processing unit that performs at least one process of moving, enlarging or reducing, rotating, and changing the degree of blurring of the first image, the second image, or the intermediate image.
An image compositing unit that generates a composite image by weighting and adding the first image, the second image, or the intermediate image that has undergone the processing.
Equipped with
The intermediate image acquisition unit is an image processing device that displays information indicating the shooting angle of view of the intermediate image on a display device and acquires an image shot in response to the information as the intermediate image . The image processing apparatus according to claim 1, wherein the second image acquisition unit acquires the second image by cutting out a portion of the first image corresponding to the zoom center and the zoom range. The intermediate image acquisition unit acquires a plurality of images having different shooting angles of view as the specified number of the intermediate images, and the image processing unit performs the processing on each of the plurality of images. 2. The image processing apparatus according to 2. The zoom setting unit is any one of claims 1 to 3 that moves the zoom center according to the deviation between the zoom center indicated by the information and the center of the image captured in response to the information. The image processing apparatus described in the section. The image processing apparatus according to any one of claims 1 to 4, wherein the image processing unit moves, enlarges, reduces, or rotates the intermediate image and / or the second image by affine transformation. The image processing device according to any one of claims 1 to 5, wherein the zoom setting unit sets the zoom center and the zoom range based on a user's instruction input. The image processing device according to any one of claims 1 to 6, wherein the image processing unit changes the degree of blurring based on a user's instruction input. The image processing apparatus according to any one of claims 1 to 7, wherein the zoom setting unit sets the shooting angle of view of the intermediate image according to the distance between the zoom center and the center of the first image. The image processing device according to any one of claims 1 to 8, further comprising a display control unit for displaying the composite image on the display device. An image pickup unit that captures a subject image with a shooting lens including a zoom lens,
An image pickup apparatus comprising the image processing apparatus according to any one of claims 1 to 9.
The zoom setting unit controls the zoom lens according to the zoom center and the zoom range.
The intermediate image acquisition unit is an image pickup device that acquires the intermediate image corresponding to the zoom center and the zoom range by the image pickup unit. The first image acquisition process for acquiring the first image and
A zoom setting process for setting the zoom center and the zoom range for the first image, and
A second image acquisition step of acquiring a second image corresponding to the zoom center and the zoom range, and
An intermediate image acquisition step of acquiring a specified number of intermediate images having a narrower shooting angle of view than the first image and a wider shooting angle of view than the second image.
An image processing step of performing at least one process of moving, enlarging or reducing, rotating, and changing the degree of blurring of the first image, the second image, or the intermediate image.
An image synthesizing step of weighting and adding the first image, the second image, or the intermediate image to which the processing has been performed to generate one image.
Have,
In the intermediate image acquisition step, an image processing method in which information indicating the shooting angle of view of the intermediate image is displayed on a display device, and an image shot in response to the information is acquired as the intermediate image.

Images