JP2020080518A - Image processing apparatus - Google Patents

Abstract

To provide an image processing apparatus capable of generating a favorable panning image even if a background and a motion of a subject being follow-photographed are close to each other.SOLUTION: A motion blur prevention part 200 of an image processing apparatus comprises: reference region designation means for designating a reference region; motion blur correction target region specification means 205 for automatically specifying a motion blur correction target region having a prescribed association with the reference region; reference motion blur specification means 202 for specifying a reference motion blur from the motion blur correction target region; and motion blur control means 203 for changing a motion blur inside the motion blur correction target region so that the reference motion blur becomes a prescribed motion blue, and performing control so that a motion blur change amount inside the motion blur correction target region and a motion blur change amount outside the motion blur correction target region are different from each other.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing device and an image processing method, and more particularly to a technique for controlling motion blur of a subject that occurs in an image.

  There is a technique called panning as a shooting technique that expresses the sense of speed of a moving subject. In this follow shot technique, the exposure time is set to be longer than usual, and the photographer follows the camera so that the photographer follows the movement of the subject. In the follow shot image, the moving subject is stationary, and the background is expressed as a motion blur. On the other hand, there is a technique for generating such a follow shot image by image processing. This technology analyzes the movement between multiple images of a moving subject and adds motion blur based on that movement to generate an image that looks as if it was actually panned by image processing. It is a technology to do.

  As a technique for adding motion blur by this image processing, there is Patent Document 1. According to Japanese Patent Laid-Open No. 2004-242242, a technique is disclosed in which a plurality of consecutive images are captured and the amount of movement of the background is calculated from the images between the plurality of images to add motion blur to the background.

JP, 2010-15483, A

In such panning, it is desirable that the moving subject be stationary and the background be blurred. However, during follow-up shooting, if the movement of the subject cannot be tracked, the position of the subject shifts between multiple images, and motion blur is added to the subject according to the amount of movement, so the moving subject is blurred. There is a problem that it will end.
In particular, when the background and the subject of the follow-up shot are close to each other in motion, the amount of movement cannot be well separated, and the image processing cannot properly shake the subject during the follow-up shot.

  The present invention has been made in view of the above problems, and easily generates a follow shot image in which the motion blur of the subject during follow shooting is reduced even when the background and the follow subject are close in motion. It is an object of the present invention to realize an image processing device capable of

In the image processing device, a reference area designating unit for designating a reference area,
Motion-blur correction target area specifying means for automatically specifying a motion-blur correction target area having a predetermined relationship with the reference area;
A reference motion blur identifying unit that identifies a reference motion blur from the motion blur correction target area;
A motion blur control that changes the motion blur in the motion blur correction target area so that the reference motion blur becomes a predetermined motion blur, and controls the variation amount of the motion blur to be different inside and outside the motion blur correction target area. And a control means.

  According to the present invention, it is possible to easily generate a follow shot image in which the motion blur of the subject at the time of the follow shooting is reduced even when the background and the follow subject are in close motion.

FIG. 3 is a diagram showing a configuration example of the image processing apparatus of the first embodiment. 3 is a diagram showing a configuration example of a motion blur control unit 200 in Embodiment 1. FIG. 6 is a flowchart showing a processing flow of the motion blur control unit 200 in the first embodiment. The figure which shows the captured image in a representative scene. Step S302 Flowchart showing a processing flow of automatic motion blur control. 6 is a flowchart showing a processing flow of the motion vector calculation unit 201. The figure which shows the calculation method of a motion vector. The figure which shows the motion vector in a representative scene. The figure which shows a motion blur conversion characteristic. The figure which shows the area|region detection method of the object area|region of motion blurring correction. The figure which shows the motion blur provision method. Step S304 is a flowchart showing a process flow of motion blur adjustment. The figure which shows the effect of the movement blurring adjustment in a representative scene. The figure which shows a motion blur conversion characteristic. The figure which shows the user display screen at the time of performing motion blur conversion.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

The first embodiment of the present invention controls the motion blur of an image based on an instruction regarding the motion blur by the user. Hereinafter, Example 1 will be described.
FIG. 1 is a block diagram showing a configuration example when the motion blur control technique of the present embodiment is applied to an image pickup apparatus 100 as an example of an image processing apparatus. The imaging device 100 includes, for example, a camera and a smartphone. The block configuration of the first embodiment will be described below with reference to FIG. In the first embodiment, the case where the image processing apparatus is applied to the image pickup apparatus 100 will be described, but the present invention is not limited to the image pickup apparatus, and may be, for example, a PC or an image reproducing apparatus having no image pickup function. ..

  The control unit 101 is, for example, a CPU as a computer, and reads an operation control program (computer program) for each block included in the imaging apparatus 100 from a ROM 102 as a storage medium described later, expands it in a RAM 103 described later, and executes the RAM 103. As a result, the control unit 101 controls the operation of each block included in the imaging device 100. The ROM 102 is an electrically erasable and recordable non-volatile memory, and stores the operation control program of each block included in the imaging apparatus 100, as well as parameters necessary for the operation of each block. The RAM 103 is a rewritable volatile memory, and is used for developing a program executed by the control unit 101 and the like, and temporarily storing data generated by the operation of each block included in the imaging apparatus 100.

  The optical system 104 includes a lens group including a zoom lens and a focus lens, and forms a subject image on an image pickup surface of an image pickup unit 105 described later. The image pickup unit 105 is, for example, an image pickup device such as a CCD or a CMOS sensor, photoelectrically converts an optical image formed on the image pickup surface of the image pickup unit 105 by the optical system 104, and converts the obtained analog image signal into an A/D signal. It is output to the conversion unit 106. The A/D converter 106 converts the input analog image signal into digital image data. The digital image data output from the A/D conversion unit 106 is temporarily stored in the RAM 103. The image processing unit 107 applies various image processing such as white balance adjustment, color interpolation, and gamma processing to the image data stored in the RAM 103. The image processing unit 107 also includes a motion blur control unit 200, and generates a motion blur image in which motion blur is added to the image stored in the recording unit 108.

  The recording unit 108 includes a removable memory card and the like. The recording unit 108 records the image data processed by the image processing unit 107 as a recorded image via the RAM 103. The recording unit 108 can also output the recorded image data to the image processing unit 107 via the RAM 103. The display unit 109 is a display device such as an LCD, and displays an image temporarily stored in the RAM 103 and an image recorded in the recording unit 108, and an operation user interface image for receiving an instruction from the user. And so on. The instruction input unit 110 is a touch panel, a mouse, a keyboard, or the like, and receives an instruction from the user.

Next, the operation of the image processing unit 107, which is a feature of this embodiment, will be described in detail. In this embodiment, an example of controlling motion blur for a captured image with a short exposure time will be described.
First, a configuration example of the motion blur control unit 200 included in the image processing unit 107 will be described with reference to FIG. The motion blur control unit 200 adds motion blur to the image data recorded in the recording unit 108 to generate a motion blur image.
FIG. 2 is a diagram showing a configuration example of the motion blur control unit 200. The motion blur control unit 200 includes a motion vector calculation unit 201, a reference motion vector identification unit 202 as a reference motion blur identification unit, a motion blur conversion characteristic calculation unit 203, a motion blur imparting unit 204, and a motion blur correction target area identification unit 205. It is configured.
Next, the processing of the motion blur control unit 200 will be described with reference to the flowchart of FIG.

  In step S301 of FIG. 3, the control unit 101 determines the exposure time when the image capturing unit 105 captures an image. Then, the image capturing unit 105 captures (captures) a plurality of images based on the determined exposure time and records the images in the recording unit 108. In this embodiment, an example will be described in which 60 images are picked up per second as an image pickup frame rate. That is, the image capturing unit 105 captures one image every 1/60 seconds. The exposure time may be determined by the user via the user interface of the display unit 109 or may be determined by the control unit 101 by automatic exposure control. As a method of determining the exposure time by the automatic exposure control, for example, there is a method of determining the exposure time based on the photometric value for each predetermined area of the digital image data captured by the image capturing unit 105. Although the details will be described later, the exposure time determined here is shorter than that in the normal follow shot, and an image with less moving body blur is captured. An example of an image to be captured will be described with reference to FIG. FIG. 4 shows an example in which the photographer shakes the imaging device 100 so as to follow a running dog, and performs continuous shooting (follow shooting) while following, and FIG. 4A shows the Nth frame. A captured image, FIG. 4B shows a captured image of the (N+1)th frame. Note that N is a positive integer. Further, since the captured images of the Nth frame and the N+1th frame are captured with a short exposure time, no large motion blur occurs in the captured image.

During follow-up shooting, it is very difficult to follow the details of the running dog perfectly. In the example of FIG. 4, the movements of the front and rear feet of the running dog are different from those of the dog's torso, and therefore change between the Nth frame and the N+1th frame.
In step S302 of FIG. 3, the motion blur control unit 200 adds motion blur to the captured image of the Nth frame captured in step S301, and displays the motion blur image on the display unit 109. Details of the motion blur addition in step S302 will be described later with reference to FIG.

In step S303 of FIG. 3, the user confirms the motion blur image displayed on the display unit 109 in step S302, and the instruction input unit 110 receives an instruction as to whether or not the motion blur adjustment is necessary. If the instruction that the adjustment of the motion blur is necessary is not received in step S303, the flow of FIG. 3 ends. In step S303, when the instruction that the movement blurring adjustment is necessary is received, the process proceeds to step S304 in FIG.
In step S304 of FIG. 3, the motion blur is adjusted on the captured image of the Nth frame captured in the process of step S301 based on the user instruction regarding the motion blur, and the motion blur image is displayed on the display unit 109. The method of adjusting the motion blur in step S304 will be described later with reference to FIG.

In step S305 of FIG. 3, the user confirms the motion blur image displayed on the display unit 109 by the process of step S304, and the instruction input unit 110 receives an instruction as to whether or not the motion blur adjustment is completed. If the instruction that the adjustment of the motion blur is completed is accepted in step S305, the flow of FIG. 3 ends. On the other hand, if the instruction that the adjustment of the motion blur is completed is not accepted in step S305, the process returns to step S304 of FIG. 3 and the motion blur adjustment is repeated.
Next, details of the process of automatically adding motion blur in step S302 of FIG. 3 will be described with reference to the flowchart of FIG.

  In step S501 of FIG. 5, the control unit 101 calculates target motion blur information. The target motion blur information is information indicating how many seconds the exposure time corresponds to the target motion blur. The target motion blur information changes the length of the motion blur. For example, the length of the motion blur corresponding to 1/60 seconds is twice the length of the motion blur corresponding to 1/120 seconds.

In step S502 of FIG. 5, the motion vector calculation unit 201 of FIG. 2 calculates the motion vector between the captured images recorded in the recording unit 108. Then, the calculated motion vector is output to the reference motion vector specifying unit 202 and the motion blur providing unit 204 in FIG. Here, a method of calculating the motion vector will be described with reference to FIGS. 6, 7, and 8.
FIG. 6 is a flowchart showing the motion vector calculation processing of step S502 by the motion vector calculation unit 201, and FIG. 7 is a diagram showing a motion vector calculation method by the block matching method. In this embodiment, a block matching method will be described as an example of a motion vector calculation method, but the motion vector calculation method is not limited to this. For example, an optical flow method may be used.

In step S601 of FIG. 6, the motion vector calculation unit 201 of FIG. 2 inputs the two captured images recorded in the recording unit 108. Then, the captured image of the Nth frame is set as the reference captured image, and the captured image of the (N+1)th frame that is temporally later is set as the reference captured image.
In step S602 of FIG. 6, a reference block 702 of N×N pixels is arranged in the reference captured image 701 as shown in FIG.
In step S603 of FIG. 6, as shown in FIG. 7, in the reference captured image 703, the surrounding (N+n)×(N+n) pixels of the same coordinates 704 as the reference block 702 of the reference captured image 701 are set in the search range 705.

In step S604 of FIG. 6, the correlation calculation is performed between the reference block of N×n pixels having different coordinates existing in the search range 705 of the reference captured image 703 and the standard block 702 of the standard captured image 701 to calculate the correlation value. The correlation value is calculated based on the sum of absolute differences between images for the pixels of the standard block 702 and the reference block. That is, the coordinate with the smallest sum of absolute differences is the coordinate with the highest correlation value. The method of calculating the correlation value is not limited to the sum of absolute differences. For example, the correlation value may be calculated based on the sum of squared differences or the normal cross-correlation value. The example of FIG. 7 shows that the reference block 706 has the highest correlation.
In step S605 of FIG. 6, a motion vector is calculated based on the reference block coordinates showing the highest correlation value. In the example of FIG. 7, the difference between the center coordinates of the standard block 702 of the standard captured image 701 and the center coordinates of the reference block 706 is the motion vector.

In step S606 of FIG. 6, while moving the reference block 702 of FIG. 7, the processes of step S602, step S603, step S604, and step S605 are repeated to calculate motion vectors of all pixels of the reference captured image 701. Note that the motion vector may be calculated for each predetermined pixel instead of calculating the motion vectors of all pixels.
FIG. 8 shows an example of the motion vector between the captured images calculated based on the above method. FIG. 8 is a diagram showing motion vectors based on difference information when the Nth frame captured image of FIG. 4A is the reference captured image and the Nth frame captured image of FIG. 4B is the reference captured image. .. The arrow in FIG. 8 indicates a motion vector, the length of the arrow indicates the length of the motion vector, and the direction of the arrow indicates the direction of the motion vector. In the example of FIG. 8, the motion vector for each pixel is not drawn, but only representative motion vectors are simplified and shown.

  In the example of FIG. 8, the imaging device 100 is swung from right to left, and the follow-up image of the dog running in the center is taken. Therefore, the stationary object in the background is detected as a motion vector in the right direction, and the vicinity of the running dog's torso (800) has the same motion as the imaging device 100, and the relative speed between the imaging device 100 and the dog is 0. 0 is calculated as the motion vector. However, with respect to the swing of the limbs of the dog, a motion vector other than 0 appears because the speed and the motion are different from the follow shooting direction. The motion vectors of the front legs are represented by 801, 802, and the motion vectors of the hind legs are 803, 804. The front legs (801, 802) and the rear legs (803, 804) are motion vectors in different directions. Furthermore, it is shown that the motion vector of the hind legs has the same motion amount as the background vector. The motion vector has a direction. For this reason, in this embodiment, for example, a downward motion vector in the horizontal direction and a downward motion vector in the vertical direction are negative, and conversely, the right and upward motion vectors are positive.

In step S503 of FIG. 5, the motion blur conversion characteristic calculation unit 203 of FIG. 2 calculates a characteristic for converting a motion vector into motion blur imparting information based on the target motion blur information. The motion blur adding information is composed of a horizontal motion blur amount and a vertical motion blur amount, and the motion blur direction for the motion blur imparting unit 204 to impart motion blur (corresponding to the direction of the motion vector). And information that determines the amount of motion blur (corresponding to the length of the motion vector). A method of calculating the motion blur conversion characteristic will be described with reference to FIG. FIG. 9 is a diagram showing motion blur conversion characteristics. The horizontal axis represents the amount of movement of the motion vector in the horizontal direction, and the vertical axis represents the amount of motion blur in the horizontal direction. Note that FIG. 9 shows a conversion characteristic for converting the horizontal component of the motion vector into the horizontal motion blur amount for simplification. Originally, a motion vector is composed of a horizontal movement amount and a vertical movement amount, but the conversion method and conversion characteristics in the vertical direction are calculated by the same method as in the horizontal direction, and therefore description thereof is omitted.
The motion blur conversion characteristic L901 in FIG. 9 is represented by the following equation (1), and is the motion blur conversion characteristic when the target motion blur information is equivalent to 1/60 second.
Horizontal motion blur amount = Horizontal movement amount ...Equation (1)

Since the frame rate of the captured image is 60 per second, the amount of movement in the horizontal direction is the amount of movement in 1/60 seconds. The motion blur conversion characteristic L901 calculates a motion blur amount corresponding to the movement amount in 1/60 seconds.
Further, the motion blur conversion characteristic L902 in FIG. 9 is represented by Expression (2), and is the motion blur conversion characteristic when the target motion blur information is 1/120.
Horizontal movement blur amount = 1/2 × horizontal movement amount ...Equation (2)
The motion blur conversion characteristic L902 calculates a motion blur amount corresponding to the movement amount in 1/120 seconds, which is half the movement amount in 1/60 seconds.

  That is, when the conversion is performed based on the motion blur conversion characteristic L902, a motion blur amount equivalent to one half of the motion blur amount converted based on the motion blur conversion characteristic 901 is added. By changing the motion blur conversion characteristic in this way, the motion blur amount can be controlled. In step S504 of FIG. 5, the motion blur providing unit 204 refers to the motion blur conversion characteristic calculated by the motion blur conversion characteristic calculation unit 203 and the motion vector for each pixel calculated by the motion vector calculation unit 201. Then, a motion blur is added to the captured image based on both vectors, and the motion blur image is output to the display unit 109. The motion blur adding method can be realized by performing a filter process using a motion vector described in JP 2010-15483 A, for example. The filtering process for adding the motion blur can be performed in pixel units or in pixel block units of a plurality of pixels. Further, the motion blur can be changed by changing the tap number (filter size) of the filter when performing the filter processing based on the size and direction of the motion vector and performing the filter processing.

FIG. 11 is a diagram showing a method of adding motion blur. The motion blur adding information L1101 in FIG. 11 indicates the motion blur adding information of the pixel A of interest. The motion blur imparting information L1101 is motion blur imparting information that passes from the pixel A through the pixel B and the pixel C and reaches the pixel D, and the motion vector represents 4 pixels in the horizontal direction and 4 pixels in the vertical direction. There is. In step S505 of FIG. 5, the motion blur image generated by the motion blur imparting unit 204 in step S504 is displayed on the display unit 109.
The process of the automatic motion blur control in step S302 of FIG. 3 has been described above.

Next, the motion blur adjustment method in step S304 of FIG. 3 will be described in detail with reference to the flowchart of FIG. The motion blur adjustment is performed in order to fulfill the user's request to adjust the motion blur amount for each subject.
In step S1201 of FIG. 12, the control unit 101 receives a designation (instruction) of a designated reference region as a motion blur adjustment instruction from the user via the instruction input unit 110. At this time, the instruction input unit 110 functions as a reference area designating unit. Further, in step S1202 of FIG. 12, the control unit 101 receives, via the instruction input unit 110, a designated motion blur instruction in the designated reference region as a motion blur adjustment instruction from the user. A method of receiving the motion blur adjustment instruction will be described with reference to FIG.

  13A shows an image before motion blur adjustment, FIG. 13B shows an image after motion blur adjustment in which the motion blur amount less than the blur amount of the designated reference area is reduced, and FIG. 7 shows an image after motion blur adjustment in which the amount of motion blur only around the reference area (limbs) is reduced. As shown in FIG. 13B, when the motion blur amount less than or equal to the blur amount of the designated reference region is reduced, the motion blur amount is also reduced at the same time in a background region or the like having a motion blur amount close to the blur amount of the designated reference region. Will end up. Therefore, in the present embodiment, as shown in FIG. 13C, the motion blurring amount is controlled only in the peripheral (limb) region including the designated reference region and related to the subject in the designated reference region. As a result, the motion blur can be adjusted by focusing only on the area desired by the user.

  The image before motion blur adjustment in FIG. 13A is the motion blur image displayed on the display unit 109 after the automatic motion blur control in step S302 in FIG. The user confirms the motion blur image displayed on the display unit 109 and determines whether or not the motion blur image needs to be adjusted (determination in step S303 in FIG. 3). Here, in the example of the image before motion blur adjustment in FIG. 13A, motion blur occurs not only in the background but also in the limbs of the dog running the main subject. FIG. 13A is a diagram in which motion blur is added based on the motion vector of FIG. 8, and the size of the motion vector of the limbs of the dog is not zero. Generally, in a follow shot image, an image in which the main subject is stationary without any motion blur and the background is subject to motion blur is compared to an image in which the main subject has motion blur compared to the image in which the main subject is motion blur. It is said that a sense of speed is outstanding and preferable. Therefore, in this embodiment, the user confirms a motion blur image such as the image before motion blur adjustment in FIG. 13A, and gives an instruction to adjust the motion blur image by a touch operation on the touch panel, a click operation by the mouse, or the like. ..

  Specifically, as shown in L1301 and L1302 of FIG. 13A, the user performs a touch operation or a click operation on an area in which the motion blur is desired to be adjusted. At that time, whether to adjust the motion blur of the designated reference region to be small or large is designated based on the manner of the touch operation or the manner of the click operation. For example, when the user makes a single-touch or single-click designation, the control unit 101 receives an instruction to reduce the movement blur as the designated movement blur of the designated reference area. When the user makes a double-touch or double-click designation, the control unit 101 accepts a designated motion blur in the designated reference region as an instruction to increase the motion blur. It should be noted that the method of designating the designation reference area does not have to be touch or click as described above, but may be designated by inputting coordinate information on the screen with a ten-key or the like.

For example, in the image before motion blur adjustment in FIG. 13A, when the motion blur of the dog is adjusted to be small, the user specifies a part of the limbs (L1301 and L1302) of the running dog with a single click. Then, the control unit 101 accepts the coordinates designated as the designated reference region, automatically detects the designated reference region and the surrounding region, and receives the designated movement blur as an instruction to reduce the motion blur. In the following description, an example will be described in which the user is designated by single-clicking on the running dog to reduce the motion blur of the dog.
It should be noted that the designated reference area received by the control unit 101 does not have to be only one coordinate designated by the user by clicking. For example, when the user specifies to circle the whole running dog or a part thereof, the control unit 101 may accept the enclosed area as the designated reference area.

  In step S502′ of FIG. 12, the motion vector calculation unit 201 calculates the motion vector between the captured images recorded in the recording unit 108. Specifically, the motion vector calculation unit 201 inputs the imaged image of the Nth frame of FIG. 4A and the imaged image of the N+1th frame of FIG. 4B, and based on the difference information between them, as shown in FIG. Calculate the motion vector. The method of calculating the motion vector in step S502' in FIG. 12 is the same as that in step S502 in FIG. 5, and description thereof will be omitted. Further, the motion vector calculated in the automatic motion blur control process of step S302 of FIG. 3 may be recorded in the recording unit 108 and used in step S502′ of FIG.

In step S1203 of FIG. 12, the reference motion vector identification unit 202 identifies the reference motion vector based on the designated reference area accepted by the control unit 101 in step S1201 and the motion vector calculated by the motion vector calculation unit 201. Specifically, in FIG. 8, when four areas of dog motion vectors 801 to 804 that are designated reference areas are selected, the motion vectors of the selected four areas are specified as reference motion vectors (reference motion blur). .. When there are a plurality of motion vectors in the designated reference area, a histogram of the motion vectors in the designated reference area may be acquired and the most frequent motion vector may be used as the reference motion vector.
At this time, when the direction and magnitude of the motion vector are different, four reference motion vectors are used.

  Next, in step S1204 of FIG. 12, the motion blur conversion characteristic calculation unit 203 of FIG. 2 calculates the motion blur conversion characteristic based on the target motion blur, the reference motion vector, and the designated motion blur. A method of calculating the motion blur conversion characteristic will be described with reference to FIG. The motion blur conversion characteristic of FIG. 14 shows an example of the motion blur conversion characteristic when the control unit 101 calculates a motion blur equivalent to 1/60 seconds as the target motion blur. It should be noted that the motion vector is composed of a two-dimensional direction such as a horizontal direction and a vertical direction and a moving amount, but in the present embodiment, the moving amount of the horizontal motion vector is converted into a horizontal motion blur amount for simplification of description. It shows the conversion characteristics to be. A motion blur amount conversion method and a conversion characteristic using a motion vector composed of a two-dimensional direction and a movement amount are also calculated by the same process, and a description thereof will be omitted.

  In the motion blur conversion characteristic of FIG. 14A, a broken line L901 indicates the motion blur conversion characteristic when the target motion blur is equivalent to 1/60 seconds. The motion blur conversion characteristic L901 is the motion blur conversion characteristic calculated by the motion blur conversion characteristic calculation unit 203 in step S302 of FIG. 3, and the motion blur image generated based on this motion blur conversion characteristic is the motion blur conversion characteristic of FIG. It is an image before adjustment. The motion blur conversion characteristic calculation unit 203 changes the motion blur conversion characteristic L901 on the basis of the reference motion vector and the designated motion blur to reduce the motion blur in the area having the size of the reference motion vector. Calculate L1401. Specifically, the reference motion vectors 801 and 802 in FIG. 8 are horizontal left vectors, and 803 and 804 are horizontal horizontal right vectors. Assuming that the length is 5 pixels, a straight line of a characteristic that makes the motion amount within the range of 5 pixels within the 5 pixels in the horizontal direction 0 is generated so that the motion blur amount corresponding to the reference motion vector becomes 0. , A motion blur conversion characteristic L1401.

Next, in step S1205 of FIG. 12, the motion blur correction target area identification unit 205 of FIG. 2 automatically identifies the motion blur correction target area having the motion blur conversion characteristic determined in step S503. This will be described with reference to FIG.
In FIG. 10, reference numeral 1001 denotes a captured image in which a motion vector is displayed in the reference captured frame, which is output to the display unit 109. At this time, it is assumed that the control unit 101 receives an area in which the user wants to adjust the motion blur, and the designated reference area is 1002. An enlarged image of the pixel position of the captured image is 1003. For the motion blur correction target area, the surrounding area within a predetermined range where the feature amount is similar to the designated reference area is calculated, and the area including the similar area is automatically set as the motion blur correction target area. Specify to. An example of the obtained motion blur correction target area is shown in FIG. Since the motion vectors 1011 and 1012 in FIG. 10B are different, they are the motion blur correction target regions as separate regions. On the contrary, the regions of the front legs (801, 802) and the rear legs (803, 804) are included as the same motion blur correction target regions, respectively.

The characteristic amount is luminance or color information close to the image signal value of the designated reference region, and is added to the motion blur correction target region as a similar region when the difference between the characteristic amounts is less than a predetermined threshold value. That is, it is determined by the following formula.
(Feature amount of designated reference area-feature amount of surrounding pixel positions) <similarity determination threshold value...Equation (3)

  The other difference in the feature amount is determined by whether or not the motion vector direction, size, depth using subject distance information, difference in distance from the designated reference region within the screen, etc. is less than a predetermined similarity determination threshold value. .. In addition, it is possible to use a feature amount (image recognition result) used for general subject discrimination. Further, by determining the surrounding area related to the designated reference area within a predetermined range (less than a predetermined similarity determination threshold value) from the designated reference area, it is possible to prevent the influence of the motion blur adjustment from spreading to an unexpected range. It is possible.

Further, since 1001 displays the image after the motion blur adjustment, it is possible to select the region in which the motion blur is desired to be changed by looking at the image after the motion blur addition, and intuitively specify the position of the designated reference region.
In step S504′ of FIG. 12, the motion blur providing unit 204 uses the motion vector information obtained in step S502′ for the motion blur correction target area obtained in step S1205, based on the motion blur conversion characteristics obtained in step S1204. To add motion blur. The motion blur imparting method in step S504′ of FIG. 12 is performed by the same process as in step S504 of FIG. 5, and description thereof will be omitted.

In step S505′ of FIG. 12, the motion blur image obtained in step S504′ is displayed on the display unit 109. This motion blur image is shown in FIG. The image after motion blur adjustment in FIG. 13C is an image in which the motion blur around the limbs of the dog is stationary and the background motion blur itself does not change compared to the image before motion blur adjustment in FIG. 13A. Become. This is a motion blur conversion characteristic in which a part of the limbs of the dog designated by the user is set as the designated reference region, and the amount of motion blur in the motion blur correction target region including the peripheral region of the limbs of the designated reference region becomes zero. This is because the motion blur is controlled. As described above, by setting the motion blur conversion characteristics in the motion blur correction target area including the designated reference area designated by the user, it is possible to adjust the motion blur to the user's preference. In the above example, the motion blur is controlled by the motion blur conversion characteristic such that the motion blur amount of the motion blur correction target region including the peripheral regions of the limbs of the designated reference region is 0. Any control may be used. That is, the amount of change in the motion blur may be varied inside and outside the motion blur correction target area.
The motion blur adjustment in step S304 of FIG. 3 has been described above.
Next, a modified example (second embodiment) of the motion blur adjustment in step S304 of FIG. 3 will be described.

  In the description of the first embodiment, the example in which the motion blur is weakened in the motion blur correction target area has been described. On the contrary, in the second embodiment, an example in which the motion blur is further increased will be described. For example, after the motion blur adjustment is performed in step S304 of FIG. 3 to reduce the motion blur in the motion blur correction target area, the image blur desired by the user cannot be obtained because the motion blur is too reduced, and the motion blur is desired to be increased. There are cases. At this time, the process is performed again in step S304 of FIG.

  In this case, the motion blur conversion characteristic set to add motion blur is, for example, as shown in FIG. As to the motion blur conversion characteristic when the motion blur is applied, in order to impart a motion blur larger than that of L902, the motion blur is imparted by controlling the motion blur conversion characteristic of L1402 in FIG. 14B. In this way, since the motion blur correction target area includes movements in both the left and right directions as the horizontal direction, it is necessary to give a larger motion blur to the motion vector having the plus and minus directions of the motion blur correction target area.

  A captured image displayed on the display unit 109 when the motion blur adjustment is performed will be described with reference to FIG. In FIG. 15, an image with motion blur after the processing of S302 or S305 in FIG. 3 is displayed on the display unit 109, and it is determined whether the motion blur adjustment is performed and how to adjust the motion blur (adjust the motion blur). FIG. 7 is a diagram illustrating a display of a motion blur correction target area and adjustment of strength). FIG. 15A is an image resulting from motion blur output after the process of step S302 or step S305 of FIG. At this time, if the user further attempts to partially intensify the motion blur on the front and rear legs of the dog, touch the positions of the front and rear legs of the dog. The position touched by the user is indicated by the designated reference area (L1501, L1502). The related surrounding area including the designated reference area is automatically specified as the motion blur correction target area as described above. Since there is no motion blur in the motion blur correction target area, the cursor position for instructing blur (blurring) in the blur control bar 1500 showing the adjustment amount of the motion blur amount in FIG. Has become. That is, it indicates that there is no motion blur in the motion blur correction target area.

  Then, when the user wants to add motion blur to the motion blur correction target area, the user moves the cursor to the position 1512 of the blur control bar 1500 in FIG. 15B. In accordance with this movement of the cursor, step S304 of FIG. 3 is executed, the conversion processing of the motion blur conversion characteristic of L1402 of FIG. 14B is executed for the motion blur correction target area, and the motion blur correction target area is set. Add blur. The display after the adjustment of the motion blur is shown in FIG. As described above, by displaying the information regarding the adjustment of the motion blur on the display unit 109, the user adjusts the strength of the motion blur to the desired motion blur for which region, whether or not to adjust the motion blur. It becomes possible to do.

  The examples have been described above. According to the embodiment, it is possible to generate a follow shot image in which the motion blur of the follow subject is reduced even when the background and the follow subject are close in motion.

  In addition, in the embodiment, the embodiment in which the motion blur is controlled by capturing the short-time exposure image with small motion blur and imparting the motion blur has been described. It's not limited. For example, the motion blur may be controlled by capturing a long-second exposure image with large motion blur and reducing the motion blur. As a method for reducing motion blur, a known technique such as a blind deconvolution method is used, and a description thereof will be omitted. (The blind deconvolution method is a method of analyzing the characteristics of motion blur in an image and convolving the inverse characteristic of the motion blur characteristic with the image to reduce motion blur.)

  In the embodiment, the example in which the designation of the motion blur amount is received as the designated motion blur has been described, but the information received as the designated motion blur is not limited to this. For example, the designation of the motion blur direction may be accepted as the designated motion blur. In this case, the motion blur conversion characteristic calculation unit 203 calculates a characteristic for converting the direction of the motion blur corresponding to the reference motion vector into the direction of the designated motion blur. This makes it possible to adjust the direction of motion blur.

In the embodiment, the method of specifying the reference motion vector has been described with the information related to the area where the user designation is accepted, but the method is not limited to this. For example, the image capturing apparatus 100 may specify the motion vector of the focused area in which the focus position is adjusted as the reference motion vector.
Further, in the embodiment, the motion blur is automatically generated based on a plurality of images and the motion blur is applied to one image, but the motion blur may be applied using only one image. That is, it goes without saying that, for example, a motion blur of a predetermined direction or a predetermined amount may be added to the area designated by the user for the one image.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the gist thereof.
For example, in the embodiment, a plurality of images read from the recording unit that is attachable to and detachable from the image pickup apparatus is acquired, and a motion blur image is formed in the image pickup apparatus using the plurality of images. It need not be removable, and may be an external recording unit provided in a cloud or the like. Further, the above-described recording unit may be connected to an external information processing device instead of the image capturing device, and a motion blur image may be formed in the information processing device using an image read from the recording unit.

  In addition, a program (software) that implements a part or all of the control according to the present invention to realize the functions of the above-described embodiments may be supplied to the imaging device or the information processing device via a network or various storage media. Then, the computer (or the CPU, MPU, or the like) in the imaging device or the information processing device may read out and execute the program. In that case, the program and the storage medium storing the program constitute the present invention.

100 image pickup device 101 control unit 102 ROM
103 RAM
104 optical system 105 image pickup unit 106 A/D conversion unit 107 image processing unit 108 recording unit 109 display unit 110 instruction input unit 200 motion blur control unit 201 motion vector calculation unit 202 reference motion vector identification unit 203 motion blur conversion characteristic calculation unit 204 Motion blurring unit 205 Motion blur correction target area specifying unit

Claims (17)

A reference area designating means for designating a reference area,
Motion-blur correction target area specifying means for automatically specifying a motion-blur correction target area having a predetermined relationship with the reference area;
A reference motion blur identifying unit that identifies a reference motion blur from the motion blur correction target area;
A motion blur control that changes the motion blur in the motion blur correction target area so that the reference motion blur becomes a predetermined motion blur, and controls the variation amount of the motion blur to be different inside and outside the motion blur correction target area. An image processing apparatus comprising: a control unit.   The image processing apparatus according to claim 1, wherein the motion blur correction target area specifying unit receives designation of the reference area by a user.   The image processing apparatus according to claim 2, wherein the motion blur correction target area specifying unit receives a designation of coordinate information of the reference area by a user.   The image processing apparatus according to claim 1, further comprising an instruction unit for instructing a change in a motion blur amount or a direction in the motion blur correction target area.   The image processing apparatus according to claim 4, wherein the instruction unit receives the instruction from the user.   2. The motion blur control means changes the motion blur by changing the tap number of the filter when performing the filtering process based on the magnitude and direction of the motion vector and performing the filtering process. 5. The image processing device according to any one of 5.   The area having the predetermined relationship is an area within a predetermined range within the screen from the reference area, and a difference between a magnitude and a direction of a motion vector of the reference area is less than a predetermined threshold value. The image processing apparatus according to claim 1.   The image processing apparatus according to claim 1, wherein the area having the predetermined relationship is an area in which a difference in subject distance from the subject in the reference area is less than a predetermined threshold value.   The image processing apparatus according to claim 1, wherein the area having the predetermined association is an area having a difference from the color or the brightness of the reference area that is less than a predetermined threshold value.   When the plurality of reference regions are designated by the reference region designating unit, the motion blur correction target region identifying unit automatically identifies a region having the predetermined relationship with each of the reference regions. The image processing apparatus according to claim 1.   11. The motion blur control means controls the motion blur so as to reduce the motion blur of a pixel having a motion blur whose difference from at least the reference motion blur is less than a predetermined threshold value. The image processing device according to claim 1.   11. The motion blur control means controls the motion blur so as to increase the motion blur of a pixel having a motion blur whose difference from at least the reference motion blur is less than a predetermined threshold value. The image processing device according to claim 1.   13. The image processing apparatus according to claim 1, wherein the reference motion blur identifying unit identifies the motion blur by calculating a motion vector based on difference information between images. ..   14. The image according to claim 1, further comprising display means for displaying motion blur information before changing the motion blur in the motion blur correction target area by the motion blur control means. Processing equipment. A reference area specifying step for specifying a reference area,
A motion blur correction target area specifying step of automatically specifying a motion blur correction target area having a predetermined relationship with the reference area;
A reference motion blur identifying step of identifying a reference motion blur from the motion blur correction target area;
The motion blur in the motion blur correction target area is changed so that the reference motion blur becomes a predetermined motion blur, and the change amount of the motion blur is controlled to be different inside and outside the motion blur correction target area. An image processing method comprising: a motion blur control step.   A computer program for causing a computer to function as each unit of the image processing apparatus according to claim 1. A computer-readable storage medium storing the computer program according to claim 16.