JP7143191B2 - Image processing device - Google Patents

Description

The present invention relates to an image processing apparatus and image processing method, and more particularly to a technique for controlling motion blurring of a subject that occurs in an image.

There is a technique called panning as a photographing technique for expressing the sense of speed of a moving subject. In this panning technique, the exposure time is set longer than usual, and the photographer follows the movement of the subject with the camera. In images captured in follow-up photography in this way, the moving subject remains stationary, and the background appears flowing due to motion blur. On the other hand, there is a technique for generating such panning images by image processing. This technology analyzes the movement between multiple images of a moving subject and applies motion blur based on that movement to create an image that looks as if it were actually panned. It is a technology to

Japanese Patent Application Laid-Open No. 2002-200300 discloses a technique for imparting motion blur by this image processing. Japanese Patent Application Laid-Open No. 2002-200001 discloses a technique of capturing a plurality of consecutive images and calculating the amount of movement of the background from the images between the plurality of images, thereby imparting motion blur to the background.

JP 2010-15483 A

In such a panning shot, it is desirable that the moving subject is still and the background is blurred. However, during follow-up photography, if the movement of the subject cannot be followed, the position of the subject shifts between the multiple images, and motion blur is applied to the subject according to the amount of movement, resulting in blurring of the moving subject. There is a problem of squeezing.
In particular, when the movement of the background and the subject being followed up is close to each other, the amount of movement cannot be separated well, and image processing cannot effectively remove the blurring of the subject during the follow up shooting.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and it is an object of the present invention to easily generate a panning image in which movement blurring of a subject during follow-up photography is reduced even when the movement of the background and the follow-up subject is close to each other. An object of the present invention is to realize an image processing apparatus capable of

In the image processing device, reference area designating means for designating a reference area;
motion blur correction target area identifying means for automatically identifying a motion blur correction target area having a predetermined relationship with the reference area;
reference motion blur identifying means for identifying a reference motion blur from the motion blur correction target area;
The motion blur is controlled such that the motion blur within the motion blur correction target area is changed so that the reference motion blur becomes a predetermined motion blur, and the change amount of the motion blur is different between inside and outside the motion blur correction target area. and a control means.

According to the present invention, even when the movement of the background and the follow subject is close, it is possible to easily generate a panning image in which motion blurring of the subject during follow shooting is reduced.

1 is a diagram showing a configuration example of an image processing apparatus according to a first embodiment; FIG. 4 is a diagram showing a configuration example of a motion blur control unit 200 according to the first embodiment; FIG. 4 is a flowchart showing a processing flow of a motion blur control unit 200 according to the first embodiment; FIG. 4 is a diagram showing a captured image in a representative scene; FIG. 10 is a flow chart showing a processing flow of automatic motion blur control in step S302; FIG. 4 is a flowchart showing a processing flow of a motion vector calculation unit 201; FIG. 4 is a diagram showing a method of calculating a motion vector; FIG. 4 is a diagram showing motion vectors in a representative scene; FIG. 5 is a diagram showing motion blurring conversion characteristics; FIG. 4 is a diagram showing a method of detecting a motion blur correction target area; FIG. 10 is a diagram showing a motion blur imparting method; FIG. 10 is a flowchart showing a processing flow of motion blur adjustment in step S304; FIG. FIG. 10 is a diagram showing the effect of motion blur adjustment in a representative scene; FIG. 5 is a diagram showing motion blurring conversion characteristics; FIG. 10 is a diagram showing a user display screen when motion blur conversion is performed;

Preferred embodiments of the present invention will be described below with reference to the drawings.

Embodiment 1 of the present invention controls motion blur of an image based on a user's instruction regarding motion blur. Example 1 will be described below.
FIG. 1 is a block diagram showing a configuration example when the motion blur control technique of this embodiment is applied to an imaging device 100 as an example of an image processing device. The imaging device 100 includes, for example, a camera and a smart phone. A block configuration of the first embodiment will be described below with reference to FIG. In the first embodiment, a case where the image processing apparatus is applied to the imaging apparatus 100 will be described as an example of the image processing apparatus. .

The control unit 101 is, for example, a CPU as a computer, and reads an operation control program (computer program) for each block included in the imaging apparatus 100 from a ROM 102 as a storage medium described later, develops it in a RAM 103 described later, and executes it. Thereby, the control unit 101 controls the operation of each block included in the imaging device 100 . The ROM 102 is an electrically erasable/recordable non-volatile memory, and stores an operation control program for each block included in the imaging apparatus 100, as well as parameters required for the operation of each block. A RAM 103 is a rewritable volatile memory, and is used for development of programs executed by the control unit 101 and the like, temporary storage of data generated by the operation of each block included in the imaging apparatus 100, and the like.

The optical system 104 is composed of a lens group including a zoom lens and a focus lens, and forms a subject image on an imaging surface of an imaging unit 105, which will be described later. The imaging unit 105 is, for example, an imaging device such as a CCD or CMOS sensor, photoelectrically converts an optical image formed on the imaging surface of the imaging unit 105 by the optical system 104, and converts the obtained analog image signal into an A/D signal. Output to conversion unit 106 . The A/D converter 106 converts the input analog image signal into digital image data. Digital image data output from the A/D converter 106 is temporarily stored in the RAM 103 . The image processing unit 107 applies various image processing such as white balance adjustment, color interpolation, and gamma processing to the image data stored in the RAM 103 . The image processing unit 107 also includes a motion blurring control unit 200 and generates a motion blurring image by adding motion blurring to the image stored in the recording unit 108 .

Recording unit 108 includes a detachable memory card or the like. A recording unit 108 records the image data processed by the image processing unit 107 as a recorded image via the RAM 103 . The recording unit 108 can also output the recorded image data to the image processing unit 107 via the RAM 103 . A display unit 109 is a display device such as an LCD, and displays images temporarily stored in the RAM 103 and images recorded in the recording unit 108, and displays an operation user interface image for receiving instructions from the user. etc. The instruction input unit 110 is a touch panel, a mouse, a keyboard, or the like, and receives instructions from the user.

Next, the operation of the image processing unit 107, which is a feature of this embodiment, will be described in detail. In this embodiment, an example of controlling motion blur for a captured image with a short exposure time will be described.
First, a configuration example of the motion blur control unit 200 included in the image processing unit 107 will be described with reference to FIG. A motion blur control unit 200 applies motion blur to the image data recorded in the recording unit 108 to generate a motion blurred image.
FIG. 2 is a diagram showing a configuration example of the motion blur control unit 200. As shown in FIG. The motion blur control unit 200 includes a motion vector calculation unit 201, a reference motion vector identification unit 202 as reference motion blur identification means, a motion blur conversion characteristic calculation unit 203, a motion blur imparting unit 204, and a motion blur correction target area identification unit 205. It is configured.
Next, processing of the motion blur control unit 200 will be described with reference to the flowchart of FIG.

In step S301 of FIG. 3, the control unit 101 determines the exposure time when the imaging unit 105 takes an image. Then, the image capturing unit 105 captures (photographs) a plurality of images based on the determined exposure time, and records them in the recording unit 108 . In this embodiment, an example in which 60 images are captured per second as the imaging frame rate will be described. That is, the imaging unit 105 captures one image every 1/60th of a second. The exposure time may be determined by the user via the user interface of the display unit 109, or may be determined by the control unit 101 through automatic exposure control. A method of determining the exposure time by automatic exposure control includes, for example, a method of determining the exposure time based on the photometric value for each predetermined area of the digital image data captured by the imaging unit 105 . Although the details will be described later, the exposure time determined here is set to be shorter than that of normal panning, and an image with little moving object blurring is captured. An example of an image to be captured will be described with reference to FIG. FIG. 4 shows an example in which the photographer swings the imaging device 100 so as to follow a running dog and performs continuous shooting (follow-up shooting) while following the dog. FIG. A captured image, and FIG. 4B shows a captured image of the N+1th frame. Note that N is a positive integer. Also, since the captured images of the Nth frame and the N+1th frame are captured with a short exposure time, no large motion blur occurs in the captured images.

During follow-up photography, it is extremely difficult to perfectly follow and photograph the details of a running dog. In the example of FIG. 4, the movement of the front and back legs of the running dog is different from that of the body of the dog, and therefore changes between the N-th frame and the N+1-th frame.
In step S<b>302 in FIG. 3 , the motion blur control unit 200 imparts motion blur to the captured image of the Nth frame captured in step S<b>301 and displays the motion blurred image on the display unit 109 . Details of the motion blur imparting in step S302 will be described later with reference to FIG.

In step S303 of FIG. 3, the user confirms the motion-blurred image displayed on the display unit 109 in step S302, and the instruction input unit 110 receives an instruction as to whether adjustment of motion blur is necessary. In step S303, if the instruction that motion blur adjustment is necessary is not accepted, the flow of FIG. 3 ends. In step S303, if an instruction that motion blurring adjustment is necessary is accepted, the process proceeds to step S304 in FIG.
In step S<b>304 in FIG. 3 , motion blur is adjusted for the captured image of the Nth frame captured in the processing of step S<b>301 based on the user's instruction regarding motion blur, and the motion blurred image is displayed on the display unit 109 . The motion blur adjustment method in step S304 will be described later with reference to FIG.

In step S305 of FIG. 3, the user confirms the motion-blurred image displayed on the display unit 109 by the process of step S304, and the instruction input unit 110 receives an instruction as to whether or not the motion-blur adjustment has been completed. In step S305, if an instruction that the adjustment of motion blur has been completed is received, the flow of FIG. 3 ends. On the other hand, in step S305, if the instruction that the motion blurring adjustment has been completed is not received, the process returns to step S304 in FIG. 3 to repeat the motion blurring adjustment.
Next, details of the process of automatically imparting motion blur in step S302 of FIG. 3 will be described with reference to the flowchart of FIG.

In step S501 of FIG. 5, the control unit 101 calculates target motion blur information. The target motion blur information is information indicating how many seconds of exposure time the motion blur should be imparted as a target. This target motion blur information changes the length of the motion blur. For example, the length of motion blur equivalent to 1/60 second is twice the length of motion blur equivalent to 1/120 second.

In step S502 in FIG. 5, the motion vector calculation unit 201 in FIG. 2 calculates a motion vector between captured images recorded in the recording unit . Then, the calculated motion vector is output to the reference motion vector specifying unit 202 and the motion blur imparting unit 204 in FIG. Here, a motion vector calculation method will be described with reference to FIGS. 6, 7 and 8. FIG.
FIG. 6 is a flowchart showing the motion vector calculation processing in step S502 by the motion vector calculation unit 201, and FIG. 7 is a diagram showing a motion vector calculation method by the block matching method. In this embodiment, a block matching method will be described as an example of a motion vector calculation method, but the motion vector calculation method is not limited to this. For example, an optical flow method may be used.

In step S601 in FIG. 6, the motion vector calculation unit 201 in FIG. 2 inputs two captured images recorded in the recording unit . Then, the captured image of the Nth frame is set as the reference captured image, and the captured image of the N+1th frame later in time is set as the reference captured image.
In step S602 of FIG. 6, a reference block 702 of N×N pixels is arranged in a reference captured image 701 as shown in FIG.
In step S603 of FIG. 6, in the reference captured image 703, (N+n)×(N+n) pixels around the same coordinates 704 as the reference block 702 of the reference captured image 701 are set as the search range 705 as shown in FIG.

In step S604 of FIG. 6, a correlation calculation is performed between a reference block of N×n pixels at different coordinates existing in the search range 705 of the reference captured image 703 and the reference block 702 of the reference captured image 701 to calculate a correlation value. A correlation value is calculated based on the sum of the inter-image absolute difference values for the pixels of the standard block 702 and the reference block. That is, the coordinate with the smallest sum of absolute difference values is the coordinate with the highest correlation value. Note that the method of calculating the correlation value is not limited to the sum of absolute differences. For example, the correlation value may be calculated based on the sum of squared differences or normal cross-correlation value. In the example of FIG. 7, reference block 706 shows the highest correlation.
At step S605 in FIG. 6, a motion vector is calculated based on the reference block coordinates showing the highest correlation value. In the example of FIG. 7, the motion vector is the difference between the coordinates 704 of the reference block 702 of the reference captured image 701 and the center coordinates of the reference block 706 .

In step S606 of FIG. 6, while moving the reference block 702 of FIG. 7, the processing of steps S602, S603, S604, and S605 is repeated to calculate motion vectors of all pixels of the reference captured image 701. FIG. Instead of calculating motion vectors for all pixels, motion vectors may be calculated for each predetermined pixel.
FIG. 8 shows an example of motion vectors between captured images calculated based on the above method. 8A and 8B are diagrams showing motion vectors based on difference information when the captured image of the Nth frame in FIG. 4A is the reference captured image and the captured image of the Nth frame in FIG. 4B is the reference captured image. . The arrow in FIG. 8 indicates the motion vector, the length of the arrow indicates the length of the motion vector, and the direction of the arrow indicates the direction of the motion vector. In the example of FIG. 8, motion vectors for each pixel are not drawn, and only representative motion vectors are shown in a simplified manner.

In the example of FIG. 8, the user swings the imaging device 100 from right to left to capture a follow-up shot of a running dog in the center. Therefore, a stationary object in the background is detected as a motion vector in the right direction. 0 is calculated as a motion vector. However, motion vectors other than 0 appear for the swinging of the dog's limbs because the speed and motion are different from those in the follow-up shooting direction. The motion vectors of the front legs are represented by 801 and 802, and the motion vectors of the rear legs are 803 and 804, respectively. Front legs (801, 802) and rear legs (803, 804) have motion vectors in different directions. Furthermore, it shows that the motion vector of the hind leg has the same amount of motion as the vector of the background. A motion vector has a direction. Therefore, in the present embodiment, for example, leftward motion vectors in the horizontal direction and downward motion vectors in the vertical direction are assumed to be negative, while rightward and upward motion vectors are positive.

In step S503 in FIG. 5, the motion blurring conversion characteristic calculation unit 203 in FIG. 2 calculates a characteristic for converting a motion vector into motion blurring addition information based on the target motion blurring information. The motion blur imparting information consists of the amount of motion blur in the horizontal direction and the amount of motion blur in the vertical direction. and the motion blur amount (corresponding to the length of the motion vector). A method for calculating the motion blurring conversion characteristic will be described with reference to FIG. FIG. 9 is a diagram showing motion blur conversion characteristics. The horizontal axis represents the amount of movement of the motion vector in the horizontal direction, and the vertical axis represents the amount of motion blur in the horizontal direction. For the sake of simplification, FIG. 9 shows conversion characteristics for converting the horizontal component of the motion vector into the amount of motion blur in the horizontal direction. Originally, a motion vector consists of a horizontal movement amount and a vertical movement amount, but since the conversion method and conversion characteristics for the vertical direction are calculated in the same manner as for the horizontal direction, description thereof will be omitted.
A motion blurring conversion characteristic L901 in FIG. 9 is represented by the following equation (1), and is a motion blurring conversion characteristic when the target motion blurring information is equivalent to 1/60 second.
Horizontal motion blur amount = horizontal movement amount (1)

Since the frame rate of captured images is 60 frames per second, the amount of movement in the horizontal direction is the amount of movement in 1/60 second. The motion blurring conversion characteristic L901 calculates the motion blurring amount corresponding to the moving amount in 1/60 second.
Also, the motion blur conversion characteristic L902 in FIG. 9 is represented by Equation (2), and is the motion blur conversion characteristic when the target motion blur information is equivalent to 1/120.
Horizontal motion blur amount = 1/2 x horizontal movement amount (2)
The motion blurring conversion characteristic L902 calculates the motion blurring amount corresponding to the moving amount in 1/120 second, which is half the moving amount in 1/60 second.

That is, when conversion is performed based on the motion blurring conversion characteristic L902, a motion blurring amount equivalent to half the motion blurring amount converted based on the motion blurring conversion characteristic 901 is added. By changing the motion blurring conversion characteristics in this way, it is possible to control the motion blurring amount. In step S<b>504 in FIG. 5 , the motion blur applying unit 204 refers to the motion blur conversion characteristics calculated by the motion blur conversion characteristics calculation unit 203 and the motion vectors for each pixel calculated by the motion vector calculation unit 201 . Then, motion blur is applied to the captured image based on both vectors, and the motion blurred image is output to the display unit 109 . The motion blur imparting method can be realized by performing filter processing using a motion vector, for example, described in Japanese Patent Laid-Open No. 2010-15483. The filter processing for imparting motion blur can be performed in units of pixels or in units of pixel blocks made up of a plurality of pixels. In addition, motion blur can be changed by changing the number of filter taps (filter size) when filtering is performed based on the magnitude and direction of the motion vector.

FIG. 11 is a diagram showing a method of imparting motion blur. Motion blurring addition information L1101 in FIG. 11 indicates motion blurring addition information for the pixel A of interest. Further, this motion blurring addition information L1101 is motion blurring addition information that reaches pixel D from pixel A through pixels B and C, and a motion vector represents 4 pixels in the horizontal direction and 4 pixels in the vertical direction. there is In step S505 of FIG. 5, the motion blurring image generated by the motion blur imparting unit 204 in step S504 is displayed on the display unit 109. FIG.
The automatic motion blur control processing in step S302 of FIG. 3 has been described above.

Next, the motion blur adjustment method in step S304 of FIG. 3 will be described in detail with reference to the flowchart of FIG. Motion blur adjustment is performed to meet a user's request to adjust the amount of motion blur for each subject.
In step S1201 of FIG. 12, the control unit 101 receives designation (instruction) of a designated reference area as a motion blur adjustment instruction from the user via the instruction input unit 110. FIG. At this time, the instruction input unit 110 functions as a reference area specifying means. Further, in step S1202 in FIG. 12, the control unit 101 receives a specified motion blurring instruction in the specified reference area as a motion blurring adjustment instruction from the user via the instruction inputting unit 110. FIG. A method of accepting a motion blur adjustment instruction will be described with reference to FIG.

FIG. 13A shows an image before motion blur adjustment, FIG. 13B shows an image after motion blur adjustment in which the motion blur amount is reduced to a value equal to or less than the blur amount of the specified reference area, and FIG. FIG. 10 shows an image after motion blur adjustment in which the amount of motion blur only around the reference area (hands and feet) is reduced. As shown in FIG. 13B, when the motion blur amount less than or equal to the blur amount of the designated reference area is reduced, the motion blur amount of the background area having the motion blur amount close to the blur amount of the designated reference area is also reduced at the same time. end up Therefore, in this embodiment, as shown in FIG. 13C, the amount of motion blurring is controlled only in peripheral (hands and feet) areas that include the specified reference area and are related to the subject in the specified reference area. As a result, it is possible to adjust the motion blur by focusing only on the area desired by the user.

FIG. 13A shows an image before motion blurring adjustment, which is a motion blurring image displayed on the display unit 109 after the automatic motion blurring control in step S302 of FIG. The user confirms the motion-blurred image displayed on the display unit 109, and determines whether adjustment of the motion-blurred image is necessary (determination in step S303 in FIG. 3). Here, in the example of the image before motion blur adjustment in FIG. 13A, motion blur occurs not only in the background but also in the hands and feet of the main subject, the running dog. FIG. 13A is a diagram in which motion blur is added based on the motion vectors in FIG. In general, in panning images, the main subject is still without motion blurring, and the background is motion-blurred. It is said that the sense of speed stands out and is preferable. Therefore, in this embodiment, the user confirms a motion-blurred image such as the image before motion-blurring adjustment in FIG. .

Specifically, as indicated by L1301 and L1302 in FIG. 13A, the user touches or clicks an area to be adjusted for motion blur. At that time, it is specified whether to adjust the movement blur of the designated reference area to be small or large, based on the way of touch operation, click operation, or the like. For example, when the user designates with a single touch or single click, the control unit 101 receives an instruction to reduce the motion blur as the designated motion blur of the designated reference area. Also, when the user double-touches or double-clicks to specify, the control unit 101 receives an instruction to increase the motion blur as the specified motion blur of the specified reference area. It should be noted that the method of specifying the specification reference area need not be the touch or click as described above, and may be specified by, for example, inputting coordinate information on the screen using a numeric keypad or the like.

For example, in the image before motion blurring adjustment in FIG. 13A, when the motion blurring of the dog is to be adjusted to be small, the user designates a part of the limbs (L1301, L1302) of the running dog with a single click. Then, the control unit 101 accepts the specified coordinates as a specified reference area, automatically detects the specified reference area and its surrounding areas, and accepts an instruction to reduce the motion blur as a specified motion blur. In the following description, an example will be described in which the user designates a running dog by single-clicking and adjusts the motion blur of the dog to be small.
Note that the designation reference area accepted by the control unit 101 may not be limited to the coordinates of only one location designated by the user by clicking. For example, when the user designates that the whole or a part of the running dog is surrounded by a circle, the control unit 101 may accept the surrounded area as the designation reference area.

In step S502′ of FIG. 12, the motion vector calculation unit 201 calculates a motion vector between the captured images recorded in the recording unit 108. FIG. Specifically, the motion vector calculation unit 201 inputs the captured image of the Nth frame in FIG. 4A and the captured image of the N+1th frame in FIG. Calculate the motion vector. It should be noted that the motion vector calculation method in step S502′ of FIG. 12 is calculated by the same processing as in step S502 of FIG. 5, and a description thereof will be omitted. Also, the motion vector calculated during the automatic motion blur control processing in step S302 of FIG. 3 may be recorded in the recording unit 108 and used in step S502' of FIG.

In step S1203 of FIG. 12, the reference motion vector identifying unit 202 identifies a reference motion vector based on the specified reference region received by the control unit 101 in step S1201 and the motion vector calculated by the motion vector calculation unit 201. FIG. Specifically, in FIG. 8, when four regions of dog motion vectors 801 to 804 are selected as designated reference regions, the motion vectors of the four selected regions are specified as reference motion vectors (reference motion blur). . If a plurality of motion vectors exist within the designated reference area, a histogram of motion vectors within the designated reference area may be obtained, and the motion vector with the highest frequency may be used as the reference motion vector.
At this time, if the motion vectors have different directions and magnitudes, four reference motion vectors are used.

Next, in step S1204 in FIG. 12, the motion blurring conversion characteristic calculation unit 203 in FIG. 2 calculates motion blurring conversion characteristics based on the target motion blurring, the reference motion vector, and the designated motion blurring. A method of calculating motion blurring conversion characteristics will be described with reference to FIG. The motion blurring conversion characteristics of FIG. 14 show an example of the motion blurring conversion characteristics when the control unit 101 calculates the motion blur corresponding to 1/60 second as the target motion blurring. A motion vector consists of two-dimensional directions, ie, horizontal and vertical directions, and a movement amount. In this embodiment, the movement amount of the motion vector in the horizontal direction is converted into the amount of motion blur in the horizontal direction for the sake of simplification of the explanation. It shows the conversion characteristics that A motion blur amount conversion method and conversion characteristics using a motion vector consisting of a two-dimensional direction and a movement amount are calculated by the same processing, and the description is omitted.

In FIG. 14(A) motion blur conversion characteristics, dashed line L901 indicates the motion blur conversion characteristics when the target motion blur is equivalent to 1/60 seconds. A motion blurring conversion characteristic L901 is a motion blurring conversion characteristic calculated by the motion blurring conversion characteristic calculation unit 203 in step S302 in FIG. This is an image before adjustment. The motion blurring conversion characteristics calculation unit 203 changes the motion blurring conversion characteristics L901 based on the reference motion vector and the designated motion blurring, thereby calculating the motion blurring conversion characteristics so as to reduce the motion blur in the area having the size of the reference motion vector. Calculate L1401. Specifically, reference motion vectors 801 and 802 in FIG. 8 represent horizontal left vectors, and reference motion vectors 803 and 804 represent horizontal right vectors. Assuming that the length is 5 pixels, a straight line having the motion blurring conversion characteristic L901 that sets the motion blurring amount within a range of 5 pixels in the horizontal direction to 0 is generated so that the motion blurring amount corresponding to the reference motion vector becomes 0. , is calculated as the motion blur conversion characteristic L1401.

Next, in step S1205 in FIG. 12, the motion blur correction target area specifying unit 205 in FIG. 2 automatically specifies a motion blur correction target area for applying the motion blur conversion characteristics determined in step S503. Description will be made with reference to FIG.
In FIG. 10 , reference numeral 1001 denotes a captured image in which a motion vector is displayed in a reference captured frame, which is output to the display unit 109 . At this time, it is assumed that the control unit 101 receives an area in which the user wishes to adjust the motion blur, and that the designation reference area is 1002 . 1003 is an enlarged pixel position of the captured image. The motion blur correction target area is calculated by calculating the surrounding area within a predetermined range where the feature value is similar to the specified reference area, and automatically selecting the area including the similar area as the motion blur correction target area. to be specified. FIG. 10B shows an example of the determined motion blur correction target area. Since 1011 and 1012 in FIG. 10B have different motion vectors, they are separate motion blur correction target areas. Conversely, the regions of the front legs (801, 802) and the rear legs (803, 804) are included as the same motion blur correction target regions.

Luminance and color information close to the image signal value of the specified reference area is used as the feature amount, and if the difference in the feature amount is less than a predetermined threshold, the area is added to the motion blur correction target area as a similar area. That is, it is determined by the following formula.
(feature amount of designated reference area−feature amount of surrounding pixel positions)<similarity determination threshold Expression (3)

Other feature amount differences are determined based on whether the motion vector direction, size, depth using subject distance information, difference in distance within the screen from the designated reference area, etc. are less than a predetermined similarity determination threshold. . In addition, it is possible to use a feature amount (image recognition result) that is generally used for subject discrimination. In addition, by determining the surrounding area related to the designated reference area within a predetermined range (less than a predetermined similarity determination threshold) from the designated reference area, it is possible to prevent the effects of motion blur adjustment from spreading to an unexpected range. It is possible.

Further, 1001 displays the image after motion blur adjustment, so that the user can select an area whose motion blur is to be changed by looking at the image after applying motion blur, and can intuitively specify the position of the specified reference area.
In step S504′ of FIG. 12, the motion blur imparting unit 204 uses the motion vector information obtained in S502′ based on the motion blur conversion characteristics obtained in S1204 for the motion blur correction target area obtained in S1205. to add motion blur. The motion blurring method in step S504′ of FIG. 12 is performed by the same process as in step S504 of FIG. 5, and the description thereof is omitted.

In step S505' of FIG. 12, the motion-blurred image obtained in step S504' is displayed on the display unit 109. FIG. This motion-blurred image is shown in FIG. Compared to the image before motion blur adjustment in FIG. 13(A), motion blur around the limbs of the dog is still in the image after motion blur adjustment in FIG. Become. This motion blurring conversion characteristic is such that the amount of motion blur in the motion blur correction target region including the peripheral region of the limbs of the specified reference region is zero. This is because motion blur is controlled. In this way, by setting the motion blur conversion characteristics for the motion blur correction target region including the reference region designated by the user, it is possible to adjust the motion blur to the user's preference. In the above example, motion blur is controlled with the motion blur conversion characteristic so that the motion blur amount of the motion blur correction target area including the peripheral areas of the limbs of the specified reference area becomes 0. control. That is, the change amount of the motion blur should be made different between the inside and outside of the motion blur correction target area.
The motion blur adjustment in step S304 of FIG. 3 has been described above.
Next, a modified example (second embodiment) of motion blur adjustment in step S304 of FIG. 3 will be described.

In the description of the first embodiment, an example of weakening motion blur in the motion blur correction target area has been described. Conversely, in a second embodiment, an example in which motion blur is increased will be described. For example, after performing the motion blur adjustment in step S304 in FIG. 3 to reduce the motion blur in the motion blur correction target area, the motion blur is reduced too much and the image effect desired by the user cannot be obtained. Sometimes. At this time, the processing is performed again in step S304 of FIG.

In this case, it is assumed that the motion blurring conversion characteristics set to apply motion blurring are as shown in FIG. 14B, for example. Compared to L902, the motion blurring conversion characteristics when motion blurring is applied are controlled to achieve the motion blurring conversion characteristics of L1402 in FIG. 14B in order to apply larger motion blurring. In this way, since the motion blur correction target area includes motion in both the left and right directions as a horizontal direction, it is necessary to impart greater motion blur to the motion vectors having the plus and minus directions of the motion blur correction target area.

A captured image displayed on the display unit 109 when motion blur adjustment is performed will be described with reference to FIG. 15 . FIG. 15 shows on the display unit 109 the motion-blurred image after processing in S302 or S305 of FIG. FIG. 10 is a diagram for explaining the display of the motion blur correction target area and intensity adjustment). FIG. 15A is an image resulting from adding motion blur, which is output after processing in step S302 or step S305 in FIG. At this time, if the user wants to further strengthen the motion blurring of the front and rear legs of the dog, the user touches the position of the front and rear legs of the dog. The position touched by the user is indicated by designated reference areas (L1501, L1502). A related surrounding area including this specified reference area is automatically identified as a motion blur correction target area as described above. Since there is no motion blur in this motion blur correction target area, the cursor position for instructing blur in the blur control bar 1500 indicating the adjustment amount of the motion blur amount in FIG. It has become. In other words, it indicates that there is no motion blur in the motion blur correction target area.

Subsequently, when the user wishes to add motion blur to the motion blur correction target area, the user moves the cursor to the position 1512 of the blur control bar 1500 in FIG. 15B. Step S304 of FIG. 3 is executed in accordance with the movement of the cursor, and conversion processing of the motion blur conversion characteristics of L1402 of FIG. Give blur. The display after motion blur adjustment is shown in FIG. 15(B). As described above, the display unit 109 displays the information related to motion blur adjustment, so that the user can decide whether or not to perform motion blur adjustment, and adjust the strength of motion blur to a preferred motion blur for which region. It becomes possible to

The embodiments have been described above. According to the embodiment, even when the movement of the background and the follow subject is close to each other, it is possible to generate a panning image in which motion blurring of the follow subject is reduced.

In the embodiment, an embodiment has been described in which motion blur is controlled by picking up a short-time exposure image with little motion blur and imparting motion blur. It is not limited. For example, motion blur may be controlled by capturing a long-second exposure image with large motion blur and reducing motion blur. A known technique such as a blind deconvolution method is used as a method for reducing motion blur, and the description thereof is omitted. (The blind deconvolution method is a method for reducing motion blur by analyzing the characteristics of motion blur in an image and convolving the inverse characteristics of the motion blur characteristics on the image.)

In the embodiment, an example of accepting the specification of the amount of motion blur as the specified motion blur has been described, but the information accepted as the specified motion blur is not limited to this. For example, designation of the motion blur direction may be accepted as the designated motion blur. In this case, the motion blurring conversion characteristic calculation unit 203 calculates a characteristic for converting the direction of motion blurring corresponding to the reference motion vector to the direction of the designated motion blurring. This makes it possible to adjust the direction of motion blur.

In the embodiment, the reference motion vector identification method is described as information related to the area for which the user designation is accepted, but the method is not limited to this. For example, the motion vector of the in-focus area in which the imaging device 100 is focused may be specified as the reference motion vector.
Further, in the embodiment, motion blurring is automatically generated based on a plurality of images and motion blurring is applied to one image, but blurring may be applied using only one image. That is, it goes without saying that motion blur of a predetermined direction or a predetermined amount may be imparted to a region designated by the user, for example, with respect to the one image.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist.
For example, in the embodiment, a plurality of images read from a recording unit detachable from the imaging device are obtained, and a motion-blurred image is formed in the imaging device using the plurality of images. It may not be detachable, and may be an external recording unit provided in a cloud or the like. Alternatively, the recording unit as described above may be connected to an external information processing device instead of the imaging device, and an image read from the recording unit may be used to form a motion-blurred image within the information processing device.

Also, a program (software) that implements the functions of the above-described embodiments for part or all of the control in the present invention may be supplied to the imaging device or the information processing device via a network or various storage media. Then, the computer (or CPU, MPU, etc.) in the imaging device or information processing device may read and execute the program. In that case, the program and the storage medium storing the program constitute the present invention.

100 imaging device 101 control unit 102 ROM
103 RAM
104 optical system 105 imaging unit 106 A/D conversion unit 107 image processing unit 108 recording unit 109 display unit 110 instruction input unit 200 motion blur control unit 201 motion vector calculation unit 202 reference motion vector identification unit 203 motion blur conversion characteristic calculation unit 204 Motion blur imparting unit 205 Motion blur correction target area specifying unit

Claims (17)

reference area designating means for designating a reference area;
motion blur correction target area identifying means for automatically identifying a motion blur correction target area having a predetermined relationship with the reference area;
reference motion blur identifying means for identifying a reference motion blur from the motion blur correction target area;
The motion blur is controlled such that the motion blur within the motion blur correction target area is changed so that the reference motion blur becomes a predetermined motion blur, and the change amount of the motion blur is different between inside and outside the motion blur correction target area. and a control means. 2. The image processing apparatus according to claim 1, wherein said motion blur correction target area specifying means accepts designation of said reference area by a user. 3. The image processing apparatus according to claim 2, wherein said motion blur correction target area specifying means receives designation of coordinate information of said reference area by a user. 2. The image processing apparatus according to claim 1, further comprising instruction means for instructing a change in motion blur amount or direction within said motion blur correction target area. 5. The image processing apparatus according to claim 4, wherein said instruction means receives said instruction from a user. 2. The motion blur control means changes the motion blur by performing filtering by changing the number of taps of a filter when performing filtering based on the magnitude and direction of the motion vector. 6. The image processing device according to any one of 5. The area having the predetermined relationship is an area within a predetermined range within the screen from the reference area, and a difference between the magnitude or direction of the motion vector of the reference area and the reference area is less than a predetermined threshold. 2. The image processing apparatus according to claim 1. 2. The image processing apparatus according to claim 1, wherein the area having the predetermined relationship is an area in which a difference in subject distance from the subject in the reference area is less than a predetermined threshold. 2. The image processing apparatus according to claim 1, wherein the area having the predetermined relationship is an area having a difference in color or luminance from the reference area less than a predetermined threshold. When a plurality of reference areas are specified by the reference area specifying means, the motion blur correction target area specifying means automatically specifies an area having the predetermined relationship with each of the reference areas. 2. The image processing apparatus according to claim 1. 11. The motion blur control means controls the motion blur so as to reduce the motion blur of at least a pixel having a motion blur whose difference from the reference motion blur is less than a predetermined threshold. The image processing device according to any one of . 11. The motion blur control means controls the motion blur so as to increase the motion blur of at least a pixel having a motion blur whose difference from the reference motion blur is less than a predetermined threshold. The image processing device according to any one of . 13. The image processing apparatus according to any one of claims 1 to 12, wherein the reference motion blur identifying means identifies the motion blur by calculating a motion vector based on difference information between images. . 14. The image according to any one of claims 1 to 13, further comprising display means for displaying motion blur information before the motion blur in said motion blur correction target area is changed by said motion blur control means. processing equipment. a reference area specifying step for specifying a reference area;
a motion blur correction target area identifying step of automatically identifying a motion blur correction target area having a predetermined relationship with the reference area;
a reference motion blur identifying step of identifying a reference motion blur from the motion blur correction target area;
The motion blur within the motion blur correction target area is changed so that the reference motion blur becomes a predetermined motion blur, and control is performed so that the change amount of the motion blur is different between inside and outside the motion blur correction target area. and a motion blur control step. A computer program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 14. 17. A computer readable storage medium storing the computer program according to claim 16.

Images