JP2019110386A - Image processing system, its control method and program - Google Patents

Abstract

To make it difficult for a moving body to be disappeared in an image process for imparting a motion change thereto.SOLUTION: To produce a motion-changed image by using respective captured images of a plurality of frames continuous in a time series, a moving-body information acquisition unit 302 calculates a motion vector of a moving body, as movement information relating to movement of the moving body in the captured images. A subject information acquisition unit 303 acquires overlap information between frames of the moving body and information about difference between the moving body and background, as subject information of a subject of the moving body in the captured images. On the basis of the movement information acquired by the moving body information acquisition unit 302 and the subject information acquired by the subject information acquisition unit 303, a motion change intensity determination unit 304 evaluates a degree of disappearance of the moving body when a motion change is imparted thereto, and determines the number of compositions, as a motion change intensity, according to the evaluation result.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image processing apparatus, a control method thereof and a program.

  As a technique for performing motion expression, there is one that generates a pseudo long second moving image which is expressed as if one frame is pseudo shot in a long second. This is a technology for applying motion blur to a plurality of frames of a moving image, for example, to add motion blur, and to generate a moving image with an effect such as giving a blur to a moving object.

  In Patent Document 1, a specific image data and a combination image data to be combined with the specific image data are selected from among a plurality of image data captured by the imaging unit, and based on the movement amount of the subject, A configuration is disclosed that determines the number of combined image data for combination to be combined with the specific image data. This makes it possible to create, for example, an image representing the movement trajectory of a person shooting toward a basketball goal, and an image representing the trajectory of a ballerina jump.

Patent No. 5948960 gazette Patent No. 5589527 gazette

The prior art disclosed in Patent Document 1 represents a trajectory of a series of movements from the start of movement to the end of movement of the moving object, and when the amount of movement is large, the number of combined sheets is increased to perform synthesis.
However, if the number of combined images increases, the moving object may disappear in the image after combining and it may be difficult to see. For example, if the movement of the moving object is fast, the overlapping of the moving object between the images decreases, and the specific gravity of the non-moving background increases each time the image is synthesized. The image is thin and the moving object is easily disappeared.

  The present invention has been made in view of the above-described points, and it is an object of the present invention to make it difficult for a moving object to disappear in image processing for applying motion blur.

  The image processing apparatus according to the present invention includes an acquisition unit that acquires information on moving objects in a plurality of images continuous in time series, and a motion blur that is evaluated based on the information acquired by the acquisition unit. Motion blur is applied using the image based on the motion blur intensity determining means for determining the motion blur intensity according to the ease of disappearance of the moving object, and the motion blur intensity determined by the motion blur intensity determining means And motion blur applying means for generating a motion blur imparted image.

  According to the present invention, it is possible to make it difficult for a moving object to disappear in image processing for applying motion blur.

It is a figure showing composition of an imaging device concerning a 1st embodiment. It is a figure for demonstrating a pseudo | simulation long second moving image. It is a figure showing functional composition of an imaging device concerning a 1st embodiment. It is a flowchart which shows the process which an imaging device performs in 1st Embodiment. It is a figure for demonstrating calculation of a motion vector. It is a figure for demonstrating calculation of the overlap area | region of a moving body. It is a characteristic view which shows the example of the relationship between the evaluation value of the ease of elimination, and each information. It is a characteristic view which shows the example of the relationship between the synthetic number and the evaluation value of the easiness of disappearance. It is a flowchart which shows the process which an imaging device performs in 2nd Embodiment. It is a characteristic view showing an example of a relation between priority and each information. It is a flowchart which shows the process which an imaging device performs in 3rd Embodiment. It is a figure for demonstrating determination of the number of taps of filter processing, and a filter factor. It is a flowchart which shows the process which an imaging device performs in 4th Embodiment. It is a figure for demonstrating determination of the number of compositings of a transition period image. It is a figure for demonstrating the determination of the length of a transition period, and the synthetic | combination number of objects of a transition period. It is a flowchart which shows the process which an imaging device performs in 5th Embodiment. It is a figure for demonstrating determination of the filter strength of a transition period image. It is a figure for demonstrating determination of the filter strength of a transition period image. It is a figure which shows the function structure of the image processing part of the imaging device which concerns on 6th Embodiment. It is a flowchart which shows the process which an imaging device performs in 6th Embodiment. It is a figure for explaining composition of a picture. It is a figure for explaining composition of a picture. It is a figure for explaining composition of a picture. It is a figure which shows the example of the number of compositings for every area | region.

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. In the embodiments described below, an example in which an imaging apparatus functions as an image processing apparatus to which the present invention is applied will be described.
First Embodiment
FIG. 1 is a diagram showing the configuration of an imaging device 100 according to the embodiment.
The system control unit 101 includes, for example, a CPU, reads an operation program from the ROM 102, and develops and executes the operation program in the RAM 103 to control the operation of each unit included in the imaging apparatus 100. The ROM 102 is a rewritable non-volatile memory, and stores, in addition to the operation program, parameters and the like necessary for the operation of each unit. The RAM 103 is a rewritable volatile memory, and is used as a temporary storage area of data output in the operation of each unit included in the imaging device 100.

  The optical system 104 is configured of a lens group including a zoom lens and a focus lens. A subject image obtained via the optical system 104 forms an image on the imaging unit 105. The imaging unit 105 is formed of an imaging element such as a CCD or CMOS sensor, for example, photoelectrically converts an optical image formed by the optical system 104, and sequentially obtains a pair of analog image signals obtained to the A / D conversion unit 106. Output. The A / D conversion unit 106 converts the input analog image signal into a digital image signal, and outputs the obtained digital image data to the RAM 103.

  An image processing unit 107 executes various image processing such as white balance adjustment, color interpolation, gamma processing, electronic zoom processing, and the like on the image data recorded in the RAM 103 or the like. Also, the image processing unit 107 executes image processing for applying motion blur. The display unit 108 is a display device such as an LCD, and displays an operation user interface for receiving an image recorded in the RAM 103 and the recording unit 109 or an instruction from a user. The recording unit 109 is formed of a recording medium such as a memory card or a hard disk, and stores final output image data subjected to image processing.

In the first embodiment, when generating a pseudo long second moving image, the ease of disappearance of the moving object when the motion blur is applied is evaluated, and the motion blur intensity is determined according to the evaluation result. In the present embodiment, the motion blur intensity is determined by determining the number of combined images.
The pseudo long second moving image will be described with reference to FIG. FIG. 2 shows images 201 to 204 of 1 to 4 frames of the captured moving image, and images 211 and 212 after combination.
In the pseudo long second moving image, moving average combination is performed on a plurality of frames of the moving image in order to express motion blur of a moving object (a bus or a pedestrian in the illustrated example). The image 211 is an image obtained by combining the images 201 to 203 of 1 to 3 frames, and is a motion blur imparted image to which motion blur is added. The image 212 is an image obtained by combining the images 202 to 204 of 2 to 4 frames, and is a motion blur imparted image to which motion blur is added.

As described above, by combining the images while sequentially shifting the frames, a motion blur is applied, and a moving image with an effect of giving a blur to the moving object is generated. Motion blur intensity changes according to the number of combined images. If the number of combined images is small, the moving object is hard to disappear, but the amount of movement is small, so the motion blur intensity becomes weak. On the other hand, if the number of combined sheets is large, the moving object is likely to disappear, but the moving amount becomes large, so the motion blur intensity becomes strong.
In addition, when the moving object moves fast, the overlapping of moving objects between frames decreases, and the specific gravity of the non-moving background increases with each composition, so the background in the image after composition becomes dark, and the moving object Becomes thin, and the moving object is likely to disappear.

FIG. 3 is a diagram showing a functional configuration of the imaging device 100 according to the first embodiment.
Reference numeral 301 denotes a captured image acquisition unit, which acquires captured images of a plurality of frames continuous in time series, which constitute a moving image.
Reference numeral 302 denotes a moving body information acquisition unit, which acquires movement information on the movement of the moving body in the captured image acquired by the captured image acquisition unit 301. As movement information, the motion vector of the moving body is calculated and used for evaluation of the disappearance of the moving body. In the present embodiment, the moving body information acquisition unit 302 functions as a moving body information acquisition unit in the present invention.
A subject information acquisition unit 303 acquires subject information as a subject of a moving object in the captured image acquired by the captured image acquisition unit 301. As the subject information, overlapping information between the frames of the moving object and difference information between the moving object and the background are acquired, and are used for evaluation of the disappearance of the moving object. In the present embodiment, the subject information acquisition unit 303 functions as a subject information acquisition unit in the present invention.

Reference numeral 304 denotes a motion blur strength determination unit, which indicates the ease of disappearance of the moving object when the motion blur is applied based on the movement information acquired by the moving object information acquisition unit 302 and the subject information acquired by the object information acquisition unit 303. evaluate. Then, the motion blur strength determination unit 304 determines the combined number as the motion blur strength according to the evaluation result. In the present embodiment, the motion blur strength determination unit 304 functions as motion blur strength determination means in the present invention.
A motion blur imparting unit 305 generates a motion blur imparted image using the captured image acquired by the captured image acquisition unit 301 based on the combined number determined by the motion blur strength determination unit 304. The motion blur imparted image is an image of a frame for a pseudo long second moving image, and a pseudo long sec moving image is generated by generating a motion blur imparted image of a plurality of consecutive frames in time series. In the present embodiment, the motion blur applying unit 305 functions as a motion blur applying unit in the present invention.
An image output unit 306 outputs the pseudo long second moving image generated by the motion blur adding unit 305.

FIG. 4 is a flowchart showing processing performed by the imaging device 100 in the first embodiment. The flowchart in FIG. 4 is realized by the system control unit 101 reading out an operation program from the ROM 102, expanding it in the RAM 103, and executing it.
In the first embodiment, an example in which there is only one moving object in the captured image acquired by the captured image acquisition unit 301 will be described.

  In step S401, the captured image acquisition unit 301 acquires captured images of a plurality of consecutive frames in time series.

In step S402, the moving body information acquisition unit 302 calculates the motion vector of the moving body in the captured image acquired in step S401.
FIG. 5 is a diagram for explaining calculation of a motion vector. FIGS. 5A and 5B show a captured image 501 of the Nth frame acquired by the captured image acquisition unit 301 and a captured image 502 of the N + 1th frame. Further, FIG. 5C shows a state where the captured image 501 of the Nth frame is divided into blocks.
The captured images 501 and 502 are divided into blocks of an arbitrary size (L × M), and correlation values are calculated as differences between the regions. In order to calculate the correlation value between frames, SAD (Sum of Absolute Difference) is calculated for each pixel with respect to an evaluation frame of an arbitrary range centered on the pixel of interest, and the difference absolute value of the pixel values in the evaluation frame Accumulate. Specifically, when a pair of pixel values between the captured images 501 and 502 in the evaluation frame is expressed as f (i, j) and g (i, j) (i and j are coordinates), the formula (1) ) _{Is used} as the evaluation value _R.sub.SAD .

Since the portion where the evaluation value R _SAD decreases is a similar region, the region is searched, and the motion vector 503 of the moving object is calculated from the positional relationship of the blocks as shown in FIG. 5 (d).
In this embodiment, although the calculation method of the evaluation value using SAD which is the difference absolute value of pixel value was explained, it is not restricted to this, and other methods may be used.

In step S403, the subject information acquisition unit 303 acquires overlap information between frames of the moving object in the captured image acquired in step S401, and difference information between the moving object and the background.
The ratio of the overlapping area between the frames of the moving body is calculated as the overlapping information between the frames of the moving body.
FIG. 6 is a diagram for explaining the calculation of the overlapping area of the moving object. Based on the motion vector calculated in step S402, an area similar to a block having motion is detected, and the areas are put together as one moving object. In order to detect the area here, a clustering process as disclosed in Patent Document 2 is used. In this clustering process, distribution analysis of luminance and color information is performed, the similarity between adjacent blocks is determined, and when it is determined that they are similar, it is determined that they are the same block. By clustering processing, the same cluster is determined as one area of a mobile.

FIG. 6A shows a captured image 601 of the Nth frame divided into blocks of size (L × M) in step S402. Further, FIG. 6B shows a mobile region 603 which is a result of clustering blocks similar to the block corresponding to the movement vector 602. Also, the average value of the motion vectors included in the blocks of the moving object region 603 is linked as the motion vector of the moving object. Then, using the motion vector detected in N frames, the area to which the moving object in the N + 1 frame has moved is determined.
Next, the coordinates in the X-axis direction of each region are calculated using the mobile region detected in the Nth frame and the mobile region detected in the N + 1th frame. As shown in FIGS. 6C and 6D, the X coordinate position of the moving object in the N frame captured image 601 is XL _n , XR _n , and the X coordinate position of the moving object in the N + 1 _th captured image 604 is It becomes XL _{n + 1} and XR _{n + 1} . Then, as shown in FIGS. 6 (e) and 6 (f), the size LenX of the overlapping area is calculated using the equations (2) and (3) from the calculated X coordinates in consideration of the moving direction.
Then, from the calculated size LenX of the overlapping area and the total size of the moving body area, the ratio of the overlapping area between the frames of the moving body is calculated.

Also, as the difference information between the moving object and the background, the difference value of the contrast between the moving object and the background is calculated.
The edge strength of the moving object is used as contrast information of the moving object. This detects an edge signal by applying a band pass filter to an image, and integrates the absolute value of the edge signal within an arbitrary block size. When the integrated value of the edge signal which is the calculated value is large, that is, when the texture is large, the evaluation value Fr becomes large. The evaluation value Fr is calculated using equation (4). fr (i, j) is the value of the edge signal at each coordinate within an arbitrary block size.
The evaluation value of the moving object and the evaluation value of the background are calculated using Equation (4), and the difference value is taken as the difference value of the contrast between the moving object and the background.

In step S404, the motion blur strength determination unit 304 obtains an evaluation value of the ease of removal of the moving object when the motion blur is applied.
The evaluation value of the easiness of disappearance is an index that is defined as the degree to which the moving body passes through and becomes thin when the images are combined. In order to obtain the evaluation value, the motion vector of the moving body calculated in step S402, the ratio of the overlapping area between the frames of the moving body calculated in step S403, and the difference value of the contrast between the moving body and the background are used.
FIG. 7 shows the evaluation value of extinction and each information (proportion of overlapping area between frames of moving object, difference value of contrast between moving object and background, moving speed of moving object detected from motion vector ) Is a characteristic diagram showing the relationship. FIGS. 7 (a) to 7 (c) show graphs showing evaluation values of the ease of disappearance in each information. The horizontal axis of the graph represents each information, and the vertical axis represents the evaluation value of the ease of disappearance. Further, FIG. 7D shows the weight set for each piece of information. The evaluation value of the ease of erasure used here is a value represented in the range of 0 to 1, and the evaluation value of the ease of erasure is assigned to the calculated value of each information.

From the relationship between each information and the disappearance of the moving body, the evaluation value of the disappearance is calculated.
When the ratio of overlapping areas between frames of moving objects is low, there are few parts where moving objects overlap between frames, so when the number of combined sheets increases, the specific gravity of the background increases and the moving objects appear thin, and after combining The moving object tends to disappear in the image of. Therefore, as shown in FIG. 7A, the evaluation value of the ease of disappearance is increased. On the other hand, when the ratio of the overlapping area between the frames of the moving object is high, the moving object is hard to disappear, so the evaluation value of the ease of elimination is reduced.

  When the difference value of the contrast between the moving object and the background is large, when the number of combined images increases, the specific gravity of the background increases and the background with high contrast appears darker than the moving object. Therefore, as shown in FIG. 7 (b), the evaluation value of the easiness of disappearance is increased. On the other hand, when the difference value between the contrast of the moving object and the background is small, the moving object is hard to disappear, so the evaluation value of the ease of elimination is reduced.

  When the movement of the moving object is fast, if the number of combined images increases, the specific gravity of the background increases, the moving object appears thin, and the moving object tends to disappear in the image after combination. Therefore, as shown in FIG. 7C, the evaluation value of the ease of disappearance is increased. On the other hand, when the movement of the moving object is slow, the moving object is hard to disappear, so the evaluation value of the ease of elimination is reduced.

When using a plurality of evaluation values obtained from each piece of information, prioritization is performed for each piece of information, and the evaluation value R _rm of the easiness of disappearance is calculated using Equation (5). r _n is an evaluation value of the eliminability of each information, b _n is a weight set for each information, and n is the number of acquired information. As shown in FIG. 7D, a weight is set for each information as b _n . Also, these weights may be specified by the user.
In the present embodiment, although three information is used, the present invention is not limited to this. For example, when only one of the pieces of information is used, the evaluation value of the easiness of disappearance obtained according to FIGS. 7A to 7C may be used as the evaluation value R _{rm of the} easiness of _deletion as it is.
As the evaluation value R _{rm of the} ease of disappearance thus obtained is larger, it is evaluated that the moving body is more likely to disappear in the combined image.

In step S405, the motion blur strength determination unit 304 determines the combined number as the motion blur strength in accordance with the evaluation value of the easiness of extinction obtained in step S404.
FIG. 8 is a characteristic diagram showing an example of the relationship between the combined number and the evaluation value of the easiness of disappearance. When the evaluation value of the easiness of disappearance is large, the moving object disappears when the combined number is increased, and therefore the combined number is decreased to weaken the blurring intensity. On the other hand, when the evaluation value of the ease of disappearance is small, the number of combined sheets is increased in order to increase the blur intensity. As described above, by determining the combined number according to the ease of disappearance of the moving body when the motion blur is applied, it becomes possible to apply the motion blur without the moving body disappearing. Note that the upper limit of the combined number may be calculated from the shutter speed of a captured moving image with respect to the shutter speed at long seconds when pseudo seconds are used.

  In step S406, the motion blur applying unit 305 combines the captured image acquired in step S401 based on the combined number determined in step S405, and generates a motion blur added image.

  After the number of combined images is determined in step S405 in a period for generating a pseudo long second moving image, thereafter, the combined number is uniformly applied, and images are overlapped while being sequentially shifted to perform combining, A pseudo long second moving image may be generated. Alternatively, the period for generating the pseudo long second moving image may be divided appropriately, and the number of combined images may be determined for each divided period.

  As described above, since the composite number is determined as the motion blur intensity according to the disappearance of the moving body when the motion blur is applied, the moving body is less likely to disappear in the image processing to which the motion blur is applied. be able to.

Second Embodiment
In the first embodiment, an example in which it is assumed that there is only one moving body in a captured image, but a plurality of moving bodies may be shown. In the second embodiment, when there are a plurality of moving objects, an example of determining a moving object (hereinafter referred to as a subject of interest) as a reference for determining the motion blur intensity will be described. In the second embodiment, even when a plurality of moving objects appear in the captured image, the subject of interest can be determined, and the pseudo long second moving image can be generated in consideration of the ease of disappearance of the subject of interest. .
The description of the configuration and the processing operation of the imaging device 100 will be omitted, and the descriptions of the components common to the first embodiment will be omitted, and differences from the first embodiment will be mainly described.

The subject information acquisition unit 303 acquires subject information for determining a subject of interest in addition to the subject information acquired in the first embodiment as subject information. As subject information for determining a subject of interest, size information of a moving body, main subject information which is information as to whether or not it is a main subject, and ratio information in which the moving body appears in all frames are used. .
The motion blur strength determination unit 304 determines a target subject among the plurality of moving objects based on the subject information acquired by the subject information acquisition unit 303. Then, similarly to the first embodiment, the ease of disappearance of the object of interest when motion blur is applied is evaluated, and the number of combined images is determined according to the evaluation result.

FIG. 9 is a flowchart showing processing performed by the imaging device 100 in the second embodiment. The flowchart of FIG. 9 is realized by the system control unit 101 reading out an operation program from the ROM 102, expanding it in the RAM 103, and executing it.
Steps S901 to S902 and S905 to S907 are the same as steps S401 to S402 and S404 to S406 in FIG. 4, and the description thereof will be omitted.
In step S903, the subject information acquisition unit 303 acquires overlap information between frames of moving objects in the captured image acquired in step S901, and difference information between the moving object and the background, as in the first embodiment. In addition to that, the subject information acquisition unit 303 acquires the size information of the moving object, the main object information, and the ratio information of the moving object in all the frames.

The mobile body size information is acquired, for example, by defining the number of pixels included in the mobile body region described in the first embodiment as the mobile body size.
The main subject information is acquired, for example, as the saliency. By using a generally known method, the difference between the features of the subject and the subject around the subject is calculated as the degree of saliency in order to determine whether or not the area is visually noticeable. In the calculation of the difference, the image is divided into arbitrary regions, and the feature amount of the block being focused on and the feature amounts of the blocks around it are used. If the difference is large, the correlation is low, so it is judged as a visually noticeable object. Here, the feature amount includes at least one or more of the luminance value, the hue value, and the edge intensity in the block area. An area having a high degree of saliency calculated is determined as a main subject.
The ratio information in which the moving object appears in all the frames is obtained by calculating the number of frames in which the moving object area appears. The moving body detected by the size information of the moving body is tracked, and the number of frames in which it is captured is calculated for each moving body region. Then, using the calculated number of frames, the proportion of the mobile object in all the frames is calculated.
In addition, the method of acquiring these subject information is an example, and is not limited to this.

In step S904, the motion blur strength determination unit 304 determines the subject of interest based on the size information of the moving object acquired in step S903, the main subject information, and the ratio information of the moving object in all frames.
FIG. 10 is a characteristic diagram showing the relationship between the priority and each information (the size of the moving object, the degree of saliency, and the ratio of being shown). FIGS. 10A to 10C show graphs showing the priority in each information. The horizontal axis of the graph represents each information, and the vertical axis represents the priority. Further, FIG. 10D shows the weight set for each piece of information. The priority used here is a value represented in the range of 0 to 1, and the priority is assigned to the calculated value of each information.

The evaluation value W (N) of the subject is calculated using equation (6) using the priority of each of the three information items for each moving object. a _n is a priority for each information, α _n is a weight set to the priority for each information, and n is the number of acquired information. As shown in FIG. 10D, a weight is set for each piece of information as α _n . Also, these weights may be specified by the user.
Among the evaluation values W (N) of the moving objects calculated in this manner, the moving object having the largest evaluation value is taken as the target subject.

  In this embodiment, size information of the moving object, main object information, and ratio information showing the moving object in all frames are used as object information for determining the object of interest. Thereby, it is possible to set a moving object estimated to be focused by the user as a focused subject. However, subject information for determining a subject of interest is not limited to this, and any one of the information may be used, or other information may be used. For example, overlap information between frames of a moving object or difference information between a moving object and a background may be used as object information for determining a subject of interest. In this case, the moving object that is most likely to disappear in the image after combination can be set as the target subject.

Third Embodiment
In the first and second embodiments, a motion blur added image is generated by combining the images, and the combined number is determined as the motion blur intensity. In the third embodiment, an example will be described in which a motion-blurred image is generated by performing filter processing on an image, and the number of taps and the filter coefficient, which are filter parameters of the filter processing, are determined as the motion blur intensity.
The description of the configuration and the processing operation of the imaging device 100 will be omitted, and the descriptions of the components common to the first embodiment will be omitted, and differences from the first embodiment will be mainly described.

As in the first embodiment, the motion blur strength determination unit 304 evaluates the disappearance of the moving object when the motion blur is applied. Then, according to the evaluation result, the motion blur strength determination unit 304 determines, as the motion blur strength, the number of taps and the filter coefficient of the filter processing to which motion blur is to be applied.
The motion blur imparting unit 305 generates a motion blur imparted image using the captured image acquired by the captured image acquisition unit 301 based on the number of filter processing taps and the filter coefficient determined by the motion blur strength determination unit 304.

FIG. 11 is a flowchart showing processing performed by the imaging device 100 in the third embodiment. The flowchart in FIG. 11 is realized by the system control unit 101 reading out an operation program from the ROM 102, expanding it in the RAM 103, and executing it.
Steps S1101 to S1104 are the same as steps S401 to S404 in FIG. 4, and the description thereof is omitted.
In step S1105, the motion blur strength determination unit 304 determines the number of filter processing taps and the filter coefficient as the motion blur strength based on the evaluation value of the easiness of cancellation obtained in step S1104.
FIG. 12 is a diagram for describing determination of the number of taps of filter processing and the filter coefficient. FIGS. 12A and 12B show a captured image 1201 of the Nth frame acquired by the captured image acquisition unit 301 and a captured image 1202 of the N + 1th frame. Further, FIG. 12C shows an outline of the process of changing the number of taps of the filter process, and FIG. 12D shows an outline of the process of changing the filter coefficient of the filter process.
First, the number of taps t is determined based on the motion vector in order to determine the number of taps of the filter processing as a reference.

When changing the number of taps, as shown in FIG. 12C, the number of taps t is changed based on the evaluation value of the ease of disappearance, and the number of taps T to be used in the filtering process is finally calculated. When the evaluation value of the ease of disappearance is large, the number of taps is reduced to weaken the motion blur intensity. On the other hand, when the evaluation value of the ease of disappearance is small, the number of taps is not changed.
When changing the filter coefficient, as shown in FIG. 12 (d), the filter coefficient within the tap number t determined based on the motion vector is changed based on the evaluation value of the easiness of disappearance to Make adjustments. When the evaluation value of the ease of disappearance is large, the filter coefficient near the moving object is increased in order to weaken the motion blur intensity. On the other hand, when the evaluation value of the ease of disappearance is small, the filter coefficient is not changed.
As described above, by determining the number of taps and the filter coefficient in accordance with the ease of disappearance of the moving body when the motion blur is applied, it becomes possible to apply the motion blur without the moving body disappearing.

  In step S1106, the motion blur applying unit 305 performs a filtering process on the captured image acquired in step S1101 based on the number of taps and the filter coefficient determined in step S1105, and generates a motion blur added image.

Fourth Embodiment
For example, when a series of moving images is switched from a normal moving image to a pseudo long second moving image, or when the combined number greatly changes between the previous and subsequent frames in the pseudo long moving image, the motion blur intensity changes suddenly to give an unnatural impression. Sometimes it becomes a movie.
The fourth embodiment aims to switch the motion blur intensity so as not to give an unnatural impression.
The description of the configuration and the processing operation of the imaging device 100 will be omitted, and the descriptions of the components common to the first embodiment will be omitted, and differences from the first embodiment will be mainly described.

FIG. 13 is a flowchart showing processing performed by the imaging device 100 in the fourth embodiment. The flowchart in FIG. 13 is realized by the system control unit 101 reading out an operation program from the ROM 102, loading it in the RAM 103, and executing it.
In step S1301, the system control unit 101 determines whether or not a composition target image, which is a composition target image, remains. If the compositing target image remains, the processing proceeds to step S1302. If no compositing target image remains, the present processing ends.
In step S1302, the system control unit 101 acquires the number of images to be synthesized.

  In step S1303, the system control unit 101 determines whether or not there is a change in the combined number of frames before and after. For example, when switching from a normal moving image to a pseudo long second moving image in a series of moving images, the previous frame is a frame not to be combined, and the subsequent frame is a frame to be combined. In addition, the number of combined images may be changed between the preceding and succeeding frames in the pseudo long second moving image. If there is a change in the number of combined sheets, the process proceeds to step S1304. If there is no change in the number of combined sheets, the process proceeds to step S1305. In this case, it may be determined whether or not the change in the combined number is equal to or more than a predetermined number, for example, as the change in the combined number.

  In step S1304, the system control unit 101 sets a transition period according to the change in the number of combined images in the previous and subsequent frames, and determines the number of combined images of the transition period. In the transition period image, in order to gradually change the motion blur intensity between the preceding and succeeding frames, a combined number different from the combined number acquired in step S1302 is determined.

FIG. 14 is a diagram for explaining the determination of the number of combined images of transition period images. FIG. 14A shows a characteristic line in which the horizontal axis represents time (frames acquired in time series) and the vertical axis represents the number of combined images.
A characteristic line 1401 is a characteristic line when the transition period image is not generated. Here, a case is shown where the frame with the number of composites 1 (not composited) is switched to the frame with the number of composites 4. FIG. 14B shows the correspondence between the characteristic line 1401 and the frame to be synthesized. FIG. 14B shows the state of synthesis at successive times T1 to T5 in time series. According to the characteristic line 1401, the number of combined sheets is changed in the periods T1 to T2. As shown in FIG. 14B, the combination is not performed at time T1 because the number of combination is one. At time T2 after that, the number of combined sheets increases to four. Then, after time T3, the combined number is maintained at four.

On the other hand, the characteristic line 1402 is a characteristic line when generating the transition period image. FIG. 14C shows the correspondence between the characteristic line 1402 and the frame to be synthesized. FIG. 14C shows the state of synthesis at successive times T1 to T5 in time series. According to the characteristic line 1402, the combined number is gradually increased to 1, 2, 3 and 4 over time T1 to time T4. Then, after time T5, the combined number is kept at four.
As described above, the transition period 1411 is set between the preceding and succeeding frames, and a transition period image in which the number of combined images affecting the motion blur intensity is gradually generated is generated. Therefore, the motion blur intensity is set so as not to give an unnatural impression. It can be switched.

  Although the case where the number of combined sheets increases is described here, the same applies to the case where the number of combined sheets decreases as shown in FIG. A characteristic line 1403 is a characteristic line in the case where the transition period image is not generated, and shows a case where the frame of the number of composites 4 is switched to the frame of the number of composites 1 (not composited). On the other hand, the characteristic line 1404 is a characteristic line when generating the transition period image, and gradually decreases to 4, 3, 2, and 1 as the combined number from time T1 to time T4.

  In step S1305, under the control of the system control unit 101, the image processing unit 107 combines the combination target image with the combined number acquired in step S1302 or the combined number determined in step S1304.

The length of the transition period may be variable, and the length may be changed according to, for example, the motion vector of the moving object.
FIG. 15 is a diagram for describing determination of the length of the transition period and the number of combined images of the transition period. FIG. 15 shows a characteristic line in which the horizontal axis represents time (frames acquired in time series) and the vertical axis represents the number of combined images.
A characteristic line 1501 is a characteristic line when the transition period image is not generated. On the other hand, characteristic lines 1502 to 1504 are characteristic lines in the case of generating the transition period image, and the transition periods 1511 to 1513 are lengthened in the order of the characteristic lines 1502, 1503 and 1504.

When the number of combined images changes between the previous and next frames, the difference between the frames may or may not be visually noticeable. Therefore, when the difference between the frames is visually noticeable, the transition period is lengthened to make the change in the number of combined images more gradual.
Whether or not the difference between the frames is visually noticeable can be determined based on the motion vector of the moving object. The method of calculating the motion vector is as described in the first embodiment. With respect to the motion vectors of the (L × M) blocks thus obtained, the largest motion vector is selected, and the characteristic lines 1501 to 1504 are selected based on the magnitude thereof. For example, as the motion vector is larger (the motion of the moving body is larger), a characteristic line with a longer transition period is selected. If the motion vector is very small (if there is almost no movement of the moving object), the characteristic line 1501 may be selected. If the movement of the moving body is scarce, the difference between the frames is visually inconspicuous even when the number of combined images changes between the previous and next frames.

Although the characteristic line is selected based on the magnitude of the largest motion vector, the present invention is not limited to this. For example, the characteristic line may be selected based on the magnitude of the change in the number of combined frames in the previous and subsequent frames. In this case, a characteristic line is selected to make the transition period longer as the change in the number of combined images in the previous and subsequent frames is larger.
In addition, the characteristic line may be selected for each area in which the motion vector is calculated. Thereby, different motion blur can be applied to each area according to the movement of the moving body.

Fifth Embodiment
Also in the fifth embodiment, as in the fourth embodiment, the object is to switch the motion blur intensity so as not to give an unnatural impression.
The description of the configuration and the processing operation of the imaging device 100 will be omitted, and the descriptions of the components common to the first embodiment will be omitted, and differences from the first embodiment will be mainly described.

FIG. 16 is a flowchart showing processing performed by the imaging device 100 in the fifth embodiment. The flowchart in FIG. 16 is realized by the system control unit 101 reading out an operation program from the ROM 102, loading it in the RAM 103, and executing it.
In step S1601, the system control unit 101 determines whether an image to be subjected to filter processing remains. If there is an image to be subjected to the filter process, the process proceeds to step S1602. If there is no image to be subjected to the filter process, the process ends.
In step S1602, the system control unit 101 acquires the filter strength of the image to be subjected to filter processing. Here, the filter strength is represented by the number of taps of the filter processing.

  In step S1603, the system control unit 101 determines whether or not there is a change in filter strength in the previous and subsequent frames. For example, when switching from a normal moving image to a pseudo long second moving image in a series of moving images, the previous frame is a frame not subjected to the filtering process, and the subsequent frame is a frame subjected to the filtering process. In addition, the filter strength may change between the preceding and succeeding frames in the pseudo long second moving image. If there is a change in the filter strength, the process proceeds to step S1604. If there is no change in the filter strength, the process proceeds to step S1605. In this case, for example, it may be determined whether the change in the filter strength is equal to or greater than a predetermined strength as the change in the filter strength.

  In step S1604, the system control unit 101 sets a transition period according to the change in filter strength in the previous and subsequent frames, and determines the filter strength of the transition period image. In the transition period image, in order to gradually change the motion blur intensity between the preceding and succeeding frames, a filter intensity different from the filter intensity acquired in step S1602 is determined.

FIG. 17 is a diagram for explaining the determination of the filter strength of the transition period image. FIG. 17A shows a characteristic line in which the horizontal axis is time (frame acquired in time series) and the vertical axis is filter strength (number of taps).
A characteristic line 1701 is a characteristic line when the transition period image is not generated. Here, the case where the frame with one tap (without the influence of the adjacent pixels) is switched to the frame with seven taps is shown. According to the characteristic line 1701, the number of taps is changed in the periods T1 to T2.

On the other hand, the characteristic line 1702 is a characteristic line when generating the transition period image. FIG. 17B shows the correspondence between the characteristic line 1702 and the number of taps. FIG. 17B shows the state of filter processing at time T1 to T4 continuous in time series. According to the characteristic line 1702, the number of taps gradually increases to 1, 3, 5, 7 from time T1 to time T4. Then, after time T4, the number of taps is maintained at seven.
In this way, the transition period 1711 between the previous and subsequent frames is set, and a transition period image in which the filter intensity affecting the motion blur intensity is gradually changed is generated, so that the motion blur intensity is prevented so as to give an unnatural impression. It can be switched.

  In step S1605, under the control of the system control unit 101, the image processing unit 107 performs filter processing on an image to be subjected to filter processing with the filter strength acquired in step S1602 or the filter strength determined in step S1604.

In addition, although it demonstrated that the number of taps of filter processing was changed as control of filter intensity | strength, the same effect can be acquired by changing a filter coefficient.
FIG. 18 is a diagram for explaining the determination of the filter strength of the transition period image. FIG. 18 (a) shows a certain scene, and FIG. 18 (b) shows changes in filter coefficients in the scene. FIG. 18B shows the state of the filter processing at time T1 to T5 continuous in time series. When the filter strength is controlled by the filter coefficient, it is realized by blurring with a weight in the moving direction of the moving body. Therefore, first, the motion vector of the moving object is obtained. In the example of FIG. 18A, the motion vector of the moving body 1800 in the left direction is detected. At this time, at time T1, a filter coefficient having a large weight of the target pixel is selected. At subsequent times T2 to T5, the filter coefficient is gradually changed so that the weight of the pixel in the right direction of the pixel of interest becomes large.

Sixth Embodiment
In the sixth embodiment, an embodiment will be described in which a motion blur is added to a moving image captured by the user performing a zoom-in operation with the optical system 104 during shooting. Although the zoom-in operation is described as an example in the present embodiment, the present invention is similarly applicable to a moving image shot by performing the zoom-out operation.
The description of the configuration and the processing operation of the imaging device 100 will be omitted, and the descriptions of the components common to the first embodiment will be omitted, and differences from the first embodiment will be mainly described.

FIG. 19 is a diagram showing a functional configuration of the image processing unit 107 of the imaging device 100 according to the present embodiment.
The image acquisition unit 1901 acquires, for example, a captured image from the RAM 103.
The imaging information acquisition unit 1902 acquires various imaging information such as a focal length, a zoom magnification, and a movement of the imaging device 100 with respect to the captured image acquired by the image acquisition unit 1901.
The determination unit 1903 determines how to apply motion blur based on the captured image acquired by the image acquisition unit 1901 and the various types of imaging information acquired by the imaging information acquisition unit 1902.
The motion blur imparting unit 1904 generates a motion blur imparted image using the captured image acquired by the image acquiring unit 1901 based on the determination result of the determination unit 1903.
Although the image processing unit 107 has been described as functioning as the units 1901 to 1904, for example, the system control unit 101 may function as the judging unit 1903.

FIG. 20 is a flowchart showing processing performed by the imaging device 100 in the sixth embodiment.
In step S2001, the determination unit 1903 determines whether a zoom operation has been performed in a predetermined time unit in a moving image. Specifically, it is determined whether or not the zoom operation has been performed with reference to the value of the focal length for each frame constituting the moving image. If a change in focal length is detected, it is determined that a zoom operation is present, and the process proceeds to step S2002. If a change in focal length is not detected, it is determined that a zoom operation is not performed, and the process proceeds to step S2009.

In step S2002, the determination unit 1903 determines whether or not there is a section in which the zoom operation is interrupted (hereinafter referred to as a zoom interruption period) in the zoom operation period determined in step S2001. Specifically, it is detected how many frames there are sections in which the focal length does not change between the sections in which the focal length changes.
In step S2003, the determination unit 1903 determines whether or not the zoom break section detected in step S2002 is equal to or more than a predetermined number of frames T1. If the zoom interruption section is less than T1, the processing proceeds to step S2004, and if the zoom interruption section is T1 or more, the processing proceeds to step S2005.

In step S2004, under the control of the determination unit 1903, the motion blur applying unit 1904 excludes the frame of the zoom interruption section detected in step S2002, selects a frame to be used for composition, and generates a motion blur imparted image. This is because when the image of the frame belonging to the section in which the focal distance is not changed is used for the composition, the intensity of the motion blur is not uniform in the motion blur added image.
FIG. 21 is a diagram for explaining frame thinning combination in step S2004. The horizontal axis in FIG. 21 represents time [frame] and the vertical axis represents focal length [mm], and represents changes in focal length for each frame. In order to generate the motion blur added image corresponding to the frame 2101, the image of the frame 2101 and the images of the frames before and after that are synthesized, but at that time, the frame 2102 of the zoom interruption section is excluded and the zoom interruption section The outer frame 2103 is selected. Since the frame 2102 belongs to a section in which the focal distance does not change, when it is used for composition, the strength of the motion blur is not uniform in the motion blur added image.

In step S2005, the determination unit 1903 determines whether or not the zoom interruption section detected in step S2002 is equal to or more than T1 and less than a predetermined number of frames T2 [frame]. If the zoom interruption section is not less than T1 and less than T2, the process proceeds to step S2006, and if the zoom interruption section is not greater than T1 and less than T2, the process proceeds to step S2009.
In step S2006, the determination unit 1903 determines whether or not there is movement of the subject in the zoom interruption section detected in step S2002. Specifically, for example, as described in the first embodiment, detection of the moving body area is performed in block units, and movement is performed if the ratio of the moving body area to the image is a predetermined rate or more (for example, 30%) It is determined that there is Note that the method of determining the movement of the subject is not limited to this, and another method may be used. If it is determined that the subject has moved, the process proceeds to step S2007. If it is determined that the subject does not move, the process proceeds to step S2004.

  In step S2007, under the control of the determination unit 1903, the motion blur applying unit 1904 enlarges and crops the image of the frame of the zoom interruption section detected in step S2002, and then performs synthesis to generate a motion blur imparted image. This is because, when the motion of the subject is large, if the composition is performed by thinning out the frames as in step S2004, the subject with the motion becomes a sparse afterimage, and a natural motion blur effect can not be obtained. is there. In order to avoid this, by enlarging and cropping the image of the frame of the zoom interruption section, it is possible to generate an image in which the focal distance is changed in a pseudo manner and use it for composition.

FIG. 22 is a diagram for explaining frame enlargement and combining in step S2007. The horizontal axis in FIG. 22 represents time, and the vertical axis represents focal length, which represents the change in focal length for each frame. As shown in FIG. 22A, in the zoom interruption section and the subsequent sections until the zoom end, by correcting the focal length indicated by the solid line to a focal length indicated by the dotted line, a frame group without zoom interruption It can be done.
Moreover, as a correction pattern of a focal distance, what is shown to the dotted line of FIG.22 (b) may be used. That is, in order to make the change of the focal length constant, not only the zoom interruption section but also the image of the frame at the time before the start of the zoom is enlarged and cropped, and then the composition is performed. This makes it possible to generate a moving image with a slower zoom speed than the original moving image.
The correction pattern may be arbitrarily selected by the user, or may be automatically selected adaptively according to the scene. For example, if the difference between the focal length at the start of zooming and the focal length at the end of zooming is larger than a predetermined value (for example, 3 times), the zoom speed does not change or is dotted to make it faster as shown in FIG. Make the slope steeper. On the other hand, if the difference between the focal length at the start of zooming and the focal length at the end of zooming is smaller than a predetermined value, the zoom speed is reduced as shown in FIG. Also, the correction pattern may be selected according to the distance of the subject as well as the difference in focal length. For example, with reference to distance measurement information acquired in autofocus processing at the time of shooting, the subject is far away In this case, control such as increasing the zoom speed is also possible.
Further, as shown in FIG. 22C, it is also possible to correct in the direction of reduction instead of enlargement. In this case, in order to match the angle of view with the image of the non-reduced frame, it is necessary to hold in advance a wider angle of view than the angle of view to be displayed and recorded.

In step S2008, under the control of the determination unit 1903, the motion blur applying unit 1904 selects a frame to be used for composition so that the motion blur intensity applied during the zoom operation period becomes constant, and generates a motion blur imparted image. Do. Specifically, a frame to be used for composition is selected according to the amount of change of the focal length so that the appearance of motion blur does not become uneven even if the zoom speed is not constant.
FIG. 23 is a diagram for explaining the fixed combination in step S2008. The horizontal axis in FIG. 23 is time, and the vertical axis is focal length, which represents a change in focal length for each frame. In the zoom operation period, the zoom speed (slope of change in focal length) is different between the zoom section 1 and the zoom section 2. In this case, in the zoom section 1, in order to generate a motion blur applied image corresponding to the frame 2301, the image of the frame 2301 and the images of, for example, two frames 2302 before and after that are synthesized. Further, in the section 2, in order to generate a motion blur added image corresponding to the frame 2303, the image of the frame 2303 and the images of, for example, two frames 2304 adjacent to each other in a skipping manner before and after that are combined. This makes it possible to match the difference in focal length between the images of the respective frames to be combined, and the appearance of the motion blur imparted by the combination becomes uniform.

  In step S2009, under the control of the determination unit 1903, the motion blur applying unit 1904 performs motions by combining images of a predetermined number of adjacent frames on the time axis without thinning out or enlarging the frames. Generate a blurring image.

  In this embodiment, even when the zoom operation at the time of shooting is interrupted or uneven, the natural motion blur effect desired by the user can be obtained by controlling the composition of the images according to the change of the focal length. It is possible to generate moving images.

  Although the configuration for uniformly combining the images of the selected frame when generating the motion blur added image is described, the present invention is not limited to this, and the number of frames to be combined is made variable according to the area in the image. May be For example, when performing a zoom-in operation, a main subject to be noted is often present near the center of the image, and the combination may make the motion blur too strong and difficult to view. Therefore, as shown in FIG. 24, the number of composites in the center of the image is small, and the number of composites is large as the image height is high, so that the blur effect by the zoom operation is produced and the motion blur is strong for the main subject. It becomes possible to control so as not to be excessive.

As mentioned above, although the present invention was explained with an embodiment, the above-mentioned embodiment shows only an example of the embodiment in the case of carrying out the present invention, and the technical scope of the present invention is interpreted restrictively by these. It is a must-have. That is, the present invention can be implemented in various forms without departing from the technical concept or the main features thereof.
For example, in the embodiment, an imaging apparatus will be described as an example of the image processing apparatus, but the application destination of the present invention is not limited to the imaging apparatus, for example, a personal computer (PC) or the like to which a captured image by the imaging apparatus is input It is also good.
(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

  100: imaging apparatus, 101: system control unit, 102: ROM, 103: RAM, 104: optical system, 105: imaging unit, 106: A / D conversion unit, 107: image processing unit, 108: display unit, 109: Recording unit 301: captured image acquisition unit 302: moving body information acquisition unit 303: subject information acquisition unit 304: motion blur intensity determination unit 305: motion blur imparting unit 306: image output unit

Claims (19)

Acquisition means for acquiring information on moving objects in a plurality of images continuous in time series;
Motion blur intensity determination means for determining the motion blur intensity according to the ease of disappearance of the moving object when motion blur is applied, which is evaluated based on the information acquired by the acquisition means;
An image processing apparatus comprising: motion blur applying means for generating a motion blur applied image to which a motion blur is applied using the image based on the motion blur intensity determined by the motion blur intensity determination means .   The image processing apparatus according to claim 1, further comprising: a moving object information acquisition unit configured to acquire movement information on movement of the moving object in the image as the acquisition unit. The moving body information acquisition means calculates a motion vector of the moving body as the movement information,
The image processing apparatus according to claim 2, wherein the motion blur strength determination unit evaluates the disappearance of the moving object based on the motion vector.   4. The image processing apparatus according to claim 3, wherein the motion blur strength determination unit evaluates that the motion of the moving body detected from the motion vector is more likely to disappear as the motion of the moving body is faster. The image is a captured image,
The image processing apparatus according to any one of claims 1 to 4, further comprising subject information acquisition means for acquiring subject information as the subject of the moving object in the captured image as the acquisition means. The subject information acquisition means acquires, as the subject information, at least one of overlap information between images of the moving object and difference information between the moving object and the background, as the object information.
The image processing according to claim 5, wherein the motion blur strength determination unit evaluates the ease of disappearance of the moving object based on at least one of the overlap information and the difference information. apparatus.   7. The image processing apparatus according to claim 6, wherein the motion blur strength determination unit evaluates that the smaller the overlap between the images of the moving objects is, the more easily they disappear.   7. The image processing apparatus according to claim 6, wherein the motion blur strength determination unit evaluates that the difference between the moving object and the background is more likely to disappear as the difference is larger.   The image processing apparatus according to any one of claims 1 to 8, wherein the motion blur strength determination unit weakens the motion blur strength when it is evaluated that the moving object is easily disappeared.   The motion blur strength determination means determines the motion body serving as the reference for determining the motion blur strength based on the information acquired by the acquisition means when there are a plurality of the moving bodies, and then adds the motion blur. The image processing apparatus according to any one of claims 1 to 9, wherein the motion blur intensity is determined in accordance with the disappearance of the moving object as the reference at the same time.   The motion blur strength determining means determines the size of the moving body as the reference moving body, information on whether or not the moving body is a main subject, and the moving body is captured in the image. The image processing apparatus according to claim 10, wherein at least one of the ratio information is used. The motion blur applying means applies motion blur by combining the images;
The image processing apparatus according to any one of claims 1 to 11, wherein the motion blur strength determination unit determines a combined number as the motion blur strength. The motion blur applying means applies motion blur by applying a filtering process to the image,
The image processing apparatus according to any one of claims 1 to 11, wherein the motion blur intensity determination unit determines a filter parameter of the filter process as the motion blur intensity.   The image processing apparatus according to claim 13, wherein the filter parameter is at least one of the number of taps of the filter processing and a filter coefficient.   The image processing apparatus according to any one of claims 1 to 14, wherein the motion blur applying unit generates a plurality of the motion blur imparted images which are continuous in time series.   The motion blur applying means sets a transition period between the front and back images when the motion blur intensity is changed to a predetermined intensity or more between the front and back images, and the motion blur intensity is gradually changed in the transition period. The image processing apparatus according to any one of claims 1 to 15, wherein   17. The image processing apparatus according to claim 16, wherein the motion blur applying unit makes the length of the transition period variable. An acquisition step of acquiring information on a moving object in a plurality of successive images in time series;
A motion blur intensity determination step of determining a motion blur intensity according to the ease of disappearance of the moving object when the motion blur is applied, which is evaluated based on the information acquired in the acquisition step;
An image processing apparatus characterized by further comprising: a motion blur applying step of generating a motion blur imparted image to which a motion blur is applied using the image based on the motion blur intensity determined in the motion blur intensity determination step; Control method. Acquisition means for acquiring information on moving objects in a plurality of images continuous in time series;
Motion blur intensity determination means for determining the motion blur intensity according to the ease of disappearance of the moving object when motion blur is applied, which is evaluated based on the information acquired by the acquisition means;
A program for causing a computer to function as motion blur imparting means for generating a motion blur imparted image to which motion blur is imparted using the image based on the motion blur intensity determined by the motion blur intensity determination means.

Images