JP2002366949A - Device and method for processing image, recording medium and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To detect a mixture ratio showing a mixed state of a plurality of objects. SOLUTION: A weight generating part 422 generates weight the value of which sequentially increases from a prescribed initial value in response to one frame. A weighted frame difference operating part 423 calculates a weighted pixel value by multiplying the weight supplied from the weight generating part 422. The weighted frame difference operating part 423 subtracts the weighted pixel value of a corresponding pixel from the pixel value of a pixel under consideration of an input image to calculate a weighted difference. A correlation value operating part 426 operates the correlation value between weighted difference image data supplied from a motion compensating part 424 and subjected to motion compensation and weighted difference image data supplied from a frame memory 425 and delayed by one frame in each pixel in each case of weight that sequentially increases from the initial value. A maximum value deciding part 427 detects weight in which correlation becomes maximum and outputs the weight as an estimated mixture ratio.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and method, a recording medium, and a program, and more particularly, to an image processing apparatus and method, a recording medium, and a method that consider a difference between a signal detected by a sensor and the real world. About the program.

[0002]

2. Description of the Related Art A technique for detecting an event in the real world by a sensor and processing sampling data output from an image sensor is widely used.

For example, in an image obtained by capturing an object moving in front of a predetermined stationary background with a video camera, motion blur occurs when the moving speed of the object is relatively high.

[0004]

When an object moves in front of a stationary background, not only the motion blur due to the mixing of the image of the moving object itself but also the image of the background and the image of the moving object move. Mixing occurs. Conventionally, it has not been considered to detect a mixed state between a background image and a moving object image.

SUMMARY OF THE INVENTION The present invention has been made in view of such a situation, and it is possible to detect a mixture ratio indicating a mixed state of a plurality of objects such as a background image and a moving object image. The purpose is to:

[0006]

An image processing apparatus according to the present invention generates a plurality of weights corresponding to a range of a mixture ratio value indicating a mixture state in pixel data of a plurality of objects in the real world, Weighting information generating means for generating weighting information indicating the generated individual weights; and weighting information between each pixel of the target frame of the image data and each pixel of an adjacent frame adjacent to the target frame of the image data. Weighted difference based on the weight
A weighted difference image data detecting unit that outputs as weighted difference image data corresponding to the frame of interest, and a motion vector indicating a relative motion between the pixel data of the frame of interest and the pixel data of the adjacent frame. The relative position between the weighted difference image data and the weighted difference image data of the adjacent frame is matched, and a target block including at least one pixel centered on each target pixel of the weighted difference image data of the target frame; A weighted difference image correlation data detecting means for calculating a correlation with a corresponding block consisting of at least one pixel of the difference image data and outputting it as weighted difference image correlation data; and a predetermined weight consisting of at least one pixel of the weighted difference image correlation data The correlation between the weighted difference image data for each unit Detecting the weight becomes maximum, characterized in that it comprises a mixing ratio detecting means for outputting a mixing ratio corresponding to the unit of the frame of interest.

The weighting information generating means sequentially generates weighting information indicating a plurality of different weights corresponding to the plurality of different mixing ratios, and the weighting difference image data detecting means corresponds to the plurality of different weights. ,
The weighted difference is detected and output as weighted difference image data corresponding to the frame of interest. The weighted difference image correlation data detecting means calculates a correlation corresponding to each of a plurality of different weights, and calculates the weighted difference image correlation data. The mixture ratio detection means detects a weight having a maximum correlation between the weighted difference image data among a plurality of different weights, and outputs the detected weight as a mixture ratio corresponding to a predetermined unit of the frame of interest. can do.

The weighted difference image correlation data detecting means detects, as the correlation, at least one attention block of at least one pixel centered on each attention pixel of the attention frame weight difference image data and at least one of the weight difference image data of adjacent frames. It is possible to calculate the absolute value of the difference with the corresponding block composed of pixels and output the result as weighted difference image correlation data.

The weighted difference image correlation data detecting means detects, as a correlation, at least one target block of at least one pixel centered on each target pixel of the target frame weighted difference image data and at least one of the weighted difference image data of adjacent frames. It is possible to calculate the sum of absolute differences with the corresponding block composed of pixels and output the result as weighted difference image correlation data.

The weighted difference image data detecting means determines that the pixel of the frame of interest is a mixed area in which a plurality of objects are mixed and belongs to a covered background area formed on the front end side in the movement direction of the foreground object. In this case, a weighted difference based on the weight indicated by the weight information is detected between each pixel of the frame of interest and each pixel of the frame before the frame of interest, and a plurality of objects are mixed in the pixels of the frame of interest. If it is a mixed area and belongs to an uncovered background area formed on the rear end side in the movement direction of the foreground object,
A weighted difference based on the weight indicated by the weighting information can be detected between each pixel of the frame of interest and each pixel of the frame next to the frame of interest.

The image processing apparatus mixes a foreground area consisting of a foreground object among a plurality of objects in image data, a background area consisting of a background object of a plurality of objects in image data, and a plurality of objects. A mixed background, a covered background region formed on the front end side in the movement direction of the foreground object, and a mixed region,
Identify the uncovered background area formed on the rear end side in the movement direction of the foreground object, and specify the foreground area,
An area information generating means for generating area information indicating a background area and a mixed area including a covered background area and an uncovered background area can be further provided.

[0012] The image processing apparatus may further include a motion vector detecting means for detecting a motion vector indicating a relative motion between the pixel data of the frame of interest and the pixel data of the adjacent frame.

[0013] The mixture ratio detecting means can detect a mixture ratio of pixels of a frame of interest belonging to a mixture region where a plurality of objects are mixed.

[0014] The image processing apparatus may further include a separation unit for separating at least a foreground object of the plurality of objects from the pixel data based on the mixture ratio.

[0015] The image processing apparatus can further include a motion blur adjusting means for adjusting the amount of motion blur of the separated foreground object.

The image processing apparatus may further include a synthesizing means for synthesizing another desired object and an object serving as a separated foreground based on the mixture ratio.

According to the image processing method of the present invention, a plurality of weights are generated in correspondence with a range of a value of a mixing ratio indicating a mixed state in pixel data of a plurality of objects in the real world. Weighting information generating step of generating weighting information shown, and a weighting difference based on the weight indicated by the weighting information between each pixel of the frame of interest of the image data and each pixel of an adjacent frame adjacent to the frame of interest of the image data , And a weighted difference image data detecting step of outputting as weighted difference image data corresponding to the frame of interest, and a motion vector indicating a relative motion between the pixel data of the frame of interest and the pixel data of the adjacent frame. , The relative position between the weighted difference image data of the frame of interest and the weighted difference image data of the adjacent frame. Calculating the correlation between the block of interest consisting of at least one pixel centered on each pixel of interest in the weighted difference image data of the frame of interest and the corresponding block consisting of at least one pixel of the weighted difference image data of the adjacent frame, Detecting a correlation data between the weighted difference images to be output as correlation data between the weighted difference images;
A mixing ratio detecting step of detecting a weight at which the correlation between the weighted difference image data becomes maximum for each predetermined unit of pixels and outputting the weight as a mixing ratio corresponding to the unit of the frame of interest.

The weighting information generating step sequentially generates weighting information corresponding to a plurality of different mixing ratios and indicating a plurality of different weights. The weighting difference image data detecting step includes a step of detecting a weighted difference image data corresponding to the plurality of different weights. Detecting a weighted difference, outputting the data as weighted difference image data corresponding to the frame of interest, and calculating a correlation data between the weighted difference images. Output as data, the mixing ratio detecting step detects a weight having a maximum correlation between the weighted difference image data among a plurality of different weights, and outputs the detected weight as a mixing ratio corresponding to a predetermined unit of the frame of interest. Can be

The step of detecting the correlation data between the weighted difference images includes, as a correlation, at least one block of interest including at least one pixel centered on each pixel of interest of the weighted difference image data of the frame of interest and at least one of the weighted difference image data of adjacent frames. It is possible to calculate the absolute value of the difference with the corresponding block composed of pixels and output the result as weighted difference image correlation data.

The step of detecting the correlation data between the weighted difference images includes, as a correlation, at least one block of the weighted difference image data of the frame of interest centered on at least one pixel of interest and at least one of the weighted difference image data of the adjacent frame. It is possible to calculate the sum of absolute differences with the corresponding block composed of pixels and output the result as weighted difference image correlation data.

The weighted difference image data detecting step includes:
When the pixel of the frame of interest is a mixed area where a plurality of objects are mixed and belongs to a covered background area formed on the front end side in the movement direction of the foreground object, each pixel of the frame of interest and the frame of interest Between each pixel of the previous frame and the weighted difference indicated by the weight information, the pixel of the frame of interest is a mixed area where a plurality of objects are mixed, and the motion of the foreground object If the pixel belongs to the uncovered background area formed on the rear end side in the direction, a weighting difference based on the weight indicated by the weighting information is detected between each pixel of the frame of interest and each pixel of the frame next to the frame of interest. You can make it.

In the image processing method, a foreground area consisting of a foreground object among a plurality of objects in image data, a background area consisting of a background object of a plurality of objects in image data, and a plurality of objects are mixed. A mixed background, a covered background region formed on the front end side in the movement direction of the foreground object, and a mixed region,
Identify the uncovered background area formed on the rear end side in the movement direction of the foreground object, and specify the foreground area,
An area information generating step of generating area information indicating a background area and a mixed area including a covered background area and an uncovered background area may be further provided.

The image processing method may further include a motion vector detecting step of detecting a motion vector indicating a relative motion between the pixel data of the frame of interest and the pixel data of the adjacent frame.

In the mixing ratio detecting step, a mixing ratio of pixels of a frame of interest belonging to a mixed region where a plurality of objects are mixed can be detected.

The image processing method may further include a separation step of separating at least a foreground object of the plurality of objects from the pixel data based on the mixture ratio.

The image processing method may further include a motion blur adjustment step of adjusting the amount of motion blur of the separated foreground object.

The image processing method may further include a synthesizing step of synthesizing another desired object and a separated foreground object based on the mixture ratio.

According to the program of the recording medium of the present invention, a plurality of weights are generated in correspondence with a range of a value of a mixing ratio indicating a mixed state in pixel data of a plurality of objects in the real world. Weighting information generating step of generating weighting information indicating: weighting based on the weight indicated by the weighting information between each pixel of a target frame of image data and each pixel of an adjacent frame adjacent to the target frame of image data A weighted difference image data detecting step of detecting a difference and outputting the weighted difference image data as the weighted difference image data corresponding to the frame of interest, and a motion vector indicating a relative motion between the pixel data of the frame of interest and the pixel data of the adjacent frame. The difference between the weighted difference image data of the frame of interest and the weighted difference image data of the adjacent frame. And the correlation between the target block consisting of at least one pixel centered on each target pixel of the weighted difference image data of the target frame and the corresponding block consisting of at least one pixel of the weighted difference image data of the adjacent frame. Calculating the correlation data between the weighted difference images and outputting the correlation data as the weighted difference image correlation data; and determining that the correlation between the weighted difference image data is maximum for each predetermined unit including at least one pixel of the weighted difference image correlation data. And a mixing ratio detecting step of detecting the weight of the target frame and outputting it as a mixing ratio corresponding to the unit of the frame of interest.

The weighting information generating step sequentially generates weighting information corresponding to a plurality of different mixing ratios and indicating a plurality of different weights, and the weighting difference image data detecting step corresponds to the plurality of different weights. Detecting a weighted difference, outputting the data as weighted difference image data corresponding to the frame of interest, and calculating a correlation data between the weighted difference images. Output as data, the mixing ratio detecting step detects a weight having a maximum correlation between the weighted difference image data among a plurality of different weights, and outputs the detected weight as a mixing ratio corresponding to a predetermined unit of the frame of interest. Can be

The step of detecting the correlation data between the weighted difference images includes, as a correlation, at least one block of the weighted difference image data of the frame of interest centered on at least one pixel and at least one of the weighted difference image data of the adjacent frame. It is possible to calculate the absolute value of the difference with the corresponding block composed of pixels and output the result as weighted difference image correlation data.

The step of detecting the correlation data between the weighted difference images includes, as a correlation, at least one block of the weighted difference image data of the frame of interest, which is made up of at least one pixel centered on each pixel of interest, and at least one of the weighted difference image data of the adjacent frame It is possible to calculate the sum of absolute differences with the corresponding block composed of pixels and output the result as weighted difference image correlation data.

The weighted difference image data detecting step includes:
When the pixel of the frame of interest is a mixed area where a plurality of objects are mixed and belongs to a covered background area formed on the front end side in the movement direction of the foreground object, each pixel of the frame of interest and the frame of interest Between each pixel of the previous frame and the weighted difference indicated by the weight information, the pixel of the frame of interest is a mixed area where a plurality of objects are mixed, and the motion of the foreground object If the pixel belongs to the uncovered background area formed on the rear end side in the direction, a weighting difference based on the weight indicated by the weighting information is detected between each pixel of the frame of interest and each pixel of the frame next to the frame of interest. You can make it.

In the program, a foreground area composed of objects serving as foreground among a plurality of objects in image data, a background area composed of objects serving as a background among a plurality of objects in image data, and a plurality of objects are mixed. A mixed area, a covered background area formed on the front end side of the foreground object in the direction of movement, and a mixed area, an uncovered background area formed on the rear end side of the foreground object in the movement direction And generating area information indicating a mixed area including a foreground area, a background area, and a covered background area and an uncovered background area.

The program may further include a motion vector detecting step of detecting a motion vector indicating a relative motion between the pixel data of the frame of interest and the pixel data of the adjacent frame.

In the mixing ratio detecting step, a mixing ratio of a pixel of a frame of interest belonging to a mixed region where a plurality of objects are mixed can be detected.

The program may further include a separation step of separating at least a foreground object of the plurality of objects from the pixel data based on the mixture ratio.

[0037] The program may further include a motion blur adjusting step of adjusting the amount of motion blur of the separated foreground object.

The program may further include a synthesizing step of synthesizing the desired other object and the separated foreground object based on the mixture ratio.

The program according to the present invention generates a plurality of weights corresponding to a range of a mixture ratio value indicating a mixture state in pixel data of a plurality of objects in the real world, and assigns weights indicating the generated individual weights. A weighting information generating step of generating information, and detecting a weighting difference based on the weight indicated by the weighting information between each pixel of the target frame of the image data and each pixel of an adjacent frame adjacent to the target frame of the image data A weighted difference image data detecting step of outputting as weighted difference image data corresponding to the frame of interest, and a motion vector indicating a relative motion between pixel data of the frame of interest and pixel data of an adjacent frame. Relative position between weighted difference image data of frame and weighted difference image data of adjacent frame At the same time, the correlation between the target block consisting of at least one pixel centered on each target pixel of the weighted difference image data of the target frame and the corresponding block consisting of at least one pixel of the weighted difference image data of the adjacent frame is calculated and weighted. A step of detecting weighted difference image correlation data to be output as difference image correlation data;
And a computer for executing a mixing ratio detecting step of detecting a weight at which the correlation between the weighted difference image data becomes maximum for each predetermined unit of pixels and outputting the detected weight as a mixing ratio corresponding to the unit of the frame of interest. And

The weighting information generating step sequentially generates weighting information corresponding to a plurality of different mixing ratios and indicating a plurality of different weights. The weighting difference image data detecting step corresponds to the plurality of different weights. Detecting a weighted difference, outputting the data as weighted difference image data corresponding to the frame of interest, and calculating a correlation data between the weighted difference images. Output as data, the mixing ratio detecting step detects a weight having a maximum correlation between the weighted difference image data among a plurality of different weights, and outputs the detected weight as a mixing ratio corresponding to a predetermined unit of the frame of interest. Can be

The step of detecting the correlation data between the weighted difference images includes, as a correlation, at least one block of the weighted difference image data of the frame of interest, which is made up of at least one pixel centered on each pixel of interest, and at least one of the weighted difference image data of the adjacent frame. It is possible to calculate the absolute value of the difference with the corresponding block composed of pixels and output the result as weighted difference image correlation data.

The step of detecting the correlation data between the weighted difference images includes, as the correlation, at least one block of the weighted difference image data of the frame of interest, which is made up of at least one pixel centered on each pixel of interest, and at least one of the weighted difference image data of the adjacent frame. It is possible to calculate the sum of absolute differences with the corresponding block composed of pixels and output the result as weighted difference image correlation data.

The weighted difference image data detecting step includes:
When the pixel of the frame of interest is a mixed area where a plurality of objects are mixed and belongs to a covered background area formed on the front end side in the movement direction of the foreground object, each pixel of the frame of interest and the frame of interest Between each pixel of the previous frame and the weighted difference indicated by the weight information, the pixel of the frame of interest is a mixed area where a plurality of objects are mixed, and the motion of the foreground object If the pixel belongs to the uncovered background area formed on the rear end side in the direction, a weighting difference based on the weight indicated by the weighting information is detected between each pixel of the frame of interest and each pixel of the frame next to the frame of interest. You can make it.

In the program, a foreground area composed of objects serving as foreground among a plurality of objects in image data, a background area composed of objects serving as a background among a plurality of objects in image data, and a plurality of objects are mixed. A mixed area, a covered background area formed on the front end side of the foreground object in the direction of movement, and a mixed area, an uncovered background area formed on the rear end side of the foreground object in the movement direction And generating area information indicating a mixed area including a foreground area, a background area, and a covered background area and an uncovered background area.

[0045] The program may further include a motion vector detecting step of detecting a motion vector indicating a relative motion between the pixel data of the frame of interest and the pixel data of the adjacent frame.

In the mixture ratio detecting step, the mixture ratio of the pixels of the frame of interest belonging to the mixture region where the plurality of objects are mixed can be detected.

The program may further include a separation step of separating at least a foreground object of the plurality of objects from the pixel data based on the mixture ratio.

[0048] The program may further include a motion blur adjustment step of adjusting the amount of motion blur of the separated foreground object.

The program may further include a synthesizing step of synthesizing the desired other object and the separated foreground object based on the mixture ratio.

In the image processing apparatus, method, and program according to the present invention, a plurality of weights are generated in correspondence with a range of a mixing ratio value indicating a mixing state in pixel data of a plurality of objects in the real world; Weighting information indicating the generated individual weights is generated, and weighting based on the weight indicated by the weighting information is performed between each pixel of the target frame of the image data and each pixel of an adjacent frame adjacent to the target frame of the image data. The difference is detected and output as weighted difference image data corresponding to the frame of interest, and the weighted difference of the frame of interest is calculated according to a motion vector indicating the relative motion between the pixel data of the frame of interest and the pixel data of the adjacent frame. The relative position of the image data and the weighted difference image data of the adjacent frame A target block consisting of at least one pixel centered on each pixel of interest of the weighted difference image data, correlation with a corresponding block of at least one pixel of the weighted difference image data between adjacent frames is calculated,
The weight is output as the weighted difference image correlation data, and for each predetermined unit including at least one pixel of the weighted difference image correlation data, a weight that maximizes the correlation between the weighted difference image data is detected and corresponds to the unit of the frame of interest. Is output as the mixing ratio.

[0051]

FIG. 1 is a diagram showing the configuration of an embodiment of a signal processing unit according to the present invention. CPU (Central P
rocessing Unit) 21 is a ROM (Read Only Memory) 2
2 or executes various processes according to a program stored in the storage unit 28. RAM (Random Access Memor
y) 23 stores programs and data executed by the CPU 21 as appropriate. These CPU 21, ROM 22, and RAM 23 are interconnected by a bus 24.

An input / output interface 25 is connected to the CPU 21 via a bus 24. The input / output interface 25 is connected to an input unit 26 including a keyboard, a mouse, a microphone, and the like, and an output unit 27 including a display, a speaker, and the like. The CPU 21 executes various processes in response to a command input from the input unit 26. Then, the CPU 21 outputs an image, a sound, and the like obtained as a result of the processing to the output unit 27.

The storage unit 28 connected to the input / output interface 25 is constituted by a hard disk, for example, and stores programs executed by the CPU 21 and various data. The communication unit 29 communicates with an external device via the Internet or another network. In the case of this example, the communication unit 29 functions as an acquisition unit that captures the output of the sensor.

Further, a program may be obtained through the communication unit 29 and stored in the storage unit 28.

The drive 30 connected to the input / output interface 25 includes a magnetic disk 51, an optical disk 5
2. When the magneto-optical disk 53, the semiconductor memory 54, and the like are mounted, they are driven to acquire programs, data, and the like recorded therein. The acquired programs and data are transferred and stored in the storage unit 28 as necessary.

Next, a more detailed example of a signal processing device for specifying a region where significant information is buried from the data acquired by the sensor and extracting the buried significant information will be described. . In the example below,
The CCD line sensor or the CCD area sensor corresponds to the sensor, the area information and the mixing ratio correspond to the significant information, and the mixed foreground and background and the motion blur correspond to the distortion in the mixed area.

FIG. 2 is a block diagram showing the signal processing unit.

It does not matter whether each function of the signal processing unit is realized by hardware or software.
That is, each block diagram in this specification may be considered as a hardware block diagram or a functional block diagram using software.

Here, the motion blur refers to a distortion included in an image corresponding to a moving object, which is caused by the movement of the object in the real world to be imaged and the characteristics of the image pickup by the sensor. .

In this specification, an image to be imaged, which corresponds to an object in the real world, is called an image object.

The input image supplied to the signal processing unit is supplied to an object extracting unit 101, a region specifying unit 103, a mixture ratio calculating unit 104, and a foreground / background separating unit 105.

The object extraction unit 101 roughly extracts an image object corresponding to a foreground object included in the input image, and supplies the extracted image object to the motion detection unit 102. The object extraction unit 101
For example, by detecting the outline of the image object corresponding to the foreground object included in the input image, the image object corresponding to the foreground object is roughly extracted.

The object extracting section 101 roughly extracts an image object corresponding to a background object included in the input image, and supplies the extracted image object to the motion detecting section 102. The object extraction unit 101
For example, the image object corresponding to the background object is roughly extracted from the difference between the input image and the extracted image object corresponding to the foreground object.

Also, for example, the object extracting unit 101
Is configured to roughly extract an image object corresponding to a foreground object and an image object corresponding to a background object from a difference between a background image stored in a background memory provided therein and an input image. It may be.

The motion detecting section 102 calculates the motion vector of the image object corresponding to the coarsely extracted foreground object, for example, by a method such as a block matching method, a gradient method, a phase correlation method, or a pel recursive method. The calculated motion vector and the position information of the motion vector (information for specifying the position of the pixel corresponding to the motion vector) are supplied to the region specifying unit 103, the mixture ratio calculating unit 104, and the motion blur extracting unit 106.

The motion vector output by the motion detecting section 102 includes information corresponding to the motion amount v.

Further, for example, the motion detection unit 102 calculates the motion vector for each image object together with the pixel position information for specifying the pixel in the image object by the motion blur adjustment unit 10.
6 may be output.

The amount of movement v is a value representing the change in the position of the image corresponding to the moving object in units of pixel intervals. For example, the image of the object corresponding to the foreground is set to 4
When moving so as to be displayed at a position separated by pixels, the motion amount v of the image of the object corresponding to the foreground is
It is set to 4.

The object extracting unit 101 and the motion detecting unit 102 are required when adjusting the amount of motion blur corresponding to a moving object.

The area specifying unit 103 specifies each of the pixels of the input image as one of a foreground area, a background area, and a mixed area, and for each pixel, selects one of the foreground area, the background area, and the mixed area. Information that belongs to
(Referred to as area information) to the mixture ratio calculation unit 104, the foreground / background separation unit 105, and the motion blur adjustment unit 106.

The mixture ratio calculation unit 104 calculates a mixture ratio (hereinafter, referred to as a mixture ratio α) corresponding to the pixels included in the mixture region 63 based on the input image and the region information supplied from the region identification unit 103. The calculated mixture ratio is supplied to the foreground / background separation unit 105.

The mixture ratio α is a value indicating a ratio of a component of an image corresponding to a background object (hereinafter, also referred to as a background component) in a pixel value, as shown in Expression (3) described later.

The foreground / background separation unit 105 includes the area specifying unit 10
3 and the mixture ratio calculation unit 104
Input image into a foreground component image composed of only components of the image corresponding to the foreground object (hereinafter also referred to as a foreground component) and a background component image composed of only the background component based on the mixture ratio α supplied from And supplies the foreground component image to the motion-blur adjusting unit 106 and the selecting unit 107. Note that the separated foreground component image may be used as a final output. It is possible to specify only the foreground and the background without considering the conventional mixed area, and obtain a more accurate foreground and background as compared with the separated system.

The motion-blur adjusting unit 106 determines a processing unit indicating one or more pixels included in the foreground component image based on the motion amount v and the area information obtained from the motion vector. The processing unit is data that specifies a group of pixels to be processed for adjusting the amount of motion blur.

The motion-blur adjusting unit 106 calculates the amount of motion-blur adjustment input to the signal processing unit, the foreground component image supplied from the foreground / background separating unit 105, the motion vector supplied from the motion detecting unit 102, and its positional information. And based on the processing unit, remove the motion blur included in the foreground component image, reduce the amount of motion blur, or adjust the amount of motion blur included in the foreground component image such as increasing the amount of motion blur Selecting the foreground component image in which the amount of motion blur has been adjusted
7 is output. The motion vector and its position information may not be used.

The selection unit 107 determines whether the foreground component image supplied from the foreground / background separation unit 105 and the amount of motion blur supplied from the motion blur adjustment unit 106 are based on a selection signal corresponding to the user's selection, for example. One of the adjusted foreground component images is selected, and the selected foreground component image is output.

Next, an input image supplied to the signal processing section will be described with reference to FIGS.

FIG. 3 is a diagram for explaining image pickup by a sensor. The sensor is, for example, a CCD (Cha
(rge-Coupled Device) It consists of a CCD video camera equipped with an area sensor. The object corresponding to the foreground in the real world moves horizontally between the object corresponding to the background and the sensor in the real world, for example, from left to right in the figure.

The sensor takes an image of the object corresponding to the foreground together with the object corresponding to the background. The sensor outputs the captured image in units of one frame. For example, the sensor outputs an image consisting of 30 frames per second. The exposure time of the sensor can be 1/30 second. The exposure time is a period from when the sensor starts converting the input light into electric charges to when the sensor finishes converting the input light into electric charges. Hereinafter, the exposure time is also referred to as a shutter time.

FIG. 4 is a diagram illustrating the arrangement of pixels.
In FIG. 4, A to I indicate individual pixels. The pixels are arranged on a plane corresponding to the image. One detection element corresponding to one pixel is arranged on the sensor. When the sensor captures an image, one detection element outputs a pixel value corresponding to one pixel forming the image. For example, the position of the detection element in the X direction corresponds to the position in the horizontal direction on the image, and the position of the detection element in the Y direction corresponds to the position in the vertical direction on the image.

As shown in FIG. 5, for example, a detection element such as a CCD converts input light into electric charges for a period corresponding to the shutter time, and accumulates the converted electric charges. The amount of charge is substantially proportional to the intensity of the input light and the time during which the light is input. The detection element adds the electric charge converted from the input light to the electric charge already stored in a period corresponding to the shutter time. That is, the detection element integrates the input light for a period corresponding to the shutter time, and accumulates an amount of charge corresponding to the integrated light.
It can be said that the detection element has an integration effect with respect to time.

The electric charge accumulated in the detecting element is converted into a voltage value by a circuit (not shown), and the voltage value is further converted into a pixel value such as digital data and output. Therefore, the individual pixel values output from the sensor are obtained by integrating the spatially extended part of the object corresponding to the foreground or background with respect to the shutter time, and projecting the value projected into a one-dimensional space. Have.

The signal processing section has significant information buried in the output signal by the operation of accumulating such a sensor.
For example, the mixture ratio α is extracted. The signal processing unit adjusts the amount of distortion caused by mixing of the foreground image objects themselves, for example, the amount of motion blur. The signal processing unit adjusts the amount of distortion caused by mixing of the foreground image object and the background image object.

FIG. 6 is a diagram for explaining an image obtained by capturing an object corresponding to a moving foreground and an object corresponding to a stationary background. FIG.
(A) shows an object corresponding to a foreground with motion;
An image obtained by capturing an object corresponding to a stationary background is shown. In the example shown in FIG. 6A, the object corresponding to the foreground is moving horizontally from left to right with respect to the screen.

FIG. 6B shows one of the images shown in FIG.
FIG. 6 is a model diagram in which pixel values corresponding to two lines are developed in the time direction. The horizontal direction in FIG. 6B corresponds to the spatial direction X in FIG.

The pixels in the background area have their pixel values composed only of the background components, ie, the image components corresponding to the background objects. Pixels in the foreground area are
The pixel value is composed of only the foreground components, that is, the components of the image corresponding to the foreground object.

The pixels of the mixed area have their pixel values made up of the background component and the foreground component. Since the pixel value of the mixed area is composed of the background component and the foreground component, it can be said that the mixed area is a distortion area. The mixed area is further classified into a covered background area and an uncovered background area.

The covered background area is a mixed area at a position corresponding to the front end of the foreground object in the traveling direction with respect to the foreground area, and is an area where the background component is covered by the foreground as time passes. Say.

On the other hand, the uncovered background area is a mixed area at a position corresponding to the rear end of the foreground object in the traveling direction with respect to the foreground area. Means the area where appears.

As described above, the image including the foreground area, the background area, or the covered background area or the uncovered background area is input to the area specifying unit 103,
The input image is input to the mixture ratio calculation unit 104 and the foreground / background separation unit 105.

FIG. 7 is a diagram for explaining the background area, foreground area, mixed area, covered background area, and uncovered background area as described above. In the case of corresponding to the image shown in FIG. 6, the background area is a stationary part, the foreground area is a moving part, the covered background area of the mixed area is a part that changes from the background to the foreground, The uncovered background area is a part that changes from the foreground to the background.

FIG. 8 shows, in the time direction, pixel values of adjacent pixels arranged in a line in an image obtained by capturing an object corresponding to a stationary foreground and an object corresponding to a stationary background. FIG. For example, as pixels arranged adjacently in one column,
Pixels arranged on one line can be selected.

The pixel values F01 to F04 shown in FIG. 8 are the pixel values of the pixels corresponding to the stationary foreground object. The pixel values B01 to B04 shown in FIG. 8 are the pixel values of the pixels corresponding to the stationary background object.

In the vertical direction in FIG. 8, time elapses from the top to the bottom in the figure. The position on the upper side of the rectangle in FIG. 8 corresponds to the time when the sensor starts converting the input light into electric charge, and the position on the lower side of the rectangle in FIG. Corresponding to the time at which the conversion of.
That is, the distance from the upper side to the lower side of the rectangle in FIG.
Corresponds to the shutter time.

Hereinafter, a case where the shutter time and the frame interval are the same will be described as an example.

The horizontal direction in FIG. 8 corresponds to the spatial direction X described in FIG. More specifically, in the example shown in FIG. 8, the distance from the left side of the rectangle described as “F01” to the right side of the rectangle described as “B04” in FIG. 8 is eight times the pixel pitch, That is, it corresponds to the interval between eight consecutive pixels.

When the foreground object and the background object are stationary, the light input to the sensor does not change during the period corresponding to the shutter time.

Here, the period corresponding to the shutter time is 2
Divide into two or more periods of equal length. For example, if the number of virtual divisions is 4, the model diagram shown in FIG. 8 can be represented as the model shown in FIG. The number of virtual divisions is set according to the amount of movement v of the object corresponding to the foreground within the shutter time. For example, the number of virtual divisions is set to 4 corresponding to the motion amount v of 4, and the period corresponding to the shutter time is divided into four.

The uppermost row in the figure corresponds to the first divided period after the shutter is opened. The second row from the top in the figure corresponds to the second divided period from when the shutter has opened. The third row from the top in the figure corresponds to the third divided period from when the shutter has opened. The fourth row from the top in the figure corresponds to the fourth divided period from when the shutter has opened.

Hereinafter, the shutter time divided according to the movement amount v is also referred to as shutter time / v.

When the object corresponding to the foreground is stationary, the light input to the sensor does not change, so that the foreground component F01 / v is equal to the value obtained by dividing the pixel value F01 by the number of virtual divisions. Similarly, when the object corresponding to the foreground is stationary, the foreground component F02 / v is equal to a value obtained by dividing the pixel value F02 by the number of virtual divisions, and the foreground component F03 / v is obtained by dividing the pixel value F03 by the virtual value. Equal to the value divided by the number of divisions, the foreground component F04 / v is
It is equal to a value obtained by dividing the pixel value F04 by the number of virtual divisions.

When the object corresponding to the background is stationary, the light input to the sensor does not change. Therefore, the background component B01 / v is equal to a value obtained by dividing the pixel value B01 by the number of virtual divisions. Similarly, when the object corresponding to the background is stationary, the background component B02 / v is equal to the value obtained by dividing the pixel value B02 by the virtual division number, and B03 / v is the pixel value B03 obtained by dividing the pixel value B03 by the virtual division number. B04 / v is equal to the value obtained by dividing the pixel value B04 by the virtual division number.

That is, when the object corresponding to the foreground is stationary, the light corresponding to the object of the foreground input to the sensor does not change during the period corresponding to the shutter time. Foreground component F01 / v corresponding to time / v, and second foreground component F0 corresponding to shutter time / v after shutter is opened
1 / v, the third foreground component F01 / v corresponding to the shutter time / v when the shutter is opened, and the fourth foreground component F01 / v corresponding to the shutter time / v when the shutter is opened. Is
It has the same value. F02 / v to F04 / v also have the same relationship as F01 / v.

When the object corresponding to the background is stationary, the light corresponding to the background object input to the sensor does not change during the period corresponding to the shutter time. A background component B01 / v corresponding to v, a second background component B01 / v corresponding to the shutter time / v when the shutter is opened, and a third component corresponding to the shutter time / v when the shutter is opened. The background component B01 / v has the same value as the background component B01 / v corresponding to the fourth shutter time / v after the shutter is opened. B02 / v to B04 / v also have the same relationship.

Next, the case where the object corresponding to the foreground moves and the object corresponding to the background is stationary will be described.

FIG. 10 is a model diagram in which pixel values of pixels on one line including a covered background area are developed in the time direction when the object corresponding to the foreground moves rightward in the figure. is there. In FIG. 10, the motion amount v of the foreground is 4. Since one frame is a short time, the object corresponding to the foreground is a rigid body,
It can be assumed that they are moving at a constant speed. In FIG. 10, the image of the object corresponding to the foreground moves so as to be displayed four pixels to the right in the next frame with reference to a certain frame.

In FIG. 10, the leftmost pixel to the fourth pixel from the left belong to the foreground area. In FIG. 10, the fifth to seventh pixels from the left belong to the mixed area that is the covered background area. FIG.
In, the rightmost pixel belongs to the background area.

Since the object corresponding to the foreground moves so as to cover the object corresponding to the background with the passage of time, the components included in the pixel values of the pixels belonging to the covered background area correspond to the shutter time. At some point during the period, the background component changes to the foreground component.

For example, in FIG. 10, a pixel value with a thick frame
M is represented by equation (1).

M = B02 / v + B02 / v + F07 / v + F06 / v (1)

For example, the fifth pixel from the left includes a background component corresponding to one shutter time / v and a foreground component corresponding to three shutter times / v.
The mixture ratio α of the second pixel is 1/4. Since the sixth pixel from the left includes a background component corresponding to two shutter times / v and a foreground component corresponding to two shutter times / v, the mixture ratio α of the sixth pixel from the left is , 1/2. Since the seventh pixel from the left includes a background component corresponding to three shutter times / v and a foreground component corresponding to one shutter time / v, the mixture ratio α of the seventh pixel from the left is , 3/4.

Since it can be assumed that the object corresponding to the foreground is a rigid body and the foreground image moves at a constant speed so as to be displayed four pixels to the right in the next frame, for example, the fourth object from the left in FIG. The foreground component F07 / v of the pixel of the first shutter time / v after the shutter is opened is shown in FIG.
It is equal to the foreground component of the fifth pixel from the left in 0 corresponding to the second shutter time / v from when the shutter has opened. Similarly, the foreground component F07 / v is the foreground component of the sixth pixel from the left in FIG. 10 corresponding to the third shutter time / v from when the shutter has opened, and the seventh pixel from the left in FIG. Of the pixel of
It is equal to the foreground component corresponding to the fourth shutter time / v after the shutter is opened.

Since it can be assumed that the object corresponding to the foreground is a rigid body and the foreground image moves at a constant speed so as to be displayed four pixels to the right in the next frame, for example, the third object from the left in FIG. The foreground component F06 / v of the pixel of the first shutter time / v from when the shutter has opened is shown in FIG.
It is equal to the foreground component of the fourth pixel from the left, corresponding to the second shutter time / v from when the shutter has opened. Similarly, the foreground component F06 / v is the foreground component of the fifth pixel from the left in FIG. 10 corresponding to the third shutter time / v from when the shutter has opened, and the sixth pixel from the left in FIG. Of the pixel of
It is equal to the foreground component corresponding to the fourth shutter time / v after the shutter is opened.

Since it can be assumed that the object corresponding to the foreground is a rigid body and the foreground image moves at a constant speed so as to be displayed four pixels to the right in the next frame, for example, the second from the left in FIG. The foreground component F05 / v of the pixel of the first shutter time / v from when the shutter has opened is shown in FIG.
It is equal to the foreground component of the third pixel from the left corresponding to the second shutter time / v from when the shutter has opened. Similarly, the foreground component F05 / v is the foreground component of the fourth pixel from the left in FIG. 10 corresponding to the third shutter time / v from when the shutter is opened, and the fifth pixel from the left in FIG. Of the pixel of
It is equal to the foreground component corresponding to the fourth shutter time / v after the shutter is opened.

Since it can be assumed that the object corresponding to the foreground is a rigid body and the foreground image moves at a constant speed so as to be displayed four pixels to the right in the next frame, for example, the leftmost pixel in FIG. The foreground component F04 / v of the first shutter time / v from when the shutter is opened is the foreground component of the second pixel from the left in FIG. 10 corresponding to the second shutter time / v from when the shutter is opened. be equivalent to. Similarly, the foreground component F04 / v is the third pixel from the left in FIG.
The foreground component corresponding to the third shutter time / v after the shutter is opened, and the foreground component corresponding to the fourth shutter time / v of the fourth pixel from the left in FIG. , Respectively.

Since the foreground area corresponding to the moving object includes the motion blur, it can be said that the area is a distortion area.

FIG. 11 is a model diagram in which pixel values of pixels on one line including the uncovered background area are developed in the time direction when the foreground moves rightward in the figure. In FIG. 11, the foreground motion amount v
Is 4. Since one frame is a short time, it can be assumed that the object corresponding to the foreground is rigid and moves at a constant speed. In FIG. 11, the image of the object corresponding to the foreground moves to the right by four pixels in the next frame with reference to a certain frame.

In FIG. 11, the leftmost pixel to the fourth pixel from the left belong to the background area. In FIG. 11, the fifth through seventh pixels from the left belong to the mixed area that is the uncovered background. FIG.
In, the rightmost pixel belongs to the foreground area.

Since the object corresponding to the foreground, which has covered the object corresponding to the background, moves so as to be removed from the front of the object corresponding to the background with the passage of time, the pixels belonging to the uncovered background area At some point during the period corresponding to the shutter time, the component included in the value is changed from the foreground component to the background component.

For example, in FIG. 11, a pixel value with a thick frame
M ′ is represented by equation (2).

M ′ = F02 / v + F01 / v + B26 / v + B26 / v (2)

For example, the fifth pixel from the left includes a background component corresponding to three shutter times / v and a foreground component corresponding to one shutter time / v.
The mixture ratio α of the third pixel is 3/4. Since the sixth pixel from the left includes a background component corresponding to two shutter times / v and a foreground component corresponding to two shutter times / v, the mixture ratio α of the sixth pixel from the left is , 1/2. Since the seventh pixel from the left includes a background component corresponding to one shutter time / v and a foreground component corresponding to three shutter times / v, the mixture ratio α of the seventh pixel from the left is , 1/4.

When the equations (1) and (2) are generalized, the pixel value M is expressed by the equation (3).

[0124]

(Equation 1) Here, α is a mixture ratio. B is a pixel value of the background, and Fi / v is a foreground component.

Since it can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed, and the movement amount v is 4, for example, the shutter of the fifth pixel from the left in FIG. First foreground component of shutter time / v F01 /
v is equal to the foreground component of the sixth pixel from the left in FIG. 11 corresponding to the second shutter time / v from when the shutter has opened. Similarly, F01 / v is the foreground component of the seventh pixel from the left in FIG. 11 corresponding to the third shutter time / v from when the shutter has opened, and the eighth pixel from the left in FIG. ,
It is equal to the foreground component corresponding to the fourth shutter time / v after the shutter is opened.

Since it can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed, and the number of virtual divisions is 4, for example, the shutter of the sixth pixel from the left in FIG. First foreground component F0 of shutter time / v
2 / v is equal to the foreground component of the seventh pixel from the left in FIG. 11 corresponding to the second shutter time / v from when the shutter has opened. Similarly, the foreground component F02 / v is equal to the foreground component of the eighth pixel from the left in FIG. 11 corresponding to the third shutter time / v from when the shutter has opened.

Since it can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed, and the amount of movement v is 4, for example, the shutter of the seventh pixel from the left in FIG. First foreground component of shutter time / v F03 /
v is equal to the foreground component of the eighth pixel from the left in FIG. 11 corresponding to the second shutter time / v from when the shutter has opened.

In the description of FIGS. 9 to 11, the number of virtual divisions has been described as four, but the number of virtual divisions corresponds to the amount of motion v. The movement amount v generally corresponds to the moving speed of the object corresponding to the foreground. For example, when the object corresponding to the foreground is moving so as to be displayed four pixels to the right in the next frame with respect to a certain frame, the motion amount v is set to 4. The virtual division number is set to 4 corresponding to the movement amount v. Similarly, for example, when the object corresponding to the foreground is moving so as to be displayed 6 pixels to the left in the next frame with respect to a certain frame, the motion amount v is set to 6, and the number of virtual divisions is , 6.

FIG. 12 and FIG.
The relationship between a mixed area including a foreground area, a background area, a covered background area, or an uncovered background area, and foreground components and background components corresponding to the divided shutter times is shown.

FIG. 12 shows an example in which pixels of a foreground area, a background area, and a mixed area are extracted from an image including a foreground corresponding to an object moving in front of a stationary background. In the example shown in FIG. 12, the object corresponding to the foreground is moving horizontally with respect to the screen.

The frame # n + 1 is a frame next to the frame #n, and the frame # n + 2 is a frame next to the frame # n + 1.

The pixels of the foreground area, the background area, and the mixed area, which are extracted from any of the frames #n to # n + 2, are extracted. FIG. 13 shows a model developed in the time direction.

Since the object corresponding to the foreground moves, the pixel value of the foreground area is composed of four different foreground components corresponding to the shutter time / v period. For example, the leftmost pixel among the pixels in the foreground area shown in FIG. 13 includes F01 / v, F02 / v, F03 / v, and F04 / v. That is, pixels in the foreground area include motion blur.

Since the object corresponding to the background is stationary, the light input to the sensor corresponding to the background does not change during the period corresponding to the shutter time. In this case, the pixel value of the background area does not include motion blur.

The pixel value of a pixel belonging to a mixed area composed of a covered background area or an uncovered background area is composed of a foreground component and a background component.

Next, when the image corresponding to the object is moving, the pixel values of pixels that are adjacently arranged in one row in a plurality of frames and located at the same position on the frame are determined in the time direction. The following is an explanation of the model developed in. For example, when an image corresponding to an object is moving horizontally with respect to the screen, pixels arranged on one line of the screen can be selected as pixels arranged adjacently in one row.

FIG. 14 is a diagram showing three adjacent frames of three frames of an image of an object corresponding to a stationary background, which are pixels arranged at the same position on the frame. FIG. 4 is a model diagram in which values are developed in a time direction. Frame #n is the next frame after frame # n-1, and frame # n + 1 is the next frame after frame #n. Other frames are similarly referred to.

The pixel values B01 to B12 shown in FIG. 14 are the pixel values of the pixels corresponding to the stationary background object. Since the object corresponding to the background is stationary, the pixel values of the corresponding pixels do not change in frames # n-1 to n + 1. For example, the pixel in frame #n and the pixel in frame # n + 1 corresponding to the position of the pixel having the pixel value of B05 in frame # n-1 each have the pixel value of B05.

FIG. 15 is a diagram showing three pixels adjacent to one another in three frames of an image obtained by capturing an object corresponding to a foreground moving to the right in the figure together with an object corresponding to a stationary background. FIG. 4 is a model diagram in which pixel values of pixels at the same position on a frame are developed in the time direction. The model shown in FIG. 15 includes a covered background area.

In FIG. 15, it can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed. The quantity v is 4, and the number of virtual divisions is 4.

For example, the foreground component of the leftmost pixel of frame # n-1 in FIG. 15 corresponding to the first shutter time / v from when the shutter has opened is F12 / v, and the second component from the left in FIG. The foreground component of the pixel of the second shutter time / v after the shutter is opened also becomes F12 / v. The foreground component of the third pixel from the left in FIG. 15 at the third shutter time / v from when the shutter has opened, and the fourth pixel from the left in FIG.
The foreground component of the fourth shutter time / v after the shutter is opened is F12 / v.

The foreground component of the leftmost pixel of frame # n-1 in FIG. 15 corresponding to the second shutter time / v from when the shutter has opened is F11 / v, and the second component from the left in FIG. The foreground component of the pixel at the third shutter time / v from when the shutter has opened is also F11 / v. The foreground component of the third pixel from the left in FIG. 15 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is F11 / v.

The foreground component of the leftmost pixel of frame # n-1 in FIG. 15 corresponding to the third shutter time / v from when the shutter has opened is F10 / v, which is the second to the left in FIG. The foreground component of the pixel at the fourth shutter time / v from when the shutter has opened is also F10 / v. The fourth shutter time of the leftmost pixel of frame # n-1 in FIG. 15 from when the shutter has opened
The foreground component of / v is F09 / v.

Since the object corresponding to the background is stationary, the background component of the second pixel from the left of frame # n-1 in FIG. 15 corresponding to the first shutter time / v from when the shutter has opened is B01 / v. The background components of the third pixel from the left of frame # n-1 in FIG. 15 corresponding to the first and second portions of the shutter time / v from when the shutter has opened are B02 / v. FIG.
The background component of the fourth pixel from the left of the middle frame # n-1 corresponding to the first to third shutter time / v from when the shutter has opened is B03 / v.

In the frame # n-1 in FIG. 15, the leftmost pixel belongs to the foreground area, and the second to fourth pixels from the left belong to the mixed area which is a covered background area.

The fifth through twelfth pixels from the left of frame # n-1 in FIG. 15 belong to the background area, and their pixel values are B04 through B11, respectively.

The first through fifth pixels from the left of frame #n in FIG. 15 belong to the foreground area. Frame #n
The foreground component of shutter time / v in the foreground area of
One of F05 / v to F12 / v.

It can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed. Since the foreground image moves so as to be displayed four pixels to the right in the next frame, frame #n in FIG. The foreground component of the fifth pixel from the left of the first shutter time / v from when the shutter has opened is
F12 / v, the foreground component of the sixth pixel from the left in FIG. 15 corresponding to the second shutter time / v from when the shutter has opened is also
It becomes F12 / v. The foreground component of the seventh pixel from the left in FIG. 15 at the third shutter time / v from when the shutter has opened, and the fourth shutter time of the eighth pixel from the left in FIG. 15 since the shutter has opened. The foreground component of / v is F12 / v.

The foreground component of the fifth pixel from the left of frame #n in FIG. 15 corresponding to the second shutter time / v from when the shutter has opened is F11 / v, and the sixth pixel from the left in FIG. 15 is F11 / v. The foreground component of the pixel at the third shutter time / v from when the shutter has opened is also F11 / v. The foreground component of the seventh pixel from the left in FIG. 15 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is F11 / v.

The foreground component of the fifth pixel from the left of frame #n in FIG. 15 corresponding to the third shutter time / v from when the shutter has opened is F10 / v, and the sixth pixel from the left in FIG. 15 is F10 / v. The foreground component of the pixel at the fourth shutter time / v from when the shutter has opened is also F10 / v. The foreground component of the fifth pixel from the left of frame #n in FIG. 15 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is F09 / v.

Since the object corresponding to the background is stationary, the background component of the sixth pixel from the left of frame #n in FIG. 15 corresponding to the first portion of the shutter time / v from when the shutter has opened is B05 / v Becomes 7 from the left of frame #n in FIG.
The background component of the first pixel at the first and second shutter time / v from when the shutter has opened is B06 / v. The background components of the eighth pixel from the left of frame #n in FIG. 15 corresponding to the first to third portions of the shutter time / v from when the shutter has opened are:
B07 / v.

In frame #n in FIG. 15, the sixth through eighth pixels from the left belong to the mixed area which is a covered background area.

The ninth to twelfth pixels from the left of frame #n in FIG. 15 belong to the background area, and the pixel value is
B08 to B11, respectively.

The first through ninth pixels from the left of frame # n + 1 in FIG. 15 belong to the foreground area. flame
The foreground component of the shutter time / v in the foreground area of # n + 1 is one of F01 / v to F12 / v.

It can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed. Since the foreground image moves so as to be displayed four pixels to the right in the next frame, frame #n in FIG. Of the ninth pixel from the left of +1
The foreground component of the first shutter time / v after the shutter is opened is F12 / v, and the foreground component of the tenth pixel from the left in FIG. It becomes F12 / v. The foreground component of the eleventh pixel from the left in FIG. 15 corresponding to the third shutter time / v from when the shutter has opened, and the fourth shutter time since the shutter opening of the twelfth pixel from the left in FIG. The foreground component of / v is F12 / v.

The foreground component of the ninth pixel from the left of frame # n + 1 in FIG. 15 corresponding to the second portion of the shutter time / v from when the shutter has opened is F11 / v, and as shown in FIG. From 1
The foreground component of the 0th pixel at the third shutter time / v from when the shutter has opened is also F11 / v. The foreground component of the eleventh pixel from the left in FIG. 15 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is F11 / v.

The foreground component of the ninth pixel from the left in frame # n + 1 in FIG. 15 corresponding to the third shutter time / v from when the shutter has opened is F10 / v, and from the left in FIG. 4th shutter time / v after shutter opened for 10th pixel
Is also F10 / v. Frame # n + in FIG.
The foreground component of the ninth pixel from the left in FIG. 1 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is F09 / v.

Since the object corresponding to the background is stationary, the background component of the tenth pixel from the left of frame # n + 1 in FIG. 15 corresponding to the first portion of the shutter time / v from when the shutter has opened is B09 / v. The background components of the eleventh pixel from the left of frame # n + 1 in FIG. 15 corresponding to the first and second portions of the shutter time / v from when the shutter has opened are B10 / v. The background components of the twelfth pixel from the left of frame # n + 1 in FIG. 15 corresponding to the first to third portions of the shutter time / v from when the shutter has opened are B11 / v.

In frame # n + 1 in FIG. 15, the tenth through twelfth pixels from the left correspond to the mixed area which is the covered background area.

FIG. 16 is a model diagram of an image in which foreground components are extracted from the pixel values shown in FIG.

FIG. 17 shows three adjacent pixels of three frames of an image obtained by capturing a foreground corresponding to an object moving to the right in the figure together with a stationary background. FIG. 10 is a model diagram in which pixel values of pixels at the same position are developed in the time direction. In FIG.
An uncovered background area is included.

In FIG. 17, it can be assumed that the object corresponding to the foreground is a rigid body and is moving at a constant speed. Since the object corresponding to the foreground is moving so as to be displayed on the right side by four pixels in the next frame, the motion amount v is 4.

For example, the shutter time / v of the leftmost pixel of frame # n-1 in FIG.
17 is F13 / v, and the second shutter time / v of the second pixel from the left in FIG.
Is also F13 / v. The third shutter time / v from when the shutter has opened for the third pixel from the left in FIG.
17 and the foreground component of the fourth pixel from the left in FIG. 17 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is F13 / v.

The foreground component of the second pixel from the left of frame # n-1 in FIG. 17 corresponding to the first shutter time / v from when the shutter has opened is F14 / v, and the third pixel from the left in FIG. 17 is F14 / v. The foreground component of the pixel of the second shutter time / v after the shutter is opened also becomes F14 / v. The foreground component of the third pixel from the left in FIG. 17 corresponding to the first portion of the shutter time / v from when the shutter has opened is F15 / v.

Since the object corresponding to the background is stationary, the leftmost pixel of frame # n-1 in FIG.
Second to fourth shutter time / v after shutter opens
Is B25 / v. Frame # n- in FIG.
The background components of the second pixel from the left, the third and fourth portions of the shutter time / v from when the shutter has opened are B26 / v
Becomes The background component of the third pixel from the left of frame # n-1 in FIG. 17 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is B27 / v.

In frame # n-1 in FIG. 17, the leftmost pixel to the third pixel belong to the mixed area which is the uncovered background area.

The fourth through twelfth pixels from the left of frame # n-1 in FIG. 17 belong to the foreground area. The foreground component of the frame is one of F13 / v to F24 / v.

The leftmost pixel to the fourth pixel from the left of frame #n in FIG. 17 belong to the background area, and the pixel value is
B25 to B28, respectively.

It can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed. Since the foreground image moves so as to be displayed four pixels to the right in the next frame, frame #n in FIG. The foreground component of the fifth pixel from the left of the first shutter time / v from when the shutter has opened is
F13 / v, and the foreground component of the sixth pixel from the left in FIG. 17 corresponding to the second shutter time / v from when the shutter has opened is also
It becomes F13 / v. The foreground component of the seventh pixel from the left in FIG. 17 corresponding to the third shutter time / v from when the shutter has opened, and the fourth shutter time since the shutter opening of the eighth pixel from the left in FIG. 17 The foreground component of / v is F13 / v.

The foreground component of the sixth pixel from the left of frame #n in FIG. 17 corresponding to the first shutter time / v from when the shutter has opened is F14 / v, and the seventh pixel from the left in FIG. 17 is F14 / v. The foreground component of the second shutter time / v after the shutter is opened also becomes F14 / v. The foreground component of the eighth pixel from the left in FIG. 17 corresponding to the first shutter time / v from when the shutter has opened is F15 / v.

Since the object corresponding to the background is stationary, the second to fourth shutter times of the fifth pixel from the left of frame #n in FIG.
The background component of / v is B29 / v. Frame # in FIG. 17
The background components of the sixth pixel from the left of n corresponding to the third and fourth shutter time / v from when the shutter has opened are B30 / v. Of the seventh pixel from the left of frame #n in FIG.
The background component of the fourth shutter time / v from when the shutter has opened is B31 / v.

In frame #n in FIG.
The seventh through seventh pixels belong to a mixed area that is an uncovered background area.

The eighth through twelfth pixels from the left of frame #n in FIG. 17 belong to the foreground area. flame
The value corresponding to the shutter time / v period in the foreground area #n is one of F13 / v to F20 / v.

The leftmost pixel to the eighth pixel from the left of frame # n + 1 in FIG. 17 belong to the background area, and have pixel values B25 to B32, respectively.

It can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed. Since the foreground image moves so as to be displayed four pixels to the right in the next frame, frame #n in FIG. Of the ninth pixel from the left of +1
The foreground component of the first shutter time / v after the shutter is opened is F13 / v, and the foreground component of the tenth pixel from the left in FIG. 17 corresponding to the second shutter time / v after the shutter is opened is also F13 / v. It becomes F13 / v. The foreground component of the eleventh pixel from the left in FIG. 17 corresponding to the third shutter time / v from when the shutter has opened, and the fourth shutter time since the shutter opening of the twelfth pixel from the left in FIG. The foreground component of / v is F13 / v.

The foreground component of the tenth pixel from the left of frame # n + 1 in FIG. 17 corresponding to the first shutter time / v from when the shutter has opened is F14 / v, and the eleventh pixel from the left in FIG. The foreground component of the pixel of the second shutter time / v after the shutter is opened also becomes F14 / v. The foreground component of the twelfth pixel from the left in FIG. 17 corresponding to the first shutter time / v from when the shutter has opened is F15 / v.

Since the object corresponding to the background is stationary, the background of the ninth pixel from the left of frame # n + 1 in FIG. 17 corresponding to the second through fourth portions of the shutter time / v from when the shutter has opened. Is B33 / v. The 10th pixel from the left of frame # n + 1 in FIG.
The background component of the 4th and 4th shutter time / v is B3
4 / v. The background component of the eleventh pixel from the left of frame # n + 1 in FIG. 17 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is B35 / v.

In frame # n + 1 in FIG. 17, the ninth to eleventh pixels from the left belong to the mixed area which is the uncovered background area.

The twelfth pixel from the left of frame # n + 1 in FIG. 17 belongs to the foreground area. The foreground components of the shutter time / v in the foreground area of frame # n + 1 are F13 / v to F13 / v
16 / v.

FIG. 18 is a model diagram of an image in which foreground components are extracted from the pixel values shown in FIG.

Referring back to FIG. 2, the area specifying unit 103 uses the pixel values of a plurality of frames to set a flag for each pixel indicating that the pixel belongs to the foreground area, background area, covered background area, or uncovered background area. Is supplied to the mixture ratio calculation unit 104 and the motion blur adjustment unit 106 as region information.

The mixture ratio calculation unit 104 calculates the mixture ratio α for each pixel of the pixels included in the mixed region based on the pixel values of a plurality of frames and the region information, and uses the calculated mixture ratio α for the foreground / background separation. To the unit 105.

The foreground / background separation unit 105 extracts a foreground component image consisting of only foreground components based on the pixel values of a plurality of frames, area information, and the mixture ratio α, and supplies the extracted foreground component image to the motion blur adjustment unit 106. .

The motion-blur adjusting unit 106 controls the foreground component image supplied from the foreground / background separating unit 105 and the motion detecting unit 102.
Motion vector supplied from the
The amount of motion blur included in the foreground component image is adjusted on the basis of the area information supplied from, and a foreground component image in which the amount of motion blur is adjusted is output.

The processing for adjusting the amount of motion blur by the signal processing unit will be described with reference to the flowchart in FIG. In step S11, based on the input image, the region identification unit 103 determines, for each pixel of the input image, region information indicating whether the pixel belongs to any of a foreground region, a background region, a covered background region, or an uncovered background region. Execute the process for specifying the area to be generated. The details of the area specifying process will be described later. The region specifying unit 103 supplies the generated region information to the mixture ratio calculation unit 104.

In step S11, based on the input image, the area specifying unit 103 determines, for each pixel of the input image, a foreground area, a background area, or a mixed area (covered background area or uncovered background area). May be generated. In this case, the foreground / background separation unit 105 and the motion blur adjustment unit 106 determine whether the mixed area is a covered background area or an uncovered background area based on the direction of the motion vector. For example, when the foreground area, the mixed area, and the background area are sequentially arranged in correspondence with the direction of the motion vector, the mixed area is determined as a covered background area, and the background area is determined according to the direction of the motion vector. When the area, the mixed area, and the foreground area are arranged in this order, the mixed area is determined to be an uncovered background area.

In step S12, the mixture ratio calculation unit 1
A step 04 calculates a mixture ratio α for each pixel included in the mixed region based on the input image and the region information. Details of the process of calculating the mixture ratio will be described later. The mixture ratio calculator 104 supplies the calculated mixture ratio α to the foreground / background separator 105.

In step S13, the foreground / background separation unit 105 extracts a foreground component from the input image based on the area information and the mixture ratio α, and supplies the foreground component image to the motion blur adjustment unit 106.

In step S14, the motion-blur adjusting unit 106 determines which of uncovered background area, foreground area, and covered background area is a continuous pixel lined up in the motion direction based on the motion vector and the area information. A processing unit indicating a position of an image belonging to the genus is generated, and the amount of motion blur included in the foreground component corresponding to the processing unit is adjusted. Details of the process of adjusting the amount of motion blur will be described later.

In step S15, the signal processing section
It is determined whether or not processing has been completed for the entire screen, and when it is determined that processing has not been completed for the entire screen,
Proceeding to step S14, the process of adjusting the amount of motion blur for the foreground component corresponding to the processing unit is repeated.

If it is determined in step S15 that the processing has been completed for the entire screen, the processing ends.

As described above, the signal processing unit can separate the foreground from the background and adjust the amount of motion blur included in the foreground. That is, the signal processing unit can adjust the amount of motion blur included in the sample data that is the pixel value of the foreground pixel.

Hereinafter, the area specifying unit 103 and the mixture ratio calculating unit 1
04, foreground / background separation unit 105, and motion blur adjustment unit 1
06 will be described.

FIG. 20 is a block diagram showing the structure of the area specifying unit 103. Area specifying unit 10 whose configuration is shown in FIG.
No. 3 does not use a motion vector. Frame memory 20
1 stores an input image in frame units. When the processing target is the frame #n, the frame memory 201 stores the frame # n-, which is two frames before the frame #n.
2. Frame # n-, which is the frame immediately before frame #n
1, frame #n, frame # n + 1 which is one frame after frame #n, and frame # n + 2 which is two frames after frame #n.

The still / movement determining unit 202-1 determines the pixel value of the pixel of frame # n + 2 at the same position as the pixel on the image that is the target of specifying the area of frame #n and the pixel value of frame #n. The pixel value of the pixel of frame # n + 1 at the same position as the position of the pixel to be specified in the image is stored in the frame memory 2
First, the absolute value of the difference between the read pixel values is calculated. The static / movement determining unit 202-1 determines whether the absolute value of the difference between the pixel value of the frame # n + 2 and the pixel value of the frame # n + 1 is greater than a preset threshold Th, When it is determined that the absolute value of the difference is larger than the threshold Th, a still / moving determination indicating a motion is supplied to the area determination unit 203-1. Frame #n
When it is determined that the absolute value of the difference between the pixel value of the pixel of +2 and the pixel value of the pixel of frame # n + 1 is equal to or smaller than the threshold Th, the static / movement determining unit 202-1 determines whether or not the static Judgment is made to the area judgment unit 2
03-1.

The still / movement determining unit 202-2 determines the pixel value of the pixel of frame # n + 1 at the same position as the position of the pixel to be specified for the region of frame #n, and the pixel value of frame #n. The pixel value of the target pixel is read from the frame memory 201, and the absolute value of the difference between the pixel values is calculated. The static / moving determining unit 202-2 determines whether the absolute value of the difference between the pixel value of the frame # n + 1 and the pixel value of the frame #n is larger than a preset threshold Th. If the absolute value of the difference is determined to be larger than the threshold value Th, a still / moving determination indicating a motion is supplied to the region determination unit 203-1 and the region determination unit 203-2. When it is determined that the absolute value of the difference between the pixel value of the pixel of frame # n + 1 and the pixel value of the pixel of frame #n is equal to or smaller than the threshold Th, the still / moving determination unit 202-2 indicates that the camera is still. The static / movement determination is supplied to the region determination unit 203-1 and the region determination unit 203-2.

The still / moving judgment section 202-3 is located at the same position as the pixel value of the pixel which is the target of specifying the region of the frame #n and the position on the image of the pixel which is the target of specifying the region of the frame #n. The pixel value of the pixel of frame # n-1 is stored in frame memory 2
01, the absolute value of the difference between the pixel values is calculated.
The static / moving determining unit 202-3 determines whether the absolute value of the difference between the pixel value of the frame #n and the pixel value of the frame # n-1 is greater than a preset threshold Th. If the absolute value of the difference is determined to be larger than the threshold Th, the static / moving determination indicating the motion is performed by the region determination unit 203-2 and the region determination unit 203.
-3. Pixel value of pixel of frame #n and frame
When it is determined that the absolute value of the difference between the pixel value of the pixel # n-1 and the pixel value is equal to or smaller than the threshold Th, the still / moving determination unit 202-3 performs the still / moving determination indicating stillness by the region determining unit 203-2. And to the area determination unit 203-3.

The still / movement determining unit 202-4 determines the pixel value of the pixel of frame # n-1 at the same position as the pixel on the image that is the target of specifying the region of frame #n, and the pixel value of frame #n. The pixel value of the pixel of frame # n-2 at the same position as the position of the pixel which is the target of region identification on the image is stored in the frame memory 2
01, the absolute value of the difference between the pixel values is calculated.
The still / moving determination unit 202-4 determines whether the absolute value of the difference between the pixel value of the frame # n-1 and the pixel value of the frame # n-2 is greater than a preset threshold Th, When it is determined that the absolute value of the difference between the pixel values is greater than the threshold Th, a still / moving determination indicating motion is supplied to the region determination unit 203-3. Frame #n
When it is determined that the absolute value of the difference between the pixel value of the pixel −1 and the pixel value of the pixel of frame # n−2 is equal to or smaller than the threshold Th, the static / moving determining unit 202-4 determines that the static The motion determination is supplied to the area determination unit 203-3.

The area determining section 203-1 is composed of the static / moving determining section 20.
When the still / movement determination supplied from 2-1 indicates stillness and the still / movement determination supplied from the still / movement determination unit 202-2 indicates movement, the pixel that is the target of region identification in frame #n Is determined to belong to the uncovered background area, and the uncovered background area determination flag corresponding to the pixel whose area is determined is set to “1” indicating that the pixel belongs to the uncovered background area.

The area judgment section 203-1 is composed of the static / movement judgment section 20.
When the still / movement determination supplied from 2-1 indicates a motion or the still / movement determination supplied from the still / movement determination unit 202-2 indicates stillness, the region is a target for area identification in frame #n. It is determined that the pixel does not belong to the uncovered background area, and “0” indicating that the pixel does not belong to the uncovered background area is set in the uncovered background area determination flag corresponding to the pixel whose area is determined.
Set.

The area determination unit 203-1 determines "
The uncovered background area determination flag in which 1 ”or“ 0 ”is set is supplied to the determination flag storage frame memory 204.

[0202] The area determining section 203-2 includes the static / moving determining section 20.
When the still / movement determination supplied from 2-2 indicates stillness and the still / movement determination supplied from still / movement determination unit 202-3 indicates stillness, the pixel which is the target of region identification in frame #n Is determined to belong to the still area, and “1” indicating that the pixel belongs to the still area is set in the still area determination flag corresponding to the pixel whose area is determined.

[0203] The area determining section 203-2 is provided with the static / moving determining section 20.
When the static / movement determination supplied from 2-2 indicates a motion, or when the static / movement determination supplied from the static / movement determination unit 202-3 indicates a motion, the region is a target for area identification in frame #n. It is determined that the pixel does not belong to the still area, and “0” indicating that the pixel does not belong to the still area is set in the still area determination flag corresponding to the pixel whose area is determined.

The area determining unit 203-2 outputs "
The still area determination flag in which “1” or “0” is set is supplied to the determination flag storage frame memory 204.

The area determining section 203-2 is provided with the static / moving determining section 20.
When the still / movement determination supplied from 2-2 indicates a movement and the still / movement determination supplied from the still / movement determination unit 202-3 indicates a movement, the pixel which is the target of the region identification in the frame #n Is determined to belong to the motion area, and “1” indicating that the pixel belongs to the motion area is set in the motion area determination flag corresponding to the pixel whose area is determined.

The area determining section 203-2 is provided with the static / moving determining section 20.
When the static / movement determination supplied from 2-2 indicates stillness, or when the static / movement determination supplied from static / movement determination unit 202-3 indicates stillness, it is a target for area identification in frame #n. It is determined that the pixel does not belong to the motion area, and “0” indicating that the pixel does not belong to the motion area is set in the motion area determination flag corresponding to the pixel whose area is determined.

[0207] The area determination unit 203-2 determines "
The motion area determination flag in which 1 ”or“ 0 ”is set is supplied to the determination flag storage frame memory 204.

The area determining section 203-3 is provided with the static / moving determining section 20.
When the still / moving judgment supplied from 2-3 indicates a motion and the still / moving judgment supplied from the still / moving judging section 202-4 indicates still, a pixel which is an area identification target in the frame #n Is determined to belong to the covered background area, and “1” indicating that the pixel belongs to the covered background area is set in the covered background area determination flag corresponding to the pixel whose area is determined.

[0209] The area determination section 203-3 is provided with the static / movement determination section 20.
When the still / movement determination supplied from 2-3 indicates stillness or when the still / movement determination supplied from the still / movement determination unit 202-4 indicates motion, the region is a target for area identification in frame #n. It is determined that the pixel does not belong to the covered background area, and “0” indicating that the pixel does not belong to the covered background area is set in a covered background area determination flag corresponding to the pixel whose area is determined.

[0210] The area determination unit 203-3 determines "
The covered background area determination flag in which 1 ”or“ 0 ”is set is stored in the determination flag storage frame memory 20.
4

The judgment flag storage frame memory 204 is
Uncovered background area determination flag supplied from area determination section 203-1, still area determination flag supplied from area determination section 203-2, area determination section 203-
2 and a covered background area determination flag supplied from the area determination unit 203-3.

The judgment flag storage frame memory 204 is
The stored uncovered background area determination flag, still area determination flag, moving area determination flag, and covered background area determination flag are stored in the combining unit 2.
05. Based on the uncovered background area determination flag, the still area determination flag, the motion area determination flag, and the covered background area determination flag supplied from the determination flag storage frame memory 204, the combining unit 205 Area information indicating that the area belongs to any of the covered background area, the still area, the moving area, and the covered background area is generated and supplied to the determination flag storage frame memory 206.

The judgment flag storage frame memory 206
The area information supplied from the synthesizing unit 205 is stored, and the stored area information is output.

Next, an example of the processing of the area specifying unit 103 will be described with reference to FIG.
This will be described with reference to FIGS.

When the object corresponding to the foreground is moving, the position on the screen of the image corresponding to the object changes for each frame. As shown in FIG. 21, in frame #n, the image corresponding to the object located at the position indicated by Yn (x, y) is Yn + 1 (x, y) in frame # n + 1 which is the next frame. located at y).

FIG. 24 shows a model diagram in which the pixel values of the pixels arranged in one row adjacent to the motion direction of the image corresponding to the foreground object are developed in the time direction. For example, when the motion direction of the image corresponding to the foreground object is horizontal with respect to the screen, the model diagram in FIG. 22 shows a model in which pixel values of adjacent pixels on one line are developed in the time direction.

In FIG. 22, the line in frame #n is the same as the line in frame # n + 1.

In frame #n, the foreground components corresponding to the objects included in the second through thirteenth pixels from the left are the sixth through seventeenth pixels from the left in frame # n + 1. included.

In the frame #n, the pixels belonging to the covered background area are the eleventh to thirteenth pixels from the left, and the pixels belonging to the uncovered background area are the second to fourth pixels from the left. .
In frame # n + 1, the pixels belonging to the covered background area are the fifteenth through seventeenth pixels from the left, and the pixels belonging to the uncovered background area are the sixth through eighth pixels from the left.

In the example shown in FIG. 22, since the foreground component included in frame #n has moved by four pixels in frame # n + 1, the motion amount v is 4. The number of virtual divisions is
It corresponds to the movement amount v and is 4.

Next, changes in pixel values of pixels belonging to the mixed area before and after the frame of interest will be described.

In the frame #n shown in FIG. 23 where the background is stationary and the foreground motion amount v is 4, the pixels belonging to the covered background area are the 15th to 17th pixels from the left.
The pixel of the th. Since the motion amount v is 4, the fifteenth through seventeenth pixels from the left in the immediately preceding frame # n-1 include only background components and belong to the background area. In the immediately preceding frame # n-2, the fifteenth through seventeenth pixels from the left include only background components and belong to the background area.

Here, since the object corresponding to the background is stationary, the pixel value of the fifteenth pixel from the left of frame # n-1 is the pixel value of the fifteenth pixel from the left of frame # n-2. Does not change from. Similarly, 1 from the left of frame # n-1
The pixel value of the sixth pixel does not change from the pixel value of the sixteenth pixel from the left of frame # n-2, and the pixel value of the seventeenth pixel from the left of frame # n-1 is frame # n- 17 from the left of 2
It does not change from the pixel value of the th pixel.

That is, a frame corresponding to a pixel belonging to the covered background area in frame #n
The pixels of # n-1 and frame # n-2 are composed of only background components, and the pixel values do not change. Therefore, the absolute value of the difference is almost zero. Therefore, frame # n-1 and frame # n-1 corresponding to the pixels belonging to the mixed area in frame #n
The still / movement determination for the pixel # n-2 is performed by the still / movement determination unit 202-4.
Is determined to be stationary.

Since the pixels belonging to the covered background area in frame #n include foreground components, the pixel values are different from those of frame # n-1 consisting only of background components. Therefore, the still / movement determination for the pixel belonging to the mixed area in frame #n and the corresponding pixel in frame # n-1 is determined to be a movement by the still / movement determination unit 202-3.

As described above, the region determination unit 203-3 is supplied with the result of the static / moving determination indicating the motion from the static / moving determining unit 202-3, and receives the static / moving determination indicating the stillness from the static / moving determining unit 202-4. When the result is supplied, it is determined that the corresponding pixel belongs to the covered background area.

In the frame #n shown in FIG. 24 where the background is stationary and the foreground motion amount v is 4, the pixels included in the uncovered background area are the second to fourth pixels from the left. Since the motion amount v is 4, in the next frame # n + 1, the second to fourth pixels from the left include only background components and belong to the background area. Further, in the next frame # n + 2, the second to fourth pixels from the left include only the background component and belong to the background area.

Here, since the object corresponding to the background is stationary, the pixel value of the second pixel from the left of frame # n + 2 is the pixel value of the second pixel from the left of frame # n + 1. Does not change from. Similarly, the pixel value of the third pixel from the left of frame # n + 2 does not change from the pixel value of the third pixel from the left of frame # n + 1, and the pixel value of the third pixel from the left of frame # n + 2 does not change. Does not change from the pixel value of the fourth pixel from the left of frame # n + 1.

That is, the pixels of frame # n + 1 and frame # n + 2 corresponding to the pixels belonging to the uncovered background area in frame #n are composed of only the background components and the pixel values do not change. The absolute value of the difference is almost zero. Therefore, the still / movement determination for the pixels of frame # n + 1 and frame # n + 2 corresponding to the pixels belonging to the mixed area in frame #n is performed by the still / movement determination unit 20
According to 2-1, it is determined to be stationary.

Since the pixels belonging to the uncovered background area in frame #n include the foreground components,
A case where only the background component in frame # n + 1 is included,
Pixel values are different. Therefore, the still / movement determination for the pixels belonging to the mixed area in frame #n and the corresponding pixels in frame # n + 1 is determined to be a motion by the still / movement determination unit 202-2.

As described above, the area determination unit 203-1 is supplied with the result of the static / movement determination indicating the motion from the static / movement determination unit 202-2, and receives the static / dynamic determination indicating the stillness from the static / movement determination unit 202-1. When the result is supplied, it is determined that the corresponding pixel belongs to the uncovered background area.

FIG. 25 is a diagram showing the determination conditions of the area specifying unit 103 in frame #n. The pixel of frame # n-2 at the same position as the position of the pixel to be determined in frame #n on the image and the pixel of frame #n in the same position as the position of the pixel to be determined in frame #n The pixel of a certain frame # n-1 is determined to be still, and the pixel of the frame # n-1 and the pixel of the frame #n at the same position as the position of the pixel to be determined of the frame #n on the image Is determined to be a motion, the area specifying unit 103 determines that the pixel to be determined in the frame #n belongs to the covered background area.

The pixel of frame # n-1 and the pixel of frame #n at the same position as the position of the pixel to be determined for frame #n on the image are determined to be stationary, and the pixel of frame #n is determined to be still. When the pixel of the frame # n + 1 located at the same position as the position of the pixel to be determined in the frame #n on the image is determined to be still, the region identifying unit 103 determines the frame #n Is determined to belong to the still area.

The pixel of frame # n-1 and the pixel of frame #n at the same position as the position of the pixel to be determined for frame #n on the image are determined to be moving, and the pixel of frame #n And the pixel of frame # n + 1 located at the same position as the position of the pixel to be determined in frame #n on the image, is determined to be moving. Is determined to belong to the motion area.

The pixel of frame #n and the pixel of frame # n + 1 located at the same position on the image as the pixel to be determined of frame #n are determined to be moving, and the determination of frame #n is performed. The pixel of frame # n + 1 located at the same position as the position of the target pixel on the image, and the frame # n + located at the same position on the image of the pixel targeted for determination of frame #n When the two pixels are determined to be still, the region specifying unit 103 determines that the pixel to be determined in the frame #n belongs to the uncovered background region.

FIG. 26 is a diagram showing an example of a result of specifying an area by the area specifying unit 103. In FIG. In FIG. 26A, pixels determined to belong to the covered background area are displayed in white. In FIG. 26B, pixels determined to belong to the uncovered background area are displayed in white.

In FIG. 26C, the pixels determined to belong to the motion area are displayed in white. FIG.
In (D), the pixels determined to belong to the still area are displayed in white.

FIG. 27 is a diagram showing, as an image, area information indicating a mixed area among the area information output from the frame memory 206 for storing the determination flag. In FIG. 27, the pixels determined to belong to the covered background area or the uncovered background area, that is, the pixels determined to belong to the mixed area, are displayed in white.
The area information indicating the mixed area output from the determination flag storage frame memory 206 indicates the mixed area and the part having the texture surrounded by the non-textured part in the foreground area.

Next, the area specifying processing of the area specifying unit 103 will be described with reference to the flowchart in FIG. In step S201, the frame memory 201 acquires images of frames # n-2 to # n + 2 including the frame #n to be determined.

In step S202, the static / movement determining unit 2
02-3, the pixel of frame # n-1 and the pixel at the same position of frame #n determine whether or not it is still, and if it is determined that it is still, proceed to step S203, 2
Determines whether the pixel of frame #n and the pixel at the same position of frame # n + 1 are still.

In step S203, when it is determined that the pixel of frame #n and the pixel at the same position of frame # n + 1 are still, the process proceeds to step S204, and the region determination unit 20
Step 3-2 sets “1” indicating that the pixel belongs to the still area in the still area determination flag corresponding to the pixel whose area is determined. The region determination unit 203-2 supplies the still region determination flag to the determination flag storage frame memory 204, and the procedure proceeds to step S205.

At step S202, frame # n-1
Pixel and the pixel at the same position of frame #n are determined to be moving, or in step S203, the pixel of frame #n and the pixel at the same position of frame # n + 1 are determined to be moving. In this case, since the pixel of frame #n does not belong to the still area, the process of step S204 is skipped, and the procedure proceeds to step S205.

In step S205, static / movement determination section 2
02-3, the pixel of frame # n-1 and the pixel at the same position of frame #n are used to determine whether or not a motion is detected. 2
Determines whether or not there is a motion between the pixel of frame #n and the pixel at the same position of frame # n + 1.

If it is determined in step S206 that the pixel of the frame #n and the pixel at the same position of the frame # n + 1 are determined to be moving, the process proceeds to step S207, and the area determination unit 20 determines
3-2 sets “1” indicating that the pixel belongs to the motion area in the motion area determination flag corresponding to the pixel whose area is determined. The area determination unit 203-2 supplies the motion area determination flag to the determination flag storage frame memory 204, and the procedure proceeds to step S208.

In the step S205, the frame # n-1
Pixel and the pixel at the same position of frame #n are determined to be still, or, at step S206, the pixel of frame #n and the pixel at the same position of frame # n + 1 are determined to be still. In this case, since the pixel of frame #n does not belong to the motion area, the process of step S207 is skipped, and the procedure proceeds to step S208.

[0246] In step S208, the static / motion judging section 2
02-4, the pixel of frame # n-2 and the pixel at the same position of frame # n-1 are used to determine whether or not they are stationary. 202-
No. 3 determines whether or not the pixel of frame # n-1 and the pixel at the same position of frame #n are moving.

[0247] In step S209, frame # n-1
If it is determined that there is a motion between the pixel of the frame #n and the pixel at the same position in the frame #n, the process proceeds to step S210, and the region determination unit 20
Step 3-3 sets “1” indicating that the pixel belongs to the covered background area in the covered background area determination flag corresponding to the pixel whose area is determined. The area determination unit 203-3 supplies the covered background area determination flag to the determination flag storage frame memory 204, and the procedure proceeds to step S211.

At step S208, frame # n-2
Is determined to be a motion between the pixel of the frame # n-1 and the pixel of the same position of the frame # n-1, or in step S209, the pixel of the frame # n-1 and the pixel of the same position of the frame #n are
If it is determined that the image is stationary, the pixel of frame #n does not belong to the covered background area.
0 is skipped, and the procedure proceeds to step S211.
Proceed to.

In the step S211, the static / movement judging section 2
In step 02-2, it is determined whether or not the pixel of the frame #n and the pixel at the same position of the frame # n + 1 are moving. 1
Determines whether the pixel at frame # n + 1 and the pixel at the same position in frame # n + 2 are still.

In step S212, frame # n + 1
If it is determined that the pixel of the frame # n + 2 and the pixel at the same position in the frame # n + 2 are still, the process proceeds to step S213, and the region determination unit 2
03-1 sets “1” indicating that the pixel belongs to the uncovered background area in the uncovered background area determination flag corresponding to the pixel whose area is determined. The area determination unit 203-1 supplies the uncovered background area determination flag to the determination flag storage frame memory 204, and the procedure proceeds to step S214.

In step S211, when it is determined that the pixel of frame #n and the pixel at the same position of frame # n + 1 are stationary, or in step S212, the pixel of frame # n + 1 and the frame #n With the pixel at the same position of +2,
If it is determined that the pixel is in motion, the pixel of frame #n does not belong to the uncovered background area,
Step 213 is skipped, and the procedure proceeds to step S2
Proceed to 14.

In step S214, region specifying unit 1
03 judges whether or not the area has been specified for all the pixels of frame #n. If it is determined that the area has not been specified for all the pixels of frame #n, the procedure returns to step S202. The region specifying process is repeated for other pixels.

If it is determined in step S214 that the region has been specified for all the pixels of frame #n,
Proceeding to step S215, the combining unit 205 generates area information indicating the mixed area based on the uncovered background area determination flag and the covered background area determination flag stored in the determination flag storage frame memory 204, Furthermore, area information indicating that each pixel belongs to any of the uncovered background area, the still area, the moving area, and the covered background area is generated, and the generated area information is set in the determination flag storage frame memory 206. Then, the process ends.

As described above, the area specifying unit 103 calculates the motion area,
Region information indicating belonging to a stationary region, an uncovered background region, or a covered background region can be generated.

The area specifying unit 103 generates area information corresponding to the mixed area by applying a logical sum to the area information corresponding to the uncovered background area and the covered background area, and includes the area information in the frame. For each of the pixels that have been set, area information including a flag indicating that the pixel belongs to a moving area, a still area, or a mixed area may be generated.

When the object corresponding to the foreground has a texture, the area specifying unit 103 can specify the moving area more accurately.

The area specifying unit 103 can output area information indicating a moving area as area information indicating a foreground area and area information indicating a still area as area information indicating a background area.

Although the description has been made assuming that the object corresponding to the background is stationary, the above-described processing for specifying the area can be applied even if the image corresponding to the background area includes motion. For example, when the image corresponding to the background area is moving uniformly, the area specifying unit 103 shifts the entire image in accordance with this movement, and performs the same processing as when the object corresponding to the background is stationary. I do. Also,
When the image corresponding to the background area includes a different motion for each local area, the area specifying unit 103 selects a pixel corresponding to the motion and executes the above-described processing.

FIG. 29 is a block diagram showing the structure of the area specifying unit 103. The area specifying unit 103 shown in FIG.
Do not use motion vectors. The background image generation unit 301
A background image corresponding to the input image is generated, and the generated background image is supplied to the binary object image extraction unit 302. The background image generation unit 301 generates, for example, a background image by extracting an image object corresponding to a background object included in the input image.

FIG. 30 shows an example of a model diagram in which the pixel values of the pixels arranged in one row adjacent to the motion direction of the image corresponding to the foreground object are developed in the time direction. For example, when the motion direction of the image corresponding to the foreground object is horizontal with respect to the screen, the model diagram in FIG. 30 shows a model in which pixel values of adjacent pixels on one line are developed in the time direction.

In FIG. 30, the lines in frame #n are the same as the lines in frame # n-1 and frame # n + 1.

In frame #n, the foreground components corresponding to the objects included in the sixth through seventeenth pixels from the left are the second through thirteenth pixels from the left in frame # n-1. Included in the 10th to 21st pixels from the left in frame # n + 1.

In frame # n−1, the pixels belonging to the covered background area are the eleventh to thirteenth pixels from the left.
The second pixel, which belongs to the uncovered background area, is the second to fourth pixels from the left. In frame #n, the pixels belonging to the covered background area are the fifteenth through seventeenth pixels from the left, and the pixels belonging to the uncovered background area are the sixth through eighth pixels from the left. flame
In # n + 1, the pixels belonging to the covered background area are the 19th to 21st pixels from the left, and the pixels belonging to the uncovered background area are the 10th to 12th pixels from the left.

In frame # n-1, the pixels belonging to the background area are the first pixel from the left and the fourteenth through twenty-first pixels from the left. In frame #n, the pixels belonging to the background area are the first through fifth pixels from the left and the eighteenth through twenty-first pixels from the left. In frame # n + 1, the pixels belonging to the background area are the first through ninth pixels from the left.

FIG. 30 generated by the background image generation unit 301
FIG. 31 shows an example of a background image corresponding to the example of FIG. The background image is composed of pixels corresponding to the background object,
The image component corresponding to the foreground object is not included.

The binary object image extracting section 302 generates a binary object image based on the correlation between the background image and the input image, and supplies the generated binary object image to the time change detecting section 303.

FIG. 32 shows the binary object image extracting unit 3
FIG. 2 is a block diagram showing a configuration of the second embodiment. Correlation value calculator 32
1 calculates a correlation between the background image and the input image supplied from the background image generation unit 301, generates a correlation value, and supplies the generated correlation value to the threshold processing unit 322.

The correlation value calculating section 321 is, for example, shown in FIG.
As shown in (A), a block in a 3 × 3 background image centered on X ₄ and, as shown in FIG. 33B, a center in Y ₄ corresponding to a block in the background image. 3 × 3
Applying equation (4) to the block in the input image of
Calculate the correlation value corresponding to ₄ .

[0269]

(Equation 2)

[0270]

(Equation 3)

[0271]

(Equation 4)

The correlation value calculation section 321 supplies the correlation value calculated for each pixel as described above to the threshold value processing section 322.

Further, the correlation value calculation unit 321 may, for example, as shown in FIG. 34A, block a 3 × 3 background image centered on X ₄ and a block shown in FIG. To
3 centered on Y ₄ corresponding to the block in the background image
Equation (7) may be applied to the blocks in the × 3 input image to calculate the sum of absolute differences corresponding to Y ₄ .

[0274]

(Equation 5)

The correlation value calculation unit 321 uses the sum of absolute differences calculated in this way as a correlation value,
Feed to 2.

The threshold value processing section 322 compares the pixel value of the correlation image with the threshold value th0, and sets 1 to the pixel value of the binary object image if the correlation value is equal to or smaller than the threshold value th0. If the correlation value is greater than the threshold value th0, the pixel value of the binary object image is set to 0, and a binary object image in which 0 or 1 is set as the pixel value is output. The threshold value processing unit 322 may store the threshold value th0 in advance, or the threshold value th0 input from the outside may be used.
0 may be used.

FIG. 35 is a diagram showing an example of a binary object image corresponding to the model of the input image shown in FIG.
In the binary object image, 0 is set to the pixel value of a pixel having a high correlation with the background image.

FIG. 36 is a block diagram showing a configuration of the time change detecting section 303. When determining the area for the pixel of frame #n, the frame memory 341 receives the frame # n- supplied from the binary object image extraction unit 302.
The binary object images of 1, frame #n, and frame # n + 1 are stored.

[0279] The area determination section 342
Based on the binary object images of frame # n-1, frame #n, and frame # n + 1 stored in
A region is determined for each pixel of #n, region information is generated, and the generated region information is output.

FIG. 37 is a view for explaining the judgment by the area judgment section 342. When the pixel of interest in the binary object image of frame #n is 0, the region determination unit 342
It is determined that the pixel of interest in frame #n belongs to the background area.

The pixel of interest of the binary object image of frame #n is 1, the corresponding pixel of the binary object image of frame # n-1 is 1, and the pixel of interest of frame # n + 1 is 2.
When the corresponding pixel of the value object image is 1, the area determination unit 342 determines that the pixel of interest in frame #n belongs to the foreground area.

When the pixel of interest in the binary object image in frame #n is 1 and the corresponding pixel in the binary object image in frame # n-1 is 0, the area determination unit 3
42 determines that the pixel of interest in frame #n belongs to the covered background area.

When the pixel of interest of the binary object image of frame #n is 1 and the corresponding pixel of the binary object image of frame # n + 1 is 0, the area determination unit 3
42 determines that the pixel of interest in frame #n belongs to the uncovered background area.

FIG. 38 is a diagram showing an example in which the time change detection unit 303 has determined the binary object image corresponding to the model of the input image shown in FIG. Since the pixel corresponding to frame #n of the binary object image is 0, the time change detection unit 303 determines that the first to fifth pixels from the left of frame #n
It is determined that the th pixel belongs to the background area.

Since the pixel of frame #n of the binary object image is 1 and the corresponding pixel of frame # n + 1 is 0, the time change detecting unit 303 uncovers the sixth through ninth pixels from the left. It is determined that it belongs to the background area.

The time change detection unit 303 determines that the pixel of frame #n of the binary object image is 1, the corresponding pixel of frame # n-1 is 1, and the corresponding pixel of frame # n + 1 is 1. Therefore, it is determined that the tenth to thirteenth pixels from the left belong to the foreground area.

Since the pixel of frame #n of the binary object image is 1 and the corresponding pixel of frame # n-1 is 0, the time change detection unit 303 covers the 14th to 17th pixels from the left. It is determined that it belongs to the ground area.

Since the pixel corresponding to frame #n of the binary object image is 0, the time change detecting section 303 determines that the 18th to 21st pixels from the left belong to the background area.

Next, referring to the flowchart in FIG. 39, the area specifying process of the area determination unit 103 will be described. In step S301, the background image generation unit 301 of the area determination unit 103 generates a background image by extracting, for example, an image object corresponding to a background object included in the input image based on the input image, and generates the background image. Image 2
It is supplied to the value object image extraction unit 302.

In step S302, the binary object image extracting unit 302 calculates a correlation value between the input image and the background image supplied from the background image generating unit 301 by the calculation described with reference to FIG. 33, for example. . In step S303, the binary object image extraction unit 302
Calculates a binary object image from the correlation value and the threshold value th0, for example, by comparing the correlation value with a threshold value th0.

[0291] In step S304, the time change detecting unit 303 executes the area determination processing, and the processing ends.

With reference to the flowchart in FIG. 40, the details of the area determination processing corresponding to step S304 will be described. In step S321, the time change detection unit 303
Is determined in the frame #n stored in the frame memory 341 as to whether or not the pixel of interest is 0. In the frame #n, the pixel of interest is
When it is determined to be 0, the process proceeds to step S322,
The target pixel of frame #n is set to belong to the background area, and the process ends.

[0293] If it is determined in step S321 that the pixel of interest is 1 in frame #n, the flow advances to step S323, and the area determination unit 342 of the time change detection unit 303 stores the frame in the frame memory 341. In frame #n, the pixel of interest is 1, and
In the frame # n-1, it is determined whether or not the corresponding pixel is 0. In the frame #n, the pixel of interest is 1, and in the frame # n-1, the corresponding pixel is 0. If it is determined that the target pixel of the frame #n belongs to the covered background area, the process proceeds to step S324, and the process ends.

In step S323, in the frame #n, the pixel of interest is 0, or the frame #n
When it is determined that the corresponding pixel is 1 in n−1, the process proceeds to step S325, and the area determination unit 342 of the time change detection unit 303 focuses on the frame #n stored in the frame memory 341. Pixel is 1,
Also, in frame # n + 1, it is determined whether the corresponding pixel is 0 or not, and in frame #n, the pixel of interest is
If it is 1, and it is determined that the corresponding pixel is 0 in frame # n + 1, the process proceeds to step S326, where it is set that the target pixel of frame #n belongs to the uncovered background area. , The process ends.

[0295] In step S325, in frame #n, the pixel of interest is 0, or
When it is determined that the corresponding pixel is 1 at n + 1, the process proceeds to step S327, and the region determination unit 342 of the time change detection unit 303 sets the target pixel of the frame #n as the foreground region, The process ends.

As described above, based on the correlation value between the input image and the corresponding background image, the area specifying unit 103 determines whether the pixels of the input image are the foreground area, the background area, the covered background area, and the uncovered background area. It is possible to specify which of the ground areas belongs to, and generate area information corresponding to the specified result.

FIG. 41 is a block diagram showing another configuration of area specifying section 103. Area specifying unit 103 shown in FIG.
Uses the motion vector supplied from the motion detection unit 102 and its position information. The same parts as those shown in FIG. 29 are denoted by the same reference numerals, and description thereof will be omitted.

[0298] The robust section 361 outputs the 2 frames of N frames supplied from the binary object image extracting section 302.
A robust binary object image is generated based on the value object image and output to the time change detection unit 303.

FIG. 42 is a block diagram illustrating the configuration of robust unit 361. Referring to FIG. The motion compensating unit 381 compensates for the motion of the binary object images of the N frames based on the motion vector and the positional information supplied from the motion detecting unit 102, and generates the motion-compensated binary object image. Output to switch 382.

The motion compensation of the motion compensator 381 will be described with reference to the examples of FIGS. 43 and 44. For example, when determining the region of frame #n, when the binary object images of frame # n-1, frame #n, and frame # n + 1 shown in FIG. 43 are input, the motion compensation unit 381 ,
Based on the motion vector supplied from the motion detection unit 102,
As shown in the example of FIG. 44, the binary object image of frame # n-1 and the binary object image of frame # n + 1 are motion-compensated and the motion-compensated binary object image is supplied to the switch 382. I do.

The switch 382 stores the motion-compensated binary object image of the first frame in the frame memory 3.
83-1, and outputs the motion-compensated binary object image of the second frame to the frame memory 383-2. Similarly, the switches 382 are third to N-1
The motion-compensated binary object image of the Nth frame is output to one of the frame memories 383-3 to 383- (N-1), and the motion-compensated binary object image of the Nth frame is output. To the frame memory 383-N.

[0302] The frame memory 383-1 stores the motion-compensated binary object image of the first frame, and outputs the stored binary object image to the weighting unit 384-1. The frame memory 383-2 is
The motion-compensated binary object image of the second frame is stored, and the stored binary object image is output to the weighting unit 384-2.

Similarly, each of the frame memories 383-3 to 383- (N-1) stores any one of the motion-compensated binary object images of the third to N-1st frames. Then, the stored binary object image is output to one of the weighting units 384-3 to 384- (N-1).
The frame memory 383-N stores the motion-compensated binary object image of the Nth frame, and outputs the stored binary object image to the weighting unit 384-N.

The weighting section 384-1 provides a predetermined weight to the pixel value of the motion-compensated binary object image of the first frame supplied from the frame memory 383-1.
The product is multiplied by w1 and supplied to the integrating unit 385. Weighting unit 38
4-2 is the 2 supplied from the frame memory 383-2.
The pixel value of the motion-compensated binary object image of the second frame is multiplied by a predetermined weight w2 and supplied to the integration unit 385.

Similarly, each of the weighting units 384-3 to 384- (N-1) is the third to the third memory supplied from any one of the frame memories 383-3 to 383- (N-1). The pixel value of the motion-compensated binary object image of any one of the (N-1) th frames is multiplied by one of the predetermined weights w3 to w (N-1) and supplied to the accumulator 385. Weighting unit 384-N
Multiplies the pixel value of the motion-compensated binary object image of the N-th frame supplied from the frame memory 383-N by a predetermined weight wN, and supplies the result to the integration unit 385.

The accumulating unit 385 accumulates the corresponding pixel values of the binary object image which are motion-compensated for the first to Nth frames and multiplied by one of the weights w1 to wN, respectively. A binary object image is generated by comparing the value with a predetermined threshold value th0.

As described above, the robust section 361 generates a robust binary object image from the N binary object images and supplies it to the time change detecting section 303. The configuration is shown in FIG. Even if the input image includes noise, the area specifying unit 103 can specify the area more accurately than in the case shown in FIG.

Next, the area specifying unit 10 shown in FIG.
The area specifying process 3 will be described with reference to the flowchart in FIG. Step S341 to step S
The process of step 343 is the same as step S301 to step S303 described in the flowchart of FIG. 39, and a description thereof will be omitted.

[0309] In step S344, the robust unit 361 executes robust processing.

[0310] In step S345, the time change detection unit 303 executes the area determination processing, and the processing ends. Details of the processing in step S345 are the same as the processing described with reference to the flowchart in FIG. 40, and a description thereof will be omitted.

Next, the details of the robust processing corresponding to the processing in step S344 in FIG. 45 will be described with reference to the flowchart in FIG. Step S36
In 1, the motion compensator 381 executes a motion compensation process for the input binary object image based on the motion vector supplied from the motion detector 102 and its position information. In step S362, the frame memory 383
Any of -1 to 383-N stores the motion-compensated binary object image supplied via the switch 382.

In step S363, robust unit 361 determines whether or not N binary object images are stored. If it is determined that N binary object images are not stored, step S361 is performed. Back to
The motion compensation processing of the binary object image and the storage processing of the binary object image are repeated.

If it is determined in step S363 that N binary object images have been stored, the flow advances to step S364, where the weighting units 384-1 to 384-
Each of N is weighted by multiplying each of the N binary object images by a weight of any of w1 to wN.

At step S 365, accumulating section 385
Multiplies N weighted binary object images.

[0315] In step S366, the integrating section 385
Generates a binary object image from the integrated image, for example, by comparison with a predetermined threshold value th1, and the process ends.

As described above, the area specifying unit 103 shown in FIG. 41 can generate area information based on a robust binary object image.

[0317] As described above, the area specifying unit 103 obtains, for each of the pixels included in the frame, the area information indicating that the pixel belongs to the moving area, the still area, the uncovered background area, or the covered background area. Can be generated.

FIG. 47 is a block diagram showing a configuration of the mixture ratio calculating section 104. The estimated mixture ratio processing unit 401 calculates an estimated mixture ratio for each pixel by an operation corresponding to the covered background area based on the input image, the motion vector supplied from the motion detection unit 102, and the position information thereof. The calculated estimated mixture ratio is supplied to the mixture ratio determination unit 403.

The estimated mixture ratio processing unit 402 calculates the estimated mixture ratio for each pixel based on the input image, the motion vector supplied from the motion detection unit 102, and its position information, by performing an operation corresponding to the uncovered background area. And supplies the calculated estimated mixture ratio to the mixture ratio determination unit 403.

The mixture ratio determining unit 403 includes
Sets the mixture ratio α based on the region information indicating whether the pixel for which the mixture ratio is calculated belongs to the foreground region, the background region, the covered background region, or the uncovered background region. I do. The mixture ratio determination unit 403 sets 0 to the mixture ratio α when the target pixel belongs to the foreground region, and sets 1 to the mixture ratio α when the target pixel belongs to the background region, and sets the mixture ratio α. When the pixel belongs to the covered background area, the estimated mixture ratio supplied from the estimated mixture ratio processing unit 401 is represented by the mixture ratio α.
When the target pixel belongs to the uncovered background area, the estimated mixture ratio supplied from the estimated mixture ratio processing unit 402 is set to the mixture ratio α. The mixture ratio determination unit 403 outputs a mixture ratio α set based on the region information.

FIG. 48 is a block diagram showing a configuration of the estimated mixture ratio processing unit 401.

The image input to the estimated mixture ratio processing unit 401 is input to the frame memory 421 and the weighted frame difference calculation unit 423.

The frame memory 421 stores the input image in frame units, delays the input image for a period corresponding to one frame, and supplies the stored image to the weighted frame difference calculation unit 423.

The weight generation section 422 generates a weight whose value sequentially increases from a predetermined initial value corresponding to one frame, and uses the generated weight as weight information as the weighted frame difference calculation section 423 and the maximum value. This is supplied to the determination unit 427. For example, the weight generation unit 422 sets the pre-stored initial value as the weight, and sequentially adds the pre-stored small value increment δ to the weight, thereby sequentially increasing the weight from the initial value. Generate

The weighted frame difference calculation section 423 multiplies the pixel value of the frame supplied from the frame memory 421 by the weight supplied from the weight generation section 422 to calculate a weighted pixel value. The weighted frame difference calculation unit 423 subtracts the weighted pixel value of the corresponding pixel from the pixel value of the pixel of interest in the input image,
Calculate the weighted difference.

The weighted frame difference calculator 423 calculates a plurality of weighted differences for one frame, corresponding to the weights supplied from the weight generator 422 and sequentially increasing from the initial value.

The weighted frame difference calculation section 423 uses the weighted difference calculated in this way as weighted difference image data as the motion compensation section 424 and the frame memory 42.
5

The motion compensator 424 performs motion compensation on the weighted difference image data supplied from the weighted frame difference calculator 423 based on the motion vector and the position information supplied from the motion detector 102, and performs the motion compensated weighting. The difference is supplied to the correlation value calculation unit 426.

The frame memory 425 stores a plurality of weighted difference image data for each frame, delays it for a period corresponding to one frame, and supplies the stored weighted difference image data to the correlation value calculator 426.

[0330] The correlation value calculating section 426 calculates the weighted difference image data subjected to the motion compensation supplied from the motion compensating section 424 and the one-frame delayed supplied from the frame memory 425 for each weight sequentially increasing from the initial value. The correlation value with the weighted difference image data is calculated for each pixel, and the calculated correlation value is supplied to the maximum value determination unit 427.

The correlation value calculator 426 calculates the difference between the motion-compensated weighted difference image data supplied from the motion compensator 424 and the one-frame delayed weighted difference image data supplied from the frame memory 425, for example. The absolute value is calculated for each pixel, and the calculated absolute value of the difference is supplied to the maximum value determining unit 427 as a correlation value. The correlation value output by the correlation value calculation unit 426 is also referred to as weighted difference image correlation data.

The maximum value judging section 427 includes a weight generating section 422
The correlation value calculation unit 4 in correspondence with the weight value supplied from
The correlation value supplied from 26 is stored. Maximum value judgment unit 4
27 selects a weight corresponding to the correlation value having the strongest correlation based on the stored correlation values. Maximum value judgment unit 427
Sets the selected weight to the estimated mixture ratio and outputs the estimated mixture ratio to which the selected weight is set.

Note that the maximum value judging section 427 may detect the mixture ratio in units of a block composed of a plurality of pixels.

Further, the maximum value judging section 427 may detect the mixture ratio with the block mixture ratio being constant in units of a block composed of a plurality of pixels.

Referring to FIG. 48 to FIG. 54, the estimated mixture ratio processing unit 401 corresponding to the covered background area
Will be described.

In the model of the image in which the amount of motion v of the foreground object is 4 as shown in FIG. 49, the sixth to eighth pixels from the left of frame #n belong to the mixed area.

The pixel value of the sixth pixel from the left of frame #n can be represented by equation (8). Similarly, frame #n
Can be expressed by Expression (9), and the pixel value of the eighth pixel from the left of frame #n is
It can be expressed by equation (10).

[0338]

(Equation 6)

[0339]

(Equation 7)

[0340]

(Equation 8)

Similarly, the tenth to twelfth pixels from the left of frame # n + 1 belong to the mixed area. Frame # n + 1
The pixel value of the tenth pixel from the left of can be expressed by equation (11). Similarly, the pixel value of the eleventh pixel from the left of frame # n + 1 can be expressed by equation (12), and the pixel value of the twelfth pixel from the left of frame # n + 1 is expressed by equation (1)
3).

[0342]

(Equation 9)

[0343]

(Equation 10)

[0344]

[Equation 11]

In equation (8), α05 is 1/4.
By calculating the value obtained by multiplying the pixel value of frame # n-1 by α05 and calculating the difference from the pixel value of the corresponding pixel belonging to the mixed area or background area of frame #n, as shown in FIG. , All or some of the background components included in the pixel values are removed.

At the sixth pixel from the left, all of the background components included in the pixel values are removed, leaving only the foreground components.

Similarly, in equation (11), α09 is 1 /
4 By calculating the value obtained by multiplying the pixel value of frame #n by α09 and calculating the difference from the pixel value of the corresponding pixel belonging to the mixed area of frame # n + 1, as shown in FIG. All or a part of the background components contained in is removed.

In the tenth pixel from the left, all of the background components included in the pixel values are removed, leaving only the foreground components.

In equation (9), α06 is 1/2.
By calculating a value obtained by multiplying the pixel value of frame # n-1 by α06 and calculating the difference from the pixel value of the corresponding pixel belonging to the mixed area or background area of frame #n, as shown in FIG. , All or some of the background components included in the pixel values are removed. In the sixth pixel from the left, a background component that is higher than the originally included background component is removed, so that the pixel value includes a negative value of the corresponding background component.

In the seventh pixel from the left, all of the background components included in the pixel values are removed, leaving only the foreground components.

In the equation (12), α10 is 1/2. Calculate the value obtained by multiplying the pixel value of frame #n by α10,
When the difference from the pixel value of the corresponding pixel belonging to the mixed area of frame # n + 1 is obtained, all or a part of the background component included in the pixel value is removed as shown in FIG. In the tenth pixel from the left, a background component that is higher than the originally included background component is to be removed, so that the pixel value includes a negative value of the corresponding background component.

In the eleventh pixel from the left, all of the background components included in the pixel values are removed, leaving only the foreground components.

As shown in FIG. 52, the value obtained by multiplying the pixel value of the previous frame by α05 is calculated, and the difference from the pixel value of the corresponding pixel belonging to the mixed area or background area of the next frame is obtained. , Only the foreground component remains in the sixth pixel from the left of frame #n, and from the left of frame # n + 1 indicated by a motion vector with reference to the sixth pixel from the left of frame #n. Only the foreground component remains in the tenth pixel.

The foreground component included in the sixth pixel from the left of frame #n is the same as the foreground component included in the tenth pixel from the left of frame # n + 1.

Similarly, when the value obtained by multiplying the pixel value of the previous frame by α06 is calculated, and the difference from the pixel value of the corresponding pixel belonging to the mixed area or background area of the next frame is obtained, Only the foreground component remains in the seventh pixel from the left of #n, and the eleventh pixel from the left of frame # n + 1 indicated by a motion vector based on the seventh pixel from the left of frame #n Also, only the foreground component remains.

The foreground component included in the seventh pixel from the left of frame #n is the same as the foreground component included in the eleventh pixel from the left of frame # n + 1.

As described above, the value obtained by multiplying the pixel value of frame # n-1 by a certain weight is calculated, and the difference from the pixel value of the corresponding pixel belonging to the mixed area of frame #n is obtained. When calculating the value obtained by multiplying the pixel value of #n by the same weight and calculating the difference from the pixel value of the corresponding pixel belonging to the mixed area of frame # n + 1, attention is focused on frame #n If the difference corresponding to the pixel is equal to the difference corresponding to the pixel at the position indicated by the motion vector with reference to the pixel of interest in frame # n + 1, the weight is Is equal to the mixture ratio α of the pixels.

With reference to FIG. 53, the above processing will be described with reference to equations.

The difference A between the pixel value CA of the pixel of interest belonging to the mixed area of the frame #n and the value obtained by multiplying the pixel value PA of the corresponding pixel of the frame # n-1 by the weight W is expressed by the following equation (14). ).

A = CA-PA × W (14)

The pixel value NB of the pixel of frame # n + 1 and the pixel value CB of the corresponding pixel of frame #n at the position indicated by the motion vector with reference to the pixel of interest in frame #n.
Is multiplied by the weight W, and the difference B is calculated by the calculation shown in Expression (15).

B = NB-CB × W (15)

The correlation value J can be, for example, the absolute value of the difference between the difference A and the difference B as shown in Expression (16).

[0364]

(Equation 12)

The correlation value J is a threshold value th1 stored in advance.
When it is determined that the weight ratio is equal to or smaller than the above, the weight W is equal to the mixture ratio α.

For example, as shown in FIG. 54 (A), differences A0 to A8 of 3 × 3 pixel blocks around A4 in the frame of interest and FIG. 54 (B) The difference B0 to B8 of the block of 3 × 3 pixels centered on the corresponding B4 in the frame next to the frame of interest,
By applying equation (17), the correlation value J1 corresponding to the pixel at the center of the block of the frame of interest may be calculated.

[0367]

(Equation 13)

[0368]

[Equation 14]

[0369]

(Equation 15)

In this case, for example, when it is determined that the correlation value J1 is equal to or larger than the threshold value th2 stored in advance, the weight W is equal to the mixture ratio α.

Furthermore, for example, in the frame of interest, A4
Differences A0 to A8 of 3 × 3 pixel blocks centered on
And the difference B0 to B8 of the block of 3 × 3 pixels centered on the corresponding B4 in the frame next to the frame of interest,
By applying equation (20), the sum of absolute differences J2 may be calculated as the correlation value corresponding to the pixel at the center of the block of the frame of interest.

[0372]

(Equation 16)

In this case, for example, the sum of absolute differences
When it is determined that J2 is less than the threshold value th3 stored in advance, the weight W is determined to be equal to the mixture ratio α.

In the processing corresponding to the uncovered background area executed by the estimated mixture ratio processing unit 402, the difference A is obtained by calculating the pixel value CA of the pixel of interest belonging to the mixed area of the frame #n and the frame # n + The difference between the pixel value PA and the value obtained by multiplying the pixel value PA of the corresponding pixel by the weight W, and the difference B is the frame #n Pixel value NB of +1 pixel
And a difference from the value obtained by multiplying the pixel value CB of the corresponding pixel of the frame # n + 2 by the weight W, except for the difference between the pixel value CB and the value corresponding to the covered background area.

The estimated mixture ratio processing unit 401 or the estimated mixture ratio processing unit 401 determines whether or not the position indicated by the motion vector corresponds to the pixel of interest in frame #n.
The difference B may be calculated based on the pixel of # n-1, and the estimated mixture ratio may be obtained.

FIG. 55 is a block diagram showing another configuration of the mixture ratio calculating section 104. In FIG.

[0377] Based on the area information supplied from the area specifying section 103, the selecting section 441 sends the pixels belonging to the covered background area and the corresponding pixels of the previous and subsequent frames to the estimated mixture ratio processing section 401. Supply. The selecting unit 441 supplies the pixels belonging to the uncovered background area and the corresponding pixels of the previous and subsequent frames to the estimated mixture ratio processing unit 402 based on the area information supplied from the area specifying unit 103. .

The estimated mixture ratio processing unit 401 includes a selection unit 441
Then, based on the pixel values input from, the estimated mixing ratio of the pixel of interest belonging to the covered background area is calculated, and the calculated estimated mixing ratio is supplied to the selection unit 442.

The estimated mixture ratio processing unit 402 includes a selection unit 441
Then, based on the pixel values input from, the estimated mixing ratio of the pixel of interest belonging to the uncovered background area is calculated, and the calculated estimated mixing ratio is supplied to the selection unit 442.

When the target pixel belongs to the foreground area based on the area information supplied from the area specifying section 103, the selection section 442 selects an estimated mixture ratio of 0, and selects the mixture ratio α.
If the target pixel belongs to the background area, 1
Is selected and set to the mixture ratio α. When the target pixel belongs to the covered background area, the selection unit 442 selects the estimated mixture ratio supplied from the estimated mixture ratio processing unit 401 and sets the mixture to the mixture ratio α. If it belongs to the ground area, the estimated mixture ratio supplied from the estimated mixture ratio processing unit 402 is selected and set as the mixture ratio α. The selection unit 442,
The mixture ratio α selected and set based on the region information is output.

As described above, the mixture ratio calculation unit 104 having another configuration shown in FIG. 55 can calculate the mixture ratio α for each pixel included in the image, and can output the calculated mixture ratio α.

With reference to the flowchart of FIG. 56, the process of calculating the mixture ratio α by the mixture ratio calculator 104 will be described.

[0383] In step S401, the mixture ratio calculating section 104 acquires the area information supplied from the area specifying section 103. In step S402, the estimated mixture ratio processing unit 401 executes a mixture ratio estimation process corresponding to the covered background area, and supplies the estimated mixture ratio to the mixture ratio determination unit 403. FIG. 57 shows details of the mixture ratio estimation process.
It will be described later with reference to the flowchart of FIG.

[0384] In step S403, the estimated mixture ratio processing unit 402 executes a mixture ratio estimation process corresponding to the uncovered background area, and supplies the estimated mixture ratio to the mixture ratio determination unit 403.

In step S404, the mixture ratio calculator 104 determines whether or not the mixture ratio has been estimated for the entire frame. If it is determined that the mixture ratio has not been estimated for the entire frame, the process proceeds to step S402. return,
A process of estimating the mixture ratio for the next pixel is executed.

If it is determined in step S404 that the mixture ratio has been estimated for the entire frame, the process proceeds to step S405, where the mixture ratio determination unit 403 calculates the mixture ratio α supplied from the region identification unit 103. Pixel is
Foreground area, background area, covered background area,
Alternatively, the mixture ratio α is set based on area information indicating whether the pixel belongs to any of the uncovered background areas.
The mixture ratio determination unit 403 sets 0 to the mixture ratio α when the target pixel belongs to the foreground region, and sets 1 to the mixture ratio α when the target pixel belongs to the background region, and sets the mixture ratio α. If the pixel belongs to the covered background area,
The estimated mixture ratio supplied from the estimated mixture ratio processing unit 401 is set to the mixture ratio α, and when the target pixel belongs to the uncovered background area, the estimated mixture ratio processing unit 402
Is set to the mixture ratio α, and the process ends.

As described above, the mixture ratio calculation unit 104 can calculate the mixture ratio α, which is a feature amount corresponding to each pixel, based on the region information supplied from the region identification unit 103 and the input image. .

By using the mixture ratio α, it is possible to separate the foreground component and the background component contained in the pixel value while retaining the information on the motion blur contained in the image corresponding to the moving object. Will be possible.

Next, the process of estimating the mixture ratio corresponding to the covered background area corresponding to step S402 in FIG. 56 will be described with reference to the flowchart in FIG.

At step S421, the weight generation unit 4
Reference numeral 22 sets an initial value stored in advance for the weight. The weight generation unit 422 supplies the weight for which the initial value is set as weight information to the weighted frame difference calculation unit 423 and the maximum value determination unit 427.

In step S422, the estimated mixture ratio processing unit 401 determines whether or not the weight output by the weight generation unit 422 has exceeded 1, and if it is determined that the weight has not exceeded 1, the process proceeds to step S423. move on.

In step S423, the weighted frame difference calculation unit 423 calculates a weighted difference which is a difference between the pixel value of the pixel of interest in the frame of interest and the value obtained by multiplying the pixel value of the corresponding pixel in the next frame by the weight. The calculated difference A is set to the weighted difference of the target pixel of the target frame.

In step S424, the weighted frame difference calculator 423 determines the pixel value of the pixel of the next frame of the frame of interest at the position indicated by the motion vector, based on the pixel of interest of the frame of interest, and Is calculated as a difference from the value obtained by multiplying the pixel value of the corresponding pixel by the weight, and the difference B is set to the weight corresponding to the pixel corresponding to the pixel of interest in the frame next to the frame of interest.

[0394] In step S425, the correlation value calculator 426 calculates a correlation value corresponding to the difference A and the difference B. The correlation value calculated by the correlation value calculation unit 426 is, for example,
It can be the absolute value of the difference between the difference A and the difference B.
The correlation value calculated by the correlation value calculation unit 426 is, for example, a weighted difference A0 to A8 of a block of 3 × 3 pixels,
And the corresponding weighted difference of 3 × 3 pixel blocks
The correlation value J1 calculated by applying the operation corresponding to Expression (17) to B0 to B8 can be used.

In step S 426, maximum value judging section 427 stores the correlation value supplied from correlation value calculating section 426 in association with the weight supplied from weight generating section 422.

In step S427, the weight generation unit 4
22 adds the pre-stored increment δ to the weight, returns to step S422, and repeats the process of calculating the correlation value corresponding to the weight.

That is, when the weight is 1 or less, the processing from step S423 to step S427 is repeated, the correlation value corresponding to the weight increasing for each increment δ is calculated, and the correlation value is stored in association with the weight. Is done.

If it is determined in step S422 that the weight has exceeded 1, the flow advances to step S428, and the maximum value determining unit 427 selects the weight corresponding to the maximum correlation value indicating that the correlation is strongest.

In step S429, maximum value determining section 427 sets the selected weight as the estimated mixture ratio, outputs the estimated mixture ratio with the set weight, and ends the process.

Thus, the estimated mixture ratio processing unit 401
An estimated mixture ratio corresponding to the covered background area can be calculated.

The processing corresponding to the uncovered background area in step S403 is the same as the processing for calculating the difference A and the difference B except that the pixel value of the next frame is used as the pixel value corresponding to the background. FIG. 57
Since the processing is the same as that described with reference to the flowchart of FIG.

[0402] Although the description has been made assuming that the object corresponding to the background is stationary, the above-described processing for obtaining the mixture ratio can be applied even when the image corresponding to the background area includes motion. For example, when the image corresponding to the background region is moving uniformly, the estimated mixture ratio processing unit 401 shifts the entire image corresponding to this movement, and performs the same operation as when the object corresponding to the background is stationary. To process.
Further, when the image corresponding to the background area includes a different motion for each local area, the estimated mixture ratio processing unit 401 selects a pixel corresponding to the motion as a pixel corresponding to a pixel belonging to the mixed area, and Execute the processing of

Note that the mixture ratio calculation unit 104 executes only the mixture ratio estimation process using the model corresponding to the covered background area for all the pixels, and outputs the calculated estimated mixture ratio as the mixture ratio α. You may do so. In this case, the mixture ratio α indicates the ratio of the background component with respect to the pixels belonging to the covered background region, and indicates the ratio of the foreground component with respect to the pixels belonging to the uncovered background region. For pixels belonging to the uncovered background area, the absolute value of the difference between the mixture ratio α thus calculated and 1 is calculated,
If the calculated absolute value is set to the mixture ratio α, the signal processing unit can obtain the mixture ratio α indicating the ratio of the background component for the pixels belonging to the uncovered background area.

[0404] Similarly, the mixture ratio calculation unit 104 executes only the mixture ratio estimation process using the model corresponding to the uncovered background area for all pixels, and calculates the calculated estimated mixture ratio. You may make it output as (alpha).

Next, the foreground / background separation unit 105 will be described. FIG. 58 is a block diagram illustrating an example of the configuration of the foreground / background separation unit 105. The input image supplied to the foreground / background separation unit 105 is supplied to a separation unit 601, a switch 602, and a switch 604. Information indicating the covered background area and the area information indicating the uncovered background area and supplied from the area specifying unit 103 are supplied to the separation unit 601. Area information indicating the foreground area is supplied to the switch 602. Area information indicating the background area is supplied to the switch 604.

The mixture ratio α supplied from the mixture ratio calculation unit 104 is supplied to the separation unit 601.

The separating unit 601 separates the foreground component from the input image based on the area information indicating the covered background area, the area information indicating the uncovered background area, and the mixture ratio α. The components are supplied to the synthesis unit 603, the background component is separated from the input image, and the separated background components are supplied to the synthesis unit 605.

The switch 602 is closed when a pixel corresponding to the foreground is input based on the area information indicating the foreground area, and supplies only the pixel corresponding to the foreground included in the input image to the synthesizing unit 603.

The switch 604 is closed when a pixel corresponding to the background is input, based on the area information indicating the background area, and supplies only the pixel corresponding to the background included in the input image to the synthesizing unit 605.

[0410] The synthesizing unit 603 synthesizes a foreground component image based on the foreground component supplied from the separation unit 601 and the foreground pixel supplied from the switch 602.
The synthesized foreground component image is output. Since the foreground region and the mixed region do not overlap, the combining unit 603 combines the foreground component image by applying a logical OR operation to the foreground component and the foreground pixel, for example.

The synthesizing unit 603 stores an image whose pixel values are all 0 in a built-in frame memory in the initialization processing executed at the beginning of the processing of synthesizing the foreground component image. , The foreground component image is stored (overwritten). Accordingly, in the foreground component image output by the synthesis unit 603, pixels corresponding to the background region are:
0 is stored as the pixel value.

[0412] The synthesizing unit 605 synthesizes the background component image based on the component corresponding to the background supplied from the separating unit 601 and the pixel corresponding to the background supplied from the switch 604, and converts the synthesized background component image. Output. Since the background region and the mixed region do not overlap, the combining unit 605 combines the background component image by applying a logical OR operation to the component corresponding to the background and the pixel corresponding to the background, for example.

[0413] In the initializing process executed at the beginning of the process of synthesizing the background component image, the synthesizing unit 605 stores an image whose pixel values are all 0 in a built-in frame memory, The background component image is stored (overwritten). Accordingly, in the background component image output by the synthesis unit 605, pixels corresponding to the foreground area include:
0 is stored as the pixel value.

FIG. 59 is a diagram showing an input image input to the foreground / background separation unit 105, and a foreground component image and a background component image output from the foreground / background separation unit 105.

FIG. 59A is a schematic diagram of an image to be displayed, and FIG. 59B is a diagram showing a pixel belonging to the foreground area, a pixel belonging to the background area, and a mixed area corresponding to FIG. 1 is a model diagram in which pixels of one line including pixels belonging to are developed in the time direction.

As shown in FIGS. 59 (A) and 59 (B), the background component image output from the foreground / background separation unit 105 is composed of pixels belonging to the background region and background components included in the pixels of the mixed region. Consists of

As shown in FIGS. 59 (A) and 59 (B), the foreground component image output from foreground / background separation section 105 has the foreground components included in the pixels belonging to the foreground area and the pixels in the mixed area. Consists of

[0418] The pixel value of the pixel in the mixed area is separated into a background component and a foreground component by the foreground / background separation unit 105. The separated background component forms a background component image together with the pixels belonging to the background area. The separated foreground component forms a foreground component image together with the pixels belonging to the foreground area.

[0419] As described above, in the foreground component image, the pixel value of the pixel corresponding to the background region is set to 0, and significant pixel values are set to the pixel corresponding to the foreground region and the pixel corresponding to the mixed region. Similarly, in the background component image, the pixel value of the pixel corresponding to the foreground area is set to 0, and meaningful pixel values are set to the pixel corresponding to the background area and the pixel corresponding to the mixed area.

Next, a description will be given of a process performed by the separating unit 601 to separate a foreground component and a background component from pixels belonging to the mixed area.

FIG. 60 is an image model showing foreground components and background components of two frames including a foreground corresponding to an object moving from left to right in the figure. In the image model shown in FIG. 60, the motion amount v of the foreground is 4
, And the number of virtual divisions is 4.

In frame #n, the leftmost pixel and the fourteenth through eighteenth pixels from the left consist only of background components and belong to the background area. In frame #n,
The second to fourth pixels from the left include the background component and the foreground component, and belong to the uncovered background area. In frame #n, the eleventh to thirteenth from the left
The third pixel contains background and foreground components and belongs to the covered background area. In frame #n, the fifth through tenth pixels from the left consist only of foreground components and belong to the foreground area.

In frame # n + 1, the first through fifth pixels from the left and the eighteenth pixel from the left consist only of background components and belong to the background area. In frame # n + 1, the sixth through eighth pixels from the left include the background component and the foreground component, and belong to the uncovered background area. In frame # n + 1, the fifteenth through seventeenth pixels from the left include the background component and the foreground component, and belong to the covered background area. In frame # n + 1, the ninth to fourteenth pixels from the left consist only of foreground components and belong to the foreground area.

FIG. 61 is a view for explaining the processing for separating the foreground components from the pixels belonging to the covered background area. In FIG. 61, α1 to α18 are mixing ratios corresponding to the respective pixels in frame #n.
In FIG. 61, the fifteenth through seventeenth pixels from the left belong to the covered background area.

[0425] The pixel value C15 of the fifteenth pixel from the left of frame #n is represented by equation (21).

C15 = B15 / v + F09 / v + F08 / v + F07 / v = α15 ・ B15 + F09 / v + F08 / v + F07 / v = α15 ・ P15 + F09 / v + F08 / v + F07 / v (21) Here, α15 is a mixture ratio of the fifteenth pixel from the left of frame #n. P15 is the pixel value of the fifteenth pixel from the left of frame # n-1.

Based on equation (21), one frame from the left of frame #n
The sum f15 of the foreground components of the fifth pixel is expressed by equation (22).

F15 = F09 / v + F08 / v + F07 / v = C15-α15 · P15 (22)

Similarly, the sum f16 of the foreground components of the sixteenth pixel from the left of frame #n is expressed by equation (23), and the sum f17 of the foreground components of the seventeenth pixel from the left of frame #n is
Is represented by equation (24).

F16 = C16-α16 · P16 (23) f17 = C17-α17 · P17 (24)

Thus, the foreground component fc contained in the pixel value C of the pixel belonging to the covered background area is
It is calculated by equation (25).

Fc = C−α · P (25) P is the pixel value of the corresponding pixel in the immediately preceding frame.

FIG. 62 is a view for explaining the processing for separating the foreground components from the pixels belonging to the uncovered background area. In FIG. 62, α1 to α18 are mixing ratios corresponding to each of the pixels in frame #n. In FIG. 62, the second through fourth pixels from the left belong to the uncovered background area.

[0434] Pixel value of second pixel from left of frame #n
C02 is represented by equation (26).

C02 = B02 / v + B02 / v + B02 / v + F01 / v = α2 / B02 + F01 / v = α2 / N02 + F01 / v (26) where α2 is the left of frame #n Is the mixture ratio of the second pixel from. N02 is the pixel value of the second pixel from the left of frame # n + 1.

Based on equation (26), two frames from the left of frame #n
The sum f02 of the foreground components of the ith pixel is expressed by equation (27).

F02 = F01 / v = C02-α2 · N02 (27)

Similarly, the sum f03 of the foreground components of the third pixel from the left of frame #n is expressed by equation (28), and the sum f04 of the foreground components of the fourth pixel from the left of frame #n is Is represented by equation (29).

F03 = C03-α3 · N03 (28) f04 = C04-α4 · N04 (29)

As described above, the foreground component fu included in the pixel value C of the pixel belonging to the uncovered background area is obtained.
Is calculated by Expression (30).

Fu = C−α · N (30) N is the pixel value of the corresponding pixel in the next frame.

[0442] As described above, the separation unit 601 outputs the information indicating the covered background area included in the area information.
And information indicating the uncovered background area,
The foreground component and the background component can be separated from the pixels belonging to the mixed area based on the mixing ratio α for each pixel.

FIG. 63 is a block diagram showing an example of the configuration of the separating section 601 for executing the processing described above. The image input to the separation unit 601 is stored in a frame memory 621
The area information indicating the covered background area and the uncovered background area supplied from the mixing ratio calculation unit 104 and the mixing ratio α are input to the separation processing block 622.

[0444] The frame memory 621 stores an input image in frame units. The frame memory 621 is
When the target of processing is frame #n, frame # n−1, frame #n, which is one frame before frame #n, and frame # n + 1, which is one frame after frame #n, Remember.

The frame memory 621 stores the frame # n−1,
The corresponding pixels of the frame #n and the frame # n + 1 are supplied to the separation processing block 622.

[0446] The separation processing block 622 determines the frame # n-1 and the frame #n supplied from the frame memory 621 based on the area information indicating the covered background area and the uncovered background area and the mixture ratio α.
n and the pixel value of the corresponding pixel in frame # n + 1 in FIG.
By applying the operation described with reference to FIG. 1 and FIG. 62, the foreground component and the background component are separated from the pixels belonging to the mixed area of frame #n, and supplied to the frame memory 623.

The separation processing block 622 includes an uncovered area processing section 631, a covered area processing section 632, a synthesizing section 633, and a synthesizing section 634.

The multiplier 6 of the uncovered area processing section 631
41 multiplies the mixture ratio α by the pixel value of the pixel of frame # n + 1 supplied from the frame memory 621, and
42. The switch 642 is connected to the frame memory 6
When the pixel of frame #n supplied from 21 (corresponding to the pixel of frame # n + 1) is the uncovered background area, the pixel is closed and multiplied by the mixture ratio α supplied from the multiplier 641 To the computing unit 643 and the combining unit 63
4 Frame output from switch 642
The value obtained by multiplying the pixel value of pixel # n + 1 by the mixture ratio α
It is equal to the background component of the pixel value of the corresponding pixel of #n.

The computing unit 643 obtains a foreground component by subtracting the background component supplied from the switch 642 from the pixel value of the pixel of frame #n supplied from the frame memory 621. The computing unit 643 supplies the foreground component of the pixel of frame #n belonging to the uncovered background area to the combining unit 633.

Multiplier 651 of covered area processing section 632
Multiplies the mixture ratio α by the pixel value of the pixel of frame # n−1 supplied from the frame memory 621 to generate a switch 652
Output to The switch 652 is connected to the frame memory 621
When the pixel of frame #n (corresponding to the pixel of frame # n-1) supplied from is the covered background area, the pixel value is closed and multiplied by the mixture ratio α supplied from the multiplier 651 To the synthesizer 653 and the synthesizing unit 634. The value obtained by multiplying the pixel value of the pixel of frame # n−1 output from the switch 652 by the mixture ratio α is equal to the background component of the pixel value of the corresponding pixel of frame #n.

The computing unit 653 subtracts the background component supplied from the switch 652 from the pixel value of the pixel of frame #n supplied from the frame memory 621 to obtain the foreground component. The calculator 653 supplies the foreground components of the pixels of frame #n belonging to the covered background area to the combining unit 633.

[0452] The synthesizing unit 633 calculates the arithmetic unit 6 of the frame #n.
The foreground component of the pixel belonging to the uncovered background area supplied from 43 and the foreground component of the pixel belonging to the covered background area supplied from the calculator 653 are combined and supplied to the frame memory 623. .

[0453] The synthesizing unit 634 calculates the background component of the pixel belonging to the uncovered background area and the switch 652 supplied from the switch 642 in the frame #n.
The background components of the pixels belonging to the covered background area supplied from the
To supply.

The frame memory 623 stores the foreground component and the background component of the pixels in the mixed area of the frame #n supplied from the separation processing block 622, respectively.

The frame memory 623 outputs the stored foreground components of the pixels of the mixed area of frame #n and the stored background components of the pixels of the mixed area of frame #n.

By utilizing the mixture ratio α, which is a feature quantity, it becomes possible to completely separate the foreground component and the background component contained in the pixel value.

The combining section 603 combines the foreground components of the pixels in the mixed area of frame #n output from the separating section 601 with the pixels belonging to the foreground area to generate a foreground component image. The synthesizing unit 605 generates a background component image by synthesizing the background component of the pixel in the mixed area of the frame #n and the pixel belonging to the background area output from the separating unit 601.

FIG. 64 is a diagram showing an example of a foreground component image and an example of a background component image corresponding to frame #n in FIG.

FIG. 64A shows an example of a foreground component image corresponding to frame #n in FIG. Since the leftmost pixel and the fourteenth pixel from the left consist of only the background component before the foreground and the background are separated, the pixel value is set to 0.

The second to fourth pixels from the left belong to the uncovered background area before the foreground and the background are separated, the background component is set to 0, and the foreground component is left as it is. . The eleventh to thirteenth pixels from the left belong to the covered background area before the foreground and the background are separated, the background component is set to 0, and the foreground component is left as it is. The fifth to tenth pixels from the left are left as they are because they consist only of foreground components.

FIG. 64B shows an example of a background component image corresponding to frame #n in FIG. The leftmost pixel and the fourteenth pixel from the left are left alone because they consisted only of the background component before the foreground and the background were separated.

The second to fourth pixels from the left belong to the uncovered background area before the foreground and the background are separated, the foreground component is set to 0, and the background component is left as it is. . Before the foreground and background are separated, the eleventh to thirteenth pixels from the left belong to the covered background area, the foreground component is set to 0, and the background component is left as it is. Before the foreground and the background are separated, the fifth through tenth pixels from the left consist only of the foreground components, and thus have a pixel value of 0.

Next, the process of separating the foreground and the background by the foreground / background separator 105 will be described with reference to the flowchart shown in FIG. In step S601, the frame memory 621 of the separation unit 601 acquires an input image,
The frame #n to be separated from the foreground and the background is stored together with the previous frame # n-1 and the subsequent frame # n + 1.

In step S602, the separation unit 601
The separation processing block 622 obtains the area information supplied from the mixture ratio calculation unit 104. In step S603, the separation processing block 622 of the separation unit 601 acquires the mixture ratio α supplied from the mixture ratio calculation unit 104.

[0465] In step S604, the uncovered area processing section 631 performs the processing based on the area information and the mixture ratio α.
The background component is extracted from the pixel values of the pixels belonging to the uncovered background area supplied from the frame memory 621.

[0466] In step S605, the uncovered area processing section 631 performs processing based on the area information and the mixture ratio α.
The foreground component is extracted from the pixel values of the pixels belonging to the uncovered background area supplied from the frame memory 621.

In step S606, the covered area processing unit 632 extracts a background component from the pixel values of the pixels belonging to the covered background area supplied from the frame memory 621 based on the area information and the mixture ratio α. .

In step S607, the covered area processing unit 632 extracts a foreground component from the pixel values of the pixels belonging to the covered background area supplied from the frame memory 621 based on the area information and the mixture ratio α. .

[0469] In step S608, the combining section 633
Combines the foreground components of the pixels belonging to the uncovered background area extracted in the processing of step S605 with the foreground components of the pixels belonging to the covered background area extracted in the processing of step S607. The synthesized foreground component is supplied to the synthesis unit 603. Further, the synthesizing unit 603 generates a foreground component image by synthesizing the pixels belonging to the foreground area supplied via the switch 602 and the foreground components supplied from the separating unit 601.

[0470] In step S609, the synthesizing unit 634
Combines the background component of the pixel belonging to the uncovered background area extracted in the processing of step S604 with the background component of the pixel belonging to the covered background area extracted in the processing of step S606. The synthesized background component is supplied to the synthesis unit 605. Further, the synthesizing unit 605 generates a background component image by synthesizing the pixels belonging to the background area supplied via the switch 604 and the background component supplied from the separation unit 601.

[0471] In step S610, the synthesizing unit 603
Outputs a foreground component image. In step S611, the synthesis unit 605 outputs the background component image, and the processing ends.

As described above, the foreground / background separation unit 105 separates the foreground component and the background component from the input image based on the area information and the mixture ratio α, and outputs a foreground component image consisting of only the foreground component. Also, a background component image consisting of only background components can be output.

Next, adjustment of the amount of motion blur from the foreground component image will be described.

FIG. 66 is a block diagram showing an example of the structure of the motion-blur adjusting unit 106. The motion vector and its position information supplied from the motion detection unit 102 and the region information supplied from the region identification unit 103 are stored in the processing unit determination unit 8.
01 and the modeling unit 802. The foreground component image supplied from the foreground / background separation unit 105 is supplied to the addition unit 804.

[0475] The processing unit determination unit 801 supplies the generated processing unit to the modeling unit 802 together with the motion vector based on the motion vector, its position information, and area information. The processing unit determination unit 801 supplies the generated processing unit to the adding unit 804.

The processing unit generated by the processing unit determination unit 801 starts from a pixel corresponding to the covered background area of the foreground component image and extends to a pixel corresponding to the uncovered background area as shown in FIG. A continuous pixel in the movement direction starting from a pixel corresponding to the uncovered background area or a continuous pixel aligned in the movement direction to a pixel corresponding to the covered background area is shown. The processing unit is, for example, an upper left point (a pixel designated by the processing unit and a position of a pixel located at the leftmost or uppermost position on the image) and a lower right point.
Data.

[0477] The modeling unit 802 performs modeling based on the motion vector and the input processing unit. More specifically, for example, the modeling unit 802 pre-stores a plurality of models corresponding to the number of pixels included in the processing unit, the number of virtual divisions of pixel values in the time direction, and the number of foreground components for each pixel. A model that specifies the correspondence between pixel values and foreground components as shown in FIG. 68 may be selected based on the processing unit and the number of virtual divisions of pixel values in the time direction. .

For example, if the number of pixels corresponding to the processing unit is one,
When the motion amount v within the shutter time is 5, the modeling unit 802 sets the virtual division number to 5, the leftmost pixel includes one foreground component, and the second pixel from the left. Pixel includes two foreground components, the third pixel from the left includes three foreground components, the fourth pixel from the left includes four foreground components, and the fifth pixel from the left includes five foreground components. Including the foreground component, the sixth pixel from the left contains five foreground components, the seventh pixel from the left contains five foreground components, and the eighth pixel from the left contains five foreground components. , The ninth pixel from the left contains four foreground components, the tenth pixel from the left contains three foreground components, the eleventh pixel from the left contains two foreground components, and 12 from the left.
A model is selected in which the th pixel contains one foreground component and comprises a total of eight foreground components.

[0479] When a motion vector and a processing unit are supplied, instead of selecting from a model stored in advance, the modeling unit 802 generates a model based on the motion vector and the processing unit. You may do so.

[0480] The modeling unit 802 supplies the selected model to the equation generation unit 803.

[0481] The equation generation unit 803 includes a modeling unit 802.
Generate an equation based on the model supplied from. FIG.
Referring to the model of the foreground component image shown in FIG. 8, the number of foreground components is 8, the number of pixels corresponding to the processing unit is 12, the motion amount v is 5, and the number of virtual divisions is 5. An equation generated by the equation generation unit 803 at a certain time will be described.

When the foreground component corresponding to the shutter time / v included in the foreground component image is F01 / v to F08 / v, F01 / v
To F08 / v and the pixel values C01 to C12 are given by Equation (31).
To (42).

C01 = F01 / v (31) C02 = F02 / v + F01 / v (32) C03 = F03 / v + F02 / v + F01 / v (33) C04 = F04 / v + F03 / v + F02 / v + F01 / v (34) C05 = F05 / v + F04 / v + F03 / v + F02 / v + F01 / v (35) C06 = F06 / v + F05 / v + F04 / v + F03 / v + F02 / v (36) C07 = F07 / v + F06 / v + F05 / v + F04 / v + F03 / v (37) C08 = F08 / v + F07 / v + F06 / v + F05 / v + F04 / v (38) C09 = F08 / v + F07 / v + F06 / v + F05 / v (39) C10 = F08 / v + F07 / v + F06 / v (40) C11 = F08 / v + F07 / v (41) C12 = F08 / v (42)

The equation generating section 803 deforms the generated equation to generate an equation. Equations (43) to (54) show the equations generated by the equation generation unit 803.

[0485] C01 = 1 ・ F01 / v + 0 ・ F02 / v + 0 ・ F03 / v + 0 ・ F04 / v + 0 ・ F05 / v +0 ・ F06 / v + 0 ・ F07 / v + 0 ・ F08 / v (43) C02 = 1 ・ F01 / v + 1 ・ F02 / v + 0 ・ F03 / v + 0 ・ F04 / v + 0 ・ F05 / v +0 ・ F06 / v + 0 ・ F07 / v + 0・ F08 / v (44) C03 = 1 ・ F01 / v + 1 ・ F02 / v + 1 ・ F03 / v + 0 ・ F04 / v + 0 ・ F05 / v +0 ・ F06 / v + 0 ・ F07 / v +0 ・ F08 / v (45) C04 = 1 ・ F01 / v + 1 ・ F02 / v + 1 ・ F03 / v + 1 ・ F04 / v + 0 ・ F05 / v +0 ・ F06 / v + 0 ・ F07 / v + 0 ・ F08 / v (46) C05 = 1 ・ F01 / v + 1 ・ F02 / v + 1 ・ F03 / v + 1 ・ F04 / v + 1 ・ F05 / v +0 ・ F06 / v + 0・ F07 / v + 0 ・ F08 / v (47) C06 = 0 ・ F01 / v + 1 ・ F02 / v + 1 ・ F03 / v + 1 ・ F04 / v + 1 ・ F05 / v +1 ・ F06 / v +0 ・ F07 / v + 0 ・ F08 / v (48) C07 = 0 ・ F01 / v + 0 ・ F02 / v + 1 ・ F03 / v + 1 ・ F04 / v + 1 ・ F05 / v +1 ・ F06 / v + 1 ・ F07 / v + 0 ・ F08 / v (49) C08 = 0 ・ F01 / v + 0 ・ F02 / v + 0 ・ F03 / v + 1 ・ F04 / v + 1 ・ F05 / v +1・ F06 / v + 1 ・ F07 / v + 1 ・ F08 / v (50) C09 = 0 ・ F01 / v + 0 ・ F02 / v + 0 ・ F03 / v + 0 ・ F04 / v + 1 ・ F05 / v +1 ・ F06 / v + 1 ・ F07 / v + 1 ・ F08 / v (51) C10 = 0 ・ F01 / v + 0 ・ F02 / v + 0 ・ F03 / v + 0 ・ F04 / v + 0 ・ F05 / v +1 ・ F06 / v + 1 ・ F07 / v + 1 ・ F08 / v (52) C 11 = 0 ・ F01 / v + 0 ・ F02 / v + 0 ・ F03 / v + 0 ・ F04 / v + 0 ・ F05 / v +0 ・ F06 / v + 1 ・ F07 / v + 1 ・ F08 / v ( 53) C12 = 0 ・ F01 / v + 0 ・ F02 / v + 0 ・ F03 / v + 0 ・ F04 / v + 0 ・ F05 / v +0 ・ F06 / v + 0 ・ F07 / v + 1 ・ F08 / v (54)

Formulas (43) to (54) are obtained by using formula (55).
It can also be expressed as

[0487]

[Equation 17] In Expression (55), j indicates a position of a pixel. In this example, j has a value of any one of 1 to 12. I indicates the position of the foreground value. In this example, i has a value of any one of 1 to 8. aij
Has a value of 0 or 1 corresponding to the values of i and j.

When expressed in consideration of the error, equation (55)
Can be expressed as in equation (56).

[0489]

(Equation 18) In Expression (56), ej is an error included in the target pixel Cj.

Equation (56) can be rewritten as equation (57).

[0490]

[Equation 19]

Here, in order to apply the least squares method, the sum of squares E of the error is defined as shown in Expression (58).

[0493]

(Equation 20)

To minimize the error, the sum of the squares of the error
What is necessary is that the value of the partial differentiation of E with respect to the variable Fk becomes 0. Fk is determined so as to satisfy Expression (59).

[0495]

(Equation 21)

In equation (59), since the motion amount v is a fixed value, equation (60) can be derived.

[0497]

(Equation 22)

By expanding the equation (60) and transposing it, the equation (61) is obtained.

[0499]

(Equation 23)

The expression is expanded into eight expressions obtained by substituting any one of integers 1 to 8 into k in the expression (61). The obtained eight expressions can be represented by one expression by a matrix. This equation is called a normal equation.

[0501] An example of a normal equation generated by the equation generation unit 803 based on the least squares method is shown in equation (62).

[0502]

(Equation 24)

If equation (62) is expressed as A · F = v · C, C, A, and v are known, and F is unknown. A and v are known at the time of modeling, but C becomes known by inputting pixel values in the adding operation.

By calculating the foreground component using a normal equation based on the least squares method, the error contained in the pixel C can be dispersed.

The equation generator 803 supplies the normal equation generated in this way to the adder 804.

The adding unit 804 includes the processing unit determining unit 80
Based on the processing unit supplied from 1, the pixel value C included in the foreground component image is set in the matrix expression supplied from the equation generation unit 803. The adding unit 804 supplies a matrix in which the pixel values C are set to the calculation unit 805.

[0507] The arithmetic unit 805 uses the sweep-out method (Gauss-Jord).
An elimination method such as an) is used to calculate the foreground component Fi / v from which the motion blur has been removed, and the pixel value of the foreground from which the motion blur has been removed, which is an integer from 0 to 8 69 is calculated, and the foreground component image from which motion blur has been removed, which is composed of the pixel values Fi from which motion blur has been removed, as shown in FIG. 807.

In the foreground component image from which motion blur has been removed as shown in FIG. 69, F01 to F08 are respectively set to C03 to C10 because the position of the foreground component image with respect to the screen is not changed. And can correspond to an arbitrary position.

[0509] The motion blur adding unit 806 generates a motion blur adjustment amount v 'having a value different from the motion amount v, for example, a motion blur adjustment amount v' having a half value of the motion amount v, or a value irrelevant to the motion amount v. By giving the motion blur adjustment amount v ′, the amount of motion blur can be adjusted. For example, as shown in FIG. 70, the motion blur adding unit 806 outputs the pixel value Fi of the foreground from which the motion blur has been removed.
Is divided by the motion blur adjustment amount v 'to obtain the foreground component Fi / v'
Is calculated, and the sum of the foreground components Fi / v ′ is calculated to generate a pixel value in which the amount of motion blur is adjusted. For example, when the motion blur adjustment amount v ′ is 3, the pixel value C02 is (F01) / v ′, the pixel value C03 is (F01 + F02) / v ′, and the pixel value C04 is
Is (F01 + F02 + F03) / v ', and the pixel value C05 is (F02 + F02).
03 + F04) / v '.

[0510] The motion blur adding unit 806 supplies the foreground component image in which the amount of motion blur has been adjusted to the selection unit 807.

[0511] The selection unit 807, for example, based on a selection signal corresponding to the user's selection, receives the foreground component image from which the motion blur has been removed and the motion supplied from the motion blur addition unit 806. One of the foreground component images whose blur amount has been adjusted is selected, and the selected foreground component image is output.

[0512] As described above, the motion blur adjusting unit 106 can adjust the amount of motion blur based on the selection signal and the motion blur adjustment amount v '.

For example, as shown in FIG. 71, when the number of pixels corresponding to the processing unit is 8 and the motion amount v is 4, the motion-blur adjusting unit 106 sets the matrix shown in Expression (63) Generates the expression

[0514]

(Equation 25)

[0515] The motion-blur adjusting unit 106 formulates a number corresponding to the length of the processing unit as described above, and calculates Fi, which is a pixel value in which the amount of motion blur has been adjusted. Similarly, for example, when the number of pixels included in the processing unit is 100, 1
An equation corresponding to 00 pixels is generated to calculate Fi.

[0516] Fig. 72 is a diagram showing another configuration of the motion blur adjustment unit 106. The same portions as those shown in FIG. 66 are denoted by the same reference numerals, and description thereof will be omitted.

The selection unit 821 supplies the input motion vector and its position signal as they are to the processing unit determination unit 801 and the modeling unit 802 based on the selection signal, or adjusts the size of the motion vector by motion blur adjustment. The motion vector whose magnitude has been replaced with the motion blur adjustment amount v 'instead of the amount v' and the position signal thereof are supplied to the processing unit determination unit 801 and the modeling unit 802.

[0518] By doing so, the processing unit determination unit 801 to the calculation unit 805 of the motion blur adjustment unit 106 shown in FIG.
Can adjust the amount of motion blur according to the values of the motion amount v and the motion blur adjustment amount v ′. For example, the amount of movement v
Is 5 and the motion blur adjustment amount v ′ is 3, the processing unit determination unit 801 to the calculation unit 805 of the motion blur adjustment unit 106 in FIG. 72 determine whether the foreground component whose motion amount v is 5 shown in FIG. FIG. 70 corresponding to the motion blur adjustment amount v ′ of 3
The calculation is performed according to the model shown in
v) / (Motion blur adjustment amount v ′) = 5/3, that is, an image including a motion blur corresponding to the motion amount v of approximately 1.7 is calculated. In addition,
In this case, since the calculated image does not include the motion blur corresponding to the motion amount v of 3, the motion blur adding unit 80
It should be noted that the meaning of the relationship between the motion amount v and the motion blur adjustment amount v ′ differs from the result of No. 6.

As described above, the motion-blur adjusting unit 106
An equation is generated corresponding to the motion amount v and the processing unit, and the pixel value of the foreground component image is set in the generated equation to calculate a foreground component image in which the amount of motion blur is adjusted.

Next, the process of adjusting the amount of motion blur included in the foreground component image by the motion blur adjustment unit 106 will be described with reference to the flowchart in FIG.

[0521] In step S801, the processing unit determination unit 801 of the motion blur adjustment unit 106 generates a processing unit based on the motion vector and the area information, and supplies the generated processing unit to the modeling unit 802.

[0522] In step S802, the modeling unit 802 of the motion-blur adjusting unit 106 selects or generates a model according to the motion amount v and the processing unit. Step S
In 803, the equation generator 803 creates a normal equation based on the selected model.

In step S804, the adding section 8
Step 04 sets the pixel value of the foreground component image in the created normal equation. In step S805, the adding unit 8
In step S804, it is determined whether the pixel values of all the pixels corresponding to the processing unit have been set. If it is determined that the pixel values of all the pixels corresponding to the processing unit have not been set, step S804 is performed. And the process of setting the pixel value to the normal equation is repeated.

[0524] If it is determined in step S805 that the pixel values of all the pixels in the processing unit have been set, the process proceeds to step S806, and the arithmetic unit 805 causes the adding unit 8
Based on the normal equation in which the pixel values supplied from 04 are set, the pixel values of the foreground with the amount of motion blur adjusted are calculated,
The process ends.

[0525] As described above, the motion blur adjusting unit 106 can adjust the amount of motion blur from the foreground image including motion blur based on the motion vector and the area information.

That is, it is possible to adjust the amount of motion blur included in a pixel value which is sample data.

[0527] As described above, the signal processing section shown in Fig. 2 can adjust the amount of motion blur included in the input image. The signal processing unit shown in FIG. 2 can calculate the mixture ratio α, which is the buried information, and output the calculated mixture ratio α.

FIG. 74 is a block diagram showing another example of the configuration of the motion-blur adjusting unit 106. The motion vector and its position information supplied from the motion detection unit 102 are supplied to the processing unit determination unit 901 and the correction unit 905, and the region information supplied from the region identification unit 103 is supplied to the processing unit determination unit 9
01 is supplied. The foreground component image supplied from the foreground / background separation unit 105 is supplied to the calculation unit 904.

The processing unit determination unit 901 supplies the generated processing unit to the modeling unit 902 together with the motion vector based on the motion vector, its position information, and the area information.

[0531] The modeling unit 902 performs modeling based on the motion vector and the input processing unit.

[0531] The equation generating unit 903 includes a modeling unit 902.
Generate an equation based on the model supplied from.

Referring to the foreground component image models shown in FIGS. 75 to 77, the number of foreground components is 8, the number of pixels corresponding to the processing unit is 12, and the motion amount v is 5. An example of an equation generated by the equation generation unit 903 at this time will be described.

When the foreground component corresponding to the shutter time / v included in the foreground component image is F01 / v to F08 / v, F01 / v
The relationship between F08 / v and the pixel values C01 to C12 is expressed by Expressions (31) to (42) as described above.

Looking at the pixel values C12 and C11, the pixel value
As shown in Expression (64), C12 includes only the foreground component F08 / v, and the pixel value C11 is composed of the product sum of the foreground component F08 / v and the foreground component F07 / v. Therefore, the foreground component F07 / v is
It can be obtained by equation (65).

F08 / v = C12 (64) F07 / v = C11-C12 (65)

Similarly, taking into account the foreground components included in the pixel values C10 to C01, the foreground components F06 / v to F01 / v can be obtained by the equations (66) to (71).

F06 / v = C10-C11 (66) F05 / v = C09-C10 (67) F04 / v = C08-C09 (68) F03 / v = C07-C08 + C12 (69) F02 / v = C06 -C07 + C11-C12 (70) F01 / v = C05-C06 + C10-C11 (71)

The equation generator 903 generates an equation for calculating a foreground component based on a difference between pixel values, as shown in the equations (64) to (71). Equation generator 903
Supplies the generated equation to the calculation unit 904.

The computing unit 904 sets the pixel value of the foreground component image in the equation supplied from the equation generating unit 903,
The foreground component is calculated based on the equation in which the pixel value is set. The calculation unit 904 calculates, for example, Expressions (64) to (7)
When 1) is supplied from the equation generation unit 903, the equation (6)
4) Set the pixel values C05 through C12 in the equations (71).

The calculating section 904 calculates the foreground component based on the equation in which the pixel values are set. For example, the arithmetic unit 904
Calculates the foreground components F01 / v to F08 / v as shown in FIG. 76 by the calculation based on the equations (64) to (71) in which the pixel values C05 to C12 are set. The calculation unit 904 includes:
The foreground components F01 / v to F08 / v are supplied to the correction unit 905.

The correction unit 905 multiplies the foreground component supplied from the calculation unit 904 by the motion amount v included in the motion vector supplied from the processing unit determination unit 901 to remove the pixel value of the foreground from which motion blur has been removed. Is calculated. For example, the correction unit 9
05 is a foreground component F01 / v supplied from the calculation unit 904.
To F08 / v are supplied, the foreground components F01 / v to F08 /
By multiplying each of the v by the motion amount v which is 5,
As shown in FIG. 77, the foreground pixel values from which motion blur has been removed
F01 to F08 are calculated.

[0542] The correction unit 905 supplies the foreground component image composed of the pixel values of the foreground from which the motion blur has been removed as calculated above to the motion blur addition unit 906 and the selection unit 907.

The motion-blur adding unit 906 generates a motion-blur adjustment amount v ′ having a value different from the motion amount v, for example, a motion-blur adjustment amount v ′ having a half value of the motion amount v and a value unrelated to the motion amount v. The amount of motion blur can be adjusted with the motion blur adjustment amount v ′.
For example, as shown in FIG.
Calculates the foreground component Fi / v 'by dividing the pixel value Fi of the foreground from which the motion blur has been removed by the motion blur adjustment amount v',
The sum of the foreground components Fi / v 'is calculated to generate a pixel value with the amount of motion blur adjusted. For example, the motion blur adjustment amount v ′ is 3
, The pixel value C02 is (F01) / v ′, and the pixel value C03
Is (F01 + F02) / v ', and the pixel value C04 is (F01 + F02 + F
03) / v ', and the pixel value C05 is (F02 + F03 + F04) / v'.

[0544] The motion blur adding unit 906 supplies the foreground component image in which the amount of motion blur has been adjusted to the selecting unit 907.

[0545] The selection unit 907 includes, for example, a foreground component image from which the motion blur has been removed supplied from the correction unit 905 and the motion supplied from the motion blur addition unit 906 based on a selection signal corresponding to the user's selection. One of the foreground component images whose blur amount has been adjusted is selected, and the selected foreground component image is output.

[0546] As described above, the motion blur adjusting unit 106 can adjust the amount of motion blur based on the selection signal and the motion blur adjustment amount v '.

Next, FIG. 78 shows a process of adjusting the amount of foreground motion blur by the motion blur adjustment unit 106 shown in FIG.
This will be described with reference to the flowchart of FIG.

In step S901, the processing unit determination unit 901 of the motion blur adjustment unit 106 generates a processing unit based on the motion vector and the area information, and supplies the generated processing unit to the modeling unit 902 and the correction unit 905. I do.

[0549] In step S902, the modeling unit 902 of the motion-blur adjusting unit 106 selects or generates a model corresponding to the amount of motion v and the processing unit. Step S
In 903, the equation generation unit 903 generates an equation for calculating a foreground component from a difference between pixel values of a foreground component image based on the selected or generated model.

[0550] In step S904, the arithmetic unit 904
Sets the pixel value of the foreground component image in the created equation, and extracts the foreground component from the difference between the pixel values based on the equation in which the pixel value is set. In step S905,
The calculation unit 904 determines whether all the foreground components corresponding to the processing unit have been extracted. If it is determined that all the foreground components corresponding to the processing unit have not been extracted, the process returns to step S904. The process of extracting the foreground components is repeated.

If it is determined in step S905 that all foreground components corresponding to the processing unit have been extracted,
In step S906, the correction unit 905 corrects each of the foreground components F01 / v to F08 / v supplied from the calculation unit 904 based on the motion amount v, and removes the motion blur from the foreground pixel values. F01 to F08 are calculated.

In step S907, the motion blur adding unit 906 calculates the pixel value of the foreground with the amount of motion blur adjusted, and the selecting unit 907 adjusts the image from which the motion blur has been removed or the amount of motion blur. One of the selected images is selected, the selected image is output, and the process ends.

As described above, the motion-blur adjusting unit 106 having the configuration shown in FIG. 74 can more quickly adjust the motion blur from the foreground image including the motion blur with a simpler operation.

The conventional method of partially removing motion blur such as a Wiener filter has an effect in an ideal state, but does not provide a sufficient effect on an actual image that is quantized and contains noise. On the other hand, the motion-blur adjusting unit 106 shown in FIG. 74 also has a sufficient effect on an actual image that is quantized and contains noise.
Accurate removal of motion blur can be achieved.

FIG. 79 is a block diagram showing another configuration of the function of the signal processing unit.

[0556] The same portions as those shown in Fig. 2 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

The area specifying section 103 supplies the area information to the mixture ratio calculating section 104 and the synthesizing section 1001.

The mixture ratio calculation unit 104 supplies the mixture ratio α to the foreground / background separation unit 105 and the synthesis unit 1001.

The foreground / background separating unit 105 supplies the foreground component image to the synthesizing unit 1001.

The synthesizing unit 1001 supplies an arbitrary background image and a foreground / background separating unit 105 based on the mixing ratio α supplied from the mixing ratio calculating unit 104 and the region information supplied from the region specifying unit 103. A foreground component image is synthesized, and a synthesized image in which an arbitrary background image and a foreground component image are synthesized is output.

FIG. 80 is a diagram showing the structure of the synthesizing unit 1001. The background component generation unit 1021 generates a background component image based on the mixture ratio α and an arbitrary background image, and supplies the background component image to the mixed region image synthesis unit 1022.

The mixed area image synthesizing section 1022 generates a mixed area synthesized image by synthesizing the background component image supplied from the background component generating section 1021 and the foreground component image, and generates the mixed area synthesized image. Image synthesis unit 1023
To supply.

[0563] The image synthesizing unit 1023 calculates the
The foreground component image, the mixed area synthesized image supplied from the mixed area image synthesizing unit 1022, and an arbitrary background image are synthesized to generate and output a synthesized image.

[0564] As described above, the synthesizing unit 1001 can synthesize a foreground component image with an arbitrary background image.

An image obtained by synthesizing a foreground component image with an arbitrary background image based on the mixture ratio α, which is a feature quantity, is more natural than an image obtained by simply synthesizing pixels.

FIG. 81 is a block diagram showing still another configuration of the function of the signal processing unit for adjusting the amount of motion blur. While the signal processing unit illustrated in FIG. 2 sequentially performs the region specification and the calculation of the mixture ratio α, the signal processing unit illustrated in FIG. 81 performs the region specification and the calculation of the mixture ratio α in parallel.

[0567] The same portions as those in the function shown in the block diagram of FIG. 2 are denoted by the same reference numerals, and description thereof will be omitted.

[0568] The input image is supplied to the mixture ratio calculating section 1101, foreground / background separating section 1102, area specifying section 103, and object extracting section 101.

[0566] Based on the input image, the mixture ratio calculation unit 1101 calculates the estimated mixture ratio when the pixel belongs to the covered background area and the estimated mixture ratio when the pixel belongs to the uncovered background area. The ratio is calculated for each of the pixels included in the input image, and the estimated mixture ratio when the calculated pixel is assumed to belong to the covered background area,
The estimated mixture ratio when the pixel belongs to the uncovered background area is supplied to the foreground / background separation unit 1102.

FIG. 82 is a block diagram showing an example of the configuration of the mixture ratio calculation section 1101.

The estimated mixture ratio processing unit 401 shown in FIG.
This is the same as the estimated mixture ratio processing unit 401 shown in FIG. The estimated mixture ratio processing unit 402 shown in FIG. 82 is the same as the estimated mixture ratio processing unit 402 shown in FIG.

The estimated mixture ratio processing unit 401 calculates an estimated mixture ratio for each pixel by an operation corresponding to the model of the covered background area based on the input image, and outputs the calculated estimated mixture ratio.

The estimated mixture ratio processing unit 402 calculates an estimated mixture ratio for each pixel by an operation corresponding to the model of the uncovered background area based on the input image, and outputs the calculated estimated mixture ratio.

[0574] The foreground / background separation unit 1102 calculates the estimated mixture ratio supplied from the mixture ratio calculation unit 1101 when the pixel belongs to the covered background area.
Based on the estimated mixture ratio when it is assumed that the pixel belongs to the uncovered background area, and the area information supplied from the area specifying unit 103, a foreground component image is generated from the input image, and the generated foreground component image is generated. The motion blur adjustment unit 106 and the selection unit 107 are supplied.

FIG. 83 is a block diagram showing an example of the structure of the foreground / background separation unit 1102.

The same parts as those in the foreground / background separation unit 105 shown in FIG. 58 are denoted by the same reference numerals, and description thereof will be omitted.

[0577] Based on the region information supplied from the region identification unit 103, the selection unit 1121 calculates the estimated mixture ratio supplied from the mixture ratio calculation unit 1101 when the pixel belongs to the covered background region, and One of the estimated mixture ratios when the pixel is assumed to belong to the uncovered background area is selected, and the selected estimated mixture ratio is supplied to the separation unit 601 as the mixture ratio α.

The separation unit 601 extracts a foreground component and a background component from the pixel values of the pixels belonging to the mixed area based on the mixture ratio α and the area information supplied from the selection unit 1121, and extracts the extracted foreground component. Is supplied to the synthesizing unit 603, and the background component is supplied to the synthesizing unit 605.

The separating section 601 can have the same configuration as that shown in FIG.

The combining section 603 combines and outputs the foreground component images. The combining unit 605 combines and outputs a background component image.

The motion-blur adjusting unit 106 shown in FIG. 81 can have the same configuration as that shown in FIG. 2. Based on the area information and the motion vector, the foreground component image supplied from the foreground / background separating unit 1102 is obtained. Is adjusted, and a foreground component image in which the amount of motion blur is adjusted is output.

[0583] The selection unit 107 shown in Fig. 81 is for example a foreground / background separation unit 1 based on a selection signal corresponding to the user's selection.
One of the foreground component image supplied from 102 and the foreground component image supplied from the motion blur adjusting unit 106 and having the adjusted amount of motion blur is selected, and the selected foreground component image is output.

[0583] As described above, the signal processing unit shown in Fig. 81 adjusts the amount of motion blur included in an image corresponding to a foreground object included in the input image and outputs the adjusted image. Can be. The signal processing unit shown in FIG. 81 can calculate the mixture ratio α as the buried information and output the calculated mixture ratio α, as in the first embodiment.

FIG. 84 is a block diagram showing another configuration of the function of the signal processing unit for combining the foreground component image with an arbitrary background image. The signal processing unit shown in FIG. 79 serially specifies the region and calculates the mixture ratio α, whereas the signal processing unit illustrated in FIG. 84 performs the region specification and the calculation of the mixture ratio α in parallel.

[0585] The same portions as those shown in the block diagram of FIG. 81 are denoted by the same reference numerals, and description thereof will be omitted.

[0586] The mixture ratio calculation unit 1101 shown in FIG. 84 estimates the estimated mixture ratio when the pixel belongs to the covered background area based on the input image, and assumes that the pixel belongs to the uncovered background area. The estimated mixture ratio in the case is calculated for each of the pixels included in the input image, and the estimated mixture ratio in the case where the calculated pixel belongs to the covered background region, and the pixel belongs to the uncovered background region Is supplied to the foreground / background separation unit 1102 and the synthesis unit 1201.

The foreground / background separation unit 1102 shown in FIG.
The estimated mixture ratio when the pixel is assumed to belong to the covered background region, the estimated mixture ratio when the pixel is assumed to belong to the uncovered background region, and the region identification unit, supplied from the mixture ratio calculation unit 1101. Based on the area information supplied from 103,
A foreground component image is generated from the input image, and the generated foreground component image is supplied to the combining unit 1201.

[0588] The synthesizing section 1201 includes a mixing ratio calculating section 1101.
, The estimated mixture ratio when the pixel is assumed to belong to the covered background region, and the estimated mixture ratio when the pixel is assumed to belong to the uncovered background region, the region supplied from the region identification unit 103. Based on the information, an arbitrary background image is synthesized with the foreground component image supplied from the foreground / background separation unit 1102, and a synthesized image in which the arbitrary background image and the foreground component image are synthesized is output.

FIG. 85 is a diagram showing the structure of the synthesizing section 1201. Parts that are the same as the functions illustrated in the block diagram of FIG. 80 are given the same numbers, and descriptions thereof are omitted.

[0590] Based on the area information supplied from the area specifying section 103, the selection section 1221 calculates the estimated mixture ratio supplied from the mixture ratio calculation section 1101 when the pixel belongs to the covered background area, and One of the estimated mixture ratios when the pixel is assumed to belong to the uncovered background area is selected, and the selected estimated mixture ratio is used as the mixture ratio α as the background component generation unit 10
21.

[0590] The background component generation unit 1021 shown in FIG.
A background component image is generated based on the mixture ratio α and an arbitrary background image supplied from the selection unit 1221, and is supplied to the mixed region image synthesis unit 1022.

[0593] Mixed area image synthesizing section 1022 shown in FIG. 85
Generates a mixed area composite image by synthesizing the background component image supplied from the background component generation unit 1021 and the foreground component image, and supplies the generated mixed area composite image to the image synthesis unit 1023.

[0593] The image synthesizing unit 1023 calculates the
The foreground component image, the mixed area synthesized image supplied from the mixed area image synthesizing unit 1022, and an arbitrary background image are synthesized to generate and output a synthesized image.

As described above, the synthesizing unit 1201 can synthesize a foreground component image with an arbitrary background image.

Although the mixing ratio α has been described as the ratio of the background component included in the pixel value, it may be the ratio of the foreground component included in the pixel value.

Although the direction of movement of the foreground object has been described from left to right, it is needless to say that the direction is not limited to that direction.

In the above, an example has been described in which an image of a real space having three-dimensional space and time axis information is projected onto a two-dimensional space and time and space having time axis information using a video camera. The present invention is not limited to this example, and corrects the distortion caused by the projection when more first-dimensional first information is projected onto less second-dimensional second information. It can be applied to extracting significant information, or combining images more naturally.

The sensor is not limited to a CCD but is a solid-state image sensor, for example, a BBD (Bucket Brigade Device),
CID (Charge Injection Device) or CPD (Charge
Priming Device) may be used, and the sensor is not limited to a sensor in which detection elements are arranged in a matrix.
A sensor in which the detection elements are arranged in a line may be used.

As shown in FIG. 1, the recording medium on which the program for performing the signal processing of the present invention is recorded is a magnetic disk on which the program is recorded, which is distributed in order to provide the program to the user separately from the computer. 51 (including floppy (registered trademark) disk), optical disk 52
(CD-ROM (Compact Disc-Read Only Memory), DVD (Digita
l Versatile Disc), magneto-optical disc 53 (M
D (including Mini-Disc) (trademark)) or a package medium including a semiconductor memory 54 and the like, and a program provided to a user in a state where the program is incorporated in a computer in advance. RO
M22, a hard disk included in the storage unit 28, and the like.

[0600] In this specification, the step of describing a program recorded on a recording medium may be performed not only in chronological order according to the described order, but also in chronological order. This also includes processing executed in parallel or individually.

[0601]

According to the image processing apparatus, method, and program of the present invention, a plurality of weights are assigned in correspondence with the range of the mixture ratio indicating the mixture state in the pixel data of a plurality of objects in the real world. Weighting information indicating the generated individual weight is generated, and the weight indicated by the weighting information is calculated between each pixel of the frame of interest of the image data and each pixel of an adjacent frame adjacent to the frame of interest of the image data. Is detected, and is output as weighted difference image data corresponding to the frame of interest. The weighted difference is determined based on the motion vector indicating the relative motion between the pixel data of the frame of interest and the pixel data of the adjacent frame. The relative positions of the weighted difference image data of A target block consisting of at least one pixel centered on each pixel of interest of the weighted difference image data, correlation with a corresponding block of at least one pixel of the weighted difference image data between adjacent frames is calculated,
The weighting is output as the weighted difference image correlation data, and for each predetermined unit including at least one pixel of the weighted difference image correlation data, the weight that maximizes the correlation between the weighted difference image data is detected and corresponds to the unit of the frame of interest. Since the mixture ratio is output as a mixture ratio, it is possible to detect a mixture ratio indicating a state of mixture of a plurality of objects.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration example of a signal processing unit.

FIG. 2 is a block diagram illustrating a signal processing unit.

FIG. 3 is a diagram illustrating imaging by a sensor.

FIG. 4 is a diagram illustrating an arrangement of pixels.

FIG. 5 is a diagram illustrating the operation of a detection element.

FIG. 6 is a diagram illustrating an image obtained by capturing an object corresponding to a moving foreground and an object corresponding to a stationary background.

FIG. 7 is a diagram illustrating a background area, a foreground area, a mixed area, a covered background area, and an uncovered background area.

FIG. 8 is a model diagram in which pixel values of adjacent pixels arranged in one line in an image obtained by capturing an object corresponding to a stationary foreground and an object corresponding to a stationary background are developed in the time direction. It is.

FIG. 9 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 10 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 11 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 12 is a diagram illustrating an example in which pixels in a foreground area, a background area, and a mixed area are extracted.

FIG. 13 is a diagram showing the correspondence between pixels and models in which pixel values are developed in the time direction.

FIG. 14 is a model diagram in which pixel values are developed in a time direction and a period corresponding to a shutter time is divided.

FIG. 15 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 16 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 17 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 18 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 19 is a flowchart illustrating a process of adjusting the amount of motion blur.

FIG. 20 is a block diagram showing a configuration of an area specifying unit 103.

FIG. 21 is a diagram illustrating an image when an object corresponding to the foreground is moving.

FIG. 22 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 23 is a model diagram in which pixel values are developed in a time direction and a period corresponding to a shutter time is divided.

FIG. 24 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 25 is a diagram illustrating conditions for region determination.

FIG. 26 is a diagram illustrating an example of a result of specifying an area by the area specifying unit 103;

FIG. 27 is a diagram illustrating an example of a result of specifying an area by the area specifying unit 103;

FIG. 28 is a flowchart illustrating an area specifying process.

FIG. 29 is a block diagram showing another configuration of the area specifying unit 103.

FIG. 30 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 31 is a diagram illustrating an example of a background image.

FIG. 32 is a block diagram illustrating a configuration of a binary object image extraction unit 302.

FIG. 33 is a diagram illustrating calculation of a correlation value.

FIG. 34 is a diagram illustrating calculation of a correlation value.

FIG. 35 is a diagram illustrating an example of a binary object image.

FIG. 36 is a block diagram illustrating a configuration of a time change detection unit 303.

FIG. 37 is a diagram for explaining determination by an area determination unit 342.

FIG. 38 is a diagram illustrating an example of determination by the time change detection unit 303.

FIG. 39 is a flowchart illustrating an area identification process performed by an area determination unit 103.

FIG. 40 is a flowchart illustrating details of a region determination process.

FIG. 41 is a block diagram showing still another configuration of the area specifying unit 103.

FIG. 42 is a block diagram illustrating a configuration of a robust unit 361.

FIG. 43 is a diagram illustrating motion compensation by a motion compensation unit 381.

FIG. 44 is a diagram illustrating motion compensation by a motion compensation unit 381.

FIG. 45 is a flowchart illustrating an area specifying process.

FIG. 46 is a flowchart illustrating details of robust processing.

FIG. 47 is a block diagram showing a configuration of a mixture ratio calculator 104.

FIG. 48 is a block diagram illustrating a configuration of an estimated mixture ratio processing unit 401.

FIG. 49 is a model diagram in which pixel values are developed in a time direction and a period corresponding to a shutter time is divided.

FIG. 50 is a model diagram illustrating a weighted difference.

FIG. 51 is a model diagram illustrating a weighted difference.

FIG. 52 is a model diagram illustrating the relationship between a weighted difference and a motion vector.

FIG. 53 is a model diagram illustrating calculation of a correlation value;

FIG. 54 is a diagram illustrating an example of a block of pixels for which a calculation of a correlation value is performed.

FIG. 55 is a block diagram showing another configuration of the mixture ratio calculator 104.

FIG. 56 is a flowchart illustrating processing for calculating a mixture ratio.

FIG. 57 is a flowchart illustrating a process of estimating a mixture ratio corresponding to a covered background area.

FIG. 58 is a block diagram illustrating an example of a configuration of a foreground / background separation unit 105.

FIG. 59 is a diagram showing an input image, a foreground component image, and a background component image.

FIG. 60 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 61 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 62 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 63 is a block diagram illustrating an example of a configuration of a separation unit 601.

FIG. 64 is a diagram illustrating an example of a separated foreground component image and a background component image.

FIG. 65 is a flowchart illustrating a process of separating a foreground and a background.

FIG. 66 is a block diagram illustrating an example of a configuration of a motion-blur adjusting unit 106.

FIG. 67 is a diagram illustrating a processing unit.

FIG. 68 expands pixel values of a foreground component image in the time direction,
FIG. 3 is a model diagram obtained by dividing a period corresponding to a shutter time.

FIG. 69 expands pixel values of a foreground component image in the time direction,
FIG. 3 is a model diagram obtained by dividing a period corresponding to a shutter time.

FIG. 70: Expands the pixel values of the foreground component image in the time direction,
FIG. 3 is a model diagram obtained by dividing a period corresponding to a shutter time.

FIG. 71 develops pixel values of a foreground component image in the time direction,
FIG. 3 is a model diagram obtained by dividing a period corresponding to a shutter time.

72 is a diagram illustrating another configuration of the motion blur adjustment unit 106. FIG.

73 is a flowchart illustrating a process of adjusting the amount of motion blur included in the foreground component image by the motion blur adjustment unit 106. FIG.

74 is a block diagram illustrating another configuration of the motion blur adjustment unit 106. FIG.

FIG. 75 is a diagram illustrating an example of a model that specifies correspondence between pixel values and foreground components.

FIG. 76 is a diagram illustrating calculation of a foreground component.

FIG. 77 is a diagram illustrating calculation of a foreground component.

FIG. 78 is a flowchart illustrating a process for removing motion blur in the foreground.

FIG. 79 is a block diagram showing another configuration of the function of the signal processing unit.

FIG. 80 is a diagram showing a configuration of a synthesis unit 1001.

FIG. 81 is a block diagram showing still another configuration of the function of the signal processing unit.

FIG. 82 is a block diagram showing a configuration of a mixture ratio calculation unit 1101.

FIG. 83 is a block diagram showing a configuration of a foreground / background separation unit 1102.

FIG. 84 is a block diagram showing still another configuration of the function of the signal processing unit.

FIG. 85 is a diagram showing a configuration of a synthesizing unit 1201.

[Explanation of symbols]

21 CPU, 22 ROM, 23 RAM, 26 input unit, 27 output unit, 28 storage unit, 29 communication unit, 51 magnetic disk, 52 optical disk, 53
Magneto-optical disk, 54 semiconductor memory, 101
Object extraction unit, 102 Motion detection unit, 103
Area specifying unit, 104 mixture ratio calculating unit, 105 foreground / background separating unit, 106 motion blur adjusting unit, 107
Selection unit, 201 frame memory, 202-1 to 202-4 static / movement determination unit, 203-1 to 203-3
Area determination unit, 204 determination flag storage frame memory, 205 synthesis unit, 206 determination flag storage frame memory, 301 background image generation unit, 302 2
Value object image extraction section, 303 time change detection section, 321 correlation value calculation section, 322 threshold value processing section, 341 frame memory, 342 area determination section, 361 robust conversion section, 381 motion compensation section,
382 switch, 383-1 to 383-N frame memory, 384-1 to 384-N weighting unit, 385 integrating unit, 401 estimated mixture ratio processing unit,
402 estimated mixture ratio processing unit, 403 mixture ratio determination unit,
421 frame memory, 422 weight generator, 42
3 weighted frame difference calculation unit, 424 motion compensation unit, 425 frame memory, 426 correlation value calculation unit, 427 maximum value determination unit, 441 selection unit, 44
2 selection unit, 601 separation unit, 602 switch,
603 synthesis section, 604 switch, 605 synthesis section, 621 frame memory, 622 separation processing block, 623 frame memory, 631 uncovered area processing section, 632 covered area processing section,
633 synthesis unit, 634 synthesis unit, 801 processing unit determination unit, 802 modeling unit, 803 equation generation unit, 804 addition unit, 805 operation unit, 8
06 Motion blur addition unit, 807 selection unit, 821
Selection unit, 901 processing unit determination unit, 902 modeling unit, 903 equation generation unit, 904 operation unit,
905 correction unit, 906 motion blur addition unit, 907
Selection section, 1001 synthesis section, 1021 background component generation section, 1022 mixed area image synthesis section, 1023
Image synthesis unit, 1101 Mixing ratio calculation unit, 1102
Foreground / background separator, 1121 selector, 1201 synthesizer, 1221 selector

Continued on the front page (72) Inventor Takashi Sawao 6-7-35 Kita Shinagawa, Shinagawa-ku, Tokyo Inside Sony Corporation (72) Inventor Takahiro Nagano 6-35-35 Kita Shinagawa, Shinagawa-ku, Tokyo Sony (72) Inventor Naoki Fujiwara 6-7-35 Kita-Shinagawa, Shinagawa-ku, Tokyo Sony Corporation (72) Inventor Toru Miyake 6-35, Kita-Shinagawa, Shinagawa-ku, Tokyo Sony Corporation (72) Inventor Seiji Wada 6-7-35 Kita-Shinagawa, Shinagawa-ku, Tokyo F-term in Sony Corporation (reference) 5B057 AA20 BA02 CA01 CA08 CA12 CA16 CB01 CB08 CB12 CB16 CC03 CE04 CE08 CE17 DA06 DB02 DB06 DB09 DC34 DC36 5C022 AB00 AB55 AC69 5C023 AA06 AA11 AA37 BA02 BA04 BA11 CA01 DA02 DA04 5L096 AA02 AA06 BA08 CA04 CA14 DA01 EA39 FA34 GA08 HA03 HA04

Claims (44)

[Claims] 1. An image processing apparatus for processing image data consisting of a predetermined number of pixel data obtained by an image sensor having a predetermined number of pixels having a time integration effect, wherein the pixel data of a plurality of objects in the real world is provided. Weighting information generating means for generating a plurality of weights and generating weighting information indicating the generated individual weights in correspondence with the range of the value of the mixing ratio indicating the mixed state in, and each of the frames of interest of the image data Between the pixel and each pixel of an adjacent frame adjacent to the frame of interest of the image data, detects a weighted difference based on the weight indicated by the weighting information, as weighted difference image data corresponding to the frame of interest Weighted difference image data detecting means to be output, the pixel data of the frame of interest and the adjacent frame The relative position between the weighted difference image data of the frame of interest and the weighted difference image data of the adjacent frame is adjusted according to a motion vector indicating a relative movement between the pixel data and the pixel data. Calculates a correlation between a target block of at least one pixel centered on each target pixel of the weighted difference image data of the target frame and a corresponding block of at least one pixel of the weighted difference image data of the adjacent frame, and assigns a weight. Weighted difference image correlation data detection means for outputting as difference image correlation data; and for each predetermined unit consisting of at least one pixel of the weighted difference image correlation data, the correlation between the weighted difference image data is maximized. Detecting a weight and outputting the weight as the mixture ratio corresponding to the unit of the frame of interest An image processing apparatus comprising: 2. The weighting information generating means sequentially generates the weighting information indicating a plurality of different weights respectively corresponding to the plurality of different mixing ratios, and the weighted difference image data detecting means is different from each other. In response to the plurality of weights, the weighted difference is detected and output as the weighted difference image data corresponding to the frame of interest, and the weighted difference image correlation data detecting means corresponds to the plurality of different weights Then, the correlation is calculated and output as weighted difference image correlation data, and the mixture ratio detection unit calculates, from among the plurality of different weights, the weight that maximizes the correlation between the weighted difference image data. Detecting and outputting as the mixture ratio corresponding to the predetermined unit of the frame of interest. The image processing device according to claim 1. 3. The weighted difference image correlation data detecting means, as the correlation, the target block consisting of at least one pixel centered on each target pixel of the weighted difference image data of the target frame, and the adjacent frame 2. The image processing apparatus according to claim 1, wherein an absolute value of a difference between the weighted difference image data and the corresponding block including at least one pixel is calculated and output as the weighted difference image correlation data. 4. The weighted difference image correlation data detecting means, as the correlation, the target block including at least one pixel centered on each target pixel of the weighted difference image data of the target frame, and the adjacent frame 2. The image processing apparatus according to claim 1, wherein a sum of absolute differences between the weighted difference image data and the corresponding block including at least one pixel is calculated and output as the weighted difference image correlation data. 5. The weighted difference image data detecting means, wherein the pixel of the frame of interest is formed in a mixed area in which a plurality of the objects are mixed, and is formed on a front end side of a foreground object in a movement direction. When belonging to the covered background area, between each pixel of the frame of interest and each pixel of the frame before the frame of interest, a weighted difference by the weight indicated by the weighting information is detected, and the frame of interest is detected. Each pixel of the frame of interest when the pixel of the frame of interest belongs to a mixed area where a plurality of the objects are mixed and belongs to an uncovered background area formed on the rear end side in the movement direction of the foreground object And the weight indicated by the weighting information between each pixel of the frame next to the frame of interest The image processing apparatus according to claim 1, wherein a weighted difference is detected by: 6. A foreground area formed of a foreground object of the plurality of objects in the image data, a background area formed of a background object of the plurality of objects in the image data, and a plurality of the objects A mixed background area, which is a covered background area formed on the front end side of the foreground object in the movement direction, and the mixed area is formed on the rear end side of the foreground object in the movement direction. Further includes area information generating means for specifying an uncovered background area, and generating area information indicating the foreground area, the background area, and the mixed area including the covered background area and the uncovered background area. The image according to claim 1, characterized in that: Processing equipment. 7. The apparatus according to claim 1, further comprising a motion vector detecting unit configured to detect the motion vector indicating a relative motion between the pixel data of the frame of interest and the pixel data of the adjacent frame. 2. The image processing device according to 1. 8. The apparatus according to claim 1, wherein the mixture ratio detection unit detects the mixture ratio of the pixels of the frame of interest belonging to a mixture region where a plurality of the objects are mixed. Image processing device. 9. The image processing apparatus according to claim 1, further comprising: a separation unit configured to separate at least a foreground object among the plurality of objects from the pixel data based on the mixture ratio. apparatus. 10. The image processing apparatus according to claim 9, further comprising a motion blur adjusting means for adjusting the amount of motion blur of the separated foreground object. 11. The image processing apparatus according to claim 9, further comprising combining means for combining another desired object and the separated foreground object based on the mixture ratio.
An image processing apparatus according to claim 1. 12. An image processing method for processing image data consisting of a predetermined number of pixel data obtained by an image sensor having a predetermined number of pixels having a time integration effect, wherein the pixel data of a plurality of objects in the real world is provided. Weighting information generating step of generating a plurality of weights and generating weighting information indicating the generated individual weights in association with the range of the value of the mixing ratio indicating the mixed state in Between the pixel and each pixel of an adjacent frame adjacent to the frame of interest of the image data, detects a weighted difference based on the weight indicated by the weighting information, as weighted difference image data corresponding to the frame of interest A step of detecting a weighted difference image data to be output; According to a motion vector indicating a relative motion between the pixel data of the adjacent frame and the relative position between the weighted difference image data of the frame of interest and the weighted difference image data of the adjacent frame. Calculating a correlation between a target block of at least one pixel centered on each target pixel of the weighted difference image data of the target frame and a corresponding block of at least one pixel of the weighted difference image data of the adjacent frame. Detecting the correlation data between the weighted difference images to be output as the correlation data between the weighted difference images; The weight corresponding to the unit of the frame of interest is detected. A mixing ratio detection step of outputting as a mixing ratio. 13. The weighting information generating step sequentially generates the weighting information indicating the plurality of different weights corresponding to the plurality of different mixing ratios, respectively, wherein the weighting information differs in the weighting difference image data detecting step. In response to the plurality of weights, the weighted difference is detected and output as the weighted difference image data corresponding to the frame of interest. Then, the correlation is calculated and output as weighted difference image correlation data. In the mixing ratio detection step, among the plurality of different weights, the weight at which the correlation between the weighted difference image data is maximum is Is detected and the predetermined The image processing method according to claim 12, wherein the image is output as the mixture ratio corresponding to a unit. 14. In the weighted difference image correlation data detecting step, the correlation block includes at least one pixel centered on each pixel of interest in the weighted difference image data of the frame of interest and the adjacent frame as the correlation. The image processing method according to claim 12, wherein an absolute value of a difference between the weighted difference image data and the corresponding block including at least one pixel is calculated and output as the weighted difference image correlation data. 15. In the weighted difference image correlation data detecting step, as the correlation, the focused block including at least one pixel centered on each focused pixel of the weighted difference image data of the focused frame; 13. The image processing method according to claim 12, wherein a sum of absolute differences between the weighted difference image data and the corresponding block including at least one pixel is calculated and output as the weighted difference image correlation data. . 16. In the weighted difference image data detecting step, the pixel of the frame of interest is a mixed area where a plurality of the objects are mixed,
When belonging to the covered background area formed on the front end side in the movement direction of the foreground object, the weighting information indicates between each pixel of the frame of interest and each pixel of the frame before the frame of interest. The weighted difference due to the weight is detected, and the pixel of the frame of interest is a mixed area in which a plurality of the objects are mixed, and the uncovered pixel is formed on the rear end side in the movement direction of the foreground object. When belonging to a background area, between each pixel of the frame of interest and each pixel of the frame next to the frame of interest, a weight difference by the weight indicated by the weight information is detected. 13. The image processing method according to claim 12, wherein: 17. A foreground area including a foreground object among the plurality of objects in the image data, a background area including a background object among the plurality of objects in the image data,
A mixed area in which a plurality of the objects are mixed, a covered background area formed on the front end side in the movement direction of the foreground object, and the mixed area, a rear end in the movement direction of the foreground object Area information for specifying an uncovered background area formed on the side of the section and generating area information indicating the foreground area, the background area, and the mixed area including the covered background area and the uncovered background area The image processing method according to claim 12, further comprising a generating step. 18. The method according to claim 18, further comprising detecting a motion vector indicating a relative motion between the pixel data of the frame of interest and the pixel data of the adjacent frame. 13. The image processing method according to item 12. 19. The mixture ratio detection step according to claim 12, wherein the mixture ratio of the pixels of the frame of interest belonging to a mixture region in which the plurality of objects are mixed is detected. Image processing method. 20. The image processing apparatus according to claim 12, further comprising a separating step of separating at least a foreground object among the plurality of objects from the pixel data based on the mixture ratio. Method. 21. The image processing method according to claim 20, further comprising a motion blur adjusting step of adjusting an amount of motion blur of the separated foreground object. 22. The image processing method according to claim 20, further comprising a synthesizing step of synthesizing another desired object and the separated foreground object based on the mixture ratio. 23. A program for processing image data consisting of a predetermined number of pixel data obtained by an image sensor having a predetermined number of pixels having a time integration effect, wherein the pixel data of a plurality of objects in the real world is provided. Weighting information generating step of generating a plurality of weights and generating weighting information indicating the generated individual weights in association with the range of the value of the mixing ratio indicating the mixed state in Between the pixel and each pixel of an adjacent frame adjacent to the frame of interest of the image data, detects a weighted difference based on the weight indicated by the weighting information, as weighted difference image data corresponding to the frame of interest Detecting the weighted difference image data to be output; and the pixel data of the frame of interest and the adjacent In accordance with a motion vector indicating a relative motion between the pixel data of the frame, the relative position of the weighted difference image data of the frame of interest and the weighted difference image data of the adjacent frame are aligned, Calculating a correlation between a block of interest consisting of at least one pixel centered on each pixel of interest of the weighted difference image data of the frame of interest and a corresponding block consisting of at least one pixel of the weighted difference image data of the adjacent frame; A step of detecting the correlation data between the weighted difference images, which is output as the correlation data between the weighted difference images; and the correlation between the weighted difference image data is maximized for each predetermined unit including at least one pixel of the correlation data between the weighted difference images. Detecting the weight, the mixing corresponding to the unit of the frame of interest And a mixing ratio detecting step of outputting the ratio as a ratio. 24. The weighting information generating step sequentially generates the weighting information indicating a plurality of different weights corresponding to the plurality of different mixing ratios, respectively, wherein the weighting information differs in the weighting difference image data detecting step. In response to the plurality of weights, the weighted difference is detected and output as the weighted difference image data corresponding to the frame of interest. Then, the correlation is calculated and output as weighted difference image correlation data. In the mixing ratio detection step, among the plurality of different weights, the weight at which the correlation between the weighted difference image data is maximum is Is detected and the predetermined The recording medium according to claim 23, wherein the recording medium is output as the mixture ratio corresponding to a unit. 25. The weighted difference image correlation data detecting step, wherein, as the correlation, the target block including at least one pixel centered on each target pixel of the weighted difference image data of the target frame; 24. The recording medium according to claim 23, wherein a difference absolute value between the weighted difference image data and the corresponding block including at least one pixel is calculated and output as the weighted difference image correlation data. 26. The weighted difference image correlation data detecting step, wherein the correlation includes, as the correlation, the target block including at least one pixel centered on each target pixel of the weighted difference image data of the target frame, and the adjacent frame. 24. The recording medium according to claim 23, wherein a sum of absolute differences between the weighted difference image data and the corresponding block including at least one pixel is calculated and output as the weighted difference image correlation data. 27. In the weighted difference image data detecting step, the pixel of the frame of interest is a mixed area in which a plurality of the objects are mixed,
When belonging to the covered background area formed on the front end side in the movement direction of the foreground object, the weighting information indicates between each pixel of the frame of interest and each pixel of the frame before the frame of interest. The weighted difference due to the weight is detected, and the pixel of the frame of interest is a mixed area in which a plurality of the objects are mixed, and the uncovered pixel is formed on the rear end side in the movement direction of the foreground object. When belonging to a background area, between each pixel of the frame of interest and each pixel of the frame next to the frame of interest, a weight difference by the weight indicated by the weight information is detected. 24. The recording medium according to claim 23. 28. The program according to claim 28, wherein the program comprises: a foreground area including a foreground object among the plurality of objects in the image data; a background area including a background object among the plurality of objects in the image data; A mixed background area in which the object is mixed, a covered background area formed on the front end side in the movement direction of the foreground object, and the mixed area, which is a rear end in the movement direction of the foreground object. Area information generation for identifying an uncovered background area formed on the side and generating area information indicating the foreground area, the background area, and the mixed area including the covered background area and the uncovered background area Characterized by further comprising a step 24. The recording medium according to claim 23. 29. The program according to claim 29, further comprising a motion vector detecting step of detecting the motion vector indicating a relative motion between the pixel data of the frame of interest and the pixel data of the adjacent frame. The recording medium according to claim 23, wherein 30. The mixture ratio detection step, wherein the mixture ratio of the pixel of the frame of interest belonging to a mixture region where a plurality of the objects are mixed is detected. Recording medium. 31. The program according to claim 23, wherein the program further comprises a separation step of separating at least a foreground object among the plurality of objects from the pixel data based on the mixture ratio. The recording medium according to the above. 32. The program according to claim 3, further comprising a motion blur adjusting step of adjusting a motion blur amount of the separated foreground object.
2. The recording medium according to 1. 33. The program according to claim 31, wherein the program further includes a combining step of combining another desired object and the separated object serving as the foreground based on the mixture ratio. recoding media. 34. A computer for processing image data consisting of a predetermined number of pixel data obtained by an image pickup device having a predetermined number of pixels having a time integration effect, wherein a computer mixes a plurality of objects in the real world with the pixel data. In correspondence with the value range of the mixture ratio indicating the state, a plurality of weights are generated, and a weighting information generating step of generating weighting information indicating the generated individual weights, and each pixel of the target frame of the image data, Detecting a weighted difference based on the weight indicated by the weighting information between each pixel of an adjacent frame adjacent to the frame of interest of the image data, and outputting the weighted difference image data corresponding to the frame of interest. Detecting the weighted difference image data; and determining the pixel data of the frame of interest and the adjacent frame. The relative positions of the weighted difference image data of the frame of interest and the weighted difference image data of the adjacent frame are aligned according to a motion vector indicating a relative movement between the pixel data of the frame and the pixel data of the frame. Calculating a correlation between a target block of at least one pixel centered on each target pixel of the weighted difference image data of the target frame and a corresponding block of at least one pixel of the weighted difference image data of the adjacent frame. Detecting the correlation data between the weighted difference images to be output as correlation data between the weighted difference images; and for each predetermined unit consisting of at least one pixel of the correlation data between the weighted difference images, the correlation between the weighted difference image data is maximum. And the mixture ratio corresponding to the unit of the frame of interest. And a mixing ratio detecting step of outputting the output. 35. The weighting information generating step, wherein the weighting information indicating the plurality of different weights respectively corresponding to the plurality of different mixing ratios is sequentially generated, and the weighting information differs in the weighting difference image data detecting step. In response to the plurality of weights, the weighted difference is detected and output as the weighted difference image data corresponding to the frame of interest. Then, the correlation is calculated and output as weighted difference image correlation data. In the mixing ratio detection step, among the plurality of different weights, the weight at which the correlation between the weighted difference image data is maximum is Is detected and the predetermined 35. The program according to claim 34, wherein the program is output as the mixture ratio corresponding to a unit. 36. In the weighted difference image correlation data detecting step, the target block including at least one pixel centered on each target pixel of the weighted difference image data of the target frame as the correlation, and the adjacent frame 35. The program according to claim 34, wherein a difference absolute value between the weighted difference image data and the corresponding block including at least one pixel is calculated and output as the weighted difference image correlation data. 37. The weighted difference image correlation data detecting step, wherein the correlation includes, as the correlation, the target block including at least one pixel centered on each target pixel of the weighted difference image data of the target frame; 35. The program according to claim 34, wherein a sum of absolute differences between the weighted difference image data and the corresponding block including at least one pixel is calculated and output as the weighted difference image correlation data. 38. In the weighted difference image data detecting step, the pixel of the frame of interest is a mixed area where a plurality of the objects are mixed,
When belonging to the covered background area formed on the front end side in the movement direction of the foreground object, the weighting information indicates between each pixel of the frame of interest and each pixel of the frame before the frame of interest. The weighted difference due to the weight is detected, and the pixel of the frame of interest is a mixed area in which a plurality of the objects are mixed, and the uncovered pixel is formed on the rear end side in the movement direction of the foreground object. When belonging to a background area, between each pixel of the frame of interest and each pixel of the frame next to the frame of interest, a weight difference by the weight indicated by the weight information is detected. 35. The program according to claim 34, wherein 39. A foreground area formed of a foreground object of the plurality of objects in the image data, a background area formed of a background object of the plurality of objects in the image data,
A mixed area in which a plurality of the objects are mixed, a covered background area formed on the front end side in the movement direction of the foreground object, and the mixed area, a rear end in the movement direction of the foreground object Area information for specifying an uncovered background area formed on the side of the section and generating area information indicating the foreground area, the background area, and the mixed area including the covered background area and the uncovered background area 35. The program according to claim 34, further comprising a generating step. 40. The method of claim 40, further comprising detecting a motion vector indicating a relative motion between the pixel data of the frame of interest and the pixel data of the adjacent frame. The program according to claim 34. 41. The mixture ratio detection step, wherein the mixture ratio of the pixels of the frame of interest belonging to a mixture region where a plurality of the objects are mixed is detected. Program. 42. The program according to claim 34, further comprising a separating step of separating at least a foreground object among the plurality of objects from the pixel data based on the mixture ratio. 43. The program according to claim 42, further comprising a motion blur adjusting step of adjusting an amount of motion blur of the separated foreground object. 44. The program according to claim 42, further comprising a combining step of combining another desired object and the separated foreground object based on the mixture ratio.