JP2002373336A - Device and method for processing image, recording medium and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To process an image while corresponding to the mixture of a background image and an image of a moving object. SOLUTION: An area specifying part 103 respectively specifies the pixels of an inputted image to be any among a foreground area, a background area and a mixed area, and supplies area information showing which each pixel belongs to among the foreground area, the background area and the mixed area to a mixing ratio calculating part 104, a foreground and background separating part 105 and a motion blur eliminating part 106. A motion blur elimination image processing part 108 decides a class corresponding to each pixel of the inputted image in each foreground component image separated by the foreground and background separating part 105 and subjected to motion blur elimination by the motion blur eliminating part 106 and in each background component image corrected by a correcting part 107.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and method, a recording medium, and a program, and more particularly, to an image processing apparatus and method, a recording medium, and a method that consider a difference between a signal detected by a sensor and the real world. About the program.

[0002]

2. Description of the Related Art As one of processes for generating a higher-resolution image based on an input image, there is a classification adaptive process. As an example of the classification adaptive processing, a coefficient used in a process of generating a higher-resolution image is generated in advance in the spatial direction, and a higher-resolution image is generated in the spatial direction based on the generated coefficient. Processing.

FIG. 1 shows a conventional image processing for generating coefficients used in a classification adaptive process for generating an HD (High Definition) image from an SD (Standard Definition) image. FIG. 2 is a block diagram illustrating a configuration of the device.

[0004] The frame memory 11 stores an input image, which is an HD image, in frame units. Frame memory 11
Supplies the stored HD image to the weighted average unit 12 and the corresponding pixel acquisition unit 16.

The weighted averaging unit 12 performs a quarter weighted averaging of the HD images stored in the frame memory 11 to generate an SD image, and supplies the generated SD image to the frame memory 13.

[0006] The frame memory 13 stores the SD images supplied from the weighted averaging unit 12 in frame units, and supplies the stored SD images to the classifying unit 14 and the prediction tap obtaining unit 15.

[0007] The classifying unit 14 includes a class tap obtaining unit 21 and a waveform classifying unit 22, and classifies a target pixel of the SD image stored in the frame memory 13 as a target pixel. Class tap acquisition unit 21
Acquires a predetermined number of class taps, which are pixels of the SD image corresponding to the target pixel, from the frame memory 13 and supplies the acquired class taps to the waveform classification unit 22.

FIG. 2 is a diagram for explaining class taps acquired by the class tap acquisition unit 21. The class tap acquisition unit 21 acquires eleven class taps at a predetermined position as shown in FIG.

[0009] The waveform classifying unit 22 calculates
The target pixel is classified into one of a plurality of classes, and a class number corresponding to the classified class is supplied to the prediction tap acquisition unit 15. The waveform classification unit 22 classifies the target pixel into one of the 2048 classes based on the 11 class taps.

The prediction tap acquisition unit 15 acquires a predetermined number of prediction taps, which are pixels of an SD image, corresponding to the class classified from the frame memory 13 based on the class number, and acquires the acquired prediction taps and The class number is supplied to the corresponding pixel acquisition unit 16.

FIG. 3 is a diagram for explaining the prediction taps obtained by the prediction tap obtaining unit 15. Predicted tap acquisition unit 15
Obtains nine prediction taps at predetermined positions as shown in FIG.

The corresponding pixel acquiring unit 16 acquires a pixel of the HD image corresponding to the pixel value to be predicted from the frame memory 11 based on the prediction tap and the class number, and calculates the prediction tap, the class number, and the obtained prediction. The pixel of the HD image corresponding to the pixel value to be supplied is supplied to the normal equation generating unit 17.

The normal equation generation unit 17 generates a normal equation corresponding to each class based on the prediction tap, the class number, and the acquired pixel value to be predicted, and corresponding to the relationship between the prediction tap and the pixel value to be predicted. The generated normal equation corresponding to each class is generated and supplied to the coefficient calculation unit 18.

The coefficient calculating unit 18 includes a normal equation generating unit 17
, The coefficient set corresponding to each class is calculated, and the calculated coefficient set is supplied to the coefficient set memory 19 together with the class number.

The coefficient set memory 19 stores a calculated coefficient set corresponding to a class based on the class number.

FIG. 4 is a diagram for explaining the outline of the classification adaptive processing. In the classification adaptive processing, a corresponding SD image is generated from a teacher image, which is an HD image, by a quarter weighted average processing. The generated SD image is called a student image.

Next, a coefficient set for generating an HD image from an SD image is generated based on a teacher image as an HD image and a corresponding student image as an SD image. The coefficient set includes coefficients for generating an HD image from an SD image by linear prediction or the like.

The coefficient set and SD thus generated
A quadruple-density image is generated from the image by linear prediction or the like. The process of generating a higher density image or the like from the coefficient set and the input image is also referred to as mapping.

Generated 4x density image and corresponding HD
Based on the image, SNR comparison or visual qualitative evaluation is performed.

A coefficient set generated from a specific teacher image and a corresponding student image is referred to as a specific teacher image and a self-coefficient set of the corresponding student image. Mapping using a self coefficient set is called self-mapping. A coefficient set generated from a plurality of other teacher images and corresponding student images is referred to as a cross coefficient set.

On the other hand, an image obtained by capturing an object which is a foreground moving in front of a predetermined stationary background with a video camera has motion blur when the moving speed of the object is relatively high. And the foreground are mixed.

In the conventional classification adaptation process, as shown in FIG. 5, the learning process as described above is applied to all of the foreground, the background, and the portion where the foreground and the background are mixed. One coefficient set is generated,
Mapping processing is executed based on this coefficient set.

With reference to the flowchart of FIG. 6, a conventional learning process for generating coefficients used in a process of generating an HD image from an SD image will be described. Step S11
In, the image processing apparatus determines whether or not there is an unprocessed pixel in the student image, and if it is determined that there is an unprocessed pixel in the student image, the process proceeds to step S12, and the raster image is processed from the student image in the raster scan order. Obtain the target pixel.

In step S13, the classifying unit 1
The class tap acquisition unit 21 acquires the class tap corresponding to the target pixel from the student image stored in the frame memory 13. In step S14, the waveform classifying unit 22 of the class classifying unit 14 uses the class tap to
The pixel of interest is classified. In step S15,
The prediction tap acquisition unit 15 acquires a prediction tap corresponding to the pixel of interest from the student image stored in the frame memory 13 based on the classified class.

In step S16, the corresponding pixel acquiring unit 16 determines the frame memory 11 based on the classified class.
The pixel corresponding to the pixel value to be predicted is obtained from the teacher image stored in the.

In step S17, the normal equation generating unit 17 adds the prediction tap and the pixel value of the pixel corresponding to the pixel value to be predicted to the matrix for each class based on the classified class, and proceeds to step S11. Returning, the image processing apparatus repeatedly determines whether there is any unprocessed pixel.
The matrix for each class to which the pixel values of the pixels corresponding to the prediction tap and the pixel value to be predicted are added corresponds to a normal equation for calculating a coefficient for each class.

If it is determined in step S11 that there is no unprocessed pixel in the student image, the process proceeds to step S18, where the normal equation generating unit 17 determines whether the pixel value of the pixel corresponding to the prediction tap and the pixel value to be predicted is The set matrix for each class is supplied to the coefficient calculator 18. Coefficient calculator 1
8 calculates a coefficient set for each class by solving a matrix for each class in which the prediction tap and the pixel value of the pixel corresponding to the pixel value to be predicted are set.

In step S19, the coefficient calculator 18
Stores the calculated coefficients for each class in a coefficient set memory 19.
Output to The coefficient set memory 19 stores the coefficient set for each class, and the process ends.

FIG. 7 is a block diagram showing a configuration of a conventional image processing apparatus for generating an HD image from an SD image by a class classification adaptive process.

The frame memory 31 stores an input image, which is an SD image, on a frame basis. Frame memory 31
Supplies the stored SD image to the mapping unit 32.

The SD image input to the mapping unit 32 is
The information is supplied to the class classification unit 41 and the prediction tap acquisition unit 42.

The class classification section 41 is composed of a class tap acquisition section 51 and a waveform classification section 52, and classifies a target pixel, which is a target pixel, of the SD image stored in the frame memory 31. Class tap acquisition unit 5
1 acquires a predetermined number of class taps corresponding to the pixel of interest from the frame memory 31 and supplies the acquired class taps to the waveform classification unit 52.

The waveform classifying section 52 calculates the waveform
The target pixel is classified into one of a predetermined number of classes, and a class number corresponding to the classified class is supplied to the prediction tap acquisition unit 42.

The prediction tap obtaining section 42 obtains a predetermined number of prediction taps corresponding to the classified classes from the input image stored in the frame memory 31 based on the class numbers, and obtains the obtained prediction taps. And the class number are supplied to the prediction calculation unit 43.

The prediction calculation unit 43 calculates a coefficient set from the coefficient set stored in the coefficient set memory 33 based on the class number.
Get the coefficient set corresponding to the class. Prediction calculation unit 4
3 predicts the pixel value of the predicted image by linear prediction based on the coefficient set corresponding to the class and the prediction tap.
The prediction calculation unit 43 stores the predicted pixel value in the frame memory 3
4

The frame memory 34 stores the predicted pixel values supplied from the prediction operation section 43 and outputs an HD image in which the predicted pixel values are set.

FIG. 8 is a diagram showing the pixel values of the input image and the pixel values of the output image generated by the classification adaptive processing. As shown in FIG. 8, the image generated by the classification adaptive processing includes a waveform lost due to the band limitation of the SD image. In that sense, it can be said that the process of generating a higher resolution image by the class classification adaptive process creates resolution.

With reference to the flowchart of FIG. 9, a description will be given of a conventional image creation process of generating an HD image from an SD image by an image processing apparatus that executes the classification adaptive processing.

In step S31, the image processing apparatus determines whether there is an unprocessed pixel in the input image. If it is determined that there is an unprocessed pixel in the input image, the process proceeds to step S32, where the mapping unit 32 acquires the coefficient set stored in the coefficient set memory 33. In step S33, the image processing apparatus acquires a pixel of interest from the input image in the order of raster scan.

In step S34, the class classification unit 4
The first class tap acquisition unit 51 acquires a class tap corresponding to the pixel of interest from the input image stored in the frame memory 31. In step S35, the waveform classification unit 52 of the class classification unit 41 uses the
The pixel of interest is classified into one class.

In step S36, the prediction tap obtaining section 42 determines the frame memory 3 based on the classified class.
A prediction tap corresponding to the pixel of interest is acquired from the input image stored in No. 1.

In step S37, the prediction calculation section 43
Predicts a pixel value of a predicted image by linear prediction based on a coefficient set corresponding to the classified class and a prediction tap.

In step S38, the prediction calculation section 43
Outputs the predicted pixel value to the frame memory 34. The frame memory 34 stores the pixel value supplied from the prediction operation unit 43. The procedure returns to step S31, and repeats the determination as to whether there is an unprocessed pixel.

If it is determined in step S31 that there is no unprocessed pixel in the input image, the process proceeds to step S39, where the frame memory 34 outputs the stored predicted image in which the predicted value is set, and The process ends.

[0045]

When an object moves in front of a stationary background, not only the motion blur due to the mixing of the image of the moving object itself but also the image of the background and the image of the moving object move. Mixing occurs. Conventionally, processing an image corresponding to a mixture of a background image and an image of a moving object has not been considered.

The present invention has been made in view of such a situation, and an object of the present invention is to enable processing of an image in accordance with mixing of a background image and an image of a moving object. I do.

[0047]

According to the present invention, there is provided an image processing apparatus comprising: a mixed area formed by mixing a foreground object component forming a foreground object and a background object component forming a background object based on input image data; Region specifying means for specifying a foreground region composed of a foreground object component and a non-mixed region composed of one of a background region composed of a background object component constituting a background object, and outputting region specification information corresponding to the specification result And class determining means for determining a class corresponding to each pixel data of the input image data in accordance with the area specifying information.

The image processing apparatus may further include a generation unit that processes pixel data of the input image data according to the determined class and generates coefficients used in the classification adaptive processing.

The image processing apparatus further includes a conversion means for processing the pixel data of the input image data and converting the input image data into the output image data, based on the coefficient for each class, corresponding to the determined class. be able to.

The area specifying means further specifies the covered background area and the uncovered background area, and outputs area specifying information corresponding to the specified result.
The class determining means may determine a class corresponding to the identified covered background area or uncovered background area and corresponding to the pixel data of the input image data.

According to the image processing method of the present invention, a foreground object component constituting a foreground object and a background region component constituting a background object are mixed based on input image data, and a foreground object component. A region specifying step of specifying a foreground region and a non-mixed region formed by one of the background regions composed of background object components constituting the background object, and outputting region specifying information corresponding to the specified result;
A class determining step of determining a class corresponding to each pixel data of the input image data in accordance with the area specifying information.

The image processing method may further include a generation step of processing the pixel data of the input image data corresponding to the determined class and generating coefficients used in the classification adaptive processing.

The image processing method further includes a conversion step of processing the pixel data of the input image data and converting the input image data into the output image data based on the coefficient for each class corresponding to the determined class. be able to.

The process of the area specifying step further specifies the covered background area and the uncovered background area, and outputs area specifying information corresponding to the specified result. The processing of the class determining step includes the specified covered background area. A class corresponding to the area or the uncovered background area and corresponding to the pixel data of the input image data can be determined.

According to the recording medium program of the present invention, a foreground object component constituting a foreground object and a background object component constituting a background object are mixed based on input image data, An area specifying step of specifying a non-mixed area composed of one of a foreground area and a background area composed of a background object component forming a background object, and outputting area specifying information corresponding to the specified result; And a class determining step of determining a class corresponding to each pixel data of the input image data.

The program of the recording medium processes the pixel data of the input image data according to the determined class,
A generation step for generating coefficients used in the classification adaptive processing may be further provided.

The program of the recording medium further includes a conversion step of processing the pixel data of the input image data and converting the input image data into the output image data based on the coefficient for each class in accordance with the determined class. Can be provided.

The processing of the area specifying step further specifies the covered background area and the uncovered background area, and outputs area specifying information corresponding to the specified result. The processing of the class determining step includes the specified covered background area. A class corresponding to the area or the uncovered background area and corresponding to the pixel data of the input image data can be determined.

According to the program of the present invention, based on input image data, a foreground object component constituting a foreground object and a background region component constituting a background object are mixed, and a foreground region comprising a foreground object component. A region specifying step of specifying a non-mixed region constituted by one of the background regions composed of the background object components constituting the background object, and outputting region specifying information corresponding to the specified result; and And a class determining step of determining a class corresponding to each pixel data of the input image data.

The program may further include a generation step of processing the pixel data of the input image data in accordance with the determined class and generating coefficients used in the classification adaptive processing.

The program may further include a conversion step of processing the pixel data of the input image data and converting the input image data into the output image data based on the coefficient for each class, corresponding to the determined class. it can.

The processing of the area specifying step further specifies the covered background area and the uncovered background area, and outputs area specifying information corresponding to the specified result. The processing of the class determining step includes the specified covered background area. A class corresponding to the area or the uncovered background area and corresponding to the pixel data of the input image data can be determined.

In the image processing apparatus and method, recording medium, and program according to the present invention, foreground object components constituting a foreground object and background object components constituting a background object are mixed based on input image data. A mixed region, a foreground region composed of foreground object components, and a non-mixed region composed of one of a background region composed of background object components constituting a background object are specified, and region specifying information corresponding to the specified result is output. The class corresponding to each pixel data of the input image data is determined according to the area specifying information.

[0064]

FIG. 10 is a block diagram showing the configuration of an embodiment of an image processing apparatus according to the present invention. CP
U (Central Processing Unit) 71 is a ROM (Read Only)
Various types of processing are executed in accordance with a program stored in the memory 72 or the storage unit 78. RAM (Random
Access Memory) 73 stores programs and data executed by CPU 71 as appropriate. These CPU 71, R
The OM 72 and the RAM 73 are interconnected by a bus 74.

An input / output interface 75 is connected to the CPU 71 via a bus 74. The input / output interface 75 is connected to an input unit 76 including a keyboard, a mouse, a microphone, and the like, and an output unit 77 including a display, a speaker, and the like. The CPU 71 executes various processes in response to a command input from the input unit 76. Then, the CPU 71 outputs an image, a sound, or the like obtained as a result of the processing to the output unit 77.

The storage unit 78 connected to the input / output interface 75 is composed of, for example, a hard disk and stores programs executed by the CPU 71 and various data. The communication unit 79 communicates with an external device via the Internet or another network. In the case of this example, the communication unit 79 functions as an acquisition unit that captures the output of the sensor.

A program may be obtained via the communication unit 79 and stored in the storage unit 78.

The drive 80 connected to the input / output interface 75 includes a magnetic disk 91, an optical disk 9
2. When the magneto-optical disk 93, the semiconductor memory 94, and the like are mounted, they are driven to acquire programs and data recorded therein. The acquired programs and data are transferred to and stored in the storage unit 78 as necessary.

FIG. 11 is a block diagram showing a functional configuration of the image processing apparatus according to the present invention.

Note that it does not matter whether each function of the image processing apparatus is realized by hardware or software. That is, each block diagram in this specification may be considered as a hardware block diagram or a functional block diagram using software.

Here, the motion blur refers to a distortion included in an image corresponding to a moving object, which is caused by the movement of the object in the real world to be imaged and the characteristics of the image pickup by the sensor. .

In this specification, an image to be imaged, which corresponds to an object in the real world, is called an image object.

The input image supplied to the image processing apparatus is supplied to an object extracting unit 101, a region specifying unit 103, a mixture ratio calculating unit 104, and a foreground / background separating unit 105.

The object extracting unit 101 roughly extracts an image object corresponding to a foreground object included in the input image and supplies the extracted image object to the motion detecting unit 102. The object extraction unit 101
For example, by detecting the outline of the image object corresponding to the foreground object included in the input image, the image object corresponding to the foreground object is roughly extracted.

The object extracting unit 101 roughly extracts an image object corresponding to a background object included in the input image, and supplies the extracted image object to the motion detecting unit 102. The object extraction unit 101
For example, the image object corresponding to the background object is roughly extracted from the difference between the input image and the extracted image object corresponding to the foreground object.

Further, for example, the object extracting unit 101
Is configured to roughly extract an image object corresponding to a foreground object and an image object corresponding to a background object from a difference between a background image stored in a background memory provided therein and an input image. It may be.

The motion detecting section 102 calculates the motion vector of the image object corresponding to the coarsely extracted foreground object, for example, by a method such as a block matching method, a gradient method, a phase correlation method, and a pel recursive method. The calculated motion vector and the position information of the motion vector (information for specifying the position of the pixel corresponding to the motion vector) are supplied to the region specifying unit 103 and the motion blur removing unit 106.

The motion vector output by the motion detecting section 102 includes information corresponding to the motion amount v.

Further, for example, the motion detecting section 102 outputs the motion vector for each image object together with the pixel position information for specifying the pixel in the image object.
6 may be output.

The amount of movement v is a value representing the change in the position of the image corresponding to the moving object in units of pixel intervals. For example, the image of the object corresponding to the foreground is set to 4
When moving so as to be displayed at a position separated by pixels, the motion amount v of the image of the object corresponding to the foreground is
It is set to 4.

The area specifying unit 103 specifies each of the pixels of the input image as one of a foreground area, a background area, and a mixed area. Information that belongs to
(Referred to as area information) is supplied to the mixture ratio calculation unit 104, the foreground / background separation unit 105, and the motion blur removal unit 106.
Details of the foreground area, background area, or mixed area will be described later.

Based on the input image and the region information supplied from the region specifying unit 103, the mixture ratio calculation unit 104 calculates a mixture ratio (hereinafter, mixture ratio α) corresponding to the pixels included in the mixture region.
) And supplies the calculated mixture ratio to the foreground / background separation unit 105.

The mixture ratio α is a value indicating a ratio of a component of an image corresponding to a background object (hereinafter, also referred to as a background component) in a pixel value, as shown in Expression (3) described later.

The foreground / background separation unit 105 is provided with
3 and the mixture ratio calculation unit 104
Input image into a foreground component image composed of only components of the image corresponding to the foreground object (hereinafter also referred to as a foreground component) and a background component image composed of only the background component based on the mixture ratio α supplied from Is supplied to the motion blur removing unit 106, and the background component image is corrected by the correction unit 1
07.

The motion-blur removing unit 106 determines a processing unit indicating one or more pixels included in the foreground component image based on the motion amount v and the area information known from the motion vector. The processing unit is data that specifies a group of pixels to be processed for adjusting the amount of motion blur.

The motion blur removing unit 106 includes the foreground component image supplied from the foreground / background separating unit 105 and the motion detecting unit 102.
The motion blur included in the foreground component image is removed based on the motion vector and its position information supplied from the unit and the processing unit, and the foreground component image from which the motion blur has been removed is output to the motion blur removed image processing unit 108. .

The correction unit 107 determines whether the background component image
The pixel value of the pixel corresponding to the mixed area is corrected. The pixel value of the pixel corresponding to the mixed area of the background component image is calculated by removing the foreground component from the pixel value of the pixel of the mixed area before separation. Therefore, the pixel value of the pixel corresponding to the mixed area of the background component image is reduced in accordance with the mixing ratio α as compared with the pixel value of the pixel of the adjacent background area.

The correction unit 107 corrects such a decrease in gain corresponding to the mixture ratio α of the pixel value of the pixel corresponding to the mixed area in the background component image, and converts the corrected background component image into the motion blur removed image. It is supplied to the processing unit 108.

The motion blur removal image processing unit 108 individually processes the foreground component image from which the motion blur has been removed and the corrected background component image by the class classification adaptive processing.

For example, the motion blur removal image processing unit 108
For each of the foreground component image from which motion blur has been removed and the corrected background component image, a coefficient used in the classification adaptive processing for generating a higher-resolution image is generated.

For example, the motion blur removal image processing unit 108
Applies a classification adaptive process to each of the foreground component image from which motion blur has been removed and the corrected background component image,
Create higher resolution images.

Next, an input image supplied to the image processing apparatus will be described with reference to FIGS.

FIG. 12 is a diagram for explaining the image pickup by the sensor. The sensor is, for example, a CCD (C
harge-Coupled Device) It consists of a CCD video camera equipped with an area sensor. The object corresponding to the foreground in the real world moves horizontally between the object corresponding to the background and the sensor in the real world, for example, from left to right in the figure.

The sensor images the object corresponding to the foreground together with the object corresponding to the background. The sensor outputs the captured image in units of one frame. For example, the sensor outputs an image consisting of 30 frames per second. The exposure time of the sensor can be 1/30 second. The exposure time is a period from when the sensor starts converting the input light into electric charges to when the sensor finishes converting the input light into electric charges. Hereinafter, the exposure time is also referred to as a shutter time.

FIG. 13 is a diagram for explaining the arrangement of pixels. In FIG. 13, A to I indicate individual pixels. The pixels are arranged on a plane corresponding to the image.
One detection element corresponding to one pixel is arranged on the sensor. When the sensor captures an image, one detection element outputs a pixel value corresponding to one pixel forming the image. For example, the position of the detection element in the X direction corresponds to the position in the horizontal direction on the image, and the position of the detection element in the Y direction corresponds to the position in the vertical direction on the image.

As shown in FIG. 14, for example, a detection element such as a CCD converts input light into electric charge and accumulates the converted electric charge for a period corresponding to the shutter time. The amount of charge is substantially proportional to the intensity of the input light and the time during which the light is input. The detection element converts the charge converted from the input light into a period corresponding to the shutter time,
It adds to the charge already stored. That is, the detection element integrates the input light for a period corresponding to the shutter time, and accumulates an amount of charge corresponding to the integrated light. It can be said that the detection element has an integration effect with respect to time.

The electric charge accumulated in the detecting element is converted into a voltage value by a circuit (not shown), and the voltage value is further converted into a pixel value such as digital data and output. Therefore, the individual pixel values output from the sensor are obtained by integrating the spatially extended part of the object corresponding to the foreground or background with respect to the shutter time, and projecting the value projected into a one-dimensional space. Have.

The image processing apparatus extracts significant information buried in the output signal, for example, the mixture ratio α by the accumulation operation of the sensor.

FIG. 15 is a view for explaining images obtained by capturing an object corresponding to a moving foreground and an object corresponding to a stationary background. FIG.
(A) shows an object corresponding to a foreground with motion;
An image obtained by capturing an object corresponding to a stationary background is shown. In the example shown in FIG. 15A, the object corresponding to the foreground is moving horizontally from left to right with respect to the screen.

FIG. 15B is a model diagram in which pixel values corresponding to one line of the image shown in FIG. 15A are developed in the time direction. The horizontal direction in FIG.
(A) corresponds to the spatial direction X.

The pixels in the background area have their pixel values composed only of the background components, ie, the components of the image corresponding to the background object. Pixels in the foreground area are
The pixel value is composed of only the foreground components, that is, the components of the image corresponding to the foreground object.

The pixel value of the pixel in the mixed area is composed of a background component and a foreground component. Since the pixel value of the mixed area is composed of the background component and the foreground component, it can be said that the mixed area is a distortion area. The mixed area is further classified into a covered background area and an uncovered background area.

The covered background area is a mixed area at a position corresponding to the front end of the foreground object in the traveling direction with respect to the foreground area, and is an area where the background component is covered by the foreground as time passes. Say.

On the other hand, the uncovered background area is a mixed area at a position corresponding to the rear end of the foreground object in the traveling direction with respect to the foreground area. Means the area where appears.

As described above, the image including the foreground area, the background area, or the covered background area or the uncovered background area is input to the area specifying unit 103,
The input image is input to the mixture ratio calculation unit 104 and the foreground / background separation unit 105.

FIG. 16 is a diagram for explaining the background area, foreground area, mixed area, covered background area, and uncovered background area as described above. In the case of the image shown in FIG. 15, the background area is a stationary part, the foreground area is a moving part, the covered background area of the mixed area is a part that changes from the background to the foreground, The uncovered background area is a part that changes from the foreground to the background.

FIG. 17 shows, in the time direction, pixel values of adjacent pixels arranged in one line in an image obtained by capturing an object corresponding to a stationary foreground and an object corresponding to a stationary background. FIG.
For example, the pixels arranged on one line of the screen can be selected as the pixels arranged adjacently in one column.

The pixel values F01 to F04 shown in FIG. 17 are the pixel values of the pixels corresponding to the stationary foreground object. The pixel values B01 to B04 shown in FIG. 17 are the pixel values of the pixels corresponding to the stationary background object.

In the vertical direction in FIG. 17, time elapses from top to bottom in the figure. The position of the upper side of the rectangle in FIG. 17 corresponds to the time when the sensor starts converting the input light into electric charge, and the position of the lower side of the rectangle in FIG. 17 corresponds to the electric charge of the input light. Corresponding to the time at which the conversion of. That is, the distance from the upper side to the lower side of the rectangle in FIG. 17 corresponds to the shutter time.

In the following, a case where the shutter time and the frame interval are the same will be described as an example.

The horizontal direction in FIG. 17 corresponds to the spatial direction X described in FIG. More specifically, in the example shown in FIG. 17, the distance from the left side of the rectangle described as “F01” to the right side of the rectangle described as “B04” in FIG. 17 is eight times the pixel pitch, That is, it corresponds to the interval between eight consecutive pixels.

When the foreground object and the background object are stationary, the light input to the sensor does not change during the period corresponding to the shutter time.

Here, the period corresponding to the shutter time is 2
Divide into two or more periods of equal length. For example, if the number of virtual divisions is 4, the model diagram shown in FIG. 17 can be represented as the model shown in FIG. The number of virtual divisions is the amount of movement v of the object corresponding to the foreground within the shutter time.
It is set in correspondence with, for example. For example, the motion amount v which is 4,
, The virtual division number is set to 4, and the period corresponding to the shutter time is divided into four.

The uppermost row in the figure corresponds to the first divided period after the shutter is opened. The second row from the top in the figure corresponds to the second divided period from when the shutter has opened. The third row from the top in the figure corresponds to the third divided period from when the shutter has opened. The fourth row from the top in the figure corresponds to the fourth divided period from when the shutter has opened.

Hereinafter, the shutter time divided according to the movement amount v is also referred to as shutter time / v.

When the object corresponding to the foreground is stationary, the light input to the sensor does not change, so that the foreground component F01 / v is equal to the value obtained by dividing the pixel value F01 by the number of virtual divisions. Similarly, when the object corresponding to the foreground is stationary, the foreground component F02 / v is equal to a value obtained by dividing the pixel value F02 by the number of virtual divisions, and the foreground component F03 / v is obtained by dividing the pixel value F03 by the virtual value. Equal to the value divided by the number of divisions, the foreground component F04 / v is
It is equal to a value obtained by dividing the pixel value F04 by the number of virtual divisions.

When the object corresponding to the background is stationary, the light input to the sensor does not change. Therefore, the background component B01 / v is equal to the value obtained by dividing the pixel value B01 by the number of virtual divisions. Similarly, when the object corresponding to the background is stationary, the background component B02 / v is equal to the value obtained by dividing the pixel value B02 by the virtual division number, and B03 / v is the pixel value B03 obtained by dividing the pixel value B03 by the virtual division number. B04 / v is equal to the value obtained by dividing the pixel value B04 by the virtual division number.

That is, when the object corresponding to the foreground is stationary, the light corresponding to the object of the foreground input to the sensor does not change during the period corresponding to the shutter time. Foreground component F01 / v corresponding to time / v, and second foreground component F0 corresponding to shutter time / v after shutter is opened
1 / v, the third foreground component F01 / v corresponding to the shutter time / v when the shutter is opened, and the fourth foreground component F01 / v corresponding to the shutter time / v when the shutter is opened. Is
It has the same value. F02 / v to F04 / v also have the same relationship as F01 / v.

When the object corresponding to the background is stationary, the light corresponding to the background object input to the sensor does not change during the period corresponding to the shutter time. A background component B01 / v corresponding to v, a second background component B01 / v corresponding to the shutter time / v when the shutter is opened, and a third component corresponding to the shutter time / v when the shutter is opened. The background component B01 / v has the same value as the background component B01 / v corresponding to the fourth shutter time / v after the shutter is opened. B02 / v to B04 / v also have the same relationship.

Next, the case where the object corresponding to the foreground moves and the object corresponding to the background is stationary will be described.

FIG. 19 is a model diagram in which pixel values of pixels on one line including a covered background area are developed in the time direction when an object corresponding to the foreground moves rightward in the figure. is there. In FIG. 19, the motion amount v of the foreground is 4. Since one frame is a short time, the object corresponding to the foreground is a rigid body,
It can be assumed that they are moving at a constant speed. In FIG. 19, the image of the object corresponding to the foreground moves so as to be displayed four pixels to the right in the next frame with reference to a certain frame.

In FIG. 19, the leftmost pixel to the fourth pixel from the left belong to the foreground area. In FIG. 19, the fifth to seventh pixels from the left belong to the mixed area that is the covered background area. FIG.
In, the rightmost pixel belongs to the background area.

Since the object corresponding to the foreground moves so as to cover the object corresponding to the background with the passage of time, the components included in the pixel values of the pixels belonging to the covered background area correspond to the shutter time. At some point during the period, the background component changes to the foreground component.

For example, in FIG. 19, a pixel value indicated by a thick line frame
M is represented by equation (1).

M = B02 / v + B02 / v + F07 / v + F06 / v (1)

For example, the fifth pixel from the left includes a background component corresponding to one shutter time / v and a foreground component corresponding to three shutter times / v.
The mixture ratio α of the second pixel is 1/4. Since the sixth pixel from the left includes a background component corresponding to two shutter times / v and a foreground component corresponding to two shutter times / v, the mixture ratio α of the sixth pixel from the left is , 1/2. Since the seventh pixel from the left includes a background component corresponding to three shutter times / v and a foreground component corresponding to one shutter time / v, the mixture ratio α of the seventh pixel from the left is , 3/4.

Since it can be assumed that the object corresponding to the foreground is a rigid body and the foreground image moves at a constant speed so as to be displayed four pixels to the right in the next frame, for example, the fourth object from the left in FIG. The foreground component F07 / v of the pixel of the first shutter time / v after the shutter is opened is shown in FIG.
9 is equal to the foreground component of the fifth pixel from the left corresponding to the second shutter time / v from when the shutter has opened. Similarly, the foreground component F07 / v is the foreground component of the sixth pixel from the left in FIG. 19 corresponding to the third shutter time / v from when the shutter has opened, and the seventh pixel from the left in FIG. Of the pixel of
It is equal to the foreground component corresponding to the fourth shutter time / v after the shutter is opened.

Since it can be assumed that the object corresponding to the foreground is a rigid body and the foreground image moves at a constant speed so as to be displayed four pixels to the right in the next frame, for example, the third object from the left in FIG. The foreground component F06 / v of the pixel of the first shutter time / v from when the shutter has opened is shown in FIG.
It is equal to the foreground component of the fourth pixel from the left, corresponding to the second shutter time / v from when the shutter has opened. Similarly, the foreground component F06 / v is the foreground component of the fifth pixel from the left in FIG. 19 corresponding to the third shutter time / v from when the shutter has opened, and the sixth pixel from the left in FIG. Of the pixel of
It is equal to the foreground component corresponding to the fourth shutter time / v after the shutter is opened.

Since it can be assumed that the object corresponding to the foreground is a rigid body and the foreground image moves at a constant speed so as to be displayed four pixels to the right in the next frame, for example, the second object from the left in FIG. The foreground component F05 / v of the pixel of the first shutter time / v from when the shutter has opened is shown in FIG.
It is equal to the foreground component of the third pixel from the left corresponding to the second shutter time / v from when the shutter has opened. Similarly, the foreground component F05 / v is the foreground component of the fourth pixel from the left in FIG. 19 corresponding to the third shutter time / v from when the shutter has opened, and the fifth pixel from the left in FIG. Of the pixel of
It is equal to the foreground component corresponding to the fourth shutter time / v after the shutter is opened.

Since it can be assumed that the object corresponding to the foreground is a rigid body and the foreground image moves at a constant speed so as to be displayed four pixels to the right in the next frame, for example, the leftmost pixel in FIG. The foreground component F04 / v of the first shutter time / v from when the shutter is opened is the foreground component of the second pixel from the left in FIG. 19 corresponding to the second shutter time / v from when the shutter is opened. be equivalent to. Similarly, the foreground component F04 / v is the third pixel from the left in FIG.
The foreground component corresponding to the third shutter time / v after the shutter is opened and the foreground component corresponding to the fourth shutter time / v of the fourth pixel from the left in FIG. , Respectively.

Since the foreground area corresponding to the moving object includes the motion blur, it can be said that the area is a distortion area.

FIG. 20 is a model diagram in which pixel values of pixels on one line including the uncovered background area are developed in the time direction when the foreground moves rightward in the figure. In FIG. 20, the motion amount v of the foreground is
Is 4. Since one frame is a short time, it can be assumed that the object corresponding to the foreground is rigid and moves at a constant speed. In FIG. 20, the image of the object corresponding to the foreground moves to the right by four pixels in the next frame with respect to a certain frame.

In FIG. 20, the leftmost pixel to the fourth pixel from the left belong to the background area. In FIG. 20, the fifth to seventh pixels from the left belong to the mixed area that is the uncovered background. FIG.
In, the rightmost pixel belongs to the foreground area.

Since the object corresponding to the foreground, which has covered the object corresponding to the background, moves so as to be removed from the front of the object corresponding to the background over time, the pixels belonging to the uncovered background area At some point during the period corresponding to the shutter time, the component included in the value is changed from the foreground component to the background component.

For example, in FIG. 20, a pixel value with a thick frame
M ′ is represented by equation (2).

M ′ = F02 / v + F01 / v + B26 / v + B26 / v (2)

For example, since the fifth pixel from the left contains a background component corresponding to three shutter times / v and a foreground component corresponding to one shutter time / v, the fifth pixel from the left
The mixture ratio α of the third pixel is 3/4. Since the sixth pixel from the left includes a background component corresponding to two shutter times / v and a foreground component corresponding to two shutter times / v, the mixture ratio α of the sixth pixel from the left is , 1/2. Since the seventh pixel from the left includes a background component corresponding to one shutter time / v and a foreground component corresponding to three shutter times / v, the mixture ratio α of the seventh pixel from the left is , 1/4.

If the equations (1) and (2) are generalized, the pixel value M is expressed by the equation (3).

[0139]

(Equation 1) Here, α is a mixture ratio. B is a pixel value of the background, and Fi / v is a foreground component.

Since it can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed, and the movement amount v is 4, for example, the shutter of the fifth pixel from the left in FIG. First foreground component of shutter time / v F01 /
v is equal to the foreground component of the sixth pixel from the left in FIG. 20 corresponding to the second shutter time / v from when the shutter has opened. Similarly, F01 / v is the foreground component of the seventh pixel from the left in FIG. 20 corresponding to the third shutter time / v from when the shutter has opened, and the eighth pixel from the left in FIG. ,
It is equal to the foreground component corresponding to the fourth shutter time / v after the shutter is opened.

Since it can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed, and the number of virtual divisions is 4, for example, the shutter of the sixth pixel from the left in FIG. First foreground component F0 of shutter time / v
2 / v is equal to the foreground component of the seventh pixel from the left in FIG. 20 corresponding to the second shutter time / v from when the shutter has opened. Similarly, the foreground component F02 / v is equal to the foreground component of the eighth pixel from the left in FIG. 20 corresponding to the third shutter time / v from when the shutter has opened.

Since it can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed, and the motion amount v is 4, for example, the shutter of the seventh pixel from the left in FIG. First foreground component of shutter time / v F03 /
v is equal to the foreground component of the eighth pixel from the left in FIG. 20 corresponding to the second shutter time / v from when the shutter has opened.

In the description of FIGS. 18 to 20, the number of virtual divisions is four, but the number of virtual divisions corresponds to the amount of motion v. The movement amount v generally corresponds to the moving speed of the object corresponding to the foreground. For example, when the object corresponding to the foreground is moving so as to be displayed four pixels to the right in the next frame with respect to a certain frame, the motion amount v is set to 4. The virtual division number is set to 4 corresponding to the movement amount v. Similarly, for example, when the object corresponding to the foreground is moving so as to be displayed 6 pixels to the left in the next frame with respect to a certain frame, the motion amount v is set to 6, and the number of virtual divisions is , 6.

FIG. 21 and FIG.
The relationship between a mixed area including a foreground area, a background area, a covered background area, or an uncovered background area, and foreground components and background components corresponding to the divided shutter times is shown.

FIG. 21 shows an example in which pixels of a foreground area, a background area, and a mixed area are extracted from an image including a foreground corresponding to an object moving in front of a stationary background. In the example shown in FIG. 21, the object corresponding to the foreground is moving horizontally with respect to the screen.

The frame # n + 1 is a frame next to the frame #n, and the frame # n + 2 is a frame next to the frame # n + 1.

The pixels of the foreground area, the background area, and the mixed area extracted from any of the frames #n to # n + 2 are extracted, and the motion amount v is set to 4, and the pixel value of the extracted pixel is determined. FIG. 22 shows a model developed in the time direction.

The pixel value of the foreground area is composed of four different foreground components corresponding to the shutter time / v period since the object corresponding to the foreground moves. For example, the leftmost pixel among the pixels in the foreground area shown in FIG. 22 includes F01 / v, F02 / v, F03 / v, and F04 / v. That is, pixels in the foreground area include motion blur.

[0149] Since the object corresponding to the background is stationary, the light corresponding to the background input to the sensor does not change during the period corresponding to the shutter time. In this case, the pixel value of the background area does not include motion blur.

The pixel values of the pixels belonging to the mixed area composed of the covered background area or the uncovered background area are composed of a foreground component and a background component.

Next, when the image corresponding to the object is moving, the pixel values of pixels that are adjacently arranged in one row in a plurality of frames and located at the same position on the frame are determined in the time direction. The following is an explanation of the model developed in. For example, when an image corresponding to an object is moving horizontally with respect to the screen, pixels arranged on one line of the screen can be selected as pixels arranged adjacently in one row.

FIG. 23 is a view showing three adjacent frames of three frames of an image obtained by capturing an object corresponding to a stationary background, which are pixels located at the same position on the frame. FIG. 4 is a model diagram in which values are developed in a time direction. Frame #n is the next frame after frame # n-1, and frame # n + 1 is the next frame after frame #n. Other frames are similarly referred to.

The pixel values B01 to B12 shown in FIG. 23 are the pixel values of the pixels corresponding to the stationary background object. Since the object corresponding to the background is stationary, the pixel values of the corresponding pixels do not change in frames # n-1 to n + 1. For example, the pixel in frame #n and the pixel in frame # n + 1 corresponding to the position of the pixel having the pixel value of B05 in frame # n-1 each have the pixel value of B05.

FIG. 24 is a diagram showing three pixels arranged adjacent to one another in three frames of an image obtained by capturing an object corresponding to a foreground moving to the right in the figure together with an object corresponding to a stationary background. FIG. 4 is a model diagram in which pixel values of pixels at the same position on a frame are developed in the time direction. The model shown in FIG. 24 includes a covered background area.

In FIG. 24, it can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed, and the foreground image moves so as to be displayed four pixels to the right in the next frame. The quantity v is 4, and the number of virtual divisions is 4.

For example, the foreground component of the leftmost pixel of frame # n-1 in FIG. 24 for the first shutter time / v from when the shutter has opened is F12 / v, and the second component from the left in FIG. The foreground component of the pixel of the second shutter time / v after the shutter is opened also becomes F12 / v. The foreground component of the third pixel from the left in FIG. 24 corresponding to the third shutter time / v from when the shutter has opened, and the fourth pixel from the left in FIG.
The foreground component of the fourth shutter time / v after the shutter is opened is F12 / v.

The foreground component of the leftmost pixel of frame # n-1 in FIG. 24 corresponding to the second shutter time / v from when the shutter has opened is F11 / v, which is the second to the left in FIG. The foreground component of the pixel at the third shutter time / v from when the shutter has opened is also F11 / v. The foreground component of the third pixel from the left in FIG. 24 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is F11 / v.

The foreground component of the leftmost pixel of frame # n-1 in FIG. 24 at the third shutter time / v from when the shutter has opened is F10 / v, which is the second to the left in FIG. The foreground component of the pixel at the fourth shutter time / v from when the shutter has opened is also F10 / v. The fourth shutter time of the leftmost pixel of frame # n-1 in FIG. 24 from when the shutter has opened
The foreground component of / v is F09 / v.

Since the object corresponding to the background is stationary, the background component of the second pixel from the left of frame # n-1 in FIG. 24 corresponding to the first shutter time / v from when the shutter has opened is B01 / v. The background components of the third pixel from the left of frame # n-1 in FIG. 24 corresponding to the first and second portions of the shutter time / v from when the shutter has opened are B02 / v. FIG.
The background component of the fourth pixel from the left of the middle frame # n-1 corresponding to the first to third shutter time / v from when the shutter has opened is B03 / v.

In frame # n-1 in FIG. 24, the leftmost pixel belongs to the foreground area, and the second to fourth pixels from the left belong to the mixed area which is a covered background area.

The fifth through twelfth pixels from the left of frame # n-1 in FIG. 24 belong to the background area, and their pixel values are B04 through B11, respectively.

The first through fifth pixels from the left of frame #n in FIG. 24 belong to the foreground area. Frame #n
The foreground component of shutter time / v in the foreground area of
One of F05 / v to F12 / v.

It can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed. Since the foreground image moves so as to be displayed four pixels to the right in the next frame, frame #n in FIG. The foreground component of the fifth pixel from the left of the first shutter time / v from when the shutter has opened is
F12 / v, the foreground component of the sixth pixel from the left in FIG. 24 corresponding to the second shutter time / v from when the shutter has opened is also
It becomes F12 / v. The foreground component of the seventh pixel from the left in FIG. 24 corresponding to the third shutter time / v from when the shutter has opened, and the fourth shutter time of the eighth pixel from the left in FIG. 24 having the shutter opened. The foreground component of / v is F12 / v.

The foreground component of the fifth pixel from the left of frame #n in FIG. 24 corresponding to the second shutter time / v from when the shutter has opened is F11 / v, and the sixth pixel from the left in FIG. 24 is F11 / v. The foreground component of the pixel at the third shutter time / v from when the shutter has opened is also F11 / v. The foreground component of the seventh pixel from the left in FIG. 24 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is F11 / v.

The foreground component of the fifth pixel from the left of frame #n in FIG. 24 corresponding to the third shutter time / v from when the shutter has opened is F10 / v, and the sixth pixel from the left in FIG. 24 is F10 / v. The foreground component of the pixel at the fourth shutter time / v from when the shutter has opened is also F10 / v. The foreground component of the fifth pixel from the left of frame #n in FIG. 24 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is F09 / v.

Since the object corresponding to the background is stationary, the background component of the sixth pixel from the left of frame #n in FIG. 24 corresponding to the first portion of the shutter time / v from when the shutter has opened is B05 / v Becomes 7 from the left of frame #n in FIG.
The background component of the first pixel at the first and second shutter time / v from when the shutter has opened is B06 / v. The background components of the eighth pixel from the left of frame #n in FIG. 24 corresponding to the first to third portions of the shutter time / v from when the shutter has opened are:
B07 / v.

In frame #n in FIG. 24, the sixth through eighth pixels from the left belong to a mixed area which is a covered background area.

The ninth to twelfth pixels from the left of frame #n in FIG. 24 belong to the background area, and the pixel value is
B08 to B11, respectively.

The first through ninth pixels from the left of frame # n + 1 in FIG. 24 belong to the foreground area. flame
The foreground component of the shutter time / v in the foreground area of # n + 1 is one of F01 / v to F12 / v.

It can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed. Since the foreground image moves so as to be displayed four pixels to the right in the next frame, frame #n in FIG. Of the ninth pixel from the left of +1
The foreground component of the first shutter time / v after the shutter is opened is F12 / v, and the foreground component of the tenth pixel from the left in FIG. It becomes F12 / v. The foreground component of the eleventh pixel from the left in FIG. 24 at the third shutter time / v from when the shutter has opened, and the fourth shutter time of the twelfth pixel from the left in FIG. 24 from the shutter opening. The foreground component of / v is F12 / v.

The foreground component of the ninth pixel from the left of frame # n + 1 in FIG. 24 corresponding to the second portion of the shutter time / v from when the shutter has opened is F11 / v, and as shown in FIG. From 1
The foreground component of the 0th pixel at the third shutter time / v from when the shutter has opened is also F11 / v. The foreground component of the eleventh pixel from the left in FIG. 24 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is F11 / v.

The foreground component of the ninth pixel from the left of frame # n + 1 in FIG. 24 corresponding to the third portion of the shutter time / v from when the shutter has opened is F10 / v, from the left in FIG. 4th shutter time / v after shutter opened for 10th pixel
Is also F10 / v. Frame # n + in FIG.
The foreground component of the ninth pixel from the left in FIG. 1 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is F09 / v.

Since the object corresponding to the background is stationary, the background component of the tenth pixel from the left of frame # n + 1 in FIG. 24 corresponding to the first portion of the shutter time / v from when the shutter has opened is B09 / v. The background components of the eleventh pixel from the left of frame # n + 1 in FIG. 24 corresponding to the first and second portions of the shutter time / v from when the shutter has opened are B10 / v. The background components of the twelfth pixel from the left of frame # n + 1 in FIG. 24 corresponding to the first to third portions of the shutter time / v from when the shutter has opened are B11 / v.

In the frame # n + 1 in FIG. 24, the tenth to twelfth pixels from the left correspond to the mixed area which is the covered background area.

FIG. 25 is a model diagram of an image in which foreground components are extracted from the pixel values shown in FIG.

FIG. 26 is a diagram showing three adjacent pixels of three frames of an image obtained by capturing a foreground corresponding to an object moving to the right in the figure together with a stationary background. FIG. 10 is a model diagram in which pixel values of pixels at the same position are developed in the time direction. In FIG.
An uncovered background area is included.

In FIG. 26, it can be assumed that the object corresponding to the foreground is a rigid body and is moving at a constant speed. Since the object corresponding to the foreground is moving so as to be displayed on the right side by four pixels in the next frame, the motion amount v is 4.

For example, the shutter time / v of the leftmost pixel of frame # n-1 in FIG.
The foreground component is F13 / v, and the second shutter time / v of the second pixel from the left in FIG.
Is also F13 / v. The third shutter time / v from when the shutter has opened for the third pixel from the left in FIG. 26
26, and the foreground component of the fourth pixel from the left in FIG. 26 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is F13 / v.

The foreground component of the second pixel from the left of frame # n-1 in FIG. 26 corresponding to the first shutter time / v from when the shutter has opened is F14 / v, and the third pixel from the left in FIG. 26 is F14 / v. The foreground component of the pixel of the second shutter time / v after the shutter is opened also becomes F14 / v. The foreground component of the third pixel from the left in FIG. 26 corresponding to the first portion of the shutter time / v from when the shutter has opened is F15 / v.

Since the object corresponding to the background is stationary, the leftmost pixel of frame # n-1 in FIG.
Second to fourth shutter time / v after shutter opens
Is B25 / v. Frame # n- in FIG.
The background components of the second pixel from the left, the third and fourth portions of the shutter time / v from when the shutter has opened are B26 / v
Becomes The background component of the third pixel from the left of frame # n-1 in FIG. 26 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is B27 / v.

In frame # n-1 in FIG. 26, the leftmost pixel to the third pixel belong to the mixed area which is the uncovered background area.

The fourth through twelfth pixels from the left of frame # n-1 in FIG. 26 belong to the foreground area. The foreground component of the frame is one of F13 / v to F24 / v.

The leftmost pixel to the fourth pixel from the left of frame #n in FIG. 26 belong to the background area, and the pixel value is
B25 to B28, respectively.

It can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed. Since the foreground image moves so as to be displayed four pixels to the right in the next frame, frame #n in FIG. The foreground component of the fifth pixel from the left of the first shutter time / v from when the shutter has opened is
F13 / v, the foreground component of the sixth pixel from the left in FIG. 26 corresponding to the second shutter time / v from when the shutter has opened is also
It becomes F13 / v. The foreground component of the seventh pixel from the left in FIG. 26 at the third shutter time / v from when the shutter has opened, and the fourth shutter time of the eighth pixel from the left in FIG. 26 since the shutter has opened. The foreground component of / v is F13 / v.

The foreground component of the sixth pixel from the left of frame #n in FIG. 26 corresponding to the first shutter time / v from when the shutter has opened is F14 / v, and the seventh pixel from the left in FIG. 26 is F14 / v. The foreground component of the second shutter time / v after the shutter is opened also becomes F14 / v. The foreground component of the eighth pixel from the left in FIG. 26 corresponding to the first shutter time / v from when the shutter has opened is F15 / v.

Since the object corresponding to the background is stationary, the second to fourth shutter times of the fifth pixel from the left of frame #n in FIG.
The background component of / v is B29 / v. Frame # in FIG. 26
The background components of the sixth pixel from the left of n corresponding to the third and fourth shutter time / v from when the shutter has opened are B30 / v. Of the seventh pixel from the left of frame #n in FIG.
The background component of the fourth shutter time / v from when the shutter has opened is B31 / v.

In frame #n in FIG.
The seventh through seventh pixels belong to a mixed area that is an uncovered background area.

The eighth through twelfth pixels from the left of frame #n in FIG. 26 belong to the foreground area. flame
The value corresponding to the shutter time / v period in the foreground area #n is one of F13 / v to F20 / v.

The leftmost pixel to the eighth pixel from the left of frame # n + 1 in FIG. 26 belong to the background area, and have pixel values B25 to B32, respectively.

It can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed. Since the foreground image moves so as to be displayed four pixels to the right in the next frame, frame #n in FIG. Of the ninth pixel from the left of +1
The foreground component of the first shutter time / v after the shutter is opened is F13 / v, and the foreground component of the tenth pixel from the left in FIG. It becomes F13 / v. The foreground component of the eleventh pixel from the left in FIG. 26 corresponding to the third shutter time / v from when the shutter has opened, and the twelfth pixel from the left in FIG. 26 having the fourth shutter time since the shutter has opened. The foreground component of / v is F13 / v.

The foreground component of the tenth pixel from the left of frame # n + 1 in FIG. 26 corresponding to the first shutter time / v from when the shutter has opened is F14 / v, and the eleventh pixel from the left in FIG. The foreground component of the pixel of the second shutter time / v after the shutter is opened also becomes F14 / v. The foreground component of the twelfth pixel from the left in FIG. 26 corresponding to the first shutter time / v from when the shutter has opened is F15 / v.

Since the object corresponding to the background is stationary, the ninth pixel from the left of frame # n + 1 in FIG. 26 corresponding to the 2nd to 4th background of the shutter time / v from when the shutter has opened. Is B33 / v. 26. The tenth pixel from the left of frame # n + 1 in FIG.
The background component of the 4th and 4th shutter time / v is B3
4 / v. The background component of the eleventh pixel from the left of frame # n + 1 in FIG. 26 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is B35 / v.

In frame # n + 1 in FIG. 26, the ninth to eleventh pixels from the left belong to the mixed area which is the uncovered background area.

The twelfth pixel from the left of frame # n + 1 in FIG. 26 belongs to the foreground area. The foreground components of the shutter time / v in the foreground area of frame # n + 1 are F13 / v to F13 / v
16 / v.

FIG. 27 is a model diagram of an image in which foreground components are extracted from the pixel values shown in FIG.

FIG. 28 shows a correspondence between an image divided for each pixel belonging to a foreground area, a background area, a covered background area, and an uncovered background area, and a model diagram in which pixel values of the pixels are developed in the time direction. FIG.

As shown in FIG.
Specifies a foreground area, a background area, a covered background area, and an uncovered background area of the input image.

FIG. 29 shows an image of a foreground area, an image of a background area, a foreground component image of a covered background area, a background component of a covered background area, a foreground component of an uncovered background area, and an uncovered background. FIG. 7 is a diagram showing a correspondence between an input image separated into background components of a region and a model diagram in which pixel values of pixels are developed in a time direction.

As shown in FIG. 29, in the input image, the foreground area, the background area, the covered background area, and the uncovered background area are specified by the area specifying unit 103. The input image is a foreground / background separation unit 10
5, based on the specified foreground area, background area, covered background area, and uncovered background area, and the mixture ratio α detected by the mixture ratio calculation unit 104, the image of the foreground area, the covered background area Foreground component image composed of the foreground component of the uncovered background area, and a background component composed of the image of the background area, the background component of the covered background area, and the background component of the uncovered background area Separated into images.

The separated foreground component image and background component image are processed for each image.

The foreground / background separation unit 105 converts the input image into a foreground area image, a background area image, a foreground component image of a covered background area, and a background component of a covered background area based on the area information and the mixture ratio α. The image, the foreground component image in the uncovered background area, and the background component image in the uncovered background area may be separated.

FIG. 30 is a diagram showing an example of an image divided into a foreground area, a background area, and a mixed area. The area specifying unit 103 specifies a foreground area, a background area, and a mixed area of the input image. The image processing apparatus can divide an input image into an image of a foreground area, an image of a background area, and an image of a mixed area based on area information indicating a foreground area, a background area, and a mixed area.

As shown in FIG. 31, the foreground / background separation unit 10
5 is based on the region information supplied from the region identification unit 103 and the mixture ratio α supplied from the mixture ratio calculation unit 104,
The image of the mixed area is separated into a foreground component image of the mixed area and a background component image of the mixed area.

As shown in FIG. 32, in the separated background component image, the pixel value of the mixed area is corrected, and in the separated foreground component image, motion blur is removed.

As shown in FIG. 33, the input image is divided into regions and separated into foreground components and background components. The separated input image is combined with the foreground component image and the background component image.

The motion blur included in the foreground component image is removed. In the background component image, the pixel value corresponding to the mixed area is corrected.

The foreground component image from which the motion blur has been removed and the corrected background component image are individually processed.

FIG. 34 is a flowchart for explaining the image processing of the image processing apparatus according to the present invention.

In step S101, the area specifying unit 1
Reference numeral 03 specifies a foreground area, a background area, a covered background area, and an uncovered background area of the input image based on the motion vector and its position information supplied from the motion detection unit 102 and the input image. The details of the area specifying process will be described later.

[0210] In step S102, the mixture ratio calculation unit 104 calculates the mixture ratio α based on the area information and the input image supplied from the area identification unit 103. The details of the process of calculating the mixture ratio α by the mixture ratio calculator 104 will be described later.

[0211] In step S103, the foreground / background separation unit 105 outputs the area information supplied from the area specifying unit 103 and the mixture ratio α supplied from the mixture ratio calculation unit 104.
, The input image is separated into a foreground component image composed of foreground components and a background component image composed of background components. The details of the image separation processing of the foreground / background separation unit 105 will be described later.

[0212] In step S104, the motion blur removing unit 106 determines whether the foreground / background separation unit 105 has the motion vector based on the motion vector and its position information supplied from the motion detection unit 102 and the region information supplied from the region identification unit 103. The motion blur of the supplied foreground component image is removed.

In step S105, the correction unit 107
Corrects the pixel value corresponding to the mixed area of the background component image supplied from the foreground / background separation unit 105.

[0214] In step S106, the motion-blur-removed image processing unit 108 performs image processing for each of the foreground component image from which the motion blur has been removed and the corrected background component image, and the process ends. Motion blur removal image processing unit 1
The details of the image processing executed by 08 will be described later.

As described above, the image processing apparatus according to the present invention separates an input image into a foreground component image and a background component image, removes motion blur from the foreground component image, and removes motion blur from the foreground component image. Image processing is executed for each image and background component image.

Hereinafter, the area specifying unit 103 and the mixture ratio calculating unit 1
04, foreground / background separation unit 105, motion blur removal unit 106,
The respective configurations of the motion blur removal image processing unit 108 will be described.

FIG. 35 is a block diagram showing an example of the structure of the area specifying unit 103. The area specifying unit 103 shown in FIG. 35 does not use a motion vector. The frame memory 201 stores an input image in frame units. In the frame memory 201, the processing target is frame #n
, The frame # n−2, which is the frame immediately before the frame #n, the frame # n−1, the frame #n, which is the frame immediately before the frame #n, the frame #n, the frame after the frame #n The frame # n + 1, which is a frame, and the frame # n + 2, which is two frames after the frame #n, are stored.

The still / moving judgment unit 202-1 determines the pixel value of the pixel of frame # n + 2 at the same position as the position of the pixel which is the object of specifying the area of frame #n, and the pixel value of frame #n. The pixel value of the pixel of frame # n + 1 at the same position as the position of the pixel to be specified in the image is stored in the frame memory 2
First, the absolute value of the difference between the read pixel values is calculated. The static / movement determining unit 202-1 determines whether the absolute value of the difference between the pixel value of the frame # n + 2 and the pixel value of the frame # n + 1 is greater than a preset threshold Th, When it is determined that the absolute value of the difference is larger than the threshold Th, a still / moving determination indicating a motion is supplied to the area determination unit 203-1. Frame #n
When it is determined that the absolute value of the difference between the pixel value of the pixel of +2 and the pixel value of the pixel of frame # n + 1 is equal to or smaller than the threshold Th, the static / movement determining unit 202-1 determines whether or not the static Judgment is made to the area judgment unit 2
03-1.

The still / movement determining unit 202-2 determines the pixel value of the pixel of the frame # n + 1 at the same position as the position of the pixel to be specified for the region of the frame #n on the image and the pixel value of the frame #n. The pixel value of the target pixel is read from the frame memory 201, and the absolute value of the difference between the pixel values is calculated. The static / moving determining unit 202-2 determines whether the absolute value of the difference between the pixel value of the frame # n + 1 and the pixel value of the frame #n is larger than a preset threshold Th. If the absolute value of the difference is determined to be larger than the threshold value Th, a still / moving determination indicating a motion is supplied to the region determination unit 203-1 and the region determination unit 203-2. When it is determined that the absolute value of the difference between the pixel value of the pixel of frame # n + 1 and the pixel value of the pixel of frame #n is equal to or smaller than the threshold Th, the still / moving determination unit 202-2 indicates that the camera is still. The static / movement determination is supplied to the region determination unit 203-1 and the region determination unit 203-2.

[0220] The still / moving judgment unit 202-3 is located at the same position as the pixel value of the pixel which is the target of specifying the region of the frame #n and the position of the pixel which is the target of specifying the region of the frame #n on the image. The pixel value of the pixel of frame # n-1 is stored in frame memory 2
01, the absolute value of the difference between the pixel values is calculated.
The static / moving determining unit 202-3 determines whether the absolute value of the difference between the pixel value of the frame #n and the pixel value of the frame # n-1 is greater than a preset threshold Th. If the absolute value of the difference is determined to be larger than the threshold Th, the static / moving determination indicating the motion is performed by the region determination unit 203-2 and the region determination unit 203.
-3. Pixel value of pixel of frame #n and frame
When it is determined that the absolute value of the difference between the pixel value of the pixel # n-1 and the pixel value is equal to or smaller than the threshold Th, the still / moving determination unit 202-3 performs the still / moving determination indicating stillness by the region determining unit 203-2. And to the area determination unit 203-3.

The still / movement determining unit 202-4 determines the pixel value of the pixel of frame # n-1 at the same position as the position of the pixel for specifying the area of frame #n on the image, and the pixel value of frame #n. The pixel value of the pixel of frame # n-2 at the same position as the position of the pixel which is the target of region identification on the image is stored in the frame memory 2
01, the absolute value of the difference between the pixel values is calculated.
The still / moving determination unit 202-4 determines whether the absolute value of the difference between the pixel value of the frame # n-1 and the pixel value of the frame # n-2 is greater than a preset threshold Th, When it is determined that the absolute value of the difference between the pixel values is greater than the threshold Th, a still / moving determination indicating motion is supplied to the region determination unit 203-3. Frame #n
When it is determined that the absolute value of the difference between the pixel value of the pixel −1 and the pixel value of the pixel of frame # n−2 is equal to or smaller than the threshold Th, the static / moving determining unit 202-4 determines that the static The motion determination is supplied to the area determination unit 203-3.

The area determining section 203-1 is composed of the static / moving determining section 20.
When the still / movement determination supplied from 2-1 indicates stillness and the still / movement determination supplied from the still / movement determination unit 202-2 indicates movement, the pixel that is the target of region identification in frame #n Is determined to belong to the uncovered background area, and the uncovered background area determination flag corresponding to the pixel whose area is determined is set to “1” indicating that the pixel belongs to the uncovered background area.

The area determination section 203-1 is a static / movement determination section 20.
When the still / movement determination supplied from 2-1 indicates a motion or the still / movement determination supplied from the still / movement determination unit 202-2 indicates stillness, the region is a target for area identification in frame #n. It is determined that the pixel does not belong to the uncovered background area, and “0” indicating that the pixel does not belong to the uncovered background area is set in the uncovered background area determination flag corresponding to the pixel whose area is determined.
Set.

[0224] The area judging section 203-1 determines "
The uncovered background area determination flag in which 1 ”or“ 0 ”is set is supplied to the determination flag storage frame memory 204.

[0225] The area determining section 203-2 is provided with the static / moving determining section 20.
When the still / movement determination supplied from 2-2 indicates stillness and the still / movement determination supplied from still / movement determination unit 202-3 indicates stillness, the pixel which is the target of region identification in frame #n Is determined to belong to the still area, and “1” indicating that the pixel belongs to the still area is set in the still area determination flag corresponding to the pixel whose area is determined.

The area determining section 203-2 is provided with the static / moving determining section 20.
When the static / movement determination supplied from 2-2 indicates a motion, or when the static / movement determination supplied from the static / movement determination unit 202-3 indicates a motion, the region is a target for area identification in frame #n. It is determined that the pixel does not belong to the still area, and “0” indicating that the pixel does not belong to the still area is set in the still area determination flag corresponding to the pixel whose area is determined.

The area determination section 203-2 determines "
The still area determination flag in which “1” or “0” is set is supplied to the determination flag storage frame memory 204.

The area determining section 203-2 is a static / moving determining section 20.
When the still / movement determination supplied from 2-2 indicates a movement and the still / movement determination supplied from the still / movement determination unit 202-3 indicates a movement, the pixel which is the target of the region identification in the frame #n Is determined to belong to the motion area, and “1” indicating that the pixel belongs to the motion area is set in the motion area determination flag corresponding to the pixel whose area is determined.

[0229] The area determining section 203-2 is provided with the static / moving determining section 20.
When the static / movement determination supplied from 2-2 indicates stillness, or when the static / movement determination supplied from static / movement determination unit 202-3 indicates stillness, it is a target for area identification in frame #n. It is determined that the pixel does not belong to the motion area, and “0” indicating that the pixel does not belong to the motion area is set in the motion area determination flag corresponding to the pixel whose area is determined.

[0230] The area judging section 203-2 outputs "
The motion area determination flag in which 1 ”or“ 0 ”is set is supplied to the determination flag storage frame memory 204.

[0231] The area determining section 203-3 is provided with the static / moving determining section 20.
When the still / moving judgment supplied from 2-3 indicates a motion and the still / moving judgment supplied from the still / moving judging section 202-4 indicates still, a pixel which is an area identification target in the frame #n Is determined to belong to the covered background area, and “1” indicating that the pixel belongs to the covered background area is set in the covered background area determination flag corresponding to the pixel whose area is determined.

The area determination section 203-3 is a static / motion determination section 20.
When the still / movement determination supplied from 2-3 indicates stillness or when the still / movement determination supplied from the still / movement determination unit 202-4 indicates motion, the region is a target for area identification in frame #n. It is determined that the pixel does not belong to the covered background area, and “0” indicating that the pixel does not belong to the covered background area is set in a covered background area determination flag corresponding to the pixel whose area is determined.

[0233] The area determination section 203-3 determines "
The covered background area determination flag in which 1 ”or“ 0 ”is set is stored in the determination flag storage frame memory 20.
4

The determination flag storage frame memory 204 is
Uncovered background area determination flag supplied from area determination section 203-1, still area determination flag supplied from area determination section 203-2, area determination section 203-
2 and a covered background area determination flag supplied from the area determination unit 203-3.

The determination flag storage frame memory 204 is
The stored uncovered background area determination flag, still area determination flag, moving area determination flag, and covered background area determination flag are stored in the combining unit 2.
05. Based on the uncovered background area determination flag, the still area determination flag, the motion area determination flag, and the covered background area determination flag supplied from the determination flag storage frame memory 204, the combining unit 205 Area information indicating that the area belongs to any of the covered background area, the still area, the moving area, and the covered background area is generated and supplied to the determination flag storage frame memory 206.

The determination flag storage frame memory 206
The area information supplied from the synthesizing unit 205 is stored, and the stored area information is output.

Next, an example of the processing of the area specifying unit 103 will be described with reference to FIG.
This will be described with reference to FIGS.

When the object corresponding to the foreground is moving, the position on the screen of the image corresponding to the object changes for each frame. As shown in FIG. 36, in frame #n, the image corresponding to the object located at the position indicated by Yn (x, y) is Yn + 1 (x, y) in frame # n + 1 which is the next frame. located at y).

FIG. 37 shows a model diagram in which the pixel values of pixels arranged in one row adjacent to the motion direction of the image corresponding to the foreground object are developed in the time direction. For example, when the motion direction of the image corresponding to the foreground object is horizontal with respect to the screen, the model diagram in FIG. 37 shows a model in which pixel values of adjacent pixels on one line are developed in the time direction.

In FIG. 37, the line in frame #n is the same as the line in frame # n + 1.

In frame #n, the foreground components corresponding to the objects included in the second through thirteenth pixels from the left are the sixth through seventeenth pixels from the left in frame # n + 1. included.

In frame #n, the pixels belonging to the covered background area are the eleventh to thirteenth pixels from the left, and the pixels belonging to the uncovered background area are the second to fourth pixels from the left. .
In frame # n + 1, the pixels belonging to the covered background area are the fifteenth through seventeenth pixels from the left, and the pixels belonging to the uncovered background area are the sixth through eighth pixels from the left.

In the example shown in FIG. 37, since the foreground component included in frame #n has moved by four pixels in frame # n + 1, the motion amount v is 4. The number of virtual divisions is
It corresponds to the movement amount v and is 4.

Next, changes in pixel values of pixels belonging to the mixed area before and after the frame of interest will be described.

In frame #n shown in FIG. 38 where the background is stationary and the foreground motion amount v is 4, the pixels belonging to the covered background area are the 15th through 17th pixels from the left.
The pixel of the th. Since the motion amount v is 4, the fifteenth through seventeenth pixels from the left in the immediately preceding frame # n-1 include only background components and belong to the background area. In the immediately preceding frame # n-2, the fifteenth through seventeenth pixels from the left include only background components and belong to the background area.

Here, since the object corresponding to the background is stationary, the pixel value of the fifteenth pixel from the left of frame # n-1 is the pixel value of the fifteenth pixel from the left of frame # n-2. Does not change from. Similarly, 1 from the left of frame # n-1
The pixel value of the sixth pixel does not change from the pixel value of the sixteenth pixel from the left of frame # n-2, and the pixel value of the seventeenth pixel from the left of frame # n-1 is frame # n- 17 from the left of 2
It does not change from the pixel value of the th pixel.

That is, the frame corresponding to the pixel belonging to the covered background area in frame #n
The pixels of # n-1 and frame # n-2 are composed of only background components, and the pixel values do not change. Therefore, the absolute value of the difference is almost zero. Therefore, frame # n-1 and frame # n-1 corresponding to the pixels belonging to the mixed area in frame #n
The still / movement determination for the pixel # n-2 is performed by the still / movement determination unit 202-4.
Is determined to be stationary.

Since the pixels belonging to the covered background area in frame #n include the foreground components, the pixel values are different from those of frame # n-1 consisting of only the background components. Therefore, the still / movement determination for the pixel belonging to the mixed area in frame #n and the corresponding pixel in frame # n-1 is determined to be a movement by the still / movement determination unit 202-3.

As described above, the region determining unit 203-3 is supplied with the result of the static / moving determination indicating the motion from the static / moving determining unit 202-3, and receives the static / moving determining unit 202-4 from the static / moving determining unit 202-4. When the result is supplied, it is determined that the corresponding pixel belongs to the covered background area.

In frame #n shown in FIG. 39 in which the background is stationary and the foreground motion amount v is 4, the pixels included in the uncovered background area are the second to fourth pixels from the left. Since the motion amount v is 4, in the next frame # n + 1, the second to fourth pixels from the left include only background components and belong to the background area. Further, in the next frame # n + 2, the second to fourth pixels from the left include only the background component and belong to the background area.

Here, since the object corresponding to the background is stationary, the pixel value of the second pixel from the left of frame # n + 2 is the pixel value of the second pixel from the left of frame # n + 1. Does not change from. Similarly, the pixel value of the third pixel from the left of frame # n + 2 does not change from the pixel value of the third pixel from the left of frame # n + 1, and the pixel value of the third pixel from the left of frame # n + 2 does not change. Does not change from the pixel value of the fourth pixel from the left of frame # n + 1.

That is, the pixels of frame # n + 1 and frame # n + 2 corresponding to the pixels belonging to the uncovered background area in frame #n are composed of only the background components and the pixel values do not change. The absolute value of the difference is almost zero. Therefore, the still / movement determination for the pixels of frame # n + 1 and frame # n + 2 corresponding to the pixels belonging to the mixed area in frame #n is performed by the still / movement determination unit 20
According to 2-1, it is determined to be stationary.

Since the pixels belonging to the uncovered background area in frame #n include foreground components,
A case where only the background component in frame # n + 1 is included,
Pixel values are different. Therefore, the still / movement determination for the pixels belonging to the mixed area in frame #n and the corresponding pixels in frame # n + 1 is determined to be a motion by the still / movement determination unit 202-2.

As described above, the region determining unit 203-1 is supplied with the result of the static / moving determination indicating the motion from the static / moving determining unit 202-2, and the static / dynamic determining unit 202-1 determines whether or not the static / moving determination indicating the stillness is present. When the result is supplied, it is determined that the corresponding pixel belongs to the uncovered background area.

FIG. 40 is a diagram showing the determination conditions of the area specifying unit 103 in frame #n. The pixel of frame # n-2 at the same position as the position of the pixel to be determined in frame #n on the image and the pixel of frame #n in the same position as the position of the pixel to be determined in frame #n The pixel of a certain frame # n-1 is determined to be still, and the pixel of the frame # n-1 and the pixel of the frame #n at the same position as the position of the pixel to be determined of the frame #n on the image Is determined to be a motion, the area specifying unit 103 determines that the pixel to be determined in the frame #n belongs to the covered background area.

The pixel of frame # n-1 and the pixel of frame #n at the same position as the position of the pixel to be determined for frame #n on the image are determined to be still, and the pixel of frame #n is determined to be still. When the pixel of the frame # n + 1 located at the same position as the position of the pixel to be determined for the frame #n on the image is determined to be still, the region identifying unit 103 determines the frame #n Is determined to belong to the still area.

The pixel of frame # n-1 and the pixel of frame #n at the same position as the position of the pixel to be determined in frame #n on the image are determined to be moving, and the pixel of frame #n And the pixel of frame # n + 1 located at the same position as the position of the pixel to be determined in frame #n on the image, is determined to be moving. Is determined to belong to the motion area.

The pixel of frame #n and the pixel of frame # n + 1 located at the same position on the image as the pixel to be determined of frame #n are determined to be moving, and the determination of frame #n is performed. The pixel of frame # n + 1 located at the same position as the position of the target pixel on the image, and the frame # n + located at the same position on the image of the pixel targeted for determination of frame #n When the two pixels are determined to be still, the region specifying unit 103 determines that the pixel to be determined in the frame #n belongs to the uncovered background region.

FIG. 41 is a diagram showing an example of a result of specifying an area by the area specifying unit 103. In FIG. In FIG. 41A, pixels determined to belong to the covered background area are displayed in white. In FIG. 41B, the pixels determined to belong to the uncovered background area are displayed in white.

In FIG. 41C, pixels determined to belong to the motion area are displayed in white. FIG.
In (D), the pixels determined to belong to the still area are displayed in white.

FIG. 42 is a diagram showing, as an image, the area information indicating the mixed area among the area information output from the frame memory 206 for storing the determination flag. In FIG. 42, the pixels determined to belong to the covered background area or the uncovered background area, that is, the pixels determined to belong to the mixed area, are displayed in white.
The area information indicating the mixed area output from the determination flag storage frame memory 206 indicates the mixed area and the part having the texture surrounded by the non-textured part in the foreground area.

Next, the area specifying process of the area specifying unit 103 will be described with reference to the flowchart in FIG. In step S201, the frame memory 201 acquires images of frames # n-2 to # n + 2 including the frame #n to be determined.

In step S202, static / movement determining section 2
02-3, the pixel of frame # n-1 and the pixel at the same position of frame #n determine whether or not it is still, and if it is determined that it is still, proceed to step S203, 2
Determines whether the pixel of frame #n and the pixel at the same position of frame # n + 1 are still.

If it is determined in step S203 that the pixel of frame #n and the pixel at the same position of frame # n + 1 are still, the process proceeds to step S204, where the region determination unit 20 determines
Step 3-2 sets “1” indicating that the pixel belongs to the still area in the still area determination flag corresponding to the pixel whose area is determined. The region determination unit 203-2 supplies the still region determination flag to the determination flag storage frame memory 204, and the procedure proceeds to step S205.

In step S202, frame # n-1
Pixel and the pixel at the same position of frame #n are determined to be moving, or in step S203, the pixel of frame #n and the pixel at the same position of frame # n + 1 are determined to be moving. In this case, since the pixel of frame #n does not belong to the still area, the process of step S204 is skipped, and the procedure proceeds to step S205.

[0266] In step S205, the static / movement determination unit 2
02-3, the pixel of frame # n-1 and the pixel at the same position of frame #n are used to determine whether or not a motion is detected. 2
Determines whether or not there is a motion between the pixel of frame #n and the pixel at the same position of frame # n + 1.

If it is determined in step S206 that the pixel of the frame #n and the pixel at the same position of the frame # n + 1 are determined to be moving, the process proceeds to step S207, and the area determination unit 20
3-2 sets “1” indicating that the pixel belongs to the motion area in the motion area determination flag corresponding to the pixel whose area is determined. The area determination unit 203-2 supplies the motion area determination flag to the determination flag storage frame memory 204, and the procedure proceeds to step S208.

In step S205, frame # n-1
Pixel and the pixel at the same position of frame #n are determined to be still, or, at step S206, the pixel of frame #n and the pixel at the same position of frame # n + 1 are determined to be still. In this case, since the pixel of frame #n does not belong to the motion area, the process of step S207 is skipped, and the procedure proceeds to step S208.

[0269] In step S208, the static / motion judging section 2
02-4, the pixel of frame # n-2 and the pixel at the same position of frame # n-1 are used to determine whether or not they are stationary. 202-
No. 3 determines whether or not the pixel of frame # n-1 and the pixel at the same position of frame #n are moving.

In step S209, frame # n-1
If it is determined that there is a motion between the pixel of the frame #n and the pixel at the same position in the frame #n, the process proceeds to step S210, and the region determination unit 20
Step 3-3 sets “1” indicating that the pixel belongs to the covered background area in the covered background area determination flag corresponding to the pixel whose area is determined. The area determination unit 203-3 supplies the covered background area determination flag to the determination flag storage frame memory 204, and the procedure proceeds to step S211.

At step S208, frame # n-2
Is determined to be a motion between the pixel of the frame # n-1 and the pixel of the same position of the frame # n-1, or in step S209, the pixel of the frame # n-1 and the pixel of the same position of the frame #n are
If it is determined that the image is stationary, the pixel of frame #n does not belong to the covered background area.
0 is skipped, and the procedure proceeds to step S211.
Proceed to.

[0272] In step S211, the static / movement determination unit 2
In step 02-2, it is determined whether or not the pixel of the frame #n and the pixel at the same position of the frame # n + 1 are moving. 1
Determines whether the pixel at frame # n + 1 and the pixel at the same position in frame # n + 2 are still.

In step S212, frame # n + 1
If it is determined that the pixel of the frame # n + 2 and the pixel at the same position in the frame # n + 2 are still, the process proceeds to step S213, and the region determination unit 2
03-1 sets “1” indicating that the pixel belongs to the uncovered background area in the uncovered background area determination flag corresponding to the pixel whose area is determined. The area determination unit 203-1 supplies the uncovered background area determination flag to the determination flag storage frame memory 204, and the procedure proceeds to step S214.

In step S211, when it is determined that the pixel of frame #n and the pixel at the same position of frame # n + 1 are still, or in step S212, the pixel of frame # n + 1 and the frame #n With the pixel at the same position of +2,
If it is determined that the pixel is in motion, the pixel of frame #n does not belong to the uncovered background area,
Step 213 is skipped, and the procedure proceeds to step S2
Proceed to 14.

In step S214, the area specifying unit 1
03 judges whether or not the area has been specified for all the pixels of frame #n. If it is determined that the area has not been specified for all the pixels of frame #n, the procedure returns to step S202. The region specifying process is repeated for other pixels.

If it is determined in step S214 that the area has been specified for all the pixels of frame #n,
Proceeding to step S215, the combining unit 205 generates area information indicating the mixed area based on the uncovered background area determination flag and the covered background area determination flag stored in the determination flag storage frame memory 204, Furthermore, area information indicating that each pixel belongs to any of the uncovered background area, the still area, the moving area, and the covered background area is generated, and the generated area information is set in the determination flag storage frame memory 206. Then, the process ends.

As described above, the area specifying unit 103 calculates the motion area,
Region information indicating belonging to a stationary region, an uncovered background region, or a covered background region can be generated.

The area specifying unit 103 generates area information corresponding to the mixed area by applying a logical sum to the area information corresponding to the uncovered background area and the covered background area, and includes the area information in the frame. For each of the pixels that have been set, area information including a flag indicating that the pixel belongs to a moving area, a still area, or a mixed area may be generated.

When the object corresponding to the foreground has a texture, the area specifying unit 103 can specify the moving area more accurately.

The area specifying unit 103 can output area information indicating a moving area as area information indicating a foreground area and area information indicating a still area as area information indicating a background area.

Although the description has been made assuming that the object corresponding to the background is stationary, the above-described processing for specifying the area can be applied even if the image corresponding to the background area includes motion. For example, when the image corresponding to the background area is moving uniformly, the area specifying unit 103 shifts the entire image in accordance with this movement, and performs the same processing as when the object corresponding to the background is stationary. I do. Also,
When the image corresponding to the background area includes a different motion for each local area, the area specifying unit 103 selects a pixel corresponding to the motion and executes the above-described processing.

FIG. 44 is a block diagram showing another example of the configuration of area specifying section 103. Area specifying unit 1 shown in FIG.
03 does not use a motion vector. Background image generation unit 3
01 generates a background image corresponding to the input image, and supplies the generated background image to the binary object image extraction unit 302. The background image generation unit 301 generates, for example, a background image by extracting an image object corresponding to a background object included in the input image.

FIG. 45 shows an example of a model diagram in which pixel values of pixels arranged in one row adjacent to the motion direction of the image corresponding to the foreground object are developed in the time direction. For example, when the motion direction of the image corresponding to the foreground object is horizontal with respect to the screen, the model diagram in FIG. 45 shows a model in which pixel values of adjacent pixels on one line are developed in the time direction.

In FIG. 45, the lines in frame #n are the same as the lines in frame # n-1 and frame # n + 1.

In frame #n, the foreground components corresponding to the objects included in the sixth through seventeenth pixels from the left are the second through thirteenth pixels from the left in frame # n-1. Included in the 10th to 21st pixels from the left in frame # n + 1.

In frame # n−1, the pixels belonging to the covered background area are the eleventh to thirteenth pixels from the left.
The second pixel, which belongs to the uncovered background area, is the second to fourth pixels from the left. In frame #n, the pixels belonging to the covered background area are the fifteenth through seventeenth pixels from the left, and the pixels belonging to the uncovered background area are the sixth through eighth pixels from the left. flame
In # n + 1, the pixels belonging to the covered background area are the 19th to 21st pixels from the left, and the pixels belonging to the uncovered background area are the 10th to 12th pixels from the left.

In frame # n-1, the pixels belonging to the background area are the first pixel from the left and the fourteenth through twenty-first pixels from the left. In frame #n, the pixels belonging to the background area are the first through fifth pixels from the left and the eighteenth through twenty-first pixels from the left. In frame # n + 1, the pixels belonging to the background area are the first through ninth pixels from the left.

FIG. 45 generated by the background image generation unit 301.
FIG. 46 shows an example of a background image corresponding to the example of FIG. The background image is composed of pixels corresponding to the background object,
The image component corresponding to the foreground object is not included.

The binary object image extracting unit 302 generates a binary object image based on the correlation between the background image and the input image, and supplies the generated binary object image to the time change detecting unit 303.

FIG. 47 shows the binary object image extracting unit 3
FIG. 2 is a block diagram showing a configuration of the second embodiment. Correlation value calculator 32
1 calculates a correlation between the background image and the input image supplied from the background image generation unit 301, generates a correlation value, and supplies the generated correlation value to the threshold processing unit 322.

The correlation value calculation unit 321 operates as shown in FIG.
As shown in (A), a block in a 3 × 3 background image centered on X ₄ and, as shown in FIG. 48 (B), a center in Y ₄ corresponding to the block in the background image 3 × 3
Applying equation (4) to the block in the input image of
Calculate the correlation value corresponding to ₄ .

[0292]

(Equation 2)

(Equation 3)

(Equation 4)

The correlation value calculation section 321 supplies the correlation value calculated for each pixel as described above to the threshold value processing section 322.

Further, the correlation value calculation unit 321 may, for example, as shown in FIG. 49A, block a 3 × 3 background image centered on X ₄ and a block shown in FIG. To
3 centered on Y ₄ corresponding to the block in the background image
Equation (7) may be applied to a block in the × 3 input image to calculate the absolute difference value corresponding to Y ₄ .

[0295]

(Equation 5)

The correlation value calculation section 321 uses the difference absolute value calculated in this way as a correlation value and sets a threshold value processing section 322
To supply.

The threshold value processing section 322 compares the pixel value of the correlation image with the threshold value th0, and sets 1 to the pixel value of the binary object image when the correlation value is equal to or smaller than the threshold value th0. If the correlation value is larger than the threshold value th0, the pixel value of the binary object image is set to 0, and a binary object image in which 0 or 1 is set as the pixel value is output. The threshold value processing unit 322 may store the threshold value th0 in advance, or the threshold value th0 input from the outside may be used.
0 may be used.

FIG. 50 is a diagram showing an example of a binary object image corresponding to the model of the input image shown in FIG.
In the binary object image, 0 is set to the pixel value of a pixel having a high correlation with the background image.

FIG. 51 is a block diagram showing a configuration of the time change detecting section 303. When determining the area for the pixel of frame #n, the frame memory 341 receives the frame # n- supplied from the binary object image extraction unit 302.
The binary object images of 1, frame #n, and frame # n + 1 are stored.

[0300] The area determination section 342
Based on the binary object images of frame # n-1, frame #n, and frame # n + 1 stored in
A region is determined for each pixel of #n, region information is generated, and the generated region information is output.

FIG. 52 is a diagram for explaining the determination by the area determination section 342. When the pixel of interest in the binary object image of frame #n is 0, the region determination unit 342
It is determined that the pixel of interest in frame #n belongs to the background area.

The pixel of interest of the binary object image of frame #n is 1, the corresponding pixel of the binary object image of frame # n-1 is 1, and the pixel of interest of frame # n + 1 is 2.
When the corresponding pixel of the value object image is 1, the area determination unit 342 determines that the pixel of interest in frame #n belongs to the foreground area.

When the pixel of interest of the binary object image of frame #n is 1 and the corresponding pixel of the binary object image of frame # n-1 is 0, the area determination unit 3
42 determines that the pixel of interest in frame #n belongs to the covered background area.

When the pixel of interest of the binary object image of frame #n is 1 and the corresponding pixel of the binary object image of frame # n + 1 is 0, the area determination unit 3
42 determines that the pixel of interest in frame #n belongs to the uncovered background area.

FIG. 53 is a diagram showing an example in which the time change detecting unit 303 determines the binary object image corresponding to the model of the input image shown in FIG. Since the pixel corresponding to frame #n of the binary object image is 0, the time change detection unit 303 determines that the first to fifth pixels from the left of frame #n
It is determined that the th pixel belongs to the background area.

The time change detection unit 303 uncovers the sixth through ninth pixels from the left since the pixel of frame #n of the binary object image is 1 and the corresponding pixel of frame # n + 1 is 0. It is determined that it belongs to the background area.

The time change detection unit 303 determines that the pixel of frame #n of the binary object image is 1, the corresponding pixel of frame # n-1 is 1, and the corresponding pixel of frame # n + 1 is 1. Therefore, it is determined that the tenth to thirteenth pixels from the left belong to the foreground area.

The time change detecting section 303 covers the 14th through 17th pixels from the left since the pixel of frame #n of the binary object image is 1 and the corresponding pixel of frame # n-1 is 0. It is determined that it belongs to the ground area.

[0309] Since the pixel corresponding to frame #n of the binary object image is 0, the time change detection unit 303 determines that the 18th to 21st pixels from the left belong to the background area.

Next, with reference to the flowchart of FIG. 54, the area specifying process of the area determination unit 103 will be described. In step S301, the background image generation unit 301 of the area determination unit 103 generates a background image by extracting, for example, an image object corresponding to a background object included in the input image based on the input image, and generates the background image. Image 2
It is supplied to the value object image extraction unit 302.

[0311] In step S302, the binary object image extracting unit 302 calculates the correlation value between the input image and the background image supplied from the background image generating unit 301 by the calculation described with reference to Fig. 48, for example. . In step S303, the binary object image extraction unit 302
Calculates a binary object image from the correlation value and the threshold value th0, for example, by comparing the correlation value with a threshold value th0.

[0312] In step S304, the time change detection unit 303 executes the area determination processing, and the processing ends.

With reference to the flowchart in FIG. 55, the details of the area determination processing corresponding to step S304 will be described. In step S321, the time change detection unit 303
Is determined in the frame #n stored in the frame memory 341 as to whether or not the pixel of interest is 0. In the frame #n, the pixel of interest is
When it is determined to be 0, the process proceeds to step S322,
The target pixel of frame #n is set to belong to the background area, and the process ends.

If it is determined in step S 321 that the pixel of interest is 1 in frame #n, the flow advances to step S 323, and the area determination unit 342 of the time change detection unit 303 stores the data in the frame memory 341. In frame #n, the pixel of interest is 1, and
In the frame # n-1, it is determined whether or not the corresponding pixel is 0. In the frame #n, the pixel of interest is 1, and in the frame # n-1, the corresponding pixel is 0. If it is determined that the target pixel of the frame #n belongs to the covered background area, the process proceeds to step S324, and the process ends.

In step S323, in the frame #n, the pixel of interest is 0, or the frame #n
When it is determined that the corresponding pixel is 1 in n−1, the process proceeds to step S325, and the area determination unit 342 of the time change detection unit 303 focuses on the frame #n stored in the frame memory 341. Pixel is 1,
Also, in frame # n + 1, it is determined whether the corresponding pixel is 0 or not, and in frame #n, the pixel of interest is
If it is 1, and it is determined that the corresponding pixel is 0 in frame # n + 1, the process proceeds to step S326, where it is set that the target pixel of frame #n belongs to the uncovered background area. , The process ends.

In step S325, in the frame #n, the pixel of interest is 0, or the frame #n
When it is determined that the corresponding pixel is 1 at n + 1, the process proceeds to step S327, and the region determination unit 342 of the time change detection unit 303 sets the target pixel of the frame #n as the foreground region, The process ends.

As described above, based on the correlation value between the input image and the corresponding background image, the area specifying unit 103 determines whether the pixels of the input image are the foreground area, the background area, the covered background area, and the uncovered back area. It is possible to specify which of the ground areas belongs to, and generate area information corresponding to the specified result.

FIG. 56 is a block diagram showing another configuration of area specifying section 103. Area specifying unit 103 shown in FIG.
Uses the motion vector supplied from the motion detection unit 102 and its position information. The same portions as those shown in FIG. 44 are denoted by the same reference numerals, and description thereof will be omitted.

[0319] The robust unit 361 outputs the 2 frames of N frames supplied from the binary object image extracting unit 302.
A robust binary object image is generated based on the value object image and output to the time change detection unit 303.

FIG. 57 is a block diagram illustrating the configuration of robust unit 361. The motion compensating unit 381 compensates for the motion of the binary object images of the N frames based on the motion vector and the positional information supplied from the motion detecting unit 102, and generates the motion-compensated binary object image. Output to switch 382.

With reference to the examples of FIGS. 58 and 59, the motion compensation of the motion compensator 381 will be described. For example, when determining the region of frame #n, when the binary object images of frame # n-1, frame #n, and frame # n + 1 shown in FIG. 58 are input, the motion compensation unit 381 ,
Based on the motion vector supplied from the motion detection unit 102,
As shown in FIG. 59, the binary object image of frame # n-1 and the binary object image of frame # n + 1 are motion-compensated, and the motion-compensated binary object image is supplied to the switch 382. I do.

The switch 382 stores the motion-compensated binary object image of the first frame in the frame memory 3
83-1, and outputs the motion-compensated binary object image of the second frame to the frame memory 383-2. Similarly, the switches 382 are third to N-1
The motion-compensated binary object image of the Nth frame is output to one of the frame memories 383-3 to 383- (N-1), and the motion-compensated binary object image of the Nth frame is output. To the frame memory 383-N.

The frame memory 383-1 stores the motion-compensated binary object image of the first frame, and outputs the stored binary object image to the weighting unit 384-1. The frame memory 383-2 is
The motion-compensated binary object image of the second frame is stored, and the stored binary object image is output to the weighting unit 384-2.

Similarly, each of the frame memories 383-3 to 383- (N-1) stores any one of the motion-compensated binary object images of the third to N-1st frames. Then, the stored binary object image is output to one of the weighting units 384-3 to 384- (N-1).
The frame memory 383-N stores the motion-compensated binary object image of the Nth frame, and outputs the stored binary object image to the weighting unit 384-N.

The weighting section 384-1 provides a predetermined weight to the pixel value of the motion-compensated binary object image of the first frame supplied from the frame memory 383-1.
The product is multiplied by w1 and supplied to the integrating unit 385. Weighting unit 38
4-2 is the 2 supplied from the frame memory 383-2.
The pixel value of the motion-compensated binary object image of the second frame is multiplied by a predetermined weight w2 and supplied to the integration unit 385.

Similarly, each of the weighting units 384-3 to 384- (N-1) is the third to the third one supplied from any of the frame memories 383-3 to 383- (N-1). The pixel value of the motion-compensated binary object image of any one of the (N-1) th frames is multiplied by one of the predetermined weights w3 to w (N-1) and supplied to the accumulator 385. Weighting unit 384-N
Multiplies the pixel value of the motion-compensated binary object image of the N-th frame supplied from the frame memory 383-N by a predetermined weight wN, and supplies the result to the integration unit 385.

The accumulating unit 385 accumulates the corresponding pixel values of the binary object image which are motion-compensated for the first to Nth frames and multiplied by one of the weights w1 to wN, respectively. A binary object image is generated by comparing the value with a predetermined threshold value th0.

As described above, the robust section 361 generates a robust binary object image from the N binary object images and supplies it to the time change detecting section 303. The configuration is shown in FIG. Even if the input image includes noise, the area specifying unit 103 can specify the area more accurately than in the case shown in FIG.

Next, the area specifying unit 10 shown in FIG.
The area specifying process 3 will be described with reference to the flowchart in FIG. Step S341 to step S
The process of step 343 is the same as step S301 to step S303 described in the flowchart of FIG. 54, and a description thereof will be omitted.

[0330] In step S344, the robust unit 361 executes robust processing.

[0331] In step S345, the time change detecting unit 303 executes the area determination processing, and the processing ends. Details of the process in step S345 are the same as those described with reference to the flowchart in FIG. 55, and a description thereof will be omitted.

Next, the details of the robust processing corresponding to the processing in step S344 in FIG. 60 will be described with reference to the flowchart in FIG. Step S36
In 1, the motion compensator 381 executes a motion compensation process for the input binary object image based on the motion vector supplied from the motion detector 102 and its position information. In step S362, the frame memory 383
Any of -1 to 383-N stores the motion-compensated binary object image supplied via the switch 382.

In step S363, the robust unit 361 determines whether or not N binary object images have been stored. If it is determined that the N binary object images have not been stored, the robust unit 361 determines in step S361. Back to
The motion compensation processing of the binary object image and the storage processing of the binary object image are repeated.

If it is determined in step S363 that N binary object images have been stored, the flow advances to step S364 to add weighting units 384-1 to 384-
Each of N is weighted by multiplying each of the N binary object images by a weight of any of w1 to wN.

[0335] In step S365, the integrating section 385
Multiplies N weighted binary object images.

In step S366, the accumulating section 385
Generates a binary object image from the integrated image, for example, by comparison with a predetermined threshold value th1, and the process ends.

As described above, the area specifying unit 103 shown in FIG. 56 can generate area information based on a robust binary object image.

[0338] As described above, the area specifying unit 103 obtains, for each pixel included in the frame, the area information indicating that the pixel belongs to the moving area, the still area, the uncovered background area, or the covered background area. Can be generated.

FIG. 62 is a block diagram showing an example of the configuration of the mixture ratio calculating section 104. Estimated mixture ratio processing unit 401
Calculates an estimated mixture ratio for each pixel by an operation corresponding to the model of the covered background area based on the input image, and supplies the calculated estimated mixture ratio to the mixture ratio determination unit 403.

The estimated mixture ratio processing unit 402 calculates an estimated mixture ratio for each pixel by an operation corresponding to the model of the uncovered background area based on the input image, and determines the calculated estimated mixture ratio for the mixture ratio. To the unit 403.

Since it can be assumed that the object corresponding to the foreground is moving at a constant speed within the shutter time, the mixture ratio α of the pixels belonging to the mixed region has the following properties. That is, the mixture ratio α changes linearly in accordance with the change in the position of the pixel. If the change in the pixel position is one-dimensional, the change in the mixture ratio α can be represented by a straight line. If the change in the pixel position is two-dimensional, the change in the mixture ratio α is represented by a plane. be able to.

Since the period of one frame is short, it is assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed.

In this case, the inclination of the mixture ratio α is the inverse ratio of the movement amount v within the shutter time of the foreground.

FIG. 63 shows an example of the ideal mixture ratio α. The gradient l in the mixing region having the ideal mixing ratio α can be represented as the reciprocal of the motion amount v.

As shown in FIG. 63, the ideal mixture ratio α
Has a value of 1 in the background region, a value of 0 in the foreground region, and a value exceeding 0 and less than 1 in the mixed region.

In the example of FIG. 64, the pixel value C06 of the seventh pixel from the left of frame #n is the seventh pixel value from the left of frame # n-1.
Expression (8) can be expressed by using the pixel value P06 of the th pixel.

[0347]

(Equation 6)

In Equation (8), the pixel value C06 is expressed as the pixel value M of the pixel in the mixed area, and the pixel value P06 is expressed as the pixel value B of the pixel in the background area. That is, the pixel value M of the pixel in the mixed area and the pixel value B of the pixel in the background area are respectively expressed by Equation (9).
And Expression (10).

M = C06 (9) B = P06 (10)

2 / v in the equation (8) corresponds to the mixture ratio α. Since the motion amount v is 4, the mixture ratio α of the seventh pixel from the left of frame #n is 0.5.

As described above, the pixel value C of the focused frame #n is regarded as the pixel value of the mixed area, and the pixel value P of the frame # n-1 before the frame #n is regarded as the pixel value of the background area. By considering this, Expression (3) indicating the mixture ratio α can be rewritten as Expression (11).

C = α · P + f (11) f in Expression (11) is the sum Σ _i Fi / v of the foreground components included in the pixel of interest. The variables included in equation (11) are the mixture ratio α and the sum f of the foreground components.

Similarly, FIG. 65 shows a model in which pixel values in the uncovered background area in which the amount of motion v is 4 and the number of virtual divisions in the time direction are 4 are developed in the time direction.

In the uncovered background area, similarly to the expression in the covered background area described above, the pixel value C of the frame #n of interest is regarded as the pixel value of the mixed area, and the frame #n after the frame #n n + 1
Equation (3) indicating the mixture ratio α can be expressed as Equation (12) by regarding the pixel value N of the pixel value as the pixel value of the background area.

C = α · N + f (12)

Although the background object has been described as being stationary, even when the background object is moving, the pixel value of the pixel at a position corresponding to the background movement amount v can be used. Equations (8) to (1)
2) can be applied. For example, in FIG. 64, when the motion amount v of the object corresponding to the background is 2 and the number of virtual divisions is 2, when the object corresponding to the background is moving to the right in the drawing, The pixel value B of the pixel in the background area is set to a pixel value P04.

Since the equations (11) and (12) each include two variables, the mixture ratio α cannot be obtained as it is. Here, since an image generally has a strong spatial correlation, adjacent pixels have substantially the same pixel value.

Since the foreground component has a strong spatial correlation, the equation is modified so that the sum f of the foreground component can be derived from the previous or subsequent frame, and the mixture ratio α is obtained.

The pixel value Mc of the seventh pixel from the left of frame #n in FIG. 66 can be expressed by equation (13).

[0360]

(Equation 7) 2 / v in the first term on the right side of the equation (13) corresponds to the mixture ratio α. The second term on the right side of Expression (13) is expressed as Expression (14) using the pixel value of the subsequent frame # n + 1.

[0361]

(Equation 8)

Here, it is assumed that Expression (15) is established using the spatial correlation of the foreground components.

F = F05 = F06 = F07 = F08 = F09 = F10 = F11 = F12 (15) Expression (14) can be replaced by expression (16) using expression (15).

[0364]

(Equation 9)

As a result, β can be expressed by equation (17).

Β = 2/4 (17)

In general, assuming that the foreground components relating to the mixed area are equal as shown in equation (15), the equation (18) is obtained from the relationship of the internal division ratio for all the pixels in the mixed area.
Holds.

Β = 1-α (18)

If equation (18) holds, then equation (1)
1) can be expanded as shown in equation (19).

[0370]

(Equation 10)

Similarly, if equation (18) holds, then
Equation (12) can be expanded as shown in equation (20).

[0372]

[Equation 11]

In equations (19) and (20),
Since C, N, and P are known pixel values, the only variable included in Expressions (19) and (20) is the mixture ratio α. FIG. 67 shows the relationship between C, N, and P in Expressions (19) and (20). C calculates the mixture ratio α,
This is the pixel value of the pixel of interest in frame #n. N is
The pixel value of the pixel of frame # n + 1 corresponding to the pixel of interest and the position in the spatial direction. P is the pixel value of the pixel of frame # n-1 corresponding to the pixel of interest and the position in the spatial direction.

Accordingly, since each of the equations (19) and (20) includes one variable, the mixture ratio α can be calculated using the pixel values of the pixels of the three frames. . By solving equations (19) and (20), the conditions for calculating the correct mixture ratio α are as follows:
Foreground components related to the mixed region, that is, pixels located at the boundary of the image object corresponding to the direction of motion of the foreground object in the foreground image object captured when the foreground object is stationary That is, the pixel values of the continuous pixels twice as many as the motion amount v are constant.

As described above, the mixture ratio α of the pixels belonging to the covered background area is calculated by the equation (21), and the mixture ratio α of the pixels belonging to the uncovered background area is calculated by the equation (22). You.

Α = (C-N) / (P-N) (21) α = (C-P) / (N-P) (22)

FIG. 68 is a block diagram showing the configuration of the estimated mixture ratio processing unit 401. The frame memory 421 stores the input image in units of frames, and supplies the next frame after the frame input as the input image to the frame memory 422 and the mixture ratio calculation unit 423.

[0378] The frame memory 422 stores the input image in frame units, and supplies the next frame after the frame supplied from the frame memory 421 to the mixture ratio calculation unit 423.

Therefore, when frame # n + 1 is input to the mixture ratio calculation unit 423 as an input image, the frame memory 421 supplies the frame #n to the mixture ratio calculation unit 423, and the frame memory 422 stores the frame #n. # n-1 is supplied to the mixture ratio calculation unit 423.

The mixture ratio calculator 423 calculates the pixel value of the pixel of interest in frame #n by the calculation shown in equation (21).
C, the pixel value N of the pixel of frame # n + 1 corresponding to the pixel of interest and the spatial position, and the pixel value of the pixel of frame # n-1 corresponding to the pixel of interest and the spatial position Based on the value P, an estimated mixture ratio of the pixel of interest is calculated, and the calculated estimated mixture ratio is output. For example, when the background is stationary, the mixture ratio calculation unit 423 determines the pixel value C of the pixel of interest in the frame #n, the pixel value of the frame # n + 1 having the same position in the frame as the pixel of interest. Frame #n, where the pixel value N of the pixel and the position of the pixel of interest in the frame are the same
Based on the pixel value P of the pixel of −1, the estimated mixing ratio of the pixel of interest is calculated, and the calculated estimated mixing ratio is output.

As described above, the estimated mixture ratio processing unit 401
An estimated mixture ratio can be calculated based on the input image and supplied to the mixture ratio determination unit 403.

The estimated mixture ratio processing unit 402 calculates the estimated mixture ratio of the pixel of interest by the calculation represented by the expression (21), whereas the estimated mixture ratio processing unit 401 calculates the estimated mixture ratio by the expression (22). Is the same as that of the estimated mixture ratio processing unit 401 except that the part for calculating the estimated mixture ratio of the pixel of interest is different from the operation shown in FIG.

FIG. 69 is a diagram showing an example of the estimated mixture ratio calculated by the estimated mixture ratio processing unit 401. The estimated mixture ratio shown in FIG. 69 shows the result for one line when the amount of movement v of the foreground corresponding to an object moving at a constant speed is 11.

The estimated mixture ratio in the mixture region is shown in FIG.
It can be seen that as shown in FIG.

Returning to FIG. 62, the mixture ratio determination unit 403 determines whether the pixel for which the mixture ratio α is supplied from the region identification unit 103 is a foreground region, a background region, a covered background region, or an uncovered region. The mixing ratio α is set based on area information indicating whether the pixel belongs to any of the background areas. The mixture ratio determination unit 403 sets 0 to the mixture ratio α when the target pixel belongs to the foreground region, and sets 1 to the mixture ratio α when the target pixel belongs to the background region, and sets the mixture ratio α. If the pixel belongs to the covered background area, the estimated mixture ratio supplied from the estimated mixture ratio processing unit 401 is set to the mixture ratio α, and if the target pixel belongs to the uncovered background area, the estimated mixture ratio processing unit The estimated mixture ratio supplied from 402 is set to the mixture ratio α. The mixture ratio determination unit 403 outputs a mixture ratio α set based on the region information.

FIG. 70 is a block diagram showing another configuration of mixture ratio calculating section 104. In FIG. The selecting unit 441 supplies the pixels belonging to the covered background area and the corresponding pixels of the previous and subsequent frames to the estimated mixture ratio processing unit 442 based on the area information supplied from the area specifying unit 103. The selection unit 441 supplies the pixels belonging to the uncovered background area and the corresponding pixels of the previous and subsequent frames to the estimated mixture ratio processing unit 443 based on the area information supplied from the area identification unit 103. .

The estimated mixture ratio processing unit 442 includes a selection unit 441
Based on the pixel value input from, the estimated mixture ratio of the pixel of interest belonging to the covered background area is calculated by the operation shown in Expression (21), and the calculated estimated mixture ratio is sent to the selection unit 444. Supply.

The estimated mixture ratio processing unit 443 includes a selection unit 441
Based on the pixel value input from, the estimated mixture ratio of the pixel of interest belonging to the uncovered background area is calculated by the operation shown in Expression (22), and the calculated estimated mixture ratio is selected by the selection unit 444. To supply.

When the target pixel belongs to the foreground area, the selecting section 444 selects an estimated mixture ratio of 0 based on the area information supplied from the area specifying section 103, and selects the mixture ratio α.
If the target pixel belongs to the background area, 1
Is selected and set to the mixture ratio α. When the target pixel belongs to the covered background area, the selection unit 444 selects the estimated mixture ratio supplied from the estimated mixture ratio processing unit 442 and sets the mixture to the mixture ratio α, and sets the target pixel to uncovered background. If it belongs to the ground area, the estimated mixture ratio supplied from the estimated mixture ratio processing unit 443 is selected and set as the mixture ratio α. The selection unit 444
The mixture ratio α selected and set based on the region information is output.

As described above, the mixture ratio calculator 104 having another configuration shown in FIG. 70 can calculate the mixture ratio α for each pixel included in the image, and can output the calculated mixture ratio α.

Referring to the flowchart of FIG. 71, FIG.
A process of calculating the mixture ratio α by the mixture ratio calculation unit 104 having the configuration shown in FIG. In step S401, the mixture ratio calculation unit 104 acquires the region information supplied from the region identification unit 103. In step S402, the estimated mixture ratio processing unit 401 executes a process of calculating the estimated mixture ratio using a model corresponding to the covered background area, and supplies the calculated estimated mixture ratio to the mixture ratio determination unit 403. The details of the calculation process of the mixture ratio estimation will be described later with reference to the flowchart in FIG.

In step S403, the estimated mixture ratio processing unit 402 executes a process of calculating the estimated mixture ratio using a model corresponding to the uncovered background area.
The calculated estimated mixture ratio is supplied to the mixture ratio determination unit 403.

In step S404, the mixture ratio calculation section 104 determines whether or not the mixture ratio α has been estimated for the entire frame. If it is determined that the mixture ratio α has not been estimated for the entire frame, the process proceeds to step S404. Returning to S402, a process of estimating the mixture ratio α for the next pixel is executed.

If it is determined in step S404 that the mixture ratio α has been estimated for the entire frame, the process proceeds to step S405, where the mixture ratio determination unit 403 determines that the pixel is
Foreground area, background area, covered background area,
Alternatively, the mixing ratio α is set based on the area information supplied from the area specifying unit 103, which indicates which of the uncovered background areas belongs. Mixing ratio determining unit 403
Sets 0 to the mixture ratio α when the target pixel belongs to the foreground region, and sets 1 to the mixture ratio α when the target pixel belongs to the background region, and sets the target pixel to the covered background. If it belongs to the region, the estimated mixture ratio supplied from the estimated mixture ratio processing unit 401 is set to the mixture ratio α,
If the target pixel belongs to the uncovered background area, the estimated mixture ratio supplied from the estimated mixture ratio processing unit 402 is set to the mixture ratio α, and the process ends.

As described above, the mixture ratio calculation unit 104 can calculate the mixture ratio α, which is a feature amount corresponding to each pixel, based on the region information supplied from the region identification unit 103 and the input image. .

The process of calculating the mixture ratio α by the mixture ratio calculating section 104 shown in FIG. 70 is the same as the process described in the flowchart of FIG. 71, and therefore the description thereof will be omitted.

Next, the process of estimating the mixture ratio by the model corresponding to the covered background area, corresponding to step S402 in FIG. 71, will be described with reference to the flowchart in FIG.

[0398] In step S421, the mixture ratio calculation unit 423 acquires the pixel value C of the target pixel of frame #n from the frame memory 421.

[0399] In step S422, the mixture ratio calculation unit 423 acquires the pixel value P of the pixel of frame # n-1 corresponding to the target pixel from the frame memory 422.

[0400] In step S423, the mixture ratio calculation unit 423 obtains the pixel value N of the pixel of frame # n + 1 corresponding to the target pixel included in the input image.

In step S424, the mixture ratio calculator 423 determines the pixel value C of the target pixel in frame #n,
An estimated mixture ratio is calculated based on the pixel value P of the pixel of n−1 and the pixel value N of the pixel of frame # n + 1.

In step S425, the mixture ratio calculation section 423 determines whether or not the process of calculating the estimated mixture ratio has been completed for the entire frame, and has completed the process of calculating the estimated mixture ratio for the entire frame. If it is determined that there is no pixel, the process returns to step S421, and the process of calculating the estimated mixture ratio for the next pixel is repeated.

If it is determined in step S425 that the process of calculating the estimated mixture ratio has been completed for the entire frame, the process ends.

As described above, the estimated mixture ratio processing unit 401
The estimated mixture ratio can be calculated based on the input image.

The process of estimating the mixture ratio by the model corresponding to the uncovered background region in step S403 in FIG. 71 is the same as the process shown in the flowchart of FIG. 72 using the equation corresponding to the model of the uncovered background region. Therefore, the description is omitted.

The estimated mixture ratio processing unit 44 shown in FIG.
2 and the estimated mixture ratio processing unit 443 calculate the estimated mixture ratio by executing the same processing as in the flowchart shown in FIG. 72, and thus the description thereof will be omitted.

Although the description has been made assuming that the object corresponding to the background is stationary, the above-described processing for determining the mixture ratio α can be applied even when the image corresponding to the background area includes motion. For example, when the image corresponding to the background region is moving uniformly, the estimated mixture ratio processing unit 401
The entire image is shifted according to the movement of the background, and the processing is performed in the same manner as when the object corresponding to the background is stationary. When the image corresponding to the background area includes a background motion that differs for each local area, the estimated mixture ratio processing unit 401
Performs the above-described processing by selecting a pixel corresponding to the background movement as a pixel corresponding to a pixel belonging to the mixed area.

[0408] The mixture ratio calculation unit 104 executes only the mixture ratio estimation process using the model corresponding to the covered background area for all the pixels, and outputs the calculated estimated mixture ratio as the mixture ratio α. You may do so. In this case, the mixture ratio α indicates the ratio of the background component with respect to the pixels belonging to the covered background region, and indicates the ratio of the foreground component with respect to the pixels belonging to the uncovered background region. For pixels belonging to the uncovered background area, the absolute value of the difference between the mixture ratio α thus calculated and 1 is calculated,
If the calculated absolute value is set to the mixture ratio α, the signal processing device can obtain the mixture ratio α indicating the ratio of the background component for the pixels belonging to the uncovered background area.

[0409] Similarly, the mixture ratio calculation unit 104 executes only the mixture ratio estimation process using the model corresponding to the uncovered background area for all the pixels, and calculates the calculated estimated mixture ratio. You may make it output as (alpha).

Next, another process of the mixture ratio calculating section 104 will be described.

In the shutter time, the property that the mixture ratio α changes linearly in response to the change in the pixel position due to the movement of the object corresponding to the foreground at a constant speed, An equation approximating the mixture ratio α and the sum f of the foreground components can be established. By using a plurality of pairs of pixel values of pixels belonging to the mixed region and pixel values of pixels belonging to the background region, the mixture ratio α is solved by solving an equation that approximates the mixture ratio α and the sum f of foreground components. calculate.

When the change in the mixture ratio α is approximated as a straight line, the mixture ratio α is expressed by equation (23).

Α = il + p (23) In Expression (23), i is an index in the spatial direction with the position of the pixel of interest set to 0. l is the slope of the straight line of the mixture ratio α. p is the intercept of the straight line of the mixture ratio α and the mixture ratio α of the pixel of interest. Equation (2
In 3), the index i is known, but the slope l
And the intercept p are unknown.

FIG. 73 shows the relationship between the index i, the slope l, and the intercept p.

By approximating the mixture ratio α as in equation (23), a plurality of different mixture ratios α can be calculated for a plurality of pixels.
Is represented by two variables. In the example shown in FIG. 73, the five mixture ratios for the five pixels are expressed by two variables, a slope 1 and an intercept p.

When the mixture ratio α is approximated by the plane shown in FIG. 74, when the motion v corresponding to the two directions of the image in the horizontal direction and the vertical direction is considered, the expression (23) is extended to a plane to obtain the mixture ratio α. α is represented by equation (24).

Α = jm + kq + p (24) In equation (24), j is an index in the horizontal direction with the position of the pixel of interest being 0, and k is an index in the vertical direction. m is the horizontal inclination of the plane having the mixture ratio α, and q is the vertical inclination of the plane having the mixture ratio α. p is the intercept of the plane with the mixture ratio α.

For example, in frame #n shown in FIG. 64, equations (25) to (27) hold for C05 to C07, respectively.

C05 = α05 · B05 / v + f05 (25) C06 = α06 · B06 / v + f06 (26) C07 = α07 · B07 / v + f07 (27)

The components of the foreground match in the vicinity, that is,
Assuming that F01 to F03 are equal, when F01 to F03 is replaced with Fc, the equation (28) is established.

F (x) = (1−α (x)) · Fc (28) In Expression (28), x represents a position in the spatial direction.

When α (x) is replaced by equation (24), equation (28) can be expressed as equation (29).

F (x) = (1- (jm + kq + p)) · Fc = j · (−m · Fc) + k · (−q · Fc) + ((1-p) · Fc) = js + kt + u (29)

In the equation (29), (−m · Fc), (−q · Fc)
Fc) and (1-p) · Fc are given by equations (30) to (32).
Has been replaced as shown.

S = -m · Fc (30) t = -q · Fc (31) u = (1-p) · Fc (32)

In equation (29), j is an index in the horizontal direction with the position of the pixel of interest being 0,
k is a vertical index.

As described above, since the assumption is made that the object corresponding to the foreground moves at a constant speed within the shutter time and the component corresponding to the foreground is constant in the vicinity, the sum of the components of the foreground is given by the following equation. 29).

When the mixture ratio α is approximated by a straight line, the sum of the foreground components can be expressed by equation (33).

F (x) = is + u (33)

When the sum of the mixture ratio α and the foreground component in Expression (13) is replaced by using Expressions (24) and (29), the pixel value M is expressed by Expression (34).

M = (jm + kq + p) · B + js + kt + u = jB · m + kB · q + B · p + j · s + k · t + u (34)

In equation (34), the unknown variables are the horizontal gradient m of the surface of the mixture ratio α, the vertical gradient q of the surface of the mixture ratio α, and the intercepts p, s, t of the surface of the mixture ratio α. , And u.

The pixel value M or the pixel value B is set in the normal equation shown in Expression (34) in correspondence with the pixel in the vicinity of the pixel of interest, and a plurality of pixels in which the pixel value M or the pixel value B is set are set. Is solved by the least squares method to calculate the mixture ratio α.

For example, the horizontal index j of the pixel of interest is set to 0, the vertical index k is set to 0, and for a 3 × 3 pixel near the pixel of interest,
When the pixel value M or the pixel value B is set in the normal equation shown in Expression (34), Expressions (35) to (43) are obtained. M _{-1, -1} = ( _-1 ) ・ B _{-1, -1}・ m + ( _-1 ) ・ B _{-1, -1}・ q + B _{-1, -1}・ p + ( _-1 ) ・ s + (- 1) ・ t + u (35) M _{0, -1} = (0) ・ B _{0, -1}・ m + ( _-1 ) ・ B _{0, -1}・ q + B _{0, -1}・ p + (0) ・s + ( _-1 ) .t + u (36) M _{+ 1, -1} = (+ 1) B _{+ 1, -1m} + ( _-1 ) B _{+ 1,} -1q + B _{+ 1 -1}・ p + (+ 1) ・ s + (-1) ・ t + u (37) M _-1,0 = ( _-1 ) ・ B _-1,0・ m + (0) ・ B _-1,0・ q + B _-1,0・ p + ( _-1 ) ・ s + (0) ・ t + u (38) M _0,0 = (0) ・ B _0,0・ m + (0) ・ B _0,0・ q + B _0,0・ p + (0) ・ s + (0) ・ t + u (39) M + _1,0 = (+ 1) ・ B + _1,0・ m + (0) ・ B + _1,0・ q + B _+1,0・ p + (+ 1) ・ s + (0) ・ t + u (40) M _{-1, + 1} = ( _-1 ) ・ B _{-1, + 1}・ m + (+ 1) ・ B _{-1, +} 1.q + B _{-1, +} 1.p + ( _-1 ) .s + (+ 1) .t + u (41) M _{0, + 1} = (0) .B _{0, +} 1.m + (+1) · B _{0, + 1} · q + B _{0, + 1} · p + (0) · s + (+ 1) · t + u (42) M _{+ 1, + 1} = (+ 1) · B _{+ 1, + 1} · m + (+ 1) · B _{+ 1, + 1} · q + B _{+ 1, + 1} · p + (+ 1) · s + (+ 1) · t + u (43)

Since the horizontal index j of the pixel of interest is 0 and the vertical index k of the pixel of interest is 0, the mixture ratio α of the pixel of interest is given by equation (24).
It is equal to the value at j = 0 and k = 0, that is, equal to the intercept p.

Accordingly, based on the nine equations (35) to (43), the least square method is used to calculate the horizontal gradient m, the vertical gradient q, and the intercepts p, s, t, and u, respectively. May be calculated and the intercept p may be output as the mixture ratio α.

Next, a more specific procedure for calculating the mixture ratio α by applying the least squares method will be described.

When the index i and the index k are represented by one index x, the relationship between the index i, the index k, and the index x is expressed by the following equation (44).

X = (j + 1) · 3 + (k + 1) (44)

[0440] Horizontal inclination m, vertical inclination q, intercept
Express p, s, t, and u as variables w0, w1, w2, w3, w4, and W5, respectively, and jB, kB, B, j, k, and 1 as a0, a1,
Expressed as a2, a3, a4, and a5. Considering the error ex,
Equations (35) to (43) can be rewritten as equation (45).

[0441]

(Equation 12) In Expression (45), x is any value of an integer from 0 to 8.

From equation (45), equation (46) can be derived.

[0443]

(Equation 13)

Here, in order to apply the least squares method, the sum of squares E of the error is defined as shown in Expression (47).

[0445]

[Equation 14]

In order to minimize the error, the sum of squares of the error
What is necessary is that the partial derivative of the variable Wv with respect to E becomes zero. Here, v is any value of an integer from 0 to 5. Therefore, wy is calculated so as to satisfy Expression (48).

[0447]

(Equation 15)

By substituting equation (46) into equation (48), equation (49) is obtained.

[0449]

(Equation 16)

By applying, for example, a sweeping method (Gauss-Jordan elimination method) to six equations obtained by substituting any one of integers 0 to 5 into v in equation (49), wy Is calculated. As described above, w0 is the horizontal gradient m, w1 is the vertical gradient q, w2 is the intercept p, w3
Is s, w4 is t, and w5 is u.

As described above, by applying the least squares method to the equation in which the pixel value M and the pixel value B are set, the horizontal gradient m, the vertical gradient q, the intercepts p, s, t, and u can ask.

In the description corresponding to the equations (35) to (43), the pixel value of the pixel included in the mixed area is M,
Although the pixel value of the pixel included in the background area has been described as B, the normal equation is applied to the case where the pixel of interest is included in the covered background area or the case where the pixel of interest is included in the uncovered background area. Need to be established.

For example, when calculating the mixture ratio α of the pixels included in the covered background area of the frame #n shown in FIG. 64, the pixels C04 to C08 of the frame #n and the pixels of the frame # n-1 Values P04 to P08 are set in the normal equation.

When calculating the mixture ratio α of the pixels included in the uncovered background area of frame #n shown in FIG. 65, the pixel values of C28 to C32 of the pixel of frame #n and the pixel value of the pixel of frame # n + 1 N28 to N32 are set in the normal equation.

For example, when calculating the mixture ratio α of the pixels included in the covered background area shown in FIG. 75, the following equations (50) to (58) are established. The pixel value of the pixel for calculating the mixture ratio α is Mc5. Mc1 = (-1) ・ Bc1 ・ m + (-1) ・ Bc1 ・ q + Bc1 ・ p + (-1) ・ s + (-1) ・ t + u (50) Mc2 = (0) ・ Bc2 ・ m + (- 1) ・ Bc2 ・ q + Bc2 ・ p + (0) ・ s + (-1) ・ t + u (51) Mc3 = (+ 1) ・ Bc3 ・ m + (-1) ・ Bc3 ・ q + Bc3 ・ p + (+ 1) · s + (-1) · t + u (52) Mc4 = (-1) · Bc4 · m + (0) · Bc4 · q + Bc4 · p + (-1) · s + (0) · t + u ( 53) Mc5 = (0) ・ Bc5 ・ m + (0) ・ Bc5 ・ q + Bc5 ・ p + (0) ・ s + (0) ・ t + u (54) Mc6 = (+ 1) ・ Bc6 ・ m + (0)・ Bc6 ・ q + Bc6 ・ p + (+ 1) ・ s + (0) ・ t + u (55) Mc7 = (-1) ・ Bc7 ・ m + (+ 1) ・ Bc7 ・ q + Bc7 ・ p + (-1)・ S + (+ 1) ・ t + u (56) Mc8 = (0) ・ Bc8 ・ m + (+ 1) ・ Bc8 ・ q + Bc8 ・ p + (0) ・ s + (+ 1) ・ t + u (57) Mc9 = (+ 1) ・ Bc9 ・ m + (+ 1) ・ Bc9 ・ q + Bc9 ・ p + (+ 1) ・ s + (+ 1) ・ t + u (58)

When calculating the mixture ratio α of the pixels included in the covered background area of the frame #n, the equation (5)
0) to (58), the pixel value Bc of the pixel in the background area of the pixel of frame # n−1 corresponding to the pixel of frame #n
1 to Bc9 are used.

When calculating the mixture ratio α of the pixels included in the uncovered background area shown in FIG. 75, the following equations (59) to (67) are established. Mixing ratio α
The pixel value of the pixel for which is calculated is Mu5. Mu1 = (-1) ・ Bu1 ・ m + (-1) ・ Bu1 ・ q + Bu1 ・ p + (-1) ・ s + (-1) ・ t + u (59) Mu2 = (0) ・ Bu2 ・ m + (- 1) ・ Bu2 ・ q + Bu2 ・ p + (0) ・ s + (-1) ・ t + u (60) Mu3 = (+ 1) ・ Bu3 ・ m + (-1) ・ Bu3 ・ q + Bu3 ・ p + (+ 1) · s + (-1) · t + u (61) Mu4 = (-1) · Bu4 · m + (0) · Bu4 · q + Bu4 · p + (-1) · s + (0) · t + u ( 62) Mu5 = (0) ・ Bu5 ・ m + (0) ・ Bu5 ・ q + Bu5 ・ p + (0) ・ s + (0) ・ t + u (63) Mu6 = (+ 1) ・ Bu6 ・ m + (0)・ Bu6 ・ q + Bu6 ・ p + (+ 1) ・ s + (0) ・ t + u (64) Mu7 = (-1) ・ Bu7 ・ m + (+ 1) ・ Bu7 ・ q + Bu7 ・ p + (-1)・ S + (+ 1) ・ t + u (65) Mu8 = (0) ・ Bu8 ・ m + (+ 1) ・ Bu8 ・ q + Bu8 ・ p + (0) ・ s + (+ 1) ・ t + u (66) Mu9 = (+ 1) ・ Bu9 ・ m + (+ 1) ・ Bu9 ・ q + Bu9 ・ p + (+ 1) ・ s + (+ 1) ・ t + u (67)

When calculating the mixture ratio α of the pixels included in the uncovered background area of the frame #n, in the equations (59) to (67), the frame # n + 1 corresponding to the pixel of the frame #n is calculated. The pixel values Bu1 to Bu9 of the pixels in the background area of the pixel are used.

FIG. 76 is a block diagram showing a configuration of the estimated mixture ratio processing unit 401. The image input to the estimated mixture ratio processing unit 401 is divided into a delay unit 501 and an adding unit 502.
Supplied to

[0460] The delay circuit 221 delays the input image by one frame and supplies it to the adding unit 502. Addition part 5
02, when the frame #n is input as an input image, the delay circuit 221 adds the frame # n-1 to the adding unit 50.
Feed to 2.

The adding unit 502 sets the pixel value of the pixel in the vicinity of the pixel for which the mixture ratio α is to be calculated and the pixel value of the frame # n-1 in a normal equation. For example, the adding section 50
2 sets the pixel values Mc1 to Mc9 and the pixel values Bc1 to Bc9 in the normal equation based on the equations (50) to (58). The adding unit 502 supplies the normal equation in which the pixel value is set to the calculation unit 503.

The computing unit 503 solves the normal equation supplied from the adding unit 502 by a sweeping method or the like to obtain an estimated mixture ratio, and outputs the obtained estimated mixture ratio.

As described above, the estimated mixture ratio processing unit 401
An estimated mixture ratio can be calculated based on the input image and supplied to the mixture ratio determination unit 403.

Since the estimated mixture ratio processing section 402 has the same configuration as that of the estimated mixture ratio processing section 401, description thereof will be omitted.

FIG. 77 is a diagram showing an example of the estimated mixture ratio calculated by the estimated mixture ratio processing unit 401. The estimated mixture ratio shown in FIG. 77 is such that the foreground motion v corresponding to an object moving at a constant speed is 11, and the result calculated by generating an equation in units of 7 × 7 pixel blocks is one line. Is shown.

The estimated mixture ratio in the mixture region is shown in FIG.
It can be seen that as shown in FIG.

[0467] The mixture ratio determination unit 403
The mixture ratio is set based on the region information indicating whether the pixel for which the mixture ratio is calculated belongs to any of the foreground region, the background region, the covered background region, or the uncovered background region. . When the target pixel belongs to the foreground area, the mixture ratio determination unit 403 sets 0 to the mixture ratio, and when the target pixel belongs to the background area, sets 1 to the mixture ratio. When the pixel belongs to the covered background area, the estimated mixture ratio supplied from the estimated mixture ratio processing unit 401 is set to the mixture ratio. When the target pixel belongs to the uncovered background area, the estimated mixture ratio is supplied from the estimated mixture ratio processing unit 402. The obtained estimated mixture ratio is set to the mixture ratio. Mixing ratio determining unit 403
Outputs the mixture ratio set based on the region information.

Referring to the flowchart of FIG. 78, the process of calculating the mixture ratio by the mixture ratio calculation unit 102 when the estimated mixture ratio processing unit 401 has the configuration shown in FIG. 76 will be described. In step S501, the mixture ratio calculation unit 10
2 obtains the area information supplied from the area specifying unit 101. In step S502, the estimated mixture ratio processing unit 4
01 executes a mixture ratio estimation process using a model corresponding to the covered background area, and supplies the estimated mixture ratio to the mixture ratio determination unit 403. The details of the mixture ratio estimation process will be described later with reference to the flowchart in FIG.

[0469] In step S503, the estimated mixture ratio processing unit 402 executes a mixture ratio estimation process using a model corresponding to the uncovered background area, and supplies the estimated mixture ratio to the mixture ratio determination unit 403.

[0470] In step S504, the mixture ratio calculation section 102 determines whether or not the mixture ratio has been estimated for the entire frame. If it is determined that the mixture ratio has not been estimated for the entire frame, the process proceeds to step S502. return,
A process of estimating the mixture ratio for the next pixel is executed.

[0471] If it is determined in step S504 that the mixture ratio has been estimated for the entire frame, the flow advances to step S505, where the mixture ratio determination unit 403 supplies the pixel supplied from the region identification unit 101 for which the mixture ratio is calculated. Is set based on region information indicating whether the image belongs to a foreground region, a background region, a covered background region, or an uncovered background region. When the target pixel belongs to the foreground area, the mixture ratio determination unit 403 sets 0 to the mixture ratio, and when the target pixel belongs to the background area, sets 1 to the mixture ratio. When the pixel belongs to the covered background area, the estimated mixture ratio supplied from the estimated mixture ratio processing unit 401 is set as the mixture ratio. When the target pixel belongs to the uncovered background area, the estimated mixture ratio is supplied from the estimated mixture ratio processing unit 402. The calculated estimated mixture ratio is set as the mixture ratio, and the process ends.

As described above, the mixture ratio calculation unit 102 can calculate the mixture ratio α, which is a feature amount corresponding to each pixel, based on the region information supplied from the region identification unit 101 and the input image. .

By using the mixture ratio α, it is possible to separate the foreground component and the background component contained in the pixel value while retaining the information on the motion blur contained in the image corresponding to the moving object. Will be possible.

Also, if images are synthesized based on the mixture ratio α, it becomes possible to create an image including a correct motion blur corresponding to the speed of a moving object as if the real world was actually photographed again. .

Next, the process of estimating the mixture ratio by the model corresponding to the covered background area corresponding to step S502 in FIG. 78 will be described with reference to the flowchart in FIG.

In step S521, adding section 5
02 sets a pixel value included in the input image and a pixel value included in the image supplied from the delay circuit 221 to a normal equation corresponding to the model of the covered background area.

In step S522, the estimated mixture ratio processing unit 401 determines whether or not the setting for the target pixel has been completed. If it is determined that the setting for the target pixel has not been completed, Returning to step S521, the process of setting the pixel value to the normal equation is repeated.

[0478] If it is determined in step S522 that the setting of the pixel value for the target pixel has been completed, the process proceeds to step S523, and the calculating unit 173 performs the estimated mixing based on the normal equation in which the pixel value has been set. The ratio is calculated, and the obtained estimated mixture ratio is output.

As described above, the estimated mixture ratio processing unit 401
The estimated mixture ratio can be calculated based on the input image.

The process of estimating the mixture ratio by the model corresponding to the uncovered background region in step S153 in FIG. 78 is the same as the process shown in the flowchart of FIG. 79 using the normal equation corresponding to the model in the uncovered background region. The description is omitted because it is similar.

Although the description has been made assuming that the object corresponding to the background is stationary, the above-described processing for obtaining the mixture ratio can be applied even when the image corresponding to the background area includes motion. For example, when the image corresponding to the background region is moving uniformly, the estimated mixture ratio processing unit 401 shifts the entire image corresponding to this movement, and performs the same operation as when the object corresponding to the background is stationary. To process.
Further, when the image corresponding to the background area includes a different motion for each local area, the estimated mixture ratio processing unit 401 selects a pixel corresponding to the motion as a pixel corresponding to a pixel belonging to the mixed area, and Execute the processing of

Next, the foreground / background separation unit 105 will be described. FIG. 80 is a block diagram illustrating an example of a configuration of the foreground / background separation unit 105. The input image supplied to the foreground / background separation unit 105 is supplied to a separation unit 601, a switch 602, and a switch 604. Information indicating the covered background area and the area information indicating the uncovered background area and supplied from the area specifying unit 103 are supplied to the separation unit 601. Area information indicating the foreground area is supplied to the switch 602. Area information indicating the background area is supplied to the switch 604.

The mixture ratio α supplied from the mixture ratio calculation section 104 is supplied to the separation section 601.

The separating unit 601 separates the foreground component from the input image based on the area information indicating the covered background area, the area information indicating the uncovered background area, and the mixture ratio α. The components are supplied to the synthesis unit 603, the background component is separated from the input image, and the separated background components are supplied to the synthesis unit 605.

The switch 602 is closed when a pixel corresponding to the foreground is input based on the area information indicating the foreground area, and supplies only the pixel corresponding to the foreground included in the input image to the synthesizing unit 603.

The switch 604 is closed when a pixel corresponding to the background is input, based on the area information indicating the background area, and supplies only the pixel corresponding to the background included in the input image to the synthesizing unit 605.

The combining unit 603 combines the foreground component image based on the foreground component supplied from the separation unit 601 and the pixel corresponding to the foreground supplied from the switch 602.
The synthesized foreground component image is output. Since the foreground region and the mixed region do not overlap, the combining unit 603 combines the foreground component image by applying a logical OR operation to the foreground component and the foreground pixel, for example.

The synthesizing section 603 stores an image whose pixel values are all 0 in a built-in frame memory in the initialization processing executed at the beginning of the processing of synthesizing the foreground component image. , The foreground component image is stored (overwritten). Accordingly, in the foreground component image output by the synthesis unit 603, pixels corresponding to the background region are:
0 is stored as the pixel value.

[0489] The combining unit 605 combines the background component image based on the component corresponding to the background supplied from the separating unit 601 and the pixel corresponding to the background supplied from the switch 604, and converts the combined background component image. Output. Since the background region and the mixed region do not overlap, the combining unit 605 combines the background component image by applying a logical OR operation to the component corresponding to the background and the pixel corresponding to the background, for example.

[0490] In the initialization process executed at the beginning of the process of synthesizing the background component image, the synthesizing unit 605 stores the image whose pixel values are all 0 in a built-in frame memory, and Is stored (overwritten). Accordingly, in the background component image output by the synthesis unit 605, pixels corresponding to the foreground area include:
0 is stored as the pixel value.

FIG. 81 is a diagram showing an input image input to the foreground / background separation unit 105, and a foreground component image and a background component image output from the foreground / background separation unit 105.

FIG. 81 (A) is a schematic diagram of an image to be displayed. FIG. 81 (B) shows a pixel belonging to the foreground area, a pixel belonging to the background area, and a mixed area corresponding to FIG. 81 (A). 1 is a model diagram in which pixels of one line including pixels belonging to are developed in the time direction.

As shown in FIG. 81 (A) and FIG. 81 (B), the background component image output from the foreground / background separation unit 105 is composed of pixels belonging to the background area and background components included in the pixels of the mixed area. Consists of

As shown in FIG. 81 (A) and FIG. 81 (B), the foreground component image output from the foreground / background separation unit 105 has the foreground components included in the pixels belonging to the foreground area and the pixels in the mixed area. Consists of

[0495] The pixel value of the pixel in the mixed area is separated by the foreground / background separation unit 105 into a background component and a foreground component. The separated background component forms a background component image together with the pixels belonging to the background area. The separated foreground component forms a foreground component image together with the pixels belonging to the foreground area.

As described above, in the foreground component image, the pixel value of the pixel corresponding to the background area is set to 0, and significant pixel values are set to the pixel corresponding to the foreground area and the pixel corresponding to the mixed area. Similarly, in the background component image, the pixel value of the pixel corresponding to the foreground area is set to 0, and meaningful pixel values are set to the pixel corresponding to the background area and the pixel corresponding to the mixed area.

Next, a description will be given of a process performed by the separation unit 601 to separate a foreground component and a background component from pixels belonging to the mixed area.

FIG. 82 is a model of an image showing foreground components and background components of two frames including a foreground corresponding to an object moving from left to right in the figure. In the image model shown in FIG. 82, the motion amount v of the foreground is 4
, And the number of virtual divisions is 4.

In frame #n, the leftmost pixel and the fourteenth through eighteenth pixels from the left consist only of background components and belong to the background area. In frame #n,
The second to fourth pixels from the left include the background component and the foreground component, and belong to the uncovered background area. In frame #n, the eleventh to thirteenth from the left
The third pixel contains background and foreground components and belongs to the covered background area. In frame #n, the fifth through tenth pixels from the left consist only of foreground components and belong to the foreground area.

In frame # n + 1, the first through fifth pixels from the left and the eighteenth pixel from the left consist only of background components and belong to the background area. In frame # n + 1, the sixth through eighth pixels from the left include the background component and the foreground component, and belong to the uncovered background area. In frame # n + 1, the fifteenth through seventeenth pixels from the left include the background component and the foreground component, and belong to the covered background area. In frame # n + 1, the ninth to fourteenth pixels from the left consist only of foreground components and belong to the foreground area.

FIG. 83 is a view for explaining the processing for separating the foreground components from the pixels belonging to the covered background area. In FIG. 83, α1 to α18 are mixing ratios corresponding to each of the pixels in frame #n.
In FIG. 83, the fifteenth through seventeenth pixels from the left belong to the covered background area.

[0502] The pixel value C15 of the fifteenth pixel from the left of frame #n is represented by equation (68).

C15 = B15 / v + F09 / v + F08 / v + F07 / v = α15 ・ B15 + F09 / v + F08 / v + F07 / v = α15 ・ P15 + F09 / v + F08 / v + F07 / v (68) where α15 is the mixture ratio of the fifteenth pixel from the left of frame #n. P15 is the pixel value of the fifteenth pixel from the left of frame # n-1.

[0504] Based on equation (68), one frame from the left of frame #n
The sum f15 of the foreground components of the fifth pixel is expressed by Expression (69).

F15 = F09 / v + F08 / v + F07 / v = C15-α15 · P15 (69)

Similarly, the sum f16 of the foreground components of the sixteenth pixel from the left of frame #n is expressed by equation (70), and the sum f17 of the foreground components of the seventeenth pixel from the left of frame #n is
Is represented by equation (71).

F16 = C16-α16 · P16 (70) f17 = C17-α17 · P17 (71)

Thus, the foreground component fc contained in the pixel value C of the pixel belonging to the covered background area is
It is calculated by equation (72).

Fc = C−α · P (72) P is the pixel value of the corresponding pixel in the immediately preceding frame.

FIG. 84 is a view for explaining the processing for separating the foreground components from the pixels belonging to the uncovered background area. In FIG. 84, α1 to α18 are mixture ratios corresponding to the respective pixels in frame #n. In FIG. 84, the second through fourth pixels from the left belong to the uncovered background area.

[0511] Pixel value of second pixel from left of frame #n
C02 is represented by equation (73).

C02 = B02 / v + B02 / v + B02 / v + F01 / v = α2 / B02 + F01 / v = α2 / N02 + F01 / v (73) where α2 is the left of frame #n Is the mixture ratio of the second pixel from. N02 is the pixel value of the second pixel from the left of frame # n + 1.

[0513] Based on equation (73), two frames from the left of frame #n are obtained.
The sum f02 of the foreground components of the th pixel is expressed by equation (74).

F02 = F01 / v = C02-α2 · N02 (74)

Similarly, the sum f03 of the foreground components of the third pixel from the left of frame #n is expressed by equation (75), and the sum f04 of the foreground components of the fourth pixel from the left of frame #n is Is represented by equation (76).

F03 = C03-α3 · N03 (75) f04 = C04-α4 · N04 (76)

As described above, the foreground component fu included in the pixel value C of the pixel belonging to the uncovered background area is obtained.
Is calculated by equation (77).

Fu = C−α · N (77) N is the pixel value of the corresponding pixel in the next frame.

[0519] As described above, the separating unit 601 outputs the information indicating the covered background area included in the area information.
And information indicating the uncovered background area,
The foreground component and the background component can be separated from the pixels belonging to the mixed area based on the mixing ratio α for each pixel.

FIG. 85 is a block diagram showing an example of the structure of the separating section 601 for executing the above-described processing. The image input to the separation unit 601 is stored in a frame memory 621
The area information indicating the covered background area and the uncovered background area supplied from the mixing ratio calculation unit 104 and the mixing ratio α are input to the separation processing block 622.

[0521] The frame memory 621 stores the input image in frame units. The frame memory 621 is
When the target of processing is frame #n, frame # n−1, frame #n, which is one frame before frame #n, and frame # n + 1, which is one frame after frame #n, Remember.

[0522] The frame memory 621 stores the frame # n-1,
The corresponding pixels of the frame #n and the frame # n + 1 are supplied to the separation processing block 622.

[0523] The separation processing block 622 determines the frame # n-1 and the frame #n supplied from the frame memory 621 based on the area information indicating the covered background area and the uncovered background area, and the mixture ratio α.
8 and the pixel value of the corresponding pixel of frame # n + 1 in FIG.
84, the foreground component and the background component are separated from the pixels belonging to the mixed area of the frame #n, and supplied to the frame memory 623.

[0524] The separation processing block 622 includes an uncovered area processing section 631, a covered area processing section 632, a synthesizing section 633, and a synthesizing section 634.

The multiplier 6 of the uncovered area processing unit 631
41 multiplies the mixture ratio α by the pixel value of the pixel of frame # n + 1 supplied from the frame memory 621, and
42. The switch 642 is connected to the frame memory 6
When the pixel of frame #n supplied from 21 (corresponding to the pixel of frame # n + 1) is the uncovered background area, the pixel is closed and multiplied by the mixture ratio α supplied from the multiplier 641 To the computing unit 643 and the combining unit 63
4 Frame output from switch 642
The value obtained by multiplying the pixel value of pixel # n + 1 by the mixture ratio α
It is equal to the background component of the pixel value of the corresponding pixel of #n.

The computing unit 643 obtains a foreground component by subtracting the background component supplied from the switch 642 from the pixel value of the pixel of frame #n supplied from the frame memory 621. The computing unit 643 supplies the foreground component of the pixel of frame #n belonging to the uncovered background area to the combining unit 633.

[0527] Multiplier 651 of covered area processing section 632
Multiplies the mixture ratio α by the pixel value of the pixel of frame # n−1 supplied from the frame memory 621 to generate a switch 652
Output to The switch 652 is connected to the frame memory 621
When the pixel of frame #n (corresponding to the pixel of frame # n-1) supplied from is the covered background area, the pixel value is closed and multiplied by the mixture ratio α supplied from the multiplier 651 To the synthesizer 653 and the synthesizing unit 634. The value obtained by multiplying the pixel value of the pixel of frame # n−1 output from the switch 652 by the mixture ratio α is equal to the background component of the pixel value of the corresponding pixel of frame #n.

The computing unit 653 subtracts the background component supplied from the switch 652 from the pixel value of the pixel of frame #n supplied from the frame memory 621 to obtain the foreground component. The calculator 653 supplies the foreground components of the pixels of frame #n belonging to the covered background area to the combining unit 633.

[0529] The synthesizing unit 633 calculates the arithmetic unit 6 of the frame #n.
The foreground component of the pixel belonging to the uncovered background area supplied from 43 and the foreground component of the pixel belonging to the covered background area supplied from the calculator 653 are combined and supplied to the frame memory 623. .

[0531] The synthesizing unit 634 calculates the background component of the pixel belonging to the uncovered background area and the switch 652 supplied from the switch 642 in the frame #n.
The background components of the pixels belonging to the covered background area supplied from the
To supply.

[0531] The frame memory 623 stores the foreground component and the background component of the pixels in the mixed area of the frame #n supplied from the separation processing block 622, respectively.

The frame memory 623 outputs the stored foreground components of the pixels of the mixed area of frame #n and the stored background components of the pixels of the mixed area of frame #n.

By utilizing the mixture ratio α, which is a feature quantity, it becomes possible to completely separate the foreground component and the background component contained in the pixel value.

The combining section 603 combines the foreground components of the pixels in the mixed area of frame #n output from the separating section 601 with the pixels belonging to the foreground area to generate a foreground component image. The synthesizing unit 605 generates a background component image by synthesizing the background component of the pixel in the mixed area of the frame #n and the pixel belonging to the background area output from the separating unit 601.

FIG. 86 is a diagram showing an example of a foreground component image and an example of a background component image corresponding to frame #n in FIG.

FIG. 86 (A) shows an example of the foreground component image corresponding to frame #n in FIG. Since the leftmost pixel and the fourteenth pixel from the left consist of only the background component before the foreground and the background are separated, the pixel value is set to 0.

[0537] The second to fourth pixels from the left belong to the uncovered background area before the foreground and the background are separated, the background component is set to 0, and the foreground component is left as it is. . The eleventh to thirteenth pixels from the left belong to the covered background area before the foreground and the background are separated, the background component is set to 0, and the foreground component is left as it is. The fifth to tenth pixels from the left are left as they are because they consist only of foreground components.

FIG. 86B shows an example of a background component image corresponding to frame #n in FIG. The leftmost pixel and the fourteenth pixel from the left are left alone because they consisted only of the background component before the foreground and the background were separated.

The second to fourth pixels from the left belong to the uncovered background area before the foreground and background are separated, the foreground component is set to 0, and the background component is left as it is. . Before the foreground and background are separated, the eleventh to thirteenth pixels from the left belong to the covered background area, the foreground component is set to 0, and the background component is left as it is. Before the foreground and the background are separated, the fifth through tenth pixels from the left consist only of the foreground components, and thus have a pixel value of 0.

Next, the process of separating the foreground and the background by the foreground / background separator 105 will be described with reference to the flowchart shown in FIG. In step S601, the frame memory 621 of the separation unit 601 acquires an input image,
The frame #n to be separated from the foreground and the background is stored together with the previous frame # n-1 and the subsequent frame # n + 1.

In step S602, the separation unit 601
The separation processing block 622 obtains the area information supplied from the mixture ratio calculation unit 104. In step S603, the separation processing block 622 of the separation unit 601 acquires the mixture ratio α supplied from the mixture ratio calculation unit 104.

[0542] In step S604, the uncovered area processing section 631 performs the processing based on the area information and the mixture ratio α.
The background component is extracted from the pixel values of the pixels belonging to the uncovered background area supplied from the frame memory 621.

[0543] In step S605, the uncovered area processing section 631 performs the processing based on the area information and the mixture ratio α.
The foreground component is extracted from the pixel values of the pixels belonging to the uncovered background area supplied from the frame memory 621.

[0544] In step S606, the covered area processing unit 632 extracts a background component from the pixel values of the pixels belonging to the covered background area supplied from the frame memory 621 based on the area information and the mixture ratio α. .

[0545] In step S607, the covered area processing unit 632 extracts a foreground component from the pixel values of the pixels belonging to the covered background area supplied from the frame memory 621 based on the area information and the mixture ratio α. .

[0546] In step S608, the combining section 633
Combines the foreground components of the pixels belonging to the uncovered background area extracted in the processing of step S605 with the foreground components of the pixels belonging to the covered background area extracted in the processing of step S607. The synthesized foreground component is supplied to the synthesis unit 603. Further, the synthesizing unit 603 generates a foreground component image by synthesizing the pixels belonging to the foreground area supplied via the switch 602 and the foreground components supplied from the separating unit 601.

[0547] In step S609, the synthesizing unit 634
Combines the background component of the pixel belonging to the uncovered background area extracted in the processing of step S604 with the background component of the pixel belonging to the covered background area extracted in the processing of step S606. The synthesized background component is supplied to the synthesis unit 605. Further, the synthesizing unit 605 generates a background component image by synthesizing the pixels belonging to the background area supplied via the switch 604 and the background component supplied from the separation unit 601.

In step S610, the synthesizing unit 603
Outputs a foreground component image. In step S611, the synthesis unit 605 outputs the background component image, and the processing ends.

As described above, the foreground / background separation unit 105 separates the foreground component and the background component from the input image based on the area information and the mixture ratio α, and outputs the foreground component image consisting of only the foreground component. Also, a background component image consisting of only background components can be output.

Next, the removal of motion blur from the foreground component image will be described.

FIG. 88 is a block diagram showing an example of the configuration of the motion blur removing unit 106. The motion vector and its position information supplied from the motion detection unit 102 and the region information supplied from the region identification unit 103 are stored in the processing unit determination unit 8.
01 and the modeling unit 802. The foreground component image supplied from the foreground / background separation unit 105 is supplied to the addition unit 804.

The processing unit determination unit 801 supplies the generated processing unit to the modeling unit 802 together with the motion vector based on the motion vector, its position information, and the area information. The processing unit determination unit 801 supplies the generated processing unit to the adding unit 804.

As shown in the example of FIG. 89, the processing unit generated by the processing unit determining unit 801 starts from a pixel corresponding to the covered background area of the foreground component image and extends to a pixel corresponding to the uncovered background area. A continuous pixel in the movement direction starting from a pixel corresponding to the uncovered background area or a continuous pixel aligned in the movement direction to a pixel corresponding to the covered background area is shown. The processing unit is, for example, an upper left point (a pixel designated by the processing unit and a position of a pixel located at the leftmost or uppermost position on the image) and a lower right point.
Data.

The modeling unit 802 performs modeling based on the motion vector and the input processing unit. More specifically, for example, the modeling unit 802 pre-stores a plurality of models corresponding to the number of pixels included in the processing unit, the number of virtual divisions of pixel values in the time direction, and the number of foreground components for each pixel. A model that specifies the correspondence between the pixel value and the foreground component as shown in FIG. 90 may be selected based on the processing unit and the virtual division number of the pixel value in the time direction. .

For example, if the number of pixels corresponding to the processing unit is one,
When the motion amount v within the shutter time is 5, the modeling unit 802 sets the virtual division number to 5, the leftmost pixel includes one foreground component, and the second pixel from the left. Pixel includes two foreground components, the third pixel from the left includes three foreground components, the fourth pixel from the left includes four foreground components, and the fifth pixel from the left includes five foreground components. Including the foreground component, the sixth pixel from the left contains five foreground components, the seventh pixel from the left contains five foreground components, and the eighth pixel from the left contains five foreground components. , The ninth pixel from the left contains four foreground components, the tenth pixel from the left contains three foreground components, the eleventh pixel from the left contains two foreground components, and 12 from the left.
A model is selected in which the th pixel contains one foreground component and comprises a total of eight foreground components.

[0556] When a motion vector and a processing unit are supplied, instead of selecting from a model stored in advance, the modeling unit 802 generates a model based on the motion vector and the processing unit. You may do so.

[0557] The modeling unit 802 supplies the selected model to the equation generation unit 803.

[0558] The equation generation unit 803 includes a modeling unit 802.
Generate an equation based on the model supplied from. FIG.
Referring to the model of the foreground component image indicated by 0, the number of foreground components is 8, the number of pixels corresponding to the processing unit is 12, the motion amount v is 5, the number of virtual divisions is 5, and An equation generated by the equation generation unit 803 at a certain time will be described.

When the foreground component corresponding to the shutter time / v included in the foreground component image is F01 / v to F08 / v, F01 / v
To F08 / v and the pixel values C01 to C12 are given by the following equations (78).
To (89).

C01 = F01 / v (78) C02 = F02 / v + F01 / v (79) C03 = F03 / v + F02 / v + F01 / v (80) C04 = F04 / v + F03 / v + F02 / v + F01 / v (81) C05 = F05 / v + F04 / v + F03 / v + F02 / v + F01 / v (82) C06 = F06 / v + F05 / v + F04 / v + F03 / v + F02 / v (83) C07 = F07 / v + F06 / v + F05 / v + F04 / v + F03 / v (84) C08 = F08 / v + F07 / v + F06 / v + F05 / v + F04 / v (85) C09 = F08 / v + F07 / v + F06 / v + F05 / v (86) C10 = F08 / v + F07 / v + F06 / v (87) C11 = F08 / v + F07 / v (88) C12 = F08 / v (89)

[0562] The equation generation unit 803 deforms the generated equation to generate an equation. The equations generated by the equation generation unit 803 are shown in equations (90) to (101). C01 = 1 ・ F01 / v + 0 ・ F02 / v + 0 ・ F03 / v + 0 ・ F04 / v + 0 ・ F05 / v +0 ・ F06 / v + 0 ・ F07 / v + 0 ・ F08 / v ( 90) C02 = 1 ・ F01 / v + 1 ・ F02 / v + 0 ・ F03 / v + 0 ・ F04 / v + 0 ・ F05 / v +0 ・ F06 / v + 0 ・ F07 / v + 0 ・ F08 / v (91) C03 = 1 ・ F01 / v + 1 ・ F02 / v + 1 ・ F03 / v + 0 ・ F04 / v + 0 ・ F05 / v +0 ・ F06 / v + 0 ・ F07 / v + 0 ・F08 / v (92) C04 = 1 ・ F01 / v + 1 ・ F02 / v + 1 ・ F03 / v + 1 ・ F04 / v + 0 ・ F05 / v +0 ・ F06 / v + 0 ・ F07 / v + 0 ・ F08 / v (93) C05 = 1 ・ F01 / v + 1 ・ F02 / v + 1 ・ F03 / v + 1 ・ F04 / v + 1 ・ F05 / v +0 ・ F06 / v + 0 ・ F07 / v + 0 ・ F08 / v (94) C06 = 0 ・ F01 / v + 1 ・ F02 / v + 1 ・ F03 / v + 1 ・ F04 / v + 1 ・ F05 / v +1 ・ F06 / v + 0 ・F07 / v + 0 ・ F08 / v (95) C07 = 0 ・ F01 / v + 0 ・ F02 / v + 1 ・ F03 / v + 1 ・ F04 / v + 1 ・ F05 / v +1 ・ F06 / v + 1 ・ F07 / v + 0 ・ F08 / v (96) C08 = 0 ・ F01 / v + 0 ・ F02 / v + 0 ・ F03 / v + 1 ・ F04 / v + 1 ・ F05 / v +1 ・ F06 / v + 1 ・ F07 / v + 1 ・ F08 / v (97) C09 = 0 ・ F01 / v + 0 ・ F02 / v + 0 ・ F03 / v + 0 ・ F04 / v + 1 ・ F05 / v +1 ・F06 / v + 1 ・ F07 / v + 1 ・ F08 / v (98) C10 = 0 ・ F01 / v + 0 ・ F02 / v + 0 ・ F03 / v + 0 ・ F04 / v + 0 ・ F05 / v + 1 ・ F06 / v + 1 ・ F07 / v + 1 ・ F08 / v (99) C11 = 0 ・ F01 / v + 0 ・ F0 2 / v + 0 ・ F03 / v + 0 ・ F04 / v + 0 ・ F05 / v +0 ・ F06 / v + 1 ・ F07 / v + 1 ・ F08 / v (100) C12 = 0 ・ F01 / v + 0 ・ F02 / v + 0 ・ F03 / v + 0 ・ F04 / v + 0 ・ F05 / v +0 ・ F06 / v + 0 ・ F07 / v + 1 ・ F08 / v (101)

The expressions (90) to (101) are obtained by the expression (10)
It can also be expressed as 2).

[0563]

[Equation 17] In Expression (102), j indicates a position of a pixel. In this example, j has a value of any one of 1 to 12. I indicates the position of the foreground value. In this example, i has a value of any one of 1 to 8. aij
Has a value of 0 or 1 corresponding to the values of i and j.

[0564] In consideration of the error, Expression (102)
Can be expressed as in equation (103).

[0565]

(Equation 18) In Expression (103), ej is an error included in the target pixel Cj.

[0566] Equation (103) can be rewritten as equation (104).

[0567]

[Equation 19]

Here, in order to apply the least squares method, the sum of squares E of the error is defined as shown in Expression (105).

[0569]

(Equation 20)

In order to minimize the error, the sum of squares of the error
What is necessary is that the value of the partial differentiation of E with respect to the variable Fk becomes 0. Fk is determined so as to satisfy Expression (106).

[0571]

(Equation 21)

In equation (106), since the amount of movement v is a fixed value, equation (107) can be derived.

[0573]

(Equation 22)

By expanding equation (107) and transposing, equation (108) is obtained.

[0575]

(Equation 23)

[0576] The expression is expanded into eight expressions obtained by substituting any one of integers 1 to 8 for k in the expression (108). The obtained eight expressions can be represented by one expression by a matrix. This equation is called a normal equation.

An example of a normal equation generated by the equation generating unit 803 based on the least squares method is shown in Expression (109).

[0578]

(Equation 24)

If equation (109) is expressed as A · F = v · C, C, A, and v are known, and F is unknown. A and v are known at the time of modeling, but C becomes known by inputting pixel values in the adding operation.

The error contained in the pixel C can be dispersed by calculating the foreground component by the normal equation based on the least squares method.

The equation generator 803 supplies the normal equation generated in this way to the adder 804.

The adding unit 804 includes the processing unit determining unit 80
Based on the processing unit supplied from 1, the pixel value C included in the foreground component image is set in the matrix expression supplied from the equation generation unit 803. The adding unit 804 supplies a matrix in which the pixel values C are set to the calculation unit 805.

[0583] The arithmetic unit 805 uses the sweep-out method (Gauss-Jord).
An elimination method such as an) is used to calculate the foreground component Fi / v from which the motion blur has been removed, and the pixel value of the foreground from which the motion blur has been removed, which is an integer from 0 to 8 Is calculated, and a motion blur-removed foreground component image composed of Fi, which is a pixel value from which motion blur has been removed, as shown in FIG. 91, is output.

In the foreground component image from which motion blur has been removed as shown in FIG. 91, F01 to F08 are set in C03 to C10, respectively, because the position of the foreground component image with respect to the screen is not changed. And can correspond to an arbitrary position.

For example, as shown in FIG. 92, when the number of pixels corresponding to the processing unit is 8 and the motion amount v is 4, the motion-blur removing unit 106 sets the matrix shown in the equation (110). Generates the expression

[0586]

(Equation 25)

[0587] The motion blur removal unit 106 calculates the Fi, which is a pixel value with the amount of motion blur adjusted, by formulating a number corresponding to the length of the processing unit. Similarly, for example, when the number of pixels included in the processing unit is 100, 1
An equation corresponding to 00 pixels is generated to calculate Fi.

[0588] As described above, the motion-blur removing unit 106
An equation is generated corresponding to the motion amount v and the processing unit, and the pixel value of the foreground component image is set in the generated equation, to calculate a foreground component image from which motion blur has been removed.

Next, with reference to the flowchart in FIG. 93, the processing of removing the motion blur included in the foreground component image by the motion blur removing unit 106 will be described.

[0590] In step S801, the processing unit determination unit 801 of the motion blur removal unit 106 generates a processing unit based on the motion vector and the area information, and supplies the generated processing unit to the modeling unit 802.

[0591] In step S802, the modeling unit 802 of the motion blur removing unit 106 selects or generates a model according to the amount of motion v and the processing unit. Step S
In 803, the equation generator 803 creates a normal equation based on the selected model.

In step S804, the adding section 8
Step 04 sets the pixel value of the foreground component image in the created normal equation. In step S805, the adding unit 8
In step S804, it is determined whether the pixel values of all the pixels corresponding to the processing unit have been set. If it is determined that the pixel values of all the pixels corresponding to the processing unit have not been set, step S804 is performed. And the process of setting the pixel value to the normal equation is repeated.

If it is determined in step S805 that the pixel values of all the pixels in the processing unit have been set, the process proceeds to step S806, and the arithmetic unit 805 causes the adding unit 8
Based on the normal equation in which the pixel values supplied from 04 have been set, pixel values of the foreground from which motion blur has been removed are calculated, and the process ends.

As described above, the motion blur removing unit 106 can remove the motion blur from the foreground image including the motion blur based on the motion vector and the area information.

That is, it is possible to remove the motion blur included in the pixel value which is the sample data.

Next, the correction of the background component image by the correction unit 107 will be described.

FIG. 94 is a diagram showing an example of a model of a background component image corresponding to the model of the foreground component image shown in FIG. 90.

As shown in FIG. 94, the pixel value of the pixel of the background component image corresponding to the mixed region of the original input image corresponds to the background region of the original input image since the foreground component has been removed. As compared with the pixel to be processed, the image is composed of a smaller number of background components corresponding to the mixture ratio α.

For example, in the background component image shown in FIG. 94, the pixel value C01 is composed of four background components B02 / V, and the pixel value C02 is composed of three background components B03 / V.
The pixel value C03 is composed of two background components B04 / V, and the pixel value C04 is composed of one background component B05 / V.

In the background component image shown in FIG. 94, the pixel value C09 is composed of one background component B10 / V, and the pixel value C10 is composed of two background components B11 / V.
The pixel value C11 is composed of three background components B12 / V, and the pixel value C12 is composed of four background components B13 / V.

As described above, the pixel value of the pixel corresponding to the mixed area of the original input image is constituted by a smaller number of background components than the pixel corresponding to the background area of the original input image. Therefore, the image corresponding to the mixed area of the foreground component image is, for example, a darker image than the image of the background area.

The correction unit 107 multiplies each of the pixel values of the pixels corresponding to the mixed region of the background component image by a constant corresponding to the mixing ratio α to obtain a pixel corresponding to the mixed region of the background component image. Is corrected.

For example, when the background component image shown in FIG. 94 is input, the correction unit 107 multiplies the pixel value C01 by 5/4, multiplies the pixel value C02 by 5/3, and multiplies the pixel value C11 by 5/5. / 3 and the pixel value C12 by 5/4. The correcting unit 107 sets the pixel values of the pixel values C03 to C11 to 0 in order to match the positions of the pixels with the foreground component image from which the motion blur has been removed, as shown in the example in FIG.

The correcting section 107 outputs a background component image in which the pixel values of the pixels corresponding to the mixed area are corrected, as shown in FIG. 95.

As described above, the correction unit 107 corrects the pixel value of the pixel corresponding to the mixed area of the background component image, and matches the position of the pixel with the foreground component image from which the motion blur has been removed.

FIG. 96 is a block diagram showing a configuration of the motion-blur-removed image processing unit 108 for generating a coefficient set used in the classification adaptive processing for generating a higher-resolution image in the spatial direction. For example, the motion-blur-removed image processing unit 108 having the configuration shown in FIG. 96 generates a coefficient set used in the classification adaptive processing for generating an HD image from an SD image based on the input HD image.

[0607] Background component teacher image frame memory 1001
Stores the corrected background component image of the teacher image supplied from the correction unit 107. The background component teacher image frame memory 1001 stores the stored background component image of the teacher image in the weighted average unit 1003-1 and the learning unit 1006-1.
To supply.

Foreground component teacher image frame memory 1002
Stores the foreground component image from which the motion blur of the teacher image has been removed, supplied from the motion blur removal unit 106. The foreground component teacher image frame memory 1002 supplies the stored foreground component image of the teacher image to the weighted average unit 1003-2 and the learning unit 1006-2.

The weighted averaging unit 1003-1 is supplied from the background component teacher image frame memory 1001, for example,
The background component image of the teacher image as the HD image is weighted by a quarter to generate an SD image as the student image, and the generated SD image is supplied to the background component student image frame memory 1004.

For example, the weighted average unit 1003-1
As shown in FIG. 7, four 2 × 2 (horizontal × vertical) pixels (indicated by white circles in FIG. 7) of the teacher image are defined as one unit.
The pixel values of the four pixels in each unit are added, and the added result is divided by four. The weighted averaging unit 1003-1 sets the result of the quarter weighted averaging in the pixels of the student image (portions indicated by black circles in the figure) located at the center of each unit.

[0611] Background component student image frame memory 1004
Stores the student image corresponding to the background component image of the teacher image supplied from the weighted averaging unit 1003-1. The background component student image frame memory 1004 supplies the stored student image corresponding to the background component image of the teacher image to the learning unit 1006-1.

The weighted averaging unit 1003-2 is supplied from the foreground component teacher image frame memory 1002, for example,
The foreground component image of the teacher image, which is an HD image, is weighted by a quarter to generate an SD image, which is a student image, and the generated SD image is supplied to the foreground component student image frame memory 1005.

[0613] Foreground component student image frame memory 1005
Stores a student image, which is an SD image, corresponding to the foreground component image of the teacher image, supplied from the weighted averaging unit 1003-2. The foreground component student image frame memory 1005
The stored student image corresponding to the foreground component image of the teacher image is supplied to the learning unit 1006-2.

The learning unit 1006-1 includes a background component image of the teacher image supplied from the background component teacher image frame memory 1001 and the background component student image frame memory 10
A coefficient set corresponding to the background component image is generated based on the student image corresponding to the background component image of the teacher image, and the generated coefficient set is stored in the coefficient set memory 10.
07.

The learning unit 1006-2 includes the foreground component image of the teacher image supplied from the foreground component teacher image frame memory 1002 and the foreground component student image frame memory 10.
A coefficient set corresponding to the foreground component image is generated based on the student image corresponding to the foreground component image of the teacher image supplied from the controller unit 05, and the generated coefficient set is stored in the coefficient set memory 10.
07.

The coefficient set memory 1007 stores the learning unit 10
The coefficient set corresponding to the background component image supplied from 06-1 and the coefficient set corresponding to the foreground component image supplied from the learning unit 1006-2 are stored.

The learning unit 1006-1 and the learning unit 1 will be described below.
When it is not necessary to distinguish 006-2 individually, it is simply referred to as a learning unit 1006.

FIG. 98 is a block diagram showing the structure of the learning unit 1006.

[0619] The class classification unit 1031 is composed of a class tap acquisition unit 1051 and a waveform classification unit 1052, and classifies the target pixel, which is the target pixel, of the input student image. The class tap acquisition unit 1051
A predetermined number of class taps, which are pixels of the student image corresponding to the target pixel, are acquired, and the acquired class taps are supplied to the waveform classification unit 1052.

[0620] For example, in FIG. 97, the i-th from the top, (in the figure, portions indicated by black circles) pixel of the j-th student image from the left when the the represented as X _ij, the class tap acquiring unit 10
_{Reference numeral} 51 denotes eight pixels X _{(i-1) (j-1)} , X adjacent to the upper left, upper, upper right, left, right, lower left, lower, and lower right of the target pixel Xij
_{(i-1) j} , X _{(i-1) (j + 1)} , Xi _(j-1) , Xi _{(j + 1)} , X
_{(i-1) (j-1)} , X _{(i-1) j} , X _{(i + 1) (j + 1)}
A class tap composed of a total of 9 pixels is obtained. The class tap is supplied to the waveform classification unit 1052.

In this case, the class tap is 3 × 3
It will be composed of square blocks composed of pixels, but the shape of the class classification block does not need to be square, and may be any other shape such as a rectangle, a cross, or any other shape. It is possible. Further, the number of pixels constituting the class tap is not limited to 9 pixels of 3 × 3.

The waveform classifying unit 1052 executes a class classification process for classifying the input signal into several classes based on the characteristics thereof, and determines one pixel of interest based on the class tap.
Into two classes. For example, the waveform classification unit 1052 classifies the pixel of interest into one of the 512 classes, and supplies a class number corresponding to the classified class to the prediction tap acquisition unit 1032.

Here, the class classification processing will be briefly described.

Now, for example, as shown in FIG. 99 (A), a class tap consisting of 2 × 2 pixels is constituted by a certain pixel of interest and three pixels adjacent thereto, and each pixel has one bit. (Takes any level of 0 or 1). In this case, as shown in FIG. 99 (B), a block of 4 pixels of 2 × 2 including the pixel of interest has 16 (= (2
¹ ) ⁴ ) Can be classified into patterns. Therefore, in this case, the target pixel can be classified into 16 patterns, and such pattern classification is a class classification process, and is performed by the class classification unit 1031.

Note that the class classification processing can be performed in consideration of the activity (complexity of the image) (the degree of change) of the image (class tap) and the like.

Here, generally, for example, about 8 bits are assigned to each pixel. Further, in the present embodiment, as described above, the class tap is composed of 3 × 3 nine pixels. Therefore, if the class tapping process is performed on such class taps, the class taps are classified into an enormous number of (2 ⁸ ) ⁹ classes.

Therefore, in the present embodiment, waveform classifying section 1052 performs ADR
C processing is performed, and thereby, the number of classes is reduced by reducing the number of bits of the pixels constituting the class tap.

For simplicity of explanation, as shown in FIG. 100 (A), considering a class tap composed of four pixels arranged on a straight line, in ADRC processing, the maximum value MAX of the pixel value and The minimum value MIN is detected. Then, let DR = MAX−MIN be the local dynamic range of the block constituted by the class taps, and based on this dynamic range DR, the pixel values of the pixels constituting the block of the class tap are requantized to K bits. Is done.

[0629] That is, from each pixel value in the block, subtracts the minimum value MIN, dividing the subtracted value by DR / 2 ^K.
Then, the code (A) corresponding to the resulting division value
DRC code). Specifically, for example, K
In the case where = 2, as shown in FIG. 100 (B), it is determined whether the divided value belongs to any range obtained by equally dividing the dynamic range DR by 4 (= 2 ² ).
The range of the lowest level, the range of the second lowest level,
In the case of belonging to the range of the third level from the bottom or the range of the top level, for example, 00B, 0
It is encoded into two bits such as 1B, 10B, or 11B (B represents a binary number). Then, on the decoding side, the ADRC codes 00B, 01B, 10
B or 11B is the center value L ₀₀ of the range of the lowest level obtained by dividing the dynamic range DR into four, the center value L ₀₁ of the range of the second level from the bottom, and the center value L _{01 of} the third level from the bottom. The conversion is performed to the center value L ₁₀ of the range or the center value L ₁₁ of the range of the highest level, and the minimum value MIN is added to the value to perform decoding.

Here, such ADRC processing is called non-edge matching.

The details of the ADRC processing are disclosed in, for example, Japanese Patent Application Laid-Open No. 3-53778, filed earlier by the present applicant.

By performing ADRC processing for performing re-quantization with a smaller number of bits than the number of bits allocated to the pixels constituting the class tap, the number of classes can be reduced as described above. The ADRC process is performed in the waveform classification unit 1052.

In the present embodiment, the waveform classification unit 10
At 52, a class classification process is performed based on the ADRC code. The class classification process includes, for example, DPCM (Predictive Coding) and BTC (Block Trunca).
It is also possible to perform processing on data that has been subjected to, for example, option coding, VQ (vector quantization), DCT (discrete cosine transform), and Hadamard transform.

The prediction tap acquisition unit 1032 is a unit for calculating the prediction value of the original image (teacher image) corresponding to the class from the pixels of the student image based on the class number.
The prediction tap is acquired, and the acquired prediction tap and class number are supplied to the corresponding pixel acquisition unit 1033.

For example, in FIG. 97, the pixels of the student image
X_ij(The original image centered on the black circle in the figure)
The pixel values of 2 × 2 9 pixels in the (teacher image)
From left to right and top to bottom, Y_ij(1),
Y_ij(2), Y_ij(3), Y _ijIf expressed as (4),
Pixel Y_ij(1) to Y_ijNecessary for calculating the predicted value of (4)
To calculate the coefficient, the prediction tap acquisition unit 1032 calculates
For example, pixel X of a student image_ij3 × 3 9 images centered on
Element X_{(i-1) (j-1)}, X_{(i-1) j}, X_{(i-1) (j + 1)}, X _{i (j-1)},
X_ij, X_{i (j + 1)}, X_{(i + 1) (j-1)}, X_{(i + 1) j}, X
_{(i + 1) (j + 1)}Get a square prediction tap consisting of
You.

Specifically, for example, in FIG.
The pixel Y in the teacher image ₃₃(1) to Y
₃₃Calculate coefficients necessary for calculating the predicted value of four pixels in (4)
Pixel X_{twenty two}, X_{twenty three}, X_{twenty four}, X₃₂, X₃₃, X₃₄, X
₄₂, X₄₃, X₄₄Constitutes a prediction tap (this
The target pixel in the case is X₃₃Becomes).

The corresponding pixel obtaining unit 1033 obtains the pixel value of the pixel of the teacher image corresponding to the pixel value to be predicted, based on the prediction tap and the class number, and calculates the prediction tap, the class number, and the obtained predicted value. The pixel value of the pixel of the teacher image corresponding to the pixel value is supplied to the normal equation generation unit 1034.

[0638] For example, the corresponding pixel acquiring unit
Pixel Y in image₃₃(1) to Y ₃₃(4) of four pixels
When calculating the coefficients required to calculate the predicted value,
As a pixel of the teacher image corresponding to the pixel value, a pixel Y
₃₃(1) to Y₃₃The pixel value of (4) is obtained.

The normal equation generation unit 1034 calculates a coefficient set used in the adaptive processing corresponding to the relationship between the prediction tap and the pixel value to be predicted, based on the prediction tap, the class number, and the acquired pixel value to be predicted. Is generated, and the generated normal equation is supplied to the coefficient calculation unit 1035 together with the class number.

[0640] The coefficient calculation unit 1035 solves the normal equation supplied from the normal equation generation unit 1034, and calculates a coefficient set corresponding to the classified class and used in the adaptive processing. The coefficient calculator 1035 stores the calculated coefficient set together with the class number in the coefficient set memory 100.
7

The normal equation generation unit 1034 generates a matrix corresponding to such a normal equation,
5 may calculate a coefficient set based on the generated matrix.

Here, the adaptive processing will be described.

For example, now, the prediction value E [y] of the pixel value y of the teacher image is changed to pixel values of some pixels around it (hereinafter, appropriately referred to as student data) x ₁ , x ₂ ,. And a predetermined linear prediction model defined by a linear combination of predetermined prediction coefficients w ₁ , w ₂ ,... In this case, the predicted value E [y] can be expressed by the following equation.

E [y] = w ₁ x ₁ + w ₂ x ₂ +... (111)

Therefore, for generalization, a matrix W composed of a set of prediction coefficients w, a matrix X composed of a set of student data, and a matrix Y ′ composed of a set of predicted values E [y] are

(Equation 26) Defines the following observation equation.

XW = Y ′ (112)

[0647] Then, it is considered that a least square method is applied to this observation equation to obtain a predicted value E [y] close to the pixel value y of the original image. In this case, a matrix Y composed of a set of pixel values y of the original image (hereinafter, appropriately referred to as teacher data),
And a matrix E consisting of a set of residuals e of the predicted values E [y] for the pixel values y of the original image,

[Equation 27] , The following residual equation holds from equation (112).

XW = Y + E (113)

In this case, the prediction coefficient w _i for obtaining a prediction value E [y] close to the pixel value y of the original image is a square error

[Equation 28] Can be obtained by minimizing.

Therefore, when the above-described squared error is differentiated by the prediction coefficient w _i , the result becomes 0, that is, the prediction coefficient w _i that satisfies the following equation is a prediction value E [y that is close to the pixel value y of the original image. ]
Is the optimum value.

[0651]

(Equation 29)

Therefore, first, the following equation is established by differentiating equation (113) with the prediction coefficient w _i .

[0653]

[Equation 30]

From the expressions (114) and (115), the expression (116) is obtained.

[0655]

(Equation 31)

Further, considering the relationship among student data x, prediction coefficient w, teacher data y, and residual e in the residual equation of equation (113), the following normal equation is obtained from equation (116). be able to.

[0657]

(Equation 32)

The normal equation of Expression (117) can be set by the same number as the number of prediction coefficients w to be obtained. Therefore, by solving Expression (117), the optimum prediction coefficient w can be obtained. . In solving equation (117), for example, a sweeping-out method (Gauss-Jordan elimination method) or the like can be applied.

As described above, the optimum prediction coefficient w is obtained for each class, and the prediction coefficient E close to the pixel value y of the teacher image is calculated by the equation (111) using the prediction coefficient w.
Finding [y] is adaptive processing.

The normal equation generation unit 1034 generates a normal equation for calculating the optimum prediction coefficient w for each class, and the coefficient calculation unit 1035 calculates the prediction coefficient w based on the generated normal equation. .

The adaptive processing is different from the interpolation processing in that components included in the original image that are not included in the thinned image are reproduced. That is, the adaptive processing is performed according to the equation (11).
As far as 1) is concerned, it is the same as the interpolation processing using the so-called interpolation filter, but the prediction coefficient w corresponding to the tap coefficient of the interpolation filter is obtained by learning using the teacher data y. Therefore, the components included in the original image can be reproduced. From this, it can be said that the adaptive processing has a so-called image creation action.

FIG. 101 is a view for explaining a coefficient set generated by the motion-blur-removed image processing unit 108 having the structure shown in FIG. For the input image, the area specifying unit 103 specifies a foreground area, a background area, a covered background area, and an uncovered background area.

The input image in which the region is specified and the mixture ratio α is detected by the mixture ratio calculation unit 104 is the foreground / background separation unit 1
05 separates the image into a foreground component image and a background component image.

[0666] The motion blur removing unit 106 removes the motion blur from the separated foreground component image. The pixel value corresponding to the mixed area of the separated background component image is
7, correction is made in response to removal of motion blur in the foreground component image.

The motion-blur-removed image processing unit 108 calculates a coefficient set corresponding to the foreground component image and a coefficient corresponding to the background component image based on the foreground component image from which the motion blur has been removed and the corrected background component image. Calculate sets individually.

That is, the learning unit 1006-1 calculates a coefficient set corresponding to the background component image based on the separated and corrected background component image, and the learning unit 1006-2 separates and sets the motion blur. A coefficient set corresponding to the foreground component image is calculated based on the removed foreground component image.

[0667] The coefficient set corresponding to the background component image is used for predicting the pixel value of the image corresponding to the background component image in the class classification adaptive processing for predicting the pixel value to be applied to the separated and corrected background component image. used.

The coefficient set corresponding to the foreground component image is applied to the foreground component image which has been separated from the input image and from which motion blur has been removed. Is used to predict the pixel value of.

The predicted image corresponding to the foreground component image is added with motion blur. The predicted image corresponding to the background component image is corrected according to the addition of the motion blur of the foreground component image.

[0670] The predicted image corresponding to the corrected background component image and the predicted image corresponding to the foreground component image to which motion blur has been added are combined to form one predicted image.

With reference to the flowchart in FIG. 102, description will be given of learning processing for generating a coefficient set used for predicting a pixel value by the classification adaptive processing by the motion-blur removal image processing unit 108 having the configuration shown in FIG. I do.

In step S1001, the weighted averaging unit 1003-1 and the weighted averaging unit 1003-2 generate a student image corresponding to the background component image and a student image corresponding to the foreground component image. That is, the weighted average unit 100
3-1 represents the background component image of the teacher image stored in the background component teacher image frame memory 1001, for example,
A quarter weighted average is used to generate a student image corresponding to the background component image of the teacher image.

The weighted average unit 1003-2 performs, for example, a quarter weighted average of the foreground component images of the teacher image stored in the foreground component teacher image frame memory 1002,
A student image corresponding to the foreground component image of the teacher image is generated.

In step S1002, the learning section 10
06-1 is a background component teacher image frame memory 1001
The coefficient set corresponding to the background component image is determined based on the background component image of the teacher image stored in the image data and the student image corresponding to the background component image of the teacher image stored in the background component student image frame memory 1004. Generate. Details of the process of generating the coefficient set in step S1002 will be described later with reference to the flowchart in FIG.

In step S1003, the learning unit 10
06-2 is a foreground component teacher image frame memory 1002
The coefficient set corresponding to the foreground component image is calculated based on the foreground component image of the teacher image stored in the foreground component student image stored in the foreground component student image frame memory 1005 and the student image corresponding to the foreground component image of the teacher image. Generate.

In step S1004, the learning section 10
06-1 and the learning unit 1006-2 output the coefficient set corresponding to the background component image or the coefficient set corresponding to the foreground component image to the coefficient set memory 1007, respectively. The coefficient set memory 1007 stores the coefficient set corresponding to the background component image or the coefficient set corresponding to the foreground component image, respectively, and the process ends.

As described above, the motion-blur-removed image processing unit 108 shown in FIG. 96 can generate a coefficient set corresponding to the background component image and a coefficient set corresponding to the foreground component image.

[0678] Needless to say, the processing of steps S1002 and S1003 may be executed serially or in parallel.

Next, referring to the flowchart in FIG. 103, the learning unit 10 corresponding to the processing in step S1002 will be described.
A process of generating a coefficient set corresponding to a background component image performed by the image processing unit 06-1 will be described.

In step S1021, the learning unit 10
06-1 is to determine whether there is an unprocessed pixel in the student image corresponding to the background component image. If it is determined that there is an unprocessed pixel in the student image corresponding to the background component image, step S1022 To obtain a pixel of interest from the student image corresponding to the background component image in the raster scan order.

In step S1023, the class tap acquisition unit 1051 of the class classification unit 1031 acquires a class tap corresponding to the pixel of interest from the student image stored in the background component student image frame memory 1004. In step S1024, the class classification unit 103
1 for the class tap.
The DRC process is applied, thereby reducing the number of bits of the pixels forming the class tap, and classifying the target pixel into a class. In step S1025, the prediction tap acquisition unit 1032 acquires a prediction tap corresponding to the pixel of interest from the student image stored in the background component student image frame memory 1004 based on the classified class.

In step S1026, the corresponding pixel acquisition unit 1033 determines the pixel value to be predicted from the background component image of the teacher image stored in the background component teacher image frame memory 1001 based on the classified class. Get a pixel.

In step S1027, the normal equation generating unit 1034 adds the prediction tap and the pixel value of the pixel corresponding to the pixel value to be predicted to the matrix for each class based on the classified class, and proceeds to step S1021. Returning, the learning unit 1006-1 repeatedly determines whether there is an unprocessed pixel. The matrix for each class to which the pixel value of the pixel corresponding to the prediction tap and the pixel value to be predicted is added corresponds to a normal equation for calculating a coefficient set for each class.

If it is determined in step S1021 that there is no unprocessed pixel in the student image, step S10
Proceeding to 28, the normal equation generating unit 1034 supplies the coefficient calculating unit 1035 with a matrix for each class in which the prediction tap and the pixel value of the pixel corresponding to the pixel value to be predicted are set. The coefficient calculation unit 1035 solves a matrix for each class in which the prediction tap and the pixel value of the pixel corresponding to the pixel value to be predicted are set, and calculates a coefficient set for each class corresponding to the background component image.

The coefficient calculation unit 1035 is not limited to a coefficient set for predicting a pixel value by linear prediction, but may calculate a coefficient set for predicting a pixel value by non-linear prediction.

[0686] In step S1029, the coefficient calculator 1035 outputs a coefficient set for each class corresponding to the background component image to the coefficient set memory 1007, and the process ends.

As described above, the learning unit 1006-1 can generate a coefficient set corresponding to the background component image.

The learning unit 1 corresponding to step S1003
The process of generating the coefficient set corresponding to the foreground component image according to 006-2 is performed by using
1 except that the foreground component image stored in the foreground component image and the student image corresponding to the foreground component image stored in the foreground component student image frame memory 105 are used.
Since the process is the same as that described with reference to the flowchart of FIG. 03, the description thereof is omitted.

[0689] As described above, the motion-blur-removed image processing unit 108 having the structure shown in Fig. 96 generates the coefficient set corresponding to the corrected background component image and the coefficient set corresponding to the foreground component image from which the motion blur has been removed. Can be generated individually.

FIG. 104 is a block diagram showing a configuration of the motion-blur-removed image processing unit 108 for executing a classification adaptive process to generate a higher-resolution image in the spatial direction. For example, the motion-blur-removed image processing unit 108 having the configuration shown in FIG. 104 performs a classification adaptive process based on an input image that is an SD image, and generates an HD image.

The background component image frame memory 1101
The corrected background component image supplied from the correction unit 107 is stored. The background component image frame memory 1101 is
The stored background component image is mapped to the mapping unit 1103-1.
To supply.

The foreground component image frame memory 1102
The foreground component image from which motion blur has been removed, supplied from the motion blur removal unit 106, is stored. The foreground component image frame memory 1102 supplies the stored foreground component image to the mapping unit 1103-2.

The mapping unit 1103-1 performs the class classification adaptive processing based on the coefficient set corresponding to the background component image stored in the coefficient set memory 1104, and performs the background storage stored in the background component image frame memory 1101. A predicted image corresponding to the component image is generated. The mapping unit 1103-1 converts the generated predicted image into a correction unit 1105.
To supply.

The correction unit 1105 includes the motion blur adding unit 110
6 corresponding to the motion blur to be added, the pixel value of a predetermined pixel of the predicted image corresponding to the mixed area of the background component image is set to 0, or a predetermined value corresponding to the motion blur to be added is set. , The pixel value of a predetermined pixel of the predicted image is divided. The correction unit 1105 supplies the predicted image thus corrected to the synthesis unit 1107.

The mapping unit 1103-2 performs the classification adaptation processing based on the coefficient set corresponding to the foreground component image stored in the coefficient set memory 1104, and performs the foreground component image frame memory 1102 stored in the foreground component image frame memory 1102. A predicted image corresponding to the component image is generated. The mapping unit 1103-2 supplies the generated predicted image to the motion blur adding unit 1106.

[0696] The motion-blur adding unit 1106 generates a desired motion-blur adjustment amount v ', for example, a motion-blur adjustment amount v' having a value half of the motion amount v of the input image, or a motion-blur adjustment value having a value independent of the motion amount v. By giving the adjustment amount v ′, motion blur is added to the predicted image. The motion blur adding unit 1106 calculates the pixel value Fi of the predicted image of the foreground component image from which the motion blur has been removed by the motion blur adjustment amount v ′.
, The foreground component Fi / v ′ is calculated, the sum of the foreground components Fi / v ′ is calculated, and a pixel value to which motion blur is added is generated.

For example, when the predicted image shown in FIG. 105 is input and the motion blur adjustment amount v ′ is 3, the pixel value C02 is (F01) / v ′ and the pixel value C03 is as shown in FIG. Is
(F01 + F02) / v ', and the pixel value C04 is (F01 + F02 + F03)
/ v ', and the pixel value C05 is (F02 + F03 + F04) / v'.

[0698] The motion blur adding unit 1106 combines the predicted image of the foreground component image to which the motion blur has been added in this way with the synthesizing unit 1.
107.

[0699] The synthesizing unit 1107 corresponds to the predicted image corresponding to the corrected background component image supplied from the correcting unit 1105 and the foreground component image to which motion blur has been added, supplied from the motion blur adding unit 1106. The predicted image to be synthesized is synthesized, and the synthesized predicted image is stored in the frame memory 1108.
To supply.

[0700] The frame memory 1108 is
7 and outputs the stored image as an output image.

Hereinafter, when it is not necessary to distinguish mapping section 1103-1 and mapping section 1103-2 from each other, they are simply referred to as mapping section 1103.

FIG. 107 is a block diagram showing a configuration of mapping section 1103.

The mapping processing section 1131 is composed of a class classification section 1141 for performing a classification processing, a prediction tap acquisition section 1142 for executing an adaptive processing, and a prediction calculation section 1143.

The classifying section 1141 is composed of a class tap obtaining section 1151 and a waveform classifying section 1152, and is a pixel of interest of the input image of either the background component image or the foreground component image. Are classified.

[0705] The class tap acquiring unit 1151 acquires a predetermined number of class taps corresponding to the target pixel of the input image, and supplies the acquired class taps to the waveform classifying unit 1152. For example, the class tap acquisition unit 1151 acquires nine class taps, and supplies the acquired class taps to the waveform classification unit 1152.

The waveform classification unit 1152 applies ADRC processing to the class taps, thereby reducing the number of bits of the pixels forming the class taps, and setting the pixel of interest to one of a predetermined number of classes. The class is classified into two classes, and the class number corresponding to the classified class is supplied to the prediction tap acquisition unit 1142. For example, the waveform classification unit 1152
The target pixel is classified into one of the 512 classes, and a class number corresponding to the classified class is supplied to the prediction tap acquisition unit 1142.

[0707] The prediction tap obtaining unit 1142 obtains a predetermined number of prediction taps corresponding to the class from the input image based on the class numbers, and supplies the obtained prediction taps and class numbers to the prediction calculation unit 1143. .

[0708] Based on the class number, the predictive operation unit 1143 uses the coefficient set corresponding to the background component image and the coefficient set corresponding to the foreground component image stored in the coefficient set memory 1104 to correspond to the input image, Get the coefficient set corresponding to the class. The prediction operation unit 1143
A pixel value of the predicted image is predicted by linear prediction based on a coefficient set corresponding to the class and a prediction tap corresponding to the class. The prediction calculation unit 1143 supplies the predicted pixel value to the frame memory 1132.

[0709] The prediction operation unit 1143 may predict the pixel value of the predicted image by nonlinear prediction.

[0710] The frame memory 1132 stores the predicted pixel values supplied from the mapping processing unit 1131, and outputs an image composed of the predicted pixel values.

Next, with reference to the flowchart in FIG. 108, the motion-blur-removed image processing unit 10 shown in FIG.
8 will be described.

[0712] In step S1101, the mapping unit 1103-1 uses the coefficient set corresponding to the background component image stored in the coefficient set memory 1104, based on the coefficient set.
The image corresponding to the background component image stored in the background component image frame memory 1101 is predicted by the classification adaptive processing. The details of the prediction process of the image corresponding to the background component image will be described later with reference to the flowchart in FIG.

[0713] In step S1102, the mapping unit 1103-2 performs processing based on the coefficient set corresponding to the foreground component image stored in the coefficient set memory 1104.
The image corresponding to the foreground component image stored in the foreground component image frame memory 1102 is predicted by the class classification adaptive processing.

[0714] In step S1103, the correction unit 11
05 corrects the predicted image corresponding to the background component image.

[0715] In step S1104, the motion blur adding unit 1106 adds motion blur to the predicted image corresponding to the foreground component image.

In step S1105, the synthesizing unit 11
07 combines a predicted image corresponding to the background component image and a predicted image corresponding to the foreground area. Synthesizing unit 1107
Supplies the synthesized image to the frame memory 1108. The frame memory 1108 stores the image supplied from the combining unit 1107.

[0717] In step S1106, the frame memory 1108 outputs the stored combined image, and the process ends.

As described above, the image processing apparatus having the motion-blur-removed image processing unit 108 shown in FIG. 104 generates a predicted image corresponding to the background component image, and corresponds to the foreground component image from which the motion blur has been removed. Prediction images can be generated individually.

[0719] It goes without saying that the processing of steps S1101 and S1102 may be executed serially or in parallel.

Referring to the flowchart in FIG. 109, mapping section 1103-corresponding to step S1101.
The process of predicting an image corresponding to a background component image according to No. 1 will be described.

[0721] In step S1121, the mapping unit 1103-1 determines whether there is an unprocessed pixel in the background component image. If it is determined that there is an unprocessed pixel in the background component image, the process proceeds to step S1122. Then, the mapping processing unit 1131 acquires a coefficient set corresponding to the background component image stored in the coefficient set memory 1104. In step S1123, the mapping processing unit 1131 acquires a pixel of interest from the background component image stored in the background component image frame memory 1101 in raster scan order.

[0722] In step S1124, the class tap acquisition unit 1151 of the class classification unit 1141 acquires a class tap corresponding to the pixel of interest from the background component image stored in the background component image frame memory 1101. In step S1125, the class classification unit 114
1 for the class tap.
The DRC process is applied, thereby reducing the number of bits of the pixels forming the class tap, and classifying the target pixel into a class. In step S1126, the prediction tap acquisition unit 1142 acquires a prediction tap corresponding to the target pixel from the background component image stored in the background component image frame memory 1101 based on the classified class.

[0723] In step S1127, the prediction calculation unit 1143 predicts the pixel value of the predicted image by linear prediction based on the background component image, the coefficient set corresponding to the classified class, and the prediction tap.

[0724] Note that the prediction calculation unit 1143 is not limited to linear prediction, and may predict pixel values of a predicted image by non-linear prediction.

[0725] In step S1128, the prediction calculation unit 1143 stores the predicted pixel value in the frame memory 113.
Output to 2. The frame memory 1132 stores the pixel value supplied from the prediction operation unit 1143. The procedure is
Returning to step S1121, the determination of whether or not there is an unprocessed pixel is repeated.

If it is determined in step S1121 that there is no unprocessed pixel in the background component image, the process proceeds to step S1121.
Proceeding to 1129, the frame memory 1132 outputs a predicted image corresponding to the stored background component image, and the process ends.

[0727] Thus, the mapping unit 1103-1
Can predict an image corresponding to the background component image based on the corrected background component image.

The process of generating a predicted image corresponding to the foreground component image by the mapping unit 1103-2 corresponding to step S1102 is performed by the foreground component image frame memory 11
2 except that the foreground component image stored in FIG. 02 and the coefficient set corresponding to the foreground component image are used.
Since the process is the same as that described with reference to the flowchart of FIG. 09, the description thereof is omitted.

As described above, the motion-blur-removed image processing unit 108 having the configuration shown in FIG. 104 generates a predicted image corresponding to the background component image, and individually generates the predicted image corresponding to the foreground component image from which the motion blur has been removed. Can be generated.

FIG. 110 is a block diagram showing another configuration of the function of the image processing apparatus. While the image processing apparatus shown in FIG. 11 sequentially performs the area specification and the calculation of the mixture ratio α,
The image processing apparatus shown in FIG. 110 performs region identification and calculation of the mixture ratio α in parallel.

Parts having the same functions as those shown in the block diagram of FIG. 11 are denoted by the same reference numerals, and description thereof will be omitted.

[0732] The input image is input to the object extraction unit 101,
The information is supplied to the area specifying unit 103, the mixture ratio calculating unit 1501, and the foreground / background separating unit 1502.

[0733] Based on the input image, the mixture ratio calculation unit 1501 calculates the estimated mixture ratio when the pixel belongs to the covered background area and the estimated mixture ratio when the pixel belongs to the uncovered background area. The ratio is calculated for each of the pixels included in the input image, and the estimated mixture ratio when the calculated pixel is assumed to belong to the covered background area,
The estimated mixture ratio when the pixel belongs to the uncovered background area is supplied to the foreground / background separation unit 1502.

FIG. 111 is a block diagram showing an example of the configuration of the mixture ratio calculating section 1501.

[0735] Estimated mixture ratio processing section 401 shown in FIG.
Is the same as the estimated mixture ratio processing unit 401 shown in FIG. The estimated mixture ratio processing unit 402 shown in FIG.
Is the same as the estimated mixture ratio processing unit 402 shown in FIG.

The estimated mixture ratio processing unit 401 calculates an estimated mixture ratio for each pixel by an operation corresponding to the model of the covered background area based on the input image, and outputs the calculated estimated mixture ratio.

The estimated mixture ratio processing unit 402 calculates an estimated mixture ratio for each pixel by an operation corresponding to the model of the uncovered background area based on the input image, and outputs the calculated estimated mixture ratio.

[0738] The foreground / background separation unit 1502 calculates the estimated mixture ratio supplied from the mixture ratio calculation unit 1501 when the pixel belongs to the covered background area.
Based on the estimated mixture ratio when it is assumed that the pixel belongs to the uncovered background area, and the area information supplied from the area specifying unit 103, the input image
The image is separated into a background component image and a foreground component image, and the separated images are supplied to the motion blur removal image processing unit 108.

FIG. 112 is a block diagram showing an example of the structure of the foreground / background separation unit 1502.

[0740] The same parts as those of the motion blur removing unit 106 shown in Fig. 80 are denoted by the same reference numerals, and description thereof will be omitted.

[0741] Based on the area information supplied from the area specifying section 103, the selecting section 1521 calculates the estimated mixing ratio supplied from the mixing ratio calculating section 1501 when the pixel belongs to the covered background area, and One of the estimated mixture ratios when the pixel is assumed to belong to the uncovered background area is selected, and the selected estimated mixture ratio is supplied to the separation unit 601 as the mixture ratio α.

The separating unit 601 extracts the foreground component and the background component from the pixel values of the pixels belonging to the mixed area based on the mixing ratio α and the area information supplied from the selecting unit 1521, and extracts the uncovered background area Background ingredients,
The foreground component of the uncovered background area, the background component of the covered background area, and the foreground component of the covered background area are separated.

[0743] The separation unit 601 can have the same configuration as the configuration shown in FIG.

As described above, the image processing apparatus having the configuration shown in FIG. 110 can execute processing for each of the background component image and the foreground component image in accordance with the respective properties.

As described above, in the image processing apparatus of the present invention, the input image is separated into the background component image and the foreground component image, and processing suitable for the separated images is executed. A higher resolution image is generated without generating an image.

FIG. 113 is a block diagram showing another configuration of the function of the image processing apparatus.

[0747] The same portions as those in the case shown in FIG. 11 are denoted by the same reference numerals, and description thereof will be omitted.

The object extracting unit 101 roughly extracts an image object corresponding to the foreground object included in the input image, and supplies the extracted image object to the motion detecting unit 102.

[0749] The motion detection unit 102 calculates the motion vector of the image object corresponding to the coarsely extracted foreground object by using a method such as a block matching method, a gradient method, a phase correlation method, or a pel recursive method. , And supplies the calculated motion vector and the position information of the motion vector to the area specifying unit 103.

[0750] The area specifying unit 103 specifies each of the pixels of the input image as one of a foreground area, a background area, an uncovered background area, or a mixed area including a covered background area. To the mixture ratio calculation unit 104 and the foreground / background separation unit 2
001.

The mixture ratio calculation unit 104 calculates a mixture ratio α corresponding to the pixels included in the mixture region based on the input image and the region information supplied from the region identification unit 103, and calculates the mixture ratio. This is supplied to the foreground / background separation unit 2001.

The foreground / background separation unit 2001 includes the area specifying unit 1
03 and the mixture ratio calculation unit 10
4, the image component corresponding to the foreground object and the background component image consisting only of the background component are separated on the basis of the mixture ratio α supplied from No. 4, and the image of the background area and the background of the uncovered background area are separated. (Hereinafter, referred to as a background component image of the uncovered background area), and an image including only the foreground component of the uncovered background area (hereinafter, referred to as a foreground component image of the uncovered background area). An image composed only of the background component of the covered background area (hereinafter, referred to as a background component image of the covered background area), and an image composed only of the foreground component of the covered background area (hereinafter, a foreground component of the covered background area) The image of the foreground area is divided into the separated image processing unit 20. 2 for supplying to.

The separated image processing unit 2002 supplies the background image, the uncovered background area background component image, the uncovered background area foreground component image, and the covered background area supplied from the foreground / background separation unit 2001. The background component image, the foreground component image in the covered background area, and the image in the foreground area are processed, respectively.

[0754] For example, the separated image processing unit 2002 includes a background area image, an uncovered background area background component image, an uncovered background area foreground component image, a covered background area background component image, and a covered background area. For each of the foreground component image and the image of the foreground area, a coefficient used in the classification adaptive processing for generating a higher-resolution image is generated.

[0755] For example, the separated image processing unit 2002 includes a background area image, an uncovered background area background component image, an uncovered background area foreground component image, a covered background area background component image, and a covered background area. A higher resolution image is created by applying the classification adaptive processing to each of the foreground component image and the image of the foreground region.

FIG. 114 is a view for explaining the processing of the separated image processing section 2002 shown in FIG. 113. For the input image, the area specifying unit 103 specifies a foreground area, a background area, a covered background area, and an uncovered background area.

The input image for which the region is specified and the mixture ratio α is detected by the mixture ratio calculation unit 104 is input to the foreground / background separation unit 2.
001, the foreground area image, the background area image, the foreground component image in the covered background area, the background component image in the covered background area, the foreground component image in the uncovered background area, and the background component in the uncovered background area Separated into images.

[0758] The separated image processing unit 2002 includes a separated foreground region image, a background region image, a covered background region foreground component image, a covered background region background component image, and an uncovered background region foreground component image. A coefficient set corresponding to the image of the foreground area, a coefficient set corresponding to the image of the background area, a coefficient set corresponding to the foreground component image of the covered background area, and a covered set based on the background component image of the uncovered background area. A coefficient set corresponding to the background component image in the background area, a coefficient set corresponding to the foreground component image in the uncovered background area, and a coefficient set corresponding to the background component image in the uncovered background area are individually calculated.

[0759] The coefficient set corresponding to the background area is used for predicting the pixel value of the background area in the classification adaptive processing for predicting the pixel value. The coefficient set corresponding to the background component image in the uncovered background area is used for predicting the pixel value corresponding to the background component image in the uncovered background area in the class classification adaptive processing for predicting the pixel value. The coefficient set corresponding to the foreground component image in the uncovered background area is used for predicting the pixel value corresponding to the foreground component image in the uncovered background area in the class classification adaptive processing for predicting the pixel value.

The coefficient set corresponding to the background component image in the covered background area is used for predicting the pixel value corresponding to the background component image in the covered background area in the classification adaptive processing for predicting the pixel value.
The coefficient set corresponding to the foreground component image in the covered background area is used for predicting the pixel value corresponding to the foreground component image in the covered background area in the class classification adaptive processing for predicting the pixel value.

The coefficient set corresponding to the foreground area is used for predicting the pixel value of the foreground area in the class classification adaptive processing for predicting the pixel value.

[0762] A predicted image corresponding to the background area image, a predicted image corresponding to the background component image in the uncovered background area, a predicted image corresponding to the foreground component image in the uncovered background area, and a background component in the covered background area The predicted image corresponding to the image, the predicted image corresponding to the foreground component image in the covered background area, and the predicted image corresponding to the image in the foreground area are combined into one predicted image.

FIG. 115 is a block diagram showing an example of the structure of the foreground / background separation unit 2001. Foreground / background separator 20
The input image supplied to 01 is supplied to a separation unit 2101, a switch 2102, and a switch 2103. Information indicating covered background area, and area specifying unit 103 indicating uncovered background area
Are supplied to the separation unit 2101. Area information indicating the foreground area is supplied to the switch 2102. The area information indicating the background area is stored in the switch 210
3 is supplied.

[0764] The mixture ratio α supplied from the mixture ratio calculation unit 104 is supplied to the separation unit 2101.

[0765] The separation unit 2101 separates the foreground component from the covered background region of the input image based on the region information indicating the covered background region and the mixture ratio α, and separates the background component. And outputting a foreground component image of a covered background area composed of the separated foreground components and a background component image of a covered background area composed of the separated background components.

[0766] The separation unit 2101 calculates the mixing ratio α based on the area information indicating the uncovered background area and the mixing ratio α.
A foreground component image of an uncovered background area composed of separated foreground components while separating a foreground component from an uncovered background area of an input image, and a separated background The background component image of the uncovered background area composed of the components is output.

The switch 2102 is closed when a pixel corresponding to the foreground area is input based on the area information indicating the foreground area, and outputs an image of the foreground area.

The switch 2103 is closed when a pixel corresponding to the background area is input, based on the area information indicating the background area, and outputs an image of the background area.

FIG. 116 is a block diagram showing an example of the structure of separation section 2101. The image input to the separation unit 2101 is supplied to the frame memory 2121, and the area information indicating the covered background area and the uncovered background area supplied from the mixture ratio calculation unit 104 and the mixture ratio α are stored in the separation processing block. 2122.

[0770] The frame memory 2121 stores the input image in frame units. Frame memory 2121
When the processing target is frame #n, frame # n−1, which is the frame immediately before frame #n, frame #n, and frame # n +, which is the frame immediately after frame #n 1
Is stored.

The frame memory 2121 stores the frame # n-
The pixels corresponding to 1, frame #n, and frame # n + 1 are supplied to the separation processing block 2122.

[0772] The separation processing block 2122 calculates the frame # n-1, frame #n, and frame # n-1 supplied from the frame memory 2121 based on the area information indicating the covered background area and the uncovered background area, and the mixture ratio α. Applying the calculation described with reference to FIGS. 83 and 84 to the pixel value of the corresponding pixel in frame # n + 1,
The foreground component and the background component are separated from the pixels belonging to the mixed area of the frame #n.

The separation processing block 2122 includes an uncovered area processing section 2131 and a covered area processing section 21.
32.

The multiplier 2141 of the uncovered area processing unit 2131 multiplies the mixture ratio α by the pixel value of the pixel of frame # n + 1 supplied from the frame memory 2121 and outputs the result to the switch 2142. The switch 2142 is closed when the pixel of frame #n (corresponding to the pixel of frame # n + 1) supplied from the frame memory 2121 is in the uncovered background area, and the mixture ratio supplied from the multiplier 2141 The pixel value multiplied by α is calculated by the arithmetic unit 2143
To supply. Frame output from switch 2142
The value obtained by multiplying the pixel value of pixel # n + 1 by the mixture ratio α
It is equal to the background component of the pixel value of the corresponding pixel of #n, and is output as the background component image of the uncovered background area.

The arithmetic unit 2143 is provided with a frame memory 212
The foreground component is obtained by subtracting the background component supplied from the switch 2142 from the pixel value of the pixel of frame #n supplied from # 1. Arithmetic unit 2143 outputs a foreground component image of a pixel of frame #n belonging to the uncovered background area.

The multiplier 21 of the covered area processing unit 2132
51 multiplies the mixture ratio α by the pixel value of the pixel of frame # n−1 supplied from the frame memory 2121 and outputs the result to the switch 2152. The switch 2152 is closed when the pixel of frame #n (corresponding to the pixel of frame # n-1) supplied from the frame memory 2121 is in the covered background area, and the mixture ratio α supplied from the multiplier 2151 Is supplied to the computing unit 2153. The value obtained by multiplying the pixel value of the pixel of frame # n−1 output from the switch 2152 by the mixture ratio α is equal to the background component of the pixel value of the corresponding pixel of frame #n, and the background component of the covered background area. Output as an image.

[0777] The arithmetic unit 2153 includes a frame memory 212.
The foreground component is obtained by subtracting the background component supplied from the switch 2152 from the pixel value of the pixel of frame #n supplied from # 1. Arithmetic unit 2153 outputs a foreground component image of a pixel of frame #n belonging to the covered background area.

By using the mixture ratio α, which is a feature quantity, it becomes possible to completely separate the foreground component and the background component contained in the pixel value.

FIG. 117 is a block diagram showing a configuration of a separated image processing unit 2002 for generating a coefficient set used in a class classification adaptive process for generating a higher resolution image in the spatial direction. For example, the separated image processing unit 2002 whose configuration is shown in FIG. 117, based on the input HD image,
A coefficient set used in the classification adaptive processing for generating an HD image from an SD image is generated.

[0780] Background area teacher image frame memory 2201
Stores the image of the background area of the teacher image supplied from the foreground / background separation unit 2001. The background area teacher image frame memory 2201 stores the image of the background area of the stored teacher image in the weighted average unit 2207-1 and the learning unit 2214.
-1.

[0781] The uncovered background area background component teacher image frame memory 2202 stores the background component image of the uncovered background area of the teacher image supplied from the foreground / background separation unit 2001. The uncovered background area background component teacher image frame memory 2202 weights the background component image of the uncovered background area of the stored teacher image to the weighted average unit 22.
07-2 and the learning unit 2214-2.

[0782] The uncovered background area foreground component teacher image frame memory 2203 stores the foreground component image of the uncovered background area of the teacher image supplied from the foreground / background separation unit 2001. The uncovered background area foreground component teacher image frame memory 2203 weights the foreground component image of the stored uncovered background area of the teacher image to the weighted average unit 22.
07-3 and the learning unit 2214-3.

The covered background area background component teacher image frame memory 2204 stores the foreground / background
The background component image of the covered background area of the teacher image supplied from 01 is stored. Covered background area background component teacher image frame memory 2204
Supplies the stored background component image of the covered background area of the teacher image to the weighted averaging unit 2207-4 and the learning unit 2214-4.

The covered background area foreground component teacher image frame memory 2205 stores the foreground / background
The foreground component image of the covered background area of the teacher image supplied from 01 is stored. Covered background area foreground component teacher image frame memory 2205
Supplies the stored foreground component image in the covered background area of the teacher image to the weighted average unit 2207-5 and the learning unit 2214-5.

[0785] Foreground area teacher image frame memory 2206
Stores the image of the foreground area of the teacher image supplied from the foreground / background separation unit 2001. The foreground area teacher image frame memory 2206 compares the stored image of the foreground area of the teacher image with the weighted average unit 2207-6 and the learning unit 2214.
-6.

[0786] The weighted average section 2207-1 is supplied from the background area teacher image frame memory 2201, for example,
An image of the background area of the teacher image, which is an HD image, is weighted by a quarter to generate an SD image, which is a student image, and the generated SD is generated.
The image is supplied to the background area student image frame memory 2208.

[0787] Background area student image frame memory 2208
Stores the student image corresponding to the image in the background area of the teacher image supplied from the weighted average unit 2207-1. The background area student image frame memory 2208 supplies the stored student image corresponding to the image of the background area of the teacher image to the learning unit 2214-1.

[0788] The weighted averaging unit 2207-2 converts the background component image of the uncovered background area of the HD teacher image supplied from the uncovered background area background component teacher image frame memory 2202 into, for example, 4 minutes. An SD image, which is a student image, is generated by performing a weighted average of the above, and the generated SD image is supplied to the uncovered background area background component student image frame memory 2209.

The uncovered background area background component student image frame memory 2209 stores the weighted average
A student image, which is an SD image, corresponding to the background component image in the uncovered background area of the teacher image supplied from 07-2 is stored. The uncovered background area background component student image frame memory 2209 learns the stored student image corresponding to the background component image in the uncovered background area of the teacher image.
4-2.

The weighted averaging unit 2207-3 converts the foreground component image of the uncovered background area of the teacher image, which is an HD image, supplied from the uncovered background area foreground component teacher image frame memory 2203 into, for example, 4 minutes. An SD image, which is a student image, is generated by performing a weighted average of the above, and the generated SD image is supplied to the uncovered background area foreground component student image frame memory 2210.

The uncovered background area foreground component student image frame memory 2210 stores the weighted average
A student image, which is an SD image, corresponding to the foreground component image in the uncovered background area of the teacher image supplied from 07-3 is stored. The uncovered background area foreground component student image frame memory 2210 learns the stored student image corresponding to the foreground component image of the uncovered background area of the teacher image.
Supply to 4-3.

[0792] The weighted average section 2207-4 includes the covered background area background component teacher image frame memory 22.
For example, the background component image of the covered background area of the teacher image supplied from the image generation unit 04 is subjected to, for example, a quarter weighted averaging to generate an SD image as a student image, and the generated SD image is covered by the covered background area background. The component is supplied to the student image frame memory 2211.

The covered background area background component student image frame memory 2211 stores the weighted average section 2207
And storing the student image, which is an SD image, corresponding to the background component image in the covered background area of the teacher image supplied from # 4. The covered background area background component student image frame memory 2211 stores
The student image corresponding to the background component image in the covered background area of the teacher image is supplied to the learning unit 2214-4.

[0794] The weighted average section 2207-5 includes the covered background area foreground component teacher image frame memory 22.
For example, the foreground component image of the covered background area of the teacher image supplied from the image 05 is weighted by a quarter to generate an SD image as a student image, and the generated SD image is converted to the covered background area foreground. The component is supplied to the student image frame memory 2212.

The covered background area foreground component student image frame memory 2212 has a weighted average unit 2207
The student image stored as the SD image corresponding to the foreground component image in the covered background area of the teacher image supplied from -5 is stored. The covered background area foreground component student image frame memory 2212 stores
The student image corresponding to the foreground component image in the covered background area of the teacher image is supplied to the learning unit 2214-5.

[0796] The weighted average section 2207-6 is supplied from the foreground area teacher image frame memory 2206, for example,
An image in the foreground area of the teacher image, which is an HD image, is weighted by a quarter to generate an SD image, which is a student image.
The image is supplied to the foreground area student image frame memory 2213.

[0797] Foreground area student image frame memory 2213
Stores the student image, which is an SD image, corresponding to the image in the foreground area of the teacher image, supplied from the weighted averaging unit 2207-6. Foreground area student image frame memory 2213
Supplies the stored student image corresponding to the image of the foreground area of the teacher image to the learning unit 2214-6.

[0798] The learning unit 2214-1 includes an image of the background area of the teacher image supplied from the background area teacher image frame memory 2201, and the background area student image frame memory 2
A coefficient set corresponding to the background area is generated based on the student image corresponding to the image in the background area of the teacher image supplied from 208, and the generated coefficient set is stored in the coefficient set memory 22.
15

[0799] The learning section 2214-2 includes the uncovered background area background component teacher image frame memory 22.
02, the background component image of the uncovered background area of the teacher image supplied from 02, and the uncovered background area background component student image frame memory 220
9, based on the student image corresponding to the background component image in the uncovered background area of the teacher image,
A coefficient set corresponding to the background component image in the uncovered background area is generated, and the generated coefficient set is supplied to the coefficient set memory 2215.

[0800] The learning unit 2214-3 includes the uncovered background area foreground component teacher image frame memory 22.
03 for the foreground component image of the uncovered background area of the teacher image and the uncovered background area foreground component student image frame memory 221 supplied from the source image 03.
Based on the student image corresponding to the foreground component image in the uncovered background area of the teacher image supplied from 0,
A coefficient set corresponding to the foreground component image in the uncovered background area is generated, and the generated coefficient set is supplied to the coefficient set memory 2215.

The learning section 2214-4 has a covered background area background component teacher image frame memory 2204.
The background component image of the covered background area of the teacher image supplied from the source image and the student image corresponding to the background component image of the covered background area of the teacher image supplied from the student image frame memory 2211 Based on this, a coefficient set corresponding to the background component image in the covered background area is generated, and the generated coefficient set is stored in a coefficient set memory 2215.
To supply.

The learning unit 2214-5 includes a covered background area foreground component teacher image frame memory 2205.
And the student image corresponding to the foreground component image in the covered background area of the teacher image supplied from the covered background area foreground component student image frame memory 2212. Based on this, a coefficient set corresponding to the foreground component image in the covered background area is generated, and the generated coefficient set is stored in a coefficient set memory 2215.
To supply.

[0803] The learning section 2214-6 includes the foreground area image of the teacher image supplied from the foreground area teacher image frame memory 2206 and the foreground area student image frame memory 2
A coefficient set corresponding to the foreground area is generated based on the student image corresponding to the image of the foreground area of the teacher image supplied from 213, and the generated coefficient set is stored in the coefficient set memory 22.
15

The coefficient set memory 2215 stores the
The coefficient set corresponding to the background area supplied from the learning unit 221-2, the coefficient set corresponding to the background component image of the uncovered background area supplied from the learning unit 221-2, the uncovered coefficient supplied from the learning unit 2214-3 A coefficient set corresponding to the foreground component image in the background area, a coefficient set corresponding to the background component image in the covered background area supplied from the learning unit 2214-4,
The coefficient set corresponding to the foreground component image of the covered background area supplied from the learning unit 2214-5 and the coefficient set corresponding to the foreground area supplied from the learning unit 2214-6 are stored.

Note that the learning units 2214-1 to 2214-1-
6 has the same configuration as that of the learning unit 1006, and thus the description thereof is omitted.

FIG. 118 is a block diagram showing a configuration of a separated image processing unit 2002 for executing a classification adaptive process to generate a higher-resolution image in the spatial direction. For example, the separated image processing unit 2002 shown in FIG.
Based on the input image which is an SD image, a classification adaptive process is executed to generate an HD image.

[0807] The background area frame memory 2301 stores an image of a background area composed of pixels belonging to the background area supplied from the foreground / background separation unit 2001. The background area frame memory 2301 supplies the stored image of the background area to the mapping unit 2307-1.

The uncovered background area background component image frame memory 2302 stores the foreground / background
The background component image of the uncovered background area supplied from 01 is stored. The uncovered background area background component image frame memory 2302 supplies the stored background component image of the uncovered background area to the mapping unit 2307-2.

The uncovered background area foreground component image frame memory 2303 stores the foreground / background
The foreground component image of the uncovered background area supplied from 01 is stored. The uncovered background area foreground component image frame memory 2303 supplies the stored foreground component image of the uncovered background area to the mapping unit 2307-3.

The covered background area background component image frame memory 2304 includes a foreground / background separation unit 2001.
The background component image of the covered background area supplied from is stored. The covered background area background component image frame memory 2304 supplies the stored background component image of the covered background area to the mapping unit 2307-4.

[0811] The covered background area foreground component image frame memory 2305 includes a foreground / background separation unit 2001.
The foreground component image of the covered background area supplied from is stored. The covered background area foreground component image frame memory 2305 supplies the stored foreground component image of the covered background area to the mapping unit 2307-5.

[0812] The foreground area frame memory 2306 stores the image of the foreground area composed of pixels belonging to the foreground area supplied from the foreground / background separation unit 2001. The foreground area image frame memory 2306 supplies the stored image of the foreground area to the mapping unit 2307-6.

[0813] The mapping unit 2307-1 performs a class classification adaptive process on the basis of the coefficient set corresponding to the background area stored in the coefficient set memory 2308 to execute the mapping of the background area stored in the background area frame memory 2301. Generate a predicted image corresponding to the image. Mapping unit 23
07-1 supplies the generated predicted image to the synthesizing unit 2309.

[0814] The mapping unit 2307-2 performs a class classification adaptive process on the basis of the coefficient set corresponding to the background component image of the uncovered background area, which is stored in the coefficient set memory 2308, to perform the uncovered background area background processing. A predicted image corresponding to the background component image in the uncovered background area stored in the component image frame memory 2302 is generated. The mapping unit 2307-2 combines the generated predicted image with the combining unit 230.
9.

[0815] The mapping unit 2307-3 performs an uncovered background area foreground by class classification adaptive processing based on the coefficient set corresponding to the foreground component image of the uncovered background area stored in the coefficient set memory 2308. A predicted image corresponding to the foreground component image in the uncovered background area stored in the component image frame memory 2303 is generated. The mapping unit 2307-3 combines the generated predicted image with the combining unit 230.
9.

[0816] The mapping unit 2307-4 performs the classification background adaptation processing on the basis of the coefficient set corresponding to the background component image of the covered background area, which is stored in the coefficient set memory 2308, to perform the processing of the covered background area background component image. A predicted image corresponding to the background component image of the covered background area stored in the frame memory 2304 is generated. Mapping unit 230
7-4 supplies the generated predicted image to the synthesizing unit 2309.

[0817] The mapping unit 2307-5 uses the coefficient classification corresponding to the foreground component image in the covered background area, which is stored in the coefficient set memory 2308, to perform the classified classification adaptive processing and perform the covering background area foreground component image processing. A predicted image corresponding to the foreground component image of the covered background area stored in the frame memory 2305 is generated. Mapping unit 230
7-5 supplies the generated prediction image to the synthesizing unit 2309.

[0818] The mapping unit 2307-6 uses the coefficient set corresponding to the foreground area, stored in the coefficient set memory 2308, to perform the classification adaptive processing to determine the foreground area stored in the foreground area frame memory 2306. Generate a predicted image corresponding to the image. Mapping unit 23
07-6 supplies the generated predicted image to the synthesizing unit 2309.

[0819] The combining unit 2309 includes a mapping unit 2307.
-1, the predicted image corresponding to the background component image of the uncovered background area supplied from the mapping unit 2307-2, and the predicted image corresponding to the background component image supplied from the mapping unit 2307-2. A predicted image corresponding to the foreground component image of the covered background area, a predicted image corresponding to the background component image of the covered background area supplied from the mapping unit 2307-4, and a covered background area supplied from the mapping unit 2307-5 And a predicted image corresponding to the foreground area image supplied from the mapping unit 2307-6, and the synthesized predicted image is supplied to the frame memory 2310.

[0820] The frame memory 2310 is
9 is stored, and the stored image is output as an output image.

Note that the mapping units 2307-1 to 237-1
07-6 has the same configuration as that of the mapping unit 1103, and a description thereof will be omitted.

An example of the processing result of the image processing apparatus of the present invention having the separated image processing section 2002 shown in FIG. 118 will be described with reference to the images shown in FIGS. 119 to 124.

In the processing for generating the result shown in the example, the total number of classes in the class classification adaptive processing of the image processing apparatus of the present invention is almost the same as the number of classes in the conventional class classification adaptive processing. That is, the number of classes in the conventional class classification adaptive processing is set to 2048, and the number of classes in the class classification adaptive processing of the image processing apparatus of the present invention corresponding to the image of each area is set to 512.

The number of prediction taps in the conventional classification adaptive processing and the number of prediction taps in the classification adaptive processing of each area of the image processing apparatus according to the present invention are set to nine, and are the same.

The results of prediction in the covered background area will be described with reference to FIGS. 119 to 121.

FIG. 119 (A) is a diagram showing an example of an image in a mixed area of a teacher image. FIG. 119B is a diagram illustrating a change in a pixel value corresponding to a position in a spatial direction of an image in a mixed region of a teacher image.

[0827] Fig. 120A is a diagram illustrating an example of an image of a mixed area generated by the conventional classification adaptive processing corresponding to the teacher image illustrated in Fig. 119. FIG.
(B) of FIG. 119 is a diagram illustrating a change in the pixel value corresponding to the position in the spatial direction of the image in the mixed region, which is generated by the conventional classification adaptive processing and corresponds to the teacher image illustrated in FIG. 119.

FIG. 121A shows the separated image processing unit 2 shown in FIG. 118 corresponding to the teacher image shown in FIG.
FIG. 9 is a diagram illustrating an example of an image of a mixed area generated by 002. FIG. 120B illustrates a separated image processing unit 200 corresponding to the teacher image illustrated in FIG. 119 and having a configuration illustrated in FIG. 118.
FIG. 6 is a diagram illustrating a change in pixel value corresponding to a position in a spatial direction of an image in a mixed area generated by Step No. 2;

[0827] The pixel value of the image in the mixed area generated by the conventional classification adaptive processing changes stepwise compared to the teacher image, and also changes stepwise in the generated actual image. Can be visually confirmed.

On the other hand, the pixel value of the image in the mixed area generated by the separated image processing unit 2002 having the configuration shown in FIG. 118 changes more smoothly than in the past and changes more closely to the teacher image. Is shown. Separated image processing unit 2
Even if the image generated by 002 is visually checked,
It can be confirmed that the image is smoother than before.

The separated image processing unit 20 shown in FIG.
The image in the mixed area generated by O.sub.02 changes more smoothly even when the input image is divided into the foreground area, the mixed area, or the background area and compared with the generated image.

Referring to FIG. 122 to FIG. 124, the result of prediction in the foreground area where the pixel value changes almost linearly with respect to the position of the pixel will be described.

FIG. 122 (A) is a diagram showing an example of an image in the foreground area of the teacher image in which the pixel values change almost linearly. FIG. 122 (B) shows an image of an image in the foreground area of the teacher image in which the pixel value changes almost linearly.
It is a figure showing change of a pixel value corresponding to a position of a spatial direction.

FIG. 123 (A) is a diagram showing an example of an image of the foreground area corresponding to the image of FIG. 122 generated by the conventional classification adaptive processing. FIG. 123 (B)
FIG. 122 generated by the conventional classification adaptive processing.
FIG. 10 is a diagram illustrating a change in pixel value corresponding to a position in a spatial direction of an image in a foreground area corresponding to the image of FIG.

FIG. 124 (A) is a diagram showing an example of an image of the foreground area corresponding to the image of FIG. 122 generated by the separated image processing section 2002 shown in FIG. 118.
FIG. 124B shows a change in pixel value corresponding to a position in the spatial direction of the image in the foreground area corresponding to the image in FIG. 122 and generated by the separated image processing unit 2002 having the configuration shown in FIG. FIG.

The pixel value of the image in the foreground area generated by the conventional classification adaptive processing changes stepwise as compared with the teacher image in the same manner as in the mixed area. Can be visually confirmed.

On the other hand, the pixel values of the image in the foreground area generated by the separated image processing unit 2002 having the configuration shown in FIG. 118 change more smoothly than in the past, and are very close to the teacher image. Value. Separated image processing unit 2
In the visual confirmation of the image generated in 002, no difference from the teacher image was recognized.

FIG. 125 is a flowchart for explaining image processing of the image processing apparatus shown in FIG. 113.

[0839] In step S2001, the area specifying unit 103 determines the foreground area, the background area, the covered background area, and the input image based on the motion vector and its position information supplied from the motion detection unit 102 and the input image. Identify the uncovered background area.

[0840] In step S2002, the mixture ratio calculation unit 104 calculates the mixture ratio α based on the region information and the input image supplied from the region identification unit 103.

[0841] In step S2003, the foreground / background separation unit 2001 converts the input image into the foreground region image based on the region information supplied from the region identification unit 103 and the mixture ratio α supplied from the mixture ratio calculation unit 104. , The background region image, the foreground component image of the covered background region, the background component image of the covered background region, the foreground component image of the uncovered background region, and the background component image of the uncovered background region. The details of the image separation processing of the foreground / background separation unit 2001 will be described later.

In step S2004, the separated image processing unit 2002 separates the foreground area image, the background area image, the foreground component image of the covered background area, the background component image of the covered background area, and the uncovered background. Image processing is performed for each of the foreground component image of the region and the background component image of the uncovered background region, and the process ends. Details of the image processing performed by the separated image processing unit 2002 will be described later.

[0843] As described above, the image processing apparatus according to the present invention converts an input image into a foreground area image, a background area image, a foreground component image in a covered background area, a background component image in a covered background area, and an uncovered image. The foreground component image of the background area and the background component image of the uncovered background area are separated and separated, and the separated foreground area image, background area image, foreground component image of the covered background area, and the covered background area image Image processing is performed for each of the background component image, the foreground component image in the uncovered background area, and the background component image in the uncovered background area.

Next, the process of separating the foreground and background by the foreground / background separator 2001 will be described with reference to the flowchart shown in FIG. In step S2101,
The frame memory 2121 of the separation unit 2101 acquires the input image, and sets the frame #n to be separated from the foreground and the background.
To the previous frame # n-1 and the subsequent frame # n + 1
And memorize it.

[0846] In step S2102, the separation unit 21
01 is the separation processing block 2122
To obtain the area information supplied from. Step S210
3, the separation processing block 212 of the separation unit 2101
2 obtains the mixture ratio α supplied from the mixture ratio calculation unit 104.

[0846] In step S2104, the uncovered area processing unit 2131 calculates the pixel value of the pixel belonging to the uncovered background area supplied from the frame memory 2121 based on the area information and the mixture ratio α.
The background component is extracted and output as a background component image of the uncovered background area.

[0847] In step S2105, the uncovered area processing unit 2131 calculates the pixel value of the pixel belonging to the uncovered background area supplied from the frame memory 2121 based on the area information and the mixture ratio α.
A foreground component is extracted and output as a foreground component image of the uncovered background area.

[0848] In step S2106, the covered area processing unit 2132 calculates the area based on the area information and the mixture ratio α.
A background component is extracted from the pixel values of the pixels belonging to the covered background area supplied from the frame memory 2121 and output as a background component image of the covered background area.

[0849] In step S2107, the covered area processing unit 2132 determines the area based on the area information and the mixture ratio α.
The foreground component is extracted from the pixel values of the pixels belonging to the covered background area supplied from the frame memory 2121 and output as the foreground component image of the covered background area, and the process ends.

[0850] As described above, the foreground / background separation unit 2001 calculates
Separating a foreground component and a background component from an input image based on the region information and the mixture ratio α, and outputting a foreground component image including only foreground components and a background component image including only background components. Can be.

With reference to the flowchart in FIG. 127, a description will be given of a learning process of generating a coefficient set used for predicting a pixel value by the classification adaptive processing by the separated image processing unit 2002 having the configuration shown in FIG. 117.

[0852] In step S2201, the weighted averaging units 2207-1 to 2207-6 generate a background area image, a foreground area image, an uncovered background area background component image, an uncovered background area foreground component image, A student image of a background component image of the background area and a foreground component image of the covered background area are generated. That is, the weighted average unit 220
7-1 is a student image corresponding to the image of the background area of the teacher image, for example, obtained by averaging the image of the background area of the teacher image stored in the background area teacher image frame memory 2201 by a quarter weight. Generate

The weighted average section 2207-2 weights the background component image of the uncovered background area of the teacher image stored in the uncovered background area background component teacher image frame memory 2202, for example, by a quarter weight. On average, a student image corresponding to the background component image in the uncovered background area of the teacher image is generated.

The weighted averaging unit 2207-3 weights the foreground component image of the uncovered background area of the teacher image stored in the uncovered background area foreground component teacher image frame memory 2203, for example, by a quarter weight. On average, a student image corresponding to the foreground component image in the uncovered background area of the teacher image is generated.

The weighted averaging unit 2207-4 stores the covered background area background component teacher image frame memory 22.
For example, the background component image of the covered background area of the teacher image stored in 04 is weighted by a quarter, for example, to generate a student image corresponding to the background component image of the covered background area of the teacher image. .

[0856] The weighted average section 2207-5 includes the covered background area foreground component teacher image frame memory 22.
For example, the foreground component image of the covered background area of the teacher image stored in the teacher image is weighted by a quarter to generate a student image corresponding to the foreground component image of the covered background area of the teacher image. .

[0857] The weighted average unit 2207-6 performs, for example, a quarter weighted average of the image of the foreground area of the teacher image stored in the foreground area teacher image frame memory 2206 to obtain the weight of the foreground area of the teacher image. Generate a student image corresponding to the image.

[0858] In step S2202, the learning unit 22
14-1 is a background area teacher image frame memory 2201
And a coefficient set corresponding to the background area based on the image of the background area of the teacher image stored in the image area and the student image corresponding to the image of the background area of the teacher image stored in the background area student image frame memory 2208 Generate

[0859] In step S2203, the learning unit 22
14-2 is stored in the uncovered background area background component teacher image frame memory 2202;
The background component image of the uncovered background area of the teacher image and the student image corresponding to the background component image of the uncovered background area of the teacher image stored in the uncovered background area student image frame memory 2209 are displayed. Based on this, a coefficient set corresponding to the background component image in the uncovered background area is generated.

In step S2204, the learning unit 22
14-3 is stored in the uncovered background area foreground component teacher image frame memory 2203;
The foreground component image of the uncovered background area of the teacher image and the student image corresponding to the foreground component image of the uncovered background area of the teacher image stored in the uncovered background area foreground component student image frame memory 2210 are displayed. Based on this, a coefficient set corresponding to the foreground component image in the uncovered background area is generated.

In step S2205, the learning section 22
14-4 is a background component image of the covered background area of the teacher image stored in the covered background area background component teacher image frame memory 2204;
And a coefficient corresponding to the background component image of the covered background area, based on the student image corresponding to the background component image of the covered background area of the teacher image, stored in the covered background area background component student image frame memory 2211. Generate a set.

In step S2206, the learning unit 22
14-5 is a foreground component image of the covered background area of the teacher image, which is stored in the covered background area foreground component teacher image frame memory 2205;
And a coefficient corresponding to the foreground component image in the covered background area, based on the student image corresponding to the foreground component image in the covered background area of the teacher image stored in the covered background area foreground component student image frame memory 2212. Generate a set.

In step S2207, the learning unit 22
14-6 is a foreground area teacher image frame memory 2206
The coefficient set corresponding to the foreground area is based on the image of the foreground area of the teacher image and the student image corresponding to the image of the foreground area of the teacher image stored in the foreground area student image frame memory 2213. Generate

[0864] In step S2208, the learning unit 22
14-1 to 2212-4 are a coefficient set corresponding to the background area, a coefficient set corresponding to the background component image in the uncovered background area, a coefficient set corresponding to the foreground component image in the uncovered background area, and a covered set, respectively. The coefficient set corresponding to the background component image in the background area, the coefficient set corresponding to the foreground component image in the covered background area, or the coefficient set corresponding to the foreground area is output to the coefficient set memory 2215. The coefficient set memory 2215 stores a background region, a foreground region, a background component image of an uncovered background region, a foreground component image of an uncovered background region, a background component image of a covered background region, or a foreground component image of a covered background region. Are stored, and the process ends.

[0859] As described above, the separated image processing unit 2002 having the configuration shown in FIG. 117 includes the coefficient set corresponding to the background area image, the coefficient set corresponding to the background component image in the uncovered background area, and the uncovered background area. Generate a coefficient set corresponding to the foreground component image, a coefficient set corresponding to the background component image in the covered background area, a coefficient set corresponding to the foreground component image in the covered background area, and a coefficient set corresponding to the image in the foreground area can do.

[0866] Steps S2202 to S220
Since the details of the process of FIG. 7 are the same as those described with reference to the flowchart of FIG. 103, the description thereof is omitted.

[0867] Steps S2202 to S220
It goes without saying that the processing of 2207 may be executed serially or in parallel.

Next, with reference to the flowchart of FIG. 128, the creation of an image by the separated image processing unit 2002 shown in FIG. 118 will be described.

[0869] In step S2301, the mapping unit 2307-1 performs a class classification adaptation process on the basis of the coefficient set corresponding to the background area stored in the coefficient set memory 2308 to perform the background area frame memory 230
An image corresponding to the image of the background area stored in No. 1 is predicted.

[0870] In step S2302, mapping section 2307-2 performs unclassified adaptive processing based on the coefficient set corresponding to the background component image in the uncovered background area stored in coefficient set memory 2308. An image corresponding to the background component image of the uncovered background area, which is stored in the background area background component image frame memory 2302, is predicted.

[0871] In step S2303, the mapping unit 2307-3 performs unclassified adaptive processing by class classification adaptation processing based on the coefficient set corresponding to the foreground component image in the uncovered background area stored in the coefficient set memory 2308. An image corresponding to the foreground component image in the uncovered background area, which is stored in the background area foreground component image frame memory 2303, is predicted.

[0887] In step S2304, the mapping unit 2307-4 performs the classified background adaptive processing based on the coefficient set corresponding to the background component image in the covered background area, stored in the coefficient set memory 2308, to execute the covered background processing. An image corresponding to the background component image of the covered background area stored in the area background component image frame memory 2304 is predicted.

[0873] In step S2305, the mapping unit 2307-5 performs the classified background adaptive processing based on the coefficient set corresponding to the foreground component image in the covered background area, which is stored in the coefficient set memory 2308. An image corresponding to the foreground component image of the covered background area stored in the area foreground component image frame memory 2305 is predicted.

[0887] In step S2306, the mapping unit 2307-6 performs a class classification adaptive process on the basis of the coefficient set corresponding to the foreground area stored in the coefficient set memory 2308 to execute the foreground area frame memory 230
The image corresponding to the image of the foreground area stored in No. 6 is predicted.

[0875] In step S2307, the synthesizing unit 23
09 is a predicted image corresponding to the image of the background area, a predicted image corresponding to the background component image of the uncovered background area, a predicted image corresponding to the foreground component image of the uncovered background area, and a background component of the covered background area. A predicted image corresponding to the image, a predicted image corresponding to the foreground component image in the covered background area, and a predicted image corresponding to the foreground area are synthesized. Synthesizer 2
309 supplies the synthesized image to the frame memory 2310. The frame memory 2310 includes a combining unit 2309
The image supplied from is stored.

[0887] In step S2308, the frame memory 2310 outputs the stored combined image, and the process ends.

[0877] As described above, the image processing apparatus having the separated image processing unit 2002 shown in FIG. 118 can be used to separate the background image, the uncovered background component background image, and the uncovered background region. A prediction image can be generated for each of the foreground component image, the background component image of the covered background area, the foreground component image of the covered background area, and the image of the foreground area.

[0887] Steps S2301 to S230
The details of the process 6 are the same as the processes described with reference to the flowchart in FIG. 109, and a description thereof will be omitted.

[0879] Steps S2301 to S230
It goes without saying that the processing of 2306 may be executed serially or in parallel.

FIG. 129 is a block diagram showing another configuration of the function of the image processing apparatus that separates the input image and processes each of the separated images. While the image processing apparatus illustrated in FIG. 113 sequentially performs the area specification and the calculation of the mixture ratio α, the image processing apparatus illustrated in FIG. 129 performs the area specification and the calculation of the mixture ratio α in parallel.

[0891] The same reference numerals are given to the same portions as the functions shown in the block diagram of FIG. 113, and description thereof will be omitted.

[0882] The input image is input to the object extraction unit 101,
The information is supplied to the area specifying unit 103, the mixture ratio calculating unit 1501, and the foreground / background separating unit 2501.

[0883] Based on the input image, the mixture ratio calculation unit 1501 calculates the estimated mixture ratio when the pixel belongs to the covered background area and the estimated mixture ratio when the pixel belongs to the uncovered background area. The ratio is calculated for each of the pixels included in the input image, and the estimated mixture ratio when the calculated pixel is assumed to belong to the covered background area,
The estimated mixture ratio when the pixel belongs to the uncovered background area is supplied to the foreground / background separation unit 2501.

[0884] The foreground / background separation unit 2501 calculates the estimated mixture ratio when the pixel belongs to the covered background area, supplied from the mixture ratio calculation unit 1501.
Based on the estimated mixture ratio when it is assumed that the pixel belongs to the uncovered background area, and the area information supplied from the area specifying unit 103, the input image
Background region image, uncovered background region background component image, uncovered background region foreground component image, covered background region background component image, covered background region foreground component image,
And an image in the foreground area, and supplies the separated image to the separated image processing unit 2002.

FIG. 130 is a block diagram showing an example of the configuration of the foreground / background separation unit 2501.

[0886] The same parts as those in the foreground / background separation unit 2001 shown in FIG. 115 are denoted by the same reference numerals, and description thereof will be omitted.

[0887] Based on the area information supplied from the area specifying unit 103, the selecting unit 2521 determines the estimated mixture ratio supplied from the mixture ratio calculation unit 1501 when the pixel belongs to the covered background area, and One of the estimated mixture ratios when the pixel is assumed to belong to the uncovered background area is selected, and the selected estimated mixture ratio is supplied to the separation unit 2101 as the mixture ratio α.

[0888] The separation unit 2101 extracts the foreground component and the background component from the pixel values of the pixels belonging to the mixed region based on the mixture ratio α and the region information supplied from the selection unit 2521, and extracts the uncovered background region , A foreground component image in the uncovered background area, a background component image in the covered background area, and a foreground component image in the covered background area.

[0889] Separating section 2101 can have the same configuration as that shown in FIG.

[0890] As described above, the image processing apparatus shown in Fig. 129 has a structure in which a background area image, an uncovered background area background component image, an uncovered background area foreground component image, and a covered background area background component. Processing can be executed for each of the image, the foreground component image in the covered background area, and the image in the foreground area in accordance with the respective properties.

FIG. 131 is a block diagram showing still another configuration of the functions of the image processing apparatus.

[0891] The input image supplied to the image processing apparatus is supplied to the object extracting unit 101, the region specifying unit 103, and the region processing unit 3001.

[0893] The object extracting unit 101 roughly extracts the image object corresponding to the foreground object included in the input image, and supplies the extracted image object to the motion detecting unit 102. The object extraction unit 101
For example, by detecting the outline of the image object corresponding to the foreground object included in the input image, the image object corresponding to the foreground object is roughly extracted.

[0894] The object extraction unit 101 roughly extracts an image object corresponding to a background object included in the input image, and supplies the extracted image object to the motion detection unit 102. The object extraction unit 101
For example, the image object corresponding to the background object is roughly extracted from the difference between the input image and the extracted image object corresponding to the foreground object.

The motion detector 102 calculates the motion vector of the image object corresponding to the coarsely extracted foreground object by using, for example, a block matching method, a gradient method, a phase correlation method, or a pel recursive method. , And supplies the calculated motion vector and the position information of the motion vector to the area specifying unit 103.

[0896] The area specifying unit 103 specifies each of the pixels of the input image as one of a foreground area, a background area, and a mixed area, and for each pixel, selects one of the foreground area, the background area, and the mixed area. Is supplied to the region processing unit 3001 indicating whether the region belongs to.

[0897] The area processing unit 3001 includes the area specifying unit 103
Foreground area, background area,
Alternatively, the input image is divided for each mixed area, and image processing is executed for each divided input image. For example, the area processing unit 3
001 is for each foreground area, background area, or mixed area.
The input image is divided, and for each of the divided input images, coefficients used in the classification adaptive processing for generating a higher-resolution image are generated.

[0898] For example, the area processing unit 3001 divides an input image for each foreground area, background area, or mixed area, and applies a classification adaptive process to each of the divided input images to obtain a higher resolution image. Create

FIG. 132 is a view for explaining the processing of the area processing unit 3001 shown in FIG. 131. Area processing unit 3
001 calculates a coefficient set corresponding to the background area, a coefficient set corresponding to the uncovered background area, a coefficient set corresponding to the foreground area, and a coefficient set corresponding to the covered background area individually.

[0900] The coefficient set corresponding to the background area is used for predicting the pixel value of the background area in the classification adaptive processing for predicting the pixel value. The coefficient set corresponding to the uncovered background area is used for predicting the pixel value of the uncovered background area in the class classification adaptive processing for predicting the pixel value.

The coefficient set corresponding to the covered background area is used for predicting the pixel value of the covered background area in the classification adaptive processing for predicting the pixel value. The coefficient set corresponding to the foreground area is used for predicting the pixel value of the foreground area in the class classification adaptive processing for predicting the pixel value.

[0992] The predicted image corresponding to the background image, the predicted image corresponding to the uncovered background area, the predicted image corresponding to the covered background area, and the predicted image corresponding to the foreground image are combined to form one predicted image Is done.

FIG. 133 is a block diagram showing a configuration of an area processing unit 3001 for generating a coefficient set used in the classification adaptive processing for generating a higher resolution image in the spatial direction. Teacher image frame memory 3101
Stores, for example, an input image, which is an HD image, in frame units. The teacher image frame memory 3101 supplies the stored input image to the area dividing unit 3102.

[0904] The area dividing section 3102
Background area, foreground area,
Divide the teacher image into a covered background area or an uncovered background area.

[0905] The area dividing section 3102 supplies an image composed of pixels belonging to the background area of the teacher image, which is the divided teacher image, to the background area teacher image frame memory 3103 and belongs to the uncovered background area of the teacher image. An image composed of pixels is supplied to the uncovered background area teacher image frame memory 3104, and an image composed of pixels belonging to the covered background area of the teacher image is supplied to the covered background area teacher image frame memory 3105, and the foreground of the teacher image is supplied. An image composed of pixels belonging to the area is supplied to the foreground area teacher image frame memory 3106.

[0906] Background area teacher image frame memory 3103
Stores an image composed of pixels belonging to the background area of the teacher image supplied from the area dividing unit 3102. The background area teacher image frame memory 3103 supplies an image composed of pixels belonging to the background area of the stored teacher image to the weighted average unit 3107-1 and the learning unit 3112-1.

[0907] The uncovered background area teacher image frame memory 3104 stores an image which is supplied from the area dividing unit 3102 and is composed of pixels belonging to the uncovered background area of the teacher image. Uncovered background area teacher image frame memory 310
4 is a weighted averaging unit 31 for calculating an image composed of pixels belonging to the uncovered background area of the stored teacher image.
07-2 and the learning unit 3112-2.

[0908] The covered background area teacher image frame memory 3105 stores an image which is supplied from the area dividing unit 3102 and is composed of pixels belonging to the covered background area of the teacher image. The covered background area teacher image frame memory 3105 supplies an image composed of pixels belonging to the covered background area of the stored teacher image to the weighted averaging unit 3107-3 and the learning unit 3112-3.

[0909] Foreground area teacher image frame memory 3106
Stores an image composed of pixels belonging to the foreground area of the teacher image supplied from the area dividing unit 3102. The foreground area teacher image frame memory 3106 supplies an image composed of pixels belonging to the foreground area of the stored teacher image to the weighted averaging unit 3107-4 and the learning unit 3112-4.

The weighted averaging section 3107-1 is supplied from the background area teacher image frame memory 3103.
An image composed of pixels belonging to the background area of the teacher image, which is an HD image, is weighted by a quarter to generate an SD image, which is a student image, and the generated SD image is supplied to the background area student image frame memory 3108. .

[0911] Background area student image frame memory 3108
Stores the student image corresponding to the image composed of the pixels belonging to the background area of the teacher image, supplied from the weighted averaging unit 3107-1. Background area student image frame memory 31
08 is a learning unit 3112-storing a student image corresponding to the stored image composed of pixels belonging to the background area of the teacher image.
Feed to 1.

The weighted averaging unit 3107-2 has an uncovered background area teacher image frame memory 3104.
An image composed of pixels belonging to the uncovered background area of the teacher image, which is an HD image, which is supplied from, for example, is weighted by a quarter to generate an SD image, which is a student image, and the generated SD image Is supplied to the uncovered background area student image frame memory 3109.

The uncovered background area student image frame memory 3109 has a weighted average unit 3107-2.
And stores the student image, which is an SD image, corresponding to the image composed of the pixels belonging to the uncovered background area of the teacher image, supplied from. The uncovered background area student image frame memory 3109 stores the student image corresponding to the stored image composed of pixels belonging to the uncovered background area of the teacher image in the learning unit 3.
Supply to 112-2.

The weighted averaging unit 3107-3 performs, for example, a quarter weighted averaging on the image, which is supplied from the covered background area teacher image frame memory 3105, and is composed of pixels belonging to the covered background area of the teacher image. Then, an SD image as a student image is generated, and the generated SD image is supplied to the covered background area student image frame memory 3110.

[0915] The covered background area student image frame memory 3110 stores a student image, which is an SD image, corresponding to an image composed of pixels belonging to the covered background area of the teacher image supplied from the weighted averaging unit 3107-3. I do. The covered background area student image frame memory 3110 supplies the stored student image corresponding to the image composed of the pixels belonging to the covered background area of the teacher image to the learning unit 3112-3.

The weighted averaging unit 3107-4 is supplied from the foreground area teacher image frame memory 3106, for example,
An image consisting of pixels belonging to the foreground area of the teacher image, which is an HD image, is weighted by a quarter to generate an SD image, which is a student image, and the generated SD image is supplied to the foreground area student image frame memory 3111. .

[0917] Foreground area student image frame memory 3111
Stores a student image, which is an SD image, corresponding to an image composed of pixels belonging to the foreground area of the teacher image, supplied from the weighted averaging unit 3107-4. The foreground area student image frame memory 3111 supplies the stored student image corresponding to the image composed of the pixels belonging to the foreground area of the teacher image to the learning unit 3112-4.

[0918] The learning unit 3112-1 includes an image composed of pixels belonging to the background area of the teacher image supplied from the background area teacher image frame memory 3103, and the background of the teacher image supplied from the background area student image frame memory 3108. A coefficient set corresponding to the background area is generated based on the student image corresponding to the image including the pixels belonging to the area, and the generated coefficient set is supplied to the coefficient set memory 3113.

[0919] The learning unit 3112-2 includes an image composed of pixels belonging to the uncovered background area of the teacher image supplied from the uncovered background area teacher image frame memory 3104, and an uncovered background area student image frame memory 3109. Generates a coefficient set corresponding to the uncovered background area based on the student image corresponding to the image composed of the pixels belonging to the uncovered background area of the teacher image, supplied from the teacher image, and stores the generated coefficient set in the coefficient set memory. 3113.

The learning section 3112-3 is supplied from the covered background area teacher image frame memory 3105, an image composed of pixels belonging to the covered background area of the teacher image, and the covered background area student image frame memory 3110. Was
A coefficient set corresponding to the covered background area is generated based on the student image corresponding to the image including the pixels belonging to the covered background area of the teacher image, and the generated coefficient set is supplied to the coefficient set memory 3113.

[0921] The learning unit 3112-4 includes an image composed of pixels belonging to the foreground area of the teacher image supplied from the foreground area teacher image frame memory 3106, and the foreground of the teacher image supplied from the foreground area student image frame memory 3110. A coefficient set corresponding to the foreground area is generated based on the student image corresponding to the image including the pixels belonging to the area, and the generated coefficient set is supplied to the coefficient set memory 3113.

[0922] The coefficient set memory 3113 includes the learning unit 31.
A coefficient set corresponding to the background area supplied from 12-1; a coefficient set corresponding to the uncovered background area supplied from the learning unit 3112-2; a learning unit 31
The coefficient set corresponding to the covered background area supplied from 12-3 and the coefficient set corresponding to the foreground area supplied from the learning unit 3112-4 are stored.

[0923] Learning unit 3112-1 to learning unit 3112-
4 has the same configuration as that of the learning unit 1006, and thus the description thereof is omitted.

[0924] Fig. 134 is a block diagram illustrating the configuration of an area processing unit 3001 that executes a class classification adaptive process and generates a higher-resolution image in the spatial direction. The frame memory 3201 stores, for example, an input image which is an SD image,
Stored in frame units. The frame memory 3201 is
The stored input image is supplied to the area dividing unit 3202.

[0925] The region dividing unit 3202
Background area, foreground area,
The input image is divided for each of the covered background area and the uncovered background area. That is, the region dividing unit 3202 supplies the divided input image, which is an image composed of pixels belonging to the background region, to the background region frame memory 3203, and converts the image composed of pixels belonging to the uncovered background region into an uncovered background. To the area frame memory 3204,
An image composed of pixels belonging to the covered background area is stored in the covered background area frame memory 32.
05, and supplies an image composed of pixels belonging to the foreground area to the foreground area frame memory 3206.

[0926] The background area frame memory 3203 stores an image composed of pixels belonging to the background area, supplied from the area dividing unit 3202. Background area frame memory 32
03 supplies the stored image composed of the pixels belonging to the background area to the mapping unit 3207-1.

[0927] The uncovered background area frame memory 3204 stores an image composed of pixels belonging to the uncovered background area supplied from the area dividing unit 3202. The uncovered background area frame memory 3204 supplies the stored image composed of pixels belonging to the uncovered background area to the mapping unit 3207-2.

[0928] The covered background area frame memory 3205 is supplied from the area dividing unit 3202,
An image composed of pixels belonging to the covered background area is stored. The covered background area frame memory 3205 maps the stored image composed of pixels belonging to the covered background area to the mapping unit 32.
07-3.

[0929] The foreground area frame memory 3206 stores an image composed of pixels belonging to the foreground area supplied from the area dividing unit 3202. The foreground area input image frame memory 3106 supplies the stored image composed of pixels belonging to the foreground area to the mapping unit 3207-4.

[0930] The mapping unit 3207-1 executes a class classification adaptive process on the background area stored in the background area frame memory 3203 based on the coefficient set corresponding to the background area stored in the coefficient set memory 3208. A predicted image corresponding to an image composed of pixels belonging to the image is generated. The mapping unit 3207-1 supplies the generated predicted image to the combining unit 3209.

[0931] The mapping unit 3207-2 stores the data in the uncovered background area frame memory 3204 by the class classification adaptive processing based on the coefficient set corresponding to the uncovered background area stored in the coefficient set memory 3208. A predicted image corresponding to an image composed of pixels belonging to the uncovered background area is generated. Mapping unit 3207-2
Supplies the generated prediction image to the synthesis unit 3209.

[0932] The mapping unit 3207-3 stores the data in the covered background area frame memory 3205 by the class classification adaptive processing based on the coefficient set corresponding to the covered background area stored in the coefficient set memory 3208. A predicted image corresponding to an image composed of pixels belonging to the covered background area is generated. The mapping unit 3207-3 supplies the generated predicted image to the combining unit 3209.

[0933] The mapping unit 3207-4 performs a class classification adaptation process on the basis of the coefficient set corresponding to the foreground area stored in the coefficient set memory 3208 to the foreground area stored in the foreground area frame memory 3206. A predicted image corresponding to an image composed of pixels belonging to the image is generated. The mapping unit 3207-4 supplies the generated predicted image to the combining unit 3209.

[0934] The combining unit 3209 includes a mapping unit 3207.
-1, a predicted image corresponding to an image composed of pixels belonging to the background area supplied from the mapping unit 3207-2, a predicted image corresponding to an image composed of pixels belonging to the uncovered background region supplied from the mapping unit 3207-2, and the mapping unit 32
A predicted image corresponding to an image composed of pixels belonging to the covered background area supplied from 07-3 and a predicted image corresponding to an image composed of pixels belonging to the foreground area supplied from the mapping unit 3207-4, The synthesized predicted image is supplied to the frame memory 3210.

[0935] The frame memory 3210 includes a synthesizing unit 320.
9 is stored, and the stored image is output as an output image.

[0936] Mapping units 3207-1 to 3207-
4 has the same configuration as the mapping unit 1103, and a description thereof will be omitted.

Referring to the images shown in FIGS. 135 to 140, an example of the processing result of the image processing apparatus of the present invention having the area processing unit 3001 shown in FIG. 134 will be described.

In the processing for generating the result shown in the example, the total number of classes in the class classification adaptive processing of the image processing apparatus of the present invention is the same as the number of classes in the conventional class classification adaptive processing. That is, the number of classes in the conventional class classification adaptive processing is set to 2048, and the number of classes in the class classification adaptive processing of each area of the image processing apparatus of the present invention is set to 3112.

Also, the number of prediction taps in the conventional classification adaptive processing and the number of prediction taps in the classification adaptive processing of each area of the image processing apparatus of the present invention are set to nine, and are the same.

Referring to FIGS. 135 to 137, the result of prediction in the covered background area will be described.

[0941] Fig. 135 (A) is a diagram showing an example of an image in a mixed area of a teacher image. FIG. 135 (B) is a diagram illustrating a change in a pixel value corresponding to a position in the spatial direction of an image in the mixed region of the teacher image.

FIG. 136 (A) is a diagram showing an example of a mixed area image corresponding to the teacher image shown in FIG. 135 and generated by the conventional classification adaptive processing. FIG.
(B) of FIG. 135 is a diagram illustrating a change in a pixel value corresponding to a position in a spatial direction of an image in a mixed region, which is generated by the conventional classification adaptive processing and corresponds to the teacher image illustrated in FIG. 135.

FIG. 137 (A) shows the area processing unit 300 corresponding to the teacher image shown in FIG. 135 and having the configuration shown in FIG.
FIG. 3 is a diagram illustrating an example of an image of a mixed area generated by No. 1; FIG. 136 (B) shows a change in pixel value corresponding to the position in the spatial direction of the image in the mixed area, which is generated by the area processing unit 3001 shown in FIG. 134 and corresponds to the teacher image shown in FIG. 135. FIG.

[0944] The pixel value of the image in the mixed area generated by the conventional classification adaptive processing changes stepwise compared to the teacher image, and also changes stepwise in the generated actual image. Can be visually confirmed.

On the other hand, the pixel value of the image in the mixed area generated by the area processing unit 3001 shown in FIG. 134 changes more smoothly than in the conventional case.
It shows a change closer to the teacher image. Even if the image generated by the area processing unit 3001 is visually checked, it can be confirmed that the image is smoother than in the related art.

Referring to FIGS. 138 to 140, the result of prediction in the foreground area where the pixel value changes almost linearly with the position of the pixel will be described.

FIG. 138 (A) is a diagram showing an example of an image in the foreground area of the teacher image in which the pixel value changes almost linearly. FIG. 138 (B) shows an image in the foreground area of the teacher image in which the pixel value changes almost linearly.
It is a figure showing change of a pixel value corresponding to a position of a spatial direction.

FIG. 139 (A) is a diagram showing an example of an image of the foreground area corresponding to the image of FIG. 138 generated by the conventional classification adaptive processing. FIG. 139 (B)
FIG. 138 generated by the conventional classification adaptive processing.
FIG. 10 is a diagram illustrating a change in pixel value corresponding to a position in a spatial direction of an image in a foreground area corresponding to the image of FIG.

FIG. 140A is a diagram showing an example of an image of the foreground area corresponding to the image of FIG. 138 generated by the area processing unit 3001 shown in FIG. FIG.
36 (B) is an area processing unit 300 shown in FIG.
FIG. 139 is a diagram illustrating a change in pixel value corresponding to a position in a spatial direction of an image in a foreground area corresponding to the image in FIG. 138 generated by No. 1.

The pixel values of the image in the foreground area generated by the conventional classification adaptive processing change stepwise as compared with the teacher image, similarly to the mixed area. Can be visually confirmed.

On the other hand, the pixel value of the image in the foreground area generated by the area processing unit 3001 shown in FIG. 134 changes more smoothly than in the past.
The value is very close to the teacher image. In visual confirmation of the image generated by the area processing unit 3001, no difference from the teacher image was recognized.

[0952] For a predetermined image, the SN in each region of the image generated by the conventional classification adaptive processing is obtained.
The ratio and the SN ratio in each region of the image generated by the class classification adaptive processing of the image processing device according to the present invention were obtained and compared.

[0953] The SNR in the covered background area of the image generated by the conventional classification adaptive processing is 32.1716 dB, the SNR in the uncovered background area is 31.8744 dB, and the SNR in the foreground area is , 31.8835 dB, and the SN ratio in the background area was 31.9985 dB.

On the other hand, the SN ratio in the covered background area of the image generated by the image processing apparatus according to the present invention was 32.1799 dB, the SN ratio in the uncovered background area was 31.8922 dB,
The SN ratio in the foreground region was 32.0925 dB, and the SN ratio in the background region was 32.0177 dB.

As described above, the S / N ratio of the image generated by the image processing apparatus according to the present invention is equal to that of the image generated by the conventional classification adaptive processing in any region.
High compared to SN ratio.

FIG. 141 is a flowchart for explaining image processing of the image processing apparatus shown in FIG. 131.

[0957] In step S3001, the area specifying unit 103 determines the foreground area, the background area, the covered background area, and the uncovered background area of the input image based on the motion vector and the position information supplied from the motion detection unit 102. Identify the area.

[0958] In step S3002, the area processing unit 3001 divides the input image into the specified foreground area, background area, covered background area, and uncovered background area, and divides the input image into the foreground area and the background. The image processing is executed for each of the area, the covered background area, and the uncovered background area, and the processing ends.

[0959] As described above, the image processing apparatus according to the present invention divides an input image into a foreground area, a background area, a covered background area, and an uncovered background area, and divides the input image into the foreground area and the background area. , Image processing is performed for each of the covered background area and the uncovered background area.

With reference to the flowchart in FIG. 142, a description will be given of a learning process for generating a coefficient set used for predicting a pixel value by the class classification adaptive process by the area processing unit 3001 shown in FIG. 133.

[0961] In step S3101, the region dividing unit 3102 divides the teacher image stored in the teacher image frame memory 3101 into regions based on the region information supplied from the region specifying unit 103. That is, the area dividing unit 3
Reference numeral 102 denotes an area-divided teacher image, which is an image including pixels belonging to the background area of the teacher image, and supplies the image to the background area teacher image frame memory 3103. Region dividing section 310
Reference numeral 2 denotes an undivided teacher image, which is an image composed of pixels belonging to the uncovered background area of the teacher image, is supplied to the uncovered background area teacher image frame memory 3104.

[0961] The area dividing section 3102 supplies an image, which is a teacher image divided into areas and is composed of pixels belonging to the covered background area of the teacher image, to the covered background area teacher image frame memory 3105. The region dividing unit 3102 is a region-divided teacher image.
An image composed of pixels belonging to the foreground area of the teacher image is supplied to the foreground area teacher image frame memory 3106.

[0963] In step S3102, the weighted averaging units 3107-1 to 3107-4 generate student images of the background area, foreground area, uncovered background area, and covered background area. That is, the weighted averaging unit 3107-1 performs, for example, a quarter weighted averaging of the image, which is stored in the background area teacher image frame memory 3103 and is composed of the pixels belonging to the background area of the teacher image, and obtains the weighted average of the teacher image. A student image corresponding to an image composed of pixels belonging to the background area is generated. Weighted average section 310
Reference numeral 7-2 denotes, for example, a quarter weighted averaging of an image composed of pixels belonging to the uncovered background area of the teacher image stored in the uncovered background area teacher image frame memory 3104 to obtain the teacher image. A student image corresponding to an image composed of pixels belonging to the uncovered background area is generated.

[0964] The weighted averaging unit 3107-3 performs, for example, a quarter weighted averaging of the images, which are stored in the covered background area teacher image frame memory 3105 and are composed of the pixels belonging to the covered background area of the teacher image. Thus, an image including pixels belonging to the covered background area of the teacher image is generated. Weighted average section 310
Reference numeral 7-4 denotes, for example, a quarter weighted averaging of an image composed of pixels belonging to the foreground area of the teacher image stored in the foreground area teacher image frame memory 3106 to obtain pixels belonging to the foreground area of the teacher image. A student image corresponding to the image consisting of is generated.

[0965] In step S3103, the learning unit 31
12-1 is a background area teacher image frame memory 3103
Consisting of pixels belonging to the background area of the teacher image stored in the background image and the background area student image frame memory 31
A coefficient set corresponding to the background area is generated based on the student image stored in 08 and corresponding to the image including the pixels belonging to the background area of the teacher image.

[0966] In step S3104, the learning unit 31
Reference numeral 12-2 denotes an image composed of pixels belonging to the uncovered background area of the teacher image, which is stored in the uncovered background area teacher image frame memory 3104, and is stored in the uncovered background area student image frame memory 3109. A coefficient set corresponding to the uncovered background area is generated based on the student image corresponding to the image including the pixels belonging to the uncovered background area of the teacher image.

[0967] In step S3105, the learning unit 31
12-3, an image composed of pixels belonging to the covered background area of the teacher image stored in the covered background area teacher image frame memory 3105, and stored in the covered background area student image frame memory 3110; A coefficient set corresponding to the covered background area is generated based on the student image corresponding to the image including the pixels belonging to the covered background area of the teacher image.

[0968] In step S3106, the learning unit 31
12-4 is a foreground area teacher image frame memory 3106
Consisting of pixels belonging to the foreground area of the teacher image stored in the foreground area and the student image frame memory 31
A coefficient set corresponding to the foreground area is generated based on the student image corresponding to the image composed of the pixels belonging to the foreground area of the teacher image stored in 11.

[0969] In step S3107, the learning section 31
Reference numerals 12-1 to 3112-4 denote a coefficient set corresponding to the background area, a coefficient set corresponding to the uncovered background area, a coefficient set corresponding to the covered background area, or a coefficient set corresponding to the foreground area, respectively. It is supplied to the set memory 3113. The coefficient set memory 3113 stores a coefficient set corresponding to each of the background area, foreground area, uncovered background area, or covered background area, and the process ends.

[0971] As described above, the area processing unit 3001 shown in Fig. 133 has a coefficient set corresponding to the background area, a coefficient set corresponding to the uncovered background area,
Coefficient set corresponding to the covered background area,
And a coefficient set corresponding to the foreground area.

[0971] Steps S3103 to S310
The details of the process 6 are the same as the processes described with reference to the flowchart in FIG. 103, and a description thereof will not be repeated.

[0972] Steps S3103 to S3103
It goes without saying that the process of 3106 may be executed serially or in parallel.

Next, with reference to the flowchart in FIG. 143, an image creation process of the area processing unit 3001 shown in FIG. 134 will be described.

[0974] In step S3201, the area dividing unit 3202 divides the input image into a background area, a foreground area, a covered background area, or an uncovered background area based on the area information supplied from the area specifying unit 103. . That is, the region dividing unit 3202
An image composed of pixels belonging to a background area, which is a divided input image, is supplied to a background area frame memory 3203,
An image composed of pixels belonging to the uncovered background area is supplied to the uncovered background area frame memory 3204, an image composed of pixels belonging to the covered background area is supplied to the covered background area frame memory 3205, and belonging to the foreground area. An image composed of pixels is stored in a foreground area frame memory 3206.
To supply.

[0975] In step S3202, the mapping unit 3207-1 executes the background area frame memory 320 by the class classification adaptive processing based on the coefficient set corresponding to the background area stored in the coefficient set memory 3208.
The image corresponding to the image consisting of the pixels belonging to the background area stored in 3 is predicted.

[0976] In step S3203, the mapping unit 3207-2 performs an uncovered background area frame by class classification adaptive processing based on the coefficient set corresponding to the uncovered background area stored in the coefficient set memory 3208. Memory 3204
Is predicted, the image corresponding to the image composed of the pixels belonging to the uncovered background area stored in.

[0977] In step S3204, mapping section 3207-3 performs covered background area frame memory 3205 by class classification adaptive processing based on the coefficient set corresponding to the covered background area stored in coefficient set memory 3208. Is predicted, which corresponds to the image composed of the pixels belonging to the covered background area stored in.

[0978] In step S3205, the mapping unit 3207-4 performs the class classification adaptive processing based on the coefficient set corresponding to the foreground area stored in the coefficient set memory 3208, and performs the foreground area frame memory 320
The image corresponding to the image consisting of the pixels belonging to the foreground area stored in 6 is predicted.

[0997] In step S3206, the synthesizing unit 32
09 is a predicted image corresponding to an image composed of pixels belonging to the background area, a predicted image corresponding to an image composed of pixels belonging to the uncovered background area, a predicted image corresponding to an image composed of pixels belonging to the covered background area, And a predicted image corresponding to the foreground area. The combining unit 3209 supplies the combined image to the frame memory 3210. The frame memory 3210 stores the image supplied from the combining unit 3209.

[0980] In step S3207, the frame memory 3210 outputs the stored combined image, and the process ends.

As described above, the image processing apparatus having the area processing unit 3001 shown in FIG. 134 divides an input image into each of a background area, an uncovered background area, a covered background area, and a foreground area. A prediction image can be generated for each of the divided images.

[0982] Steps S3202 to S320
The details of the process 5 are the same as the processes described with reference to the flowchart in FIG. 109, and a description thereof will be omitted.

[0983] Steps S3202 through S320
It goes without saying that the processing of 3205 may be executed serially or in parallel.

[0984] The processing executed by the motion blur removal image processing unit 108, the separated image processing unit 2002, and the area processing unit 3001 is to generate coefficients corresponding to SD images and HD images, or to convert HD images from SD images. Not only the process to generate,
For example, a coefficient for generating an image with a higher resolution in the spatial direction may be generated, and an image with a higher resolution in the spatial direction may be generated. Further, the motion blur removal image processing unit 108, the separated image processing unit 2002, and the area processing unit 3001 may execute a process of generating an image with higher resolution in the time direction.

[0985] Note that the motion-blur-removed image processing unit 108, the separated image processing unit 2002, and the area processing unit 3001
A coefficient may be generated from predetermined information, and the classification adaptive processing may be performed based on the generated coefficient.

[0986] Also, the motion blur removal image processing unit 108, the separated image processing unit 2002, and the area processing unit 3001
Based on the classification process, for example, the image size is converted to a desired size, color signals such as RGB are extracted, noise is removed, and other processes such as image compression or encoding are performed. You may. For example, based on the classified classes and the motion vectors corresponding to the respective images, the compression ratio in the direction along the motion vectors is given to the motion blur removal image processing unit 108, the separated image processing unit 2002, and the area processing unit 3001. If the compression ratio in the direction orthogonal to the motion vector is increased and the image for each region is compressed, the compression ratio can be increased with less deterioration of the image compared to the related art. .

[0987] The direction of movement of the foreground object has been described as left to right, but it is needless to say that the direction is not limited to that direction.

[0988] In the above, an example has been described in which an image of a real space having three-dimensional space and time-axis information is projected onto a two-dimensional space and time-space having time-axis information using a video camera. The present invention is not limited to this example, and can be applied to a case where more first information of the first dimension is projected onto second information of a smaller second dimension.

[0989] The sensor is not limited to a CCD, but is a solid-state image sensor, for example, a BBD (Bucket Brigade Device),
CID (Charge Injection Device) or CPD (Charge
Priming Device) may be used, and the sensor is not limited to a sensor in which detection elements are arranged in a matrix.
A sensor in which the detection elements are arranged in a line may be used.

As shown in FIG. 10, the recording medium on which the program for performing the signal processing of the present invention is recorded is a magnetic disk on which the program is recorded, which is distributed to provide the program to the user separately from the computer. 91 (including floppy (registered trademark) disk), optical disk 92
(CD-ROM (Compact Disc-Read Only Memory), DVD (Digita
l Versatile Disc), magneto-optical disc 93 (M
D (including a Mini-Disc) (trademark)) or a package medium including a semiconductor memory 94 and the like, and a program provided to a user in a state of being incorporated in a computer in advance. RO
It is composed of an M72, a hard disk included in the storage unit 78, and the like.

[0991] In this specification, the step of describing a program recorded on a recording medium may be performed in a chronological order according to the described order. This also includes processing executed in parallel or individually.

[0992]

According to the image processing apparatus and method, the recording medium, and the program of the present invention, foreground object components constituting a foreground object and background object components constituting a background object are mixed based on input image data. And a non-mixed region composed of one of a foreground region composed of a foreground object component and a background region composed of a background object component constituting a background object, and region identification information corresponding to the specified result Is output, and the class corresponding to each pixel data of the input image data is determined in accordance with the area specifying information, so that the image corresponding to the mixture of the background image and the moving object image is formed. Can be processed.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a conventional image processing apparatus.

FIG. 2 is a diagram illustrating a class tap.

FIG. 3 is a diagram illustrating a prediction tap.

FIG. 4 is a diagram illustrating an outline of a class classification adaptive process.

FIG. 5 is a diagram illustrating a conventional coefficient set.

FIG. 6 is a flowchart illustrating a conventional learning process.

FIG. 7 is a block diagram illustrating a configuration of a conventional image processing apparatus.

FIG. 8 is a diagram illustrating pixel values of an input image and pixel values of an output image generated by a class classification adaptive process.

FIG. 9 is a flowchart illustrating a conventional image creation process.

FIG. 10 is a block diagram illustrating a configuration of an embodiment of an image processing apparatus according to the present invention.

FIG. 11 is a block diagram illustrating a functional configuration of the image processing apparatus.

FIG. 12 is a diagram illustrating imaging by a sensor.

FIG. 13 is a diagram illustrating an arrangement of pixels.

FIG. 14 is a diagram illustrating the operation of the detection element.

FIG. 15 shows an object corresponding to a moving foreground;
FIG. 3 is a diagram illustrating an image obtained by capturing an object corresponding to a stationary background.

FIG. 16 is a diagram illustrating a background area, a foreground area, a mixed area, a covered background area, and an uncovered background area.

FIG. 17 is a model diagram in which pixel values of adjacent pixels arranged in one line in an image obtained by capturing an object corresponding to a stationary foreground and an object corresponding to a stationary background are developed in the time direction. It is.

FIG. 18 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 19 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 20 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 21 is a diagram illustrating an example in which pixels in a foreground area, a background area, and a mixed area are extracted.

FIG. 22 is a diagram showing the correspondence between pixels and models in which pixel values are developed in the time direction.

FIG. 23 is a model diagram in which pixel values are developed in a time direction and a period corresponding to a shutter time is divided.

FIG. 24 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 25 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 26 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 27 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 28 is a diagram showing a correspondence between a divided image and a model diagram in which pixel values of pixels are developed in the time direction.

FIG. 29 is a diagram showing a correspondence between a separated image and a model diagram in which pixel values of pixels are developed in the time direction.

FIG. 30 is a diagram illustrating an example of a divided image.

FIG. 31 is a diagram illustrating an example of a separated image.

FIG. 32 is a diagram showing a correspondence between an image from which motion blur has been removed and a model diagram in which pixel values of pixels are developed in the time direction.

FIG. 33 is a diagram illustrating processing of the image processing apparatus according to the present invention.

FIG. 34 is a flowchart illustrating image processing of the image processing apparatus according to the present invention.

FIG. 35 is a block diagram illustrating an example of a configuration of an area specifying unit 103.

FIG. 36 is a diagram illustrating an image when an object corresponding to the foreground is moving.

FIG. 37 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 38 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 39 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 40 is a diagram illustrating conditions for region determination.

41 is a diagram illustrating an example of a result of specifying an area by the area specifying unit 103. FIG.

FIG. 42 is a diagram illustrating an example of a result of specifying an area by the area specifying unit 103;

FIG. 43 is a flowchart illustrating an area specifying process.

FIG. 44 is a block diagram showing another example of the configuration of the area specifying unit 103.

FIG. 45 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 46 is a diagram illustrating an example of a background image.

FIG. 47 is a block diagram showing a configuration of a binary object image extraction unit 302.

FIG. 48 is a diagram illustrating calculation of a correlation value.

FIG. 49 is a diagram illustrating calculation of a correlation value.

FIG. 50 is a diagram illustrating an example of a binary object image.

FIG. 51 is a block diagram illustrating a configuration of a time change detection unit 303.

FIG. 52 is a view for explaining determination by an area determination unit 342.

FIG. 53 is a diagram illustrating an example of determination by the time change detection unit 303.

FIG. 54 is a flowchart illustrating an area identification process performed by an area determination unit.

FIG. 55 is a flowchart illustrating details of a region determination process.

FIG. 56 is a block diagram showing still another configuration of the area specifying unit 103.

FIG. 57 is a block diagram illustrating a configuration of a robust unit 361.

FIG. 58 is a diagram illustrating motion compensation by a motion compensation unit 381.

FIG. 59 is a diagram illustrating motion compensation by a motion compensation unit 381.

FIG. 60 is a flowchart illustrating an area specifying process.

FIG. 61 is a flowchart illustrating details of robust processing.

FIG. 62 is a block diagram illustrating an example of a configuration of a mixture ratio calculation unit 104.

FIG. 63 is a diagram showing an example of an ideal mixture ratio α.

FIG. 64 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 65 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 66 is a diagram for explaining approximation using correlation of foreground components.

FIG. 67 is a diagram illustrating the relationship between C, N, and P.

FIG. 68 is a block diagram illustrating a configuration of an estimated mixture ratio processing unit 401.

FIG. 69 is a diagram illustrating an example of an estimated mixture ratio.

FIG. 70 is a block diagram showing another configuration of the mixture ratio calculation unit 104.

FIG. 71 is a flowchart illustrating processing for calculating a mixture ratio.

FIG. 72 is a flowchart illustrating a process of calculating an estimated mixture ratio.

FIG. 73 is a diagram illustrating a straight line approximating the mixture ratio α.

FIG. 74 is a diagram illustrating a plane that approximates a mixture ratio α.

FIG. 75 is a diagram for explaining the correspondence of pixels in a plurality of frames when calculating the mixture ratio α.

FIG. 76 is a block diagram showing another configuration of the mixture ratio estimation processing unit 401.

FIG. 77 is a diagram illustrating an example of an estimated mixture ratio.

FIG. 78 is a flowchart illustrating processing for calculating a mixture ratio.

FIG. 79 is a flowchart illustrating a process of estimating a mixture ratio using a model corresponding to a covered background area.

FIG. 80 is a block diagram illustrating an example of a configuration of a foreground / background separation unit 105.

FIG. 81 is a diagram illustrating an input image, a foreground component image, and a background component image.

FIG. 82 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 83 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 84 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 85 is a block diagram illustrating an example of a configuration of a separation unit 601.

FIG. 86 is a diagram illustrating an example of a separated foreground component image and a background component image.

FIG. 87 is a flowchart illustrating a process of separating a foreground and a background.

FIG. 88 is a block diagram illustrating an example of a configuration of a motion-blur removing unit 106.

FIG. 89 is a diagram illustrating a processing unit.

FIG. 90 expands pixel values of a foreground component image in the time direction,
FIG. 3 is a model diagram obtained by dividing a period corresponding to a shutter time.

FIG. 91 expands pixel values of a foreground component image in the time direction,
FIG. 3 is a model diagram obtained by dividing a period corresponding to a shutter time.

FIG. 92 expands pixel values of a foreground component image in the time direction,
FIG. 3 is a model diagram obtained by dividing a period corresponding to a shutter time.

FIG. 93 is a flowchart illustrating a process of removing a motion blur included in a foreground component image by the motion blur removal unit 106.

FIG. 94 is a diagram illustrating a model of a background component image.

FIG. 95 is a diagram illustrating a model of a corrected background component image.

FIG. 96 is a block diagram illustrating a configuration of a motion blur removal image processing unit that generates a coefficient set.

FIG. 97 is a diagram illustrating a relationship between a teacher image and a student image.

FIG. 98 is a block diagram showing a configuration of a learning unit 1006.

FIG. 99 is a diagram for explaining class classification processing.

FIG. 100 is a diagram for explaining ADRC processing;

FIG. 101 is a diagram for describing a coefficient set generated by the motion blur removal image processing unit 108;

FIG. 102 is a flowchart illustrating a learning process of generating a coefficient set by the motion-blur removal image processing unit 108;

FIG. 103 is a flowchart illustrating a process of generating a coefficient set corresponding to a background component image.

FIG. 104 is a block diagram illustrating a configuration of a motion-blur-removed image processing unit that generates a higher-resolution image in a spatial direction by executing a class classification adaptive process.

FIG. 105 is a diagram illustrating a model of a foreground component image from which motion blur has been removed.

FIG. 106 is a diagram illustrating a model of a foreground component image to which motion blur has been added.

FIG. 107 is a block diagram showing a configuration of a mapping unit 1103.

FIG. 108 is a flowchart illustrating an image creation process of the motion blur removal image processing unit 108;

FIG. 109 is a flowchart illustrating a process of predicting an image corresponding to a background component image.

FIG. 110 is a block diagram illustrating another configuration of the functions of the image processing apparatus.

FIG. 111 is a block diagram illustrating an example of a configuration of a mixture ratio calculation unit 1501.

112 is a block diagram illustrating an example of a configuration of a foreground / background separation unit 1502. FIG.

FIG. 113 is a block diagram illustrating another configuration of the functions of the image processing apparatus.

114 is a diagram for describing processing of a separated image processing unit 2002. FIG.

115 is a block diagram illustrating an example of a configuration of a foreground / background separation unit 2001. FIG.

FIG. 116 is a block diagram illustrating an example of the configuration of a separation unit 2101.

FIG. 117: Separated image processing unit 20 for generating a coefficient set
FIG. 2 is a block diagram showing a configuration of the second embodiment.

FIG. 118 is a block diagram illustrating a configuration of a separated image processing unit 2002 that generates a higher-resolution image in a spatial direction.

FIG. 119 is a diagram illustrating an example of an image in a mixed area of a teacher image.

FIG. 120 is a diagram illustrating an example of an image of a mixed area generated by the conventional classification adaptive processing.

FIG. 121 is a diagram illustrating an example of an image of a mixed area generated by a separated image processing unit 2002.

FIG. 122 is a diagram illustrating an example of an image in a foreground area of a teacher image.

FIG. 123 is a diagram illustrating an example of an image of a foreground area generated by the conventional classification adaptive processing.

FIG. 124 is a diagram illustrating an example of an image of a foreground area generated by a separated image processing unit 2002.

125 is a flowchart illustrating image processing of the image processing apparatus illustrated in FIG. 113.

126 is a flowchart for describing processing for separating a foreground and a background by a foreground / background separation unit 2001. FIG.

FIG. 127 is a flowchart illustrating a learning process of generating a coefficient set by the separated image processing unit 2002.

FIG. 128 is a flowchart illustrating an image creation process performed by a separated image processing unit 2002.

FIG. 129 is a block diagram illustrating still another configuration of the functions of the image processing apparatus.

130 is a block diagram illustrating an example of a configuration of a foreground / background separation unit 2501. FIG.

FIG. 131 is a block diagram illustrating still another configuration of the functions of the image processing apparatus.

FIG. 132 is a view for explaining processing of an area processing unit 3001;

FIG. 133 is an area processing unit 3001 that generates a coefficient set.
FIG. 3 is a block diagram showing the configuration of FIG.

FIG. 134 is a block diagram illustrating a configuration of an area processing unit 3001 that generates a higher-resolution image in the spatial direction.

FIG. 135 is a diagram illustrating an example of an image in a mixed area of a teacher image.

FIG. 136 is a diagram illustrating an example of an image of a mixed region generated by the conventional classification adaptive processing.

FIG. 137 is a diagram illustrating an example of an image of a mixed area generated by the area processing unit 3001.

FIG. 138 is a diagram illustrating an example of an image in a foreground area of a teacher image.

FIG. 139 is a diagram illustrating an example of an image of a foreground area generated by conventional classification adaptive processing.

140 is a diagram illustrating an example of an image of a foreground area generated by an area processing unit 3001. FIG.

FIG. 141 is a flowchart illustrating image processing of the image processing apparatus according to the present invention.

FIG. 142 is a flowchart illustrating a learning process of generating a coefficient set performed by an area processing unit 3001.

FIG. 143 is a flowchart illustrating an image creation process of an area processing unit 3001;

[Explanation of symbols]

71 CPU, 72 ROM, 73 RAM, 76 input section, 77 output section, 78 storage section, 79 communication section,
91 magnetic disk, 92 optical disk, 93
Magneto-optical disk, 94 semiconductor memory, 101 object extraction unit, 102 motion detection unit, 103
Area specifying section, 104 mixture ratio calculating section, 105 foreground / background separating section, 106 motion blur removing section, 107 correcting section, 108 motion blur removing image processing section, 201 frame memory, 202-1 to 202-4 static / moving determining sections, 203-1 to 203-3 area determination unit, 20
4 Frame memory for storing judgment flag, 205 Synthesis unit, 206 Frame memory for storing judgment flag, 30
1 background image generation unit, 302 binary object image extraction unit, 303 time change detection unit, 321 correlation value calculation unit, 322 threshold processing unit, 341 frame memory, 342 area determination unit, 361 robust conversion unit, 381 motion compensation Part, 382 switch, 3
83-1 to 383-N frame memory, 384-
1 to 384-N weighting unit, 385 accumulating unit, 4
01 estimated mixture ratio processing unit, 402 estimated mixture ratio processing unit, 403 mixture ratio determination unit, 421 frame memory, 422 frame memory, 423 mixture ratio calculation unit, 441 selection unit, 442 estimated mixture ratio processing unit,
443 Estimated mixture ratio processing section, 444 Selection section, 5
01 delay circuit, 502 addition unit, 503 operation unit, 601 separation unit, 602 switch, 603
Synthesis unit, 604 switch, 605 synthesis unit,
621 frame memory, 622 separation processing block, 623 frame memory, 631 uncovered area processing unit, 632 covered area processing unit, 63
3 synthesis unit, 634 synthesis unit, 801 processing unit determination unit, 802 modeling unit, 803 equation generation unit,
804 addition unit, 805 operation unit, 1001
Background component teacher image frame memory, 1002 Foreground component teacher image frame memory, 1003-1 and 1
003-2 weighted averaging unit, 1004 background component student image frame memory, 1005 foreground component student image frame memory, 1006-1 and 1006-2 learning units, 1007 coefficient set memory, 1031 class classification unit, 1032 prediction tap acquisition unit, 1033
Corresponding pixel acquisition unit, 1034 normal equation generation unit, 1
035 coefficient calculation unit, 1051 class tap acquisition unit, 1052 waveform classification unit, 1101 background component image frame memory, 1102 foreground component image frame memory, 1103-1 and 1103-2 mapping unit, 1104 coefficient set memory, 1105 correction unit, 1106 Motion blur adding unit, 1107 combining unit, 1131 mapping processing unit, 1141 class classification unit, 1142 prediction tap acquisition unit, 1143
Prediction operation unit, 1151 class tap acquisition unit, 11
52 waveform classifier, 1501 mixture ratio calculator, 15
02 Foreground / background separator, 1521 selector, 2001
Foreground / background separation unit, 2002 Separated image processing unit, 2
101 separation unit, 2102 switch, 2103
Switch, 2201 background area teacher image frame memory, 2202 uncovered background area background component teacher image frame memory, 2203 uncovered background area foreground component teacher image frame memory, 2204 covered background area background component teacher image frame memory, 2205 covered Background area foreground component teacher image frame memory,
2206 Foreground area teacher image frame memory, 22
07-1 to 2207-6 Weighted average part, 2208
Background area student image frame memory, 2209 uncovered background area background component student image frame memory, 2210 uncovered background area foreground component student image frame memory, 2211 covered background area background component student image frame memory, 2212 covered background area Foreground component student image frame memory, 2213 Foreground area student image frame memory, 2214-1 to 2214-
6 learning unit, 2215 coefficient set memory, 230
1 background area frame memory, 2302 uncovered background area background component image frame memory,
2303 uncovered background area foreground component image frame memory, 2304 covered background area background component image frame memory, 2305
Covered background area foreground component image frame memory, 2306 foreground area frame memory, 23
07-1 to 2307-6 mapping unit, 2308
Coefficient set memory, 2309 synthesis unit, 2501
Foreground / background separation unit, 2521 selection unit, 3001 region processing unit, 3102 region division unit, 3103 background region teacher image frame memory, 3104 uncovered background region teacher image frame memory, 3
105 Covered background area teacher image frame memory, 3106 Foreground area teacher image frame memory, 3107-1 to 3107-4 Weighted average section,
3108 background area student image frame memory, 310
9 uncovered background area student image frame memory, 3110 covered background area student image frame memory, 3111 foreground area student image frame memory, 3112-1 to 3112-4 learning unit, 3113 coefficient set memory, 3206 area division unit, 3203 Background area frame memory, 3
204 uncovered background area frame memory, 3205 covered background area frame memory, 3206 foreground area frame memory, 3
207-1 to 3207-4 mapping unit, 320
8 coefficient set memory, 3209 synthesis unit

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Takashi Sawao 6-7-35 Kita-Shinagawa, Shinagawa-ku, Tokyo Inside Sony Corporation (72) Inventor Naoki Fujiwara 6-35, Kita-Shinagawa, Shinagawa-ku, Tokyo No. Sony Corporation (72) Inventor Takahiro Nagano 6-7-35 Kita Shinagawa, Shinagawa-ku, Tokyo Sony Corporation (72) Inventor Seiji Wada 6-7-35 Kita-Shinagawa, Shinagawa-ku, Tokyo Inside Sony Corporation (72) Inventor Toru Miyake 6-35 Kita Shinagawa, Shinagawa-ku, Tokyo Sony Corporation F-term (reference) 5B057 BA02 CA12 CA16 CE09 DA06 DB02 5C021 RA01 RB03 RB06 XA03 XA07 XA08 YC13 5C023 AA06 AA07 AA16 AA37 BA04 BA13 CA01 DA02 DA03 5C054 AA04 CC02 EA01 ED12 EJ07 FC13 FC14 GA00 GA04 GB14 GB15 5L096 CA02 GA08 HA01

Claims (16)

[Claims] An image processing apparatus for processing input image data consisting of a predetermined number of pixel data obtained by an image sensor having a predetermined number of pixels having a time integration effect, comprising: a foreground object based on the input image data A mixed region formed by mixing foreground object components constituting a background object component and a background object component constituting a background object; and a foreground region composed of the foreground object components,
And a region specifying means for specifying a non-mixed region constituted by one of the background regions comprising the background object components constituting the background object, and outputting region specifying information corresponding to the specified result; And a class determining means for determining a class corresponding to each pixel data of the input image data. 2. The image processing apparatus according to claim 1, further comprising a generating unit configured to process the pixel data of the input image data in accordance with the determined class and generate coefficients used in a class classification adaptive process. Item 2. The image processing device according to Item 1. 3. A conversion means for processing the pixel data of the input image data based on a coefficient for each class corresponding to the determined class and converting the input image data into output image data. The image processing apparatus according to claim 1, further comprising: 4. The area specifying means further specifies a covered background area and an uncovered background area, and outputs the area specifying information corresponding to a specified result. The image processing apparatus according to claim 1, wherein the class corresponding to the background area or the uncovered background area and corresponding to the pixel data of the input image data is determined. 5. An image processing method for processing input image data consisting of a predetermined number of pixel data obtained by an image sensor having a predetermined number of pixels having a time integration effect, comprising: A mixed region formed by mixing foreground object components constituting a background object component and a background object component constituting a background object; and a foreground region composed of the foreground object components,
And an area specifying step of specifying a non-mixed area constituted by one of the background areas composed of the background object components constituting the background object, and outputting area specifying information corresponding to the specified result; And a class determining step of determining a class corresponding to each pixel data of the input image data. 6. The method according to claim 1, further comprising the step of: processing the pixel data of the input image data according to the determined class to generate coefficients used in a classification adaptive process. Item 6. The image processing method according to Item 5. 7. A conversion step of processing the pixel data of the input image data based on the coefficient for each class and converting the input image data into output image data in accordance with the determined class. The image processing method according to claim 5, further comprising: 8. In the area specifying step, a covered background area and an uncovered background area are further specified, and the area specifying information corresponding to the specification result is output. The image processing method according to claim 5, wherein the class corresponding to the covered background area or the uncovered background area and corresponding to the pixel data of the input image data is determined. 9. An image processing program for processing input image data consisting of a predetermined number of pixel data obtained by an image sensor having a predetermined number of pixels having a time integration effect, wherein the program is based on the input image data. A mixed area formed by mixing foreground object components constituting a foreground object and a background object component constituting a background object; and a foreground area composed of the foreground object components,
And an area specifying step of specifying a non-mixed area constituted by one of the background areas made up of the background object components constituting the background object, and outputting area specifying information corresponding to the specified result. A class determining step of determining a class corresponding to each pixel data of the input image data. 10. The program further includes a generating step of processing the pixel data of the input image data according to the determined class to generate coefficients used in a classification adaptive process. The recording medium according to claim 9, wherein: 11. The program processes the pixel data of the input image data based on a coefficient for each class corresponding to the determined class, and converts the input image data into output image data. The recording medium according to claim 9, further comprising a converting step. 12. In the area specifying step, a covered background area and an uncovered background area are further specified, and the area specifying information corresponding to the specification result is output. The recording medium according to claim 9, wherein the class corresponding to the pixel data of the input image data is determined corresponding to the covered background area or the uncovered background area. 13. A computer for processing input image data consisting of a predetermined number of pixel data obtained by an image sensor having a predetermined number of pixels having a time integration effect, comprising: A mixed area formed by mixing foreground object components and a background object component forming a background object; and a foreground area including the foreground object components.
And an area specifying step of specifying a non-mixed area constituted by one of the background areas composed of the background object components constituting the background object, and outputting area specifying information corresponding to the specified result; And a class determining step of determining a class corresponding to each pixel data of the input image data. 14. The method according to claim 1, further comprising the step of: processing the pixel data of the input image data according to the determined class to generate coefficients used in a classification adaptive process. Item 14. The program according to Item 13. 15. A conversion step of processing the pixel data of the input image data based on the coefficient for each class and converting the input image data into output image data corresponding to the determined class. The program according to claim 13, further comprising: 16. In the area specifying step, a covered background area and an uncovered background area are further specified, and the area specifying information corresponding to the specification result is output. 14. The program according to claim 13, wherein the class corresponding to the covered background area or the uncovered background area and corresponding to the pixel data of the input image data is determined.