JP2001250119A - Device and method for signal processing and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain significant information from the detection signal obtained by a sensor and to adjust the distortion generated in the detection signal by the sensor according to the significant information. SOLUTION: A signal processing part 12 obtains a 2nd signal, having 2nd dimensions less than 1st dimensions, which is obtained by detecting a 1st signal of the real world having the 1st dimensions by the sensor and includes distortion for the 1st signal, and performs signal processing based upon the 2nd signal to generate a 3rd signal which has less distortion than the 2nd signal.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a signal processing apparatus and method, and a recording medium, and more particularly to a signal processing apparatus and method and a recording medium in which a difference between a signal detected by a sensor and the real world is considered.

[0002]

2. Description of the Related Art A technique for detecting an event in the real world by a sensor and performing signal processing on sampling data output from the sensor, such as data corresponding to images, sounds, temperatures, pressures, accelerations, and odors, is widely used. . Further, real-world information having a space and a time axis is acquired by a sensor and converted into data. The data acquired by the sensor is information obtained by projecting information in the real world onto a spatio-temporal dimension lower than that in the real world. Therefore, the information obtained by the projection has distortion generated by the projection. In other words, the data output by the sensor has a distortion with respect to real world information. Further, although the data has distortion due to projection, it includes significant information for correcting the distortion.

[0003] In the conventional signal processing for the sampling data obtained by such a sensor, the sampling data obtained by the sensor is considered to be the most reliable data. In consideration of the above, only approaching the original data was considered.

[0004] For example, in an image obtained by capturing a moving object in front of a predetermined stationary background with a video camera using a CCD or the like, motion blur occurs when the moving speed of the object is relatively high. Will be. That is, CC which is a sensor
When the real world is detected in D, the image that is the sampling data is distorted.

Conventionally, to suppress such motion blur, for example, the speed of an electronic shutter is increased and the exposure time is shortened. However, a motion which is a distortion generated by a sensor due to signal processing is performed. It does not suppress blurring.

[0006]

Conventionally, the sampling data output from the sensor was considered to be the most reliable data. Therefore, data of higher quality than the sampling data was created, and distortion due to projection was considered from the sampling data. No signal processing such as extraction of significant information buried therein by projection has been performed.

The present invention has been made in view of such circumstances, and has as its object to provide a signal processing apparatus capable of removing distortion from sampling data output by a sensor and extracting significant information. I do. For example, it is an object of the present invention to be able to adjust the amount of motion blur included in a detection signal for an image.

[0008]

According to a first aspect of the present invention, there is provided a signal processing apparatus, wherein a first signal, which is a real-world signal having a first dimension, is projected onto a sensor and detected by the sensor. Acquiring means for acquiring a second signal of a second dimension less than the first dimension, and performing signal processing based on the second signal to extract significant information buried by projection from the second signal. Signal processing means.

[0009] The significant information may be information for adjusting distortion caused by the projection.

The sensor is a sensor composed of a plurality of detection elements each having a time integration effect, and the acquisition means acquires a plurality of detection signals corresponding to the individual detection elements detected by the sensor as a second signal, The distortion can be a distortion due to a time integration effect.

The acquisition means acquires a plurality of detection signals in a plurality of time units detected by a plurality of detection elements of the sensor for each predetermined time unit, and the signal processing means converts the plurality of detection signals in a plurality of time units into a plurality of detection signals. Based on this, it is possible to extract significant information for the second signal at a desired time.

[0012] The second signal can be an image signal.

The signal processing means specifies a significant area including significant information buried by the projection and other areas in the second signal, and outputs an area information indicating the specified area as significant information. Means may be provided.

The area information includes a foreground area, which is another area consisting only of the foreground object components constituting the foreground object, a background area which is another area consisting only of the background object components constituting the background object, and a foreground object component. And a background area component.

The area information may include information for identifying the mixed area as a covered background area and an uncovered background area.

[0016] The signal processing means may further include significant information extracting means for extracting significant information from the area including the significant information specified by the area specifying means.

The significant information is obtained by mixing a foreground area consisting of only foreground object components constituting a foreground object, a background area consisting of only background object components constituting a background object, and a foreground object component and a background object component. The mixing ratio of the foreground object component and the background object component in the mixed region of the second signal including the mixed region may be indicated.

[0018] The signal processing means may further include a distortion amount adjusting means for adjusting an amount of distortion generated in the second signal by the projection based on the significant information.

The distortion amount adjusting means can reduce the amount of distortion.

The distortion amount adjusting means can remove the distortion.

The distortion can be a motion blur occurring in the foreground object.

The signal processing means further comprises an object motion detecting means for detecting a motion amount of the foreground object,
The distortion amount adjusting means can adjust the amount of motion blur, which is distortion, based on the amount of movement of the foreground object.

The signal processing means mixes the foreground object consisting of only the foreground object component constituting the foreground object, the background region consisting of only the background object component constituting the background object, the foreground object component and the background object component. A second mixed region
The mixing ratio of the foreground object component and the background object component in the mixed area of the signal can be extracted as significant information.

[0024] The signal processing means may further include a distortion amount adjusting means for adjusting an amount of distortion generated in the second signal by projection based on the significant information.

[0025] The distortion amount adjusting means can reduce the amount of distortion.

The distortion amount adjusting means can remove the distortion.

The distortion can be a motion blur occurring in the foreground object.

[0028] The signal processing means further comprises an object motion detecting means for detecting a motion amount of the foreground object,
The distortion amount adjusting means can adjust the amount of motion blur, which is distortion, based on the amount of movement of the foreground object.

The signal processing means, based on the second signal,
The mixture ratio is extracted as significant information, and the signal processing means
, Region information indicating the foreground region, the background region, and the mixed region can be extracted as significant information.

The signal processing means can separate a foreground object consisting only of foreground object components from a background object consisting only of background object components based on the mixture ratio and the area information.

[0031] The signal processing means may further include a distortion amount adjusting means for adjusting a motion blur which is a distortion occurring in the foreground object.

[0032] The signal processing means further comprises object motion detecting means for detecting a motion amount of the foreground object,
The distortion amount adjusting means can adjust the amount of motion blur, which is distortion, based on the amount of movement of the foreground object.

[0033] The signal processing means can combine the foreground object with an arbitrary background image.

The signal processing method according to claim 27 is characterized in that
Acquiring a first signal that is a real-world signal having the dimensions of: a second signal having a second dimension smaller than the first dimension obtained by projecting the first signal onto the sensor and detecting the first signal with the sensor; , A signal processing step of extracting significant information buried by projection from the second signal by performing signal processing based on the second signal.

[0035] In the recording medium according to the present invention, the first signal, which is a real-world signal having the first dimension, is projected onto a sensor and detected by the sensor to obtain the first dimension. By performing an acquisition step of acquiring a second signal having a smaller second dimension and performing signal processing based on the second signal, significant information buried by projection is converted to a second signal.
And a signal processing step of extracting from the signals of

[0036] The signal processing device according to claim 29 is characterized in that:
Has a second dimension that is smaller in dimension than the first dimension and is obtained by detecting a first signal that is a real-world signal having a dimension of An acquisition unit that acquires the second signal; and a signal processing unit that performs signal processing based on the second signal to generate a third signal whose distortion is reduced compared to the second signal. It is characterized by the following.

The sensor is a sensor composed of a plurality of detection elements each having a time integration effect that is a distortion, and the acquisition means uses a plurality of detection signals corresponding to each detection element detected by the sensor as a second signal. The acquired signal processing means performs signal processing based on the second signal to generate a third signal including a plurality of sample data corresponding to a plurality of detection signals, in which a time integration effect is reduced. Can be

The signal processing means detects a first object in the real world and a second object moving relatively to the first object by a sensor, and outputs the first object represented by the second signal. In the vicinity of the boundary between the first object and the second object, distortion caused by mixing of the first object and the second object caused by the time integration effect of the sensor can be reduced by signal processing.

The acquisition means acquires a plurality of time-unit detection signals detected by the plurality of detection elements of the sensor for each predetermined time unit, and the signal processing means acquires a plurality of time-unit detection signals based on the plurality of time-unit detection signals. The distortion near the boundary between the first object and the second object represented by the second signal corresponding to the desired time unit can be reduced by signal processing.

The signal processing means is mixed in the second signal when the sensor detects the first object in the real world and the second object moving relatively to the first object. One of the first object and the second object is separated from the first object and the second object, and one of the separated first object and the second object is sent to a third signal. Can be output as

The sensor can convert an electromagnetic wave including light, which is a first signal, into an image signal, which is a second signal, by photoelectric conversion.

The sensor can be a thermographic device for measuring temperature.

The sensor can be a pressure sensor.

The sensor may generate the second signal at predetermined time intervals.

The sensor is a sensor having a plurality of detection elements each having a spatial integration effect, and the acquisition means acquires a plurality of detection signals corresponding to the individual detection elements detected by the sensor as a second signal, The signal processing means includes:
By performing signal processing based on the second signal, it is possible to generate a third signal including a plurality of detection signals in which the spatial integration effect due to projection as distortion is reduced.

The sensor has at least one detecting element.
A plurality of signal processing means are arranged in a predetermined direction which is one of the two directions, and the signal processing means performs a correlation between each sample data of the second signal and two sample data of the second signal adjacent in the predetermined direction. Correlation detection means for respectively detecting
For each sample data of interest, a first sample value that is a sample value on the sample data side with the higher correlation is generated based on the sample value of the sample data with the higher correlation among the two adjacent sample data. Generating, based on the sample value of the sample data of interest and the first sample value, a second sample value that is a sample value with a smaller correlation, and assigning the first sample value and the second sample value to the sample of interest. And a double-density sample generating unit that outputs two sample values of the third signal corresponding to the third signal.

In the sensor, a plurality of detection elements are arranged in a matrix, and the predetermined direction can be at least one of the horizontal direction and the vertical direction in the matrix arrangement.

The signal processing means can double the density in both the horizontal and vertical directions.

The correlation can be a difference between the sample data.

[0050] The acquisition means may acquire the second signal from the plurality of detection elements at predetermined time intervals.

The signal processing method according to claim 44 is characterized in that:
Has a second dimension that is smaller in dimension than the first dimension and is obtained by detecting a first signal that is a real-world signal having a dimension of An acquisition step of acquiring a second signal; and a signal processing step of performing a signal processing based on the second signal to generate a third signal whose distortion is reduced as compared with the second signal. It is characterized by the following.

The program of the recording medium according to claim 45, wherein the dimension is compared with the first dimension, obtained by detecting the first signal, which is a real-world signal having the first dimension, by a sensor. An acquisition step of acquiring a second signal having a small second dimension and including distortion with respect to the first signal; and performing signal processing based on the second signal, thereby obtaining distortion in comparison with the second signal. And a signal processing step of generating a reduced third signal.

The signal processing apparatus according to claim 46, wherein, among the detection signals, a foreground area consisting only of a foreground object component constituting a foreground object, a background area consisting only of a background object component constituting a background object,
Area specifying means for specifying a mixed area formed by mixing a foreground object component and a background object component;
At least a mixture ratio detection unit that detects a mixture ratio between a foreground object component and a background object component in the mixture region, and a specification result and a mixture ratio of the region specification unit,
It is characterized by including a separating means for separating a foreground object and a background object.

The signal processing device may further include a motion blur amount adjusting means for adjusting the motion blur amount of the foreground object.

In the signal processing method according to the present invention, a foreground area consisting only of a foreground object component constituting a foreground object and a background area consisting only of a background object component constituting a background object are included in the detection signal.
An area specifying step of specifying a mixed area formed by mixing a foreground object component and a background object component; a mixing ratio detecting step of detecting a mixing ratio of at least a foreground object component and a background object component in the mixed area; The image processing apparatus further includes a separation step of separating the foreground object and the background object based on the specified result of the step processing and the mixture ratio.

The recording medium program according to claim 49 is a signal processing program for processing a predetermined number of detection signals obtained by a sensor including a predetermined number of detection elements having a time integration effect. Of the detection signals, a foreground region consisting of only the foreground object component constituting the foreground object, a background region consisting of only the background object component constituting the background object, and a mixture of the foreground object component and the background object component An area specifying step of specifying an area, a mixing ratio detecting step of detecting a mixing ratio of at least a foreground object component and a background object component in the mixed area, and a specifying result and a mixing ratio of the processing of the area specifying step, Separation step for separating foreground and background objects Characterized in that it comprises a.

According to a fifty-first aspect of the present invention, in the signal processing apparatus, a foreground region consisting of only a foreground object component constituting a foreground object and a background region consisting of only a background object component constituting a background object are included in the detection signal.
Area specifying means for specifying a mixed area formed by mixing a foreground object component and a background object component;
A mixing ratio detecting unit that detects a mixing ratio of at least a foreground object component and a background object component in the mixed region based on the specification result of the region specifying unit.

The signal processing device may further include a separating unit for separating the foreground object and the background object based on the mixture ratio.

The signal processing apparatus may further include a motion blur amount adjusting means for adjusting the amount of motion blur included in the foreground object.

The signal processing device further includes a motion detecting means for detecting at least one of the foreground object and the background object, and the motion blur adjusting means adjusts the amount of motion blur based on the detected motion. You can make it.

A signal processing method according to claim 54, wherein, among the detection signals, a foreground region consisting only of a foreground object component constituting a foreground object, a background region consisting only of a background object component constituting a background object,
A region specifying step of specifying a mixed region formed by mixing the foreground object component and the background object component; and mixing the foreground object component and the background object component in at least the mixed region based on the specification result of the processing of the region specifying step And a mixing ratio detecting step of detecting a ratio.

The program of the recording medium according to claim 55, wherein the detection signal includes a foreground area consisting of only a foreground object component constituting a foreground object, a background area consisting of only a background object component constituting a background object, A region specifying step of specifying a mixed region formed by mixing the foreground object component and the background object component; and mixing the foreground object component and the background object component in at least the mixed region based on the specification result of the processing of the region specifying step And a mixing ratio detecting step of detecting a ratio.

A signal processing apparatus according to claim 56, wherein the foreground object component and the background object component in a mixed area in which the foreground object component forming the foreground object and the background object component forming the background object are mixed. And a separation unit that separates a foreground object from a background object based on the mixture ratio.

The signal processing device may further include a motion blur amount adjusting means for adjusting the amount of motion blur included in the foreground object.

The signal processing device further includes a motion detecting means for detecting at least one of the foreground object and the background object, and the motion blur adjusting means adjusts the amount of motion blur based on the detected motion. You can make it.

A signal processing method according to claim 59, wherein the foreground object component and the background object component in a mixed area formed by mixing the foreground object component forming the foreground object and the background object component forming the background object. And a separation step of separating a foreground object and a background object based on the mixture ratio.

The program of the recording medium according to claim 60, wherein the foreground object component and the background object component in a mixed area formed by mixing the foreground object component forming the foreground object and the background object component forming the background object. And a separation step of separating a foreground object and a background object based on the mixture ratio.

A signal processing device according to claim 1 and claim 2.
In the signal processing method according to the seventh aspect and the recording medium according to the second aspect, the first signal, which is a real-world signal having the first dimension, is projected onto the sensor and detected by the sensor. A second signal of a second dimension less than the first dimension is obtained, and by performing signal processing based on the second signal, significant information buried by projection is converted to a second signal.
From the signal of

In the signal processing device according to the present invention, the signal processing method according to the present invention, and the recording medium according to the present invention, the first signal which is a real-world signal having a first dimension is provided. A second signal obtained by detecting the signal with the sensor and having a second dimension having a smaller dimension than the first dimension and including a distortion with respect to the first signal is obtained. By performing signal processing based on the third signal, a third signal with reduced distortion compared to the second signal is generated.

In the signal processing device according to claim 46, the signal processing method according to claim 48, and the recording medium according to claim 49, only the foreground object component constituting the foreground object is included in the detection signal. A foreground region, a background region consisting only of a background object component constituting the background object, and a mixed region formed by mixing the foreground object component and the background object component are specified, and at least the foreground object component and the background object in the mixed region are specified. The mixing ratio with the component is detected,
Based on the specific result and the mixture ratio, the foreground object,
The background object is separated.

In the signal processing apparatus according to claim 50, the signal processing method according to claim 54, and the recording medium according to claim 55, only the foreground object component constituting the foreground object is included in the detection signal. A foreground region, a background region consisting only of a background object component constituting the background object, and a mixed region formed by mixing the foreground object component and the background object component are specified, and based on the specified result, at least the mixed region A mixture ratio between the foreground object component and the background object component is detected.

In the signal processing device according to claim 56, the signal processing method according to claim 59, and the recording medium according to claim 60, a foreground object component forming a foreground object and a background object are formed. The mixture ratio of the foreground object component and the background object component in the mixture region where the background object component is mixed is detected, and the foreground object and the background object are separated based on the mixture ratio.

[0073]

FIG. 1 shows the principle of the present invention. As shown in the figure, a first signal which is information of the real world 1 having a space and a time axis is acquired by the sensor 2,
Be converted to data. The detection signal, which is the data 3 acquired by the sensor 2, is information obtained by projecting information of the real world 1 onto a spatio-temporal space lower than the real world. Therefore, the information obtained by the projection has distortion generated by the projection. In other words, the data 3 output by the sensor 2 has distortion with respect to the information of the real world 1. The data 3 has distortion due to projection, but includes significant information for correcting the distortion.

Therefore, in the present invention, the signal output from the sensor 2 is subjected to signal processing in the signal processing section 4 to remove, reduce, or adjust the distortion. Alternatively, in the present invention, the signal output from the sensor 2 is subjected to signal processing in the signal processing unit 4 to extract significant information.

FIG. 2 shows a configuration example of a signal processing device to which the present invention is applied. The sensor 11 includes, for example, a video camera, captures a real-world image, and outputs obtained image data to the signal processing unit 12. Signal processing unit 1
2 is configured by, for example, a personal computer or the like, processes data input from the sensor 11, adjusts the amount of distortion generated by projection, specifies an area including significant information buried by projection, Further, significant information is extracted from the specified area, and the input data is processed based on the extracted significant information.

The significance information referred to here is, for example, a mixing ratio described later.

Note that information indicating an area including significant information buried by projection can also be considered as significant information.
Here, region information described later corresponds to significant information.

The area including the significant information is, for example, a mixed area described later.

The signal processing section 12 is configured, for example, as shown in FIG. CPU (Central Processing Uuit) 21
Is a ROM (Read Only Memory) 22 or a storage unit 28
Executes various processes according to the program stored in the. A RAM (Random Access Memory) 23 has a CPU 21
And the programs executed by the program are stored as appropriate.
The CPU 21, the ROM 22, and the RAM 23 are connected to the bus 2
4 are interconnected.

An input / output interface 25 is connected to the CPU 21 via a bus 24. The input / output interface 25 is connected to an input unit 26 including a keyboard, a mouse, a microphone, and the like, and an output unit 27 including a display, a speaker, and the like. The CPU 21 executes various processes in response to a command input from the input unit 26. Then, the CPU 21 outputs an image, a sound, and the like obtained as a result of the processing to the output unit 27.

The storage unit 28 connected to the input / output interface 25 is composed of, for example, a hard disk and stores programs executed by the CPU 21 and various data. The communication unit 29 communicates with an external device via the Internet or another network. In the case of this example, the communication unit 29 functions as an acquisition unit that captures the output of the sensor 11.

Further, a program may be obtained through the communication unit 29 and stored in the storage unit 28.

The drive 30 connected to the input / output interface 25 includes a magnetic disk 51, an optical disk 5
2. When the magneto-optical disk 53, the semiconductor memory 54, and the like are mounted, they are driven to acquire programs, data, and the like recorded therein. The acquired programs and data are transferred and stored in the storage unit 28 as necessary.

Next, referring to the flowchart of FIG.
An operation performed by the signal processing device based on a program stored in the storage unit 28 as a storage medium will be described.
First, in step S1, an image of a subject obtained by the sensor 11 is obtained via the communication unit 29 or the like. The CPU 21 of the signal processing unit 12 supplies the acquired image data to the storage unit 28 and stores it.

FIG. 5 shows an image corresponding to the image data thus obtained. In this example,
This is an image in which a foreground 62 is arranged before a background 61. In this example, the foreground 62 is a toy airplane, and moves rightward in the figure at a predetermined speed in front of the stationary background 61. As a result, the image of the foreground 62 is an image having so-called motion blur. On the other hand, since the image of the background 61 is stationary, it is a clear image without motion blur. And the mixed area 63 is the background 6
The image is an image in which the object 1 and the object 62 are mixed.

Next, in step S2, the CPU 21
Detects the mixed area of the object. In the example of FIG. 5, the mixed area 63 is detected as an area where objects are mixed.

The CPU 21 determines in step S3 whether or not objects are mixed. If the objects are not mixed, that is, if there is no mixed area 63, the processing is terminated because it is not a processing target in this information processing apparatus.

On the other hand, if it is determined in step S3 that the objects are mixed, the process proceeds to step S4, where the CPU 21 obtains the mixture ratio of the objects in the detected mixed area. The mixture ratio is, for example, foreground 6
A motion vector for the second background 61 is obtained, and the motion vector is applied to the mixed area 63 so that the mixing ratio changes from 0 to 1. Further, in step S5, the CPU 21
Based on the obtained mixing ratio, a process of separating the objects in the mixed area 63 where a plurality of objects are mixed is executed.

The above processing will be further described by taking the image of FIG. 5 as an example. Now, when the pixel data on one line of the portion 63A which is a part of the right end of the mixed area 63 in FIG. 5 is plotted, the result is as shown in FIG. 6, the horizontal axis represents the X coordinate (the horizontal coordinate in FIG. 5), and the vertical axis represents the pixel value at that coordinate.

The curve L1 represents the pixel value on the line at the first timing, and the curve L2 represents the pixel value on the corresponding line at the next timing. Similarly,
The curve L3 indicates the pixel value of the corresponding line at the next timing, and the curve L4 indicates the pixel value of the corresponding line at the next timing. In other words, FIG.
5 shows changes in pixel values on a line corresponding to two timings.

The curve L1 represents the first timing in time. In this state, the foreground 62 has not been imaged yet. Therefore, the curve L1 represents the pixel of the background 61.

On the curve L1, the pixel value is about 75 near the X coordinate 140, but at the X coordinate 145, the pixel value increases to about 130. Thereafter, the pixel value decreases, and around the X coordinate 149, the pixel value becomes approximately 120.
Becomes As the X coordinate increases, the pixel value then increases again, and near the X coordinate 154, the pixel value is approximately one.
It is 60. Thereafter, the pixel value decreases and the X coordinate 1
At around 62, it is about 130. Thereafter, the pixel value becomes about 180 near the X coordinate 165, and the pixel value drops to about 125 again near the X coordinate 170. Thereafter, in the vicinity of the X coordinate 172, the pixel value becomes about 175.
, And thereafter, around the X coordinate 178, the pixel value decreases to about 60. After that, X coordinates 178 to 1
In the section up to 95, the pixel value slightly changes between 60 and 80. Then, at the coordinates further to the right of the vicinity of the X coordinate 195, the pixel value increases again to around 160.

In the curve L2 of the next frame, the pixel value of about 200 is constant up to the vicinity of the X coordinate 145.
The pixel value gradually decreases from the coordinate 145 to the X coordinate 160. At the X coordinate 160, the pixel value is about 125. Subsequent changes are similar to curve L1.

The pixel value of the curve L3 is substantially constant at a pixel value of 200 up to the vicinity of the X coordinate 158.
At around 62, after dropping to about 180, the X coordinate 1
Around 64, the pixel value increases again to about 190. Thereafter, it changes substantially in the same manner as the curve L1.

The pixel value of the curve L4 is a constant pixel value of about 200 from the vicinity of the X coordinate 140 to the vicinity of the X coordinate 170, but the pixel value decreases rapidly from the vicinity of the X coordinate 170 to the vicinity of the X coordinate 180. And about 70 around the X coordinate 170
It has become. Subsequent changes are similar to curve L1.

As described above, the pixel values of the curves L2 to L4 change in the state of the curve L1 in the case of the image of the background 61 only, while the image of the foreground 62 changes with the movement ( (Over time).

That is, as is clear from the comparison between the curve L1 and the curve L2 at the timing immediately after the curve L1, the values of the curves L2 to L4 are almost the same up to the vicinity of the X coordinate 147. The value of the curve L2 is calculated from the vicinity of the X coordinate 147,
3 and L4, which are almost the same as the value of the curve L1 near the X coordinate 159. Subsequent pixel values of the curve L2 at the X coordinate are almost the same as those of the curve L1. That is, from the X coordinate 146 to the X coordinate 1
Curve L in region R1 corresponding to section D1 up to 59
A value of 2 indicates that the tip of the foreground 62 has moved from the left end to the right end of the section D1 in a unit period.

Similarly, in the region R2 corresponding to the section D2 from the X coordinate 159 to the X coordinate 172, the pixel value of the curve L3 at the next timing corresponds to the image in which the front end of the foreground 62 has moved during that time. I have. Further, the X coordinate 17
Region R3 corresponding to section D3 from 2 to X coordinate 184
, The pixel value of the curve L4 indicates that the tip of the foreground 62 has moved in the meantime.

Accordingly, when the value obtained by weighting the pixel value of the curve L1 based on the mixture ratio of the foreground 62 to the background 61 is subtracted from the pixel value of the curve L2 in the section D1, as shown in FIG. A curve L11 is obtained. Since the value corresponding to the background 61 is subtracted from the pixel of the foreground 62 in the mixed area 63, the curve L11 is an image of the foreground on the background with the pixel value 0. Note that the horizontal axis in FIG. 7 represents the position (the left end corresponds to the left end of section D1 in FIG. 6, and the right end corresponds to the right end in section D1 in FIG. 6), and the vertical axis is the extracted foreground pixel. Represents a value.

Similarly, in section D2 of FIG.
When the pixel value of the curve L1 weighted by the mixture ratio is subtracted from the pixel value of 3, the curve L12 in FIG. 7 is obtained. In the section D3 of FIG. 6, the pixel value of the curve L1 weighted by the mixture ratio from the curve L4 Is subtracted, a curve L13 in FIG. 7 is obtained. As shown in FIG. 7, the curve L12 and the curve L13 are curves almost coincident with the curve L11. This means that the foreground 62 is moving at a substantially constant speed over the three unit timing periods, and the foreground pixel value on the black background, that is, the background with the pixel value 0, is correctly corrected by the weighted subtraction. Indicates what was requested.

The above operation will be described focusing on pixels.
As shown in FIG. In the figure, the horizontal axis represents the portion 6
3A represents the X coordinate, and the vertical axis represents the time axis from the top to the bottom. In this example, since the motion amount is 5, exposure is performed within a time corresponding to t1 to t5 within one exposure time (shutter time). b1 to bf represent pixel values of each pixel of the background 61. a1 to a6 are
Represents the pixel value of the foreground 62.

That is, at the timing t1, the pixel a1 of the foreground 62 is placed at the position of the pixels b3 to b8 of the background 61.
At the timing t2, the pixels a1 to a6 of the foreground 62 are moved rightward by one pixel, that is, to the positions of the pixels b4 to b9 of the background 61.

Similarly, as the time advances from timing t3 to timing t5, the pixels a1 to a6 of the foreground 62 sequentially move rightward by one pixel at a time.

In this case, pixel values y1 to yf obtained by averaging the pixels of each line at timings t1 to t5 are as follows:
Pixels obtained as a result of imaging (pixels with motion blur)
Its value is expressed by the following equation.

[0105]

(Equation 1)

Note that y1, y2, yd, ye, and yf are:
These are equal to the background pixels b1, b2, bd, be, and bf, respectively.

By removing the background pixels b1 to bf, the background 61 and the foreground 62 in the mixed area 63 can be separated. That is, a plurality of objects can be separated. Further, the background pixels b1 to bf are assumed to be known, for example, by using the pixel values of the preceding and following shutter times (frames), and the pixels a1 to a
6 can be obtained by solving the above equation using, for example, the least square method. As a result, a foreground image excluding the motion blur can be obtained. That is, distortion in the projected real world information can be reduced. And, by further processing such as resolution creation,
You can create clear images.

In FIG. 4, the deterministic processing is performed. That is, the next processing is performed based on the previous processing on the assumption that the result of the previous processing is correct. It is also possible to perform the processing statistically. FIG.
Represents a processing example in this case.

That is, when performing this statistical processing, in step S21, the CPU 21 acquires image data. This process is similar to the process in step S1 of FIG.

Next, at step S22, the CPU 21
Performs a process of calculating a mixture ratio between the foreground and the background from the image data acquired in step S21. Then, step S2
In 3, the CPU 21 executes a process for separating the background and the foreground based on the mixture ratio obtained in step S22.

Thus, when based on statistical processing,
Since there is no need to perform the process of determining whether or not the object is a boundary as in step S23 of FIG. 4, it is possible to more quickly separate the foreground and the background.

As described above, a clear foreground 62 image can be separated and extracted from an image with blurred motion obtained when an image of the foreground 62 moving in front of the background 61 is captured. .

Next, a more specific signal processing device that performs a process of specifying a region where significant information is buried from the data acquired by the sensor and extracting the buried significant information from the data acquired by the sensor by deterministic processing. This will be described with a simple example. In the following example, the CCD line sensor or CCD area sensor corresponds to the sensor, the area information and the mixing ratio correspond to the significant information, and the mixed foreground and background and the motion blur correspond to the distortion in the mixed area. I do.

FIG. 10 is a block diagram showing the signal processing section 12. As shown in FIG.

It should be noted that it does not matter whether each function of the signal processing unit 12 is realized by hardware or software. That is, each block diagram in this specification may be considered as a hardware block diagram or a functional block diagram using software.

Here, the motion blur refers to the movement of an object to be imaged in the real world and the motion of the sensor 1.
The distortion included in the image corresponding to the moving object, which is caused by the characteristic of the first imaging.

In this specification, an image to be imaged that corresponds to an object in the real world is called an image object.

The input image supplied to the signal processing unit 12 is
The information is supplied to the object extracting unit 101, the area specifying unit 103, the mixture ratio calculating unit 104, and the foreground / background separating unit 105.

The object extracting unit 101 roughly extracts an image object corresponding to a foreground object included in the input image, and supplies the extracted image object to the motion detecting unit 102. The object extraction unit 101
For example, by detecting the outline of the image object corresponding to the foreground object included in the input image, the image object corresponding to the foreground object is roughly extracted.

The object extraction unit 101 roughly extracts an image object corresponding to a background object included in the input image, and supplies the extracted image object to the motion detection unit 102. The object extraction unit 101
For example, the image object corresponding to the background object is roughly extracted from the difference between the input image and the extracted image object corresponding to the foreground object.

Further, for example, the object extracting unit 101
Is configured to roughly extract an image object corresponding to a foreground object and an image object corresponding to a background object from a difference between a background image stored in a background memory provided therein and an input image. It may be.

The motion detecting section 102 calculates the motion vector of the image object corresponding to the coarsely extracted foreground object, for example, by a method such as a block matching method, a gradient method, a phase correlation method, or a pel recursive method. The calculated motion vector and the position information of the motion vector (information for specifying the position of the pixel corresponding to the motion vector) are supplied to the motion-blur extracting unit 106.

The motion vector output from the motion detecting section 102 includes information corresponding to the motion amount v.

Further, for example, the motion detection unit 102 calculates the motion vector for each image object together with the pixel position information for specifying the pixel in the image object by the motion blur adjustment unit 10.
6 may be output.

The amount of movement v is a value representing the change in the position of the image corresponding to the moving object in units of pixel intervals. For example, the image of the object corresponding to the foreground is set to 4
When moving so as to be displayed at a position separated by pixels, the motion amount v of the image of the object corresponding to the foreground is
It is set to 4.

The object extracting unit 101 and the motion detecting unit 102 are used when the motion blur adjusting unit 106 adjusts the amount of motion blur corresponding to a moving object.

The area specifying unit 103 specifies each of the pixels of the input image as one of a foreground area, a background area, and a mixed area, and for each pixel, selects one of the foreground area, the background area, and the mixed area. Information that belongs to
(Referred to as area information) to the mixture ratio calculation unit 104, the foreground / background separation unit 105, and the motion blur adjustment unit 106.

The mixture ratio calculation unit 104 calculates a mixture ratio (hereinafter, referred to as a mixture ratio α) corresponding to the pixels included in the mixture region 63 based on the input image and the region information supplied from the region identification unit 103. The calculated mixture ratio is supplied to the foreground / background separation unit 105.

The mixture ratio α is a value indicating the ratio of a component of an image corresponding to a background object (hereinafter, also referred to as a background component) to a pixel value, as shown in Expression (13) described later.

The foreground / background separation unit 105 is provided with
3 and the mixture ratio calculation unit 104
Input image into a foreground component image composed of only components of the image corresponding to the foreground object (hereinafter also referred to as a foreground component) and a background component image composed of only the background component based on the mixture ratio α supplied from And supplies the foreground component image to the motion-blur adjusting unit 106 and the selecting unit 107. Note that the separated foreground component image may be used as a final output. It is possible to specify only the foreground and the background without considering the conventional mixed area, and obtain a more accurate foreground and background as compared with the separated system.

The motion-blur adjusting unit 106 determines a processing unit indicating one or more pixels included in the foreground component image based on the motion amount v and the area information known from the motion vector. The processing unit is data that specifies a group of pixels to be processed for adjusting the amount of motion blur.

The motion-blur adjusting unit 106 includes the signal processing unit 12
Are included in the foreground component image based on the motion blur adjustment amount input to the foreground component image supplied from the foreground / background separation unit 105, the motion vector supplied from the motion detection unit 102 and its position information, and the processing unit. Adjust the amount of motion blur included in the foreground component image, such as removing motion blur, reducing the amount of motion blur, or increasing the amount of motion blur, and select the foreground component image with the adjusted amount of motion blur Output to the unit 107. The motion vector and its position information may not be used.

The selection unit 107 determines whether the amount of the foreground component image supplied from the foreground / background separation unit 105 and the amount of motion blur supplied from the motion blur adjustment unit 106 are based on a selection signal corresponding to the user's selection, for example. One of the adjusted foreground component images is selected, and the selected foreground component image is output.

Next, an input image supplied to the signal processing section 12 will be described with reference to FIGS.

FIG. 11 is a diagram for explaining image pickup by a sensor. The sensor 11 is, for example, a solid-state image sensor CC
CCD with D (Charge-Coupled Device) area sensor
It is composed of a video camera and the like. The object corresponding to the foreground in the real world moves horizontally between the object corresponding to the background and the sensor 11 in the real world, for example, from left to right in the drawing.

The sensor 11 images the object corresponding to the foreground together with the object corresponding to the background. The sensor 11 outputs the captured image in units of one frame. For example, the sensor 11 outputs an image composed of 30 frames per second. The exposure time of the sensor 11 is 1/3
It can be 0 seconds. The exposure time is a period from the time when the sensor 11 starts to convert the input light to the electric charge to the time when the conversion of the input light to the electric charge is completed. Less than,
The exposure time is also called a shutter time.

FIG. 12 is a diagram for explaining the arrangement of pixels. In FIG. 12, A to I indicate individual pixels. The pixels are arranged on a plane corresponding to the image.
One detection element corresponding to one pixel is arranged on the sensor 11. When the sensor 11 captures an image,
One detection element outputs a pixel value corresponding to one pixel forming the image. For example, the position of the detection element in the X direction corresponds to the position in the horizontal direction on the image, and the position of the detection element in the Y direction corresponds to the position in the vertical direction on the image.

As shown in FIG. 13, for example, a detection element such as a CCD converts input light into electric charges for a period corresponding to the shutter time and accumulates the converted electric charges. The amount of charge is substantially proportional to the intensity of the input light and the time during which the light is input. The detection element converts the charge converted from the input light into a period corresponding to the shutter time,
It adds to the charge already stored. That is, the detection element integrates the input light for a period corresponding to the shutter time, and accumulates an amount of charge corresponding to the integrated light. It can be said that the detection element has an integration effect with respect to time.

The electric charge accumulated in the detecting element is converted into a voltage value by a circuit (not shown), and the voltage value is further converted into a pixel value such as digital data and output. Therefore, the individual pixel values output from the sensor 11 are the result of integrating a certain spatially expanding part of the object corresponding to the foreground or background with respect to the shutter time.
It has values projected into a one-dimensional space.

The signal processing section 12 is provided with
, The significant information buried in the output signal, for example, the mixture ratio α is extracted. Signal processing unit 12
Adjusts the amount of distortion caused by mixing of the foreground image objects themselves, for example, the amount of motion blur. In addition, the signal processing unit 12 adjusts the amount of distortion caused by mixing of the foreground image object and the background image object.

FIG. 14 is a diagram for explaining images obtained by capturing an object corresponding to a moving foreground and an object corresponding to a stationary background. FIG.
(A) shows an object corresponding to a foreground with motion;
An image obtained by capturing an object corresponding to a stationary background is shown. In the example shown in FIG. 14A, the object corresponding to the foreground is moving horizontally from left to right with respect to the screen.

FIG. 14B is a model diagram in which pixel values corresponding to one line of the image shown in FIG. 14A are developed in the time direction. The horizontal direction in FIG.
(A) corresponds to the spatial direction X.

The pixels in the background area have their pixel values composed only of the background components, ie, the image components corresponding to the background objects. Pixels in the foreground area are
The pixel value is composed only of the foreground component, that is, the component of the image corresponding to the foreground object.

The pixel value of the pixel in the mixed area is composed of the background component and the foreground component. Since the pixel value of the mixed area is composed of the background component and the foreground component, it can be said that the mixed area is a distortion area. The mixed area is further classified into a covered background area and an uncovered background area.

The covered background area is a mixed area at a position corresponding to the front end of the foreground object in the traveling direction with respect to the foreground area, and is an area in which the background component is covered by the foreground as time passes. Say.

On the other hand, the uncovered background area is a mixed area at a position corresponding to the rear end of the foreground object in the traveling direction with respect to the foreground area. Means the area where appears.

As described above, the image including the foreground area, the background area, or the covered background area or the uncovered background area is input to the area specifying unit 103,
The input image is input to the mixture ratio calculation unit 104 and the foreground / background separation unit 105.

FIG. 15 is a diagram for explaining the background area, foreground area, mixed area, covered background area, and uncovered background area as described above. In the case of the image shown in FIG. 14, the background area is a stationary part, the foreground area is a moving part, the covered background area of the mixed area is a part that changes from the background to the foreground, The uncovered background area is a part that changes from the foreground to the background.

FIG. 16 shows, in the time direction, pixel values of adjacent pixels arranged in one line in an image obtained by capturing an object corresponding to a stationary foreground and an object corresponding to a stationary background. FIG.
For example, the pixels arranged on one line of the screen can be selected as the pixels arranged adjacently in one column.

The pixel values of F01 to F04 shown in FIG. 16 are the pixel values of the pixels corresponding to the stationary foreground object. The pixel values B01 to B04 shown in FIG. 16 are the pixel values of the pixels corresponding to the stationary background object.

In the vertical direction in FIG. 16, time elapses from top to bottom in the figure. The position on the upper side of the rectangle in FIG. 16 corresponds to the time when the sensor 11 starts converting the input light into electric charge, and the position on the lower side of the rectangle in FIG. This corresponds to the time when the conversion into electric charge ends. That is, the distance from the upper side to the lower side of the rectangle in FIG. 16 corresponds to the shutter time.

Hereinafter, a case where the shutter time and the frame interval are the same will be described as an example.

The horizontal direction in FIG. 16 corresponds to the spatial direction X described in FIG. More specifically, in the example shown in FIG. 16, the distance from the left side of the rectangle described as “F01” to the right side of the rectangle described as “B04” in FIG. 16 is eight times the pixel pitch, That is, it corresponds to the interval between eight consecutive pixels.

When the foreground object and the background object are stationary, the light input to the sensor 11 does not change during the period corresponding to the shutter time.

Here, the period corresponding to the shutter time is 2
Divide into two or more periods of equal length. For example, if the number of virtual divisions is 4, the model diagram shown in FIG. 16 can be represented as the model shown in FIG. The number of virtual divisions is the amount of movement v of the object corresponding to the foreground within the shutter time.
It is set in correspondence with, for example. For example, the motion amount v which is 4,
, The virtual division number is set to 4, and the period corresponding to the shutter time is divided into four.

The uppermost row in the figure corresponds to the first divided period after the shutter is opened. The second row from the top in the figure corresponds to the second divided period from when the shutter has opened. The third row from the top in the figure corresponds to the third divided period from when the shutter has opened. The fourth row from the top in the figure corresponds to the fourth divided period from when the shutter has opened.

Hereinafter, the shutter time divided according to the amount of movement v is also referred to as shutter time / v.

When the object corresponding to the foreground is stationary, the light input to the sensor 11 does not change.
The foreground component F01 / v is equal to a value obtained by dividing the pixel value F01 by the number of virtual divisions. Similarly, when the object corresponding to the foreground is stationary, the foreground component F02 / v is equal to a value obtained by dividing the pixel value F02 by the number of virtual divisions, and the foreground component F03 / v has a pixel value F02.
Equal to the value obtained by dividing 03 by the number of virtual divisions, and the foreground component F04 / v
Is equal to the value obtained by dividing the pixel value F04 by the number of virtual divisions.

When the object corresponding to the background is stationary, the light input to the sensor 11 does not change.
The background component B01 / v is equal to a value obtained by dividing the pixel value B01 by the number of virtual divisions. Similarly, when the object corresponding to the background is stationary, the background component B02 / v is equal to the value obtained by dividing the pixel value B02 by the virtual division number, and B03 / v is the pixel value B03 obtained by dividing the pixel value B03 by the virtual division number. B04 / v is equal to the value obtained by dividing the pixel value B04 by the virtual division number.

That is, when the object corresponding to the foreground is stationary, the light corresponding to the object of the foreground input to the sensor 11 does not change during the period corresponding to the shutter time. The foreground component F01 / v corresponding to the shutter time / v, the second foreground component F01 / v corresponding to the shutter time / v when the shutter is opened, and the third foreground component F01 / v corresponding to the shutter time / v.
The foreground component F01 / v corresponding to v has the same value as the fourth foreground component F01 / v corresponding to the shutter time / v after the shutter is opened. F02 / v to F04 / v also have the same relationship as F01 / v.

When the object corresponding to the background is stationary, the light corresponding to the background object input to the sensor 11 does not change during the period corresponding to the shutter time.
The background component B01 / v corresponding to / v
The background component B01 / v corresponding to the second shutter time / v
And the third background component B01 / v corresponding to the shutter time / v after the shutter is opened, and the fourth background component B
The background component B01 / v corresponding to the shutter time / v has the same value. B02 / v to B04 / v also have the same relationship.

Next, the case where the object corresponding to the foreground moves and the object corresponding to the background is stationary will be described.

FIG. 18 is a model diagram in which pixel values of pixels on one line including the covered background area are developed in the time direction when the object corresponding to the foreground moves rightward in the figure. is there. In FIG. 18, the motion amount v of the foreground is 4. Since one frame is a short time, the object corresponding to the foreground is a rigid body,
It can be assumed that they are moving at a constant speed. In FIG. 18, the image of the object corresponding to the foreground moves so as to be displayed four pixels to the right in the next frame with reference to a certain frame.

In FIG. 18, the leftmost pixel to the fourth pixel from the left belong to the foreground area. In FIG. 18, the fifth to seventh pixels from the left belong to the mixed area that is the covered background area. FIG.
In, the rightmost pixel belongs to the background area.

Since the object corresponding to the foreground moves so as to cover the object corresponding to the background with the passage of time, the components included in the pixel values of the pixels belonging to the covered background area correspond to the shutter time. At some point during the period, the background component changes to the foreground component.

For example, in FIG. 18, pixel values indicated by thick frames
M is represented by equation (11).

M = B02 / v + B02 / v + F07 / v + F06 / v (11)

For example, since the fifth pixel from the left contains a background component corresponding to one shutter time / v and a foreground component corresponding to three shutter times / v, the fifth pixel from the left
The mixture ratio α of the second pixel is 1/4. Since the sixth pixel from the left includes a background component corresponding to two shutter times / v and a foreground component corresponding to two shutter times / v, the mixture ratio α of the sixth pixel from the left is , 1/2. Since the seventh pixel from the left includes a background component corresponding to three shutter times / v and a foreground component corresponding to one shutter time / v, the mixture ratio α of the seventh pixel from the left is , 3/4.

Since it can be assumed that the object corresponding to the foreground is a rigid body and the foreground image moves at a constant speed so as to be displayed four pixels to the right in the next frame, for example, the fourth object from the left in FIG. The foreground component F07 / v of the pixel of the first shutter time / v after the shutter is opened is shown in FIG.
8 is equal to the foreground component of the fifth pixel from the left corresponding to the second shutter time / v from when the shutter has opened. Similarly, the foreground component F07 / v is the foreground component of the sixth pixel from the left in FIG. 18 corresponding to the third shutter time / v from when the shutter has opened, and the seventh pixel from the left in FIG. Of the pixel of
It is equal to the foreground component corresponding to the fourth shutter time / v after the shutter is opened.

Since it can be assumed that the object corresponding to the foreground is a rigid body and the foreground image moves at a constant speed so as to be displayed four pixels to the right in the next frame, for example, the third object from the left in FIG. The foreground component F06 / v of the pixel of the first shutter time / v from when the shutter has opened is shown in FIG.
It is equal to the foreground component of the fourth pixel from the left, corresponding to the second shutter time / v from when the shutter has opened. Similarly, the foreground component F06 / v is the fifth pixel from the left in FIG. 18, the foreground component corresponding to the third shutter time / v from when the shutter has opened, and the sixth pixel from the left in FIG. Of the pixel of
It is equal to the foreground component corresponding to the fourth shutter time / v after the shutter is opened.

Since it can be assumed that the object corresponding to the foreground is a rigid body and the foreground image moves at a constant speed so as to be displayed four pixels to the right in the next frame, for example, the second object from the left in FIG. The foreground component F05 / v of the pixel of the first shutter time / v from when the shutter has opened is shown in FIG.
It is equal to the foreground component of the third pixel from the left corresponding to the second shutter time / v from when the shutter has opened. Similarly, the foreground component F05 / v is the foreground component of the fourth pixel from the left in FIG. 18 corresponding to the third shutter time / v from when the shutter has opened, and the fifth foreground component F05 / v in FIG. Of the pixel of
It is equal to the foreground component corresponding to the fourth shutter time / v after the shutter is opened.

Since it can be assumed that the object corresponding to the foreground is a rigid body and the foreground image moves at a constant speed so as to be displayed four pixels to the right in the next frame, for example, the leftmost pixel in FIG. The foreground component F04 / v of the first shutter time / v from when the shutter is opened is the foreground component of the second pixel from the left in FIG. 18 corresponding to the second shutter time / v from when the shutter is opened. be equivalent to. Similarly, the foreground component F04 / v is the third pixel from the left in FIG.
The foreground component corresponding to the third shutter time / v after the shutter is opened, and the foreground component corresponding to the fourth shutter time / v of the fourth pixel from the left in FIG. , Respectively.

Since the foreground area corresponding to the moving object includes the motion blur as described above, it can be said that the area is a distortion area.

FIG. 19 is a model diagram in which the pixel values of the pixels on one line including the uncovered background area are developed in the time direction when the foreground moves to the right in the figure. In FIG. 19, the foreground motion amount v
Is 4. Since one frame is a short time, it can be assumed that the object corresponding to the foreground is rigid and moves at a constant speed. In FIG. 19, the image of the object corresponding to the foreground moves to the right by four pixels in the next frame with reference to a certain frame.

In FIG. 19, the leftmost pixel to the fourth pixel from the left belong to the background area. In FIG. 19, the fifth through seventh pixels from the left belong to the mixed area that is the uncovered background. FIG.
In, the rightmost pixel belongs to the foreground area.

Since the object corresponding to the foreground, which covered the object corresponding to the background, has been moved so as to be removed from the front of the object corresponding to the background over time, the pixels belonging to the uncovered background area At some point during the period corresponding to the shutter time, the component included in the value is changed from the foreground component to the background component.

For example, in FIG. 19, a pixel value with a bold frame
M ′ is represented by equation (12).

M ′ = F02 / v + F01 / v + B26 / v + B26 / v (12)

For example, since the fifth pixel from the left contains a background component corresponding to three shutter times / v and a foreground component corresponding to one shutter time / v, the fifth pixel from the left
The mixture ratio α of the third pixel is 3/4. Since the sixth pixel from the left includes a background component corresponding to two shutter times / v and a foreground component corresponding to two shutter times / v, the mixture ratio α of the sixth pixel from the left is , 1/2. Since the seventh pixel from the left includes a background component corresponding to one shutter time / v and a foreground component corresponding to three shutter times / v, the mixture ratio α of the seventh pixel from the left is , 1/4.

When formulas (11) and (12) are generalized, the pixel value M is expressed by formula (13).

[0181]

(Equation 2) Here, α is a mixture ratio. B is a pixel value of the background, and Fi / v is a foreground component.

Since it can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed, and the amount of movement v is 4, for example, the shutter of the fifth pixel from the left in FIG. First foreground component of shutter time / v F01 /
v is equal to the foreground component of the sixth pixel from the left in FIG. 19 corresponding to the second shutter time / v from when the shutter has opened. Similarly, F01 / v is the foreground component of the seventh pixel from the left in FIG. 19 corresponding to the third shutter time / v from when the shutter has opened, and the eighth pixel from the left in FIG. ,
It is equal to the foreground component corresponding to the fourth shutter time / v after the shutter is opened.

Since it can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed, and the number of virtual divisions is 4, for example, the shutter of the sixth pixel from the left in FIG. First foreground component F0 of shutter time / v
2 / v is equal to the foreground component of the seventh pixel from the left in FIG. 19 corresponding to the second shutter time / v from when the shutter has opened. Similarly, the foreground component F02 / v is equal to the foreground component of the eighth pixel from the left in FIG. 19 corresponding to the third shutter time / v from when the shutter has opened.

Since it can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed, and the movement amount v is 4, for example, the shutter of the seventh pixel from the left in FIG. First foreground component of shutter time / v F03 /
v is equal to the foreground component of the eighth pixel from the left in FIG. 19 corresponding to the second shutter time / v from when the shutter has opened.

In the description of FIGS. 17 to 19, the number of virtual divisions is four, but the number of virtual divisions corresponds to the amount of motion v. The movement amount v generally corresponds to the moving speed of the object corresponding to the foreground. For example, when the object corresponding to the foreground is moving so as to be displayed four pixels to the right in the next frame with respect to a certain frame, the motion amount v is set to 4. The virtual division number is set to 4 corresponding to the movement amount v. Similarly, for example, when the object corresponding to the foreground is moving so as to be displayed 6 pixels to the left in the next frame with respect to a certain frame, the motion amount v is set to 6, and the number of virtual divisions is , 6.

FIG. 20 and FIG.
The relationship between a mixed area including a foreground area, a background area, a covered background area, or an uncovered background area, and foreground components and background components corresponding to the divided shutter times is shown.

FIG. 20 shows an example in which pixels of a foreground area, a background area, and a mixed area are extracted from an image including a foreground corresponding to an object moving in front of a stationary background. In the example shown in FIG. 20, the object corresponding to the foreground is moving horizontally with respect to the screen.

The frame # n + 1 is a frame next to the frame #n, and the frame # n + 2 is a frame next to the frame # n + 1.

The pixels of the foreground area, the background area, and the mixed area, which are extracted from any of the frames #n to # n + 2, are extracted. FIG. 21 shows a model developed in the time direction.

The pixel value of the foreground area is composed of four different foreground components corresponding to the shutter time / v period because the object corresponding to the foreground moves. For example, the leftmost pixel of the pixels in the foreground area shown in FIG. 21 is composed of F01 / v, F02 / v, F03 / v, and F04 / v. That is, pixels in the foreground area include motion blur.

Since the object corresponding to the background is stationary, the light corresponding to the background input to the sensor 11 does not change during the period corresponding to the shutter time. In this case, the pixel value of the background area does not include motion blur.

The pixel values of the pixels belonging to the mixed area composed of the covered background area or the uncovered background area are composed of a foreground component and a background component.

Next, when the image corresponding to the object is moving, the pixel values of the pixels that are adjacently arranged in one row in a plurality of frames and located at the same position on the frame are determined in the time direction. The following is an explanation of the model developed in. For example, when an image corresponding to an object is moving horizontally with respect to the screen, pixels arranged on one line of the screen can be selected as pixels arranged adjacently in one row.

FIG. 22 shows three adjacent pixels of three frames of an image obtained by capturing an object corresponding to a stationary background, the pixels being located at the same position on the frame. FIG. 4 is a model diagram in which values are developed in a time direction. Frame #n is the next frame after frame # n-1, and frame # n + 1 is the next frame after frame #n. Other frames are similarly referred to.

The pixel values B01 to B12 shown in FIG. 22 are the pixel values of the pixels corresponding to the stationary background object. Since the object corresponding to the background is stationary, the pixel values of the corresponding pixels do not change in frames # n-1 to n + 1. For example, the pixel in frame #n and the pixel in frame # n + 1 corresponding to the position of the pixel having the pixel value of B05 in frame # n-1 each have the pixel value of B05.

FIG. 23 is a diagram showing three pixels adjacent to one another in three frames of an image obtained by capturing an object corresponding to a foreground moving to the right in the figure together with an object corresponding to a stationary background. FIG. 4 is a model diagram in which pixel values of pixels at the same position on a frame are developed in the time direction. The model shown in FIG. 23 includes a covered background area.

In FIG. 23, it can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed, and the foreground image moves so as to be displayed four pixels to the right in the next frame. The quantity v is 4, and the number of virtual divisions is 4.

For example, the foreground component of the leftmost pixel of frame # n-1 in FIG. 23 corresponding to the first shutter time / v from when the shutter has opened is F12 / v, and the second component from the left in FIG. The foreground component of the pixel of the second shutter time / v after the shutter is opened also becomes F12 / v. The foreground component of the third pixel from the left in FIG. 23 at the third shutter time / v from when the shutter has opened, and the fourth pixel from the left in FIG.
The foreground component of the fourth shutter time / v after the shutter is opened is F12 / v.

The foreground component of the leftmost pixel in frame # n-1 in FIG. 23 corresponding to the second shutter time / v from when the shutter has opened is F11 / v, which is the second to the left in FIG. The foreground component of the pixel at the third shutter time / v from when the shutter has opened is also F11 / v. The foreground component of the third pixel from the left in FIG. 23 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is F11 / v.

The foreground component of the leftmost pixel in frame # n-1 in FIG. 23 at the third shutter time / v from when the shutter has opened is F10 / v, which is the second to the left in FIG. The foreground component of the pixel at the fourth shutter time / v from when the shutter has opened is also F10 / v. The fourth shutter time of the leftmost pixel of frame # n-1 in FIG. 23 from when the shutter has opened
The foreground component of / v is F09 / v.

Since the object corresponding to the background is stationary, the background component of the second pixel from the left of frame # n-1 in FIG. 23 corresponding to the first portion of the shutter time / v from when the shutter has opened is B01 / v. The background components of the third pixel from the left of frame # n-1 in FIG. 23 corresponding to the first and second portions of the shutter time / v from when the shutter has opened are B02 / v. FIG.
The background component of the fourth pixel from the left of the middle frame # n-1 corresponding to the first to third shutter time / v from when the shutter has opened is B03 / v.

In frame # n-1 in FIG. 23, the leftmost pixel belongs to the foreground area, and the second to fourth pixels from the left belong to the mixed area which is the covered background area.

The fifth through twelfth pixels from the left of frame # n-1 in FIG. 23 belong to the background area, and their pixel values are B04 through B11, respectively.

The first through fifth pixels from the left of frame #n in FIG. 23 belong to the foreground area. Frame #n
The foreground component of shutter time / v in the foreground area of
One of F05 / v to F12 / v.

It can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed. Since the foreground image moves so as to be displayed four pixels to the right in the next frame, frame #n in FIG. The foreground component of the fifth pixel from the left of the first shutter time / v from when the shutter has opened is
F12 / v, the foreground component of the sixth pixel from the left in FIG. 23 corresponding to the second shutter time / v from when the shutter has opened is also
It becomes F12 / v. The foreground component of the seventh pixel from the left in FIG. 23 corresponding to the third shutter time / v from when the shutter has opened, and the fourth shutter time since the shutter opening of the eighth pixel from the left in FIG. 23 The foreground component of / v is F12 / v.

The foreground component of the fifth pixel from the left of frame #n in FIG. 23 corresponding to the second shutter time / v from when the shutter has opened is F11 / v, and the sixth pixel from the left in FIG. 23 is F11 / v. The foreground component of the pixel at the third shutter time / v from when the shutter has opened is also F11 / v. The foreground component of the seventh pixel from the left in FIG. 23 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is F11 / v.

The foreground component of the fifth pixel from the left of frame #n in FIG. 23 corresponding to the third shutter time / v from when the shutter has opened is F10 / v, and the sixth pixel from the left in FIG. 23 is F10 / v. The foreground component of the pixel at the fourth shutter time / v from when the shutter has opened is also F10 / v. The foreground component of the fifth pixel from the left of frame #n in FIG. 23 corresponding to the fourth shutter time / v from when the shutter has opened is F09 / v.

Since the object corresponding to the background is stationary, the background component of the sixth pixel from the left of frame #n in FIG. 23 corresponding to the first portion of the shutter time / v from when the shutter has opened is B05 / v Becomes 23 from the left of frame #n in FIG.
The background component of the first pixel at the first and second shutter time / v from when the shutter has opened is B06 / v. The background components of the eighth pixel from the left of frame #n in FIG. 23 corresponding to the first to third portions of the shutter time / v from when the shutter has opened are:
B07 / v.

In frame #n in FIG. 23, the sixth through eighth pixels from the left belong to a mixed area which is a covered background area.

[0210] The ninth to twelfth pixels from the left of frame #n in Fig. 23 belong to the background area, and the pixel value is
B08 to B11, respectively.

[0211] The first through ninth pixels from the left of frame # n + 1 in Fig. 23 belong to the foreground area. flame
The foreground component of the shutter time / v in the foreground area of # n + 1 is one of F01 / v to F12 / v.

It can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed. Since the foreground image moves so as to be displayed four pixels to the right in the next frame, frame #n in FIG. Of the ninth pixel from the left of +1
The foreground component of the first shutter time / v after the shutter is opened is F12 / v, and the foreground component of the tenth pixel from the left in FIG. It becomes F12 / v. The foreground component of the eleventh pixel from the left in FIG. 23 corresponding to the third shutter time / v from when the shutter has opened, and the fourth shutter time since the shutter opening of the twelfth pixel from the left in FIG. 23 The foreground component of / v is F12 / v.

The foreground component of the ninth pixel from the left of frame # n + 1 in FIG. 23 corresponding to the second portion of the shutter time / v from when the shutter has opened is F11 / v, and as shown in FIG. From 1
The foreground component of the 0th pixel at the third shutter time / v from when the shutter has opened is also F11 / v. The foreground component of the eleventh pixel from the left in FIG. 23 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is F11 / v.

The foreground component of the ninth pixel from the left of frame # n + 1 in FIG. 23 corresponding to the third portion of the shutter time / v from when the shutter has opened is F10 / v, from the left in FIG. 4th shutter time / v after shutter opened for 10th pixel
Is also F10 / v. Frame # n + in FIG.
The foreground component of the ninth pixel from the left in FIG. 1 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is F09 / v.

Since the object corresponding to the background is stationary, the background component of the tenth pixel from the left of frame # n + 1 in FIG. 23 corresponding to the first portion of the shutter time / v from when the shutter has opened is B09 / v. The background components of the eleventh pixel from the left of frame # n + 1 in FIG. 23 corresponding to the first and second portions of the shutter time / v from when the shutter has opened are B10 / v. The background components of the twelfth pixel from the left of frame # n + 1 in FIG. 23 corresponding to the first through third portions of the shutter time / v from when the shutter has opened are B11 / v.

In frame # n + 1 in FIG. 23, the tenth through twelfth pixels from the left correspond to the mixed area which is the covered background area.

FIG. 24 is a model diagram of an image in which foreground components are extracted from the pixel values shown in FIG.

FIG. 25 is a view showing three adjacent pixels of three frames of an image obtained by capturing a foreground corresponding to an object moving to the right in the figure together with a stationary background. FIG. 10 is a model diagram in which pixel values of pixels at the same position are developed in the time direction. In FIG.
An uncovered background area is included.

In FIG. 25, it can be assumed that the object corresponding to the foreground is a rigid body and is moving at a constant speed. Since the object corresponding to the foreground is moving so as to be displayed on the right side by four pixels in the next frame, the motion amount v is 4.

For example, the first shutter time / v of the leftmost pixel of frame # n-1 in FIG.
Of the foreground is F13 / v, and the second shutter time / v of the second pixel from the left in FIG.
Is also F13 / v. The third shutter time / v from when the shutter has opened for the third pixel from the left in FIG.
25, and the foreground component of the fourth pixel from the left in FIG. 25 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is F13 / v.

The foreground component of the second pixel from the left of frame # n-1 in FIG. 25 corresponding to the first shutter time / v from when the shutter has opened is F14 / v, and the third pixel from the left in FIG. 25 is F14 / v. The foreground component of the pixel of the second shutter time / v after the shutter is opened also becomes F14 / v. The foreground component of the third pixel from the left in FIG. 25 corresponding to the first portion of the shutter time / v from when the shutter has opened is F15 / v.

Since the object corresponding to the background is stationary, the leftmost pixel of frame # n-1 in FIG.
Second to fourth shutter time / v after shutter opens
Is B25 / v. Frame # n- in FIG.
The background components of the second pixel from the left, the third and fourth portions of the shutter time / v from when the shutter has opened are B26 / v
Becomes The background component of the third pixel from the left of frame # n-1 in FIG. 25 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is B27 / v.

In the frame # n-1 in FIG. 25, the leftmost pixel to the third pixel belong to the mixed area which is the uncovered background area.

The fourth through twelfth pixels from the left of frame # n-1 in FIG. 25 belong to the foreground area. The foreground component of the frame is one of F13 / v to F24 / v.

The leftmost pixel to the fourth pixel from the left of frame #n in FIG. 25 belong to the background area, and the pixel value is
B25 to B28, respectively.

It can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed. Since the foreground image moves so as to be displayed four pixels to the right in the next frame, frame #n in FIG. The foreground component of the fifth pixel from the left of the first shutter time / v from when the shutter has opened is
F13 / v, the foreground component of the sixth pixel from the left in FIG. 25 corresponding to the second shutter time / v from when the shutter has opened is also
It becomes F13 / v. The foreground component of the seventh pixel from the left in FIG. 25 at the third shutter time / v from when the shutter has opened, and the fourth shutter time of the eighth pixel from the left in FIG. 25 having the shutter opened. The foreground component of / v is F13 / v.

The foreground component of the sixth pixel from the left of frame #n in FIG. 25 corresponding to the first shutter time / v from when the shutter has opened is F14 / v, and the seventh pixel from the left in FIG. 25 is F14 / v. The foreground component of the second shutter time / v after the shutter is opened also becomes F14 / v. The foreground component of the eighth pixel from the left in FIG. 25 corresponding to the first shutter time / v from when the shutter has opened is F15 / v.

Since the object corresponding to the background is stationary, the second to fourth shutter times of the fifth pixel from the left of frame #n in FIG.
The background component of / v is B29 / v. Frame # in FIG. 25
The background components of the sixth pixel from the left of n corresponding to the third and fourth shutter time / v from when the shutter has opened are B30 / v. Of the seventh pixel from the left of frame #n in FIG.
The background component of the fourth shutter time / v from when the shutter has opened is B31 / v.

In frame #n in FIG. 25, 5
The seventh through seventh pixels belong to a mixed area that is an uncovered background area.

The eighth through twelfth pixels from the left of frame #n in FIG. 25 belong to the foreground area. flame
The value corresponding to the shutter time / v period in the foreground area #n is one of F13 / v to F20 / v.

The leftmost pixel to the eighth pixel from the left of frame # n + 1 in FIG. 25 belong to the background area, and have pixel values B25 to B32, respectively.

It can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed. Since the foreground image moves so as to be displayed four pixels to the right in the next frame, frame #n in FIG. Of the ninth pixel from the left of +1
The foreground component of the first shutter time / v after the shutter is opened is F13 / v, and the foreground component of the tenth pixel from the left in FIG. 25 at the second shutter time / v after the shutter is opened is also F13 / v. It becomes F13 / v. The foreground component of the eleventh pixel from the left in FIG. 25 corresponding to the third shutter time / v from when the shutter has opened, and the twelfth pixel from the left in FIG. 25 having the fourth shutter time since the shutter has opened. The foreground component of / v is F13 / v.

The foreground component of the tenth pixel from the left of frame # n + 1 in FIG. 25 corresponding to the first shutter time / v from when the shutter has opened is F14 / v, and the eleventh pixel from the left in FIG. 25 is F14 / v. The foreground component of the pixel of the second shutter time / v after the shutter is opened also becomes F14 / v. The foreground component of the twelfth pixel from the left in FIG. 25 corresponding to the first shutter time / v from when the shutter has opened is F15 / v.

Since the object corresponding to the background is stationary, the background of the ninth pixel from the left of frame # n + 1 in FIG. 25 corresponding to the second to fourth portions of the shutter time / v from when the shutter has opened. Is B33 / v. The shutter of the tenth pixel from the left of frame # n + 1 in FIG.
The background component of the 4th and 4th shutter time / v is B3
4 / v. The background component of the eleventh pixel from the left of frame # n + 1 in FIG. 25 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is B35 / v.

In frame # n + 1 in FIG. 25, the ninth to eleventh pixels from the left belong to the mixed area which is the uncovered background area.

The twelfth pixel from the left of frame # n + 1 in FIG. 25 belongs to the foreground area. The foreground components of the shutter time / v in the foreground area of frame # n + 1 are F13 / v to F13 / v
16 / v.

FIG. 26 is a model diagram of an image in which foreground components are extracted from the pixel values shown in FIG.

Returning to FIG. 10, the area specifying unit 103 uses the pixel values of a plurality of frames to set a flag indicating that the pixel belongs to the foreground area, background area, covered background area, or uncovered background area for each pixel. Is supplied to the mixture ratio calculation unit 104 and the motion blur adjustment unit 106 as region information.

The mixture ratio calculation unit 104 calculates the mixture ratio α for each pixel of the pixels included in the mixed region based on the pixel values of a plurality of frames and the region information, and uses the calculated mixture ratio α for the foreground / background separation. To the unit 105.

The foreground / background separation unit 105 extracts a foreground component image consisting of only foreground components based on the pixel values of a plurality of frames, area information, and the mixture ratio α, and supplies the extracted foreground component image to the motion blur adjustment unit 106. .

The motion-blur adjusting unit 106 controls the foreground component image supplied from the foreground / background separating unit 105 and the motion detecting unit 102.
Motion vector supplied from the
The amount of motion blur included in the foreground component image is adjusted on the basis of the area information supplied from, and a foreground component image in which the amount of motion blur is adjusted is output.

The processing for adjusting the amount of motion blur by the signal processing unit 12 will be described with reference to the flowchart in FIG. In step S101, the area specifying unit 103
Based on the input image, an area specifying process for generating area information indicating whether the pixel belongs to a foreground area, a background area, a covered background area, or an uncovered background area for each pixel of the input image is executed. The details of the area specifying process will be described later with reference to the flowchart in FIG. The region specifying unit 103 supplies the generated region information to the mixture ratio calculation unit 104.

In step S101, based on the input image, the area specifying unit 103 determines, for each pixel of the input image, a foreground area, a background area, or a mixed area (covered background area or uncovered background area). May be generated. In this case, the foreground / background separation unit 105 and the motion blur adjustment unit 106 determine whether the mixed area is a covered background area or an uncovered background area based on the direction of the motion vector. For example, when the foreground area, the mixed area, and the background area are sequentially arranged in correspondence with the direction of the motion vector, the mixed area is determined as a covered background area, and the background area is determined according to the direction of the motion vector. When the area, the mixed area, and the foreground area are arranged in this order, the mixed area is determined to be an uncovered background area.

In step S102, the mixture ratio calculator 104 calculates a mixture ratio α for each pixel included in the mixed region based on the input image and the region information. Details of the mixture ratio calculation process will be described later with reference to the flowchart in FIG. The mixture ratio calculator 104 supplies the calculated mixture ratio α to the foreground / background separator 105.

In step S103, the foreground / background separation unit 105 extracts a foreground component from the input image based on the region information and the mixture ratio α, and supplies the extracted foreground component image to the motion blur adjustment unit 106.

[0246] In step S104, the motion-blur adjusting unit 106 determines which of uncovered background area, foreground area, and covered background area is a continuous pixel lined up in the motion direction based on the motion vector and the area information. A processing unit indicating a position on the image of the object belonging to the genus is generated, and the amount of motion blur included in the foreground component corresponding to the processing unit is adjusted. Details of the process of adjusting the amount of motion blur will be described later with reference to the flowchart in FIG.

In step S105, the signal processing unit 1
Step 2 determines whether or not the processing has been completed for the entire screen. If it is determined that the processing has not been completed for the entire screen, the process advances to step S104 to perform a motion on the foreground component corresponding to the processing unit. The process of adjusting the amount of blur is repeated.

If it is determined in step S106 that the processing has been completed for the entire screen, the processing ends.

As described above, the signal processing section 12 can separate the foreground and the background and adjust the amount of motion blur included in the foreground. That is, the signal processing unit 12 can adjust the amount of motion blur included in the sample data that is the pixel value of the foreground pixel.

Hereinafter, the area specifying unit 103 and the mixture ratio calculating unit 1
04, foreground / background separation unit 105, and motion blur adjustment unit 1
An example of each of the components 06 will be described.

FIG. 28 is a block diagram showing an example of the structure of the area specifying unit 103. The frame memory 121 stores the input image in frame units. When the processing target is the frame #n, the frame memory 121 outputs a frame # n-2 which is a frame immediately before the frame #n, a frame # n-1 which is a frame immediately before the frame #n, flame
#n, frame # n + which is one frame after frame #n
1 and frame # n + 2, which is two frames after frame #n.

The still / moving judgment unit 122-1 determines the pixel value of the pixel of frame # n + 2 at the same position as the position of the pixel as the target of specifying the region of frame #n, and the pixel value of frame #n. The pixel value of the pixel of frame # n + 1 at the same position as the position of the pixel which is the target of region identification on the image is stored in the frame memory
21 to calculate the absolute value of the difference between the read pixel values. The static / movement determining unit 122-1 determines whether the absolute value of the difference between the pixel value of the frame # n + 2 and the pixel value of the frame # n + 1 is greater than a preset threshold Th, When it is determined that the absolute value of the difference is larger than the threshold Th, a still / moving determination indicating a motion is supplied to the area determination unit 123-1. Frame #n
When it is determined that the absolute value of the difference between the pixel value of the pixel of +2 and the pixel value of the pixel of frame # n + 1 is equal to or smaller than the threshold Th, the static / movement determining unit 122-1 determines whether or not the static The judgment is made to the area judgment unit 1
23-1.

The still / moving judgment unit 122-2 determines the pixel value of the pixel of frame # n + 1 at the same position as the position of the pixel for specifying the region of frame #n on the image and the pixel value of frame #n. The pixel value of the target pixel is read from the frame memory 121, and the absolute value of the difference between the pixel values is calculated. The static / moving determining unit 122-2 determines whether the absolute value of the difference between the pixel value of the frame # n + 1 and the pixel value of the frame #n is larger than a preset threshold Th. If it is determined that the absolute value of the difference is larger than the threshold Th, a still / moving determination indicating a motion is supplied to the region determination unit 123-1 and the region determination unit 123-2. When it is determined that the absolute value of the difference between the pixel value of the pixel of frame # n + 1 and the pixel value of the pixel of frame #n is equal to or smaller than the threshold Th, the still / moving determination unit 122-2 indicates that the camera is still. The static / movement determination is supplied to the region determination unit 123-1 and the region determination unit 123-2.

The still / moving judgment section 122-3 is located at the same position as the pixel value of the pixel which is the target of specifying the region of the frame #n and the position of the pixel which is the target of specifying the region of the frame #n on the image. The pixel value of the pixel of frame # n-1 is stored in frame memory 1
21 to calculate the absolute value of the difference between the pixel values.
The still / moving determination unit 122-3 determines whether the absolute value of the difference between the pixel value of the frame #n and the pixel value of the frame # n-1 is greater than a preset threshold Th. If the absolute value of the difference is determined to be larger than the threshold Th, the static / movement determination indicating the motion is performed by the region determination unit 123-2 and the region determination unit 123-2.
-3. Pixel value of pixel of frame #n and frame
When it is determined that the absolute value of the difference between the pixel value of the pixel # n-1 and the pixel value is equal to or smaller than the threshold Th, the static / moving determination unit 122-3 performs a static / moving determination indicating stillness by the region determining unit 123-2. And to the area determination unit 123-3.

The static / movement determining unit 122-4 determines the pixel value of the pixel of the frame # n-1 at the same position as the pixel on the image, which is the object of specifying the region of the frame #n, and the pixel value of the frame #n. The pixel value of the pixel of frame # n-2 at the same position as the position of the pixel to be specified in the image is stored in the frame memory 1.
21 to calculate the absolute value of the difference between the pixel values.
The static / movement determining unit 122-4 determines whether the absolute value of the difference between the pixel value of the frame # n-1 and the pixel value of the frame # n-2 is larger than a preset threshold Th, When it is determined that the absolute value of the difference between the pixel values is greater than the threshold Th, a still / moving determination indicating a motion is supplied to the region determination unit 123-3. Frame #n
When it is determined that the absolute value of the difference between the pixel value of the pixel of −1 and the pixel value of the pixel of frame # n−2 is equal to or smaller than the threshold Th, the static / movement determining unit 122-4 determines that the static value indicates static. The motion determination is supplied to the area determination unit 123-3.

The area judging section 123-1 comprises the static / moving judging section 12
When the still / movement determination supplied from 2-1 indicates stillness and the still / movement determination supplied from still / movement determination section 122-2 indicates movement, a pixel which is a target of region identification in frame #n Is determined to belong to the uncovered background area, and the uncovered background area determination flag corresponding to the pixel whose area is determined is set to “1” indicating that the pixel belongs to the uncovered background area.

The area determining unit 123-1 is a static motion determining unit 12
When the still / movement determination supplied from 2-1 indicates a motion or the still / movement determination supplied from the still / movement determination unit 122-2 indicates stillness, the region is to be specified in the frame #n. It is determined that the pixel does not belong to the uncovered background area, and “0” indicating that the pixel does not belong to the uncovered background area is set in the uncovered background area determination flag corresponding to the pixel whose area is determined.
Set.

The area judging unit 123-1 outputs “
The uncovered background area determination flag in which “1” or “0” is set is supplied to the determination flag storage frame memory 124.

The area determination section 123-2 is a static / motion determination section 12.
When the still / movement determination supplied from 2-2 indicates stillness and the still / movement determination supplied from still / movement determination unit 122-3 indicates stillness, a pixel which is a target of region identification in frame #n Is determined to belong to the still area, and “1” indicating that the pixel belongs to the still area is set in the still area determination flag corresponding to the pixel whose area is determined.

The area determination section 123-2 is a
When the static / movement determination supplied from 2-2 indicates a motion, or when the static / movement determination supplied from the static / movement determination unit 122-3 indicates a motion, the region is a target of area identification in frame #n. It is determined that the pixel does not belong to the still area, and “0” indicating that the pixel does not belong to the still area is set in the still area determination flag corresponding to the pixel whose area is determined.

The area determination unit 123-2 determines "
The still area determination flag in which 1 ”or“ 0 ”is set is supplied to the determination flag storage frame memory 124.

The area determination section 123-2 is a
When the still / movement determination supplied from 2-2 indicates a movement and the still / movement determination supplied from the still / movement determination unit 122-3 indicates a movement, the pixel which is the target of the region identification in the frame #n Is determined to belong to the motion area, and “1” indicating that the pixel belongs to the motion area is set in the motion area determination flag corresponding to the pixel whose area is determined.

The area determination section 123-2 is a static / motion determination section 12.
When the static / movement determination supplied from 2-2 indicates stillness, or when the static / movement determination supplied from static / movement determination unit 122-3 indicates stillness, it is an area identification target in frame #n. It is determined that the pixel does not belong to the motion area, and “0” indicating that the pixel does not belong to the motion area is set in the motion area determination flag corresponding to the pixel whose area is determined.

The area determining unit 123-2 determines "
The motion area determination flag in which 1 ”or“ 0 ”is set is supplied to the determination flag storage frame memory 124.

The area determination section 123-3 is a static / motion determination section 12.
When the still / movement determination supplied from 2-3 indicates a motion and the still / movement determination supplied from the still / movement determination unit 122-4 indicates stillness, the pixel which is the target of the region identification in the frame #n Is determined to belong to the covered background area, and “1” indicating that the pixel belongs to the covered background area is set in the covered background area determination flag corresponding to the pixel whose area is determined.

The area determination section 123-3 is a static / motion determination section 12.
When the still / movement determination supplied from 2-3 indicates stillness or when the still / movement determination supplied from the still / movement determination unit 122-4 indicates movement, the region is a target for area identification in frame #n. It is determined that the pixel does not belong to the covered background area, and “0” indicating that the pixel does not belong to the covered background area is set in a covered background area determination flag corresponding to the pixel whose area is determined.

The area determination section 123-3 outputs "
The covered background area determination flag in which 1 ”or“ 0 ”is set is stored in the determination flag storage frame memory 12
4

The determination flag storage frame memory 124
The uncovered background area determination flag supplied from the area determination unit 123-1, the still area determination flag supplied from the area determination unit 123-2, and the area determination unit 123-
2 and the covered background area determination flag supplied from the area determination unit 123-3.

The judgment flag storage frame memory 124
The stored uncovered background area determination flag, stationary area determination flag, moving area determination flag, and covered background area determination flag are stored in the combining unit 1.
25. Based on the uncovered background area determination flag, the still area determination flag, the motion area determination flag, and the covered background area determination flag supplied from the determination flag storage frame memory 124, the combining unit 125 Region information indicating that the region belongs to any of the covered background region, the still region, the moving region, and the covered background region is generated and supplied to the determination flag storage frame memory 126.

The determination flag storage frame memory 126 is
The area information supplied from the synthesizing unit 125 is stored, and the stored area information is output.

Next, an example of the processing of the area specifying unit 103 will be described with reference to FIG.
This will be described with reference to FIGS.

When the object corresponding to the foreground is moving, the position on the screen of the image corresponding to the object changes for each frame. As shown in FIG. 29, in frame #n, the image corresponding to the object located at the position indicated by Yn (x, y) is Yn + 1 (x, y) in frame # n + 1 which is the next frame. located at y).

FIG. 30 shows a model diagram in which the pixel values of the pixels arranged in one row adjacent to the motion direction of the image corresponding to the foreground object are developed in the time direction. For example, when the motion direction of the image corresponding to the foreground object is horizontal with respect to the screen, the model diagram in FIG. 30 shows a model in which pixel values of adjacent pixels on one line are developed in the time direction.

In FIG. 30, the line in frame #n is the same as the line in frame # n + 1.

In frame #n, the foreground components corresponding to the objects included in the second through thirteenth pixels from the left are the sixth through seventeenth pixels from the left in frame # n + 1. included.

In frame #n, the pixels belonging to the covered background area are the eleventh to thirteenth pixels from the left, and the pixels belonging to the uncovered background area are the second to fourth pixels from the left. .
In frame # n + 1, the pixels belonging to the covered background area are the fifteenth through seventeenth pixels from the left, and the pixels belonging to the uncovered background area are the sixth through eighth pixels from the left.

In the example shown in FIG. 30, since the foreground component included in frame #n has moved by four pixels in frame # n + 1, the motion amount v is 4. The number of virtual divisions is
It corresponds to the movement amount v and is 4.

Next, changes in pixel values of pixels belonging to the mixed area before and after the frame of interest will be described.

In frame #n shown in FIG. 31 in which the background is stationary and the foreground motion amount v is 4, the pixels belonging to the covered background area are the 15th to 17th pixels from the left.
The pixel of the th. Since the motion amount v is 4, the fifteenth through seventeenth pixels from the left in the immediately preceding frame # n-1 include only background components and belong to the background area. In the immediately preceding frame # n-2, the fifteenth through seventeenth pixels from the left include only background components and belong to the background area.

Here, since the object corresponding to the background is stationary, the pixel value of the fifteenth pixel from the left of frame # n-1 is the pixel value of the fifteenth pixel from the left of frame # n-2. Does not change from. Similarly, 1 from the left of frame # n-1
The pixel value of the sixth pixel does not change from the pixel value of the sixteenth pixel from the left of frame # n-2, and the pixel value of the seventeenth pixel from the left of frame # n-1 is 17 from the left of 2
It does not change from the pixel value of the th pixel.

That is, a frame corresponding to a pixel belonging to the covered background area in frame #n
The pixels of # n-1 and frame # n-2 are composed of only background components, and the pixel values do not change. Therefore, the absolute value of the difference is almost zero. Therefore, frame # n-1 and frame # n-1 corresponding to the pixels belonging to the mixed area in frame #n
The still / movement determination for the pixel # n-2 is performed by the still / movement determination unit 122-4.
Is determined to be stationary.

Since the pixels belonging to the covered background area in frame #n include the foreground components, the pixel values are different from those of frame # n-1 consisting of only the background components. Therefore, the still / movement determination for the pixels belonging to the mixed area in frame #n and the corresponding pixels in frame # n−1 is determined to be a movement by the still / movement determination unit 122-3.

As described above, the region determining unit 123-3 is supplied with the result of the static / moving determination indicating the motion from the static / moving determining unit 122-3, and the static / dynamic determining unit 122-4 performs the static / moving determination indicating the stillness. When the result is supplied, it is determined that the corresponding pixel belongs to the covered background area.

In frame #n shown in FIG. 32 where the background is stationary and the foreground motion amount v is 4, the pixels included in the uncovered background area are the second to fourth pixels from the left. Since the motion amount v is 4, in the next frame # n + 1, the second to fourth pixels from the left include only the background component and belong to the background area. Further, in the next frame # n + 2, the second to fourth pixels from the left include only the background component and belong to the background area.

Here, since the object corresponding to the background is stationary, the pixel value of the second pixel from the left of frame # n + 2 is the pixel value of the second pixel from the left of frame # n + 1. Does not change from. Similarly, the pixel value of the third pixel from the left of frame # n + 2 does not change from the pixel value of the third pixel from the left of frame # n + 1, and the pixel value of the third pixel from the left of frame # n + 2 does not change. Does not change from the pixel value of the fourth pixel from the left of frame # n + 1.

That is, the pixels of frame # n + 1 and frame # n + 2 corresponding to the pixels belonging to the uncovered background area in frame #n are composed only of the background components and the pixel values do not change. The absolute value of the difference is almost zero. Therefore, the still / movement determination for the pixels in frame # n + 1 and frame # n + 2 corresponding to the pixels belonging to the mixed area in frame #n is performed by the still / movement determination unit 12
By 2-1, it is determined to be stationary.

Since the pixels belonging to the uncovered background area in frame #n include the foreground components,
A case where only the background component in frame # n + 1 is included,
Pixel values are different. Therefore, the still / movement determination for the pixels belonging to the mixed area in frame #n and the corresponding pixels in frame # n + 1 is determined to be a motion by the still / movement determination unit 122-2.

As described above, the area determination section 123-1 is supplied with the result of the static / movement determination indicating the motion from the static / movement determination section 122-2, and receives the static / movement determination indicating the stillness from the static / movement determination section 122-1. When the result is supplied, it is determined that the corresponding pixel belongs to the uncovered background area.

FIG. 33 is a diagram showing determination conditions of the area specifying unit 103 in frame #n. The pixel of frame # n-2 at the same position as the position of the pixel to be determined in frame #n on the image and the pixel of frame #n in the same position as the position of the pixel to be determined in frame #n The pixel of a certain frame # n-1 is determined to be still, and the pixel of the frame # n-1 and the pixel of the frame #n at the same position as the position of the pixel to be determined of the frame #n on the image Is determined to be a motion, the area specifying unit 103 determines that the pixel to be determined in the frame #n belongs to the covered background area.

The pixel of frame # n-1 and the pixel of frame #n located at the same position on the image of the pixel to be determined of frame #n are determined to be still, and the pixel of frame #n When the pixel of the frame # n + 1 located at the same position as the position of the pixel to be determined in the frame #n on the image is determined to be still, the region identifying unit 103 determines the frame #n Is determined to belong to the still area.

The pixel of frame # n-1 and the pixel of frame #n at the same position as the position of the pixel to be determined for frame #n on the image are determined to be moving, and the pixel of frame #n And the pixel of frame # n + 1 located at the same position as the position of the pixel to be determined in frame #n on the image, is determined to be moving. Is determined to belong to the motion area.

The pixel of frame #n and the pixel of frame # n + 1 located at the same position on the image as the pixel to be determined of frame #n are determined to be moving, and the determination of frame #n is performed. The pixel of frame # n + 1 located at the same position as the position of the target pixel on the image, and the frame # n + located at the same position on the image of the pixel targeted for determination of frame #n When the two pixels are determined to be still, the region specifying unit 103 determines that the pixel to be determined in the frame #n belongs to the uncovered background region.

FIG. 34 is a diagram showing an example of a result of specifying an area by the area specifying unit 103. In FIG. In FIG. 34A, pixels determined to belong to the covered background area are displayed in white. In FIG. 34B, the pixels determined to belong to the uncovered background area are displayed in white.

In FIG. 34C, pixels determined to belong to the motion area are displayed in white. FIG.
In (D), the pixels determined to belong to the still area are displayed in white.

FIG. 35 is a diagram showing, as an image, area information indicating a mixed area among the area information output from the frame memory 126 for storing the determination flag. In FIG. 35, pixels determined to belong to the covered background area or the uncovered background area, that is, pixels determined to belong to the mixed area, are displayed in white.
The area information indicating the mixed area output from the determination flag storage frame memory 126 indicates a mixed area and a part having a texture surrounded by a non-textured part in the foreground area.

Next, the area specifying process of the area specifying unit 103 will be described with reference to the flowchart in FIG. In step S121, the frame memory 121 acquires images of frames # n-2 to # n + 2 including the frame #n to be determined.

[0297] In step S122, the static / motion judging section 1
22-3 determines whether the pixel of frame # n-1 and the pixel at the same position of frame #n are still or not, and if it is determined to be still, proceeds to step S123 and proceeds to step S123. 2
Determines whether the pixel of frame #n and the pixel at the same position of frame # n + 1 are still.

If it is determined in step S123 that the pixel of frame #n and the pixel at the same position of frame # n + 1 are still, the process proceeds to step S124, and the area determination unit 12
Step 3-2 sets “1” indicating that the pixel belongs to the still area in the still area determination flag corresponding to the pixel whose area is determined. The area determination unit 123-2 supplies the still area determination flag to the determination flag storage frame memory 124, and the procedure proceeds to step S125.

At step S122, frame # n-1
Pixel and the pixel at the same position in frame #n are determined to be moving, or in step S123, the pixel is determined to be moving in the frame #n and the pixel at the same position in frame # n + 1. In this case, since the pixel of frame #n does not belong to the still area, the process of step S124 is skipped, and the procedure proceeds to step S125.

[0300] In step S125, the static / motion judging section 1
22-3 determines whether or not the pixel of the frame # n-1 and the pixel at the same position of the frame #n are moving. 2
Determines whether or not there is a motion between the pixel of frame #n and the pixel at the same position of frame # n + 1.

If it is determined in step S126 that the pixel of frame #n and the pixel at the same position of frame # n + 1 are motion, the process proceeds to step S127, and the area determination unit 12
3-2 sets “1” indicating that the pixel belongs to the motion area in the motion area determination flag corresponding to the pixel whose area is determined. The area determination unit 123-2 supplies the motion area determination flag to the determination flag storage frame memory 124, and the procedure proceeds to step S128.

At step S125, frame # n-1
Pixel and the pixel at the same position in frame #n are determined to be still, or in step S126, the pixel in frame #n and the pixel in the same position in frame # n + 1 are determined to be still. In this case, since the pixel of frame #n does not belong to the motion area, the process of step S127 is skipped, and the procedure proceeds to step S128.

[0303] In step S128, the static / motion judging section 1
22-4, the pixel of frame # n-2 and the pixel at the same position of frame # n-1 are used to determine whether or not they are still. If it is determined that they are still, the flow proceeds to step S129, 122-
No. 3 determines whether or not the pixel of frame # n-1 and the pixel at the same position of frame #n are moving.

At step S129, frame # n-1
If it is determined that there is a motion between the pixel of the frame #n and the pixel at the same position in the frame #n, the process proceeds to step S130, and the region determination unit 12
Step 3-3 sets “1” indicating that the pixel belongs to the covered background area in the covered background area determination flag corresponding to the pixel whose area is determined. The area determination unit 123-3 supplies the covered background area determination flag to the determination flag storage frame memory 124, and the procedure proceeds to step S131.

In step S128, frame # n-2
Is determined to be a motion between the pixel of the frame # n-1 and the pixel at the same position of the frame # n-1, or, at step S129, the pixel of the frame # n-1 and the pixel at the same position of the frame #n are:
If it is determined that the pixel is stationary, the pixel of frame #n does not belong to the covered background area.
0 is skipped, and the procedure proceeds to step S131.
Proceed to.

[0306] In step S131, the static / motion judging section 1
22-2 determines whether or not the pixel of the frame #n and the pixel at the same position of the frame # n + 1 are moving. If it is determined that the motion is detected, the process proceeds to step S132, and the static / moving determining unit 122- 1
Determines whether the pixel at frame # n + 1 and the pixel at the same position in frame # n + 2 are still.

[0307] In step S132, the frame # n + 1
If the pixel of the frame # n + 2 and the pixel at the same position are determined to be still, the process proceeds to step S133, and the region determination unit 1
23-1 sets "1" to the uncovered background area determination flag corresponding to the pixel whose area is determined, indicating that the pixel belongs to the uncovered background area. The area determination unit 123-1 supplies the uncovered background area determination flag to the determination flag storage frame memory 124, and the procedure proceeds to step S134.

If it is determined in step S131 that the pixel of frame #n and the pixel at the same position in frame # n + 1 are stationary, or in step S132, the pixel of frame # n + 1 and the pixel of frame #n With the pixel at the same position of +2,
If it is determined that the pixel is in motion, the pixel of frame #n does not belong to the uncovered background area,
133 is skipped, and the procedure proceeds to step S1.
Proceed to 34.

In step S134, the area specifying unit 1
03 judges whether or not the area has been specified for all the pixels of frame #n. If it is determined that the area has not been specified for all the pixels of frame #n, the procedure returns to step S122. The region specifying process is repeated for other pixels.

[0310] If it is determined in step S134 that the area has been specified for all the pixels of frame #n,
Proceeding to step S135, the combining unit 125 generates area information indicating the mixed area based on the uncovered background area determination flag and the covered background area determination flag stored in the determination flag storage frame memory 124, Furthermore, area information indicating that each pixel belongs to any of the uncovered background area, the still area, the motion area, and the covered background area is generated, and the generated area information is set in the determination flag storage frame memory 126. Then, the process ends.

[0311] As described above, the area specifying unit 103 calculates the motion area,
Region information indicating belonging to a stationary region, an uncovered background region, or a covered background region can be generated.

[0312] The area specifying unit 103 generates area information corresponding to the mixed area by applying a logical sum to the area information corresponding to the uncovered background area and the covered background area, and includes the area information in the frame. For each of the pixels that have been set, area information including a flag indicating that the pixel belongs to a moving area, a still area, or a mixed area may be generated.

When the object corresponding to the foreground has a texture, the area specifying unit 103 can specify the moving area more accurately.

The area specifying unit 103 can output area information indicating a moving area as area information indicating a foreground area and area information indicating a still area as area information indicating a background area.

Although the description has been made assuming that the object corresponding to the background is stationary, the above-described processing for specifying the area can be applied even if the image corresponding to the background area includes motion. For example, when the image corresponding to the background area is moving uniformly, the area specifying unit 103 shifts the entire image in accordance with this movement, and performs the same processing as when the object corresponding to the background is stationary. I do. Also,
When the image corresponding to the background area includes a different motion for each local area, the area specifying unit 103 selects a pixel corresponding to the motion and executes the above-described processing.

FIG. 37 is a block diagram showing an example of the configuration of the mixture ratio calculating section 104. Estimated mixture ratio processing unit 201
Calculates an estimated mixture ratio for each pixel by an operation corresponding to the model of the covered background area based on the input image, and supplies the calculated estimated mixture ratio to the mixture ratio determination unit 203.

The estimated mixture ratio processing unit 202 calculates an estimated mixture ratio for each pixel by an operation corresponding to the model of the uncovered background area based on the input image, and determines the calculated estimated mixture ratio for the mixture ratio. To the unit 203.

Since it can be assumed that the object corresponding to the foreground is moving at a constant speed within the shutter time, the mixture ratio α of the pixels belonging to the mixed area has the following properties. That is, the mixture ratio α changes linearly in accordance with the change in the position of the pixel. If the change in the pixel position is one-dimensional, the change in the mixture ratio α can be represented by a straight line. If the change in the pixel position is two-dimensional, the change in the mixture ratio α is represented by a plane. be able to.

Since the period of one frame is short, it is assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed.

In this case, the inclination of the mixture ratio α is the inverse ratio of the movement amount v within the shutter time of the foreground.

FIG. 38 shows an example of the ideal mixture ratio α. The gradient l in the mixing region having the ideal mixing ratio α can be represented as the reciprocal of the motion amount v.

As shown in FIG. 38, the ideal mixture ratio α
Has a value of 1 in the background region, a value of 0 in the foreground region, and a value exceeding 0 and less than 1 in the mixed region.

In the example of FIG. 39, the pixel value C06 of the seventh pixel from the left of frame #n is the seventh pixel value from the left of frame # n-1.
Expression (14) can be expressed by using the pixel value P06 of the th pixel.

[0324]

(Equation 3)

In equation (14), the pixel value C06 is expressed as the pixel value M of the pixel in the mixed area, and the pixel value P06 is expressed as the pixel value B of the pixel in the background area. That is, the pixel value M of the pixel in the mixed area
And the pixel value B of the pixel in the background area are given by the formula (1)
5) and Expression (16).

M = C06 (15) B = P06 (16)

[0327] 2 / v in the equation (14) corresponds to the mixture ratio α. Since the motion amount v is 4, the mixture ratio α of the seventh pixel from the left of frame #n is 0.5.

As described above, the pixel value C of the focused frame #n is regarded as the pixel value of the mixed area, and the pixel value P of the frame # n-1 before the frame #n is regarded as the pixel value of the background area. By considering this, Expression (13) indicating the mixture ratio α is rewritten as Expression (17).

C = α · P + f (17) f in Expression (17) is the sum Σ _i Fi / v of the foreground components included in the pixel of interest. The variables included in equation (17) are the mixture ratio α and the sum f of the foreground components.

Similarly, FIG. 40 shows a model obtained by expanding pixel values in the uncovered background area in which the amount of motion v is 4 and the number of virtual divisions in the time direction is 4, in the time direction.

In the uncovered background area, similarly to the expression in the covered background area, the pixel value C of the frame #n of interest is regarded as the pixel value of the mixed area, and the frame #n after the frame #n n + 1
Expression (13) indicating the mixture ratio α can be expressed as Expression (18) by regarding the pixel value N of the pixel value as the pixel value of the background region.

C = α · N + f (18)

Although the background object has been described as being stationary, even when the background object is moving, the pixel value of the pixel at the position corresponding to the background movement amount v can be used. Equations (14) through (1)
8) can be applied. For example, in FIG. 39, when the motion amount v of the object corresponding to the background is 2 and the number of virtual divisions is 2, when the object corresponding to the background is moving to the right in the drawing, the expression (16) The pixel value B of the pixel in the background area is set to a pixel value P04.

Since the equations (17) and (18) each include two variables, the mixture ratio α cannot be obtained as it is. Here, since an image generally has a strong spatial correlation, adjacent pixels have substantially the same pixel value.

Therefore, since the foreground components have a strong spatial correlation, the equation is modified so that the sum f of the foreground components can be derived from the previous or subsequent frame, and the mixture ratio α is obtained.

The pixel value Mc of the seventh pixel from the left of frame #n in FIG. 41 can be expressed by equation (19).

[0337]

(Equation 4) 2 / v in the first term on the right side of the equation (19) corresponds to the mixture ratio α. The second term on the right side of Expression (19) is expressed as Expression (20) using the pixel value of the subsequent frame # n + 1.

[0338]

(Equation 5)

Here, it is assumed that Expression (21) is established using the spatial correlation of the foreground components.

F = F05 = F06 = F07 = F08 = F09 = F10 = F11 = F12 (21) Expression (20) can be replaced by expression (22) using expression (21).

[0341]

(Equation 6)

As a result, β can be expressed by equation (23).

Β = 2/4 (23)

In general, assuming that foreground components related to the mixed area are equal as shown in equation (21), equation (24) is obtained from the relationship of the internal division ratio for all pixels in the mixed area.
Holds.

Β = 1-α (24)

If equation (24) holds, then equation (1)
7) can be expanded as shown in equation (25).

[0347]

(Equation 7)

Similarly, if equation (24) holds, then
Equation (18) can be expanded as shown in equation (26).

[0349]

(Equation 8)

In equations (25) and (26),
Since C, N, and P are known pixel values, the only variable included in Expressions (25) and (26) is the mixture ratio α. FIG. 42 shows the relationship between C, N, and P in Expressions (25) and (26). C calculates the mixture ratio α,
This is the pixel value of the pixel of interest in frame #n. N is
The pixel value of the pixel of frame # n + 1 corresponding to the pixel of interest and the position in the spatial direction. P is the pixel value of the pixel of frame # n-1 corresponding to the pixel of interest and the position in the spatial direction.

Therefore, since each of the equations (25) and (26) includes one variable, the mixture ratio α can be calculated using the pixel values of the pixels of the three frames. . By solving equations (25) and (26), the conditions for calculating the correct mixture ratio α are as follows:
Foreground components related to the mixed region, that is, pixels located at the boundary of the image object corresponding to the direction of motion of the foreground object in the foreground image object captured when the foreground object is stationary That is, the pixel values of the continuous pixels twice as many as the motion amount v are constant.

As described above, the mixture ratio α of the pixels belonging to the covered background area is calculated by the equation (27), and the mixture ratio α of the pixels belonging to the uncovered background area is calculated by the equation (28). You.

Α = (C-N) / (P-N) (27) α = (C-P) / (N-P) (28)

FIG. 43 is a block diagram showing a configuration of the estimated mixture ratio processing section 201. The frame memory 221 stores the input image in frame units, and supplies the next frame after the frame input as the input image to the frame memory 222 and the mixture ratio calculation unit 223.

The frame memory 222 stores the input image in frame units, and supplies the next frame from the frame supplied from the frame memory 221 to the mixture ratio calculation unit 223.

Therefore, when frame # n + 1 is input as an input image to the mixture ratio calculator 223, the frame memory 221 supplies the frame #n to the mixture ratio calculator 223, and the frame memory 222 # n−1 is supplied to the mixture ratio calculation unit 223.

The mixture ratio calculator 223 calculates the pixel value of the pixel of interest in frame #n by the calculation shown in equation (27).
C, the pixel value N of the pixel of frame # n + 1 corresponding to the pixel of interest and the spatial position, and the pixel value of the pixel of frame # n-1 corresponding to the pixel of interest and the spatial position Based on the value P, an estimated mixture ratio of the pixel of interest is calculated, and the calculated estimated mixture ratio is output. For example, when the background is stationary, the mixture ratio calculating unit 223 determines the pixel value C of the pixel of interest in frame #n, the pixel value of frame # n + 1 in the same position as the pixel of interest in the frame. The pixel value N of the pixel, and the position of the pixel of interest and the position in the frame are the same, frame #n
Based on the pixel value P of the pixel of −1, the estimated mixing ratio of the pixel of interest is calculated, and the calculated estimated mixing ratio is output.

Thus, the estimated mixture ratio processing unit 201
An estimated mixture ratio can be calculated based on the input image and supplied to the mixture ratio determination unit 203.

The estimated mixture ratio processing unit 202 calculates the estimated mixture ratio of the pixel of interest by the calculation shown in Expression (27), whereas the estimated mixture ratio processing unit 201 calculates Expression (28). Is the same as that of the estimated mixture ratio processing unit 201 except that the part for calculating the estimated mixture ratio of the pixel of interest is different from the operation shown in FIG.

FIG. 44 is a diagram showing an example of the estimated mixture ratio calculated by the estimated mixture ratio processing unit 201. The estimated mixture ratio shown in FIG. 44 shows a result in a case where the foreground motion amount v corresponding to an object moving at a constant speed is 11, for one line.

The estimated mixture ratio is shown in FIG.
It can be seen that as shown in FIG.

Returning to FIG. 37, the mixture ratio determination unit 203 determines whether the pixel for which the mixture ratio α is supplied from the region identification unit 103 is a foreground region, a background region, a covered background region, or an uncovered region. The mixing ratio α is set based on area information indicating whether the pixel belongs to any of the background areas. If the target pixel belongs to the foreground area, the mixture ratio determination unit 203 sets 0 to the mixture ratio α, and if the target pixel belongs to the background area, sets 1 to the mixture ratio α and becomes the target. If the pixel belongs to the covered background area, the estimated mixture ratio supplied from the estimated mixture ratio processing unit 201 is set to the mixture ratio α, and if the target pixel belongs to the uncovered background area, the estimated mixture ratio processing unit The estimated mixture ratio supplied from 202 is set to the mixture ratio α. The mixture ratio determination unit 203 outputs a mixture ratio α set based on the region information.

FIG. 45 is a block diagram showing another configuration of mixture ratio calculating section 104. In FIG. The selecting unit 231 supplies the pixels belonging to the covered background area and the corresponding pixels of the previous and subsequent frames to the estimated mixture ratio processing unit 232 based on the area information supplied from the area specifying unit 103. The selection unit 231 supplies the pixels belonging to the uncovered background area and the corresponding pixels of the previous and subsequent frames to the estimated mixture ratio processing unit 233 based on the area information supplied from the area identification unit 103. .

The estimated mixture ratio processing unit 232 includes a selection unit 231
Based on the pixel value input from, the estimated mixture ratio of the pixel of interest belonging to the covered background area is calculated by the calculation shown in Expression (27), and the calculated estimated mixture ratio is sent to the selection unit 234. Supply.

The estimated mixture ratio processing unit 233 includes a selection unit 231
Based on the pixel value input from, the estimated mixture ratio of the pixel of interest belonging to the uncovered background area is calculated by the operation shown in Expression (28), and the calculated estimated mixture ratio is selected by the selection unit 234. To supply.

When the target pixel belongs to the foreground area based on the area information supplied from the area specifying unit 103, the selection unit 234 selects an estimated mixture ratio of 0, and selects the mixture ratio α.
If the target pixel belongs to the background area, 1
Is selected and set to the mixture ratio α. When the target pixel belongs to the covered background area, the selection unit 234 selects the estimated mixture ratio supplied from the estimated mixture ratio processing unit 232 and sets the mixture to the mixture ratio α. If it belongs to the ground area, the estimated mixture ratio supplied from the estimated mixture ratio processing unit 233 is selected and set as the mixture ratio α. The selection unit 234
The mixture ratio α selected and set based on the region information is output.

As described above, the mixture ratio calculation unit 104 having another configuration shown in FIG. 45 can calculate the mixture ratio α for each pixel included in the image, and can output the calculated mixture ratio α.

Referring to the flowchart of FIG. 46, FIG.
The process of calculating the mixture ratio α by the mixture ratio calculator 104 shown in FIG. 7 will be described. In step S151, the mixture ratio calculation unit 104 acquires the region information supplied from the region identification unit 103. In step S152, the estimated mixture ratio processing unit 201 executes a process of calculating the estimated mixture ratio using a model corresponding to the covered background area, and supplies the calculated estimated mixture ratio to the mixture ratio determination unit 203. Details of the calculation process of the mixture ratio estimation will be described later with reference to the flowchart in FIG.

[0369] In step S153, the estimated mixture ratio processing unit 202 executes a process of calculating the estimated mixture ratio using a model corresponding to the uncovered background area.
The calculated estimated mixture ratio is supplied to the mixture ratio determination unit 203.

In step S154, the mixture ratio calculator 104 determines whether the mixture ratio α has been estimated for the entire frame. If it is determined that the mixture ratio α has not been estimated for the entire frame, the process proceeds to step S154. Returning to S152, processing for estimating the mixture ratio α for the next pixel is executed.

If it is determined in step S154 that the mixture ratio α has been estimated for the entire frame, the process proceeds to step S155, where the mixture ratio determination unit 203 determines that the pixel is
Foreground area, background area, covered background area,
Alternatively, the mixing ratio α is set based on the area information supplied from the area specifying unit 103, which indicates which of the uncovered background areas belongs. Mixing ratio determining unit 203
Sets 0 to the mixture ratio α when the target pixel belongs to the foreground region, and sets 1 to the mixture ratio α when the target pixel belongs to the background region, and sets the target pixel to the covered background. If it belongs to the region, the estimated mixture ratio supplied from the estimated mixture ratio processing unit 201 is set to the mixture ratio α,
If the target pixel belongs to the uncovered background area, the estimated mixture ratio supplied from the estimated mixture ratio processing unit 202 is set to the mixture ratio α, and the process ends.

As described above, the mixture ratio calculation unit 104 can calculate the mixture ratio α, which is a feature amount corresponding to each pixel, based on the region information supplied from the region identification unit 103 and the input image. .

The process of calculating the mixture ratio α by the mixture ratio calculator 104 having the configuration shown in FIG. 45 is the same as the process described in the flowchart of FIG. 46, and a description thereof will not be repeated.

Next, the process of estimating the mixture ratio using the model corresponding to the covered background area, corresponding to step S152 in FIG. 46, will be described with reference to the flowchart in FIG.

[0375] In step S171, the mixture ratio calculation unit 223 acquires the pixel value C of the target pixel of frame #n from the frame memory 221.

In step S172, the mixture ratio calculation unit 223 obtains the pixel value P of the pixel of frame # n-1 corresponding to the target pixel from the frame memory 222.

[0377] In step S173, the mixture ratio calculation unit 223 obtains the pixel value N of the pixel of frame # n + 1 corresponding to the target pixel included in the input image.

[0378] In step S174, the mixture ratio calculation unit 223 determines the pixel value C of the target pixel in frame #n,
An estimated mixture ratio is calculated based on the pixel value P of the pixel of n−1 and the pixel value N of the pixel of frame # n + 1.

In step S175, the mixture ratio calculating section 223 determines whether or not the process of calculating the estimated mixture ratio has been completed for the entire frame, and has completed the process of calculating the estimated mixture ratio for the entire frame. If it is determined that there is no pixel, the process returns to step S171, and the process of calculating the estimated mixture ratio for the next pixel is repeated.

If it is determined in step S175 that the process of calculating the estimated mixture ratio has been completed for the entire frame, the process ends.

Thus, the estimated mixture ratio processing unit 201
The estimated mixture ratio can be calculated based on the input image.

The process of estimating the mixture ratio in the model corresponding to the uncovered background region in step S153 in FIG. 46 is the same as the process shown in the flowchart in FIG. 47 using the equation corresponding to the model in the uncovered background region. Therefore, the description is omitted.

The estimated mixture ratio processing unit 23 shown in FIG.
47 and the estimated mixture ratio processing unit 233 calculate the estimated mixture ratio by executing the same processing as that in the flowchart shown in FIG. 47, and the description thereof will be omitted.

Although the description has been made assuming that the object corresponding to the background is stationary, the above-described processing for obtaining the mixture ratio α can be applied even when the image corresponding to the background area includes motion. For example, when the image corresponding to the background region is moving uniformly, the estimated mixture ratio processing unit 201
The entire image is shifted according to the movement of the background, and the processing is performed in the same manner as when the object corresponding to the background is stationary. When the image corresponding to the background region includes a background motion that differs for each local area, the estimated mixture ratio processing unit 201
Performs the above-described processing by selecting a pixel corresponding to the background movement as a pixel corresponding to a pixel belonging to the mixed area.

The configuration of the mixture ratio calculating section 104 shown in FIG. 37 or FIG. 45 is an example.

Also, the mixture ratio calculation unit 104 executes only the mixture ratio estimation process using the model corresponding to the covered background area for all the pixels, and outputs the calculated estimated mixture ratio as the mixture ratio α. You may do so. In this case, the mixture ratio α indicates the ratio of the background component with respect to the pixels belonging to the covered background region, and indicates the ratio of the foreground component with respect to the pixels belonging to the uncovered background region. For pixels belonging to the uncovered background area, the absolute value of the difference between the mixture ratio α thus calculated and 1 is calculated,
If the calculated absolute value is set to the mixture ratio α, the signal processing unit 1
2 can determine the mixture ratio α indicating the ratio of the background component for the pixels belonging to the uncovered background area.

Similarly, the mixture ratio calculation unit 104 executes only the mixture ratio estimation process using the model corresponding to the uncovered background area for all the pixels, and calculates the calculated estimated mixture ratio. You may make it output as (alpha).

Next, the foreground / background separation unit 105 will be described. FIG. 48 is a block diagram illustrating an example of a configuration of the foreground / background separation unit 105. The input image supplied to the foreground / background separation unit 105 is supplied to a separation unit 251, a switch 252, and a switch 254. The information indicating the covered background area and the area information indicating the uncovered background area and supplied from the area specifying unit 103 are supplied to the separation unit 251. Area information indicating the foreground area is supplied to the switch 252. Area information indicating the background area is supplied to the switch 254.

The mixture ratio α supplied from the mixture ratio calculation section 104 is supplied to the separation section 251.

The separating section 251 separates the foreground component from the input image based on the area information indicating the covered background area, the area information indicating the uncovered background area, and the mixture ratio α. The component is supplied to the combining unit 253, the background component is separated from the input image, and the separated background component is supplied to the combining unit 255.

The switch 252 is closed when a pixel corresponding to the foreground is input, based on the area information indicating the foreground area, and supplies only the pixel corresponding to the foreground included in the input image to the synthesizing unit 253.

The switch 254 is closed when a pixel corresponding to the background is input, based on the area information indicating the background area, and supplies only the pixel corresponding to the background included in the input image to the synthesizing unit 255.

The synthesizing section 253 synthesizes a foreground component image based on the foreground component supplied from the separation section 251 and the pixel corresponding to the foreground supplied from the switch 252.
The synthesized foreground component image is output. Since the foreground area and the mixed area do not overlap, the synthesis unit 253 synthesizes a foreground component image by applying a logical OR operation to, for example, a component corresponding to the foreground and a pixel corresponding to the foreground.

The synthesizing unit 253 stores an image whose pixel values are all 0 in a built-in frame memory in the initialization processing executed at the beginning of the processing of synthesizing the foreground component image. , The foreground component image is stored (overwritten). Accordingly, in the foreground component image output by the synthesis unit 253, pixels corresponding to the background region are:
0 is stored as the pixel value.

The synthesizing unit 255 synthesizes a background component image based on the component corresponding to the background supplied from the separating unit 251 and the pixel corresponding to the background supplied from the switch 254, and converts the synthesized background component image. Output. Since the background region and the mixed region do not overlap, the synthesis unit 255 synthesizes the background component image by, for example, applying a logical OR operation to the component corresponding to the background and the pixel corresponding to the background.

The synthesizing unit 255 stores an image whose pixel values are all 0 in a built-in frame memory in an initialization process executed at the beginning of the background component image synthesizing process. The background component image is stored (overwritten). Accordingly, in the background component image output by the synthesis unit 255, pixels corresponding to the foreground area include:
0 is stored as the pixel value.

FIG. 49 is a diagram showing an input image input to the foreground / background separation unit 105, and a foreground component image and a background component image output from the foreground / background separation unit 105.

FIG. 49A is a schematic diagram of an image to be displayed, and FIG. 49B is a diagram showing a pixel belonging to the foreground region, a pixel belonging to the background region, and a mixed region corresponding to FIG. 1 is a model diagram in which pixels of one line including pixels belonging to are developed in the time direction.

As shown in FIGS. 49 (A) and 49 (B), the background component image output from foreground / background separation unit 105 is composed of the background component included in the background region and the background component included in the pixel of the mixed region. Consists of

As shown in FIGS. 49 (A) and 49 (B), the foreground component image output from foreground / background separator 105 is composed of pixels belonging to the foreground area and foreground components included in the pixels of the mixed area. Consists of

The pixel values of the pixels in the mixed area are separated into a background component and a foreground component by the foreground / background separation unit 105. The separated background component forms a background component image together with the pixels belonging to the background area. The separated foreground component forms a foreground component image together with the pixels belonging to the foreground area.

As described above, in the foreground component image, the pixel value of the pixel corresponding to the background area is set to 0, and significant pixel values are set to the pixel corresponding to the foreground area and the pixel corresponding to the mixed area. Similarly, in the background component image, the pixel value of the pixel corresponding to the foreground area is set to 0, and meaningful pixel values are set to the pixel corresponding to the background area and the pixel corresponding to the mixed area.

Next, the process of separating the foreground component and the background component from the pixels belonging to the mixed area, which is performed by the separating unit 251 will be described.

FIG. 50 is an image model showing foreground components and background components of two frames including a foreground corresponding to an object moving from left to right in the figure. In the image model shown in FIG. 50, the motion amount v of the foreground is 4
, And the number of virtual divisions is 4.

In frame #n, the leftmost pixel and the fourteenth through eighteenth pixels from the left consist only of background components and belong to the background area. In frame #n,
The second to fourth pixels from the left include background components and foreground components, and belong to the uncovered background area. In frame #n, eleventh to thirteenth from the left
The second pixel contains background and foreground components and belongs to the covered background area. In frame #n, the fifth through tenth pixels from the left consist only of foreground components and belong to the foreground area.

In frame # n + 1, the first through fifth pixels from the left and the eighteenth pixel from the left consist only of background components and belong to the background area. In frame # n + 1, the sixth through eighth pixels from the left include the background component and the foreground component, and belong to the uncovered background area. In frame # n + 1, the fifteenth through seventeenth pixels from the left include the background component and the foreground component, and belong to the covered background area. In frame # n + 1, the ninth to fourteenth pixels from the left consist only of foreground components and belong to the foreground area.

FIG. 51 is a view for explaining the processing for separating the foreground components from the pixels belonging to the covered background area. In FIG. 51, α1 to α18 are mixing ratios corresponding to the respective pixels in frame #n.
In FIG. 51, the fifteenth through seventeenth pixels from the left belong to the covered background area.

The pixel value C15 of the fifteenth pixel from the left of frame #n is represented by equation (29).

C15 = B15 / v + F09 / v + F08 / v + F07 / v = α15 ・ B15 + F09 / v + F08 / v + F07 / v = α15 ・ P15 + F09 / v + F08 / v + F07 / v (29) Here, α15 is a mixture ratio of the fifteenth pixel from the left of frame #n. P15 is the pixel value of the fifteenth pixel from the left of frame # n-1.

[0410] Based on equation (29), one frame from the left of frame #n
The sum f15 of the foreground components of the fifth pixel is represented by Expression (30).

F15 = F09 / v + F08 / v + F07 / v = C15-α15 · P15 (30)

Similarly, the sum f16 of the foreground components of the sixteenth pixel from the left of frame #n is expressed by equation (31), and the sum f17 of the foreground components of the seventeenth pixel from the left of frame #n is
Is represented by equation (32).

F16 = C16-α16 · P16 (31) f17 = C17-α17 · P17 (32)

Thus, the foreground component fc contained in the pixel value C of the pixel belonging to the covered background area is
It is calculated by equation (33).

Fc = C-α · P (33) P
Is the pixel value of the corresponding pixel in the previous frame.

FIG. 52 is a view for explaining the processing for separating the foreground components from the pixels belonging to the uncovered background area. In FIG. 52, α1 to α18 are mixing ratios corresponding to the respective pixels in frame #n. In FIG. 52, the second to fourth pixels from the left belong to the uncovered background area.

[0417] The pixel value of the second pixel from the left of frame #n
C02 is represented by equation (34).

C02 = B02 / v + B02 / v + B02 / v + F01 / v = α2 · B02 + F01 / v = α2 · N02 + F01 / v (34) where α2 is the left of frame #n Is the mixture ratio of the second pixel from. N02 is the pixel value of the second pixel from the left of frame # n + 1.

[0419] Based on equation (34), two frames from the left of frame #n
The sum f02 of the foreground components of the th pixel is expressed by Expression (35).

F02 = F01 / v = C02-α2 · N02 (35)

Similarly, the sum f03 of the foreground components of the third pixel from the left of frame #n is expressed by equation (36), and the sum f04 of the foreground components of the fourth pixel from the left of frame #n is Is represented by equation (37).

F03 = C03-α3 · N03 (36) f04 = C04-α4 · N04 (37)

As described above, the foreground component fu included in the pixel value C of the pixel belonging to the uncovered background area is obtained.
Is calculated by Expression (38).

Fu = C−α · N (38) N is the pixel value of the corresponding pixel in the next frame.

[0425] As described above, the separation unit 251 outputs the information indicating the covered background area included in the area information,
And information indicating the uncovered background area,
The foreground component and the background component can be separated from the pixels belonging to the mixed area based on the mixing ratio α for each pixel.

FIG. 53 is a block diagram showing an example of the structure of the separating section 251 for executing the processing described above. The image input to the separation unit 251 is stored in the frame memory 301.
The area information indicating the covered background area and the uncovered background area supplied from the mixing ratio calculation unit 104 and the mixing ratio α are input to the separation processing block 302.

[0447] The frame memory 301 stores an input image in frame units. The frame memory 301 is
When the target of processing is frame #n, frame # n−1, frame #n, which is one frame before frame #n, and frame # n + 1, which is one frame after frame #n, Remember.

[0428] The frame memory 301 stores the frame # n-1,
The pixels corresponding to the frame #n and the frame # n + 1 are supplied to the separation processing block 302.

The separation processing block 302 determines the frames # n−1 and # supplied from the frame memory 301 based on the area information indicating the covered background area and the uncovered background area and the mixture ratio α.
n and the pixel value of the corresponding pixel in frame # n + 1 in FIG.
By applying the calculation described with reference to FIG. 1 and FIG. 52, foreground components and background components are separated from pixels belonging to the mixed area of frame #n, and supplied to the frame memory 303.

The separation processing block 302 includes an uncovered area processing section 311, a covered area processing section 312, a synthesis section 313, and a synthesis section 314.

The multiplier 3 of the uncovered area processing unit 311
21 multiplies the pixel value of the pixel of frame # n + 1 supplied from the frame memory 301 by the mixture ratio α
22. The switch 322 is connected to the frame memory 3
When the pixel of frame #n supplied from 01 (corresponding to the pixel of frame # n + 1) is the uncovered background area, the pixel is closed and multiplied by the mixture ratio α supplied from the multiplier 321 To the arithmetic unit 322 and the combining unit 31
4 Frame output from switch 322
The value obtained by multiplying the pixel value of pixel # n + 1 by the mixture ratio α
It is equal to the background component of the pixel value of the corresponding pixel of #n.

The arithmetic unit 323 obtains a foreground component by subtracting the background component supplied from the switch 322 from the pixel value of the pixel of frame #n supplied from the frame memory 301. The calculator 323 supplies the foreground component of the pixel of frame #n belonging to the uncovered background area to the combining unit 313.

The multiplier 331 of the covered area processing unit 312
Multiplies the pixel value of the pixel of frame # n-1 supplied from the frame memory 301 by the
Output to The switch 332 is connected to the frame memory 301
When the pixel of frame #n (corresponding to the pixel of frame # n-1) supplied from is a covered background area, the pixel value is closed and multiplied by the mixture ratio α supplied from the multiplier 331. 333 and the combining unit 314. The value obtained by multiplying the pixel value of the pixel of frame # n−1 output from the switch 332 by the mixture ratio α is equal to the background component of the pixel value of the corresponding pixel of frame #n.

The computing unit 333 subtracts the background component supplied from the switch 332 from the pixel value of the pixel of frame #n supplied from the frame memory 301 to obtain a foreground component. The calculator 333 supplies the foreground component of the pixel of frame #n belonging to the covered background area to the combining unit 313.

[0435] The synthesizing unit 313 calculates the arithmetic unit 3 of the frame #n.
The foreground component of the pixel belonging to the uncovered background area supplied from 23 and the foreground component of the pixel belonging to the covered background area supplied from the calculator 333 are combined and supplied to the frame memory 303. .

[0436] The synthesizing unit 314 calculates the background component of the pixel belonging to the uncovered background area and the switch 332 supplied from the switch 322 in the frame #n.
The background components of the pixels belonging to the covered background area supplied from the
To supply.

[0437] The frame memory 303 stores the foreground component and the background component of the pixels in the mixed area of the frame #n supplied from the separation processing block 302, respectively.

The frame memory 303 outputs the stored foreground components of the pixels in the mixed area of frame #n and the stored background components of the pixels in the mixed area of frame #n.

By utilizing the mixture ratio α, which is a feature quantity, it becomes possible to completely separate the foreground component and the background component contained in the pixel value.

[0440] The combining section 253 combines the foreground components of the pixels in the mixed area of frame #n and the pixels belonging to the foreground area output from the separating section 251 to generate a foreground component image. The combining unit 255 combines the background component of the pixel in the mixed area of the frame #n and the pixel belonging to the background area output from the separating unit 251 to generate a background component image.

FIG. 54 is a diagram showing an example of a foreground component image and an example of a background component image corresponding to frame #n in FIG.

FIG. 54A shows an example of a foreground component image corresponding to frame #n in FIG. Since the leftmost pixel and the fourteenth pixel from the left consist of only the background component before the foreground and the background are separated, the pixel value is set to 0.

The second to fourth pixels from the left belong to the uncovered background area before the foreground and the background are separated, the background component is set to 0, and the foreground component is left as it is. . The eleventh to thirteenth pixels from the left belong to the covered background area before the foreground and the background are separated, the background component is set to 0, and the foreground component is left as it is. The fifth to tenth pixels from the left are left as they are because they consist only of foreground components.

FIG. 54B shows an example of a background component image corresponding to frame #n in FIG. The leftmost pixel and the fourteenth pixel from the left are left alone because they consisted only of the background component before the foreground and the background were separated.

The second to fourth pixels from the left belong to the uncovered background area before the foreground and background are separated, the foreground component is set to 0, and the background component is left as it is. . Before the foreground and background are separated, the eleventh to thirteenth pixels from the left belong to the covered background area, the foreground component is set to 0, and the background component is left as it is. Before the foreground and the background are separated, the fifth through tenth pixels from the left consist only of the foreground components, and thus have a pixel value of 0.

Next, the process of separating the foreground and background by the foreground / background separation unit 105 will be described with reference to the flowchart shown in FIG. In step S201, the frame memory 301 of the separation unit 251 acquires an input image,
The frame #n to be separated from the foreground and the background is stored together with the previous frame # n-1 and the subsequent frame # n + 1.

In step S202, the separation unit 251
The separation processing block 302 acquires the area information supplied from the mixture ratio calculation unit 104. In step S203, the separation processing block 302 of the separation unit 251 acquires the mixture ratio α supplied from the mixture ratio calculation unit 104.

[0448] In step S204, the uncovered area processing unit 311 performs processing based on the area information and the mixture ratio α.
The background component is extracted from the pixel values of the pixels belonging to the uncovered background area supplied from the frame memory 301.

[0449] In step S205, the uncovered area processing unit 311 performs the processing based on the area information and the mixture ratio α.
The foreground component is extracted from the pixel values of the pixels belonging to the uncovered background area supplied from the frame memory 301.

In step S206, the covered area processing unit 312 extracts a background component from the pixel values of the pixels belonging to the covered background area supplied from the frame memory 301 based on the area information and the mixture ratio α. .

In step S207, the covered area processing unit 312 extracts a foreground component from the pixel values of the pixels belonging to the covered background area supplied from the frame memory 301 based on the area information and the mixture ratio α. .

In step S208, the synthesizing unit 313
Combines the foreground components of the pixels belonging to the uncovered background area extracted in the processing of step S205 and the foreground components of the pixels belonging to the covered background area extracted in the processing of step S207. The synthesized foreground component is supplied to the synthesis unit 253. Further, the synthesizing unit 253 generates a foreground component image by synthesizing the pixels belonging to the foreground area supplied via the switch 252 and the foreground components supplied from the separation unit 251.

In step S209, the synthesizing unit 314
Synthesizes the background component of the pixel belonging to the uncovered background area extracted in the processing of step S204 and the background component of the pixel belonging to the covered background area extracted in the processing of step S206. The synthesized background component is supplied to the synthesis unit 255. Further, the synthesizing unit 255 synthesizes the pixels belonging to the background area supplied via the switch 254 and the background component supplied from the separating unit 251 to generate a background component image.

In step S210, combining section 253
Outputs a foreground component image. In step S211, the combining unit 255 outputs the background component image, and the process ends.

As described above, the foreground / background separation unit 105 separates the foreground component and the background component from the input image based on the region information and the mixture ratio α, and outputs the foreground component image composed of only the foreground component. Also, a background component image consisting of only background components can be output.

Next, adjustment of the amount of motion blur from the foreground component image will be described.

FIG. 56 is a block diagram showing an example of the configuration of the motion-blur adjusting unit 106. The motion vector and its position information supplied from the motion detection unit 102 and the region information supplied from the region identification unit 103 are processed by the processing unit determination unit 3
51 and the modeling unit 352. The foreground component image supplied from the foreground / background separation unit 105 is supplied to the addition unit 354.

The processing unit determination unit 351 supplies the generated processing unit to the modeling unit 352 together with the motion vector based on the motion vector, its position information, and the area information. The processing unit determination unit 351 supplies the generated processing unit to the adding unit 354.

As shown in the example of FIG. 57, the processing unit generated by the processing unit determination unit 351 starts from a pixel corresponding to the covered background area of the foreground component image and extends to a pixel corresponding to the uncovered background area. A continuous pixel in the movement direction starting from a pixel corresponding to the uncovered background area or a continuous pixel aligned in the movement direction to a pixel corresponding to the covered background area is shown. The processing unit is, for example, an upper left point (a pixel designated by the processing unit and a position of a pixel located at the leftmost or uppermost position on the image) and a lower right point.
Data.

[0460] The modeling unit 352 performs modeling based on the motion vector and the input processing unit. More specifically, for example, the modeling unit 352 previously stores a plurality of models corresponding to the number of pixels included in the processing unit, the number of virtual divisions of pixel values in the time direction, and the number of foreground components for each pixel. A model that specifies the correspondence between a pixel value and a foreground component as shown in FIG. 58 may be selected based on the processing unit and the number of virtual divisions of the pixel value in the time direction. .

For example, if the number of pixels corresponding to the processing unit is one,
When the motion amount v during the shutter time is 5, the modeling unit 352 sets the virtual division number to 5, the leftmost pixel includes one foreground component, and the second pixel from the left. Pixel includes two foreground components, the third pixel from the left includes three foreground components, the fourth pixel from the left includes four foreground components, and the fifth pixel from the left includes five foreground components. Including the foreground component, the sixth pixel from the left contains five foreground components, the seventh pixel from the left contains five foreground components, and the eighth pixel from the left contains five foreground components. , The ninth pixel from the left contains four foreground components, the tenth pixel from the left contains three foreground components, the eleventh pixel from the left contains two foreground components, and 12 from the left.
A model is selected in which the th pixel contains one foreground component and comprises a total of eight foreground components.

Note that the modeling unit 352 does not select a model stored in advance but generates a model based on the motion vector and the processing unit when the motion vector and the processing unit are supplied. You may do so.

[0463] The modeling unit 352 supplies the selected model to the equation generation unit 353.

[0464] The equation generating unit 353 includes a modeling unit 352.
Generate an equation based on the model supplied from. FIG.
Referring to the model of the foreground component image shown in FIG. 8, the number of foreground components is 8, the number of pixels corresponding to the processing unit is 12, the motion amount v is 5, and the number of virtual divisions is 5. An equation generated by the equation generating unit 353 at a certain time will be described.

When the foreground component corresponding to the shutter time / v included in the foreground component image is F01 / v to F08 / v, F01 / v
To F08 / v and the pixel values C01 to C12 are given by Equation (39)
To (50).

C01 = F01 / v (39) C02 = F02 / v + F01 / v (40) C03 = F03 / v + F02 / v + F01 / v (41) C04 = F04 / v + F03 / v + F02 / v + F01 / v (42) C05 = F05 / v + F04 / v + F03 / v + F02 / v + F01 / v (43) C06 = F06 / v + F05 / v + F04 / v + F03 / v + F02 / v (44) C07 = F07 / v + F06 / v + F05 / v + F04 / v + F03 / v (45) C08 = F08 / v + F07 / v + F06 / v + F05 / v + F04 / v (46) C09 = F08 / v + F07 / v + F06 / v + F05 / v (47) C10 = F08 / v + F07 / v + F06 / v (48) C11 = F08 / v + F07 / v (49) C12 = F08 / v (50)

[0467] The equation generating section 353 deforms the generated equation to generate an equation. The equations generated by the equation generation unit 353 are shown in equations (51) to (62). C01 = 1 ・ F01 / v + 0 ・ F02 / v + 0 ・ F03 / v + 0 ・ F04 / v + 0 ・ F05 / v +0 ・ F06 / v + 0 ・ F07 / v + 0 ・ F08 / v ( 51) C02 = 1 ・ F01 / v + 1 ・ F02 / v + 0 ・ F03 / v + 0 ・ F04 / v + 0 ・ F05 / v +0 ・ F06 / v + 0 ・ F07 / v + 0 ・ F08 / v (52) C03 = 1 ・ F01 / v + 1 ・ F02 / v + 1 ・ F03 / v + 0 ・ F04 / v + 0 ・ F05 / v +0 ・ F06 / v + 0 ・ F07 / v + 0 ・F08 / v (53) C04 = 1 ・ F01 / v + 1 ・ F02 / v + 1 ・ F03 / v + 1 ・ F04 / v + 0 ・ F05 / v +0 ・ F06 / v + 0 ・ F07 / v + 0 ・ F08 / v (54) C05 = 1 ・ F01 / v + 1 ・ F02 / v + 1 ・ F03 / v + 1 ・ F04 / v + 1 ・ F05 / v +0 ・ F06 / v + 0 ・ F07 / v + 0 ・ F08 / v (55) C06 = 0 ・ F01 / v + 1 ・ F02 / v + 1 ・ F03 / v + 1 ・ F04 / v + 1 ・ F05 / v +1 ・ F06 / v + 0 ・F07 / v + 0 ・ F08 / v (56) C07 = 0 ・ F01 / v + 0 ・ F02 / v + 1 ・ F03 / v + 1 ・ F04 / v + 1 ・ F05 / v +1 ・ F06 / v + 1 ・ F07 / v + 0 ・ F08 / v (57) C08 = 0 ・ F01 / v + 0 ・ F02 / v + 0 ・ F03 / v + 1 ・ F04 / v + 1 ・ F05 / v +1 ・ F06 / v + 1 ・ F07 / v + 1 ・ F08 / v (58) C09 = 0 ・ F01 / v + 0 ・ F02 / v + 0 ・ F03 / v + 0 ・ F04 / v + 1 ・ F05 / v +1 ・F06 / v + 1 ・ F07 / v + 1 ・ F08 / v (59) C10 = 0 ・ F01 / v + 0 ・ F02 / v + 0 ・ F03 / v + 0 ・ F04 / v + 0 ・ F05 / v + 1 ・ F06 / v + 1 ・ F07 / v + 1 ・ F08 / v (60) C11 = 0 ・ F01 / v + 0 ・ F0 2 / v + 0 ・ F03 / v + 0 ・ F04 / v + 0 ・ F05 / v +0 ・ F06 / v + 1 ・ F07 / v + 1 ・ F08 / v (61) C12 = 0 ・ F01 / v + 0 ・ F02 / v + 0 ・ F03 / v + 0 ・ F04 / v + 0 ・ F05 / v +0 ・ F06 / v + 0 ・ F07 / v + 1 ・ F08 / v (62)

Formulas (51) to (62) are obtained by using formula (63).
It can also be expressed as

[0469]

(Equation 9) In Expression (63), j indicates a position of a pixel. In this example, j has a value of any one of 1 to 12. I indicates the position of the foreground value. In this example, i has a value of any one of 1 to 8. aij
Has a value of 0 or 1 corresponding to the values of i and j.

When expressed in consideration of the error, equation (63)
Can be expressed as in equation (64).

[0471]

(Equation 10) In Expression (64), ej is an error included in the target pixel Cj.

The equation (64) can be rewritten as the equation (65).

[0473]

[Equation 11]

Here, in order to apply the least squares method, the sum of squares E of the error is defined as shown in Expression (66).

[0475]

(Equation 12)

In order to minimize the error, the sum of squares of the error
What is necessary is that the value of the partial differentiation of E with respect to the variable Fk becomes 0. Fk is determined so as to satisfy Expression (67).

[0477]

(Equation 13)

In equation (67), since the motion amount v is a fixed value, equation (68) can be derived.

[0479]

[Equation 14]

By expanding the equation (68) and transposing it, the equation (69) is obtained.

[0481]

(Equation 15)

[0447] The expression is expanded into eight expressions obtained by substituting any one of integers 1 to 8 into k in the expression (69). The obtained eight expressions can be represented by one expression by a matrix. This equation is called a normal equation.

An example of a normal equation generated by the equation generating section 353 based on the least squares method is shown in equation (70).

[0484]

(Equation 16)

If equation (70) is expressed as A · F = v · C, C, A, v are known, and F is unknown. A and v are known at the time of modeling, but C becomes known by inputting pixel values in the adding operation.

The error contained in the pixel C can be dispersed by calculating the foreground component by the normal equation based on the least square method.

The equation generating section 353 supplies the normal equation generated in this way to the adding section 354.

The adding unit 354 includes the processing unit determining unit 35
Based on the processing unit supplied from 1, the pixel value C included in the foreground component image is set to the matrix expression supplied from the equation generation unit 353. The adding unit 354 supplies the matrix in which the pixel values C are set to the calculation unit 355.

[0489] The arithmetic unit 355 performs the sweep-out method (Gauss-Jord).
An elimination method such as an) is used to calculate the foreground component Fi / v from which the motion blur has been removed, and the pixel value of the foreground from which the motion blur has been removed, which is an integer from 0 to 8 Of the foreground component image from which the motion blur has been removed, which is composed of the pixel values Fi from which the motion blur has been removed, as shown in the example of FIG. 357.

In the foreground component image from which the motion blur is removed as shown in FIG. 59, each of F01 to F08 is set for each of C03 to C10 because the position of the foreground component image with respect to the screen is not changed. And can correspond to an arbitrary position.

The motion-blur adding unit 356 generates a motion-blur adjustment amount v ′ having a value different from the motion amount v, for example, a motion-blur adjustment amount v ′ having a half value of the motion amount v, or a value unrelated to the motion amount v. By giving the motion blur adjustment amount v ′, the amount of motion blur can be adjusted. For example, as shown in FIG. 60, the motion-blur adding unit 356 outputs the pixel value Fi of the foreground from which the motion blur has been removed.
Is divided by the motion blur adjustment amount v 'to obtain the foreground component Fi / v'
Is calculated, and the sum of the foreground components Fi / v ′ is calculated to generate a pixel value in which the amount of motion blur is adjusted. For example, when the motion blur adjustment amount v ′ is 3, the pixel value C02 is (F01) / v ′, the pixel value C03 is (F01 + F02) / v ′, and the pixel value C04 is
Is (F01 + F02 + F03) / v ', and the pixel value C05 is (F02 + F02).
03 + F04) / v '.

[0492] The motion blur adding unit 356 supplies the foreground component image in which the amount of motion blur has been adjusted to the selecting unit 357.

[0493] The selection section 357, for example, based on a selection signal corresponding to the user's selection, receives the foreground component image from which the motion blur has been removed and the motion supplied from the motion blur addition section 356. One of the foreground component images whose blur amount has been adjusted is selected, and the selected foreground component image is output.

As described above, the motion-blur adjusting unit 106 can adjust the amount of motion blur based on the selection signal and the motion-blur adjustment amount v ′.

For example, as shown in FIG. 61, when the number of pixels corresponding to the processing unit is 8 and the motion amount v is 4, the motion-blur adjusting unit 106 sets the matrix shown in the equation (71). Generates the expression

[0496]

[Equation 17]

[0497] The motion-blur adjusting unit 106 calculates Fi, which is a pixel value in which the amount of motion blur is adjusted, by formulating a number corresponding to the length of the processing unit. Similarly, for example, when the number of pixels included in the processing unit is 100, 1
An equation corresponding to 00 pixels is generated to calculate Fi.

FIG. 62 is a diagram showing another configuration of the motion-blur adjusting unit 106. The same portions as those shown in FIG. 56 are denoted by the same reference numerals, and description thereof will be omitted.

The selecting section 361 supplies the input motion vector and its position signal to the processing unit determining section 351 and the modeling section 352 as they are, or adjusts the magnitude of the motion vector based on the selection signal. The motion vector whose magnitude is replaced by the motion blur adjustment amount v ′ and its position signal instead of the amount v ′ are supplied to the processing unit determination unit 351 and the modeling unit 352.

By doing so, the processing unit determination unit 351 through the operation unit 355 of the motion blur adjustment unit 106 shown in FIG.
Can adjust the amount of motion blur according to the values of the motion amount v and the motion blur adjustment amount v ′. For example, the amount of movement v
Is 5, and the motion blur adjustment amount v ′ is 3, the processing unit determination unit 351 to the calculation unit 355 of the motion blur adjustment unit 106 in FIG. FIG. 60 corresponding to the motion blur adjustment amount v ′ of 3 for the image
The calculation is performed according to the model shown in
v) / (Motion blur adjustment amount v ′) = 5/3, that is, an image including a motion blur corresponding to the motion amount v of approximately 1.7 is calculated. In addition,
In this case, since the calculated image does not include the motion blur corresponding to the motion amount v of 3, the motion blur adding unit 35
It should be noted that the meaning of the relationship between the motion amount v and the motion blur adjustment amount v ′ differs from the result of No. 6.

[0501] As described above, the motion-blur adjusting unit 106
An equation is generated corresponding to the motion amount v and the processing unit, and the pixel value of the foreground component image is set in the generated equation to calculate a foreground component image in which the amount of motion blur is adjusted.

Next, the process of adjusting the amount of motion blur included in the foreground component image by the motion blur adjustment unit 106 will be described with reference to the flowchart in FIG.

[0503] In step S251, the processing unit determination unit 351 of the motion blur adjustment unit 106 generates a processing unit based on the motion vector and the area information, and supplies the generated processing unit to the modeling unit 352.

[0504] In step S252, the modeling unit 352 of the motion-blur adjusting unit 106 selects or generates a model in accordance with the motion amount v and the processing unit. Step S
In 253, the equation generation unit 353 creates a normal equation based on the selected model.

At step S254, adding section 3
Reference numeral 54 sets the pixel value of the foreground component image in the created normal equation. In step S255, the adding unit 3
Step 54 determines whether or not the pixel values of all the pixels corresponding to the processing unit have been set, and if it is determined that the pixel values of all the pixels corresponding to the processing unit have not been set. And the process of setting the pixel value to the normal equation is repeated.

[0506] If it is determined in step S255 that the pixel values of all the pixels in the processing unit have been set, the process proceeds to step S256, and the arithmetic unit 355 causes the adding unit 3
Based on the normal equation in which the pixel value supplied from 54 is set, the pixel value of the foreground with the amount of motion blur adjusted is calculated,
The process ends.

As described above, the motion-blur adjusting unit 106 can adjust the amount of motion blur from the foreground image including the motion blur based on the motion vector and the area information.

That is, it is possible to adjust the amount of motion blur included in a pixel value which is sample data.

The motion-blur adjusting unit 106 shown in FIG.
Is an example and is not the only configuration.

[0510] As described above, the signal processing unit 12 shown in Fig. 10 can adjust the amount of motion blur included in the input image. Signal processing unit 12 whose configuration is shown in FIG.
Calculates the mixture ratio α, which is the buried information, and can output the calculated mixture ratio α.

[0511] FIG. 64 is a block diagram showing another configuration of the function of the signal processing unit 12.

[0512] The same portions as those shown in Fig. 10 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

[0513] The area specifying unit 103 supplies the area information to the mixture ratio calculating unit 104 and the synthesizing unit 371.

[0514] The mixture ratio calculation unit 104 supplies the mixture ratio α to the foreground / background separation unit 105 and the synthesis unit 371.

[0515] The foreground / background separation unit 105 supplies the foreground component image to the synthesis unit 371.

[0516] The combining section 371 generates an arbitrary background image and a foreground / background separating section 1 based on the mixing ratio α supplied from the mixing ratio calculating section 104 and the area information supplied from the area specifying section 103.
The foreground component image supplied from step 05 is synthesized, and a synthesized image in which an arbitrary background image and a foreground component image are synthesized is output.

FIG. 65 is a diagram showing the structure of the synthesizing unit 371. The background component generation unit 381 generates a background component image based on the mixture ratio α and an arbitrary background image, and supplies the background component image to the mixed region image synthesis unit 382.

[0518] The mixed area image synthesizing section 382 generates a mixed area synthesized image by synthesizing the background component image supplied from the background component generating section 381 and the foreground component image, and generates the mixed area synthesized image. It is supplied to the image synthesizing unit 383.

[0519] The image combining section 383 combines the foreground component image, the mixed area combined image supplied from the mixed area image combining section 382, and an arbitrary background image based on the area information.
Generate and output a composite image.

As described above, the synthesizing unit 371 can synthesize a foreground component image with an arbitrary background image.

An image obtained by synthesizing a foreground component image with an arbitrary background image based on the mixture ratio α, which is a characteristic amount, is more natural than an image obtained by simply synthesizing pixels.

FIG. 66 is a block diagram showing still another configuration of the function of the signal processing unit 12 for adjusting the amount of motion blur. The signal processing unit 12 shown in FIG.
Are sequentially performed, whereas the signal processing unit 12 illustrated in FIG. 66 performs the area specification and the calculation of the mixture ratio α in parallel.

[0523] The same portions as those shown in the block diagram of FIG. 10 are denoted by the same reference numerals, and description thereof will be omitted.

[0524] The input image is supplied to the mixture ratio calculating section 401, the foreground / background separating section 402, the area specifying section 103, and the object extracting section 101.

[0525] The mixture ratio calculating section 401 calculates the mixture ratio based on the input image.
An estimated mixture ratio when the pixel is assumed to belong to the covered background region and an estimated mixture ratio when the pixel is assumed to belong to the uncovered background region are calculated for each of the pixels included in the input image. The estimated mixture ratio when the calculated pixel belongs to the covered background area and the estimated mixture ratio when the pixel belongs to the uncovered background area are calculated by the foreground / background separation unit 40.
Feed to 2.

FIG. 67 is a block diagram showing an example of the configuration of the mixture ratio calculating section 401.

The estimated mixture ratio processing unit 201 shown in FIG.
This is the same as the estimated mixture ratio processing unit 201 shown in FIG. The estimated mixture ratio processing unit 202 shown in FIG. 67 is the same as the estimated mixture ratio processing unit 202 shown in FIG.

The estimated mixture ratio processing unit 201 calculates an estimated mixture ratio for each pixel by an operation corresponding to the model of the covered background area based on the input image, and outputs the calculated estimated mixture ratio.

The estimated mixture ratio processing unit 202 calculates an estimated mixture ratio for each pixel by an operation corresponding to the model of the uncovered background area based on the input image, and outputs the calculated estimated mixture ratio.

The foreground / background separation unit 402
01, the estimated mixture ratio when the pixel is assumed to belong to the covered background region, the estimated mixture ratio when the pixel is assumed to belong to the uncovered background region, and the region identification unit 1.
A foreground component image is generated from the input image on the basis of the area information supplied from 03, and the generated foreground component image is supplied to the motion blur adjustment unit 106 and the selection unit 107.

FIG. 68 is a block diagram showing an example of the structure of the foreground / background separation unit 402.

[0532] The same parts as those in the foreground / background separation unit 105 shown in Fig. 48 are denoted by the same reference numerals, and description thereof will be omitted.

[0533] The selection unit 421, based on the region information supplied from the region identification unit 103, estimates the mixture ratio supplied from the mixture ratio calculation unit 401 when it is assumed that the pixel belongs to the covered background region, and One of the estimated mixture ratios when the pixel is assumed to belong to the uncovered background area is selected, and the selected estimated mixture ratio is supplied to the separation unit 251 as the mixture ratio α.

The separation unit 251 extracts a foreground component and a background component from the pixel values of the pixels belonging to the mixed region based on the mixture ratio α and the region information supplied from the selection unit 421, and extracts the extracted foreground component. Is supplied to the synthesizing unit 253, and the background component is supplied to the synthesizing unit 255.

The separating section 251 can have the same configuration as the configuration shown in FIG.

The combining section 253 combines and outputs a foreground component image. The combining unit 255 combines and outputs the background component image.

[0537] The motion-blur adjusting unit 106 shown in Fig. 66 can have the same configuration as that shown in Fig. 10. The foreground component image supplied from the foreground / background separating unit 402 based on the area information and the motion vector. Is adjusted, and a foreground component image in which the amount of motion blur is adjusted is output.

[0538] The selection unit 107 shown in FIG. 66, for example, based on a selection signal corresponding to the user's selection, outputs the foreground / background separation unit 4
Either the foreground component image supplied from 02 or the foreground component image supplied from the motion-blur adjusting unit 106 with the amount of motion blur adjusted is selected, and the selected foreground component image is output.

As described above, the signal processing unit 12 shown in FIG. 66 adjusts the amount of motion blur included in an image corresponding to a foreground object included in an input image and outputs the adjusted image. be able to. The signal processing unit 12 shown in FIG. 66 can calculate the mixture ratio α, which is the buried information, and output the calculated mixture ratio α, as in the first embodiment.

FIG. 69 is a block diagram showing another configuration of the function of the signal processing section 12 for synthesizing a foreground component image with an arbitrary background image. While the signal processing unit 12 shown in FIG. 64 serially performs the area specification and the calculation of the mixture ratio α, FIG.
The signal processing unit 12 shown in (1) performs region identification and calculation of the mixture ratio α in parallel.

The same reference numerals are given to the same portions as those shown in the block diagram of FIG. 66, and the description thereof will be omitted.

[0542] The mixture ratio calculation unit 401 shown in Fig. 69 estimates the estimated mixture ratio when the pixel belongs to the covered background area based on the input image, and assumes that the pixel belongs to the uncovered background area. The estimated mixture ratio in the case is calculated for each of the pixels included in the input image, and the estimated mixture ratio in the case where the calculated pixel belongs to the covered background region, and the pixel belongs to the uncovered background region Is supplied to the foreground / background separation unit 402 and the synthesis unit 431.

The foreground / background separation unit 402 shown in FIG. 69 calculates the estimated mixture ratio supplied from the mixture ratio calculation unit 401 when it is assumed that the pixel belongs to the covered background region, and the pixel in the uncovered background region. A foreground component image is generated from the input image based on the estimated mixture ratio when it is assumed to belong and the area information supplied from the area specifying unit 103, and the generated foreground component image is supplied to the combining unit 431.

The combining unit 431 supplies the estimated mixture ratio when the pixel belongs to the covered background area and the estimated mixture ratio when the pixel belongs to the uncovered background area, supplied from the mixture ratio calculation unit 401. An arbitrary background image and a foreground component image supplied from the foreground / background separation unit 402 are combined based on the estimated mixture ratio and the region information supplied from the region identification unit 103, and the arbitrary background image and the foreground component image are combined. And outputs a composite image obtained by combining the above.

FIG. 70 is a diagram showing the structure of the synthesizing section 431. Parts that are the same as the functions illustrated in the block diagram of FIG. 65 are given the same numbers, and descriptions thereof will be omitted.

[0546] Based on the region information supplied from the region identification unit 103, the selection unit 441 provides the estimated mixture ratio supplied from the mixture ratio calculation unit 401 when the pixel belongs to the covered background region, and One of the estimated mixture ratios when the pixel is assumed to belong to the uncovered background area is selected, and the selected estimated mixture ratio is supplied to the background component generation unit 381 as the mixture ratio α.

The background component generation section 381 shown in FIG. 70 generates a background component image based on the mixture ratio α and an arbitrary background image supplied from the selection section 441, and supplies the background component image to the mixed area image synthesis section 382. .

The mixed area image synthesizing unit 382 shown in FIG.
Generates a mixed region composite image by combining the background component image supplied from the background component generation unit 381 with the foreground component image, and supplies the generated mixed region composite image to the image combination unit 383.

The image synthesizing section 383 synthesizes the foreground component image, the mixed area synthesized image supplied from the mixed area image synthesizing section 382, and an arbitrary background image based on the area information.
Generate and output a composite image.

Thus, the synthesizing section 431 can synthesize the foreground component image with an arbitrary background image.

Although the mixture ratio α has been described as the ratio of the background component included in the pixel value, it may be the ratio of the foreground component included in the pixel value.

Also, the direction of movement of the foreground object has been described from left to right, but it is needless to say that the direction is not limited to that direction.

Next, an example will be described in which the amount of motion blur included in the temperature data or the pressure data is adjusted by the same processing as that of the signal processing unit 12 described above.

FIG. 71 is a diagram showing a configuration example of a signal processing device according to the present invention. The thermographic device 451 converts the infrared rays radiated from the object to be measured into
Detected by built-in infrared sensor such as infrared CCD,
A signal corresponding to the detected infrared wavelength or intensity is generated. The thermography device 451 converts the generated signal from analog to digital, compares the generated signal with reference data corresponding to a reference temperature, generates temperature data indicating the temperature of each part of the object, and converts the generated temperature data into a signal processing unit 452.
Output to

[0555] The thermographic device 451 is provided with the sensor 11
Similarly, has an integrating effect on space and on time.

[0556] The thermographic device 451 is the signal processing unit 4
The temperature data supplied to 52 has the same configuration as the image data of the moving image, and is a value indicating the temperature of each part of the object to be measured (corresponding to the pixel value of the image data).
Are two-dimensionally arranged in the spatial direction (corresponding to a frame of image data), and further arranged in the time direction.

The signal processing unit 452 adjusts the distortion included in the input temperature data and caused by the movement of the object to be measured. For example, the signal processing unit 452 extracts a more accurate temperature of a desired portion of the object to be measured.

FIG. 72 is a flowchart for explaining the process of adjusting the amount of motion blur by the signal processing unit 452.
In step S301, the signal processing unit 452 acquires temperature data in which values indicating temperatures are two-dimensionally arranged, corresponding to each part of the object to be measured. The signal processing unit 452 generates data indicating movement based on the temperature data.

In step S302, the signal processing unit 4
Reference numeral 52 denotes a foreground region consisting of only a value indicating a temperature corresponding to an object whose temperature is desired to be measured, a background region consisting of only a value representing a temperature corresponding to an object other than the desired object, and a temperature corresponding to a desired object. The temperature data area is specified in a mixed area formed by mixing the information of the object and the temperature information corresponding to the object other than the desired object.

In step S303, the signal processing unit 4
52 determines whether or not the value indicating the temperature included in the temperature data belongs to the mixed region. If it is determined that the value belongs to the mixed region, the process proceeds to step S304, and the process proceeds to step S304 in FIG.
The mixture ratio α is calculated by a process similar to the process of 102.

In step S305, the signal processing unit 4
Reference numeral 52 denotes a process similar to the process of step S103 in FIG. 27, which separates temperature information corresponding to the object whose temperature is desired to be measured, and the procedure proceeds to step S306.

The process of separating the temperature information in step S305 is based on Kirchhoff's law or Stephans
Based on the law indicating the relationship between the temperature of an object and the emitted infrared rays, such as Boltzmann's law, the temperature information is converted into the amount of infrared energy emitted from the object whose temperature is desired to be measured and converted. The energy amount of infrared rays may be separated, and the separated energy amount may be converted into temperature again. The signal processing unit 452 can separate the temperature information with higher accuracy as compared with the case where the temperature information is separated as it is by converting the information into the amount of infrared energy and separating.

If it is determined in step S303 that the value indicating the temperature included in the temperature data does not belong to the mixed area, there is no need to perform processing for separating the temperature information corresponding to the object whose temperature measurement is desired. S3
04 and step S305 are skipped, and the procedure proceeds to step S306.

[0564] In step S306, the signal processing unit 4
52 is a value indicating the temperature belonging to the foreground area and temperature information corresponding to the object for which the temperature measurement is desired.
Generate temperature data corresponding to an object for which temperature measurement is desired.

[0565] In step S307, the signal processing unit 4
52 is a process similar to the process of step S251 in FIG. 63, and generates a model corresponding to the generated temperature data.

[0566] In step S308, the signal processing unit 4
Reference numeral 52 denotes a step S2 in FIG. 63 based on the generated model.
By the same processing as the processing from 52 to S255, the amount of motion blur included in the temperature data corresponding to the object whose temperature measurement is desired is adjusted, and the processing ends.

As described above, the signal processing unit 452 adjusts the amount of motion blur included in the temperature data generated by the movement of the object to be measured, and calculates a more accurate temperature value of each part of the object. can do.

FIG. 73 is a diagram showing a configuration example of a signal processing device according to the present invention. In this example, weight measurement is performed. The pressure area sensor 501 includes a plurality of pressure sensors, and measures a load on a unit area of a plane, that is, a pressure. The pressure area sensor 501 includes, for example, a plurality of pressure sensors 511-1-1 to 511 as shown in FIG.
-M-N is two-dimensionally arranged on the floor. When the object 512 whose weight is to be measured moves on the pressure area sensor 501, the pressure area sensor 50
1 detects the pressure applied to each of the pressure sensors 511-1-1 to 511-M-N,
It generates weight data indicating the weight corresponding to each of the measurement ranges of 1-1-1 to 511-MN, and outputs the generated weight data to the signal processing unit 502.

[0566] Pressure sensors 511-1-1 to 511-M
Each of -N is constituted by, for example, a sensor using birefringence, so-called photoelasticity, which is generated by distortion when an external force is applied to a transparent elastic body.

[0570] The pressure area sensor 501 may be constituted entirely by one sensor utilizing photoelasticity.

FIG. 75 is a view for explaining loads applied to each of the pressure sensors 511-m-1 to 511-m-N constituting the pressure area sensor 501 and corresponding to the weight of each part of the object 512.

The load a corresponding to the weight of the leftmost portion of the object 512 in the drawing is calculated by the pressure sensor 511.
-M-1. The load b corresponding to the weight of the portion of the object 512 located second from the left in the drawing is:
It is applied to the pressure sensor 511-m-2. The load c corresponding to the weight of the portion of the object 512 located third from the left in the drawing is applied to the pressure sensor 511-m-3. The load d corresponding to the weight of the portion of the object 512 located at the fourth position from the left in the drawing is the pressure sensor 511-
m-4.

The load e corresponding to the weight of the portion of the object 512 located fifth from the left in the figure is applied to the pressure sensors 511-m-5. The load f corresponding to the weight of the portion of the object 512 located sixth from the left in the drawing is applied to the pressure sensor 511-m-6. The load g corresponding to the weight of the portion of the object 512 located seventh from the left in the drawing is applied to the pressure sensor 511-m-7.

[0574] The weight data output from the pressure area sensor 501 includes the pressure sensors 511-1-1 through 511-MN.
, And a value indicating the weight arranged two-dimensionally in the spatial direction.

FIG. 76 shows that when the pressure area sensor 501 has an integrating effect and the object 512 is moving,
FIG. 4 is a diagram illustrating an example of weight data output by a pressure area sensor 501.

The pressure sensor 511-m-1 applies a load a corresponding to the weight of the leftmost portion of the object 512 in the figure in the unit time to be measured, and indicates the weight included in the weight data. The value a is output as the value.

[0577] The pressure sensor 511-m-2 outputs the object 512 from the left in the figure in the unit time to be measured.
A load b corresponding to the weight of the portion located at the second position is added, and then a load a corresponding to the weight of the leftmost portion of the object 512 in the drawing is added. The value a + b is output as the value.

[0578] The pressure sensor 511-m-3 detects the object 512 from the left in the figure in the unit time to be measured.
A load c corresponding to the weight of the portion positioned second is added, and a load b corresponding to the weight of the portion positioned second from the left of the object 512 in the drawing is added. Since a load a corresponding to the weight of the portion located on the left is applied, a value a + b + c is output as a value indicating the weight included in the weight data.

[0579] The pressure sensor 511-m-4 outputs the object 512 from the left in the figure in the unit time to be measured.
A load d corresponding to the weight of the third position is added, and a load c corresponding to the weight of the third position of the object 512 from the left in the figure is added. A load b corresponding to the weight of the portion located at the third position is applied, and then the object 51
Since a load a corresponding to the weight of the leftmost portion in FIG. 2 is applied, a value a + b + c + d is output as a value indicating the weight included in the weight data.

[0580] The pressure sensor 511-m-5 detects the object 512 from the left in the figure in the unit time to be measured.
A load e corresponding to the weight of the portion located at the third position is added, and a load d corresponding to the weight of the fourth portion of the object 512 from the left in the drawing is added. A load c corresponding to the weight of the portion located at the third position is applied, and then the object 51
Since a load b corresponding to the weight of the second portion from the left in FIG. 2 is applied, a value b + c + d + e is output as a value indicating the weight included in the weight data.

[0581] The pressure sensor 511-m-6 detects the object 512 from the left in the figure in the unit time to be measured.
A load f corresponding to the weight of the portion positioned at the nth position is added, and a load e corresponding to the weight of the portion positioned at the fifth position of the object 512 from the left in the drawing is added. A load d corresponding to the weight of the third position is applied, and then the object 51
Since a load c corresponding to the weight of the third position from the left in FIG. 2 is applied, a value c + d + e + f is output as a value indicating the weight included in the weight data.

[0582] The pressure sensor 511-m-7 detects the object 512 from the left in the figure in the unit time to be measured.
A load g corresponding to the weight of the portion located at the nth position is added, and a load f corresponding to the weight of the portion located at the sixth position of the object 512 from the left in the figure is added. A load e corresponding to the weight of the third position is applied, and then the object 51
Since a load d corresponding to the weight of the fourth position from the left in FIG. 2 is applied, a value d + e + f + g is output as a value indicating the weight included in the weight data.

[0583] The pressure sensor 511-m-8 detects the object 512 from the left in the figure in the unit time to be measured.
A load g corresponding to the weight of the portion located at the nth position is added, and a load f corresponding to the weight of the portion located at the sixth position of the object 512 from the left in the figure is added. Since a load e corresponding to the weight of the portion located at the second position is applied, a value e + f + g is output as a value indicating the weight included in the weight data.

[0584] The pressure sensor 511-m-9 detects the object 512 from the left in the figure in the unit time to be measured.
A load g corresponding to the weight of the portion located at the nth position is added, and a load f corresponding to the weight of the portion located at the sixth position from the left of the object 512 in the drawing is added, indicating the weight included in the weight data. The value f + g is output as the value.

The pressure sensor 511-m-10 applies a load g corresponding to the weight of the portion of the object 512 located at the seventh position from the left in the drawing in the unit time to be measured. As a value indicating
Output the value g.

[0586] The pressure area sensor 501 is
The value a output from 11-m-1 and the pressure sensor 511-m-
2, a + b + c output by the pressure sensor 511-m-3, a + b + c + d output by the pressure sensor 511-m-4, b + c + d + e output by the pressure sensor 511-m-5, and the pressure sensor 511- The value c + d + e + f output by m-6, the pressure sensor 511-m-7
Output value d + e + f + g, pressure sensor 511-m−
8, the output e + f + g, the pressure sensor 511-m-9
And the pressure sensor 511-m-1
The gravity data including the value g output by 0 is output.

[0587] The signal processing unit 502
From the weight data supplied from 01, the distortion generated by the movement of the object 512 to be measured is adjusted. For example, the signal processing unit 502 extracts a more accurate weight of a desired portion of the object 512 to be measured. For example, the signal processing unit 502 calculates the value a, the value a +
b, value a + b + c, value a + b + c + d, value b + c + d +
e, value c + d + e + f, value d + e + f + g, value e + f +
g, the value f + g, and the weight data
a, load b, load c, load d, load e, load f, and load g
Is extracted.

With reference to the flowchart in FIG. 77, a description will be given of a load calculation process executed by the signal processing unit 502.

In step S401, the signal processing unit 5
02 acquires the weight data output from the pressure area sensor 501. In step S402, the signal processing unit 5
02 determines whether the load of the object 512 is applied to the pressure area sensor 501 based on the weight data acquired from the pressure area sensor 501, and determines that the load of the object 512 is applied to the pressure area sensor 501. If so, the process advances to step S403 to acquire the movement of the object 512 based on the change in the weight data.

[0590] In step S404, the signal processing unit 5
In step 02, data for one line of the pressure sensor 511 included in the weight data is acquired along the direction of movement acquired in the process of step S403.

[0591] In step S405, the signal processing unit 5
02 calculates the load corresponding to the weight of each part of the object 512, and the processing ends. The signal processing unit 502
For example, a load corresponding to the weight of each part of the object 512 is calculated by the same processing as the processing described with reference to the flowchart in FIG.

If it is determined in step S402 that the load of the object 512 is not applied to the pressure area sensor 501, there is no weight data to be processed, and the process ends.

As described above, the weight measuring system can calculate an accurate load corresponding to the weight of each part of the moving object.

Next, the signal processing unit 12 for generating a higher resolution image in the spatial direction will be described.

FIG. 78 is a block diagram showing a configuration for generating a high-resolution image by increasing the number of pixels per frame as another function of the signal processing unit 12.

The frame memory 701 stores an input image in frame units, and stores the stored image in the pixel value generation unit 7.
02 and the correlation calculation unit 703.

[0597] The correlation operation section 703 is provided in the frame memory 70.
The pixel value generation unit 70 calculates a correlation value between pixel values of horizontally adjacent pixel data included in the image supplied from No. 1
Feed to 2. The pixel value generation unit 702 includes a correlation calculation unit 70
Based on the pixel values of the three pieces of pixel data arranged side by side based on the correlation value supplied from No. 3, a component of a double-density image in the horizontal direction is calculated from the pixel value of the central pixel, and is calculated as the pixel value To generate a horizontal double-density image. The pixel value generation unit 702 supplies the generated horizontal double-density image to the frame memory 704.

[0598] The frame memory 704 stores the pixel value
The horizontal density image supplied from 02 is stored in frame units, and the stored horizontal density image is supplied to the pixel value generation unit 705 and the correlation calculation unit 706.

[0599] The correlation operation unit 706 is provided in the frame memory 70.
The correlation value of pixel values of pixel data adjacent in the vertical direction included in the horizontal double-density image supplied from 4 is calculated and supplied to the pixel value generation unit 705. The pixel value generation unit 705 uses the correlation values supplied from the correlation calculation unit 703 to calculate the density of the vertically doubled image from the pixel value of the central pixel based on the pixel values of the three pixel data arranged vertically. The component is calculated, and a double-density image is generated by using the component as a pixel value. The pixel value generation unit 705 outputs the generated double-density image.

Next, a process of generating a horizontal double-density image by the pixel value generation unit 702 will be described.

FIG. 79 is a view for explaining the arrangement of the pixels provided in the sensor 11, which is a CCD, and the area corresponding to the pixel data of the horizontal double-density image, corresponding to FIG. In FIG. 79, A to I indicate individual pixels.
The regions a to r are light receiving regions in which each of the pixels A to I is vertically halved. When the width of the light receiving area of the pixels A to I is 2l, the width of the areas a to r is l. The pixel value generation unit 702 calculates pixel values of pixel data corresponding to the regions a to r.

FIG. 80 is a view for explaining pixel data corresponding to light input to the areas a to r. F '(x) in FIG. 80
Corresponds to the input light and the minute spatial section,
This shows an ideal pixel value when viewed spatially.

Assuming that the pixel value of one pixel data is represented by a uniform integration of the ideal pixel value f ′ (x), the pixel value Y1 of the pixel data corresponding to the region i is expressed by the formula (1) 72),
The pixel value Y2 of the pixel data corresponding to the area j is given by the equation (73)
And the pixel value Y3 of the pixel E is represented by Expression (74).

[0604]

(Equation 18)

[0605]

[Equation 19]

[0606]

(Equation 20) In Equations (72) to (74), x1, x2, and x3
Are the spatial coordinates of the respective boundaries of the light receiving area of the pixel E, the area i, and the area j.

By transforming equation (74), equation (7)
5) and equation (76) can be derived.

Y1 = 2 · Y3-Y2 (75) Y2 = 2 · Y3-Y1 (76) Therefore, if the pixel value Y3 of the pixel E and the pixel value Y2 of the pixel data corresponding to the area j are known, the following equation is obtained. According to (75), the pixel value Y1 of the pixel data corresponding to the area i can be calculated. If the pixel value Y3 of the pixel E and the pixel value Y1 of the pixel data corresponding to the region i are known, the pixel value Y2 of the pixel data corresponding to the region j can be calculated by the equation (76).

As described above, if the pixel value corresponding to the pixel and one of the pixel values of the pixel data corresponding to the two regions of the pixel can be known, the other values corresponding to the two regions of the pixel can be obtained. The pixel value of the pixel data can be calculated.

Referring to FIG. 81, calculation of pixel values of pixel data corresponding to two regions of one pixel will be described. FIG. 81A shows a relationship between a pixel D, a pixel E, and a pixel F and a spatially ideal pixel value f ′ (x).

The pixels D, E and F have a spatial integration effect, and one pixel outputs one pixel value. Therefore, as shown in FIG. Output the pixel value. The pixel value output by the pixel E corresponds to a value obtained by integrating the pixel value f ′ (x) in the range of the light receiving region.

The correlation operation unit 703 calculates the correlation value between the pixel value of the pixel D and the pixel value of the pixel E, and the pixel value of the pixel E and the pixel value of the pixel F
Then, a correlation value with the pixel value is generated and supplied to the pixel value generation unit 702. The correlation value calculated by the correlation calculator 703 is calculated based on, for example, a difference value between the pixel value of the pixel D and the pixel value of the pixel E, or a difference value between the pixel value of the pixel E and the pixel value of the pixel F. You. When the pixel values of adjacent pixels are closer,
It can be said that the correlation between those pixels is stronger. That is, a smaller value of the pixel value difference value indicates a stronger correlation.

Therefore, when the difference value between the pixel value of the pixel D and the pixel value of the pixel E or the difference value between the pixel value of the pixel E and the pixel value of the pixel F is directly used as a correlation value, a smaller difference value is obtained. A correlation value of indicates a stronger correlation.

For example, when the correlation between the pixel value of the pixel D and the pixel value of the pixel E is stronger than the correlation between the pixel value of the pixel E and the pixel value of the pixel F, as shown in FIG. Value generator 7
In the case of 02, the pixel value of the pixel D is divided by 2, and the resulting value is used as the pixel data of the area i.

As shown in FIG. 81 (D), the pixel value generation unit 702 calculates the pixel E by the equation (75) or (76).
The pixel value of the pixel data of the area j is calculated based on the pixel value of the pixel data of the area i and the pixel value of the pixel data of the area i.

The pixel value generation unit 702 calculates the pixel value of the pixel data of the area g and the pixel value of the pixel data of the area h for the pixel D, and calculates the pixel data of the area i for the pixel E. The pixel value of the pixel data of the area j, and the pixel value of the pixel data of the area j are calculated. The pixel values of the pixel data are sequentially calculated for the pixels in the screen as described above, a horizontal double-density image including the calculated pixel values of the pixel data is generated, and the generated horizontal double-density image is supplied to the frame memory 704. I do.

The pixel value generation unit 705 is
Similarly to 2, from the correlation between the pixel values of the three pixels arranged vertically and the pixel values of the three pixels of the horizontal double-density image supplied from the correlation operation unit 706, two light receiving areas of the pixels are vertically arranged. A double-density image is generated by calculating the pixel value of the image data corresponding to the divided region.

When the image shown in FIG. 82 is an input image, the pixel value generating section 702 generates a horizontal double-density image shown in FIG. 83.

The pixel value generation unit 705 generates an image shown in FIG. 84 when the image shown in FIG. 82 is input, and generates a horizontal double-density image shown in FIG.
A double-density image whose example is shown in FIG. 85 is generated.

FIG. 86 is a flowchart for explaining the process of generating a double-density image by the signal processing section 12 having the configuration shown in FIG. In step S601, the signal processing unit 12
Acquires an input image and stores it in the frame memory 701.

[0621] In step S602, the correlation operation unit 7
03 selects one pixel in the screen as a pixel of interest,
Based on the pixel values stored in the frame memory 701,
A correlation value of a pixel horizontally adjacent to the target pixel is obtained. In step S603, the pixel value generation unit 702
Generates the pixel value of the pixel data on one side of the horizontal double-density image from the pixel value with the stronger correlation, that is, the larger correlation value, based on the correlation value supplied from the correlation calculator 703.

In step S604, the pixel value generation unit 702 generates pixel values of other pixel data of the horizontal double-density image based on the characteristics of the CCD. Specifically, the pixel value generation unit 702 calculates the pixel value calculated in step S603 and the pixel data of the input image based on Expressions (75) and (76) described with reference to FIG. Based on the pixel value,
A pixel value of another image data of the horizontal double-density image is calculated. The pixel data of the horizontal double-density image corresponding to the target pixel, which is generated in the processing of steps S603 and S604, is stored in the frame memory 704.

[0623] In step S605, the pixel value generation unit 702 determines whether or not processing of the entire screen has been completed.
If it is determined that the processing of the entire screen has not been completed, the procedure returns to step S602, the next pixel is selected as the target pixel, and the processing of generating a horizontal double-density image is repeated.

If it is determined in step S605 that the processing of the entire screen has been completed, the process proceeds to step S606, where the correlation calculation unit 706 selects one pixel in the screen as a pixel of interest and stores it in the frame memory 704. Based on the pixel values of the horizontal double-density image, a correlation value of a pixel vertically adjacent to the target pixel is obtained. Step S60
7, the pixel value generation unit 705 includes a correlation calculation unit 706
The pixel value of one side of the double-density image is generated from the pixel value having a strong correlation based on the correlation value supplied from.

In step S608, the pixel value generation unit 705 generates other pixel values of the double-density image based on the characteristics of the CCD as in step S604. Specifically, the pixel value generation unit 702 calculates the equation (7) described with reference to FIG.
Based on 5) and equation (76), the pixel value of the other image data of the double-density image is calculated based on the pixel value calculated in the process of step S607 and the pixel value of the pixel data of the horizontal double-density image. .

[0626] In step S609, the pixel value generation unit 705 determines whether or not processing of the entire screen has been completed.
If it is determined that the processing of the entire screen has not been completed, the procedure returns to step S606, the next pixel is selected as the target pixel, and the processing of generating a double-density image is repeated.

If it is determined in step S609 that the processing of the entire screen has been completed, the pixel value generation unit 705
Outputs the generated double-density image, and the process ends.

Thus, the signal processing section 12 shown in FIG. 78 doubles the number of pixels in the vertical direction of the image and doubles the number of pixels in the horizontal direction from the input image. A dense image can be generated.

As described above, the signal processing unit 12 shown in FIG. 78 generates a spatially high-resolution image by performing signal processing in consideration of pixel correlation and integration effect on CCD space. be able to.

In the above description, an example is given in which an image of a real space having three-dimensional space and time axis information is projected onto a two-dimensional space and time and space having time axis information using a video camera. The present invention is not limited to this example, and corrects the distortion caused by the projection when more first-dimensional first information is projected onto less second-dimensional second information. It can be applied to extracting significant information, or combining images more naturally.

[0631] The sensor 11 is a solid-state image sensor, not limited to a CCD, for example, a BBD (Bucket Brigade Device).
e), CID (Charge Injection Device), or CPD (Ch
arge Priming Device).
The sensor is not limited to a sensor in which the detection elements are arranged in a matrix, but may be a sensor in which the detection elements are arranged in one row.

As shown in FIG. 3, the recording medium on which the program for performing the signal processing of the present invention is recorded is a magnetic disk on which the program is recorded, which is distributed separately from the computer in order to provide the program to the user. 51 (including floppy disk), optical disk 52 (CD-ROM (Com
paut Disk-Read Only Memory), DVD (Digital VersatileD
isk)), magneto-optical disk 53 (MD (Mini-Dis
k)), or a package medium including a semiconductor memory 54 or the like, and also provided to a user in a state where the program is incorporated in a computer in advance.
8 and the like.

In this specification, the step of describing a program recorded on a recording medium is not limited to processing performed in chronological order according to the described order, but is not necessarily performed in chronological order. This also includes processing executed in parallel or individually.

[0634]

As described above, according to the signal processing device of claim 1, the signal processing method of claim 27, and the recording medium of claim 28, the real world having the first dimension A first signal, which is a signal of
A second signal of a second dimension smaller than the first dimension obtained by detection by the sensor is obtained, and by performing signal processing based on the second signal, significant information buried by projection is converted to a second signal. Since the information is extracted from the signal, significant information can be extracted.

According to the signal processing apparatus of claim 29, the signal processing method of claim 44, and the recording medium of claim 45, the first signal which is a real-world signal having a first dimension. A second signal having a second dimension, which is smaller in dimension than the first dimension and obtained by detecting the signal of the second signal and having a distortion with respect to the first signal, is obtained. By performing the signal processing based on the third signal, the third signal whose distortion is reduced as compared with the second signal is generated, so that the distortion of the signal can be reduced.

According to the signal processing apparatus of claim 46, the signal processing method of claim 48, and the recording medium of claim 49, only the foreground object component constituting the foreground object is included in the detection signal. , A background region consisting only of the background object component constituting the background object, and a mixed region formed by mixing the foreground object component and the background object component are specified, and at least the foreground object component and the background in the mixed region The mixture ratio with the object component is detected,
Based on the specific result and the mixture ratio, the foreground object,
Since the background object is separated from the background object, the foreground object and the background object, which are higher quality data, can be used.

According to the signal processing device of claim 50, the signal processing method of claim 54, and the recording medium of claim 55, only the foreground object component constituting the foreground object is included in the detection signal. , A background region consisting of only the background object component constituting the background object, and a mixed region formed by mixing the foreground object component and the background object component are specified. Based on the specified result, at least the mixed region Since the mixture ratio of the foreground object component and the background object component in is detected, the mixture ratio that is significant information can be extracted.

According to the signal processing device of claim 56, the signal processing method of claim 59, and the recording medium of claim 60, the foreground object component forming the foreground object and the background object forming the foreground object component In the mixed area where the background object component
Since the mixture ratio between the foreground object component and the background object component is detected and the foreground object and the background object are separated based on the mixture ratio, the foreground object and the background object that are higher quality data are used. Will be able to

[Brief description of the drawings]

FIG. 1 is a diagram illustrating the principle of the present invention.

FIG. 2 is a block diagram illustrating a configuration example of a system to which the present invention is applied.

FIG. 3 is a block diagram illustrating a configuration example of a signal processing unit in FIG. 2;

FIG. 4 is a flowchart illustrating the operation of the system in FIG. 2;

FIG. 5 is a diagram showing an example of an image acquired in step S1 of FIG.

FIG. 6 is a diagram illustrating pixel values of a mixed area.

FIG. 7 is a diagram illustrating a result obtained by subtracting a background image component in sections D1 to D3 of FIG. 6;

FIG. 8 is a diagram illustrating the structure of motion blur.

FIG. 9 is a flowchart illustrating another processing example of the system in FIG. 2;

FIG. 10 is a block diagram showing a signal processing unit 12;

FIG. 11 is a diagram illustrating imaging by a sensor.

FIG. 12 is a diagram illustrating an arrangement of pixels.

FIG. 13 is a diagram illustrating the operation of the detection element.

FIG. 14 shows an object corresponding to a moving foreground;
FIG. 3 is a diagram illustrating an image obtained by capturing an object corresponding to a stationary background.

FIG. 15 is a diagram illustrating a background area, a foreground area, a mixed area, a covered background area, and an uncovered background area.

FIG. 16 is a model diagram in which pixel values of adjacent pixels arranged in one column in an image obtained by capturing an object corresponding to a stationary foreground and an object corresponding to a stationary background are developed in the time direction. It is.

FIG. 17 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 18 is a model diagram in which pixel values are developed in a time direction and a period corresponding to a shutter time is divided.

FIG. 19 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 20 is a diagram illustrating an example in which pixels in a foreground area, a background area, and a mixed area are extracted.

FIG. 21 is a diagram showing correspondence between pixels and models in which pixel values are developed in the time direction.

FIG. 22 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 23 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 24 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 25 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 26 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 27 is a flowchart illustrating a process of adjusting the amount of motion blur.

FIG. 28 is a block diagram illustrating an example of a configuration of an area specifying unit 103.

FIG. 29 is a diagram illustrating an image when an object corresponding to the foreground is moving.

FIG. 30 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 31 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 32 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 33 is a diagram illustrating conditions for region determination.

FIG. 34 is a diagram illustrating an example of a result of specifying an area by the area specifying unit 103;

35 is a diagram illustrating an example of a result of specifying an area by the area specifying unit 103. FIG.

FIG. 36 is a flowchart illustrating an area specifying process.

FIG. 37 is a block diagram illustrating an example of a configuration of a mixture ratio calculation unit 104.

FIG. 38 is a diagram showing an example of an ideal mixture ratio α.

FIG. 39 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 40 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 41 is a diagram illustrating approximation using correlation of foreground components.

FIG. 42 is a diagram illustrating the relationship between C, N, and P.

FIG. 43 is a block diagram illustrating a configuration of an estimated mixture ratio processing unit 201.

FIG. 44 is a diagram illustrating an example of an estimated mixture ratio.

FIG. 45 is a block diagram showing another configuration of the mixture ratio calculation unit 104.

FIG. 46 is a flowchart illustrating a process of calculating a mixture ratio.

FIG. 47 is a flowchart illustrating a process of calculating an estimated mixture ratio.

FIG. 48 is a block diagram illustrating an example of a configuration of a foreground / background separation unit 105.

FIG. 49 is a diagram showing an input image, a foreground component image, and a background component image.

FIG. 50 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 51 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 52 is a model diagram in which pixel values are developed in the time direction and a period corresponding to a shutter time is divided.

FIG. 53 is a block diagram illustrating an example of a configuration of a separation unit 251.

FIG. 54 is a diagram illustrating an example of a separated foreground component image and a background component image.

FIG. 55 is a flowchart illustrating a process of separating a foreground and a background.

FIG. 56 is a block diagram illustrating an example of a configuration of a motion blur adjustment unit 106.

FIG. 57 is a diagram illustrating a processing unit.

FIG. 58 expands the pixel values of the foreground component image in the time direction,
FIG. 3 is a model diagram obtained by dividing a period corresponding to a shutter time.

FIG. 59 expands pixel values of a foreground component image in the time direction,
FIG. 3 is a model diagram obtained by dividing a period corresponding to a shutter time.

FIG. 60 expands pixel values of a foreground component image in the time direction,
FIG. 3 is a model diagram obtained by dividing a period corresponding to a shutter time.

FIG. 61 expands pixel values of a foreground component image in the time direction,
FIG. 3 is a model diagram obtained by dividing a period corresponding to a shutter time.

62 is a diagram illustrating another configuration of the motion blur adjustment unit 106. FIG.

FIG. 63 is a flowchart illustrating a process of adjusting the amount of motion blur included in the foreground component image by the motion blur adjustment unit 106.

FIG. 64 is a block diagram illustrating another configuration of the function of the signal processing unit 12.

FIG. 65 is a diagram illustrating a configuration of a synthesis unit 371.

FIG. 66 is a block diagram showing still another configuration of the function of the signal processing unit 12;

FIG. 67 is a block diagram illustrating a configuration of a mixture ratio calculation unit 401.

68 is a block diagram illustrating a configuration of a foreground / background separation unit 402. FIG.

FIG. 69 is a block diagram showing another configuration of the function of the signal processing unit 12.

FIG. 70 is a diagram illustrating a configuration of a combining unit 431.

FIG. 71 is a diagram illustrating a configuration example of a signal processing device according to the present invention.

FIG. 72 is a flowchart illustrating processing for adjusting the amount of motion blur by the signal processing unit 452.

FIG. 73 is a diagram illustrating a configuration example of a signal processing device according to the present invention.

74 is a diagram showing a configuration of a pressure area sensor 501. FIG.

FIG. 75 is a diagram illustrating a load applied to the pressure area sensor 501.

FIG. 76 is a diagram illustrating an example of weight data output by a pressure area sensor 501.

FIG. 77 is a flowchart illustrating a load calculation process performed by a signal processing unit 502;

FIG. 78 is a block diagram showing a configuration for generating an image in which the number of pixels per frame is increased as another function of the signal processing unit 12.

FIG. 79 is a diagram illustrating an arrangement of pixels and an area corresponding to a pixel of the horizontal double-density image.

FIG. 80 is a diagram illustrating components of an image corresponding to light input to regions a to r.

FIG. 81 is a diagram illustrating calculation of components of an image corresponding to two regions of one pixel.

FIG. 82 is a diagram illustrating an example of an input image.

Fig. 83 is a diagram illustrating an example of a horizontal double-density image.

FIG. 84 is a diagram illustrating an example of a vertically doubled image.

FIG. 85 is a diagram illustrating an example of a double-density image.

86 is a flowchart illustrating a process of generating a double-density image by the signal processing unit 12 illustrated in FIG. 78.

[Explanation of symbols]

11 sensors, 12 signal processing units, 21 CPUs,
22 ROM, 23 RAM, 26 input unit, 27 output unit, 28 storage unit, 29 communication unit, 51 magnetic disk, 52 optical disk, 53 magneto-optical disk, 54 semiconductor memory, 61 background, 62 foreground, 63 mixed area, 101 object Extraction unit, 102 motion detection unit, 103 area identification unit,
104 mixture ratio calculator, 105 foreground / background separator,
106 motion blur adjustment unit, 107 selection unit, 121
Frame memory, 122-1 to 122-4 static / moving determining section, 123-1 to 123-3 area determining section, 1
24 determination flag storage frame memory, 125 synthesis unit, 126 determination flag storage frame memory, 201
Estimated mixture ratio processor, 202 Estimated mixture ratio processor,
203 mixing ratio determination unit, 221 frame memory,
222 frame memory, 223 mixture ratio calculator,
231 selection unit, 232 estimated mixture ratio processing unit, 23
3 Estimated mixture ratio processing section, 234 Selection section, 251
Separation unit, 252 switch, 253 synthesis unit, 2
54 switches, 255 synthesis unit, 301 frame memory, 302 separation processing block, 303 frame memory, 311 uncovered area processing unit,
312 Covered area processing unit, 313 synthesis unit, 31
4 synthesis unit, 351 processing unit determination unit, 352 modeling unit, 353 equation generation unit, 354 addition unit, 355 calculation unit, 356 motion blur addition unit,
357 selection section, 361 selection section, 371 synthesis section, 381 background component generation section, 382 mixed area image synthesis section, 383 image synthesis section, 401 mixing ratio calculation section, 402 foreground / background separation section, 421 selection section,
431 synthesis section, 441 selection section, 451 thermography device, 452 signal processing section, 501 pressure area sensor, 502 signal processing section, 511-1-
1 to 511-MN pressure sensor, 701 frame memory, 702 pixel value generation unit, 703 correlation operation unit, 704 frame memory, 705 pixel value generation unit, 706 correlation operation unit

 ──────────────────────────────────────────────────続 き Continuing from the front page (72) Inventor Junichi Ishibashi 6-7-35 Kita-Shinagawa, Shinagawa-ku, Tokyo Inside Sony Corporation (72) Takashi Sawao 6-35, Kita-Shinagawa, Shinagawa-ku, Tokyo Inside Sony Corporation (72) Inventor Naoki Fujiwara 6-7-35 Kita-Shinagawa, Shinagawa-ku, Tokyo Inside Sony Corporation (72) Inventor Takahiro Nagano 6-7-35 Kita-Shinagawa, Shinagawa-ku, Tokyo Soni -Inside Co., Ltd. (72) Inventor Toru Miyake 6-35 Kita Shinagawa, Shinagawa-ku, Tokyo Inside Sony Corporation

Claims (60)

[Claims] A first signal, which is a real-world signal having a first dimension, is projected onto a sensor, and a second signal having a second dimension less than the first dimension obtained by detection by the sensor. And a signal processing unit for performing signal processing based on the second signal to extract significant information buried by projection from the second signal. Signal processing device. 2. The signal processing apparatus according to claim 1, wherein the significant information is information for adjusting distortion caused by projection. 3. The sensor according to claim 1, wherein the sensor includes a plurality of detection elements each having a time integration effect, and the acquisition unit outputs a plurality of detection signals corresponding to each of the detection elements detected by the sensor. The signal processing apparatus according to claim 2, wherein the distortion is obtained as the second signal, and the distortion is a distortion due to a time integration effect. 4. The method according to claim 1, wherein the obtaining unit obtains the plurality of detection signals in a plurality of time units detected by a plurality of detection elements of the sensor for each predetermined time unit. The signal processing apparatus according to claim 3, wherein the significant information for the second signal at a desired time is extracted based on a plurality of detection signals in units. 5. The signal processing device according to claim 1, wherein the second signal is an image signal. 6. The signal processing unit specifies a significant region including significant information buried by the projection and another region in the second signal, and converts the region information indicating the specified region into the significant region. The signal processing apparatus according to claim 1, further comprising an area specifying unit that outputs the information as information. 7. The area information includes a foreground area, which is another area only including a foreground object component forming a foreground object, and a background area, which is another area including only a background object component forming a background object. 7. The signal processing device according to claim 6, wherein the signal processing device indicates a mixed region that is the significant region obtained by mixing the foreground object component and the background object component. 8. The signal processing apparatus according to claim 7, wherein the area information includes information for identifying the mixed area as a covered background area and an uncovered background area. 9. The apparatus according to claim 6, wherein the signal processing unit further includes a significant information extracting unit that extracts the significant information from a region including the significant information specified by the region specifying unit. Signal processing device. 10. The significant information includes a foreground area consisting only of a foreground object component constituting a foreground object, a background area consisting only of a background object component constituting a background object, the foreground object component and the background object component. The signal processing apparatus according to claim 9, wherein the signal processing device indicates a mixing ratio of the foreground object component and the background object component in the mixing region in the second signal including a mixing region formed by mixing the two. 11. The apparatus according to claim 11, wherein the signal processing unit further includes a distortion amount adjusting unit that adjusts an amount of distortion generated in the second signal by the projection based on the significant information. 10. The signal processing device according to 9. 12. The signal processing device according to claim 11, wherein the distortion amount adjusting unit reduces the amount of the distortion. 13. The signal processing apparatus according to claim 11, wherein the distortion amount adjusting unit removes the distortion. 14. The signal processing apparatus according to claim 11, wherein the distortion is a motion blur occurring in the foreground object. 15. The signal processing means further comprises an object motion detecting means for detecting a motion amount of the foreground object, and the distortion amount adjusting means comprises a motion which is the distortion based on the motion amount of the foreground object. The signal processing device according to claim 14, wherein the amount of blur is adjusted. 16. The signal processing means includes: a foreground region consisting of only a foreground object component constituting a foreground object; a background region consisting of only a background object component constituting a background object; the foreground object component and the background object component. And extracting, as the significant information, a mixture ratio of the foreground object component and the background object component in the mixed region in the second signal including a mixed region obtained by mixing the foreground object component and the background object component. 2. The signal processing device according to 1. 17. The apparatus according to claim 1, wherein the signal processing unit further includes a distortion amount adjusting unit that adjusts an amount of distortion generated in the second signal by the projection based on the significant information. The signal processing device according to item 16. 18. The signal processing device according to claim 17, wherein the distortion amount adjusting unit reduces the amount of the distortion. 19. The signal processing device according to claim 17, wherein the distortion amount adjusting unit removes the distortion. 20. The signal processing apparatus according to claim 17, wherein the distortion is a motion blur occurring in the foreground object. 21. The signal processing means further comprises an object motion detecting means for detecting a motion amount of the foreground object, and the distortion amount adjusting means comprises a motion which is the distortion based on the motion amount of the foreground object. 21. The signal processing device according to claim 20, wherein the amount of blur is adjusted. 22. The signal processing unit extracts the mixture ratio as the significant information based on the second signal, and the signal processing unit extracts the foreground region and the background from the second signal. Area information indicating an area and the mixed area,
2. The method according to claim 1, wherein the information is extracted as the significant information.
7. The signal processing device according to 6. 23. The apparatus according to claim 23, wherein the signal processing unit separates the foreground object including only the foreground object component from the background object including only the background object component based on the mixture ratio and the area information. The signal processing device according to claim 22, characterized in that: 24. The apparatus according to claim 23, wherein the signal processing unit further includes a distortion amount adjusting unit that adjusts a motion blur as the distortion occurring in the foreground object.
The signal processing device according to claim 1. 25. The signal processing means further comprises an object motion detecting means for detecting a motion amount of the foreground object, and the distortion amount adjusting means comprises a motion which is the distortion based on the motion amount of the foreground object. The signal processing device according to claim 24, wherein the amount of blur is adjusted. 26. The signal processing apparatus according to claim 23, wherein the signal processing unit combines the foreground object with an arbitrary background image. 27. A first signal, which is a real-world signal having a first dimension, is projected onto a sensor, and a second signal less than the first dimension obtained by detecting with the sensor.
And a signal processing step of performing signal processing based on the second signal to extract significant information buried by projection from the second signal. A signal processing method characterized by the above-mentioned. 28. A first signal, which is a real-world signal having a first dimension, is projected onto a sensor, and a second signal less than the first dimension obtained by detecting with the sensor.
And a signal processing step of performing signal processing based on the second signal to extract significant information buried by projection from the second signal. A recording medium on which a computer-readable program is recorded. 29. having a second dimension, obtained by detecting a first signal, which is a real-world signal having a first dimension, with a sensor, the dimension being smaller than the first dimension, Acquiring means for acquiring a second signal including distortion with respect to the first signal; and performing signal processing based on the second signal to reduce the distortion with respect to the second signal. 3. A signal processing device comprising: signal processing means for generating the signal of (3). 30. The sensor, comprising: a plurality of detection elements each having a time integration effect as the distortion, wherein the acquisition unit includes a plurality of detection elements corresponding to each of the detection elements detected by the sensor. Is obtained as the second signal, and the signal processing means performs signal processing based on the second signal, thereby reducing a time integration effect and corresponding to the plurality of detection signals. 30. The method according to claim 29, further comprising: generating the third signal comprising the sample data of
The signal processing device according to claim 1. 31. When the sensor detects a first object in the real world and a second object that moves relatively to the first object, the signal processing unit executes the second object. In the vicinity of a boundary between the first object and the second object represented by a signal, distortion caused by mixing of the first object and the second object caused by a time integration effect of the sensor is represented by the signal. 31. The signal processing apparatus according to claim 30, wherein the signal processing apparatus reduces the processing. 32. The acquisition unit acquires the detection signals of a plurality of time units detected by a plurality of detection elements of the sensor for each predetermined time unit, and the signal processing unit acquires a plurality of the time signals. The second unit corresponding to the desired time unit based on the detection signal of the unit;
32. The signal processing device according to claim 31, wherein the signal processing reduces distortion near the boundary between the first object and the second object represented by the following signal. 33. When the sensor detects a first object in the real world and a second object that moves relatively to the first object, the signal processing unit performs the second object processing. One of the first object and the second object is separated from the first object and the second object mixed in a signal, and the separated first object and the second object are separated. 31. The signal processing apparatus according to claim 30, wherein one of the two objects is output as the third signal. 34. The signal processing according to claim 29, wherein the sensor converts an electromagnetic wave including light, which is the first signal, into an image signal, which is the second signal, by photoelectric conversion. apparatus. 35. The signal processing device according to claim 29, wherein the sensor is a thermography device for measuring a temperature. 36. The signal processing device according to claim 29, wherein the sensor is a pressure sensor. 37. The signal processing apparatus according to claim 29, wherein the sensor generates the second signal at predetermined time intervals. 38. The sensor according to claim 38, wherein the sensor includes a plurality of detection elements each having a spatial integration effect, wherein the acquisition unit outputs a plurality of detection signals corresponding to each of the detection elements detected by the sensor. The signal processing unit obtains the second signal, and performs signal processing based on the second signal, so that the second signal includes a plurality of detection signals in which a spatial integration effect due to the projection as the distortion is reduced. 30. The signal processing apparatus according to claim 29, wherein the signal processing apparatus generates three signals. 39. The sensor, wherein a plurality of the plurality of detection elements are arranged in a predetermined direction which is at least one direction, and wherein the signal processing unit is configured to perform processing on each sample data of the second signal. A correlation detecting means for respectively detecting a correlation between two sample data of the second signal adjacent in the predetermined direction; and for each sample data of interest, of the two adjacent sample data, Generating, based on the sample value of the sample data having the larger correlation, a first sample value which is a sample value on the sample data side having the larger correlation, and generating a sample value of the sample data of interest and the first sample value; And generating a second sample value that is a sample value on the smaller correlation side based on the first sample value and the second sample value. 39. The signal processing apparatus according to claim 38, further comprising: a double-density sample generating unit that outputs two sample values of the third signal corresponding to the sample of interest. 40. The sensor, wherein a plurality of the detection elements are arranged in a matrix, and the predetermined direction is at least one of a horizontal direction and a vertical direction in the matrix arrangement. The signal processing device according to claim 39. 41. The signal processing apparatus according to claim 40, wherein said signal processing means doubles a density in both a horizontal direction and a vertical direction. 42. The signal processing apparatus according to claim 39, wherein the correlation is a difference between the sample data. 43. The signal processing apparatus according to claim 39, wherein the acquisition unit acquires the second signal at predetermined time intervals from the plurality of detection elements. 44. A second dimension having a smaller dimension than the first dimension, obtained by detecting a first signal that is a real-world signal having a first dimension with a sensor, An acquisition step of acquiring a second signal including a distortion with respect to the first signal, and a signal processing based on the second signal, whereby a second signal having the distortion reduced compared to the second signal is obtained. And a signal processing step of generating a third signal. 45. A second dimension having a smaller dimension than the first dimension, obtained by detecting a first signal that is a real-world signal having a first dimension with a sensor, An acquisition step of acquiring a second signal including a distortion with respect to the first signal, and a signal processing based on the second signal, whereby a second signal having the distortion reduced compared to the second signal is obtained. And a signal processing step of generating the signal of (3). A recording medium on which a computer-readable program is recorded. 46. A signal processing device for processing a predetermined number of detection signals obtained by a sensor including a predetermined number of detection elements having a time integration effect, wherein a foreground object constituting a foreground object among the detection signals Region specifying means for specifying a foreground region consisting only of components, a background region consisting only of background object components constituting a background object, and a mixed region consisting of a mixture of the foreground object component and the background object component; and A mixture ratio detection unit that detects a mixture ratio between the foreground object component and the background object component in the mixture region; a foreground object and the background object based on a specification result of the region specification unit and the mixture ratio. Signal separating means for separating Equipment. 47. The signal processing apparatus according to claim 46, further comprising a motion blur amount adjusting means for adjusting a motion blur amount of the foreground object. 48. A signal processing method for processing a predetermined number of detection signals obtained by a sensor including a predetermined number of detection elements having a time integration effect, wherein a foreground object constituting a foreground object among the detection signals An area specifying step of specifying a foreground area consisting of only a component, a background area consisting only of a background object component constituting a background object, and a mixed area obtained by mixing the foreground object component and the background object component; and A mixture ratio detection step of detecting a mixture ratio between the foreground object component and the background object component in the mixture region; and a foreground object based on a result of the process of the region identification step and the mixture ratio. Separating step from the background object A signal processing method characterized by the above-mentioned. 49. A signal processing program for processing a predetermined number of detection signals obtained by a sensor including a predetermined number of detection elements having a time integration effect, wherein a foreground object is included in the detection signals. Area specification for specifying a foreground area consisting only of constituent foreground object components, a background area consisting only of background object components constituting a background object, and a mixed area consisting of a mixture of the foreground object component and the background object component A mixture ratio detecting step of detecting a mixture ratio of the foreground object component and the background object component in at least the mixed region; and a foreground based on a specification result of the process of the region specifying step and the mixture ratio. Separation step for separating the object from the background object A recording medium on which a computer-readable program is recorded. 50. A signal processing apparatus for processing a predetermined number of detection signals obtained by a sensor including a predetermined number of detection elements having a time integration effect, wherein a foreground object constituting a foreground object among the detection signals A region specifying means for specifying a foreground region consisting of only a component, a background region consisting of only a background object component constituting a background object, and a mixed region formed by mixing the foreground object component and the background object component; A signal processing apparatus comprising: a mixture ratio detection unit configured to detect a mixture ratio of the foreground object component and the background object component in at least the mixed region based on a specification result of the region specification unit. 51. The signal processing apparatus according to claim 50, further comprising a separating unit configured to separate the foreground object and the background object based on the mixture ratio. 52. The signal processing apparatus according to claim 50, further comprising a motion blur amount adjusting means for adjusting an amount of motion blur included in the foreground object. 53. The image processing apparatus further includes: a motion detection unit configured to detect a motion of at least one of the foreground object and the background object;
53. The signal processing apparatus according to claim 52, wherein the amount of motion blur is adjusted. 54. A signal processing method for processing a predetermined number of detection signals obtained by a sensor including a predetermined number of detection elements having a time integration effect, wherein a foreground object constituting a foreground object among the detection signals An area specifying step of specifying a foreground area consisting of only a component, a background area consisting only of a background object component constituting a background object, and a mixed area formed by mixing the foreground object component and the background object component; A signal processing method comprising: detecting a mixture ratio of at least the foreground object component and the background object component in the mixed region based on a result of the process of the region specifying step. 55. A signal processing program for processing a predetermined number of detection signals obtained by a sensor including a predetermined number of detection elements having a time integration effect, wherein a foreground object is included in the detection signals. Area specification for specifying a foreground area consisting only of constituent foreground object components, a background area consisting only of background object components constituting a background object, and a mixed area consisting of a mixture of the foreground object component and the background object component And a mixing ratio detecting step of detecting a mixing ratio of the foreground object component and the background object component in at least the mixed region based on a specification result of the process of the region specifying step. Recording medium on which a readable program is recorded . 56. A signal processing device for processing a predetermined number of detection signals obtained by a sensor including a predetermined number of detection elements having a time integration effect, comprising: a foreground object component forming a foreground object;
Mixing ratio detection means for detecting a mixing ratio of the foreground object component and the background object component in the mixed region formed by mixing background object components constituting a background object; and A signal processing apparatus comprising: a separating unit configured to separate an object from the background object. 57. The signal processing apparatus according to claim 56, further comprising a motion blur amount adjusting means for adjusting an amount of motion blur included in the foreground object. 58. The image processing apparatus further comprising: motion detection means for detecting at least one of the foreground object and the background object, wherein the motion-blur adjusting means calculates
58. The signal processing device according to claim 57, wherein an amount of motion blur is adjusted. 59. A signal processing method for processing a predetermined number of detection signals obtained by a sensor including a predetermined number of detection elements having a time integration effect, comprising: a foreground object component forming a foreground object;
A mixing ratio detecting step of detecting a mixing ratio between the foreground object component and the background object component in the mixed region formed by mixing background object components constituting a background object; and A signal processing method, comprising: a separating step of separating an object from the background object. 60. A signal processing program for processing a predetermined number of detection signals obtained by a sensor including a predetermined number of detection elements having a time integration effect, comprising: a foreground object component forming a foreground object; ,
A mixing ratio detecting step of detecting a mixing ratio between the foreground object component and the background object component in the mixed region formed by mixing background object components constituting a background object; and A recording medium in which a computer-readable program is recorded, comprising a separating step of separating an object and the background object.