JP2003018564A - Communication device and method, communication system, recording medium, and program - Google Patents

Abstract

(57) [Summary] [Problem] To freely modify an image. SOLUTION: A motion blur adjusting unit 2101 adjusts a foreground component image separated by a separation processing unit 11 based on a motion blur amount, and outputs the adjusted foreground component image to a combining unit 2102. The background component image separated by the separation processing unit 11 is output to the synthesis unit 2102 as it is. The synthesis unit 2102 synthesizes the motion-blurred image input from the motion-blur adjustment unit 2101 and the motion-blurred background component image directly input from the separation processing unit 11, and outputs the synthesized image. Output as a modified image.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a communication device and method, a communication system, a recording medium, and a program, and a communication device and method, a communication system, and a recording medium in consideration of a difference between a signal detected by a sensor and the real world. ,
And about the program.

[0002]

2. Description of the Related Art A technique of detecting a phenomenon in the real world with a sensor and processing sampling data output from an image sensor is widely used.

For example, in an image obtained by picking up an object moving in front of a predetermined stationary background with a video camera, when the moving speed of the object is relatively high, motion blur occurs.

Further, in the above-described image motion blur removal or adjustment, some motion blur is allowed after adjusting the shutter, and the motion blur may be used as a visual effect, but it is generally more than necessary. When motion blur occurs in the background, a Wiener filter is used to remove this motion blur.

[0005]

When an object moves in front of a stationary background, not only the motion blur caused by the mixture of the images of the moving object itself, but also the background image and the image of the moving object. Mixing occurs. Conventionally, a process corresponding to a state of mixture of a background image and a moving object image has not been considered.

The present invention has been made in view of such a situation, and by setting and adjusting the degree of the mixed state, motion blur is removed and a clear image can be obtained. By synthesizing the image in which the motion blur is adjusted and the background image, a more natural image can be corrected.

[0007]

A first communication device of the present invention comprises request information input means for inputting request information of a user,
Foreground consisting of foreground object components forming a foreground object of image data consisting of pixel values determined for each pixel based on the request information and according to the amount of light forming the image that is temporally integrated. A mixture ratio estimating unit that estimates a mixture ratio indicating a mixture ratio of a component image and a background component image that is a background object component that forms a background object; and, based on the mixture ratio, image data, a foreground component image, A separating unit that separates the background component image, a foreground component image that is separated by the separating unit, an arbitrary background component image, or a background component image that is separated by the separating unit and an arbitrary foreground component image Synthesized by mixing ratio,
It is characterized in that it is provided with a combining means for generating a combined image and an output means for outputting the combined image combined by the combining means.

In addition to the user's request information, a motion blur adjustment amount for adding motion blur to the foreground component image is further input to the request information input means, and the motion blur adjustment amount is added to the foreground component image. It is possible to further provide motion blur adding means for adding corresponding motion blur, and the combining means includes the foreground component image to which motion blur is added by the motion blur adding means and an arbitrary background component image,
It is possible to generate a composite image by combining them at a predetermined mixing ratio.

In the request information input means, in addition to the user's request information, an arbitrary background component image is further input,
The synthesizing unit may synthesize the foreground component image and the arbitrary background component image with the mixture ratio estimated by the mixture ratio estimating unit to generate a synthesized image.

According to the request information, it is possible to further provide a charging means for executing a charging process.

[0011] The billing means generates billing information including an identifier of a user, an identifier of a communication device, and an amount of money corresponding to the request information in accordance with the request information, and based on the billing information, the user A billing process can be executed for a financial account.

It is possible to further provide a point storage means for storing points for each user used for the billing process. The billing means has a user identifier and a communication device identifier according to request information. , And billing information including money amount information corresponding to the composite image is generated, and the billing process can be executed by subtracting the number of points stored in the point storage means by the amount corresponding to the money amount information. .

The output means may output the composite image to the communication device of the user who has completed the charging processing after the charging processing is completed by the charging means.

The request information input means inputs predetermined image data in addition to the user's request information, and the output means completes the charging process by the charging means for the combined image combined by the combining means. After that, it can be output to the communication device of the user who has completed the charging process.

According to a first communication method of the present invention, a request information input step of inputting request information of a user, and a light forming an image integrated pixel by pixel and temporally based on the request information. The ratio of the mixture of the foreground component image consisting of the foreground object component forming the foreground object and the background component image forming the background object component forming the background object of the image data consisting of pixel values determined according to the A mixture ratio estimating step of estimating a mixture ratio shown, based on the mixture ratio, the image data, the foreground component image, a separation step of separating the background component image, the foreground component image separated by the process of the separation step, An arbitrary background component image or the background component image separated by the process of the separation step and an arbitrary foreground component image are combined at a predetermined mixing ratio to generate a combined image. Characterized in that it comprises a synthetic steps of, and an output step of outputting the synthesized image synthesized in the process of synthesis steps.

The program of the first recording medium of the present invention is
A request information input control step for controlling the input of user request information, and a pixel value determined for each pixel based on the request information and according to the amount of light forming an image that is temporally integrated. A foreground component image composed of foreground object components that form a foreground object of image data consisting of
A mixture ratio estimation control step for controlling estimation of a mixture ratio indicating a mixture ratio with a background component image composed of background object components constituting a background object, and a foreground component image of the image data based on the mixture ratio, and a background. Separation control step for controlling separation into component images, foreground component image separated by processing of separation control step, arbitrary background component image, or background component image separated by separation control step, and arbitrary foreground A combination with a component image at a predetermined mixing ratio; a combination control step for controlling the generation of the combined image; and an output control step for controlling the output of the combined image combined in the processing of the combination control step. Characterize.

A first program of the present invention comprises a request information input control step of controlling input of request information of a user,
Foreground consisting of foreground object components forming a foreground object of image data consisting of pixel values determined for each pixel based on the request information and according to the amount of light forming the image that is temporally integrated. A mixture ratio estimation control step that controls estimation of a mixture ratio that indicates a mixture ratio of a component image and a background component image that is a background object component that forms a background object, and a foreground component of image data based on the mixture ratio. An image, a separation control step for controlling separation into a background component image, a foreground component image separated by the processing of the separation control step, an arbitrary background component image, or
The background component image separated by the separation control step and the arbitrary foreground component image are combined at a predetermined mixing ratio, the combination control step for controlling the generation of the combined image, and the combination control step are combined. And an output control step of controlling the output of the composite image.

In the communication system of the present invention, the first communication device includes request information input means for inputting user request information,
The second communication includes: request information transmitting means for transmitting the request information input by the request information input means to the second transmitting device; and receiving means for receiving the composite image transmitted from the second communication device. The device receives a request information receiving unit that receives request information transmitted from the first transmitting device, and determines the amount of light that constitutes an image integrated pixel by pixel based on the request information. A mixing ratio indicating the mixing ratio of the foreground component image composed of the foreground object components forming the foreground object and the background component image composed of the background object components forming the background object, of the image data composed of the pixel values determined accordingly. , A separating means for separating the image data into a foreground component image and a background component image based on the mixing ratio, and a foreground component separated by the separating means. The image, the arbitrary background component image, or the background component image separated by the separating unit, and the arbitrary foreground component image are combined at a predetermined mixing ratio to generate a combined image. And a composite image transmitting means for transmitting the composite image thus combined to the first communication device.

In addition to the user's request information, the request information input means is further made to input a motion blur adjustment amount for adding motion blur to the foreground component image, and the second communication device displays the foreground component image. On the other hand, it is possible to further provide motion blur adding means for adding motion blur corresponding to the motion blur adjustment amount, and the combining means, the foreground component image to which motion blur is added by the motion blur adding means, It is possible to synthesize an arbitrary background component image with a predetermined mixing ratio and generate a synthesized image.

In addition to the user's request information, the request information input means further inputs an arbitrary background component image,
The synthesizing unit may synthesize the foreground component image and the arbitrary background component image with the mixture ratio estimated by the mixture ratio estimating unit to generate a synthesized image.

The second communication device may be further provided with a charging means for executing a charging process according to the request information.

The billing means generates billing information including the identifier of the user, the identifier of the second communication device, and the amount of money corresponding to the demand information according to the demand information, and based on the billing information, The charging process can be executed for the financial account of the user.

The second communication device may be further provided with a point storage means for storing the points for each user used for the billing process, and the billing means is used according to request information. By generating the billing information including the person's identifier, the identifier of the second communication device, and the amount information corresponding to the composite image, and subtracting the number of points stored in the point storage means by the amount corresponding to the amount information. The billing process can be executed.

The composite image transmitting means may transmit the composite image to the first communication device of the user who has completed the charging processing after the charging processing is completed by the charging means.

The request information input means inputs predetermined image data in addition to the user's request information, and the output means completes the charging process of the combined image combined by the combining means by the charging means. After that, the charging process can be output to the first communication device of the user.

The communication method of the communication system of the present invention is the first
And a request information transmitting step of transmitting the request information input in the processing of the request information inputting step to the second transmitting apparatus. A receiving step of receiving the composite image transmitted from the second communication apparatus, the communication method of the second communication apparatus including a request information receiving step of receiving the request information transmitted from the first transmitting apparatus; , A foreground object component forming a foreground object of image data having a pixel value determined for each pixel based on the request information and according to the amount of light forming the image that is temporally integrated. A mixing ratio estimating step of estimating a mixing ratio indicating a mixing ratio of a foreground component image and a background component image including a background object component forming a background object, and Then, the image data is separated into a foreground component image and a background component image by a separation step, the foreground component image separated by the processing in the separation step, and any background component image or separated by the processing in the separation step. The background component image and the arbitrary foreground component image are combined at a predetermined mixing ratio to generate a combined image, and the combined image combined in the processing of the combining step is transmitted to the first communication device. And a composite image transmitting step.

The program of the second recording medium of the present invention is
The program controlling the first communication device controls the input of the request information of the user, and the second step of the request information input in the process of the request information input step.
And a reception control step of controlling reception of a composite image transmitted from the second communication device, and a program for controlling the second communication device. A request information reception control step of controlling reception of request information transmitted from the first transmitter, and an amount of light forming an image integrated pixel by pixel and temporally based on the request information. A foreground component image composed of foreground object components constituting a foreground object, of image data composed of pixel values determined according to
A mixture ratio estimation control step for controlling estimation of a mixture ratio indicating a mixture ratio with a background component image composed of background object components constituting a background object, and a foreground component image of the image data based on the mixture ratio, and a background. A separation control step for controlling separation into component images, a foreground component image separated by the processing of the separation control step, any background component image, or a background component image separated by the processing of the separation step, and any Compositing with a foreground component image at a predetermined mixture ratio, and a compositing control step for controlling the generation of the composite image,
And a synthetic image transmission control step of controlling transmission of the synthetic image synthesized by the processing of the synthesis control step to the first communication device.

The second program of the present invention is input to the computer controlling the first communication device in the processing of the request information input control step of controlling the input of the request information of the user and the request information input step. The request information transmission control step of controlling the transmission of the request information to the second transmission device and the reception control step of controlling the reception of the composite image transmitted from the second communication device are executed to execute the second communication. A request information reception control step of controlling reception of request information transmitted from the first transmission device to a computer controlling the device, and an image integrated temporally for each pixel based on the request information The foreground component image composed of the foreground object components forming the foreground object and the background object of the image data composed of the pixel values determined according to the amount of light forming the A mixture ratio estimation control step of controlling estimation of the mixing ratio indicating the ratio of mixing of the background component image consisting of background object components forming, based on the mixture ratio,
Image data, a foreground component image, a separation control step for controlling separation into a background component image, a foreground component image separated by the processing of the separation control step, and an arbitrary background component image,
Alternatively, the background component image separated in the process of the separation step and the arbitrary foreground component image are combined at a predetermined mixing ratio, a combination control step of controlling the generation of the combined image, and a combination control step. And a composite image transmission control step of controlling the transmission of the combined image to the first communication device.

A second communication device of the present invention comprises request information input means for inputting user request information, and request information transmitting means for transmitting the request information input by the request information input means to another communication device. , And a receiving unit that receives a composite image transmitted from another communication device.

A second communication method of the present invention is a request information input step of inputting request information of a user, and a request information transmission step of transmitting the request information input in the processing of the request information input step to another communication device. And a receiving step of receiving a composite image transmitted from another communication device.

The program of the third recording medium of the present invention is
A request information input control step for controlling input of request information of a user, a request information transmission control step for controlling transmission of request information whose input is controlled in the processing of the request information input control step to another communication device, And a reception control step of controlling reception of a composite image transmitted from another communication device, the computer-readable program being recorded.

A third program of the present invention comprises a request information input control step for controlling the input of user request information,
A request information transmission control step for controlling the transmission of the request information whose input is controlled in the processing of the request information input control step to another communication device, and a reception for controlling the reception of the composite image transmitted from the other communication device. The control step is executed by a computer.

A third communication device of the present invention is a request information receiving means for receiving request information transmitted from another communication device, and is integrated for each pixel temporally based on the request information. Foreground component image of the foreground object component of the foreground object and background component image of the background object component of the background object of the image data composed of pixel values determined according to the amount of light A mixture ratio estimating means for estimating a mixture ratio indicating a mixture ratio of the image data, a separating means for separating the image data into a foreground component image and a background component image based on the mixture ratio, and a foreground separated by the separating means. A component image, an arbitrary background component image, or a background component image separated by the separating means and an arbitrary foreground component image are combined at a predetermined mixing ratio to generate a combined image. A synthesizing unit that, characterized in that it comprises a composite image transmitting means for transmitting the synthesized image synthesized in the other communication device by synthetic means.

According to a third communication method of the present invention, a request information receiving step of receiving request information transmitted from another communication device, and on the basis of the request information, is integrated pixel by pixel and temporally. A foreground component image composed of foreground object components constituting a foreground object, of image data consisting of pixel values determined according to the amount of light constituting the image,
A mixture ratio estimating step of estimating a mixture ratio indicating a mixture ratio with a background component image composed of a background object component forming a background object, and image data is converted into a foreground component image and a background component image based on the mixture ratio. A separation step of separating, a foreground component image separated by the processing of the separation step, an arbitrary background component image, or a background component image separated by the processing of the separation step, and an arbitrary foreground component image It is characterized by including a synthesizing step of synthesizing at a mixing ratio to generate a synthetic image, and a synthetic image transmitting step of transmitting the synthetic image synthesized by the processing of the synthesizing step to another communication device.

The program of the fourth recording medium of the present invention is
A request information reception control step for controlling reception of request information transmitted from another communication device, and based on the request information, for each pixel and according to the amount of light forming an image that is temporally integrated. Of the foreground component image of the foreground object component of the foreground object and the background component image of the background object component of the background object of the image data of the pixel values determined by A mixture ratio estimation control step for controlling the estimation, a separation control step for controlling separation of the image data from the foreground component image and the background component image based on the mixture ratio, and a foreground separated by the processing of the separation control step. Predetermined mixture of the component image, an arbitrary background component image, or the background component image separated by the processing of the separation control step, and an arbitrary foreground component image And a composite image transmission control step of controlling transmission of the composite image composited by the processing of the composite control step to another communication device. And

A fourth program of the present invention is a request information reception control step for controlling reception of request information transmitted from another communication device, and on a pixel-by-pixel and temporally basis based on the request information. A foreground component image consisting of a foreground object component forming a foreground object and a background consisting of a background object component forming a background object, of image data consisting of pixel values determined according to the amount of light forming the integrated image. A mixture ratio estimation control step for controlling estimation of a mixture ratio indicating a mixture ratio with a component image, and a separation control step for controlling separation of image data from a foreground component image and a background component image based on the mixture ratio. And the foreground component image separated by the process of the separation control step and any background component image, or the background component image separated by the process of the separation control step And an arbitrary foreground component image at a predetermined mixture ratio, a combination control step for controlling the generation of the combined image, and the transmission of the combined image combined in the processing of the combination control step to another communication device. And a synthetic image transmission control step for controlling the.

In the first communication device and method and the program of the present invention, the user's request information is input, and an image integrated pixel by pixel and temporally is constructed based on the request information. The ratio of the mixture of the foreground component image composed of the foreground object components forming the foreground object and the background component image composed of the background object components forming the background object, of the image data composed of pixel values determined according to the amount of light. , The image data is separated into a foreground component image and a background component image based on the mixture ratio, and the separated foreground component image and any background component image or the separated background Component images,
The arbitrary foreground component image is combined with a predetermined mixing ratio to generate a combined image, and the combined image is output.

In the communication system, method, and program of the present invention, the first communication device inputs the user's request information, the input request information is transmitted to the second transmitting device, and the second transmitting device transmits the second request information. The composite image transmitted from the communication device is received, the second communication device receives the request information transmitted from the first transmission device, and based on the request information, for each pixel and temporally. A foreground component image of a foreground object component of a foreground object, and a background object component of a background object of the image data of pixel values determined according to the amount of light constituting the image A mixture ratio indicating a mixture ratio with the background component image is estimated, and the image data is separated into the foreground component image and the background component image based on the mixture ratio, and separated. The foreground component image and the arbitrary background component image, or the separated background component image and the arbitrary foreground component image are combined at a predetermined mixing ratio to generate a combined image, and the combined image is combined. Sent to the first communication device.

In the second communication device and method and the program of the present invention, the user's request information is input, the request information is transmitted to another communication device, and the composite image is transmitted from the other communication device. Is received.

In the third communication device and method and the program of the present invention, request information transmitted from another communication device is received, and based on the request information, it is integrated pixel by pixel and temporally. Image data consisting of pixel values that are determined according to the amount of light that makes up the captured image A mixing ratio indicating the mixing ratio with the image is estimated, and based on the mixing ratio, the image data is
The foreground component image and the background component image are separated, and the separated foreground component image and the arbitrary background component image or the separated background component image and the arbitrary foreground component image are combined at a predetermined mixing ratio. Then, a combined image is generated, and the combined image is transmitted to another communication device.

[0041]

1 is a diagram showing an embodiment of an image processing system according to the present invention.

The image processing system of the present invention includes, for example, a separation processing server 11, a motion detection server 12, a region specifying server 13, a mixture ratio calculation server 14, a foreground / background separation processing server 15, a motion blur on a network 1 such as the Internet. Coordination server 16, encoding server 17, storage server 1
8-1, 18-2, combination server 19, correction server 20,
The purchase server 21, the sale server 22, the search server 23, the billing server 24, the financial server (for customers) 25, the financial server (for providers) 26, the client computer 27, and the camera terminal devices 28-1 to 28-n. , Are connected and can exchange data with each other.
Separation processing server 11, motion detection server 12, region identification server 13, mixture ratio calculation server 14, foreground / background separation processing server 15, motion blur adjustment server 16, encoding server 1.
7, synthesis server 19, correction server 20, purchase server 2
1, sale server 22, search server 23, billing server 2
4. The financial server (for customer) 25 and the financial server (for provider) 26 are a separation service, a motion detection service, a region identification service, a mixture ratio calculation service, a foreground / background separation service, a motion blur adjustment service, and a code, respectively. Conversion service, synthesis service, modification service, purchase service,
Selling service, search service, billing service, and
A server managed or operated by a provider who provides financial services (for customers and providers).
In the following description, the storage servers 18-1, 18-
2 and the camera terminal devices 28-1 to 28-n are simply referred to as the storage server 18 and the camera terminal device 28 when it is not necessary to distinguish them individually. The same applies to other servers and devices.

FIG. 2 shows the separation processing server 11 according to the present invention.
It is a figure which shows the structure of. CPU (Central Processing Uni
The t) 41 executes various processes according to a program stored in the ROM (Read Only Memory) 42 or the storage unit 48. RAM (Random Access Memory) 43 has a C
Programs and data executed by the PU 41 are appropriately stored. These CPU 41, ROM 42, and RAM 43 are
They are interconnected by a bus 44.

An input / output interface 45 is also connected to the CPU 41 via a bus 44. The input / output interface 45 is connected to an input unit 46 including a keyboard, a mouse, a microphone and the like, and an output unit 47 including a display and a speaker. The CPU 41 executes various kinds of processing in response to a command input from the input unit 46. Then, the CPU 41 outputs the image, sound, etc. obtained as a result of the processing to the output unit 47.

The storage unit 48 connected to the input / output interface 45 is composed of, for example, a hard disk, and stores programs executed by the CPU 41 and various data. The communication unit 49 communicates with an external device via the Internet or other network.

Further, the program may be acquired via the communication section 49 and stored in the storage section 48.

The drive 50 connected to the input / output interface 45 includes a magnetic disk 61 and an optical disk 6.
2. When the magneto-optical disk 63, the semiconductor memory 64, or the like is mounted, they are driven to acquire the program or data recorded therein. The acquired programs and data are transferred to and stored in the storage unit 48 as needed.

The motion detection server 12, the area identification server 13, the mixture ratio calculation server 14, the foreground / background separation processing server 15, the motion blur adjustment server 16, the encoding server 17, the storage servers 18-1, 18-2, and the composition. Server 19, correction server 20, purchase server 21, sale server 22, search server 23, billing server 24, financial server (for customer) 25,
The basic configuration of the financial server (for provider) 26 and the client computer 27 is the same as that of the separation processing server 11, and the description thereof will be omitted.

FIG. 3 shows a camera terminal device 28 according to the present invention.
It is a figure which shows the structure of. The configuration of the camera terminal device 28 is
A sensor 76a and a GPS (Global Posi
tioning System) 76b is provided, and the output unit 77 has
The configuration is the same as that of the separation processing server 11 except that an LCD (Liquid Crystal Display) 77a is provided. That is, the CPU 71, the ROM of the camera terminal device 28
72, RAM 73, bus 74, input / output interface 7
5, input unit 76, output unit 77, storage unit 78, communication unit 7
9, drive 80, magnetic disk 91, optical disk 9
2. Magneto-optical disk 93 and semiconductor memory 94
Are the CPU 41, ROM 42, and
It corresponds to the RAM 43, the bus 44, the input / output interface 45, the input unit 46, the output unit 47, the storage unit 48, the communication unit 49, the drive 50, the magnetic disk 61, the optical disk 62, the magneto-optical disk 63, and the semiconductor memory 64, respectively. is doing.

The sensor 76a is an image pickup device and outputs the picked-up image to the input unit 76. The GPS 76b detects position information (latitude and longitude) on the earth based on a signal transmitted from a geostationary satellite (not shown), and outputs the detected position information to the input unit 76. The LCD 77a displays the image output from the output unit.

Next, the separation processing server 11 will be described with reference to FIGS. 4 and 5.

As shown in FIG. 4, the separation processing server 11 separates an image input from the client computer 27 or the like via the network 1 into a foreground component image and a background component image by a method described later, ID for each of the input image, foreground component image, and background component image
A storage server 18 that generates a file with an (Identifier), outputs it to the client computer 27, and stores it by itself,
Output to and store in, or output to and stored in another server on the network. Here, the foreground component image refers to an image having a motion component in the input image, and the background component image refers to a still portion image in the input image that does not include the motion component. At this time, the billing processing unit 11a executes billing processing of the charge relating to the separation processing to the billing server 24 via the network 1. Further, as shown in FIG. 5, when an image ID designating an image is input instead of an image, the separation processing server 11 inputs a search server 23 and a storage server 1 described later on the network 1.
8 or by searching its own storage unit (for example, the storage unit 78 in FIG. 3) to read out the image data corresponding to the input image ID and separating it into the foreground component image and the background component image, The IDs corresponding to each are added and stored by themselves, or output to another server on the network 1 and the processing corresponding to that server is executed.

In the following description, an image ID will be described as an example of information for specifying an image, but any information that can specify an image may be used, and for example, image position information described later may be used.

Next, the motion detection server 12 will be described with reference to FIGS.

As shown in FIG. 6, the object extraction unit 12a of the motion detection server 12 extracts an image object in an image input from the client computer 27 or the like and outputs it to the motion detection unit 12b. The motion detection unit 12b detects the motion vector and position information of the input image object and outputs it to the client computer 27, stores it by itself, or outputs it to another server on the network 1 and supports it. To execute the process. At this time, the billing processing unit 12c performs billing processing on the billing server 24 via the network 1 for a fee relating to the process of detecting the motion vector and position information for each image object. Note that in this specification, an image corresponding to an object in the real world, which is an image capturing target, is referred to as an object.

Further, as shown in FIG. 7, when an image ID designating an image is input instead of an image, the motion detection server 12 searches a server 23 on the network 1 which will be described later.
Or an image input by accessing the storage server 18 or searching its own storage unit (for example, the storage unit 78 in FIG. 3).
The image data corresponding to the ID is read and the same processing as above is executed.

Next, the area specifying server 13 will be described with reference to FIGS.

As shown in FIG. 8, the area specifying server 13 is inputted via the network 1 by, for example, an image inputted from the client computer 27 or the like and object designation information designating an object in the image. Each of the pixels of the image is specified as a foreground area, a background area, or a mixed area, and information indicating whether each pixel belongs to the foreground area, the background area, or the mixed area (hereinafter, area information Called)),
The data is output to the client computer 27, stored by itself, or output to another server on the network 1 to execute the process corresponding to the server. At this time, the billing processing unit 13a executes billing processing for the fee relating to the area specifying processing to the billing server 24 via the network 1. Further, as shown in FIG. 9, when an image ID designating an image is input instead of an image, the area specifying server 13 searches a search server 2 to be described later on the network 1.
3 or access the storage server 18 or search its own storage unit (for example, the storage unit 78 in FIG. 3), call the image corresponding to the input image ID, and correspond to the object designation information of the image. Output area information.

Next, the mixing ratio calculation server 14 will be described with reference to FIGS.

As shown in FIG. 10, the mixing ratio calculation server 14 receives an image input from the client computer 27 or the like via the network 1, object designation information for designating an object in the image, and an area. Based on the information, a mixture ratio (hereinafter, referred to as a mixture ratio α) corresponding to the pixels included in the mixture area is calculated,
The calculated mixing ratio is output to the client computer 27, stored by itself, or output to another server on the network, and the processing corresponding to the server is executed. At this time, the billing processing unit 14a executes the billing process for the charge relating to the mixture ratio calculation process to the billing server 24 via the network 1. Further, as shown in FIG. 11, when the image ID designating an image is input instead of the image, the mixture ratio calculation server 14 accesses a search server 23 or a storage server 18 described later on the network 1. , Searches its own storage unit (for example, the storage unit 78 in FIG. 3) and calls the image corresponding to the input image ID,
The same processing as described above is executed.

Next, the foreground / background separation processing server 15 will be described with reference to FIGS.

As shown in FIG. 12, the foreground / background separation processing server 15 receives an image input from the client computer 27 or the like via the network 1, object specifying information for specifying an object in the image, and area information. , And a foreground component image consisting only of image components corresponding to foreground objects (hereinafter also referred to as foreground components) and a background component (hereinafter also referred to as background components) based on the mixture ratio α. The input image is separated into the background component image and the image is assigned to each image and output to the client computer 27, stored on its own, or output to another server on the network and supported by that server. To execute the process. At this time, the billing processing unit 15a executes the billing process for the fee relating to the foreground / background separation process to the billing server 24 via the network 1. Further, as shown in FIG. 13, when an image ID designating an image is input instead of an image, the foreground / background separation processing server 15 accesses a search server 23 and a storage server 18 described later on the network 1. Or
The own storage unit (for example, the storage unit 78 in FIG. 3) is searched, the image corresponding to the input image ID is called, and the same processing as described above is executed.

Next, the motion-blur adjusting server 16 will be described with reference to FIGS.

As shown in FIG. 14, the motion-blur adjusting server 16 receives a foreground component image input from the client computer 27 or the like via the network 1, for example.
The motion blur included in the foreground component image is removed based on the motion vector and the motion blur amount, such as removing motion blur included in the foreground component image, reducing the amount of motion blur, or increasing the amount of motion blur. The foreground component image in which the amount of motion blur is adjusted is generated, and an ID is assigned to each image and output to the client computer 27, stored by itself, or output to another server on the network. Then, the processing corresponding to the server is executed. At this time, the billing processing unit 16a executes the billing process for the fee relating to the motion blur adjustment process to the billing server 24 via the network 1. Further, as shown in FIG. 15, when the foreground component image ID designating the foreground component image is input in place of the foreground component image, the motion blurring adjustment server 16 searches the server 1 and the storage server 23 described later on the network 1. The server 18 is accessed, or its own storage unit (for example, the storage unit 78 in FIG. 3) is searched, the foreground component image corresponding to the input foreground component image ID is called, and the same processing as described above is executed. To do.

Next, the encoding server 17 will be described with reference to FIGS.

As shown in FIG. 16, the encoding server 17 separates an image input from the client computer 27 or the like via the network 1 into a foreground component image and a background component image, and assigns IDs to them. Then, it can be stored by itself or output to another server on the network and stored, and the URL of that server (Uni
(for example, versatile resource locator), and foreground component image position information consisting of a code indicating the position on the network 1 of the server where the background component image is output (stored), and background component image position information. , And output together with information such as the motion vector of these images, position information, and mixture ratio. The information output by the encoded information may be any of the encoded information, the image, and the image and the encoded information, and the output information can be changed as necessary. Further, at this time, the billing processing unit 17a executes billing processing of the charge relating to the encoding processing to the billing server 24 via the network 1. Further, as shown in FIG. 17, when an image ID designating an image is input instead of the image, the encoding server 17 receives a search server 23 and a storage server 1 on the network 1 described later.
8 or retrieves its own storage unit (for example, the storage unit 78 in FIG. 3), calls the image corresponding to the input image ID, and executes the same processing as described above.

Next, the storage server 18 will be described with reference to FIGS.

The storage server 18, as shown in FIG.
The images connected to the network 1 and transmitted from various servers are accumulated, and the image position information corresponding to the accumulated images is output together with the image ID. Based on this image position information, for example, the client computer 27
Can access through the network 1 and retrieve a desired image. That is, as shown in FIG. 19, for example, the client computer 27 uses the storage server 18 of the network 1 based on the image position information.
It is possible to read out the desired image by accessing to and specifying the image ID corresponding to the desired image. In this specification, the image position information and the image ID will be described separately, but the image position information may be a part of the image ID. In this case, the image position information is determined by the image ID. May be recognized, and which server on the network 1 is stored (stored or processed) may be recognized. Also, the storage server 18
In addition to the image data, a motion vector, position information, a mixture ratio, and a motion blur amount may be stored.

Next, the synthesis server 19 will be described with reference to FIGS.

The synthesizing server 19 is, as shown in FIG.
For example, two images such as images A and B input from the client computer 27 or the like via the network 1, a motion vector, position information, a mixture ratio,
Also, the images A and B are combined from the motion blur amount to generate a combined image (A + B), and the client computer 27
Output to, memorize by itself, or network 1
Output to the other server above and execute the process corresponding to that server. In this case, one of the images A and B is treated as a foreground component image and the other is treated as a background component image, so that the two are combined. At this time, the billing processing unit 19a executes the billing process for the fee relating to the combining process to the billing server 24 via the network 1. Further, as shown in FIG. 21, the synthesizing server 19 replaces the images A and B with images A-ID and B- which specify images.
When the ID is input, the search server 23 or the storage server 18 described later on the network 1 is accessed, or its own storage unit (for example, the storage unit 78 in FIG. 3) is searched, and the input image A- Call the image corresponding to ID, B-ID,
The same processing as described above is executed.

Next, the correction server 20 will be described with reference to FIGS.

The correction server 20, as shown in FIG.
An image input from the client computer 27 or the like via the network 1 is converted into a motion vector,
Further, the correction is performed based on the position information, the mixture ratio, and the motion blur amount, and the corrected image is generated and output to the client computer 27, stored by itself, or output to another server on the network. Causes the server to execute the corresponding process. At this time, the charging processing unit 20a
To the billing server 24 via the network 1,
Charge processing for the charge relating to the correction processing is executed. Also, FIG.
As shown by 3, the correction server 20 uses the image instead of the image.
When an image ID that specifies an image is entered, the network 1
It accesses the search server 23 and the storage server 18, which will be described later, or has its own storage unit (for example, the storage unit 78 in FIG. 3).
Is searched, the image corresponding to the input image ID is called, and the same processing as described above is executed.

Next, the purchase server 21 will be described with reference to FIG.

The purchase server 21, as shown in FIG.
An image ID for designating an image desired to be purchased by operating a client computer 27 or the like via a network 1 by a user wishing to purchase a predetermined image, for example.
Is input, the corresponding image is sent to the separation processing server 11, the storage server 18, the synthesizing server 19 on the network 1,
Alternatively, the correction server 20 or the like is accessed to call the corresponding image and output it to the client computer 27. At this time, the billing processing unit 21a executes the billing process for the fee for the image to be purchased, to the billing server 24 via the network 1.

Next, the selling server 22 will be described with reference to FIG.

The selling server 22, as shown in FIG.
Via the network 1, for example, the separation processing server 1
1. When an image generated by the synthesizing server 19 or the correction server 20 and owned by a predetermined user, which is a separated image, a composite image, or an image for which the sale of the revised image is desired, is input, the image for which the sale is desired is input. Is stored in the separation processing server 11, the storage server 18, the combination server 19, or the correction server 20 on the network 1, and the charging processing unit 22a sells the image to the charging server 24 via the network 1. The fee-charging process is executed (in this case, the provider of the sale processing service executes the payment process for the price corresponding to the sold image to the user who desires the sale).

Next, referring to FIG. 26, the search server 26
Will be described.

The search server 26 uses, for example, the information indicating the characteristics of the image desired by the user by the client computer 1 and the physical position information of the camera terminal devices 28-1 to 28-n to find the camera on the network 1. An image currently captured by the terminal devices 28-1 to 28-n or an image captured in advance is searched for and output to the client computer 1 as a request image. At this time, the billing processing unit 23 a uses the billing server 2 via the network 1.
For 4, the charge processing for the fee relating to the search processing is executed.

In the present specification, the term "encoding" refers to conversion into information on a foreground component image, a background component image, a motion vector, position information, a motion blur amount, and a mixture ratio, which is obtained based on image data. This is coded, and the data is called coded data.

FIG. 27 is a block diagram showing the separation processing server 11.

It does not matter whether each function of the separation processing server 11 is realized by hardware or software. That is, each block diagram in this specification may be considered as a block diagram of hardware or a functional block diagram of software.

The input image supplied to the separation processing server 11 is supplied to the object extracting unit 101, the region specifying unit 103, the mixture ratio calculating unit 104, and the foreground / background separating unit 105.

The object extracting unit 101 roughly extracts the image object corresponding to the foreground object included in the input image, and supplies the extracted image object to the motion detecting unit 102. The object extraction unit 101
For example, by detecting the contour of the image object corresponding to the foreground object included in the input image, the image object corresponding to the foreground object is roughly extracted.

The object extracting section 101 roughly extracts the image object corresponding to the background object included in the input image, and supplies the extracted image object to the motion detecting section 102. The object extraction unit 101
For example, the image object corresponding to the background object is roughly extracted from the difference between the input image and the extracted image object corresponding to the foreground object.

In addition, for example, the object extraction unit 101
To roughly extract the image object corresponding to the foreground object and the image object corresponding to the background object from the difference between the background image stored in the background memory provided inside and the input image. You may

The motion detection unit 102 calculates the motion vector of the image object corresponding to the roughly extracted foreground object by a method such as a block matching method, a gradient method, a phase correlation method, and a per-recursive method. The calculated motion vector and position information of the motion vector (information specifying the position of the pixel corresponding to the motion vector) are supplied to the area specifying unit 103 and the motion blur adjusting unit 106.

The motion vector output from the motion detector 102 includes information corresponding to the motion amount v.

Further, for example, the motion detecting unit 102 calculates the motion vector of each image object together with the pixel position information for specifying the pixel in the image object, and the motion blur adjusting unit 10
6 may be output.

The motion amount v is a value that represents the change in the position of the image corresponding to the moving object in units of pixel intervals. For example, the image of the object corresponding to the foreground is 4 in the next frame with respect to a certain frame.
When moving so as to be displayed at a pixel-separated position, the motion amount v of the image of the object corresponding to the foreground is
It is set to 4.

The object extracting unit 101 and the motion detecting unit 102 are necessary when adjusting the amount of motion blur corresponding to a moving object.

The area specifying unit 103 specifies each of the pixels of the input image as either the foreground area, the background area, or the mixed area, and selects the foreground area, the background area, or the mixed area for each pixel. The information indicating whether to belong to the mixture ratio calculation unit 104, the foreground / background separation unit 105, and the motion blur adjustment unit 106 is supplied.

The mixture ratio calculation unit 104 calculates the mixture ratio corresponding to the pixels included in the mixed region based on the input image and the region information supplied from the region specifying unit 103, and the calculated mixture ratio is the foreground. It is supplied to the background separation unit 105.

The mixing ratio α is a value indicating the ratio of the image component (hereinafter, also referred to as the background component) corresponding to the background object in the pixel value, as shown in the equation (3) described later.

The foreground / background separating unit 105 includes the area specifying unit 10
3 and the region information supplied from the mixture ratio calculation unit 104.
Based on the mixture ratio α supplied from, the input image is a foreground component image consisting only of image components corresponding to the foreground object (hereinafter also referred to as foreground component) and a background component image consisting only of background components. Are separated and the foreground component image is supplied to the motion-blur adjusting unit 106 and the selecting unit 107. It should be noted that the separated foreground component image may be the final output. It is possible to specify only the foreground and the background without considering the conventional mixed area, and obtain a more accurate foreground and background than the method of separating.

The motion-blur adjusting unit 106 determines a processing unit indicating one or more pixels included in the foreground component image, based on the amount of motion v and the area information which can be known from the motion vector. The processing unit is data that designates a group of pixels that are the target of the processing for adjusting the amount of motion blur.

The motion-blur adjusting unit 106 detects the motion-blur adjustment amount input to the separation processing server 11 and the foreground / background separating unit 10.
5, the motion blur included in the foreground component image is removed, and the amount of motion blur is reduced, based on the foreground component image supplied from No. 5, the motion vector and its position information supplied from the motion detection unit 102, and the processing unit. , Or the amount of motion blur included in the foreground component image is adjusted by increasing the amount of motion blur, and the foreground component image with the adjusted amount of motion blur is output to the selection unit 107. The motion vector and its position information may not be used.

Here, the motion blur refers to a distortion included in an image corresponding to a moving object, which is caused by the movement of the object in the real world to be imaged and the image pickup characteristics of the sensor. .

The selection unit 107 determines the foreground component image supplied from the foreground / background separation unit 105 and the amount of motion blur supplied from the motion blur adjustment unit 106 based on the selection signal corresponding to the user's selection, for example. Either one of the adjusted foreground component images is selected and the selected foreground component image is output.

Next, the input image supplied to the separation processing server 11 will be described with reference to FIGS. 28 to 43.

FIG. 28 is a diagram for explaining the image pickup by the sensor 76a. The sensor 76a is composed of, for example, a CCD video camera including a CCD (Charge-Coupled Device) area sensor that is a solid-state image sensor. The object corresponding to the foreground in the real world is between the object corresponding to the background and the sensor in the real world,
For example, it moves horizontally from the left side to the right side in the figure.

The sensor 76a images the object corresponding to the foreground together with the object corresponding to the background.
The sensor 76a outputs the captured image in 1-frame units. For example, the sensor 76a outputs an image composed of 30 frames per second. The exposure time of the sensor 76a is
It can be 1/30 second. The exposure time depends on the sensor 7
6a is a period from the start of conversion of input light into electric charges to the end of conversion of input light into electric charges. Hereinafter, the exposure time is also referred to as the shutter time.

FIG. 29 is a diagram for explaining the arrangement of pixels. In FIG. 29, A to I indicate individual pixels. The pixels are arranged on a plane corresponding to the image.
One detection element corresponding to one pixel is the sensor 76a.
It is placed on top. When the sensor 76a captures an image, one detection element outputs a pixel value corresponding to one pixel forming the image. For example, the position of the detection element in the X direction corresponds to the position in the horizontal direction on the image, and the position of the detection element in the Y direction.
The position in the direction corresponds to the position in the vertical direction on the image.

As shown in FIG. 30, the detection element, which is, for example, a CCD, converts the input light into electric charge and stores the converted electric charge for a period corresponding to the shutter time. The amount of electric charge is almost proportional to the intensity of the input light and the time during which the light is input. The detection element, during the period corresponding to the shutter time, the charge converted from the input light,
Add to the already stored charge. That is, the detection element integrates the input light for a period corresponding to the shutter time, and accumulates an amount of charges corresponding to the integrated light. It can be said that the detection element has an integration effect with respect to time.

The electric charge accumulated in the detecting element is converted into a voltage value by a circuit (not shown), and the voltage value is further converted into a pixel value such as digital data and output. Therefore, the individual pixel values output from the sensor 76a are the values projected onto a one-dimensional space, which is the result of integrating, with respect to the shutter time, a portion of the object corresponding to the foreground or the background that has a spatial spread. Have.

The separation processing server 11 extracts the significant information buried in the output signal, for example, the mixture ratio α, by the accumulation operation of the sensor 76a. The separation processing server 11 adjusts the amount of distortion caused by the foreground image objects themselves being mixed, for example, the amount of motion blur. Further, the separation processing server 11 adjusts the amount of distortion caused by the mixture of the foreground image object and the background image object.

FIG. 31 is a diagram for explaining an image obtained by picking up an object corresponding to a moving foreground and an object corresponding to a stationary background. Figure 31
(A) is an object corresponding to a moving foreground,
The image obtained by imaging the object corresponding to the stationary background is shown. In the example illustrated in FIG. 31A, the object corresponding to the foreground is moving horizontally from left to right with respect to the screen.

FIG. 31B is a model diagram in which the pixel values corresponding to one line of the image shown in FIG. 31A are expanded in the time direction. The horizontal direction of FIG.
This corresponds to the spatial direction X of (A).

The pixel value of the pixel in the background area is formed only from the background component, that is, the image component corresponding to the background object. The pixels in the foreground area are
The pixel value is configured only from the foreground component, that is, the image component corresponding to the foreground object.

The pixel value of the pixel in the mixed area is composed of the background component and the foreground component. It can be said that the mixed region is a distorted region because its pixel value is composed of the background component and the foreground component. The mixed area is further classified into a covered background area and an uncovered background area.

The covered background area is a mixed area at a position corresponding to the front end of the foreground object in the traveling direction with respect to the foreground area, and the background component is covered with the foreground as time passes. Say.

On the other hand, the uncovered background area is a mixed area at a position corresponding to the rear end portion of the foreground object in the traveling direction of the foreground area, and the background component corresponding to the passage of time. It means the area where appears.

As described above, the image including the foreground area, the background area, or the covered background area or the uncovered background area is determined by the area specifying unit 103,
It is input as an input image to the mixture ratio calculation unit 104 and the foreground / background separation unit 105.

FIG. 32 is a diagram for explaining the background area, foreground area, mixed area, covered background area, and uncovered background area as described above. In the case corresponding to the image shown in FIG. 31, the background area is a still portion, the foreground area is a moving portion, and the covered background area of the mixed area is a portion that changes from the background to the foreground. The uncovered background area is the portion that changes from the foreground to the background.

FIG. 33 shows, in the time direction, the pixel values of pixels arranged in a row adjacent to each other in an image obtained by picking up an object corresponding to a stationary foreground and an object corresponding to a stationary background. FIG.
For example, the pixels lined up on one line of the screen can be selected as the pixels lined up adjacently in one line.

The pixel values F01 to F04 shown in FIG. 33 are the pixel values of the pixels corresponding to the stationary foreground object. The pixel values B01 to B04 illustrated in FIG. 33 are pixel values of pixels corresponding to a stationary background object.

The vertical direction in FIG. 33 corresponds to time,
Time passes from the top to the bottom in the figure. The position of the upper side of the rectangle in FIG. 33 corresponds to the time when the sensor 76a starts converting the input light into electric charges, and the position of the lower side of the rectangle in FIG. 33 indicates the position of the light input by the sensor 76a. Corresponds to the time when the conversion to electric charge is completed. That is, the distance from the upper side to the lower side of the rectangle in FIG. 33 corresponds to the shutter time.

In the following, the case where the shutter time and the frame interval are the same will be described as an example.

The horizontal direction in FIG. 33 corresponds to the spatial direction X described in FIG. More specifically, in the example shown in FIG. 33, the distance from the left side of the rectangle described as “F01” in FIG. 33 to the right side of the rectangle described as “B04” is 8 times the pixel pitch, That is, it corresponds to the interval of eight consecutive pixels.

When the foreground object and the background object are stationary, the light input to the sensor 76a does not change during the period corresponding to the shutter time.

Here, the period corresponding to the shutter time is set to 2
Divide into one or more equal length periods. For example, assuming that the number of virtual divisions is 4, the model diagram shown in FIG. 33 can be represented as the model shown in FIG. The number of virtual divisions is set corresponding to the amount of movement v of the object corresponding to the foreground within the shutter time. For example, the virtual division number is set to 4 corresponding to the motion amount v of 4, and the period corresponding to the shutter time is divided into 4.

The uppermost row in the figure corresponds to the first divided period when the shutter is opened. The second row from the top in the figure corresponds to the second divided period when the shutter is opened. The third row from the top in the figure corresponds to the third divided period when the shutter is opened. The fourth row from the top in the drawing corresponds to the fourth divided period when the shutter is opened.

Hereinafter, the shutter time divided corresponding to the motion amount v is also referred to as shutter time / v.

When the object corresponding to the foreground is stationary, the light input to the sensor 76a does not change, so the foreground component F01 / v is equal to the pixel value F01 divided by the virtual division number. Similarly, when the object corresponding to the foreground is stationary, the foreground component F02 / v is equal to the pixel value F02 divided by the virtual division number, and the foreground component F03 / v is the pixel value F03. Equal to the number of divisions, the foreground component F
04 / v is equal to the pixel value F04 divided by the number of virtual divisions.

Since the light input to the sensor 76a does not change when the object corresponding to the background is stationary, the background component B01 / v is equal to the pixel value B01 divided by the virtual division number. Similarly, when the object corresponding to the background is stationary, the background component B02 / v is equal to the pixel value B02 divided by the virtual division number, and B03 / v is the pixel value B03 in the virtual division number. B04 / v is equal to the pixel value B04 divided by the number of virtual divisions.

That is, when the object corresponding to the foreground is stationary, the light corresponding to the object in the foreground input to the sensor 76a does not change in the period corresponding to the shutter time.
Foreground component F01 / v corresponding to shutter time / v, second foreground component F01 / v corresponding to shutter time / v, and third shutter time / v corresponding to shutter time / v Foreground component F01 / v corresponding to the foreground component and the fourth foreground component F01 / v corresponding to the shutter time / v after the shutter is opened.
And have the same value. F02 / v to F04 / v also have the same relationship as F01 / v.

When the object corresponding to the background is stationary, the light corresponding to the object in the background input to the sensor 76a does not change during the period corresponding to the shutter time. The background component B01 / v corresponding to / v and the second background component B01 / v corresponding to the shutter time / v when the shutter is open.
, The third component with the shutter open, the background component B01 / v corresponding to the shutter time / v, and the fourth component with the shutter open,
The background component B01 / v corresponding to the shutter time / v has the same value. B02 / v to B04 / v have the same relationship.

Next, the case where the object corresponding to the foreground is moving and the object corresponding to the background is stationary will be described.

FIG. 35 is a model diagram in which the pixel values of the pixels on one line including the covered background area are expanded in the time direction when the object corresponding to the foreground moves toward the right side in the drawing. is there. In FIG. 35, the motion amount v of the foreground is 4. Since one frame is a short time, the object corresponding to the foreground is a rigid body,
It can be assumed that they are moving at a constant speed. In FIG. 35, the image of the object corresponding to the foreground moves so as to be displayed on the right side by four pixels in the next frame with reference to a certain frame.

In FIG. 35, the leftmost pixel through the fourth pixel from the left belong to the foreground area. In FIG. 35, the fifth pixel from the left to the seventh pixel from the left belong to the mixed area which is the covered background area. Fig. 35
In, the rightmost pixel belongs to the background area.

Since the object corresponding to the foreground is moving over time so as to cover up the object corresponding to the background, the component included in the pixel value of the pixel belonging to the covered background area corresponds to the shutter time. At some point in the period, the background component is replaced by the foreground component.

For example, the pixel value with a bold frame in FIG.
M is represented by Formula (1).

[0132] M = B02 / v + B02 / v + F07 / v + F06 / v (1)

For example, the fifth pixel from the left includes the background component corresponding to one shutter time / v, and the foreground component corresponding to three shutter times / v.
The mixture ratio α of the th pixel is 1/4. The sixth pixel from the left includes the background component corresponding to two shutter times / v and the foreground component corresponding to two shutter times / v, so the mixture ratio α of the sixth pixel from the left is , 1/2. Since the seventh pixel from the left includes the background component corresponding to three shutter times / v and the foreground component corresponding to one shutter time / v, the mixture ratio α of the seventh pixel from the left is , 3/4.

Since the object corresponding to the foreground is a rigid body and it can be assumed that the foreground image moves at a constant speed so as to be displayed on the right side by 4 pixels in the next frame, for example, the fourth from the left in FIG. The first foreground component F07 / v of the shutter time / v of the pixel of
This is equal to the foreground component of the fifth pixel from the left in 5 corresponding to the second shutter time / v from when the shutter has opened. Similarly, the foreground component F07 / v is the foreground component of the sixth pixel from the left in FIG. 35 corresponding to the third shutter time / v from when the shutter has opened, and the seventh component from the left in FIG. Of pixels,
It is equal to the foreground component corresponding to the fourth shutter time / v with the shutter open.

Since it can be assumed that the object corresponding to the foreground is a rigid body and the foreground image moves at a constant speed so that it is displayed on the right side of 4 pixels in the next frame, for example, the third from the left in FIG. 35, the foreground component F06 / v of the first pixel of the shutter time / v after the shutter is opened is shown in FIG.
It is equal to the foreground component of the fourth pixel from the left corresponding to the second shutter time / v from when the shutter has opened. Similarly, the foreground component F06 / v is the foreground component of the fifth pixel from the left in FIG. 35 corresponding to the third shutter time / v from when the shutter has opened, and the sixth from the left in FIG. Of pixels,
It is equal to the foreground component corresponding to the fourth shutter time / v with the shutter open.

Since the object corresponding to the foreground is a rigid body and it can be assumed that the foreground image moves at a constant speed so that it is displayed on the right side by 4 pixels in the next frame, for example, the second from the left in FIG. 35, the foreground component F05 / v of the first shutter time / v after the shutter is opened is shown in FIG.
It is equal to the foreground component of the third pixel from the left corresponding to the second shutter time / v from when the shutter has opened. Similarly, the foreground component F05 / v is the foreground component of the fourth pixel from the left in FIG. 35 corresponding to the third shutter time / v from when the shutter has opened, and the fifth component from the left in FIG. Of pixels,
It is equal to the foreground component corresponding to the fourth shutter time / v with the shutter open.

Since it can be assumed that the object corresponding to the foreground is a rigid body and the image of the foreground moves at a constant speed so as to be displayed on the right side by 4 pixels in the next frame, for example, the leftmost pixel in FIG. The foreground component F04 / v of the first shutter time / v from when the shutter is opened is the foreground component of the second pixel from the left in FIG. 35 corresponding to the second shutter time / v of the shutter opened. be equivalent to. Similarly, the foreground component F04 / v is the third pixel from the left in FIG.
The foreground component corresponding to the third shutter time / v from when the shutter is opened, and the foreground component corresponding to the fourth shutter time / v from the left of the fourth pixel in FIG. , Each equal.

Since the foreground area corresponding to the moving object includes motion blur as described above, it can be said to be a distortion area.

FIG. 36 is a model diagram in which the pixel values of pixels on one line including the uncovered background area are expanded in the time direction when the foreground moves toward the right side in the drawing. In FIG. 36, the motion amount v of the foreground
Is 4. Since one frame is a short time, it can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed. In FIG. 36, the image of the object corresponding to the foreground moves to the right by four pixels in the next frame with respect to a certain frame.

In FIG. 36, the leftmost pixel through the fourth pixel from the left belong to the background area. In FIG. 36, the fifth pixel from the left to the seventh pixel from the left belong to the mixed area that is the uncovered background. Fig. 36
In, the rightmost pixel belongs to the foreground area.

Since the object corresponding to the foreground covering the object corresponding to the background is moved so as to be removed from the front of the object corresponding to the background over time, the pixel of the pixel belonging to the uncovered background area The component included in the value changes from the foreground component to the background component at a certain point in the period corresponding to the shutter time.

For example, the pixel value with a bold frame in FIG.
M'is represented by Formula (2).

[0143] M '= F02 / v + F01 / v + B26 / v + B26 / v (2)

For example, the fifth pixel from the left includes the background component corresponding to three shutter times / v, and the foreground component corresponding to one shutter time / v.
The mixture ratio α of the th pixel is 3/4. The sixth pixel from the left includes the background component corresponding to two shutter times / v and the foreground component corresponding to two shutter times / v, so the mixture ratio α of the sixth pixel from the left is , 1/2. The seventh pixel from the left includes the background component corresponding to one shutter time / v, and the foreground component corresponding to three shutter times / v. Therefore, the mixture ratio α of the seventh pixel from the left is , 1/4.

When the equations (1) and (2) are generalized, the pixel value M is represented by the equation (3).

[0146]

[Equation 1] Here, α is a mixing ratio. B is the background pixel value, and Fi / v is the foreground component.

Since the object corresponding to the foreground is a rigid body and can be assumed to move at a constant speed, and the motion amount v is 4, for example, the shutter of the fifth pixel from the left in FIG. 36 is opened. First, foreground component F01 / with shutter time / v
v is equal to the foreground component of the sixth pixel from the left in FIG. 36 corresponding to the second shutter time / v from when the shutter has opened. Similarly, F01 / v corresponds to the foreground component of the seventh pixel from the left in FIG. 36 corresponding to the third shutter time / v from when the shutter has opened, and the eighth pixel from the left in FIG. ,
It is equal to the foreground component corresponding to the fourth shutter time / v with the shutter open.

Since the object corresponding to the foreground is a rigid body and can be assumed to move at a constant speed, and the number of virtual divisions is 4, for example, when the shutter of the sixth pixel from the left in FIG. 36 opens. First foreground component F0 with shutter time / v
2 / v is equal to the foreground component of the seventh pixel from the left in FIG. 36 corresponding to the second shutter time / v from when the shutter has opened. Similarly, the foreground component F02 / v is equal to the foreground component of the eighth pixel from the left in FIG. 36 corresponding to the third shutter time / v from when the shutter has opened.

Since the object corresponding to the foreground is a rigid body and can be assumed to move at a constant speed, and the amount of movement v is 4, for example, the shutter of the seventh pixel from the left in FIG. First, foreground component F03 / with shutter time / v
v is equal to the foreground component of the eighth pixel from the left in FIG. 36 corresponding to the second shutter time / v from when the shutter has opened.

In the description of FIGS. 34 to 36, the number of virtual divisions is four, but the number of virtual divisions corresponds to the motion amount v. The amount of movement v generally corresponds to the moving speed of the object corresponding to the foreground. For example, when the object corresponding to the foreground is moving so as to be displayed on the right side by four pixels in the next frame with respect to a certain frame, the motion amount v is set to 4. The virtual division number is set to 4 corresponding to the motion amount v. Similarly, for example, when the object corresponding to the foreground is moving so as to be displayed on the left side by 6 pixels in the next frame with respect to a certain frame, the motion amount v is set to 6 and the virtual division number is , 6, and so on.

37 and 38 described above,
The relationship between a mixed area including a foreground area, a background area, a covered background area, or an uncovered background area, and a foreground component and a background component corresponding to divided shutter times is shown.

FIG. 37 shows an example in which the pixels of the foreground area, the background area, and the mixed area are extracted from the image including the foreground corresponding to the object moving in front of the stationary background. In the example shown in FIG. 37, the object corresponding to the foreground is moving horizontally with respect to the screen.

The frame # n + 1 is a frame next to the frame #n, and the frame # n + 2 is a frame next to the frame # n + 1.

Pixels in the foreground area, background area, and mixed area extracted from any one of the frames #n to # n + 2 are extracted, and the pixel value of the extracted pixel is set to 4 as the motion amount v. A model expanded in the time direction is shown in FIG.

The pixel value of the foreground area is composed of four different foreground components corresponding to the shutter time / v period because the object corresponding to the foreground moves. For example, the leftmost pixel of the pixels in the foreground area shown in FIG. 38 is composed of F01 / v, F02 / v, F03 / v, and F04 / v. That is, the pixels in the foreground area include motion blur.

Since the object corresponding to the background is stationary, the light corresponding to the background input to the sensor 76a does not change during the period corresponding to the shutter time. In this case, the pixel value of the background area does not include motion blur.

The pixel values of the pixels belonging to the mixed area consisting of the covered background area or the uncovered background area are composed of the foreground component and the background component.

Next, when the image corresponding to the object is moving, the pixel values of the pixels in a plurality of frames which are arranged in a row adjacent to each other and which are at the same position on the frame are set in the time direction. The model developed in 1. will be explained. For example, when the image corresponding to the object is moving horizontally with respect to the screen, the pixels lined up on one line of the screen can be selected as the pixels lined up adjacently in one line.

FIG. 39 shows pixels in three frames of an image obtained by picking up an object corresponding to a still background, which are arranged in a row adjacent to each other, and are located at the same position on the frame. It is the model figure which expanded the value in the time direction. The frame #n is a frame next to the frame # n-1, and the frame # n + 1 is a frame next to the frame #n. Other frames are also referred to in the same manner.

The pixel values B01 to B12 shown in FIG. 39 are the pixel values of the pixels corresponding to the stationary background object. Since the object corresponding to the background is stationary, the pixel value of the corresponding pixel in frame # n-1 to frame n + 1 does not change. For example, the pixel in frame #n and the pixel in frame # n + 1, which correspond to the position of the pixel having the pixel value of B05 in frame # n-1, have the pixel value of B05, respectively.

FIG. 40 shows pixels arranged in a row adjacent to each other in three frames of an image in which an object corresponding to a stationary background and an object corresponding to the foreground moving to the right in the figure are picked up. FIG. 11 is a model diagram in which pixel values of pixels at the same position on a frame are expanded in the time direction. The model shown in FIG. 40 includes a covered background area.

In FIG. 40, it can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed, and the foreground image moves so as to be displayed on the right side by 4 pixels in the next frame. The quantity v is 4 and the number of virtual divisions is 4.

For example, the foreground component of the leftmost pixel of frame # n-1 in FIG. 40 at the first shutter time / v from when the shutter has opened is F12 / v, which is the second from the left in FIG. The foreground component of the pixel at the second shutter time / v after the shutter is opened is also F12 / v. The foreground component of the third pixel from the left in FIG. 40, which corresponds to the third shutter time / v from when the shutter has opened, and the fourth pixel from the left in FIG.
The foreground component at the fourth shutter time / v from when the shutter has opened is F12 / v.

The foreground component of the leftmost pixel of frame # n-1 in FIG. 40 corresponding to the second portion of the shutter time / v from when the shutter has opened is F11 / v, which is the second from the left in FIG. The foreground component of the pixel at the third shutter time / v from when the shutter has opened is also F11 / v. The foreground component of the third pixel from the left in FIG. 40 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is F11 / v.

The foreground component of the leftmost pixel of frame # n-1 in FIG. 40 corresponding to the third portion of the shutter time / v from when the shutter has opened is F10 / v, which is the second from the left in FIG. The foreground component of the pixel for the fourth shutter time / v from when the shutter has opened is also F10 / v. The shutter time of the leftmost pixel of frame # n-1 in FIG.
The foreground component of / v is F09 / v.

Since the object corresponding to the background is stationary, the background component of the second pixel from the left of frame # n-1 in FIG. 40 corresponding to the first shutter time / v from when the shutter has opened is B01. / v. The background component of the third pixel from the left of frame # n-1 in FIG. 40 corresponding to the first and second portions of the shutter time / v from when the shutter has opened is B02 / v. Figure 40
The background component of the fourth pixel from the left of frame # n-1 in the first to third shutter times / v from when the shutter has opened is B03 / v.

In frame # n-1 in FIG. 40, the leftmost pixel belongs to the foreground area, and the second to fourth pixels from the left belong to the mixed area which is the covered background area.

The fifth through twelfth pixels from the left of frame # n-1 in FIG. 40 belong to the background area, and the pixel values thereof are B04 through B11, respectively.

The first through fifth pixels from the left of frame #n in FIG. 40 belong to the foreground area. Frame #n
The foreground component of shutter time / v in the foreground region of
Any of F05 / v to F12 / v.

It can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed, and since the foreground image moves so as to be displayed on the right side by 4 pixels in the next frame, the frame #n in FIG. The foreground component of the fifth pixel from the left of is the first shutter time / v from when the shutter has opened:
It becomes F12 / v, and the foreground component of the sixth pixel from the left in FIG. 40 at the second shutter time / v from when the shutter has opened is also
It will be F12 / v. The foreground component of the seventh pixel from the left in FIG. 40, which is the third shutter time / v from when the shutter is open, and the fourth shutter time, which is the eighth pixel from the left, in FIG. 40 when the shutter is open. The foreground component of / v is F12 / v.

The foreground component of the fifth pixel from the left of frame #n in FIG. 40 corresponding to the second portion of the shutter time / v from when the shutter has opened is F11 / v, and the sixth component from the left in FIG. The foreground component of the pixel at the third shutter time / v from when the shutter has opened is also F11 / v. The foreground component of the seventh pixel from the left in FIG. 40 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is F11 / v.

The foreground component of the fifth pixel from the left of frame #n in FIG. 40 corresponding to the third portion of the shutter time / v from when the shutter has opened is F10 / v, which is the sixth component from the left in FIG. The foreground component of the pixel for the fourth shutter time / v from when the shutter has opened is also F10 / v. The foreground component of the fifth pixel from the left of frame #n in FIG. 40 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is F09 / v.

Since the object corresponding to the background is stationary, the background component of the sixth pixel from the left of frame #n in FIG. 40 at the first shutter time / v from when the shutter has opened is B05 / v. Becomes From left of frame #n in Figure 40
The background component of the second pixel at the first and second shutter times / v from when the shutter has opened is B06 / v. The background components of the eighth pixel from the left of frame #n in FIG. 40, which are the first through third shutter times / v from when the shutter has opened, are:
It will be B07 / v.

In frame #n in FIG. 40, the sixth through eighth pixels from the left belong to the mixed area, which is a covered background area.

The ninth through twelfth pixels from the left of frame #n in FIG. 40 belong to the background area, and the pixel values are
They are B08 to B11, respectively.

The first through ninth pixels from the left of frame # n + 1 in FIG. 40 belong to the foreground area. flame
The foreground component of shutter time / v in the # n + 1 foreground area is any one of F01 / v to F12 / v.

It can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed, and since the foreground image moves so as to be displayed on the right side by 4 pixels in the next frame, frame #n in FIG. Of the 9th pixel from the left of +1
The foreground component of the first shutter time / v after the shutter is opened is F12 / v, and the foreground component of the tenth pixel from the left in FIG. It will be F12 / v. The foreground component of the 11th pixel from the left in FIG. 40 for the third shutter time / v from when the shutter has opened, and the 4th shutter time of the 12th pixel from the left in FIG. 40 for which the shutter has opened The foreground component of / v is F12 / v.

The foreground component of the ninth pixel from the left of frame # n + 1 in FIG. 40 during the second shutter time / v from when the shutter has opened is F11 / v. From 1
The foreground component of the 0th pixel at the third shutter time / v from when the shutter has opened is also F11 / v. The foreground component of the eleventh pixel from the left in FIG. 40, which is the fourth portion of the shutter time / v from when the shutter has opened is F11 / v.

The foreground component of the ninth pixel from the left of frame # n + 1 in FIG. 40 corresponding to the third portion of the shutter time / v from when the shutter has opened is F10 / v, from the left in FIG. The 10th pixel is the 4th shutter time / v after the shutter is opened
The foreground component of is also F10 / v. Frame # n + in Figure 40
The foreground component of the ninth pixel from the left of 1 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is F09 / v.

Since the object corresponding to the background is stationary, the background component of the tenth pixel from the left of frame # n + 1 in FIG. 40 at the first shutter time / v from when the shutter has opened is B09. / v. The background component of the eleventh pixel from the left of frame # n + 1 in FIG. 40 corresponding to the first and second portions of the shutter time / v from when the shutter has opened is B10 / v. The background components of the 12th pixel from the left of frame # n + 1 in Fig. 40 corresponding to the first through third portions of the shutter time / v from when the shutter has opened are B11 / v.

In frame # n + 1 in FIG. 40, the tenth to twelfth pixels from the left correspond to the mixed area which is the covered background area.

FIG. 41 is a model diagram of an image obtained by extracting the foreground components from the pixel values shown in FIG.

FIG. 42 shows pixels arranged in a row adjacent to each other in three frames of an image obtained by capturing a foreground corresponding to an object moving to the right side in the drawing together with a stationary background. It is the model figure which expanded the pixel value of the pixel of the same position above in the time direction. In FIG. 42,
Includes uncovered background area.

In FIG. 42, it can be assumed that the object corresponding to the foreground is a rigid body and is moving at a constant speed. Since the object corresponding to the foreground is moving so as to be displayed on the right side by four pixels in the next frame, the movement amount v is 4.

For example, the shutter time / v of the leftmost pixel of frame # n-1 in FIG.
The foreground component of F is 13 / v, which is the second shutter time / v of the second pixel from the left in FIG.
The foreground component of is also F13 / v. The third pixel from the left in FIG. 42 is the third shutter time / v from when the shutter has opened.
42 and the foreground component of the fourth pixel from the left in FIG. 42 corresponding to the fourth portion of the shutter time / v from when the shutter has opened are F13 / v.

The foreground component of the second pixel from the left of frame # n-1 in FIG. 42 corresponding to the first portion of the shutter time / v from when the shutter has opened is F14 / v, which is the third component from the left in FIG. The foreground component of the second pixel at the shutter time / v after the shutter is opened is also F14 / v. The foreground component of the third pixel from the left in FIG. 42 corresponding to the first portion of the shutter time / v from when the shutter has opened is F15 / v.

Since the object corresponding to the background is stationary, the leftmost pixel of frame # n-1 in FIG.
Second to fourth shutter time / v after the shutter opens
The background component is B25 / v. Frame # n- in FIG. 42
The background component of the second pixel from the left of 1 at the third and fourth portions of the shutter time / v from when the shutter has opened is B26 / v
Becomes The background component of the third pixel from the left of frame # n-1 in FIG. 42 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is B27 / v.

In frame # n-1 in FIG. 42, the leftmost pixel through the third pixel belong to the mixed area which is an uncovered background area.

The fourth through twelfth pixels from the left of frame # n-1 in FIG. 42 belong to the foreground area. The foreground component of the frame is any of F13 / v to F24 / v.

The leftmost pixel through the fourth pixel from the left of frame #n in FIG. 42 belong to the background area, and the pixel value is
They are B25 to B28, respectively.

It can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed, and since the foreground image moves so as to be displayed on the right side by 4 pixels in the next frame, frame #n in FIG. The foreground component of the fifth pixel from the left of is the first shutter time / v from when the shutter has opened:
F13 / v, and the foreground component of the sixth pixel from the left in FIG. 42 corresponding to the second portion of the shutter time / v from when the shutter has opened is also
It becomes F13 / v. The foreground component of the seventh pixel from the left in FIG. 42 for which the shutter has opened and the third shutter time / v, and the eighth pixel from the left in FIG. 42 for which the shutter has opened and the fourth shutter time The foreground component of / v is F13 / v.

The foreground component of the sixth pixel from the left of frame #n in FIG. 42 at the first shutter time / v from when the shutter has opened is F14 / v, and the seventh pixel from left in FIG. The foreground component of the second shutter time / v after the shutter opens is also F14 / v. The foreground component of the eighth pixel from the left in FIG. 42 corresponding to the first portion of the shutter time / v from when the shutter has opened is F15 / v.

Since the object corresponding to the background is stationary, the second to fourth shutter time after the shutter has opened for the fifth pixel from the left of frame #n in FIG.
The background component of / v is B29 / v. Frame # in Figure 42
The background component of the sixth pixel from the left of n corresponding to the third and fourth portions of the shutter time / v from when the shutter has opened is B30 / v. Of the seventh pixel from the left of frame #n in FIG. 42,
The background component at the fourth shutter time / v from when the shutter has opened is B31 / v.

In frame #n in FIG. 42, 5 from the left
The seventh pixel to the seventh pixel belong to the mixed area which is the uncovered background area.

The eighth through twelfth pixels from the left of frame #n in FIG. 42 belong to the foreground area. flame
The value corresponding to the period of shutter time / v in the foreground area of #n is any one of F13 / v to F20 / v.

The leftmost pixel through the eighth pixel from the left of frame # n + 1 in FIG. 42 belong to the background area, and the pixel values are B25 through B32, respectively.

It can be assumed that the object corresponding to the foreground is a rigid body and moves at a constant speed, and since the foreground image moves so as to be displayed on the right side by 4 pixels in the next frame, frame #n in FIG. Of the 9th pixel from the left of +1
The foreground component at the first shutter time / v after the shutter is opened is F13 / v, and the foreground component at the tenth pixel from the left in FIG. 42 at the second shutter time / v after the shutter is opened is also It becomes F13 / v. The foreground component of the 11th pixel from the left in FIG. 42, which is the third shutter time / v from when the shutter is open, and the 12th pixel from the left in FIG. 42, which is the fourth shutter time from the open shutter The foreground component of / v is F13 / v.

The foreground component of the tenth pixel from the left of frame # n + 1 in FIG. 42 corresponding to the first portion of the shutter time / v from when the shutter has opened is F14 / v, which is the 11th pixel from the left in FIG. The foreground component of the second pixel at the shutter time / v after the shutter is opened is also F14 / v. The foreground component of the twelfth pixel from the left in FIG. 42 corresponding to the first portion of the shutter time / v from when the shutter has opened is F15 / v.

Since the object corresponding to the background is stationary, the background of the ninth pixel from the left of frame # n + 1 in FIG. 42, which is the second to fourth shutter time / v from when the shutter has opened. The component of is B33 / v. The shutter opens for the 10th pixel from the left of frame # n + 1 in FIG.
The background component of the second and fourth shutter times / v is B3
It becomes 4 / v. The background component of the eleventh pixel from the left of frame # n + 1 in FIG. 42 corresponding to the fourth portion of the shutter time / v from when the shutter has opened is B35 / v.

In frame # n + 1 in FIG. 42, the ninth through eleventh pixels from the left belong to the mixed area, which is an uncovered background area.

The twelfth pixel from the left of frame # n + 1 in FIG. 42 belongs to the foreground area. The foreground components of shutter time / v in the foreground area of frame # n + 1 are F13 / v to F13 / v.
It is either 16 / v.

FIG. 43 is a model diagram of an image obtained by extracting the foreground components from the pixel values shown in FIG.

Returning to FIG. 27, the area specifying unit 103 uses the pixel values of a plurality of frames to set a flag indicating that it belongs to the foreground area, the background area, the covered background area, or the uncovered background area for each pixel. And is supplied as region information to the mixture ratio calculation unit 104 and the motion blur adjustment unit 106.

The mixture ratio calculation unit 104 calculates the mixture ratio α for each pixel of the pixels included in the mixed region based on the pixel values of a plurality of frames and the region information, and uses the calculated mixture ratio α as the foreground / background separation. It is supplied to the unit 105.

The foreground / background separating section 105 extracts a foreground component image consisting of only foreground components based on the pixel values of a plurality of frames, area information, and the mixture ratio α, and supplies it to the motion blur adjusting section 106. .

The motion blur adjusting unit 106 detects the foreground component image supplied from the foreground / background separating unit 105, and the motion detecting unit 102.
The motion vector supplied from
The amount of motion blur included in the foreground component image is adjusted based on the area information supplied from the foreground component image, and the foreground component image with the adjusted amount of motion blur is output.

The process of adjusting the amount of motion blur by the separation processing server 11 will be described with reference to the flowchart of FIG. In step S11, the area specifying unit 103
Based on the input image, an area specifying process is performed for generating, for each pixel of the input image, area information indicating whether the pixel belongs to the foreground area, the background area, the covered background area, or the uncovered background area. Details of the area specifying process will be described later. The area specifying unit 103 supplies the generated area information to the mixture ratio calculating unit 104.

In step S11, the area specifying unit 103 distinguishes the foreground area, the background area, or the mixed area (covered background area or uncovered background area) for each pixel of the input image based on the input image. Area information indicating which one of (No) is included may be generated. In this case, the foreground / background separation unit 105 and the motion blur adjustment unit 106 determine whether the mixed region is a covered background region or an uncovered background region based on the direction of the motion vector. For example, when the foreground area, the mixed area, and the background area are arranged in order corresponding to the direction of the motion vector, the mixed area is determined to be a covered background area, and the background corresponding to the direction of the motion vector is determined. When the region, the mixed region, and the foreground region are lined up in order, the mixed region is determined as the uncovered background region.

At step S12, the mixture ratio calculation unit 1
04 calculates the mixing ratio α for each pixel included in the mixed area based on the input image and the area information. Details of the processing for calculating the mixing ratio will be described later. The mixture ratio calculation unit 104 supplies the calculated mixture ratio α to the foreground / background separation unit 105.

In step S13, the foreground / background separating unit 105 extracts the foreground component from the input image based on the area information and the mixture ratio α and supplies it to the motion blur adjusting unit 106 as a foreground component image.

In step S14, the motion-blur adjusting unit 106 is a continuous pixel lined up in the motion direction based on the motion vector and the area information, and is any one of the uncovered background area, the foreground area, and the covered background area. A processing unit indicating the position on the image of that belonging to the crab is generated, and the amount of motion blur included in the foreground component corresponding to the processing unit is adjusted. Details of the process of adjusting the amount of motion blur will be described later.

In step S15, the separation processing server 11 determines whether or not the processing has been completed for the entire screen. If it is determined that the processing has not been completed for the entire screen, the process proceeds to step S14 and the processing unit is changed. The process of adjusting the amount of motion blur for the corresponding foreground component is repeated.

When it is determined in step S15 that the processing has been completed for the entire screen, the processing ends.

In this way, the separation processing server 11 can separate the foreground and the background and adjust the amount of motion blur included in the foreground. That is, the separation processing server 11
The amount of motion blur included in the sample data, which is the pixel value of the foreground pixel, can be adjusted.

Hereinafter, the area specifying unit 103 and the mixture ratio calculating unit 1
04, foreground / background separation unit 105, and motion blur adjustment unit 1
The respective configurations of 06 will be described.

FIG. 45 is a block diagram showing an example of the structure of the area specifying unit 103. The area specifying unit 103 having the configuration shown in FIG. 45 does not use the motion vector. The frame memory 201 stores the input image on a frame-by-frame basis. In the frame memory 201, the processing target is frame #n.
, Frame # n-2, which is the frame two frames before frame #n, and frame # n-1, which is the frame one frame before frame #n, frame #n, and one frame after frame #n. The frame # n + 1, which is the frame, and the frame # n + 2, which is the frame two frames after the frame #n, are stored.

The static / moving judgment unit 202-1 determines the pixel value of the pixel of the frame # n + 2 at the same position as the position of the pixel of the frame #n which is the area specifying target on the image, and the pixel of the frame #n. The pixel value of the pixel of frame # n + 1 located at the same position on the image as the pixel for which the region is to be specified is stored in the frame memory 2
Then, the absolute value of the difference between the read pixel values is calculated. The static / moving determination unit 202-1 determines whether or not the absolute value of the difference between the pixel value of frame # n + 2 and the pixel value of frame # n + 1 is greater than a preset threshold Th. When it is determined that the absolute value of the difference is larger than the threshold value Th, the static / moving determination indicating the movement is supplied to the area determination unit 203-1. Frame #n
When it is determined that the absolute value of the difference between the pixel value of the pixel of +2 and the pixel value of the pixel of frame # n + 1 is less than or equal to the threshold value Th, the stationary / moving determination unit 202-1 determines that the stationary / moving indicates stationary. Area determination unit 2
03-1 is supplied.

The static / moving judgment unit 202-2 determines the pixel value of the pixel of frame # n + 1 at the same position as the position on the image of the pixel which is the object of region identification of frame #n, and the value of frame #n. The pixel value of the target pixel is read from the frame memory 201, and the absolute value of the difference between the pixel values is calculated. The stationary / moving determination unit 202-2 determines whether or not the absolute value of the difference between the pixel value of frame # n + 1 and the pixel value of frame #n is larger than a preset threshold Th, When it is determined that the absolute value of the difference is larger than the threshold value Th, the static / moving determination indicating the movement is supplied to the area determination unit 203-1 and the area determination unit 203-2. When it is determined that the absolute value of the difference between the pixel value of the pixel of frame # n + 1 and the pixel value of the pixel of frame #n is less than or equal to the threshold value Th, the stationary / moving determination unit 202-2 indicates stationary. The static / moving determination is supplied to the area determination unit 203-1 and the area determination unit 203-2.

The static / moving determination unit 202-3 is located at the same position as the pixel value of the pixel for which the area #n is to be specified and the position of the pixel for which the area is specified for the frame #n on the image. The pixel value of the pixel of frame # n-1 is stored in the frame memory 2
The absolute value of the difference between the pixel values is calculated by reading from 01.
The static / moving determination unit 202-3 determines whether or not the absolute value of the difference between the pixel value of frame #n and the pixel value of frame # n-1 is greater than a preset threshold Th, and the pixel value When it is determined that the absolute value of the difference is larger than the threshold value Th, the static / moving determination indicating the movement is performed as the area determination unit 203-2 and the area determination unit 203
-3. Pixel value of pixel in frame #n and frame
When it is determined that the absolute value of the difference between the pixel value of the pixel # n-1 and the pixel value is less than or equal to the threshold value Th, the static / moving determination unit 202-3 determines the static / moving determination indicating stillness to the region determination unit 203-2. And the area determination unit 203-3.

The static / moving judgment unit 202-4 determines the pixel value of the pixel of frame # n-1 at the same position as the position on the image of the pixel which is the object of region identification of frame #n, and the value of frame #n. The frame memory 2 stores the pixel value of the pixel of frame # n-2 at the same position as the position of the pixel of the region specifying target on the image.
The absolute value of the difference between the pixel values is calculated by reading from 01.
The static / moving determination unit 202-4 determines whether or not the absolute value of the difference between the pixel value of frame # n-1 and the pixel value of frame # n-2 is larger than a preset threshold Th, When it is determined that the absolute value of the difference between the pixel values is larger than the threshold value Th, the static / moving determination indicating the movement is supplied to the area determination unit 203-3. Frame #n
When it is determined that the absolute value of the difference between the pixel value of the pixel of -1 and the pixel value of the pixel of frame # n-2 is less than or equal to the threshold Th, the static / moving determination unit 202-4 determines that the static The motion determination is supplied to the area determination unit 203-3.

The area determination unit 203-1 is the stationary / moving determination unit 20.
When the static / moving determination supplied from 2-1 indicates stillness and the static / moving determination supplied from the static / moving determination unit 202-2 indicates movement, a pixel which is a target of region identification in frame #n Is determined to belong to the uncovered background area, and "1" indicating that it belongs to the uncovered background area is set to the uncovered background area determination flag corresponding to the pixel of which the area is determined.

The area determining unit 203-1 is the stationary / moving determining unit 20.
When the static / movement determination supplied from 2-1 indicates movement, or the static / movement determination supplied from the static / moving determination unit 202-2 indicates static, it is a target for region identification in frame #n. It is determined that the pixel does not belong to the uncovered background area, and "0" indicating that the pixel does not belong to the uncovered background area in the uncovered background area determination flag corresponding to the determined pixel of the area.
To set.

The area determination unit 203-1 is thus
The uncovered background area determination flag in which “1” or “0” is set is supplied to the determination flag storage frame memory 204.

The area determining unit 203-2 is the stationary / moving determining unit 20.
When the static / moving determination supplied from 2-2 indicates static and the static / moving determination supplied from the static-moving determining unit 202-3 indicates static, the pixel that is the target of region identification in frame #n Is determined to belong to the still area, and the still area determination flag corresponding to the pixel of the area to be determined is set to "1" indicating that it belongs to the still area.

The area determining unit 203-2 is the stationary / moving determining unit 20.
When the static / movement determination supplied from 2-2 indicates a motion, or when the static / movement determination supplied from the static-movement determination unit 202-3 indicates a motion, it is a target for region identification in frame #n. It is determined that the pixel does not belong to the still area, and "0" indicating that the pixel does not belong to the still area is set in the still area determination flag corresponding to the pixel of which the area is determined.

The area determination unit 203-2 thus reads "
The still area determination flag in which “1” or “0” is set is supplied to the determination flag storage frame memory 204.

The area determination unit 203-2 is the stationary / moving determination unit 20.
When the static / moving determination supplied from 2-2 indicates a motion and the static / moving determination supplied from the static / moving determination unit 202-3 indicates a motion, a pixel which is a target of region identification in frame #n Is determined to belong to the motion area, and "1" indicating that it belongs to the motion area is set in the motion area determination flag corresponding to the pixel of which the area is determined.

The area determining unit 203-2 is the stationary / moving determining unit 20.
When the static / moving determination supplied from 2-2 indicates static, or when the static / moving determination supplied from the static-moving determining unit 202-3 indicates static, it is a target for region identification in frame #n. It is determined that the pixel does not belong to the motion area, and "0" indicating that the pixel does not belong to the motion area is set in the motion area determination flag corresponding to the pixel of which the area is determined.

The area determination unit 203-2 thus reads "
The motion area determination flag in which “1” or “0” is set is supplied to the determination flag storage frame memory 204.

The area determination unit 203-3 is a stationary / moving determination unit 20.
When the static / moving determination supplied from 2-3 indicates motion and the static / moving determination supplied from the static / moving determination unit 202-4 indicates static, the pixel that is the target of region identification in frame #n Is determined to belong to the covered background area, and the covered background area determination flag corresponding to the pixel of which the area is determined is set to "1" indicating that it belongs to the covered background area.

The area determination unit 203-3 is equivalent to the stationary / moving determination unit 20.
When the static / moving determination supplied from 2-3 indicates static, or when the static / moving determination supplied from the static-moving determining unit 202-4 indicates movement, it is a target for region identification in frame #n. It is determined that the pixel does not belong to the covered background area, and "0" indicating that the pixel does not belong to the covered background area is set in the covered background area determination flag corresponding to the pixel of which the area is determined.

The area determination unit 203-3 is thus
The covered background area determination flag in which “1” or “0” is set is stored as the determination flag in the frame memory 20.
Supply to 4.

The judgment flag storage frame memory 204 is
The uncovered background area determination flag supplied from the area determination unit 203-1, the still area determination flag supplied from the area determination unit 203-2, the area determination unit 203-
The motion area determination flag supplied from No. 2 and the covered background area determination flag supplied from the area determination unit 203-3 are stored.

The judgment flag storage frame memory 204 is
The synthesizing unit 2 stores the stored uncovered background area determination flag, still area determination flag, moving area determination flag, and covered background area determination flag.
Supply to 05. The synthesizing unit 205 determines that each pixel is based on the uncovered background area determination flag, the still area determination flag, the moving area determination flag, and the covered background area determination flag supplied from the determination flag storage frame memory 204. Area information indicating that it belongs to any of the covered background area, the still area, the moving area, and the covered background area is generated and supplied to the determination flag storage frame memory 206.

The judgment flag storage frame memory 206 is
The area information supplied from the composition unit 205 is stored, and the stored area information is output.

Next, an example of processing of the area specifying unit 103 is shown in FIG.
6 to 50, description will be made.

When the object corresponding to the foreground is moving, the position of the image corresponding to the object on the screen changes for each frame. As shown in FIG. 46, in frame #n, the image corresponding to the object located at the position indicated by Yn (x, y) is Yn + 1 (x, y in frame # n + 1, which is the next frame. Located in y).

FIG. 22 shows a model diagram in which the pixel values of pixels arranged in one row adjacent to the moving direction of the image corresponding to the foreground object are expanded in the time direction. For example, when the motion direction of the image corresponding to the foreground object is horizontal to the screen, the model diagram in FIG. 47 shows a model in which the pixel values of adjacent pixels on one line are expanded in the time direction.

In FIG. 47, the line in frame #n is the same as the line in frame # n + 1.

In frame #n, the foreground components corresponding to the objects contained in the second through thirteenth pixels from the left are located in the sixth through seventeenth pixels from the left in frame # n + 1. included.

In frame #n, the pixels belonging to the covered background area are the 11th to 13th pixels from the left, and the pixels belonging to the uncovered background area are the 2nd to 4th pixels from the left. .
In frame # n + 1, the pixels belonging to the covered background area are the 15th to 17th pixels from the left, and the pixels belonging to the uncovered background area are the 6th to 8th pixels from the left.

In the example shown in FIG. 47, since the foreground component contained in frame #n has moved by 4 pixels in frame # n + 1, the motion amount v is 4. The number of virtual partitions is
It corresponds to the movement amount v and is 4.

Next, the change in the pixel value of the pixels belonging to the mixed area before and after the frame of interest will be described.

In frame #n shown in FIG. 48 in which the background is stationary and the motion amount v of the foreground is 4, the pixels belonging to the covered background area are the fifteenth to seventeenth pixels from the left.
It is the th pixel. Since the motion amount v is 4, the fifteenth to seventeenth pixels from the left in the immediately preceding frame # n−1 include only the background component and belong to the background region. In frame # n-2, which is one frame before, the fifteenth to seventeenth pixels from the left include only the background component and belong to the background area.

Here, since the object corresponding to the background is stationary, the pixel value of the fifteenth pixel from the left of frame # n-1 is the pixel value of the fifteenth pixel from the left of frame # n-2. Does not change from Similarly, 1 from the left of frame # n-1
The pixel value of the sixth pixel does not change from the pixel value of the 16th pixel from the left of frame # n-2, and the pixel value of the 17th pixel from the left of frame # n-1 is the same as that of frame # n- 17 from left of 2
It does not change from the pixel value of the second pixel.

That is, the frame corresponding to the pixels belonging to the covered background area in frame #n
The pixels of # n-1 and frame # n-2 are composed of only the background component, and the pixel values do not change, so the absolute value of the difference is almost 0. Therefore, frame # n-1 and frame # n-1 corresponding to the pixels belonging to the mixed area in frame #n
The static / moving determination for the pixel # n-2 is performed by the static / moving determining unit 202-4.
Is determined to be stationary.

Pixels belonging to the covered background area in frame #n include foreground components, and therefore have different pixel values from those in the case of only background components in frame # n-1. Therefore, the still / moving determination unit 202-3 determines that the pixel belonging to the mixed area in the frame #n and the corresponding pixel in the frame # n−1 are moving.

As described above, the area determination unit 203-3 is supplied with the result of the static / moving determination indicating the movement from the static / moving determination unit 202-3, and the static / moving determination unit 202-3 determines the static / moving determination indicating the static. When the result is provided, it is determined that the corresponding pixel belongs to the covered background area.

In frame #n shown in FIG. 49 in which the background is stationary and the motion amount v of the foreground is 4, the pixels included in the uncovered background area are the second to fourth pixels from the left. Since the motion amount v is 4, the second to fourth pixels from the left in the subsequent frame # n + 1 include only the background component and belong to the background region. In the frame # n + 2 that is one frame further after, the second to fourth pixels from the left include only the background component and belong to the background area.

Since the object corresponding to the background is stationary, the pixel value of the second pixel from the left of frame # n + 2 is the pixel value of the second pixel from the left of frame # n + 1. Does not change from Similarly, the pixel value of the third pixel from the left of frame # n + 2 does not change from the pixel value of the third pixel from the left of frame # n + 1, and it is the fourth pixel from the left of frame # n + 2. The pixel value of the pixel is unchanged from the pixel value of the fourth pixel from the left of frame # n + 1.

That is, the pixels of frame # n + 1 and frame # n + 2, which correspond to the pixels belonging to the uncovered background area in frame #n, are composed of only the background component and the pixel value does not change. The absolute value of the difference is almost zero. Therefore, the still / moving determination unit 20 performs the still / moving determination on the pixels of the frame # n + 1 and the frame # n + 2 corresponding to the pixels belonging to the mixed area in the frame #n.
It is determined to be stationary according to 2-1.

Since the pixels belonging to the uncovered background area in frame #n include the foreground components,
In the case where it consists only of the background component in frame # n + 1,
Pixel value is different. Therefore, the still / moving determination unit 202-2 determines that the pixel belonging to the mixed area in the frame #n and the corresponding pixel in the frame # n + 1 are moving.

As described above, the area determining unit 203-1 is supplied with the result of the static / moving determination indicating the motion from the static / moving determining unit 202-2, and the static / moving determination unit 202-1 determines the static / moving determination indicating the static. When the result is supplied, it is determined that the corresponding pixel belongs to the uncovered background area.

FIG. 50 is a diagram showing the judgment conditions of the area specifying unit 103 in frame #n. The pixel of frame # n-2 at the same position on the image as the position of the pixel that is the target of frame #n determination, and the same position as the position on the image of the pixel that is the target of determination of frame #n A pixel in frame # n-1 is determined to be still, and a pixel in frame # n-1 and a pixel in frame #n that are at the same position on the image as the pixel to be determined in frame #n When and are determined to be motions, the region identifying unit 103 determines that the pixel to be the determination target of the frame #n belongs to the covered background region.

The pixel of frame # n-1 and the pixel of frame #n, which are at the same position on the image as the pixel to be judged in frame #n, are determined to be stationary, and the pixel of frame #n is determined. When it is determined that the pixel of the frame # n + 1 at the same position on the image as the pixel of the determination target of the frame #n is still, the area specifying unit 103 determines that the determination of the frame #n is performed. It is determined that the target pixel belongs to the still region.

It is determined that the pixel of frame # n-1 and the pixel of frame #n at the same position on the image as the pixel to be determined in frame #n and the pixel of frame #n are moving. When it is determined that the pixel of frame # n + 1 and the pixel of frame # n + 1 located at the same position on the image as the determination target of frame #n are in motion, the area specifying unit 103 determines whether frame #n is determined. It is determined that the target pixel of (1) belongs to the motion area.

It is determined that the pixel of frame #n and the pixel of frame # n + 1 at the same position as the position of the pixel to be determined in frame #n on the image are in motion and the determination of frame #n is made. The pixel of frame # n + 1 located at the same position on the image as the pixel of interest, and the frame # n + located at the same position on the image as the pixel of interest for frame #n determination. When it is determined that the 2nd pixel is still, the area specifying unit 103 determines that the pixel that is the determination target of the frame #n belongs to the uncovered background area.

FIG. 51 is a diagram showing an example of the result of specifying the area by the area specifying unit 103. In FIG. 51A, the pixels determined to belong to the covered background area are displayed in white. In FIG. 51B, pixels determined to belong to the uncovered background area are displayed in white.

In FIG. 51C, the pixels determined to belong to the motion area are displayed in white. FIG. 51
In (D), the pixels determined to belong to the still area are displayed in white.

FIG. 52 is a diagram showing, as an image, area information indicating a mixed area in the area information output from the judgment flag storage frame memory 206. In FIG. 52, the pixels determined to belong to the covered background area or the uncovered background area, that is, the pixels determined to belong to the mixed area are displayed in white.
The area information indicating the mixed area output from the determination flag storage frame memory 206 indicates a mixed area and a textured portion surrounded by a texture-free portion in the foreground area.

Next, the area specifying processing of the area specifying unit 103 will be described with reference to the flowchart of FIG. In step S201, the frame memory 201 acquires images of the frame # n-2 to the frame # n + 2 including the frame #n to be determined.

In step S202, the stationary / moving determination unit 2
02-3 determines whether or not the pixel of frame # n-1 and the pixel at the same position of frame #n are still. If it is determined to be still, the process proceeds to step S203, and the stationary / moving determination unit 202- Two
Determines whether the pixel of frame #n and the pixel at the same position of frame # n + 1 are still.

If it is determined in step S203 that the pixel of frame #n and the pixel of the same position in frame # n + 1 are still, the process proceeds to step S204, and the area determination unit 20
3-2 sets “1” indicating that the pixel belongs to the still area in the still area determination flag corresponding to the pixel of which the area is determined. The area determination unit 203-2 supplies the still area determination flag to the determination flag storage frame memory 204, and the procedure proceeds to step S205.

In step S202, frame # n-1
If it is determined that there is a motion between the pixel of # 1 and the pixel of the same position of frame #n, or in step S203, the pixel of frame #n and the pixel of the same position of frame # n + 1 are determined to be motion. In this case, since the pixel of frame #n does not belong to the still area, the process of step S204 is skipped and the procedure proceeds to step S205.

In step S205, the stationary / moving determination unit 2
02-3 determines whether or not there is a motion between the pixel of frame # n-1 and the pixel at the same position of frame #n. If it is determined to be motion, the process proceeds to step S206, and the stationary / moving determination unit 202- Two
Determines whether the pixel of frame #n and the pixel at the same position of frame # n + 1 are in motion.

If it is determined in step S206 that the pixel of frame #n and the pixel at the same position of frame # n + 1 are in motion, the process proceeds to step S207 and the area determination unit 20
3-2 sets “1” indicating that the pixel belongs to the motion area, in the motion area determination flag corresponding to the pixel of which the area is determined. The area determination unit 203-2 supplies the motion area determination flag to the determination flag storage frame memory 204, and the procedure proceeds to step S208.

In step S205, frame # n-1
When it is determined that the pixel of the frame #n and the pixel of the same position of the frame #n are static, or in step S206, the pixel of the frame #n and the pixel of the same position of the frame # n + 1 are determined to be static. In this case, since the pixel of frame #n does not belong to the motion area, the process of step S207 is skipped and the procedure proceeds to step S208.

In step S208, the stationary / moving determination unit 2
02-4 determines whether or not the pixel of frame # n-2 and the pixel of the same position of frame # n-1 are still, and when it is determined to be still, the process proceeds to step S209, and the static / moving determination unit 202-
In No. 3, it is determined whether or not the pixel of frame # n-1 and the pixel at the same position of frame #n are moving.

In step S209, frame # n-1
If it is determined that there is a motion between the pixel of the pixel and the pixel of the same position of frame #n, the process proceeds to step S210, and the area determination unit 20
3-3 sets the covered background area determination flag corresponding to the pixel of which the area is determined to "1" indicating that the area belongs to the covered background area. The area determination unit 203-3 supplies the covered background area determination flag to the determination flag storage frame memory 204, and the procedure proceeds to step S211.

In step S208, frame # n-2
If it is determined that there is a motion between the pixel of the frame # n-1 and the pixel of the same position of the frame # n-1, or in step S209, the pixel of the frame # n-1 and the pixel of the same position of the frame #n,
If it is determined to be still, the pixels of frame #n do not belong to the covered background area, so step S21
The process of 0 is skipped, and the procedure is step S211.
Proceed to.

In step S211, the stationary / moving determination unit 2
02-2 determines whether or not there is a motion between the pixel of frame #n and the pixel at the same position of frame # n + 1. If it is determined to be motion, the process proceeds to step S212, and the stationary / moving determination unit 202- 1
Determines whether the pixel of frame # n + 1 and the pixel of the same position of frame # n + 2 are still.

In step S212, frame # n + 1
When it is determined that the pixel of the pixel and the pixel of the same position of the frame # n + 2 are still, the process proceeds to step S213, and the area determination unit 2
03-1 sets "1" indicating belonging to the uncovered background area in the uncovered background area determination flag corresponding to the pixel of which the area is determined. The area determination unit 203-1 supplies the uncovered background area determination flag to the determination flag storage frame memory 204, and the procedure proceeds to step S214.

When it is determined in step S211 that the pixel of frame #n and the pixel of the same position in frame # n + 1 are still, or in step S212, the pixel of frame # n + 1 and the pixel of frame # n + 1 With the pixel at the same position of +2,
If it is determined to be moving, the pixels of frame #n do not belong to the uncovered background area, so step S
The processing of step 213 is skipped, and the procedure is step S2.
Proceed to 14.

In step S214, the area specifying unit 1
03 determines whether or not areas have been specified for all pixels of frame #n, and if it is determined that areas have not been specified for all pixels of frame #n, the procedure returns to step S202, The region specifying process is repeated for other pixels.

If it is determined in step S214 that the regions have been specified for all the pixels of frame #n,
In step S215, the combining unit 205 generates area information indicating a mixed area based on the uncovered background area determination flag and the covered background area determination flag stored in the determination flag storage frame memory 204, Furthermore, area information indicating that each pixel belongs to any of an uncovered background area, a still area, a moving area, and a covered background area is generated, and the generated area information is set in the determination flag storage frame memory 206. Then, the process ends.

In this way, the area specifying unit 103 determines, for each of the pixels included in the frame, the motion area,
Area information indicating that the area belongs to a static area, an uncovered background area, or a covered background area can be generated.

The area specifying unit 103 applies the logical sum to the area information corresponding to the uncovered background area and the covered background area to generate the area information corresponding to the mixed area and includes it in the frame. Region information including a flag indicating that the pixel belongs to a moving region, a still region, or a mixed region may be generated for each of the pixels.

When the object corresponding to the foreground has a texture, the area specifying unit 103 can specify the moving area more accurately.

The area specifying unit 103 can output area information indicating a moving area as area information indicating a foreground area, and area information indicating a still area as area information indicating a background area.

Although it has been described that the object corresponding to the background is stationary, the above-described processing for specifying the area can be applied even if the image corresponding to the background area includes motion. For example, when the image corresponding to the background area is moving uniformly, the area specifying unit 103 shifts the entire image according to this movement, and performs the same processing as when the object corresponding to the background is stationary. To do. Also,
When the image corresponding to the background region includes a motion that differs locally, the region identifying unit 103 selects a pixel corresponding to the motion and executes the above-described processing.

FIG. 54 is a block diagram showing another example of the structure of the area specifying unit 103. The area specifying unit 103 illustrated in FIG. 54 does not use the motion vector. The background image generation unit 301 generates a background image corresponding to the input image and supplies the generated background image to the binary object image extraction unit 302. The background image generation unit 301 extracts, for example, an image object corresponding to a background object included in the input image to generate a background image.

FIG. 55 shows an example of a model diagram in which the pixel values of pixels arranged in one row adjacent to the moving direction of the image corresponding to the foreground object are expanded in the time direction. For example, when the motion direction of the image corresponding to the foreground object is horizontal to the screen, the model diagram in FIG. 55 shows a model in which the pixel values of adjacent pixels on one line are expanded in the time direction.

In FIG. 55, the line in frame #n is the same as the line in frame # n-1 and frame # n + 1.

In frame #n, the foreground components corresponding to the objects contained in the sixth through seventeenth pixels from the left are in the second through thirteenth pixels from the left in frame # n-1. It is included in the 10th to 21st pixels from the left in frame # n + 1.

In frame # n-1, the pixels belonging to the covered background area are the 11th to 13th pixels from the left.
Pixels belonging to the uncovered background area are the second to fourth pixels from the left. In frame #n, the pixels belonging to the covered background area are the 15th to 17th pixels from the left, and the pixels belonging to the uncovered background area are the 6th to 8th pixels from the left. flame
In # n + 1, the pixels belonging to the covered background area are the 19th to 21st pixels from the left, and the pixels belonging to the uncovered background area are the 10th to 12th pixels from the left.

In frame # n-1, the pixels belonging to the background area are the first pixel from the left and the 14th to 21st pixels from the left. In frame #n, the pixels belonging to the background area are the first to fifth pixels from the left and the eighteenth to 21st pixels from the left. In frame # n + 1, the pixels belonging to the background area are the first to ninth pixels from the left.

FIG. 55 generated by the background image generation unit 301.
56 shows an example of a background image corresponding to the example of FIG. The background image is composed of pixels corresponding to the background objects,
Does not include image components corresponding to foreground objects.

The binary object image extraction unit 302 generates a binary object image based on the correlation between the background image and the input image, and supplies the generated binary object image to the time change detection unit 303.

FIG. 57 shows the binary object image extraction unit 3
It is a block diagram which shows the structure of 02. Correlation value calculator 32
Reference numeral 1 calculates the correlation between the background image and the input image supplied from the background image generation unit 301, generates a correlation value, and supplies the generated correlation value to the threshold processing unit 322.

The correlation value calculation unit 321 is, for example, as shown in FIG.
As shown in (A), a block in a 3 × 3 background image centered on X ₄ and a center in Y ₄ corresponding to a block in the background image as shown in FIG. Done 3x3
(4) is applied to the block in the input image of
Calculate the correlation value corresponding to ₄ .

[0291]

[Equation 2]

[Equation 3]

[Equation 4]

The correlation value calculation unit 321 supplies the correlation value thus calculated for each pixel to the threshold value processing unit 322.

Further, the correlation value calculation unit 321 may, for example, as shown in FIG. 59 (A), a block in a 3 × 3 background image centered on X ₄ and as shown in FIG. 59 (B). To
3 centered on Y ₄ corresponding to the block in the background image
Equation (7) may be applied to the block in the input image of × 3 to calculate the absolute difference value corresponding to Y ₄ .

[0294]

[Equation 5]

The correlation value calculation unit 321 uses the difference absolute value thus calculated as the correlation value, and the threshold value processing unit 322.
Supply to.

The threshold value processing unit 322 compares the pixel value of the correlation image with the threshold value th0, and when the correlation value is less than or equal to the threshold value th0, sets 1 to the pixel value of the binary object image. If the correlation value is larger than the threshold value th0, the pixel value of the binary object image is set to 0, and the binary object image in which 0 or 1 is set to the pixel value is output. The threshold value processing unit 322 may store the threshold value th0 in advance, or the threshold value th0 input from the outside.
You may make it use 0.

FIG. 60 is a diagram showing an example of a binary object image corresponding to the model of the input image shown in FIG.
In the binary object image, a pixel value of 0 is set for a pixel having a high correlation with the background image.

FIG. 61 is a block diagram showing the structure of the time change detector 303. The frame memory 341 supplies the frame # n-supplied from the binary object image extraction unit 302 when determining the area for the pixel of the frame #n.
The binary object images of 1, frame #n, and frame # n + 1 are stored.

The area determination unit 342 is connected to the frame memory 34.
Frames based on the binary object images of frame # n-1, frame #n, and frame # n + 1 stored in 1
A region is determined for each pixel of #n, region information is generated, and the generated region information is output.

FIG. 62 is a diagram for explaining the judgment of the area judgment unit 342. When the pixel of interest of the binary object image of frame #n is 0, the area determination unit 342
It is determined that the pixel of interest in frame #n belongs to the background area.

The pixel of interest in the binary object image of frame #n is 1, the corresponding pixel of the binary object image of frame # n-1 is 1, and the pixel of 2 in frame # n + 1.
When the corresponding pixel of the value object image is 1, the area determination unit 342 determines that the pixel of interest of frame #n belongs to the foreground area.

When the pixel of interest in the binary object image of frame #n is 1 and the corresponding pixel of the binary object image of frame # n-1 is 0, the area determination unit 3
42 determines that the pixel of interest in frame #n belongs to the covered background area.

When the pixel of interest in the binary object image of frame #n is 1 and the corresponding pixel of the binary object image of frame # n + 1 is 0, the area determination unit 3
42 determines that the pixel of interest in frame #n belongs to the uncovered background area.

FIG. 63 is a diagram showing an example of judgment made by the time change detection unit 303 on the binary object image corresponding to the model of the input image shown in FIG. Since the pixel corresponding to frame #n of the binary object image is 0, the time change detection unit 303 is the first to fifth pixels from the left of frame #n.
The th pixel is determined to belong to the background area.

Since the pixel of frame #n of the binary object image is 1 and the corresponding pixel of frame # n + 1 is 0, the time change detection unit 303 uncovers the sixth to ninth pixels from the left. It is determined to belong to the background area.

The time change detection unit 303 determines that the pixel of frame #n of the binary object image is 1, the corresponding pixel of frame # n-1 is 1, and the corresponding pixel of frame # n + 1 is 1. Therefore, it is determined that the tenth to thirteenth pixels from the left belong to the foreground area.

Since the time change detection unit 303 has the pixel 1 of frame #n of the binary object image and the corresponding pixel 0 of frame # n-1, the 14th to 17th pixels from the left are covered back. It is determined that it belongs to the ground area.

Since the corresponding pixel of frame #n of the binary object image is 0, the time change detecting unit 303 determines that the 18th to 21st pixels from the left belong to the background area.

Next, the area specifying processing of the area determining unit 103 will be described with reference to the flowchart in FIG. In step S301, the background image generation unit 301 of the area determination unit 103 generates a background image by extracting, for example, an image object corresponding to a background object included in the input image based on the input image, and the generated background Image 2
It is supplied to the value object image extraction unit 302.

In step S302, the binary object image extraction unit 302 calculates the correlation value between the input image and the background image supplied from the background image generation unit 301, for example, by the calculation described with reference to FIG. . In step S303, the binary object image extraction unit 302
Calculates a binary object image from the correlation value and the threshold value th0 by, for example, comparing the correlation value and the threshold value th0.

In step S304, the time change detection unit 303 executes the area determination processing, and the processing ends.

Details of the area determination processing corresponding to step S304 will be described with reference to the flowchart in FIG. In step S321, the time change detection unit 303
Area determination unit 342 determines whether the pixel of interest is 0 in frame #n stored in frame memory 341, and the pixel of interest in frame #n is
If it is determined to be 0, the process proceeds to step S322,
It is set that the pixel of interest in frame #n belongs to the background area, and the process ends.

When it is determined in step S321 that the pixel of interest in frame #n is 1, the process proceeds to step S323, and the area determination unit 342 of the time change detection unit 303 is stored in the frame memory 341. In frame #n, the pixel of interest is 1, and
In frame # n-1, it is determined whether or not the corresponding pixel is 0. In frame #n, the pixel of interest is 1, and in frame # n-1, the corresponding pixel is 0. If it is determined that the pixel of interest in frame #n belongs to the covered background area, the process ends.

In step S323, the pixel of interest is 0 in frame #n, or frame #n
When it is determined that the corresponding pixel is 1 in n−1, the process proceeds to step S325, and the area determination unit 342 of the time change detection unit 303 focuses on the frame #n stored in the frame memory 341. One pixel,
Also, in frame # n + 1, it is determined whether or not the corresponding pixel is 0, and in frame #n, the pixel of interest is
If it is 1 and it is determined that the corresponding pixel is 0 in frame # n + 1, the process proceeds to step S326, and it is set that the pixel of interest in frame #n belongs to the uncovered background area. , The process ends.

[0315] In step S325, the pixel of interest is 0 in frame #n, or frame #n
When it is determined that the corresponding pixel is 1 in n + 1, the process proceeds to step S327, the area determination unit 342 of the time change detection unit 303 sets the pixel of interest in frame #n as the foreground area, and The process ends.

As described above, the area specifying unit 103 determines that the pixels of the input image are the foreground area, the background area, the covered background area, and the uncovered background based on the correlation value between the input image and the corresponding background image. It is possible to specify which of the ground areas it belongs to and generate area information corresponding to the specified result.

FIG. 66 is a block diagram showing another structure of the area specifying unit 103. Area specifying unit 103 shown in FIG. 66
Uses the motion vector and its position information supplied from the motion detection unit 102. The same parts as those shown in FIG. 54 are designated by the same reference numerals, and the description thereof will be omitted.

The robustization unit 361 outputs the 2 of the N frames supplied from the binary object image extraction unit 302.
A robust binary object image is generated based on the value object image, and is output to the time change detection unit 303.

FIG. 67 is a block diagram illustrating the structure of the robust unit 361. The motion compensating unit 381 compensates the motion of the binary object image of N frames based on the motion vector supplied from the motion detecting unit 102 and its position information, and outputs the motion-compensated binary object image. Output to the switch 382.

Motion compensation of the motion compensator 381 will be described with reference to the examples of FIGS. 68 and 69. For example, when determining the area of frame #n, if the binary object images of frame # n-1, frame #n, and frame # n + 1 shown in FIG. 68 are input, the motion compensation unit 381 ,
Based on the motion vector supplied from the motion detection unit 102,
As shown in the example in FIG. 69, the binary object image of frame # n−1 and the binary object image of frame # n + 1 are motion-compensated and the motion-compensated binary object image is supplied to the switch 382. To do.

The switch 382 stores the motion-compensated binary object image of the first frame in the frame memory 3
83-1 and outputs the motion-compensated binary object image of the second frame to the frame memory 383-2. Similarly, the switches 382 are the third to N-1th.
The motion-compensated binary object image of the th frame is output to one of the frame memories 383-3 to 383- (N-1), and the motion-compensated binary object image of the Nth frame is output. To the frame memory 383-N.

The frame memory 383-1 stores the motion-compensated binary object image of the first frame and outputs the stored binary object image to the weighting unit 384-1. The frame memory 383-2 is
The motion-compensated binary object image of the second frame is stored, and the stored binary object image is output to the weighting unit 384-2.

Similarly, each of the frame memories 383-3 to 383- (N-1) stores any one of the motion-compensated binary object images of the third frame to the (N-1) th frame. Then, the stored binary object image is output to one of the weighting units 384-3 to 384- (N-1).
The frame memory 383-N stores the motion-compensated binary object image of the Nth frame, and outputs the stored binary object image to the weighting unit 384-N.

The weighting unit 384-1 weights the pixel value of the motion-compensated binary object image of the first frame supplied from the frame memory 383-1 with a predetermined weight.
It is multiplied by w1 and supplied to the integrating unit 385. Weighting unit 38
4-2 is 2 supplied from the frame memory 383-2.
The pixel value of the motion-compensated binary object image of the th frame is multiplied by a predetermined weight w2 and supplied to the integration unit 385.

Similarly, each of the weighting units 384-3 to 384- (N-1) is the third to the third memory supplied from any of the frame memories 383-3 to 383- (N-1). The pixel value of the motion-compensated binary object image of any of the N-1th frames is multiplied by any one of the predetermined weights w3 to w (N-1), and the result is supplied to the integrating unit 385. Weighting unit 384-N
Multiplies the pixel value of the motion-compensated binary object image of the Nth frame supplied from the frame memory 383-N by a predetermined weight wN, and supplies the multiplied value to the accumulating unit 385.

The accumulating unit 385 accumulates the corresponding pixel values of the binary object image, which are motion-compensated for the 1st to Nth frames and multiplied by any of the weights w1 to wN, respectively, and accumulate the accumulated pixels. A binary object image is generated by comparing the value with a predetermined threshold th0.

As described above, the robust unit 361 generates a robust binary object image from the N binary object images and supplies it to the time change detecting unit 303. The configuration is shown in FIG. Even if the input image contains noise, the area specifying unit 103 can specify the area more accurately than in the case shown in FIG. 54.

Next, the area specifying unit 10 whose structure is shown in FIG.
The area specifying processing of No. 3 will be described with reference to the flowchart of FIG. Steps S341 to S
The processing of 343 is the same as that of steps S301 to S303 described in the flowchart of FIG. 64, and therefore description thereof will be omitted.

In step S344, the robust unit 361 executes robust processing.

In step S345, the time change detection unit 303 executes the area determination processing, and the processing ends. Details of the processing in step S345 are the same as the processing described with reference to the flowchart in FIG. 65, and therefore description thereof will be omitted.

Details of the robust process corresponding to the process of step S344 of FIG. 70 will be described below with reference to the flowchart of FIG. Step S36
In No. 1, the motion compensation unit 381 executes the motion compensation processing of the input binary object image based on the motion vector supplied from the motion detection unit 102 and its position information. In step S362, the frame memory 383
Any one of -1 to 383-N stores the motion-compensated binary object image supplied via the switch 382.

In step S363, the robust unit 361 determines whether or not N binary object images are stored, and if it is determined that N binary object images are not stored, step S361. Back to
The process of motion compensation of the binary object image and the process of storing the binary object image are repeated.

If it is determined in step S363 that N binary object images have been stored, the process proceeds to step S364, and the weighting units 384-1 to 384- are used.
Each N is weighted by multiplying each of the N binary object images by one of the weights w1 to wN.

In step S365, the integrating section 385
Integrates the N weighted binary object images.

[0335] In step S366, the accumulating unit 385.
Generates a binary object image from the integrated images by, for example, comparison with a predetermined threshold value th1, and the process ends.

As described above, the area specifying unit 103 shown in FIG. 66 can generate area information based on the robust binary object image.

As described above, the area specifying unit 103 provides the area information indicating that each of the pixels included in the frame belongs to the moving area, the still area, the uncovered background area, or the covered background area. Can be generated.

FIG. 72 is a block diagram showing an example of the configuration of the mixture ratio calculating unit 104. Estimated mixing ratio processing unit 401
Calculates the estimated mixture ratio for each pixel by calculation corresponding to the model of the covered background area based on the input image, and supplies the calculated estimated mixture ratio to the mixture ratio determination unit 403.

The estimated mixture ratio processing unit 402 calculates the estimated mixture ratio for each pixel by the calculation corresponding to the model of the uncovered background area based on the input image, and determines the calculated estimated mixture ratio. It is supplied to the unit 403.

Since it can be assumed that the object corresponding to the foreground is moving at a constant speed within the shutter time, the mixture ratio α of the pixels belonging to the mixed area has the following properties. That is, the mixture ratio α changes linearly according to the change in the position of the pixel. If the change of the pixel position is one-dimensional, the change of the mixture ratio α can be expressed by a straight line, and if the change of the pixel position is two-dimensional, the change of the mixture ratio α is expressed by a plane. be able to.

Since the period of one frame is short, it is assumed that the object corresponding to the foreground is a rigid body and is moving at a constant speed.

In this case, the gradient of the mixture ratio α is the inverse ratio of the movement amount v within the shutter time of the foreground.

FIG. 73 shows an example of the ideal mixing ratio α. The slope l in the mixing region of the ideal mixing ratio α can be expressed as the reciprocal of the motion amount v.

As shown in FIG. 73, the ideal mixture ratio α
Has a value of 1 in the background area, a value of 0 in the foreground area, and a value greater than 0 and less than 1 in the mixed area.

In the example of FIG. 74, the pixel value C06 of the seventh pixel from the left of frame #n is 7 from the left of frame # n-1.
It can be expressed by Expression (8) using the pixel value P06 of the th pixel.

[0346]

[Equation 6]

In equation (8), the pixel value C06 is expressed as the pixel value M of the pixel in the mixed area, and the pixel value P06 is expressed as the pixel value B of the pixel in the background area. That is, the pixel value M of the pixel in the mixed area and the pixel value B of the pixel in the background area are respectively expressed by equation (9).
And can be expressed as in Expression (10).

[0348] M = C06 (9) B = P06 (10)

2 / v in the equation (8) corresponds to the mixing ratio α. Since the motion amount v is 4, the mixture ratio α of the seventh pixel from the left of frame #n is 0.5.

As described above, the pixel value C of the focused frame #n is regarded as the pixel value of the mixed area, and the pixel value P of the frame # n−1 before the frame #n is set as the pixel value of the background area. By observing it, the equation (3) indicating the mixture ratio α can be rewritten as the equation (11).

C = α · P + f (11) f in Expression (11) is the sum Σ _i Fi / v of the foreground components included in the pixel of interest. The variables included in the equation (11) are the mixture ratio α and the sum f of the foreground components.

Similarly, FIG. 75 shows a model in which the pixel value is expanded in the time direction in which the amount of motion v is 4 and the number of virtual divisions in the time direction is 4 in the uncovered background area.

In the uncovered background area, similarly to the above-described expression in the covered background area, the pixel value C of the frame #n of interest is regarded as the pixel value of the mixed area. n + 1
By regarding the pixel value N of the above as the pixel value of the background region, the equation (3) indicating the mixture ratio α can be expressed as the equation (12).

[0354] C = α ・ N + f (12)

Although it has been described that the background object is stationary, even when the background object is moving, by using the pixel value of the pixel at the position corresponding to the background motion amount v, Expressions (8) to (1
2) can be applied. For example, in FIG. 74, when the motion amount v of the object corresponding to the background is 2 and the virtual division number is 2, when the object corresponding to the background is moving to the right side in the figure, The pixel value B of the pixel in the background area is set to the pixel value P04.

Since the equations (11) and (12) each include two variables, the mixture ratio α cannot be obtained as it is. Here, since an image generally has a strong spatial correlation, adjacent pixels have substantially the same pixel value.

Since the foreground components have a strong spatial correlation, the formula is modified so that the sum f of the foreground components can be derived from the previous or subsequent frame, and the mixture ratio α is obtained.

The pixel value Mc of the seventh pixel from the left of frame #n in FIG. 76 can be expressed by equation (13).

[0359]

[Equation 7] 2 / v of the first term on the right side of the equation (13) corresponds to the mixing ratio α. The second term on the right side of Expression (13) is expressed as Expression (14) using the pixel value of the subsequent frame # n + 1.

[0360]

[Equation 8]

Here, it is assumed that the expression (15) is established by utilizing the spatial correlation of the foreground components.

F = F05 = F06 = F07 = F08 = F09 = F10 = F11 = F12 (15) The formula (14) can be replaced as the formula (16) by using the formula (15).

[0363]

[Equation 9] As a result, β can be expressed by equation (17).

[0364] β = 2/4 (17)

Generally, assuming that the foreground components related to the mixed area are equal to each other, as shown in the expression (15), the expression (18) is calculated from the internal division ratio for all pixels in the mixed area.
Is established.

[0366] β = 1-α (18)

If the equation (18) holds, the equation (1
1) can be expanded as shown in equation (19).

[0368]

[Equation 10]

Similarly, if equation (18) holds, then
Equation (12) can be expanded as shown in equation (20).

[0370]

[Equation 11]

In equations (19) and (20),
Since C, N, and P are known pixel values, the variable included in the equations (19) and (20) is only the mixture ratio α. 77 shows the relationship between C, N, and P in equations (19) and (20). C calculates the mixing ratio α,
This is the pixel value of the pixel of interest in frame #n. N is
This is the pixel value of the pixel of frame # n + 1, which corresponds to the position of the pixel of interest in the spatial direction. P is the pixel value of the pixel of frame # n-1 whose position in the spatial direction corresponds to the pixel of interest.

Therefore, since one variable is included in each of the equations (19) and (20), the mixture ratio α can be calculated using the pixel values of the pixels of the three frames. . The conditions for calculating the correct mixture ratio α by solving the equations (19) and (20) are:
Pixels located at the boundary of the foreground object corresponding to the direction of movement of the foreground object in the foreground image object captured when the foreground components related to the mixed area are equal, that is, when the foreground object is stationary. That is, the pixel value of the number of consecutive pixels that is twice the motion amount v is constant.

As described above, the mixture ratio α of pixels belonging to the covered background area is calculated by the equation (21), and the mixture ratio α of pixels belonging to the uncovered background area is calculated by the equation (22). It

[0374]       α = (C-N) / (P-N) (21)       α = (C-P) / (N-P) (22)

FIG. 78 is a block diagram showing the structure of the estimated mixing ratio processing unit 401. The frame memory 421 stores the input image on a frame-by-frame basis, and supplies the frame immediately after the frame input as the input image to the frame memory 422 and the mixing ratio calculation unit 423.

The frame memory 422 stores the input image on a frame-by-frame basis, and supplies the frame one frame after the frame supplied from the frame memory 421 to the mixture ratio calculation unit 423.

Therefore, when the frame # n + 1 is input as the input image to the mixture ratio calculation unit 423, the frame memory 421 supplies the frame #n to the mixture ratio calculation unit 423, and the frame memory 422 displays the frame #n. # n-1 is supplied to the mixing ratio calculation unit 423.

The mixing ratio calculation unit 423 calculates the pixel value of the pixel of interest in frame #n by the calculation shown in equation (21).
C, the pixel value N of the pixel in frame # n + 1 corresponding to the pixel of interest and the spatial position, and the pixel value of the pixel in frame # n-1 corresponding to the pixel of interest in the spatial position The estimated mixture ratio of the pixel of interest is calculated based on the value P, and the calculated estimated mixture ratio is output. For example, when the background is stationary, the mixing ratio calculation unit 423 determines that the pixel value C of the pixel of interest in frame #n, the pixel of interest and the position in the frame are the same, The pixel value N of the pixel, and the position of the pixel of interest in the frame is the same as frame #n
The estimated mixture ratio of the pixel of interest is calculated based on the pixel value P of the pixel of -1, and the calculated estimated mixture ratio is output.

In this way, the estimated mixing ratio processing unit 401
The estimated mixture ratio can be calculated based on the input image and supplied to the mixture ratio determination unit 403.

The estimated mixture ratio processing unit 402 calculates the estimated mixture ratio of the pixel of interest by the calculation of the equation (21), whereas the estimated mixture ratio processor 401 calculates the estimated mixture ratio by the equation (22). The calculation is similar to the estimated mixture ratio processing unit 401 except that the portion for calculating the estimated mixture ratio of the pixel of interest is different, and thus the description thereof will be omitted.

FIG. 79 is a diagram showing an example of the estimated mixture ratio calculated by the estimated mixture ratio processing unit 401. The estimated mixture ratio shown in FIG. 79 shows the result for one line when the motion amount v of the foreground corresponding to the object moving at a constant speed is 11.

The estimated mixing ratio is shown in FIG.
As shown in, it can be seen that the change is almost linear.

Returning to FIG. 72, the mixture ratio determining unit 403 determines that the pixels supplied from the region specifying unit 103 for which the mixture ratio α is to be calculated are the foreground region, the background region, the covered background region, or the uncovered region. The mixture ratio α is set based on the area information indicating which of the background areas it belongs to. The mixture ratio determination unit 403 sets 0 to the mixture ratio α when the target pixel belongs to the foreground region, and sets 1 to the mixture ratio α when the target pixel belongs to the background region and becomes the target. When the pixel belongs to the covered background area, the estimated mixture ratio supplied from the estimated mixture ratio processing section 401 is set to the mixture ratio α, and when the target pixel belongs to the uncovered background area, the estimated mixture ratio processing section is set. The estimated mixing ratio supplied from 402 is set to the mixing ratio α. The mixing ratio determination unit 403 outputs the mixing ratio α set based on the area information.

FIG. 80 is a block diagram showing another configuration of the mixture ratio calculation unit 104. Based on the area information supplied from the area specifying unit 103, the selecting unit 441 supplies the pixels belonging to the covered background area and the pixels of the corresponding previous and subsequent frames to the estimated mixture ratio processing unit 442. The selection unit 441 supplies the pixels belonging to the uncovered background area and the pixels of the corresponding previous and subsequent frames to the estimated mixture ratio processing unit 443 based on the area information supplied from the area specifying unit 103. .

The estimated mixing ratio processing section 442 has a selection section 441.
The estimated mixture ratio of the pixel of interest belonging to the covered background area is calculated by the calculation shown in Expression (21) on the basis of the pixel value input from the selection unit 444. Supply.

The estimated mixing ratio processing unit 443 is the selection unit 441.
The estimated mixture ratio of the pixel of interest belonging to the uncovered background area is calculated by the calculation shown in Expression (22) based on the pixel value input from the selection unit 444. Supply to.

Based on the area information supplied from the area specifying unit 103, the selecting unit 444 selects the estimated mixing ratio of 0 when the pixel of interest belongs to the foreground region, and mixes it with the mixing ratio α.
If the target pixel belongs to the background area, 1
The estimated mixture ratio is selected and set to the mixture ratio α. When the target pixel belongs to the covered background area, the selecting unit 444 selects the estimated mixing ratio supplied from the estimated mixing ratio processing unit 442 and sets it as the mixing ratio α, and the target pixel is uncovered background. If it belongs to the ground area, the estimated mixture ratio supplied from the estimated mixture ratio processing unit 443 is selected and set to the mixture ratio α. The selection unit 444 is
The mixture ratio α selected and set based on the region information is output.

As described above, the mixing ratio calculating unit 104 having another configuration shown in FIG. 80 can calculate the mixing ratio α for each pixel included in the image and output the calculated mixing ratio α.

Referring to the flowchart of FIG. 81, FIG.
The process of calculating the mixture ratio α by the mixture ratio calculation unit 104 having the configuration shown in 2 will be described. In step S401, the mixture ratio calculating unit 104 acquires the area information supplied from the area specifying unit 103. In step S402, the estimated mixing ratio processing unit 401 executes the process of calculating the estimated mixing ratio using the model corresponding to the covered background area, and supplies the calculated estimated mixing ratio to the mixing ratio determination unit 403. Details of the processing for calculating the mixing ratio will be described later with reference to the flowchart in FIG.

In step S403, the estimated mixing ratio processing unit 402 executes the processing of the estimated mixing ratio using the model corresponding to the uncovered background area,
The calculated estimated mixture ratio is supplied to the mixture ratio determination unit 403.

In step S404, the mixture ratio calculation unit 104 determines whether or not the mixture ratio α has been estimated for the entire frame. If it is determined that the mixture ratio α has not been estimated for the entire frame, step S404 Returning to S402, the process of estimating the mixture ratio α for the next pixel is executed.

If it is determined in step S404 that the mixture ratio α has been estimated for the entire frame, the flow advances to step S405, and the mixture ratio determination unit 403 determines that the pixel is
Foreground area, background area, covered background area,
Alternatively, the mixture ratio α is set based on the area information supplied from the area specifying unit 103, which indicates which of the uncovered background areas it belongs to. Mixing ratio determination unit 403
Sets 0 to the mixture ratio α when the target pixel belongs to the foreground region, and sets 1 to the mixture ratio α when the target pixel belongs to the background region and sets the target pixel to the covered background. If it belongs to the region, the estimated mixture ratio supplied from the estimated mixture ratio processing unit 401 is set to the mixture ratio α,
When the target pixel belongs to the uncovered background area, the estimated mixture ratio supplied from the estimated mixture ratio processing unit 402 is set to the mixture ratio α, and the process ends.

As described above, the mixture ratio calculating unit 104 can calculate the mixture ratio α, which is the feature amount corresponding to each pixel, based on the area information supplied from the area specifying unit 103 and the input image. .

Since the process of calculating the mixture ratio α of the mixture ratio calculating unit 104 having the configuration shown in FIG. 80 is the same as the process described in the flowchart of FIG. 81, the description thereof will be omitted.

Next, the processing of the mixture ratio estimation by the model corresponding to the covered background area corresponding to step S402 of FIG. 81 will be described with reference to the flowchart of FIG.

In step S421, the mixture ratio calculation unit 423 acquires the pixel value C of the target pixel of frame #n from the frame memory 421.

In step S422, the mixture ratio calculation unit 423 acquires the pixel value P of the pixel of frame # n-1 corresponding to the pixel of interest from the frame memory 422.

[0398] In step S423, the mixture ratio calculation unit 423 acquires the pixel value N of the pixel of frame # n + 1 corresponding to the pixel of interest included in the input image.

[0399] In step S424, the mixture ratio calculation unit 423 determines the pixel value C of the pixel of interest in frame #n and the frame #
The estimated mixing ratio is calculated based on the pixel value P of the pixel of n−1 and the pixel value N of the pixel of frame # n + 1.

In step S425, the mixture ratio calculation unit 423 determines whether or not the process of calculating the estimated mixture ratio has been completed for the entire frame, and the process of calculating the estimated mixture ratio has been completed for the entire frame. If it is determined that there is not, the process returns to step S421, and the process of calculating the estimated mixture ratio for the next pixel is repeated.

If it is determined in step S425 that the process of calculating the estimated mixing ratio has been completed for the entire frame, the process ends.

As described above, the estimated mixing ratio processing unit 401
The estimated mixture ratio can be calculated based on the input image.

The processing of the mixture ratio estimation by the model corresponding to the uncovered background area in step S403 of FIG. 81 is the same as the processing shown in the flowchart of FIG. 82 using the equation corresponding to the model of the uncovered background area. Therefore, its explanation is omitted.

The estimated mixture ratio processing unit 44 shown in FIG.
Since the 2 and the estimated mixing ratio processing unit 443 execute the same processing as the flowchart shown in FIG. 82 to calculate the estimated mixing ratio, the description thereof will be omitted.

Although the object corresponding to the background has been described as stationary, the above-described processing for obtaining the mixture ratio α can be applied even if the image corresponding to the background area includes motion. For example, when the image corresponding to the background area is moving uniformly, the estimated mixture ratio processing unit 401
The entire image is shifted according to the movement of the background, and the same processing is performed as when the object corresponding to the background is stationary. Further, when the image corresponding to the background region includes the movement of the background that differs locally, the estimated mixture ratio processing unit 401
Selects a pixel corresponding to the motion of the background as a pixel corresponding to a pixel belonging to the mixed area, and executes the above-mentioned processing.

Further, the mixture ratio calculating unit 104 executes only the process of estimating the mixture ratio by the model corresponding to the covered background area for all the pixels, and outputs the calculated estimated mixture ratio as the mixture ratio α. You may do it. In this case, the mixture ratio α indicates the ratio of the background component for the pixels belonging to the covered background area, and indicates the ratio of the foreground component for the pixels belonging to the uncovered background area. For the pixels belonging to the uncovered background area, the absolute value of the difference between the thus calculated mixture ratio α and 1 is calculated,
If the calculated absolute value is set as the mixture ratio α, the separation processing server 11 can obtain the mixture ratio α indicating the ratio of the background components for the pixels belonging to the uncovered background area.

Similarly, the mixture ratio calculating unit 104 executes only the mixture ratio estimation processing by the model corresponding to the uncovered background region for all the pixels, and calculates the estimated mixture ratio as the mixture ratio. It may be output as α.

Next, the mixing ratio calculating unit 104 for calculating the mixing ratio α by utilizing the property that the mixing ratio α changes linearly will be described.

As described above, the equations (11) and (1
Since 2) includes two variables, the mixture ratio α cannot be obtained as it is.

Therefore, by utilizing the property that the mixture ratio α changes linearly in response to a change in pixel position caused by the object corresponding to the foreground moving at a constant speed within the shutter time, the spatial direction Is the sum of the mixture ratio α and the foreground components
Establish an equation that approximates f and. Using a plurality of sets of pixel values of pixels belonging to the mixed area and pixels belonging to the background area, an equation approximating the mixture ratio α and the sum f of the foreground components is solved.

When the change in the mixture ratio α is approximated as a straight line, the mixture ratio α is expressed by the equation (23).

Α = il + p (23) In equation (23), i is a spatial index in which the position of the pixel of interest is 0. l is the slope of the straight line of the mixture ratio α. p is a straight line intercept of the mixture ratio α and is the mixture ratio α of the pixel of interest. Formula (2
In 3), the index i is known, but the slope l
And the intercept p is unknown.

FIG. 83 shows the relationship between the index i, the slope l, and the intercept p.

By approximating the mixture ratio α as shown in equation (23), a plurality of different mixture ratios α for a plurality of pixels can be obtained.
Is represented by two variables. In the example shown in FIG. 83, five mixing ratios for five pixels are represented by two variables, a slope l and an intercept p.

When the mixture ratio α is approximated by the plane shown in FIG. 84, when the motion v corresponding to two directions of the image, that is, the horizontal direction and the vertical direction, is taken into consideration, the equation (23) is expanded to the plane to obtain the mixture ratio α. α is represented by equation (24).

Α = jm + kq + p (24) In the equation (24), j is a horizontal index in which the position of the pixel of interest is 0, and k is a vertical index. m is the inclination of the surface of the mixture ratio α in the horizontal direction, and q is the inclination of the surface of the mixture ratio α in the vertical direction. p is the intercept of the plane of the mixing ratio α.

For example, in frame #n shown in FIG. 74, equations (25) to (27) hold for C05 to C07, respectively.

[0418] C05 = α05 ・ B05 / v + f05 (25) C06 = α06 ・ B06 / v + f06 (26) C07 = α07 ・ B07 / v + f07 (27)

The foreground components match in the neighborhood, ie,
If F01 to F03 are equal, F01 to F03 are replaced with Fc, and then Expression (28) is established.

[0420] f (x) = (1-α (x)) ・ Fc (28) In Expression (28), x represents a position in the spatial direction.

Replacing α (x) with equation (24), equation (28) can be expressed as equation (29).

[0422] f (x) = (1- (jm + kq + p)) ・ Fc = j ・ (-m ・ Fc) + k ・ (-q ・ Fc) + ((1-p) ・ Fc) = js + kt + u (29)

In equation (29), (-m · Fc), (-q ·
Fc) and (1-p) · Fc are expressed by equations (30) to (32)
Has been replaced as shown.

[0424] s = -m ・ Fc (30) t = -q ・ Fc (31) u = (1-p) ・ Fc (32)

In equation (29), j is a horizontal index with the position of the pixel of interest as 0,
k is a vertical index.

As described above, since it is assumed that the object corresponding to the foreground moves at a constant speed within the shutter time and the component corresponding to the foreground is constant in the vicinity, the sum of the components of the foreground is expressed by the formula ( 29).

When the mixture ratio α is approximated by a straight line, the sum of the foreground components can be expressed by equation (33).

[0428] f (x) = is + u (33)

When the mixture ratio α and the sum of the foreground components in equation (13) are replaced by utilizing equations (24) and (29), the pixel value M is represented by equation (34).

[0430] M = (jm + kq + p) ・ B + js + kt + u = jB ・ m + kB ・ q + B ・ p + j ・ s + k ・ t + u (34)

In the equation (34), unknown variables are the inclination m of the surface of the mixture ratio α in the horizontal direction, the inclination q of the surface of the mixture ratio α in the vertical direction, and the intercepts p, s, t of the surface of the mixture ratio α. , And u.

A plurality of pixels in which the pixel value M or the pixel value B is set and the pixel value M or the pixel value B is set in the normal equation shown in the equation (34) corresponding to the pixel in the vicinity of the pixel of interest. The normal equation of is solved by the method of least squares to calculate the mixture ratio α.

For example, the horizontal direction index j of the pixel of interest is set to 0, the vertical index k is set to 0, and 3 × 3 pixels near the target pixel are
When the pixel value M or the pixel value B is set in the normal equation shown in Expression (34), Expressions (35) to (43) are obtained.

M _{-1, -1} = ( _-1 ) ・ B _{-1, -1}・ m + ( _-1 ) ・ B _{-1, -1}・ q + B _{-1, -1}・ p + ( _-1 ) ・s + (-1) ・ t + u (35) M _{0, -1} = (0) ・ B _{0, -1}・ m + ( _-1 ) ・ B _{0, -1}・ q + B _{0, -1}・ p + ( 0) ・ s + (-1) ・ t + u (36) M _{+ 1, -1} = (+ 1) ・ B _{+ 1, -1}・ m + ( _-1 ) ・ B _{+ 1, -1}・ q + B _{+ 1, -1}・ p + (+ 1) ・ s + (-1) ・ t + u (37) M _-1,0 = ( _-1 ) ・ B _-1,0・ m + (0) ・ B _{-1, 0}・ q + B _-1,0・ p + ( _-1 ) ・ s + (0) ・ t + u (38) M _0,0 = (0) ・ B _0,0・ m + (0) ・ B _0,0・ Q + B _0,0・ p + (0) ・ s + (0) ・ t + u (39) M _+1,0 = (+ 1) ・ B _+1,0・ m + (0) ・ B _{+1, 0}・ q + B _+1,0・ p + (+ 1) ・ s + (0) ・ t + u (40) M _{-1, + 1} = ( _-1 ) ・ B _{-1, + 1}・ m + (+ 1 ) ・ B _{-1, + 1}・ q + B _{-1, + 1}・ p + ( _-1 ) ・ s + (+ 1) ・ t + u (41) M _{0, + 1} = (0) ・ B _{0, + 1}・ m + (+ 1) ・ B _{0, + 1}・ q + B _{0, + 1}・ p + (0) ・ s + (+ 1) ・ t + u (42) M _{+ 1, + 1} = (+ 1)・ B _{+ 1, + 1}・ m + (+ 1) ・ B _{+ 1, + 1}・ q + B _{+ 1, + 1}・ p + (+ 1) ・ s + (+ 1) ・ t + u (43)

Since the horizontal index j of the pixel of interest is 0 and the vertical index k of it is 0, the mixture ratio α of the pixel of interest is calculated from equation (24) as follows:
It is equal to the value at j = 0 and k = 0, ie the intercept p.

Therefore, based on the nine equations (35) to (43), the horizontal gradient m, the vertical gradient q, and the intercepts p, s, t, and u are calculated by the method of least squares. The value of is calculated and the intercept p is output as the mixture ratio α.

Next, a more specific procedure for calculating the mixture ratio α by applying the least square method will be described.

When the index i and the index k are expressed by one index x, the relationship between the index i, the index k, and the index x is expressed by the equation (44).

[0439] x = (j + 1) / 3 + (k + 1) (44)

Horizontal tilt m, vertical tilt q, intercept
p, s, t, and u are expressed as variables w0, w1, w2, w3, w4, and W5, respectively, and jB, kB, B, j, k, and 1 are expressed as a0, a1,
Expressed as a2, a3, a4, and a5. Considering the error ex,
Equations (35) to (43) can be rewritten as equation (45).

[0441]

[Equation 12] In Expression (45), x is any value of integers from 0 to 8.

From equation (45), equation (46) can be derived.

[0443]

[Equation 13]

Here, in order to apply the least squares method, the sum of squares E of the errors is defined as shown in equation (47).

[0445]

[Equation 14]

In order to minimize the error, the sum of squares of the error is
It suffices if the partial differential of the variable Wv with respect to E becomes zero. Here, v is any value of an integer of 0 to 5. Therefore, wy is calculated so as to satisfy the equation (48).

[0447]

[Equation 15]

By substituting the equation (46) into the equation (48), the equation (49) is obtained.

[0449]

[Equation 16]

By applying, for example, the sweeping method (Gauss-Jordan elimination method) to the six expressions obtained by substituting any one of the integers 0 to 5 for v in the expression (49), wy To calculate. As described above, w0 is the horizontal inclination m, w1 is the vertical inclination q, w2 is the intercept p, and w3
Is s, w4 is t, and w5 is u.

As described above, by applying the least squares method to the equations in which the pixel value M and the pixel value B are set, the horizontal inclination m, the vertical inclination q, the intercepts p, s, t, and You can ask u.

In the description corresponding to the equations (35) to (43), the pixel value of the pixel included in the mixed area is M,
Although the pixel value of the pixel included in the background area is described as B, the normal equation is calculated for each of the case where the pixel of interest is included in the covered background area or the uncovered background area. Need to stand up.

For example, when obtaining the mixture ratio α of the pixels included in the covered background area of the frame #n shown in FIG. 74, the pixels C04 to C08 of the frame #n and the pixels of the pixel of the frame # n-1 are obtained. The values P04 to P08 are set in the normal equation.

When the mixture ratio α of the pixels included in the uncovered background area of frame #n shown in FIG. 75 is obtained, the pixel values of pixels C28 to C32 of frame #n and the pixels of frame # n + 1 are calculated. N28 to N32 are set to the normal equation.

Further, for example, when calculating the mixture ratio α of the pixels included in the covered background area shown in FIG. 85, the following expressions (50) to (58) are established. The pixel value of the pixel for calculating the mixture ratio α is Mc5.

[0456] Mc1 = (-1) ・ Bc1 ・ m + (-1) ・ Bc1 ・ q + Bc1 ・ p + (-1) ・ s + (-1) ・ t + u (50) Mc2 = (0) ・ Bc2 ・ m + (-1) ・ Bc2 ・ q + Bc2 ・ p + (0) ・ s + (-1) ・ t + u (51) Mc3 = (+ 1) ・ Bc3 ・ m + (-1) ・ Bc3 ・ q + Bc3 ・ p + (+ 1) ・ s + (-1) ・ t + u (52) Mc4 = (-1) ・ Bc4 ・ m + (0) ・ Bc4 ・ q + Bc4 ・ p + (-1) ・ s + (0) ・ t + u (53) Mc5 = (0) ・ Bc5 ・ m + (0) ・ Bc5 ・ q + Bc5 ・ p + (0) ・ s + (0) ・ t + u (54) Mc6 = (+ 1) ・ Bc6 ・ m + (0) ・ Bc6 ・ q + Bc6 ・ p + (+ 1) ・ s + (0) ・ t + u (55) Mc7 = (-1) ・ Bc7 ・ m + (+ 1) ・ Bc7 ・ q + Bc7 ・ p + (-1) ・ s + (+ 1) ・ t + u (56) Mc8 = (0) ・ Bc8 ・ m + (+ 1) ・ Bc8 ・ q + Bc8 ・ p + (0) ・ s + (+ 1) ・ t + u (57) Mc9 = (+ 1) ・ Bc9 ・ m + (+ 1) ・ Bc9 ・ q + Bc9 ・ p + (+ 1) ・ s + (+ 1) ・ t + u (58)

When calculating the mixture ratio α of the pixels included in the covered background area of frame #n, the expression (5
0) to Expression (58), the pixel value Bc of the pixel in the background area of the pixel of frame # n-1 corresponding to the pixel of frame #n
1 to Bc9 are used.

When calculating the mixture ratio α of the pixels included in the uncovered background area shown in FIG. 85, the following equations (59) to (67) are established. Mixing ratio α
The pixel value of the pixel for calculating is Mu5.

[0459] Mu1 = (-1) ・ Bu1 ・ m + (-1) ・ Bu1 ・ q + Bu1 ・ p + (-1) ・ s + (-1) ・ t + u (59) Mu2 = (0) ・ Bu2 ・ m + (-1) ・ Bu2 ・ q + Bu2 ・ p + (0) ・ s + (-1) ・ t + u (60) Mu3 = (+ 1) ・ Bu3 ・ m + (-1) ・ Bu3 ・ q + Bu3 ・ p + (+ 1) ・ s + (-1) ・ t + u (61) Mu4 = (-1) ・ Bu4 ・ m + (0) ・ Bu4 ・ q + Bu4 ・ p + (-1) ・ s + (0) ・ t + u (62) Mu5 = (0) ・ Bu5 ・ m + (0) ・ Bu5 ・ q + Bu5 ・ p + (0) ・ s + (0) ・ t + u (63) Mu6 = (+ 1) ・ Bu6 ・ m + (0) ・ Bu6 ・ q + Bu6 ・ p + (+ 1) ・ s + (0) ・ t + u (64) Mu7 = (-1) ・ Bu7 ・ m + (+ 1) ・ Bu7 ・ q + Bu7 ・ p + (-1) ・ s + (+ 1) ・ t + u (65) Mu8 = (0) ・ Bu8 ・ m + (+ 1) ・ Bu8 ・ q + Bu8 ・ p + (0) ・ s + (+ 1) ・ t + u (66) Mu9 = (+ 1) ・ Bu9 ・ m + (+ 1) ・ Bu9 ・ q + Bu9 ・ p + (+ 1) ・ s + (+ 1) ・ t + u (67)

When calculating the mixture ratio α of the pixels included in the uncovered background area of the frame #n, in the formulas (59) to (67), the frame # n + 1 corresponding to the pixel of the frame #n is calculated. The pixel values Bu1 to Bu9 of the pixels in the background area of the pixel are used.

FIG. 86 is a block diagram showing the structure of the estimated mixing ratio processing unit 401. The image input to the estimated mixing ratio processing unit 401 includes the delay unit 501 and the addition unit 502.
Is supplied to.

The delay circuit 221 delays the input image by one frame and supplies it to the adding section 502. Foot part 5
When the frame #n is input as the input image to 02, the delay circuit 221 adds the frame # n-1 to the adder 50.
Supply to 2.

The adder 502 sets the pixel values of the pixels near the pixel for which the mixture ratio α is calculated and the pixel value of frame # n-1 in the normal equation. For example, the adding part 50
2 sets pixel values Mc1 to Mc9 and pixel values Bc1 to Bc9 in the normal equation based on the equations (50) to (58). The addition unit 502 supplies the normal equation in which the pixel value is set to the calculation unit 503.

The computing unit 503 solves the normal equation supplied from the adding unit 502 by a sweep method or the like to obtain an estimated mixture ratio, and outputs the obtained estimated mixture ratio.

In this way, the estimated mixing ratio processing unit 401
The estimated mixture ratio can be calculated based on the input image and supplied to the mixture ratio determination unit 403.

Since the estimated mixing ratio processing unit 402 has the same structure as the estimated mixing ratio processing unit 401, its description is omitted.

FIG. 87 is a diagram showing an example of the estimated mixture ratio calculated by the estimated mixture ratio processing unit 401. The estimated mixture ratio shown in FIG. 87 has a foreground movement v corresponding to an object moving at a constant velocity of 11, and the result calculated by generating an equation in units of 7 × 7 pixel blocks is 1 line. Is shown for.

The estimated mixing ratio is shown in FIG.
As shown in, it can be seen that the change is almost linear.

Next, the process of estimating the mixture ratio by the model corresponding to the covered background area by the estimated mixture ratio processing unit 401 having the configuration shown in FIG. 86 will be described with reference to the flowchart of FIG. 88.

In step S521, the adding portion 5
02 sets the pixel value included in the input image and the pixel value included in the image supplied from the delay circuit 221 to the normal equation corresponding to the model of the covered background area.

[0471] In step S522, the estimated mixture ratio processing unit 401 determines whether or not the setting for the target pixel is completed, and if it is determined that the setting for the target pixel is not completed, Returning to step S521, the process of setting the pixel value in the normal equation is repeated.

If it is determined in step S522 that the pixel value setting for the target pixel is completed, the operation proceeds to step S523, and the arithmetic unit 173 calculates the estimated mixture based on the normal equation in which the pixel value is set. The ratio is calculated and the obtained estimated mixing ratio is output.

As described above, the estimated mixture ratio processing unit 401 having the configuration shown in FIG. 86 can calculate the estimated mixture ratio based on the input image.

The processing of the mixture ratio estimation by the model corresponding to the uncovered background area is the same as the processing shown in the flowchart of FIG. 88 using the normal equation corresponding to the model of the uncovered background area, and therefore the description thereof will be omitted. Omit it.

Although it has been described that the object corresponding to the background is stationary, the above-described processing for obtaining the mixture ratio can be applied even if the image corresponding to the background area includes motion. For example, when the image corresponding to the background area is moving uniformly, the estimated mixing ratio processing unit 401 shifts the entire image in response to this movement, similar to the case where the object corresponding to the background is stationary. To process.
In addition, when the image corresponding to the background area includes different motions locally, the estimated mixing ratio processing unit 401 selects the pixel corresponding to the motion as the pixel corresponding to the pixel belonging to the mixed area, and The process of is executed.

As described above, the mixture ratio calculating unit 102 can calculate the mixture ratio α, which is the feature amount corresponding to each pixel, based on the area information supplied from the area specifying unit 101 and the input image. .

By using the mixture ratio α, it is possible to separate the foreground component and the background component included in the pixel value while leaving the information of the motion blur included in the image corresponding to the moving object. It will be possible.

Also, by synthesizing images based on the mixture ratio α, it becomes possible to create an image including correct motion blur that matches the speed of a moving object as if the real world was actually shot again. .

Next, the foreground / background separator 105 will be described. FIG. 89 is a block diagram showing an example of the configuration of the foreground / background separator 105. The input image supplied to the foreground / background separation unit 105 is supplied to the separation unit 601, the switch 602, and the switch 604. The information indicating the covered background area and the area information supplied from the area specifying unit 103 indicating the uncovered background area are supplied to the separating unit 601. Area information indicating the foreground area is supplied to the switch 602. Area information indicating the background area is supplied to the switch 604.

The mixing ratio α supplied from the mixing ratio calculating section 104 is supplied to the separating section 601.

The separating unit 601 separates the foreground components from the input image on the basis of the area information indicating the covered background area, the area information indicating the uncovered background area, and the mixture ratio α. The components are supplied to the combining unit 603, the background components are separated from the input image, and the separated background components are supplied to the combining unit 605.

The switch 602 is closed when a pixel corresponding to the foreground is input based on the area information indicating the foreground area, and supplies only the pixel corresponding to the foreground included in the input image to the combining unit 603.

The switch 604 is closed when a pixel corresponding to the background is input based on the area information indicating the background area, and supplies only the pixel corresponding to the background included in the input image to the synthesizing unit 605.

The synthesizer 603 synthesizes the foreground component image based on the foreground component supplied from the separator 601 and the pixel corresponding to the foreground supplied from the switch 602.
The combined foreground component image is output. Since the foreground area and the mixed area do not overlap, the combining unit 603 applies a logical sum operation to the component corresponding to the foreground and the pixel corresponding to the foreground to combine the foreground component images.

The compositing unit 603 stores an image in which all the pixel values are 0 in the built-in frame memory in the initialization process which is executed first in the process of composing the foreground component images. In the synthesizing process of, the foreground component image is stored (overwritten). Therefore, in the foreground component image output by the combining unit 603, the pixel corresponding to the background region is
0 is stored as the pixel value.

The synthesizing unit 605 synthesizes the background component images based on the components corresponding to the background supplied from the separating unit 601 and the pixels corresponding to the background supplied from the switch 604, and synthesizes the background component images. Output. Since the background area and the mixed area do not overlap, the combining unit 605 applies a logical sum operation to the component corresponding to the background and the pixel corresponding to the background to combine the background component images.

The combining unit 605 stores an image in which all pixel values are 0 in the built-in frame memory in the initialization process which is executed first in the process of combining the background component images. In the synthesizing process of, the background component image is stored (overwritten). Therefore, in the background component image output by the combining unit 605, the pixel corresponding to the foreground area is
0 is stored as the pixel value.

FIG. 90 is a diagram showing an input image input to the foreground / background separation unit 105, and a foreground component image and a background component image output from the foreground / background separation unit 105.

FIG. 90 (A) is a schematic diagram of an image to be displayed, and FIG. 90 (B) shows pixels belonging to the foreground region, pixels belonging to the background region, and mixed regions corresponding to FIG. 90 (A). 3 is a model diagram in which one line of pixels including pixels belonging to is developed in the time direction.

As shown in FIGS. 90 (A) and 90 (B), the background component image output from the foreground / background separator 105 is a background component included in the pixels belonging to the background area and the pixels in the mixed area. Composed of.

As shown in FIGS. 90 (A) and 90 (B), the foreground component image output from the foreground / background separating unit 105 includes the foreground components included in the pixels belonging to the foreground area and the pixels in the mixed area. Composed of.

The pixel values of the pixels in the mixed area are separated by the foreground / background separation unit 105 into a background component and a foreground component. The separated background component constitutes a background component image together with the pixels belonging to the background area. The separated foreground component constitutes a foreground component image together with the pixels belonging to the foreground region.

As described above, in the foreground component image, the pixel values of the pixels corresponding to the background area are set to 0, and the meaningful pixel values are set for the pixels corresponding to the foreground area and the pixels corresponding to the mixed area. Similarly, in the background component image, the pixel value of the pixel corresponding to the foreground area is set to 0, and the meaningful pixel value is set to the pixel corresponding to the background area and the pixel corresponding to the mixed area.

Next, the processing executed by the separating unit 601 for separating the foreground component and the background component from the pixels belonging to the mixed area will be described.

FIG. 91 is a model of an image showing foreground components and background components of two frames including a foreground corresponding to an object moving from left to right in the figure. In the image model shown in FIG. 91, the foreground motion amount v is 4
And the number of virtual divisions is 4.

In frame #n, the leftmost pixel and the fourteenth to eighteenth pixels from the left consist of background components only and belong to the background area. In frame #n,
The second to fourth pixels from the left include the background component and the foreground component, and belong to the uncovered background area. Eleventh to thirteenth from the left in frame #n
The th pixel contains the background and foreground components and belongs to the covered background area. In frame #n, the fifth to tenth pixels from the left consist of only the foreground components and belong to the foreground area.

In frame # n + 1, the first to fifth pixels from the left and the eighteenth pixel from the left consist of the background components only and belong to the background area. In frame # n + 1, the sixth to eighth pixels from the left include the background component and the foreground component, and belong to the uncovered background area. In frame # n + 1, the fifteenth to seventeenth pixels from the left include the background component and the foreground component, and belong to the covered background area. In frame # n + 1, the ninth through fourteenth pixels from the left consist of only the foreground components and belong to the foreground area.

FIG. 92 is a view for explaining the processing for separating the foreground components from the pixels belonging to the covered background area. In FIG. 92, α1 to α18 are mixing ratios corresponding to the respective pixels in frame #n.
In FIG. 92, the fifteenth to seventeenth pixels from the left belong to the covered background area.

The pixel value C15 of the fifteenth pixel from the left of frame #n is expressed by equation (68).

C15 = B15 / v + F09 / v + F08 / v + F07 / v = α15 ・ B15 + F09 / v + F08 / v + F07 / v = α15 ・ P15 + F09 / v + F08 / v + F07 / v (68) where α15 is the mixture ratio of the fifteenth pixel from the left of frame #n. P15 is the pixel value of the 15th pixel from the left of frame # n-1.

1 from the left of frame #n based on equation (68)
The sum f15 of the foreground components of the fifth pixel is expressed by equation (69).

[0502] f15 = F09 / v + F08 / v + F07 / v = C15-α15 ・ P15 (69)

Similarly, the sum f16 of the foreground components of the 16th pixel from the left of frame #n is represented by equation (70), and the sum f17 of the foreground components of the 17th pixel from the left of frame #n is shown.
Is represented by equation (71).

[0504] f16 = C16-α16 ・ P16 (70) f17 = C17-α17 ・ P17 (71)

Thus, the foreground component fc contained in the pixel value C of the pixel belonging to the covered background area is
It is calculated by the equation (72).

Fc = C−α · P (72) P is the pixel value of the corresponding pixel in the immediately preceding frame.

FIG. 93 is a diagram for explaining the processing for separating the foreground components from the pixels belonging to the uncovered background area. In FIG. 93, α1 to α18 are the mixing ratios corresponding to the respective pixels in frame #n. In FIG. 93, the second to fourth pixels from the left belong to the uncovered background area.

[0508] Pixel value of second pixel from the left in frame #n
C02 is represented by Expression (73).

C02 = B02 / v + B02 / v + B02 / v + F01 / v = α2 · B02 + F01 / v = α2 · N02 + F01 / v (73) where α2 is the left of frame #n Is the mixing ratio of the second pixel from. N02 is the pixel value of the second pixel from the left of frame # n + 1.

From the left of frame #n, based on equation (73), 2
The sum f02 of the foreground components of the th pixel is represented by equation (74).

[0511]       f02 = F01 / v          = C02-α2 ・ N02 (74)

Similarly, the sum f03 of the foreground components of the third pixel from the left of frame #n is expressed by equation (75), and the sum f04 of the foreground components of the fourth pixel from the left of frame #n is represented. Is expressed by equation (76).

[0513]       f03 = C03-α3 ・ N03 (75)       f04 = C04-α4 ・ N04 (76)

[0514] As described above, the foreground component fu included in the pixel value C of the pixel belonging to the uncovered background area
Is calculated by equation (77).

Fu = C−α · N (77) N is the pixel value of the corresponding pixel in the frame immediately after.

[0516] As described above, the separation unit 601 includes information indicating the covered background area, which is included in the area information,
And information indicating the uncovered background area,
In addition, the foreground component and the background component can be separated from the pixels belonging to the mixed region based on the mixture ratio α of each pixel.

[0517] Fig. 94 is a block diagram showing an example of the configuration of the separation unit 601 which executes the processing described above. The image input to the separation unit 601 is stored in the frame memory 621.
The region information indicating the covered background region and the uncovered background region and the mixture ratio α supplied from the mixture ratio calculation unit 104 are input to the separation processing block 622.

The frame memory 621 stores the input image in frame units. The frame memory 621 is
When the processing target is the frame #n, the frame # n-1 which is the frame before the frame #n, the frame #n, and the frame # n + 1 which is the frame after the frame #n are Remember.

The frame memory 621 stores the frame # n-1,
The corresponding pixels of frame #n and frame # n + 1 are supplied to the separation processing block 622.

The separation processing block 622 uses the area information indicating the covered background area and the uncovered background area and the mixture ratio α to supply the frame # n-1 and the frame #n supplied from the frame memory 621.
The pixel values of the corresponding pixels of n and frame # n + 1 are shown in FIG.
2 and the operation described with reference to FIG. 93 is applied to separate the foreground component and the background component from the pixels belonging to the mixed area of the frame #n, and the separated components are supplied to the frame memory 623.

The separation processing block 622 is composed of an uncovered area processing section 631, a covered area processing section 632, a combining section 633, and a combining section 634.

Multiplier 6 of uncovered area processing unit 631
Reference numeral 41 indicates a switch 6 by multiplying the pixel value of the pixel of the frame # n + 1 supplied from the frame memory 621 by the mixture ratio α.
To 42. The switch 642 is used for the frame memory 6
When the pixel of frame #n supplied from 21 (corresponding to the pixel of frame # n + 1) is the uncovered background area, the pixel value is closed and multiplied by the mixing ratio α supplied from the multiplier 641. To the computing unit 643 and the combining unit 63
Supply to 4. Frame output from switch 642
The value obtained by multiplying the pixel value of the pixel of # n + 1 by the mixture ratio α is the frame
Equal to the background component of the pixel value of the corresponding pixel in #n.

The calculator 643 subtracts the background component supplied from the switch 642 from the pixel value of the pixel of frame #n supplied from the frame memory 621 to obtain the foreground component. The calculator 643 supplies the foreground component of the pixel of frame #n belonging to the uncovered background area to the combining unit 633.

[0524] The multiplier 651 of the covered area processing unit 632.
Multiplies the pixel value of the pixel of the frame # n−1 supplied from the frame memory 621 by the mixture ratio α, and switches 652.
Output to. The switch 652 is a frame memory 621.
When the pixel of frame #n supplied from (corresponding to the pixel of frame # n-1) is a covered background area, it is closed and the pixel value multiplied by the mixture ratio α supplied from the multiplier 651 is calculated. It is supplied to the container 653 and the combining unit 634. The value obtained by multiplying the pixel value of the pixel of frame # n−1 output from the switch 652 by the mixture ratio α is equal to the background component of the pixel value of the corresponding pixel of frame #n.

The calculator 653 subtracts the background component supplied from the switch 652 from the pixel value of the pixel of frame #n supplied from the frame memory 621 to obtain the foreground component. The calculator 653 supplies the foreground component of the pixel of frame #n belonging to the covered background area to the combining unit 633.

The synthesis unit 633 uses the arithmetic unit 6 of the frame #n.
The foreground components of the pixels belonging to the uncovered background area supplied from 43 and the foreground components of the pixels belonging to the covered background area supplied from the calculator 653 are combined and supplied to the frame memory 623. .

The synthesizer 634 supplies the background component of the pixel belonging to the uncovered background area, supplied from the switch 642, of the frame #n and the switch 652.
The background components of the pixels belonging to the covered background area, which are supplied from
Supply to.

The frame memory 623 stores the foreground component and the background component of the pixels of the mixed area of the frame #n supplied from the separation processing block 622, respectively.

The frame memory 623 outputs the stored foreground component of pixels in the mixed area of frame #n and the stored background component of pixels in the mixed area of frame #n.

By utilizing the mixture ratio α which is the feature amount, it is possible to completely separate the foreground component and the background component included in the pixel value.

The combining unit 603 combines the foreground components of the pixels in the mixed area of frame #n output from the separation unit 601 with the pixels belonging to the foreground area to generate a foreground component image. The synthesizing unit 605 synthesizes the background component of the pixel of the mixed area of the frame #n output from the separating unit 601 and the pixel belonging to the background area to generate a background component image.

FIG. 95 is a diagram showing an example of a foreground component image and an example of a background component image corresponding to the frame #n in FIG.

FIG. 95A shows an example of the foreground component image corresponding to the frame #n in FIG. Since the leftmost pixel and the 14th pixel from the left consist of only the background component before the foreground and the background are separated, the pixel value is set to 0.

The second to fourth pixels from the left belong to the uncovered background area before the foreground and the background are separated, the background component is set to 0, and the foreground component is left as it is. . The eleventh to thirteenth pixels from the left belong to the covered background area before the foreground and the background are separated, the background component is set to 0, and the foreground component is left as it is. The fifth to tenth pixels from the left are left as they are because they consist of only the foreground components.

FIG. 95 (B) shows an example of the background component image corresponding to frame #n in FIG. The leftmost pixel and the 14th pixel from the left are left as they are because they consisted of only the background component before the foreground and the background were separated.

The second to fourth pixels from the left belong to the uncovered background area before the foreground and the background are separated, the foreground component is set to 0, and the background component is left as it is. . The eleventh to thirteenth pixels from the left belong to the covered background area before the foreground and the background are separated, the foreground component is set to 0, and the background component remains. The fifth to tenth pixels from the left have a pixel value of 0 because they consisted of only the foreground components before the foreground and the background were separated.

Next, the process of separating the foreground and the background by the foreground / background separator 105 will be described with reference to the flowchart shown in FIG. In step S601, the frame memory 621 of the separation unit 601 acquires the input image,
The frame #n that is the target of separation of the foreground and the background is stored together with the preceding frame # n-1 and the subsequent frame # n + 1.

[0538] In step S602, the separation unit 601.
The separation processing block 622 of 1 acquires the area information supplied from the mixture ratio calculation unit 104. In step S603, the separation processing block 622 of the separation unit 601 acquires the mixture ratio α supplied from the mixture ratio calculation unit 104.

[0539] In step S604, the uncovered area processing unit 631 determines based on the area information and the mixture ratio α.
The background component is extracted from the pixel values of the pixels belonging to the uncovered background area, which are supplied from the frame memory 621.

In step S605, the uncovered area processing unit 631 uses the area information and the mixture ratio α as a basis.
The foreground component is extracted from the pixel values of the pixels belonging to the uncovered background area, which are supplied from the frame memory 621.

In step S606, the covered area processing unit 632 extracts the background component from the pixel values of the pixels belonging to the covered background area supplied from the frame memory 621 based on the area information and the mixture ratio α. .

In step S607, the covered area processing unit 632 extracts the foreground component from the pixel values of the pixels belonging to the covered background area supplied from the frame memory 621 based on the area information and the mixture ratio α. .

[0543] In step S608, the combining unit 633.
Combines the foreground components of the pixels belonging to the uncovered background area extracted in the processing of step S605 and the foreground components of the pixels belonging to the covered background area extracted in the processing of step S607. The combined foreground component is supplied to the combining unit 603. Further, the combining unit 603 combines the pixels belonging to the foreground region supplied via the switch 602 and the foreground components supplied from the separating unit 601 to generate a foreground component image.

[0544] In step S609, the combining unit 634.
Combines the background components of the pixels belonging to the uncovered background area extracted in the processing of step S604 and the background components of the pixels belonging to the covered background area extracted in the processing of step S606. The combined background component is supplied to the combining unit 605. Furthermore, the combining unit 605 combines the pixels belonging to the background area supplied via the switch 604 and the background components supplied from the separating unit 601 to generate a background component image.

[0545] In step S610, the combining unit 603.
Outputs the foreground component image. In step S611, the combining unit 605 outputs the background component image, and the processing ends.

In this way, the foreground / background separation unit 105 separates the foreground component and the background component from the input image based on the area information and the mixture ratio α, and the foreground component image consisting of only the foreground component, It is possible to output a background component image including only the background component.

Next, adjustment of the amount of motion blur of the foreground component image will be described.

FIG. 97 is a block diagram showing an example of the configuration of the motion-blur adjusting unit 106. The motion vector and its position information supplied from the motion detection unit 102 are supplied to the processing unit determination unit 801, the modeling unit 802, and the calculation unit 805. The area information supplied from the area specifying unit 103 is supplied to the processing unit determining unit 801. The foreground component image supplied from the foreground / background separator 105 is added by the adder 8
04.

The processing unit determination unit 801 generates a processing unit based on the motion vector, its position information, and area information, and supplies the generated processing unit to the modeling unit 802 and addition unit 804.

The processing unit generated by the processing unit determination unit 801 starts from the pixel corresponding to the covered background area of the foreground component image and ends at the pixel corresponding to the uncovered background area, as shown in FIG. It shows continuous pixels lined up in the movement direction or continuous pixels lined up in the movement direction starting from pixels corresponding to the uncovered background area and extending to pixels corresponding to the covered background area. The processing unit is, for example, 2 at the upper left point (the position of the pixel specified in the processing unit and located at the leftmost or uppermost pixel on the image) and the lower right point.
It consists of two data.

The modeling unit 802 executes modeling based on the motion vector and the input processing unit. More specifically, for example, the modeling unit 802 preliminarily prepares a plurality of models corresponding to the number of pixels included in the processing unit, the virtual number of pixel values in the time direction, and the number of foreground components for each pixel. A model that specifies the correspondence between the pixel value and the foreground component as shown in FIG. 99 is selected based on the processing unit and the number of virtual divisions of the pixel value in the time direction.

For example, the number of pixels corresponding to the processing unit is 1.
When it is 2 and the amount of movement v within the shutter time is 5, the modeling unit 802 sets the virtual number of divisions to 5, the pixel located at the leftmost includes one foreground component, and is second from the left. Pixel includes two foreground components, the third pixel from the left includes three foreground components, the fourth pixel from the left includes four foreground components, and the fifth pixel from the left is five. Including the foreground component, the sixth pixel from the left contains five foreground components, the seventh pixel from the left contains five foreground components, and the eighth pixel from the left contains five foreground components , The 9th pixel from the left contains four foreground components, the 10th pixel from the left contains 3 foreground components, the 11th pixel from the left contains 2 foreground components, and the 12th from the left
The model in which the th pixel contains one foreground component and a total of eight foreground components is selected.

[0553] Note that the modeling unit 802 does not select from a model stored in advance, but generates a model based on the motion vector and the processing unit when the motion vector and the processing unit are supplied. You may do it.

The modeling unit 802 supplies the selected model to the equation generation unit 803.

The equation generation unit 803 has a modeling unit 802.
Generates an equation based on the model supplied by. Figure 9
Referring to the model of the foreground component image shown in FIG. 9, the number of foreground components is 8, the number of pixels corresponding to the processing unit is 12, the motion amount v is 5, and the virtual division number is 5. An equation generated by the equation generation unit 803 at a given time will be described.

When the foreground component corresponding to the shutter time / v included in the foreground component image is F01 / v to F08 / v, F01 / v
To F08 / v and the pixel values C01 to C12 are expressed by the formula (78).
It is represented by formula (89).

[0557]       C01 = F01 / v (78)       C02 = F02 / v + F01 / v (79)       C03 = F03 / v + F02 / v + F01 / v (80)       C04 = F04 / v + F03 / v + F02 / v + F01 / v (81)       C05 = F05 / v + F04 / v + F03 / v + F02 / v + F01 / v (82)       C06 = F06 / v + F05 / v + F04 / v + F03 / v + F02 / v (83)       C07 = F07 / v + F06 / v + F05 / v + F04 / v + F03 / v (84)       C08 = F08 / v + F07 / v + F06 / v + F05 / v + F04 / v (85)       C09 = F08 / v + F07 / v + F06 / v + F05 / v (86)       C10 = F08 / v + F07 / v + F06 / v (87)       C11 = F08 / v + F07 / v (88)       C12 = F08 / v (89)

The equation generator 803 transforms the generated equation to generate an equation. The equations generated by the equation generation unit 803 are shown in Equations (90) to (101).

[0559] C01 = 1 / F01 / v + 0 / F02 / v + 0 / F03 / v + 0 / F04 / v + 0 / F05 / v                         + 0 / F06 / v + 0 / F07 / v + 0 / F08 / v (90) C02 = 1 ・ F01 / v + 1 ・ F02 / v + 0 ・ F03 / v + 0 ・ F04 / v + 0 ・ F05 / v                         +0 ・ F06 / v + 0 ・ F07 / v + 0 ・ F08 / v (91) C03 = 1 / F01 / v + 1 / F02 / v + 1 / F03 / v + 0 / F04 / v + 0 / F05 / v                         + 0 / F06 / v + 0 / F07 / v + 0 / F08 / v (92) C04 = 1 / F01 / v + 1 / F02 / v + 1 / F03 / v + 1 / F04 / v + 0 / F05 / v                         + 0 / F06 / v + 0 / F07 / v + 0 / F08 / v (93) C05 = 1 / F01 / v + 1 / F02 / v + 1 / F03 / v + 1 / F04 / v + 1 / F05 / v                         + 0 / F06 / v + 0 / F07 / v + 0 / F08 / v (94) C06 = 0 ・ F01 / v + 1 ・ F02 / v + 1 ・ F03 / v + 1 ・ F04 / v + 1 ・ F05 / v                         +1 / F06 / v + 0 / F07 / v + 0 / F08 / v (95) C07 = 0 ・ F01 / v + 0 ・ F02 / v + 1 ・ F03 / v + 1 ・ F04 / v + 1 ・ F05 / v                         +1 / F06 / v + 1 / F07 / v + 0 / F08 / v (96) C08 = 0 / F01 / v + 0 / F02 / v + 0 / F03 / v + 1 / F04 / v + 1 / F05 / v                         +1 / F06 / v + 1 / F07 / v + 1 / F08 / v (97) C09 = 0 ・ F01 / v + 0 ・ F02 / v + 0 ・ F03 / v + 0 ・ F04 / v + 1 ・ F05 / v                         +1 / F06 / v + 1 / F07 / v + 1 / F08 / v (98) C10 = 0 ・ F01 / v + 0 ・ F02 / v + 0 ・ F03 / v + 0 ・ F04 / v + 0 ・ F05 / v                         +1 / F06 / v + 1 / F07 / v + 1 / F08 / v (99) C11 = 0 ・ F01 / v + 0 ・ F02 / v + 0 ・ F03 / v + 0 ・ F04 / v + 0 ・ F05 / v                         +0 / F06 / v + 1 / F07 / v + 1 / F08 / v (100) C12 = 0 ・ F01 / v + 0 ・ F02 / v + 0 ・ F03 / v + 0 ・ F04 / v + 0 ・ F05 / v                         +0 ・ F06 / v + 0 ・ F07 / v + 1 ・ F08 / v (101)

Equations (90) to (101) are given by equation (10)
It can also be expressed as 2).

[0561]

[Equation 17] In Expression (102), j indicates the position of the pixel. In this example, j has a value of 1 to 12. Further, i indicates the position of the foreground value. In this example, i has a value from 1 to 8. aij
Has a value of 0 or 1 corresponding to the values of i and j.

Expressing in consideration of the error, equation (102)
Can be expressed as in equation (103).

[0563]

[Equation 18] In Expression (103), ej is an error included in the target pixel Cj.

Expression (103) can be rewritten as expression (104).

[0565]

[Formula 19]

Since the least squares method is applied, the error sum of squares E is defined as shown in equation (105).

[0567]

[Equation 20]

In order to minimize the error, the sum of squares of the error is
It suffices if the value of the partial differential with respect to E by the variable Fk becomes zero. Fk is calculated so as to satisfy the equation (106).

[0569]

[Equation 21]

In the equation (106), since the motion amount v is a fixed value, the equation (107) can be derived.

[0571]

[Equation 22]

When the expression (107) is expanded and transposed, the expression (108) is obtained.

[0573]

[Equation 23]

It is expanded into eight expressions obtained by substituting any one of the integers 1 to 8 for k in the expression (108). The obtained eight expressions can be expressed by one expression by a matrix. This equation is called a normal equation.

An example of a normal equation generated by the equation generation unit 803 based on such a least squares method is shown in equation (109).

[0576]

[Equation 24]

When the equation (109) is expressed as AF = vC, C, A, v are known and F is unknown. A and v are known at the time of modeling, but C is known by inputting pixel values in the adding operation.

The error contained in the pixel C can be dispersed by calculating the foreground component by the normal equation based on the method of least squares.

The equation generator 803 supplies the normal equation thus generated to the adder 804.

[0580] The adding unit 804 is the processing unit determining unit 80.
Based on the processing unit supplied from 1, the pixel value C included in the foreground component image is set in the matrix equation supplied from the equation generation unit 803. The adder 804 supplies the matrix in which the pixel value C is set to the calculator 805.

The operation unit 805 uses the sweep method (Gauss-Jord
The foreground component Fi / v with motion blur removed is calculated by a process based on a solution method such as (an elimination method), and is an integer from 0 to 8 that is the pixel value of the foreground with motion blur removed. 100 corresponding to the i of FIG. 100 is calculated, and the foreground component image from which motion blur has been removed, which consists of Fi that is the pixel value from which motion blur has been removed, is shown in FIG. Output to 807.

Note that in the foreground component image from which the motion blur shown in FIG. 100 has been removed, F01 is added to each of C03 to C10.
Each of F08 to F08 is set because the position of the foreground component image with respect to the screen is not changed, and can be set to any position.

The motion-blur adding unit 806 determines a motion-blur adjustment amount v ′ having a value different from the motion amount v, for example, a motion-blur adjustment amount v ′ that is half the motion amount v, or a value irrelevant to the motion amount v. The amount of motion blur can be adjusted by giving the amount of motion blur adjustment v'of. For example, as shown in FIG. 101, the motion-blur adding unit 806 causes the motion-blur-removed foreground pixel value to be removed.
By dividing Fi by the motion blur adjustment amount v ', the foreground component Fi /
v ′ is calculated, the sum of the foreground components Fi / v ′ is calculated, and a pixel value in which the amount of motion blur is adjusted is generated. For example, when the motion blur adjustment amount v ′ is 3, the pixel value C02 is (F01) / v ′, the pixel value C03 is (F01 + F02) / v ′, and the pixel value C04
Is (F01 + F02 + F03) / v ', and the pixel value C05 is (F02 + F03
03 + F04) / v '.

The motion blur adding section 806 supplies the foreground component image in which the amount of motion blur is adjusted to the selecting section 807.

The selecting unit 807, for example, based on the selection signal corresponding to the user's selection, the foreground component image supplied from the arithmetic unit 805 from which motion blur has been removed, and the motion supplied from the motion blur adding unit 806. Either one of the foreground component images whose blur amount has been adjusted is selected, and the selected foreground component image is output.

In this way, the motion blur adjusting section 106 can adjust the amount of motion blur based on the selection signal and the motion blur adjustment amount v '.

Also, for example, as shown in FIG. 102, when the number of pixels corresponding to the processing unit is 8 and the motion amount v is 4, the motion-blur adjusting unit 106 uses the matrix shown in equation (110). Generates the expression.

[0588]

[Equation 25]

The motion-blur adjusting unit 106 formulates a number equation corresponding to the length of the processing unit in this way to calculate Fi, which is a pixel value in which the amount of motion-blur is adjusted. Similarly, for example, when the number of pixels included in the processing unit is 100, 1
Fi is calculated by generating an equation corresponding to 00 pixels.

FIG. 103 is a diagram showing another configuration of the motion-blur adjusting unit 106. The same parts as those shown in FIG. 97 are designated by the same reference numerals, and the description thereof will be omitted.

Based on the selection signal, the selection unit 821 supplies the input motion vector and its position signal to the processing unit determination unit 801 and the modeling unit 802 as they are, or adjusts the size of the motion vector by motion blur. The motion vector whose position is replaced with the motion blur adjustment amount v ′ and its position signal by replacing the amount v ′ with the motion blur adjustment amount v ′ is supplied to the processing unit determination unit 801 and the modeling unit 802.

By doing so, the processing unit determination unit 801 to the calculation unit 80 of the motion blur adjustment unit 106 of FIG.
5 corresponds to the values of the motion amount v and the motion blur adjustment amount v ′,
The amount of motion blur can be adjusted. For example, the amount of movement
When v is 5 and the motion blur adjustment amount v ′ is 3, FIG.
The processing unit determination unit 801 to the calculation unit 805 of the motion blur adjustment unit 106 of 3 corresponds to the motion blur adjustment amount v ′ of 3 for the foreground component image of which the motion amount v is 5 illustrated in FIG. Perform the operation according to the model as shown in
(Motion amount v) / (Motion blur adjustment amount v ′) = 5/3, that is, an image including motion blur corresponding to the motion amount v of approximately 1.7 is calculated. In this case, since the calculated image does not include the motion blur corresponding to the motion amount v of 3, the result of the motion blur adding unit 806 is the motion amount v and the motion blur adjustment amount v ′.
It should be noted that the implications of the relationship are different.

As described above, the motion-blur adjusting unit 106
An expression is generated corresponding to the motion amount v and the processing unit, and the pixel value of the foreground component image is set in the generated expression to calculate the foreground component image in which the amount of motion blur is adjusted.

Next, the process of adjusting the amount of motion blur included in the foreground component image by the motion blur adjusting unit 106 will be described with reference to the flowchart in FIG.

In step S801, the processing unit determination unit 801 of the motion blur adjustment unit 106 generates a processing unit based on the motion vector and area information, and supplies the generated processing unit to the modeling unit 802.

In step S802, the modeling unit 802 of the motion-blur adjusting unit 106 selects or generates a model corresponding to the motion amount v and the processing unit. Step S
In 803, the equation generation unit 803 creates a normal equation based on the selected model.

[0597] In step S804, the adding portion 8 is added.
04 sets the pixel value of the foreground component image in the created normal equation. In step S805, the adding portion 8
04 determines whether or not the pixel values of all the pixels corresponding to the processing unit have been set, and when it is determined that the pixel values of all the pixels corresponding to the processing unit have not been set, step S804. Then, the process of setting the pixel value in the normal equation is repeated.

If it is determined in step S805 that the pixel values of all the pixels in the processing unit have been set, the operation proceeds to step S806, and the arithmetic unit 805 causes the adder 8
Based on the normal equation in which the pixel values supplied from 04 are set, the pixel value of the foreground in which the amount of motion blur is adjusted is calculated,
The process ends.

As described above, the motion blur adjusting section 106 can adjust the amount of motion blur from the foreground image containing motion blur based on the motion vector and the area information.

That is, the amount of motion blur included in the pixel value which is the sample data can be adjusted.

FIG. 105 is a block diagram showing another example of the configuration of the motion-blur adjusting unit 106. Motion detector 102
The motion vector and the position information thereof supplied from the processing unit determination unit 901 and the correction unit 905 are supplied to the processing unit determination unit 901, and the area information supplied from the region identification unit 103 is supplied to the processing unit determination unit 901. The foreground component image supplied from the foreground / background separation unit 105 is supplied to the calculation unit 904.

The processing unit determination unit 901 generates a processing unit based on the motion vector, its position information, and area information, and supplies the generated processing unit to the modeling unit 902 together with the motion vector.

The modeling unit 902 executes modeling based on the motion vector and the input processing unit. More specifically, for example, the modeling unit 902 preliminarily creates a plurality of models corresponding to the number of pixels included in the processing unit, the virtual number of pixel values in the time direction, and the number of foreground components for each pixel. A model that specifies the correspondence between the pixel value and the foreground component as shown in FIG. 106 is selected based on the processing unit and the number of virtual divisions of the pixel value in the time direction.

For example, if the number of pixels corresponding to the processing unit is 1
When it is 2 and the amount of movement v is 5, the modeling unit 902 sets the number of virtual divisions to 5, the leftmost pixel includes one foreground component, and the second pixel from the left is 2 pixels.
The third pixel from the left contains three foreground components, the fourth pixel from the left contains four foreground components, and the fifth pixel from the left contains five foreground components. Including, the 6th pixel from the left contains 5 foreground components, the 7th pixel from the left contains 5 foreground components, and 8 from the left
The tenth pixel contains five foreground components, the ninth pixel from the left contains four foreground components, the tenth pixel from the left contains three foreground components, and the eleventh pixel from the left is two.
A model is selected that includes two foreground components, the twelfth pixel from the left includes one foreground component, and a total of eight foreground components.

The modeling unit 902 does not select from a model stored in advance, but generates a model based on the motion vector and the processing unit when the motion vector and the processing unit are supplied. You may do it.

[0606] The equation generation unit 903 is the modeling unit 902.
Generates an equation based on the model supplied by.

Referring to the models of the foreground component images shown in FIGS. 106 to 108, the number of foreground components is 8, the number of pixels corresponding to the processing unit is 12, and the motion amount v is 5. An example of an equation generated by the equation generation unit 903 at that time will be described.

When the foreground component corresponding to the shutter time / v included in the foreground component image is F01 / v to F08 / v, F01 / v
The relationships between F08 / v to F08 / v and the pixel values C01 to C12 are expressed by the equations (78) to (89) as described above.

Focusing on the pixel values C12 and C11, the pixel values
As shown in Expression (111), C12 includes only the foreground component F08 / v, and the pixel value C11 is the sum of products of the foreground component F08 / v and the foreground component F07 / v. Therefore, the foreground component F07 / v
Can be calculated by equation (112).

[0610]       F08 / v = C12 (111)       F07 / v = C11-C12 (112)

Similarly, considering the foreground components included in the pixel values C10 to C01, the foreground components F06 / v to F01 / v can be obtained from the equations (113) to (118).

[0612]       F06 / v = C10-C11 (113)       F05 / v = C09-C10 (114)       F04 / v = C08-C09 (115)       F03 / v = C07-C08 + C12 (116)       F02 / v = C06-C07 + C11-C12 (117)       F01 / v = C05-C06 + C10-C11 (118)

The equation generation unit 903 generates the equations shown in the equations (111) to (118) for calculating the foreground component from the pixel value difference. Equation generator 90
3 supplies the generated equation to the calculation unit 904.

The calculation unit 904 sets the pixel value of the foreground component image in the equation supplied from the equation generation unit 903,
The foreground component is calculated based on the equation in which the pixel value is set. The calculation unit 904 may calculate, for example, the equations (111) to (1).
When (18) is supplied from the equation generation unit 903, the pixel values C05 to C12 are set in the equations (111) to (118).

The calculation unit 904 calculates the foreground component based on the equation in which the pixel value is set. For example, the calculation unit 904
Calculates the foreground components F01 / v to F08 / v as shown in FIG. 107 by the calculation based on the equations (111) to (118) in which the pixel values C05 to C12 are set. Computing unit 90
4 supplies the foreground components F01 / v to F08 / v to the correction unit 905.

The correction unit 905 multiplies the foreground component supplied from the calculation unit 904 by the motion amount v included in the motion vector supplied from the processing unit determination unit 901, and removes the motion blur from the foreground pixel value. To calculate. For example, the correction unit 9
05 is the foreground component F01 / v supplied from the calculation unit 904.
To F08 / v are supplied, the foreground components F01 / v to F08 /
By multiplying each of v by the amount of movement v, which is 5,
As shown in FIG. 108, the foreground pixel values F01 to F08 from which motion blur has been removed are calculated.

The correction unit 905 supplies the motion blur adding unit 906 and the selection unit 907 with the foreground component image composed of the pixel values of the foreground from which motion blur has been calculated, calculated as described above.

The motion-blur adding unit 906 determines a motion-blur adjustment amount v ′ having a value different from the motion amount v, for example, a motion-blur adjustment amount v ′ having a half value of the motion amount v, and a value unrelated to the motion amount v. The amount of motion blur can be adjusted with the amount of motion blur adjustment v '.
For example, as shown in FIG. 101, the motion blur adding unit 906
Calculates the foreground component Fi / v 'by dividing the foreground pixel value Fi from which motion blur has been removed by the motion blur adjustment amount v',
The sum of the foreground components Fi / v 'is calculated to generate a pixel value in which the amount of motion blur is adjusted. For example, the motion blur adjustment amount v ′ is 3
, The pixel value C02 is set to (F01) / v ', and the pixel value C03
Is (F01 + F02) / v ', and the pixel value C04 is (F01 + F02 + F
03) / v ', and the pixel value C05 is (F02 + F03 + F04) / v'.

The motion blur adding section 906 supplies the foreground component image in which the amount of motion blur is adjusted to the selecting section 907.

The selecting unit 907, for example, based on the selection signal corresponding to the selection by the user, the foreground component image from which the motion blur supplied from the correcting unit 905 has been removed, and the motion supplied from the motion blur adding unit 906. Either one of the foreground component images whose blur amount has been adjusted is selected, and the selected foreground component image is output.

In this way, the motion blur adjusting section 106 can adjust the amount of motion blur based on the selection signal and the motion blur adjustment amount v '.

Next, the process of adjusting the amount of motion blur of the foreground by the motion blur adjuster 106 having the configuration shown in FIG.
This will be described with reference to the flowchart of No. 09.

In step S901, the processing unit determination unit 901 of the motion blur adjustment unit 106 generates a processing unit based on the motion vector and the area information, and supplies the generated processing unit to the modeling unit 902 and the correction unit 905. To do.

In step S902, the modeling unit 902 of the motion-blur adjusting unit 106 selects or generates a model corresponding to the motion amount v and the processing unit. Step S
In 903, the equation generation unit 903 generates an equation for calculating the foreground component based on the difference in pixel value of the foreground component image based on the selected or generated model.

In step S904, the operation unit 904
Sets the pixel value of the foreground component image in the created equation, and extracts the foreground component from the difference in pixel value based on the equation in which the pixel value is set. In step S905,
The calculation unit 904 determines whether or not all foreground components corresponding to the processing unit have been extracted, and when it is determined that all foreground components corresponding to the processing unit have not been extracted, the processing unit 904 returns to step S904. , The process of extracting the foreground components is repeated.

If it is determined in step S905 that all foreground components corresponding to the processing unit have been extracted,
In step S906, the correction unit 905 corrects each of the foreground components F01 / v to F08 / v supplied from the calculation unit 904 based on the motion amount v to remove the motion blur, and the foreground pixel value. Calculate F01 to F08.

In step S907, the motion blur adding unit 906 calculates the pixel value of the foreground in which the amount of motion blur is adjusted, and the selecting unit 907 adjusts the image from which motion blur is removed or the amount of motion blur. Selected image is output, the selected image is output, and the process ends.

As described above, the motion-blur adjusting unit 106 whose configuration is shown in FIG. 105 can adjust the motion-blur from the foreground image including the motion-blur more quickly by a simpler calculation.

The conventional method for partially removing motion blur such as the Wiener filter is effective in the ideal state, but is not sufficiently effective for an actual image including quantization and noise. On the other hand, even in the motion blur adjusting unit 106 having the configuration shown in FIG. 105, a sufficient effect is recognized even on an actual image that is quantized and contains noise, and motion blur can be removed with high accuracy. Become.

As described above, the separation processing server 11 shown in FIG. 27 can adjust the amount of motion blur included in the input image.

FIG. 110 is a block diagram showing another structure of the function of the separation processing server 11.

The same parts as those shown in FIG. 27 are designated by the same reference numerals, and the description thereof will be appropriately omitted.

The area specifying unit 103 supplies the area information to the mixture ratio calculating unit 104 and the synthesizing unit 1001.

The mixing ratio calculating unit 104 supplies the mixing ratio α to the foreground / background separating unit 105 and the synthesizing unit 1001.

The foreground / background separator 105 supplies the foreground component image to the synthesizer 1001.

The synthesizing unit 1001 is supplied from an arbitrary background image and the foreground / background separating unit 105 based on the mixture ratio α supplied from the mixture ratio calculating unit 104 and the area information supplied from the area specifying unit 103. The foreground component image is combined to output a combined image in which an arbitrary background image and the foreground component image are combined.

FIG. 111 is a diagram showing the structure of the combining unit 1001. The background component generation unit 1021 generates a background component image based on the mixture ratio α and an arbitrary background image, and supplies the background component image to the mixed region image synthesis unit 1022.

The mixed area image synthesizing unit 1022 generates a mixed area synthetic image by synthesizing the background component image and the foreground component image supplied from the background component generating unit 1021, and generates the generated mixed area synthetic image. Image composition unit 1023
Supply to.

The image composition unit 1023, based on the area information,
The foreground component image, the mixed area combined image supplied from the mixed area image combining unit 1022, and an arbitrary background image are combined to generate and output a combined image.

As described above, the synthesizing unit 1001 can synthesize the foreground component image with an arbitrary background image.

The image obtained by synthesizing the foreground component image with an arbitrary background image based on the mixture ratio α which is the feature amount becomes more natural than the image obtained by simply synthesizing pixels.

FIG. 112 is a block diagram showing still another configuration of the function of the separation processing server 11 for adjusting the amount of motion blur. While the separation processing server 11 shown in FIG. 27 sequentially performs the region identification and the calculation of the mixture ratio α, the separation processing server 11 shown in FIG. 112 performs the region identification and the calculation of the mixture ratio α in parallel.

The same parts as those of the function shown in the block diagram of FIG. 27 are designated by the same reference numerals, and the description thereof will be omitted.

The input image is supplied to the mixture ratio calculating unit 1101, the foreground / background separating unit 1102, the region specifying unit 103, and the object extracting unit 101.

The mixing ratio calculating unit 1101 estimates the mixing ratio based on the input image when the pixel belongs to the covered background area and the estimated mixing ratio when the pixel belongs to the uncovered background area. A ratio is calculated for each of the pixels included in the input image, and the estimated mixture ratio in the case where the calculated pixel is assumed to belong to the covered background region,
And the estimated mixture ratio under the assumption that the pixel belongs to the uncovered background area is supplied to the foreground / background separation unit 1102.

FIG. 113 is a block diagram showing an example of the configuration of the mixture ratio calculation unit 1101.

Estimated mixture ratio processing unit 401 shown in FIG. 113.
Is the same as the estimated mixture ratio processing unit 401 shown in FIG. The estimated mixture ratio processing unit 402 shown in FIG.
It is the same as the estimated mixing ratio processing unit 402 shown in FIG.

The estimated mixture ratio processing unit 401 calculates the estimated mixture ratio for each pixel by the calculation corresponding to the model of the covered background area based on the input image, and outputs the calculated estimated mixture ratio.

The estimated mixture ratio processing unit 402 calculates the estimated mixture ratio for each pixel by the calculation corresponding to the model of the uncovered background area based on the input image, and outputs the calculated estimated mixture ratio.

The foreground / background separator 1102 supplies the estimated mixture ratio supplied from the mixture ratio calculator 1101, assuming that the pixel belongs to the covered background area,
And the estimated mixture ratio in the case where the pixel belongs to the uncovered background area, and the area information supplied from the area specifying unit 103, the foreground component image is generated from the input image, and the generated foreground component image is generated. It is supplied to the motion blur adjusting unit 106 and the selecting unit 107.

FIG. 114 is a block diagram showing an example of the structure of the foreground / background separator 1102.

The same parts as those of the foreground / background separator 105 shown in FIG. 89 are designated by the same reference numerals, and the description thereof will be omitted.

The selecting unit 1121 uses the region information supplied from the region specifying unit 103 to estimate the mixture ratio supplied from the mixture ratio calculating unit 1101, assuming that the pixel belongs to the covered background region, and One of the estimated mixing ratios in the case where the pixel is assumed to belong to the uncovered background area is selected, and the selected estimated mixing ratio is supplied to the separation unit 601 as the mixing ratio α.

The separating unit 601 extracts the foreground component and the background component from the pixel values of the pixels belonging to the mixed region based on the mixture ratio α and the region information supplied from the selecting unit 1121, and extracts the extracted foreground component. To the synthesizing unit 603 and the background component to the synthesizing unit 605.

Separation unit 601 can have the same structure as that shown in FIG.

The synthesizer 603 synthesizes the foreground component image and outputs it. The combining unit 605 combines and outputs the background component images.

The motion-blur adjusting unit 106 shown in FIG.
A configuration similar to that shown in FIG. 27 can be adopted, and the foreground / background separation unit 110 is based on the region information and the motion vector.
The amount of motion blur included in the foreground component image supplied from No. 2 is adjusted, and the foreground component image with the adjusted amount of motion blur is output.

The selection unit 107 shown in FIG. 112 is based on, for example, a selection signal corresponding to the user's selection, the foreground component image supplied from the foreground / background separation unit 1102, and the motion supplied from the motion blur adjustment unit 106. Either one of the foreground component images whose blur amount has been adjusted is selected, and the selected foreground component image is output.

As described above, the separation processing server 11 having the configuration shown in FIG. 112 adjusts the amount of motion blur included in an image corresponding to the foreground object included in the input image and outputs the adjusted image. be able to. The separation processing server 11 having the configuration shown in FIG. 112 can calculate the mixture ratio α, which is the buried information, and output the calculated mixture ratio α, as in the first embodiment.

FIG. 115 is a block diagram showing another configuration of the function of the separation processing server 11 for synthesizing the foreground component image with an arbitrary background image. Separation processing server 1 shown in FIG. 110
1 performs the region identification and the calculation of the mixture ratio α serially, whereas the separation processing server 11 shown in FIG. 115 performs the region identification and the calculation of the mixture ratio α in parallel.

The same parts as those of the function shown in the block diagram of FIG. 112 are designated by the same reference numerals, and the description thereof will be omitted.

The mixing ratio calculation unit 1101 shown in FIG.
Based on the input image, the estimated mixture ratio when assuming that the pixel belongs to the covered background region and the estimated mixture ratio when assuming that the pixel belongs to the uncovered background region are The foreground / background separation unit 1102 calculates the estimated mixture ratio calculated for each pixel and the estimated mixture ratio when the calculated pixel belongs to the covered background region and the estimated mixture ratio when the pixel belongs to the uncovered background region. And to the synthesis unit 1201.

Foreground / background separator 1102 shown in FIG.
Is an estimated mixture ratio when the pixel is assumed to belong to the covered background region and an estimated mixture ratio when the pixel is assumed to belong to the uncovered background region, and the region supplied from the mixture ratio calculation unit 1101. The foreground component image is generated from the input image based on the area information supplied from the specifying unit 103, and the generated foreground component image is supplied to the combining unit 1201.

The synthesis unit 1201 has a mixing ratio calculation unit 1101.
, The estimated mixture ratio when the pixels are assumed to belong to the covered background area, and the estimated mixture ratio when the pixels are assumed to belong to the uncovered background area, the area supplied from the area specifying unit 103. An arbitrary background image and the foreground component image supplied from the foreground / background separating unit 1102 are combined based on the information, and a combined image in which the arbitrary background image and the foreground component image are combined is output.

FIG. 116 is a diagram showing the structure of the combining unit 1201. The same numbers are given to the same parts as the functions shown in the block diagram of FIG. 111, and the description thereof will be omitted.

The selecting unit 1221 uses the region information supplied from the region specifying unit 103 to estimate the mixture ratio supplied from the mixture ratio calculating unit 1101, assuming that the pixel belongs to the covered background region, and The background component generation unit 10 selects either one of the estimated mixture ratios on the assumption that the pixel belongs to the uncovered background area and sets the selected estimated mixture ratio as the mixture ratio α.
21.

[0667] The background component generation unit 1021 shown in Fig. 116.
Generates a background component image based on the mixture ratio α and the arbitrary background image supplied from the selection unit 1221 and supplies the background component image to the mixed region image synthesis unit 1022.

Mixed area image combining section 102 shown in FIG.
2 generates a mixed area combined image by combining the background component image and the foreground component image supplied from the background component generation unit 1021, and supplies the generated mixed area combined image to the image combining unit 1023.

The image composition unit 1023, based on the area information,
The foreground component image, the mixed area combined image supplied from the mixed area image combining unit 1022, and an arbitrary background image are combined to generate and output a combined image.

As described above, the combining unit 1201 can combine the foreground component image with an arbitrary background image.

Although the mixing ratio α has been described as the ratio of the background component included in the pixel value, it may be the ratio of the foreground component included in the pixel value.

Further, the direction of movement of the foreground object has been described as being from left to right, but it goes without saying that it is not limited to that direction.

In the above, the case where the image of the real space having the three-dimensional space and the time-axis information is projected to the two-dimensional space and the space-time having the time-axis information has been described as an example. The present invention is not limited to this example, and when more first information of the first dimension is projected to less second information of the second dimension, the distortion generated by the projection is corrected. It is also possible to apply it when extracting significant information, or when synthesizing images more naturally.

The sensor 76a is not limited to the CCD but is a solid-state image sensor, for example, a BBD (Bucket Brigade Devi).
ce), CID (Charge Injection Device), CPD (Charge
Priming Device) or CMOS (Complementary Mental
Oxide Semiconductor) sensor, and the detection elements are not limited to those arranged in a matrix,
A sensor in which the detection elements are arranged in a line may be used.

Each function of the separation processing server 11 described above can be realized by distributed processing by various servers configured on the network 1 shown in FIG. That is, the object extraction unit 10
1 and the motion detection unit 102 is the motion detection server 12
The area identification unit 103 is the area identification server 13, the mixture ratio calculation unit 104 is the mixture ratio calculation server 14, the foreground / background separation processing unit 105 is the foreground / background separation processing server 15, and the motion blur adjustment unit 106 is the motion blur adjustment. The server 16 functions as a corresponding one. Therefore, the block diagram of the separation processing server 11 shown in FIG. 27 may be realized by hardware, software, or the network 1. In addition, the synthesis server 19
The same applies to the above, and its configuration corresponds to the combining unit 1201, and may be realized by hardware, realized by software, or realized by the network 1. Is.

The object extraction unit 10 described above is used.
1, the motion detecting unit 102, the region specifying unit 103, the mixture ratio calculating unit 104, the foreground / background separating processing unit 105, and the motion blur adjusting unit 106 perform the respective processes.
2, the area specifying server 13, the mixture ratio calculating server 14, the foreground / background separating processing server 15, and the motion blur adjusting server 16 are the same as those that are replaced, and the processing is also the same, so the description thereof is omitted. To do.

When the separation processing server 11 is realized as hardware or software as described above, various servers connected to the network 1 shown in FIG. 1, a client computer 27, and
It may be configured to be incorporated as a separation processing unit in each of the camera terminal devices 28. Therefore, in the following description, when the separation processing server 11 is described as a single device having a function of simply separating the input image into the foreground component image and the background component image, it is also referred to as the separation processing unit 11. And

Next, referring to the flowchart of FIG. 117, the network 1 of FIG.
The process of the image separation service input from the client computer 27 via the will be described.

In step S1001, the client computer 27 outputs information designating an image to the separation processing server 11. That is, as information for specifying an image that the user wants to separate, a specific image or an image ID specifying the image is output to the separation processing server 11.

In step S1011, the separation processing server 11 acquires the designated image. That is, when the image is transmitted from the client computer 27, when the information for designating the image is transmitted, the image corresponding to the image ID is read out and acquired from the network 1.

In step S1012, the charging processing unit 11a of the separation processing server 11 executes the charging processing together with the charging server 24 via the network 1. At the same time, in step S1021, the charging server 24
The accounting process is executed together with the separation processing server 11.

[0682] Here, the above-mentioned billing process will be described with reference to the flowchart in Fig. 118. The actual charging process is executed by the separation processing server 11 and the charging server 24, but the information necessary for various processes is also output from the client computer 27. Also explained.

In step S1101, as shown in FIG. 119, the client computer 27 specifies the service and ID information and authentication information (password or the like) for identifying the user (user who receives the image separation service). At the same time, the usage amount is transmitted to the separation processing server 11 via the network 1. That is, in the present case, FIG.
When the information designating the image is transmitted in the process of step S1001, the process of step S1101 is executed. Further, the usage fee is a fee for the separation service.

[0684] In step S1111, as shown in Fig. 119, the billing processing unit 11a of the separation processing server 11 is executed.
Receives the ID information and the authentication information, and further transmits the usage amount and its own ID to the billing server 24.

[0685] In step S1121, as shown in Fig. 119, the accounting server 24 determines the authentication information and the customer account I based on the ID transmitted from the separation processing server 11.
Inquire about D and the usage amount to the financial server 25 managed by the financial institution of the customer account.

[0686] In step S1131, as shown in Fig. 119, the financial server (for customer) 25 uses the customer account ID.
The authentication processing is executed based on the authentication information and the authentication result and the availability information are notified to the billing server 24.

In step S1122, as shown in FIG. 119, the billing server 24 transmits the authentication result and the availability information to the separation processing server 11. In the following description, the description will be made under the condition that the authentication result is satisfactory and the authentication result can be used. Further, when the information that the authentication result has a problem and the use thereof is not permitted is received, the process ends.

[0688] In step S1112, as shown in Fig. 119, the separation processing server 11 provides the service to the client computer 27 if there is no problem in the authentication result and the financial institution can be used. To do. In step S1102, the client computer 27 receives the service. That is, in this case, in step S1112, the separation processing server 1
1 separates the designated image into a foreground component image and a background component image and outputs the image to the client computer 27. In step S1102, the client computer 2
7 receives the separated foreground component image and background component image.

[0689] In step S1113, the separation processing server 11 transmits a service use notification to the billing server 24. In step S1123, the billing server 24
Notifies the financial server (for customer) 25 of the customer account ID, the usage amount, and the provider account ID.

[0690] In step S1132, the financial server (for customer) 25 transfers the usage amount from the account of the customer account ID to the provider financial server (for provider) 26.

Now, let us return to the description of the flowchart in FIG. 117.

After the separation processing server 11 and the charging server 24 perform the charging processing in steps S1012 and S1021, the separation processing server 11 executes the image separation processing in step S1013. That is, the area specifying unit 103 of the separation processing server 11 is configured as shown in FIG.
The area specifying process described with reference to the flowchart of FIG.
The mixture ratio calculation unit 104 performs the process of calculating the mixture ratio described with reference to the flowchart of FIG.
Reference numeral 05 denotes the process of separating the foreground and background described with reference to the flowchart of FIG.
The motion blur amount adjustment processing described with reference to the flowchart in FIG. 44 is executed to separate the designated image.
The process of adjusting the amount of motion blur, the process of specifying the region, the process of calculating the mixture ratio, and the process of separating the foreground and the background are all the same as those described above, and a description thereof will be omitted.

[0693] In step S1014, the separation processing server 11 assigns IDs to the separated foreground component image and background component image and transmits them to the client computer 27. In step S1002, the client computer 27 sends the separated foreground component image and background component image transmitted from the separation processing server 11 and their components.
The ID is received, stored in its own storage unit 48 (FIG. 2), and printed out as necessary. In addition, the client computer 27 stores the foreground component image and the background component image that have been separated by the separation processing server 11 in the separation processing server 11 itself in accordance with a command from the user,
Alternatively, the data may be output to the storage server 18 via the network 1 and stored (stored).

In the above description, the processing in the case where the fee related to the separation processing is transferred to the financial servers 25 and 26 by the billing server 24 has been described. However, for example, the prepaid point or the like is used by the user. Stores the point indicating that the user paid the usage fee to the separation service provider in advance in the storage unit 48 (FIG. 2), and subtracts the point each time the separation service is provided, thereby performing the charging process. It may be executed.

Now, with reference to the flowchart of FIG. 120, the charging process when the prepaid points are used will be described.

[0696] In step S1201, the client computer 27 specifies a service, ID information,
Also, the authentication information is transmitted. That is, the client computer 27 executes the same process as the process in step S1101 of FIG. 117.

[0697] In step S1211, the billing processing unit 11a of the separation processing server 11 receives the ID information and the authentication information. In step S <b> 1212, the billing processing unit 11 a, from the prepaid points corresponding to the amount paid in advance by the user of the client computer 27 stored in the storage unit 48, the points corresponding to the usage amount related to the separation process. Subtract and store.
In step S1213, the separation processing server 11
Provide services. That is, in this case, the separation processing server 11 executes the separation processing of the input image and transmits the separated foreground component image and background component image to the client computer 27.

At step S1202, the client computer 27 receives the service. That is, in this case, the client computer 27 receives the foreground component image and the background component image transmitted from the separation processing server 11.

Also, in the above, the separation processing server 1
1 has stored the prepaid points in its own storage unit 48 (FIG. 2), the same processing is performed, for example, when a card in which the prepaid points are recorded, that is, a so-called prepaid card is used. Become. In this case, the prepaid point stored in the prepaid card is read and transmitted by the client computer 27 in step S1201, and the separation processing server 11
However, it is necessary to subtract points corresponding to the usage fee from the points received by the billing process, transmit the points resulting from the subtraction to the client computer 27, and overwrite the points on the prepaid card.

Next, the processing of the motion detection service for obtaining the motion vector and position information of the image designated by the client computer 27 will be described with reference to the flowchart in FIG.

In step S1301, the client computer 27 outputs information designating an image to the motion detection server 12. That is, as the information for specifying the image that the user wants to perform the motion detection processing, the image ID that is a specific image or the image that specifies the image is output to the motion detection server 12.

In step S1311, the motion detection server 12 acquires the designated image. That is, when the image is transmitted from the client computer 27, when the information for designating the image is transmitted, the image corresponding to the image ID is read out and acquired from the network 1.

In steps S1312 and S1321, the billing processing unit 12c of the motion detection server 12 and the billing server 24 execute billing processing. Note that the charging process is the same as that in the separated service in FIGS. 118 and 120, and therefore its description is omitted.

[0704] In step S1313, the object extraction unit 12a of the motion detection server 12 extracts each object from the acquired specified image, the motion detection unit 12b detects the position information and the motion vector, and the client computer Send to 27.

[0705] In step S1302, the client computer 27 receives and stores the position information and the motion vector of the object transmitted from the motion detection server 12.

The client computer 27 stores the position information and the motion vector detected by the motion detection server 12 in the motion detection server 12 itself according to a command from the user, or via the network 1. It is also possible to output to the storage server 18 and store (store) it.

Next, referring to the flowchart of FIG. 122, the processing of the area specifying service executed by the area specifying server 13 for specifying an area from information specifying an image and an object input from the client computer 27 will be described. To do.

In step S1401, the client computer 27 outputs information designating an image and an object to the area specifying server 13. That is, as the information for specifying the image for which the user wants to specify the area, the information for specifying the object, which is either a specific image or the image ID for specifying the image, is output to the area specifying server 13.

[0709] In step S1411, the area specifying server 13 acquires the specified image. That is, when an image is transmitted from the client computer 27, and when an image ID for designating the image is transmitted, the image corresponding to the image ID is read and acquired from the network 1.

In steps S1412 and S1421, the charging processing unit 13a of the area specifying server 13 and the charging server 24 execute the charging process. Note that the charging process is the same as that in the separated service in FIGS. 118 and 120, and therefore its description is omitted.

[0711] In step S1413, the area specifying server 13 determines, based on the information designating the object,
Perform area-specific processing. Note that the region specifying process is the same as the process described with reference to the flowchart in FIG. 53, and thus description thereof will be omitted.

In step S1414, the area specifying server 13 transmits the area information obtained in the processing of step S1413 to the client computer 27.

At step S1402, the client computer 27 receives and stores the area information transmitted from the area specifying server 13.

The client computer 27 stores the area information obtained by the area specifying server 13 in the area specifying server 13 itself in accordance with a command from the user, or the storage server 18 via the network 1. It is also possible to output to and store (accumulate).

Next, referring to the flowchart in FIG. 123, the mixture ratio is calculated from the information input by the client computer 27 for designating the image and the object, and the area information, which is executed by the mixture ratio calculation server 14. The processing of the mixing ratio calculation service of is described.

In step S1501, the client computer 27 outputs the information designating the image and the object and the area information to the mixture ratio calculation server 14. That is, as the information for specifying the image for which the user wants to calculate the mixture ratio, the image is a specific image, or the image ID for specifying the image, the information for specifying the object, and the area information are the mixture ratio calculation server 14. Is output to.

[0717] In step S1511, the mixture ratio calculation server 14 acquires the designated image. That is,
When an image is transmitted from the client computer 27, and when an image ID for designating the image is transmitted, the image corresponding to the image ID is transmitted to the network 1
Read and get from above.

In steps S1512 and S1521, the charging processing unit 14a of the mixing ratio calculation server 14 and the charging server 24 execute charging processing. Note that the charging process is the same as that in the separated service in FIGS. 118 and 120, and therefore its description is omitted.

[0719] In step S1513, the mixture ratio calculation server 14 specifies the object, and
A process of calculating the mixture ratio is executed based on the area information.
The process of calculating the mixing ratio is the same as the process described with reference to the flowchart of FIG. 81, and thus the description thereof will be omitted.

[0720] In step S1514, the mixture ratio calculation server 14 transmits the mixture ratio obtained in the process of step S1513 to the client computer 27.

[0721] In step S1502, the client computer 27 receives and stores the mixture ratio transmitted from the mixture ratio calculation server 14.

The client computer 27 stores the mixing ratio calculated by the mixing ratio calculating server 14 in the mixing ratio calculating server 14 itself or stores it via the network 1 in accordance with a command from the user. Server 18
It is also possible to output to and store (accumulate).

Next, referring to the flow chart of FIG. 124, the foreground component is executed from the foreground / background separation server 15, which is information input by the client computer 27 for designating an image and an object, area information, and a mixture ratio. The processing of the service for separating the image and the background component image will be described.

In step S1601, the client computer 27 outputs information designating an image and an object, area information, and mixture ratio information to the foreground / background separation server 15. That is, as information for specifying an image for which the user wants to separate the foreground and background, a specific image, or an image ID for specifying an image, information for specifying an object, area information, and information on a mixture ratio are foreground. It is output to the background separation server 15.

In step S1611, the foreground / background separation server 15 acquires the designated image. That is, when an image is transmitted from the client computer 27, and when an image ID for designating the image is transmitted, the image corresponding to the image ID is read and acquired from the network 1.

In steps S1612 and S1621, the charging processing unit 15a of the foreground / background separation server 15 and the charging server 24 execute the charging process. Note that the charging process is the same as that in the separated service in FIGS. 118 and 120, and therefore its description is omitted.

[0727] In step S1613, the foreground / background separation server 15 executes the foreground / background separation process based on the information designating the object, the area information, and the mixture ratio. Note that the processing for separating the foreground and the background is as shown in FIG.
Since the processing is the same as the processing described with reference to the flowchart of FIG.

[0728] In step S1614, the foreground / background separation server 15 attaches IDs to the foreground component image and the background component image obtained in the process of step S1613 and sends them to the client computer 27.

In step S1602, the client computer 27 receives and stores the foreground component image and the background component image transmitted from the foreground / background separation server 15.

The client computer 27 stores the foreground component image and the background component image transmitted by the foreground / background separation server 15 in the foreground / background separation server 15 itself in accordance with a command from the user, or Storage (storage) by outputting to the storage server 18 via the network 1.
You can choose to let them do it.

Next, with reference to the flowchart in FIG. 125, designation is made from the information for designating the image input from the client computer 27, the motion vector, and the motion blur adjustment amount, which are executed by the motion blur adjustment server 16. The processing of the service for adjusting the motion blur of the generated image will be described.

In step S1701, the client computer 27 outputs the information designating the image, the motion vector, and the information on the motion blur adjustment amount to the motion blur adjustment server 16. That is, as the information for specifying the image for which the user wants to adjust the motion blur, a specific image, or an image ID for specifying the image, information for specifying the object, a motion vector, and information for the motion blur adjustment amount. Is output to the motion blur adjustment server 16.

[0733] In step S1711, the motion-blur adjusting server 16 acquires the designated image. That is, when an image is transmitted from the client computer 27, and when an image ID for designating the image is transmitted, the image corresponding to the image ID is read out and acquired from the network 1.

In steps S1712 and S1721, the billing processing unit 16a of the motion-blur adjusting server 16 and the billing server 24 execute billing processing. Note that the charging process is the same as that in the separated service in FIGS. 118 and 120, and therefore its description is omitted.

[0735] In step S1713, the motion-blur adjusting server 16 executes the process of adjusting the amount of motion-blur based on the motion vector and the information on the amount of motion-blur adjustment. The process of adjusting the amount of motion blur is the same as the process described with reference to the flowchart of FIG. 104, and thus the description thereof is omitted.

[0736] In step S1714, the motion-blur adjusting server 16 attaches an ID to the motion-blur adjusting image obtained in the process of step S1713 and transmits it to the client computer 27.

[0737] In step S1702, the client computer 27 receives and stores the motion blur adjustment image transmitted from the motion blur adjustment server 16.

The client computer 27 causes the motion-blur adjusting server 16 itself to store the motion-blur adjusting image transmitted from the motion-blur adjusting server 16 according to a command from the user, or the storage server 18 It is also possible to output to and store (accumulate).

Next, the detailed configuration of the encoding server 17 will be described with reference to FIG. The separation processing unit 2002 of the encoding server 17 inputs the input image (including the image that is input from the storage server 18 via the network 1 and the image ID that specifies the image is input). The image and the background component image are separated and output to the encoding unit 2001 together with the mixing ratio, the motion vector, and the position information. The separation processing unit 2002 is the same as the separation processing server (separation processing unit) 11 described with reference to FIG. 27, and is the same for the acquisition processing of the mixture ratio, the motion vector, and the position information. The description is omitted.

The encoding unit 2001 has a separation processing unit 2002.
The foreground component image and the background component image input by the above are output to the storage server 18 via the network 1 and stored therein, and converted into the stored position information of the storage server 18 on the network, that is, information such as URL. , Foreground component image position information and background component image position information. At this time, the encoding unit 2001 also outputs the mixing ratio, the motion vector, and the position information extracted when the foreground component image and the background component image are separated.

When the foreground component image and the background component image are converted into the foreground component image position information and the background component image position information by the encoding unit 2001, the charging processing unit 17a (FIGS. 16 and 17) A billing process is executed for the billing server 24 via the network 1. It should be noted that this charging process may be borne by a user who receives a combining service for generating a combined image by using the combining server 19 described later. On the contrary, the user who uses the encoding service bears the cost in advance so that the user does not need to pay the usage fee when using the combining service.

Next, with reference to the flowchart in FIG. 127, the processing of the encoding service executed by the encoding server 17 for encoding the image input from the client computer 27 will be described. In this description, the case where the user of the encoding service bears the usage fee will be described.

[0743] In step S1801, the client computer 27 outputs information designating an image to the encoding server 17. That is, as the information for specifying the image that the user wants to encode, a specific image, an image ID for specifying the image, or information for specifying the object is output to the encoding server 17.

[0744] In step S1811, the encoding server 17 acquires the designated image. That is, when an image is transmitted from the client computer 27, and when an image ID for designating the image is transmitted, the image corresponding to the image ID is read out and acquired from the network 1.

In steps S1812 and S1821, the charging processing unit 17a of the encoding server 17 and the charging server 24 execute the charging process. Note that the charging process is the same as that in the separated service in FIGS. 118 and 120, and therefore its description is omitted.

[0746] In step S1813, the separation processing section 2002 of the encoding server 17 executes image separation processing. The image separation process is the same as the process of step S1013 in the flowchart of FIG. 117, and thus description thereof will be omitted.

[0747] In step S1814, the encoding server 17 outputs the foreground component image and the background component image obtained in the process of step S1813 to the accumulation server 18 for storage (accumulation). In step S1831, the storage server 18 stores the transmitted foreground component image and background component image.

[0748] In step S1815, the encoding server 17 adds the motion vector and the position information to the foreground component image position information and the background component position information generated by the encoding, and transmits them to the client computer 27.

[0749] In step S1802, the client computer 27 sends the foreground component image position information, the background component image position information, transmitted from the encoding server 17,
The motion vector and the position information are received and stored.

[0750] When the input image is separated and encoded, the encoding unit 2001 encodes only the difference data when encoding an image similar to the already encoded image. Code of the image being displayed (image position information)
It may be attached to and output. For example, FIG.
In the case of synthesizing an image as shown in (1), the foreground component image 1, the foreground component image 2, and the encoding information of the first image including the encoding information of the mixture ratio 1, the foreground component image 1, and the foreground component image 3 are combined. , And when the encoded information of the second image composed of the encoded information of the mixture ratio 2 is combined, the foreground component image 1 is included in the information of any image, and therefore, when combining, The coding information of the foreground component image 1 of one image may be omitted, and as a result, the compression rate can be improved as much as the information of the foreground component image 1 is deleted as compared with the case of simply combining. .

As a result, when the first image and the second image as shown in FIG. 128 are accumulated, when the first image is accumulated first, the second image is the difference. Therefore, only the foreground component image 3 and the coded information of the mixture ratio 2 need to be stored. Therefore, in the case where the coded information of the same image is accumulated over a plurality of times, the compression rate is improved as the number of accumulated images increases.

The mixture ratio, motion vector, and position information encoded by the encoding server 17 may be information designated by the user as shown in FIG. 129. Further, as shown in FIG. 129, as the image to be encoded, the foreground component image and the background component image corresponding to the image ID designated by the user may be read from the accumulation server 18 and encoded. In this case, the encoding server 17 does not have to be provided with the separation processing unit 2002.

Although the image ID is used as the information for designating the image in this specification, the image position information may be used instead of the image ID.

Next, with reference to the flowchart in FIG. 130, the information executed by the synthesizing server 19 for designating the images A and B input from the client computer 27, the motion vector, the mixture ratio, the position information, and The processing of the service that synthesizes the designated images A and B from the motion blur adjustment amount will be described.

[0755] In step S1901, the client computer 27 outputs the information designating the images A and B, the motion vector, the mixture ratio, the position information, and the motion blur adjustment amount information to the synthesizing server 19. That is, as the information for specifying the images A and B that the user wants to combine, a specific image, or images A-ID and B-ID (the above-mentioned encoded image) that specify the images A and B are specified. Position information), the motion vector, the mixture ratio, the position information, and the motion blur adjustment amount information are output to the synthesis server 19.

[0756] In step S1911, the combining server 16 acquires the specified image. That is, when an image is transmitted from the client computer 27 and an image ID (which may be the above-mentioned encoded image position information) designating the image is transmitted, the image
The image corresponding to the ID is read and acquired from the network 1.

[0757] In steps S1912 and S1921, the charging processing unit 19a of the synthesizing server 19 and the charging server 2
4 executes a billing process. Regarding the billing process,
In FIGS. 118 and 120, the processing is the same as in the case of the separated service, and therefore its explanation is omitted. In addition, this billing process can be omitted if the user who received the coding service uses the above-described coding server 16 and bears the charge. On the contrary, instead of the user who receives the encoding service, the user who receives the combining service may bear the burden.

[0758] In step S1913, the combining server 19 executes the processing of combining the images A and B based on the motion vector, the mixture ratio, the position information, and the information on the motion blur adjustment amount.

In step S1914, the composition server 19 assigns an ID to the composite image (A + B) obtained in the processing of step S1913, and adds the ID to the client computer 2
Send to 7.

[0760] In step S1902, the client computer 27 receives and stores the combined image (A + B) transmitted from the combining server 19.

The client computer 27 stores the composite image (A + B) transmitted by the composition server 19 in the composition server 19 itself or accumulates it via the network 1 in accordance with a command from the user. It is also possible to output it to the server 18 and store (store) it.

By the way, the synthesizing server 20 is capable of synthesizing a plurality of images as described above. At this time, by using the motion blur adjustment amount as a key, the motion blur of the synthetic image is made. Can be added to generate an encrypted image. FIG. 131 shows a motion blur addition unit for encryption 20 provided for causing the synthesis server 20 to generate an encrypted image.
21 shows the configuration of No. 21.

The input information processing unit 2031 of the encryption motion blur adding unit 2021 inputs the input encrypted signal to be encrypted to the imaging unit 2032, the encryption key information to the motion blur creating unit 2033, Also, image selection information for selecting an image (background component image) to be combined with the encrypted signal as a foreground component image is output to the combining server 20.

The imaging unit 2032 is the input information processing unit 20.
When the encrypted signal input from 31 is not an image signal, the signal is converted into an image signal and the motion blur adding unit 20
To 33. That is, the encrypted signal is
Since the condition is that it is an image signal, a signal that is not an image signal is imaged in order to correspond to the processing.

The motion blur generation unit 2033 generates a motion blur adjustment amount based on the information such as the speed and direction input from the input information processing unit 2031 and adds the motion blur to the image signal input from the imaging unit 2032. And output to the synthesis server 20. The synthesizing server 20 acquires a background component image based on the image selection information input from the input information processing unit 2031 and further acquires the background component image using the image input from the motion blur adding unit 2033 as a foreground component image. And a composite image is generated and displayed. At this time, the image selection information designating the background component image may be the background component image itself, the background component image position information, or the background component image ID.

Next, referring to FIG. 132, the composition server 2
A cryptographic motion blur removing unit 2041 that decrypts the composite image encrypted by the cryptographic motion blur adding unit 2021 provided in 0 to convert the composite image into the original signal will be described. The encryption motion blur adding unit 202 shown in FIGS. 131 and 132
1 and the encryption motion blur removing unit 2041 may be considered as a functional block diagram of software built in the client computer 27, or may be considered as a block diagram of hardware. Further, the encryption motion blur adding unit 2021 and the encryption motion blur removing unit 2041 may be configured as a dedicated server on the network 1.

The separation server 11 separates the encrypted composite image into a foreground component image and a background component image, and outputs the foreground component image with motion blur added to the input information processing unit 2051.

The input information processing unit 2051 inputs the encrypted foreground component image input from the separation processing server 11 and the speed and direction information as a key for decrypting the encrypted foreground component image. Then, it is output to the motion blur removing unit 2052. The speed and the direction of the key are set when the image is two-dimensionally displayed in the x direction and the y direction.

The motion blur removing unit 2052 generates a motion blur amount based on the speed and direction information input from the input information processing unit 2051, and adds a motion blur adding unit for encryption to the encrypted foreground component image. A motion blur addition process that is the reverse of the motion blur addition process performed by 2021 is performed, and the encrypted foreground component image is decoded and output to the signal conversion unit 2053. When the encrypted signal is not an image, the signal converting unit 2053 converts the image signal input from the motion blur removing unit 2052 into the original signal and outputs the original signal.

That is, the motion blur creating unit 2033 described above.
(FIG. 131) and the motion blur removing unit 2052 substantially perform the same processing as the motion blur adding unit 806 of FIG. 97, and use the motion blur adjustment amount as an encryption key to perform mutual processing. The reverse motion blur addition process is executed. However, of the motion blur addition processing in the x direction or the y direction, which will be described later, the gain up processing is different from the motion blur addition processing that is executed first.

Here, the principle of encrypting an image signal by adding motion blur will be described. For example, as shown in FIG. 133, when the subject moves in the direction of the arrow, when the image is taken by the sensor 76a such as a CCD, a mixed region (covered background region and uncovered background region) is formed before and after the moving direction. Occurs as motion blur of the captured image (see FIG. 31 for details).
See). FIG. 134 shows an example of this phenomenon, in which an object such as that shown in FIG.
In the case of imaging with a, when the subject is moving in the left-right direction in the figure, the region of motion blur spreads depending on the speed of the subject.
Furthermore, it is shown that the colors to be imaged are diffused.
That is, when the subject is moving in the left-right direction at the speed v, it is assumed that an image as shown in FIG. 134 (B) is captured. At this time, assuming that the areas imaged when the subject is stationary are areas a0 to a0 ′, FIG.
Areas (a) to (a) in which the subject 4 (B) is imaged
1 ′, the areas of the original positions a0 to a0 ′ become lighter in color, and the areas a1 to a0 and the areas a1 to be motion blurred become
The color spreads from 0'to a1 '. Similarly, when the subject is moving at a speed of 2v (twice the speed v),
As shown in FIG. 4 (C), a region a2 that is further blurred
It is shown that the color spreads to the areas a0 to a0 and the areas a0 'to a2'. Further, when the object moves at a speed of 3v, as shown in FIG. 134 (D), the object spreads to areas a3 to a0 and areas a0 'to a3' which are blurred, and when the object moves at a speed of 4v, it moves to FIG. As shown in (E), the regions a4 to a0 and the region a that are out of motion
It spreads from 0'to a4 ', and the color becomes light as a whole. That is, since the individual pixel values output from the sensor 76a are the result of integrating a certain portion of the object to be imaged having a spatial spread with respect to the shutter time, the pixel values integrated as a whole are the same. As the speed of the subject increases, the color spreads spatially, resulting in a lighter color image. Therefore, as the color becomes lighter, the area spreads, and the motion blur area becomes larger, it becomes difficult to read the subject. Here, the color indicates the legibility, and the darker the color, the greater the legibility, and the lighter the color, the lower the legibility.

The encryption by the motion blur adjustment utilizes this property, and the motion blur in the two-dimensional direction, which cannot occur in the real world, is generated in the image for encryption. That is, as shown in FIG. 135, an image in which a black circle-shaped subject is imaged in a stationary state is shown in the uppermost leftmost column of the matrix. From this state,
For example, when motion blurring in the vertical direction is added, a black circle-shaped subject becomes an image in which motion blurring occurs in the vertical direction as shown in the top row of the middle row. Further, when the motion blur is generated in the horizontal direction, an image in which the motion blur is generated in the up, down, left, and right directions of the subject is displayed, as shown in the middle row of the middle row.

In this state, when motion (speed) in the left-right direction is further increased to add motion blur, an image in which the motion-blur region is further expanded in the left-right direction as shown in the bottom row of the middle row is obtained. Become. When motion blur is further generated in this image in the vertical direction, the motion-blur region of the black circle-shaped subject expands as shown in the lowermost row in the rightmost column, and the color becomes lighter as a whole. As a result, the possibility of reading the subject is reduced, and the image itself can be encrypted.

Next, with reference to the flowchart in FIG. 136, an encryption process using the motion-blur adjustment amount of the encryption motion-blur adding unit 2021 will be described. Note that, in the following description, as shown in FIG. 137, an example of encrypting an image in which a subject including 5 × 5 pixels is captured will be described.
Here, in FIG. 137, the pixel values a to y of each pixel of 5 × 5 pixels are used, and the vertical direction is y and the horizontal direction is x.
, And the time axis is described as time t.

[0775] In step S2001, the input information processing unit 2031 determines whether or not the encrypted signal is input, repeats the processing until input, and when it is determined that the input signal is input, the processing is Proceed to S2002.

In step S2002, the input information processing unit 2031 converts the input encrypted signal into the image forming unit 2
Output to 032. In step S2003, the imaging unit 2032 determines whether the input encrypted signal is an image signal. For example, when it is determined that the encrypted signal is not an image signal, the imaging unit 2032 converts the encrypted signal into an image signal and outputs the image signal to the motion blur adding unit 2033 in step S2004. When it is determined in step S2003 that the encrypted information is an image signal, the imaging unit 2032 outputs the input encrypted signal as it is to the motion blur adding unit 2033.

[0777] In step S2005, the input signal processing unit 2031 determines whether or not the key speed and direction information has been input, repeats the processing until the key information is input, and the speed and direction key is input. Then, the process proceeds to step S2006.

In step S2006, the motion blur adding section 2033 encrypts the input image signal in the x direction (adds motion blur).

Here, a concrete method of generating pixel values when a subject is encrypted by motion blur adjustment will be described with reference to FIGS. 137 to 149.

Here, as shown in FIG. 137, a method of encrypting the pixels a to e in the lowermost stage by causing motion blur in the x-direction will be described. At this time, if the motion amount v indicating the speed of the key is 5 (the number of virtual divisions is 5),
The lowermost pixel indicated by 38 is shown as in FIG.
That is, since the pixel value in each time direction is divided into 5,
/ 5 = a0 = a1 = a2 = a3 = a4, b / 5 = b0 =
b1 = b2 = b3 = b4, c / 5 = c0 = c1 = c2 =
c3 = c4, d / 5 = d0 = d1 = d2 = d3 = d4,
The relationship of e / 5 = e0 = e1 = e2 = e3 = e4 is satisfied. Here, the pixel value in the upper part of FIG. 139 is the pixel value at the earlier time.

Motion in the x direction with respect to the subject (in this case,
(Right direction in the figure), the arrangement of pixel values is slid at predetermined time intervals, and as a result, the arrangement shown in FIG. 140 is obtained. That is, at the timing of the movement start, the pixel values a0 to e0 are at the original positions, at the next timing, the pixel values a1 to e1 slide rightward by one pixel, and at the next timing, the pixel values a2 to e2. e2 slides one pixel further to the right, and at the next timing, pixel values a3 to e3 slide one pixel further to the right, and at the next timing,
The pixel values a4 to e4 are arranged such that the pixel values have moved in accordance with the movement of the subject, such as sliding one pixel further to the right.

Also, each pixel value on the xy plane is as shown in FIG.
It is said that the pixel values shown by are added in the time direction. However, for example, the leftmost column or the rightmost column has only the pixel values a0 and e4, and the pixel value may be very small.
When performing the same processing for the direction as well, a gain-up processing is performed so that the value does not become a very small value. An example in which this gain-up processing is performed is shown in FIG.

Here, a0 * = 5 × a0, b0 * = (5
/ 2) × b0, a0 * = (5/2) × a1, c0 * = (5 /
3) × c0, b1 * = (5/3) × b1, a2 * = (5 /
3) × a2, d0 * = (5/4) × d0, c1 * = (5 /
4) × c1, b2 * = (5/4) × b2, a3 * = (5 /
4) × a3, e1 * = (5/4) × e1, d2 * = (5 /
4) × d2, c3 * = (5/4) × c3, b4 * = (5 /
4) × b4, e2 * = (5/3) × e2, d3 * = (5 /
3) × d3, c4 * = (5/3) × c4, e3 * = (5 /
2) × e3, d4 * = (5/2) × d4, and e4 *
= 5 * e4. That is, the gain is adjusted so that the added value of each pixel becomes the pixel value of one pixel. As a result, when the pixels a to e shown in FIG. 138 are encrypted in the x direction under the condition that the motion amount v is 5 (when motion blur is added), the pixels ax to dx shown in FIG. 142 are obtained. '(Encoded), the number of horizontal pixels of the subject is 5
It will be increased from 9 to 9. Here, the pixel value is ax
= Ax *, bx = (b0 *) + (a1 *), cx = (c
0 *) + (b1 *) + (a2 *), dx = (d0 *) +
(C1 *) + (b2 *) + (a3 *), ex = (e0)
+ (D1) + (c2) + (b3) + (a4), ax '=
(E1 *) + (d2 *) + (c3 *) + (b4 *), b
x ′ = (e2 *) + (d3 *) + (c4 *), cx ′ =
(E3 *) + (d4 *) and ex = ex *.

[0784] The above-described processing is performed as shown in FIG.
If all x of 5 pixels are encrypted in the x direction,
The pixel value as shown in FIG. 143 will be obtained.
That is, the pixels ax to yx, and the pixels ax 'to dx', fx 'to ix', kx 'to nx', px 'to s.
Pixels of x ′ and ux ′ to xx ′ are obtained, and motion blur occurs in the x direction, so that spread occurs in the x direction, and 9 pixels are obtained each.

Now, let us return to the description of the flowchart in FIG. 136.

In step S2007, the motion blur adding unit 2033 encrypts the image signal encoded in the x direction in the y direction.

Here, as shown in FIG. 144, as shown in FIG.
The pixels ax, fx, kx, px, ux in the rightmost column shown by
A method of encrypting by causing motion blur in the y direction will be described. At this time, if the amount of movement v indicating the speed serving as a key is 5 (the number of virtual divisions is 5), the pixels in the rightmost column shown in FIG. 143 are shown in FIG. 144. That is, since the pixel value in each time direction is divided into 5, ax / 5 = ax0 =
ax1 = ax2 = ax3 = ax4, fx / 5 = fx0 =
fx1 = fx2 = fx3 = fx4, kx / 5 = kx0 =
kx1 = kx2 = kx3 = kx4, px / 5 = px0 =
px1 = px2 = px3 = px4, ux / 5 = ux0 =
The relationship of ux1 = ux2 = ux3 = ux4 is satisfied. Here, the pixels in the upper part of FIG. 145 have pixel values at earlier times.

When the subject is moved in the y direction, the pixel value arrangement is slid at predetermined time intervals, and as a result, the arrangement shown in FIG. 146 is obtained. That is, at the timing when the movement starts, the pixel value a
x0, fx0, kx0, px0, ux0 are original positions, and at the next timing, pixel values ax1, fx1, kx
1, px1, ux1 slides one pixel to the right, and at the next timing, pixel values ax2, fx2, kx
2, px2, ux2 slides one pixel further to the right, and at the next timing, pixel values ax3, fx
3, kx3, px3, and ux3 slide rightward by one pixel, and at the next timing, the pixel value ax
4, fx4, kx4, px4, and ux4 are arranged so that the pixel values have moved in accordance with the movement of the subject, such as sliding one pixel further to the right.

Now, let us return to the description of the flowchart in FIG. 136.

[0790] In step S2008, the composition server 19 combines the background component image to be combined with the encrypted image (foreground component image). For example, when a background component image (image consisting of one stage of pixels in the x direction) composed of pixel values B0 to B9 arranged in the y direction as shown in FIG. 147 is synthesized, as shown in FIG. The value obtained by adding the pixel values becomes the pixel value. That is, the pixel value of the image encrypted in the xy direction (with motion blur added) is combined as the pixel value of the foreground component image, and the pixel value of the combined image is combined as the pixel value of the background component image. As a result, pixel values A, F, K, P, U, A as shown in FIG.
y ′, Fy ′, Ky ′, Py ′ are obtained, and each pixel value is pixel value A = ax0 + B0 × 4/5, pixel value F
= Fx0 + ax0 + B1 × 3/5, pixel value K = kx0 +
fx1 + ax2 + B2 × 2/5, pixel value P = px0 + k
x1 + fx2 + ax3 + B3 × 1/5, pixel value U = ux
0 + px1 + kx2 + fx3 + ax4, pixel value Ay ′ =
B5 × 1/5 + ux1 + px2 + kx3 + fx4, pixel value Fy ′ = B6 × 2/5 + ux2 + px3 + kx4, pixel value Ky ′ = B7 × 3/5 + ux3 + px4, and
The pixel value Py ′ = B8 × 4/5 + ux4.

By executing these processes for all y directions, a composite image having the encrypted foreground component image as the background component image as shown in FIG. 150 is generated. That is, the input 5 × 5 pixel image is 9 × 9
Pixels (pixels A to Y, pixels Ax to Dx, pixels Fx to Ix, pixels Kx to Nx, pixels Px to Sx, pixels Ux
Through Xx, pixels Ay ′ through Ty ′, pixels Ax ′ through Dx ′,
Pixels Fx ′ to Ix ′, pixels Kx ′ to Nx ′, and pixels Px ′ to Sx ′) are converted into images.

Regarding the decryption processing, in the encryption motion blur removing unit 2041, the encryption motion blur adding unit 20 is used.
Since the motion blur addition process, which is the reverse of the process of 21, is executed, the description thereof will be omitted.

Also, in the above-mentioned step S2006, when the encryption in the x direction is performed, the gain-up process is performed and then the encryption in the y direction is performed. Therefore, decoding is performed in the y direction, then the gain is reduced, and then x
Need to decrypt in the direction. Also, the order of the encryption processing in the y direction and the x direction may be exchanged, but since the gain-up processing is performed for the direction to be encrypted first, the decryption must also correspond to the encrypted order. There is.

Next, referring to the flowchart of FIG. 151, the motion blur adding unit for encryption 2021 shown in FIG. 131 will be described.
An encryption service by the composition server 19 provided with will be described. This process is performed when the client computer 27-1 connected to the network 1 transmits an encrypted signal to the synthesis server 19, encrypts the encrypted signal, and transmits the encrypted signal to the client computer 27-2. Processing. Also, the client computer 2
In FIG. 7, it is assumed that the hardware having the image separation processing function of the separation processing server 11 having the encryption motion blur removing unit 2041 is provided or software is installed.

[0795] In step S2101, the client computer 27-1 has information to be encrypted (encoded signal), speed and direction information to be an encryption key,
And image selection information (information to select the background component image)
Is transmitted to the synthesis server 19.

[0796] In step S2111, the encryption motion blur adding unit 2021 of the combining server 19 encrypts the information (encrypted signal) to be encrypted based on the encryption key input from the client computer 27-1. Then, the encryption process for synthesizing the selected background component image is executed. The encryption process has been described with reference to the flowchart of FIG.

[0797] In step S2112, the composition server 19 transmits to the client computer 27-1, the image which has been encrypted and combined by adding motion blur.

[0798] In step S2102, the client computer 27-1 displays the composite image received from the composition server 19 to determine whether or not the image is desired by the user, and determines that the image is the desired image. If done,
In step S2103, the composition server 19 is notified that the image is the desired image. In step S2113, the composition server 19 determines whether or not the image is the desired image. For example, in this case, in step S2103, the composition server 19 has been notified that the image is the desired image. Therefore, the process is step S2114. Proceed to.

[0799] In step S2114, the charging processing unit 19a of the combining server 19 executes the charging processing together with the charging server 24. Note that the charging process is shown in FIG.
In FIG. 120, the processing is the same as in the case of the separated service, and therefore its explanation is omitted.

In step S2115, the composition server 19 sends the encrypted composite image to the client computer 27-1. In Step S2104,
The client computer 27-1 receives the encrypted composite image and sends it to the client computer 27-2 as it is.

[0801] In step S2141, the client computer 27-2 receives the encrypted composite image. In step S2142, the client computer 27-2 determines whether or not the key is input, and repeats the processing until the encryption key is input. In step S2142, when the speed and direction information is input as the encryption key, in step S2143, the motion blur removing processing unit 2041 executes the motion blur process based on the input speed and direction. Step S2
At 144, the image with motion blur removed is displayed.

If it is determined in step S2102 that the image is not the desired image, the synthesis server 19 is notified in step S2015 that the image is not the desired image, and the process returns to step S2101. Further, as a result of this processing, it is determined in step S2114 that the image is not the desired image, so the processing of the composition server 19 returns to the processing of step S2111.

In other words, by this processing, when the user of the client computer 27-2 correctly inputs the key of the speed and direction designated by the user of the client computer 27-1, the encrypted image is correctly decrypted. The displayed image is displayed. Also, a decryption service can be provided by the same system as the above-mentioned encryption service.

Next, referring to FIG. 152, the correction server 2
0 will be described.

The separation processing unit 11 of the correction server 20 regards the input image (may be the image ID, in the case of the image designated by the image ID, the image obtained by reading the corresponding image from the network 1) as the foreground component image. The background component image is separated, and the foreground component image is output to the motion-blur adjusting unit 2101 and the background component image is output to the combining unit. The motion-blur adjusting unit 2101 performs motion-blur adjustment on the input foreground component image with a specified motion-blur amount (adjusts the degree of correction), and outputs the image to the combining unit 2101. The synthesizing unit 2101 synthesizes the foreground component image that has undergone motion blur adjustment and the input background component image, and outputs it as a corrected image.

For example, it is assumed that the image as shown in FIG. 153 (A) is input to the correction server 20. That is, as shown in the right part of FIG. 153 (A), when the foreground moves in the arrow direction on the background, motion blur occurs in the traveling direction of the foreground and the opposite direction. The part of this motion blur is
As shown in the left part of FIG. 153 (A), the mixed region generated in the front part in the moving direction is a CB (Covered Ba).
ckground), and the mixed area generated in the rear part in the moving direction is UB (Uncovered Background). Note that FIG.
In the left part of FIG. 3 (A), since the time axis t is set in the vertical direction, the relationship between the accumulation state of the pixel value on the pixel and the passage of time is shown along with the movement. The separation processing unit 11 separates this input image into a foreground and a background as shown in FIG. 153 (B). At this time, the mixed area of the input image is also extracted at the same time.

The motion blur adjusting unit 2101 is shown in FIG.
The motion blur of the foreground component image as shown in (B) is adjusted to generate, for example, the foreground component image as shown in FIG. 153 (C). That is, in the present case, the motion blur is reduced (the CB and UB portions are reduced). The motion blur adjustment amount for adjusting the motion blur may be input by the user while repeating the operation several times, or may be adjusted by the motion blur adjusting unit 2101 to a predetermined value. .

The synthesis unit 2102 synthesizes the foreground component image adjusted as shown in FIG. 153 and the input background component image, and the motion blur adjusted foreground as shown in FIG. 153 (D). The component image is combined with the background component image and output.

When it is desired to change the background component image to another background component image different from that of the input image, the separated background component image is not output to the synthesizing unit 2102, and the background component image to be changed is selected. Input to the synthesis unit 2102. The correction server 20 includes the separation processing unit 11, the motion blur adjustment unit 2101, and the synthesis unit 2102 as the separation processing server 11, the motion blur adjustment server 16, and the motion blur adjustment server 16 on the network 1.
Alternatively, the composition server 19 may be used instead.

Next, referring to the flowchart in FIG. 154, the processing of the correction service executed by the correction server 20 to correct the image input from the client computer 27 will be described.

[0811] In step S2201, the client computer 27 outputs information designating an image to the correction server 20. That is, as the information for specifying the image that the user wants to correct, a specific image or an image ID specifying the image is output to the correction server 20.

In step S2211, the correction server 20 acquires the designated image to be corrected and the specified background component image, and the separation processing section 11 separates the image to be corrected into the foreground component image and the background component image. . That is,
When an image is transmitted from the client computer 27, and when an image ID for designating the image is transmitted, the image corresponding to the image ID is transmitted to the network 1
Read and get from above. Further, the separation processing unit 11 separates the acquired image into a foreground component image and a background component image.

[0813] In steps S2212 and S2221, the billing processing unit 20a of the correction server 20 and the billing server 2 are processed.
4 executes a billing process. Regarding the billing process,
In FIGS. 118 and 120, the processing is the same as in the case of the separated service, and therefore its explanation is omitted.

In step S2213, the motion-blur adjusting unit 2101 of the correction server 20 executes the motion-blur adjusting process of the foreground component image. Since the motion blur adjustment process is the same as the process described with reference to the flowchart in FIG. 104, the description thereof will be omitted.

In step S2214, the combining unit 21
Reference numeral 02 synthesizes the foreground component image whose motion blur has been adjusted with the designated background component image. In step S2215, the modification server 20 transmits the composite image obtained in the process of step S2214, that is, the modified image to the client computer 27.

[0816] In step S2202, the client computer 27 receives and stores the corrected image transmitted from the correction server 20.

The client computer 27 causes the correction server 20 itself to store the image corrected by the correction server 20 in accordance with a command from the user, or
It is also possible to output to the storage server 18 via a network and store (store) it.

Next, with reference to the flowchart of FIG. 155, the processing of the image purchasing service executed by the purchasing server 21 for purchasing the image designated by the client computer 27 will be described.

In step S2301, the client computer 27 outputs the information designating the image desired to be purchased to the purchase server 21. That is, the image ID designating the image is output to the purchase server 21 as the information designating the image the user wants to purchase.

[0820] In step S2311, the purchase server 21 acquires an image desired to be purchased. That is, the image ID transmitted from the client computer 27
The image corresponding to is read and acquired from the network 1.

[0821] In steps S2312 and S2321, the charging processing unit 21a of the purchase server 21 and the charging server 2
4 executes a billing process. Regarding the billing process,
In FIGS. 118 and 120, the processing is the same as in the case of the separated service, and therefore its explanation is omitted.

[0821] In step S2313, the purchase server 21 transmits the image acquired in the process of step S2311 to the client computer 27.

[0823] In step S2302, the client computer 27 receives and stores the image transmitted from the purchase server 21.

The client computer 27 stores the image purchased by the purchase server 21 in the purchase server 21 itself or outputs it to the storage server 18 for storage (accumulation) in response to a command from the user. You can also In addition, another client computer 2
By making it possible to send to 7, it is also possible to present an image, for example. Further, another user may separate the foreground component image, the background component image, the synthesized image, or the synthesized image by the separation processing service, the synthesis service, or the correction service as described above, or
It is also possible to purchase modified images that have been modified.

Next, with reference to the flowchart in FIG. 156, the processing of the image sale service for selling the image designated by the client computer 27, which is executed by the sale server 22, will be described.

[0826] In step S2401, the client computer 27 outputs information designating an image desired to be sold to the purchase server 21. That is, the image that the user wants to sell is output to the selling server 22.

[0827] In step S2411, the selling server 22 acquires an image desired to be sold. That is, the image transmitted from the client computer 27 is acquired.

[0828] In step S2422, the selling server 22 sets an appropriate price for the image desired to be sold. The price may be set in advance by the user, may be set in an auction format on the network 1, or may be an image of a person. Alternatively, a method of setting depending on whether or not the person being imaged is a predetermined celebrity, for example.

[0829] In steps S2413 and S2431, the charging processing unit 22a of the selling server 22 and the charging server 2
4 executes a sale charge process.

[0830] Here, the sale charge processing will be described with reference to the flowchart in FIG. 157. Note that the actual sale and charge processing is executed by the sale server 22 and the charge server 24, but the information necessary for various processing is also output from the client computer 27, so here,
The processing of the client computer 27 will also be described.

[0831] In step S2501, ID information for identifying a user (a user who requests the sale of an image) is transmitted to the sale server 22 via the network 1.

[0832] In step S2511, the selling server 22 transmits the sale amount and the ID for identifying the selling server 22 to the charging server 24 based on the ID information transmitted from the client computer 27.

[0833] In step S2521, the billing server 24 sends the financial server 26 having the provider account to the financial server 26 based on the transmitted ID for identifying the selling server 22.
Financial server 25 of customer account corresponding to purchase amount ID information
Request a transfer to.

[0834] In step S2531, the provider financial server 26 transfers the amount corresponding to the sale amount from the provider's account to the customer financial server 25 in which the customer's account is opened.

Now, return to the description of the flowchart in FIG. 156.

[0836] In step S2424, the sale server 22 notifies the client computer 27 that the sale has been completed. In step S2402, the client computer 27 receives the sale completion notification.

The sale server 22 may store the image sold by the user in the sale server 21 itself, or may output it to the storage server 18 and store (store) it. . Further, when the price is set in the auction format as described above, it may be transmitted to the client computer 27 of the successful bidder.

Next, referring to FIG. 158, the search server 2
3 will be described.

The search server 23 receives search conditions (image request signal) input from the client computer 27 or the like.
Based on the above, the image captured by the camera terminal device 1 connected to the network 1 is searched and the requested image is output. The search conditions include time, season, weather, area, place, or subject.

The control unit 2161 of the search server 23 controls the overall operation of the search server 23. Database 21
62 is a search server 2 connected to the network 1.
3 Cameras recognized by each camera terminal device 28
Corresponding to the ID, position data of each camera terminal device 28 (obtained by the GPS 76b provided in the camera terminal device 28), weather data, and data 2162b such as data of a subject being imaged are stored as a database. ing. The contents of this database 2162 are updated by the control unit 2161 controlling the communication unit 2165 at predetermined time intervals to acquire from each camera terminal device 28 via the network 1.

The storage unit 2163 stores images acquired from the camera terminal device 28 on the network 1 from the communication unit 2165, or stores information necessary for processing various images.

[0842] The request information generation unit 2164 organizes the search conditions input from the client computer 27 or the like on the network 1 to generate the conditions that can actually be searched on the database 2162. That is, for example, when the season is input as the search condition, each camera terminal device 2
Therefore, the season can be specified by the position data of 8 and the time information calculated by the time calculation unit 2166. Therefore, the request information generation unit 2164, for example,
When a search condition such as "spring" is input, position data of latitude and longitude on the earth in which the season is "spring" from the current time is generated. The control unit 2161 controls the communication unit 21.
The image corresponding to “spring” is acquired by controlling 65 to read the captured image of the camera terminal device 28 having the camera ID corresponding to this position data from the network 1.

The separation processing unit 2167 acquires the image for search included in the read image by the separation processing.
The separation processing unit 2167 has the same function as the separation processing unit 11.

Next, referring to the flowchart in FIG. 159, the processing of the search service executed by the search server 23 to search for an image based on the search condition input from the client computer 27 will be described.

[0845] In step S2601, the client computer 27 outputs the search condition to the search server 23. In step S2611, the search server 23
Receives the search condition through the communication unit 2165.

[0846] In steps S2612 and S2631, the charging processing unit 23a of the search server 23 and the charging server 2
4 executes a billing process. Regarding the billing process,
In FIGS. 118 and 120, the processing is the same as in the case of the separated service, and therefore its explanation is omitted. Further, the charging process in the processes of steps S2612 and S2631 is a charging process related to the fee for executing the search.

[0847] In step S2613, the search server 23 searches for an image corresponding to the search condition, and in step S2613.
Call the corresponding image at 2614 for the corresponding image. In step S2641, the camera terminal device 28
Sends the currently captured image to the search server 23.

That is, for example, as shown in FIG. 160, it is assumed that the client computers 27-1 to 27-5, the search server 23, and the camera terminal devices 28-1 to 28-5 are connected to the network 1. To do. At this time, the client computer 27-2
It is operated by the user, and as a search condition, "human",
When the “car” and the “building” are transmitted in the process of step S2611, the search server 23 causes the search server 23 to perform the process of step S2.
In 613, the objects of “human”, “car”, and “building” are searched as search conditions in the database 2162. That is, in the case of FIG. 160, the search server 23
Indicates that the car 2172 has the camera terminal device 28 with the camera ID = 1.
1, the human 2183 is the camera terminal device 28 with the camera ID = 2
-2 and the building 2211 are imaged by the camera terminal device 28-5 having the camera ID = 5, and in step S2614, the camera terminal device 28-5 is searched.
Get more images.

In step S2515, the search server 23 determines whether or not the called image is separated, that is, whether or not an image (object) other than the desired condition is included.

In the case of FIG. 160, the camera terminal device 28-1
In the image sent from the cloud 217 other than the search condition
2 is included, and the image transmitted from the camera terminal device 28-2 includes the house 2181 other than the search condition. Therefore, these images are separated from the image corresponding to the search condition. Since it does not exist, the process proceeds to step S2616.

[0851] In step S2616, the separation processing unit 2167 executes the separation processing. The separation process is the same as the process of step S1013 in the flowchart of FIG. 117, and thus the description thereof will be omitted.

[0852] In step S2617, the called images are combined and transmitted to the client computer 27. In step S2602, the client computer 27 acquires the image transmitted from the search server 23. In step S2603, the client computer 27 determines whether the received image is a desired image. As shown in FIG. 160, in the case of the image displayed on the display 27a-1, "human", "car", and "building" as search conditions are displayed as human 2182, car 2172, and building 2211. Since this is a desired image, the search server is notified of the desired image in step S2604.

[0853] In step S2618, the search server 23 determines whether the notification transmitted from the client computer 27 is a desired image. Figure 1
In the case of 60, the image is a desired image, and thus the process proceeds to step S2619.

[0854] In steps S2619 and S2632, the charging processing unit 23a of the search server 23 and the charging server 2
4 executes a billing process. Regarding the billing process,
In FIGS. 118 and 120, the processing is the same as in the case of the separated service, and therefore its explanation is omitted. Further, the charging process in the processes of steps S2619 and S2632 is a charging process related to the fee for transmitting the searched image. Also, step S2515.
In the case where all of the images are search conditions, the process of step S2616 is skipped.

[0855] In step S2603, for example, "home", "cloud", and "face" are specified by the client computer 27-4 as search conditions, but they are displayed on the display 27a-4. As in the image, when only the house 2181 and the cloud 2071 are displayed and the face 2201 is not displayed, it means that the image is not the desired image, so the process is step S260.
In step 5, the fact that the image is not the desired image is transmitted to the search server 23, and the process ends.

At this time, the search server 23 determines in step S
In 2618, since it is notified that the image is not the desired image, the processing ends.

[0857] In this case, although the fee for the search process is paid, the fee is not paid for transmitting the searched image.

According to the above, by adjusting the motion blur of the foreground component image, a clear foreground component image with no motion blur can be obtained, and the foreground component image obtained by adjusting the motion blur can be used as the background component. By synthesizing with the image, it is possible to generate a synthetic image that is more natural in appearance. As a result, the image can be freely modified by adjusting the motion blur of the foreground component image and synthesizing it with the background component image.

As shown in FIGS. 2 and 3, the recording medium on which the program for performing the signal processing of the present invention is recorded, in which the program is distributed separately from the computer in order to provide the program to the user. Magnetic disk 61,
91 (including flexible disk), optical disk 6
2,92 (CD-ROM (Compact Disc-Read Only Memory), DV
D (including Digital Versatile Disc)), magneto-optical discs 63 and 93 (including MD (Mini-Disc) (trademark)), or package media including semiconductor memories 64 and 94, as well as a computer ROM 42 and 72 in which a program is recorded and storage units 48 and 7 which are provided to the user in a state of being pre-installed in
It is composed of a hard disk and the like included in 8.

[0860] In this specification, the steps for writing the program to be recorded on the recording medium are not limited to the processing performed in time series according to the order described, but are not necessarily performed in time series. It also includes processing executed in parallel or individually.

Further, in this specification, the system means
It represents the entire apparatus composed of a plurality of devices.

[0862]

According to the first communication device and method and the program of the present invention, the user's request information is input,
Foreground consisting of foreground object components forming a foreground object of image data consisting of pixel values determined for each pixel based on the request information and according to the amount of light forming the image that is temporally integrated. Estimate the mixture ratio that indicates the mixture ratio of the component image and the background component image that is the background object component that constitutes the background object, and based on the mixture ratio, separate the image data into the foreground component image and the background component image. Then, the separated foreground component image and the arbitrary background component image or the separated background component image and the arbitrary foreground component image are combined at a predetermined mixing ratio to generate a combined image, and the combined image is combined. Is output.

According to the communication system and method and the program of the present invention, the first communication device inputs the request information of the user, transmits the input request information to the second transmitting device, and the second transmitting device The second communication device receives the composite image transmitted from the communication device, the second communication device receives the request information transmitted from the first transmission device, and based on the request information, for each pixel and temporally. A foreground component image of a foreground object component of a foreground object, and a background object component of a background object of the image data of pixel values determined according to the amount of light constituting the image A mixing ratio indicating a mixing ratio with the background component image is estimated, image data is separated into a foreground component image and a background component image based on the mixing ratio, and the separated foreground component image and an arbitrary Jing component image, or the background component image separated, and any foreground component image are synthesized at a predetermined mixing ratio to generate a combined image, synthesized composite image of the first
I sent it to the communication device.

According to the second communication device and method and the program of the present invention, the request information of the user is input, the request information is transmitted to the other communication device, and the composition is transmitted from the other communication device. Received images.

According to the third communication device and method and the program of the present invention, the request information transmitted from another communication device is received, and based on the request information, for each pixel,
In addition, a foreground component image of a foreground object component of a foreground object, and a background of a background object of image data of pixel values determined according to the amount of light forming the image integrated temporally. Estimate a mixture ratio indicating a mixture ratio with a background component image composed of object components, and based on the mixture ratio, separate image data into a foreground component image and a background component image, and separate the foreground component image and an arbitrary The background component image of or the separated background component image and the arbitrary foreground component image are combined at a predetermined mixing ratio to generate a combined image, and the combined image is transmitted to another communication device. I chose

[0866] In any case, as a result, it is possible to remove the motion blur and obtain a clear image, and the image with the motion blur adjusted and the background image are combined to obtain a more natural image. It is possible to freely modify the above.

[Brief description of drawings]

FIG. 1 is a diagram showing a configuration of an embodiment of an image processing system to which the present invention is applied.

2 is a diagram showing a configuration of a separation processing server in FIG.

FIG. 3 is a diagram showing a configuration of a camera terminal device of FIG.

FIG. 4 is a diagram showing a function of a separation processing server in FIG.

5 is a diagram showing functions of the separation processing server of FIG. 1. FIG.

FIG. 6 is a diagram showing functions of the motion detection server of FIG. 1.

FIG. 7 is a diagram showing functions of the motion detection server of FIG. 1.

FIG. 8 is a diagram showing functions of the area specifying server of FIG. 1.

FIG. 9 is a diagram showing functions of the area specifying server of FIG. 1.

10 is a diagram showing a function of a mixture ratio calculation server in FIG.

FIG. 11 is a diagram showing a function of a mixture ratio calculation server of FIG. 1.

12 is a diagram showing functions of the foreground / background separation processing server in FIG.

13 is a diagram showing the function of the foreground / background separation processing server of FIG.

FIG. 14 is a diagram showing the function of the motion blur adjustment server of FIG. 1.

FIG. 15 is a diagram showing the function of the motion blur adjustment server of FIG. 1.

16 is a diagram showing functions of the encoding server of FIG. 1. FIG.

17 is a diagram showing functions of the encoding server of FIG. 1. FIG.

FIG. 18 is a diagram showing functions of the storage server of FIG. 1.

19 is a diagram showing functions of the storage server in FIG. 1. FIG.

20 is a diagram showing a function of the composition server in FIG. 1. FIG.

21 is a diagram showing the function of the composition server in FIG. 1. FIG.

22 is a diagram showing the function of the correction server of FIG. 1. FIG.

FIG. 23 is a diagram showing a function of a correction server of FIG. 1.

FIG. 24 is a diagram showing functions of the purchase server of FIG. 1.

FIG. 25 is a diagram showing functions of the sale server of FIG. 1.

26 is a diagram showing functions of the search server of FIG. 1. FIG.

FIG. 27 is a block diagram showing a separation processing server.

FIG. 28 is a diagram illustrating imaging by a sensor.

FIG. 29 is a diagram illustrating an arrangement of pixels.

FIG. 30 is a diagram illustrating the operation of the detection element.

FIG. 31 shows an object corresponding to a moving foreground,
It is a figure explaining the image obtained by picturizing the object corresponding to the still background.

FIG. 32 is a diagram illustrating a background area, a foreground area, a mixed area, a covered background area, and an uncovered background area.

FIG. 33 is a model diagram in which the pixel values of pixels arranged in a row adjacent to each other in the image obtained by imaging an object corresponding to a stationary foreground and an object corresponding to a stationary background are expanded in the time direction. Is.

FIG. 34 is a model diagram in which the pixel value is expanded in the time direction and the period corresponding to the shutter time is divided.

FIG. 35 is a model diagram in which the pixel value is expanded in the time direction and the period corresponding to the shutter time is divided.

FIG. 36 is a model diagram in which the pixel value is expanded in the time direction and the period corresponding to the shutter time is divided.

FIG. 37 is a diagram showing an example in which pixels of a foreground area, a background area, and a mixed area are extracted.

FIG. 38 is a diagram showing the correspondence between pixels and a model in which pixel values are expanded in the time direction.

FIG. 39 is a model diagram in which the pixel value is expanded in the time direction and the period corresponding to the shutter time is divided.

FIG. 40 is a model diagram in which the pixel value is expanded in the time direction and the period corresponding to the shutter time is divided.

FIG. 41 is a model diagram in which the pixel value is expanded in the time direction and the period corresponding to the shutter time is divided.

FIG. 42 is a model diagram in which the pixel value is expanded in the time direction and the period corresponding to the shutter time is divided.

FIG. 43 is a model diagram in which the pixel value is expanded in the time direction and the period corresponding to the shutter time is divided.

FIG. 44 is a flowchart illustrating a process of adjusting the amount of motion blur.

45 is a block diagram showing an example of a configuration of a region specifying unit 103. FIG.

FIG. 46 is a diagram illustrating an image when an object corresponding to the foreground is moving.

FIG. 47 is a model diagram in which the pixel value is expanded in the time direction and the period corresponding to the shutter time is divided.

FIG. 48 is a model diagram in which a pixel value is expanded in the time direction and a period corresponding to a shutter time is divided.

FIG. 49 is a model diagram in which the pixel value is expanded in the time direction and the period corresponding to the shutter time is divided.

FIG. 50 is a diagram illustrating conditions for region determination.

FIG. 51 is a diagram showing an example of a result of specifying a region by the region specifying unit 103.

FIG. 52 is a diagram showing an example of a result of specifying a region by the region specifying unit 103.

FIG. 53 is a flowchart illustrating a region specifying process.

FIG. 54 is a block diagram showing another example of the configuration of the area specifying unit 103.

FIG. 55 is a model diagram in which the pixel value is expanded in the time direction and the period corresponding to the shutter time is divided.

FIG. 56 is a diagram showing an example of a background image.

FIG. 57 is a block diagram showing a configuration of a binary object image extraction unit 302.

FIG. 58 is a diagram illustrating calculation of a correlation value.

FIG. 59 is a diagram illustrating calculation of a correlation value.

FIG. 60 is a diagram showing an example of a binary object image.

FIG. 61 is a block diagram showing a configuration of a time change detection unit 303.

[Fig. 62] Fig. 62 is a diagram for explaining the determination of the region determination unit 342.

FIG. 63 is a diagram showing an example of determination by the time change detection unit 303.

FIG. 64 is a flowchart illustrating a region specifying process of a region determining unit 103.

FIG. 65 is a flowchart illustrating details of area determination processing.

66 is a block diagram showing still another configuration of the area specifying unit 103. FIG.

[Fig. 67] Fig. 67 is a block diagram illustrating a configuration of a robustization unit 361.

[Fig. 68] Fig. 68 is a diagram for describing motion compensation of the motion compensation unit 381.

[Fig. 69] Fig. 69 is a diagram for describing motion compensation of the motion compensation unit 381.

FIG. 70 is a flowchart illustrating a region specifying process.

FIG. 71 is a flowchart illustrating details of robust processing.

72 is a block diagram showing an example of the configuration of a mixing ratio calculation unit 104. FIG.

FIG. 73 is a diagram showing an example of an ideal mixing ratio α.

FIG. 74 is a model diagram in which pixel values are expanded in the time direction and a period corresponding to shutter time is divided.

FIG. 75 is a model diagram in which the pixel value is expanded in the time direction and the period corresponding to the shutter time is divided.

[Fig. 76] Fig. 76 is a diagram for describing approximation using the correlation of foreground components.

77 is a diagram for explaining the relationship between C, N and P. FIG.

FIG. 78 is a block diagram showing a configuration of an estimated mixing ratio processing unit 401.

FIG. 79 is a diagram showing an example of an estimated mixing ratio.

FIG. 80 is a block diagram showing another configuration of the mixing ratio calculation unit 104.

FIG. 81 is a flowchart illustrating a process of calculating a mixture ratio.

FIG. 82 is a flowchart illustrating a process of calculating an estimated mixing ratio.

FIG. 83 is a diagram illustrating a straight line approximating the mixture ratio α.

FIG. 84 is a diagram illustrating a plane that approximates the mixture ratio α.

[Fig. 85] Fig. 85 is a diagram for describing correspondence between pixels in a plurality of frames when a mixture ratio α is calculated.

FIG. 86 is a block diagram showing another configuration of the mixing ratio estimation processing unit 401.

FIG. 87 is a diagram showing an example of an estimated mixing ratio.

[Fig. 88] Fig. 88 is a flowchart illustrating a process of estimating a mixture ratio using a model corresponding to a covered background area.

89 is a block diagram showing an example of the configuration of the foreground / background separator 105. FIG.

FIG. 90 is a diagram showing an input image, a foreground component image, and a background component image.

FIG. 91 is a model diagram in which a pixel value is expanded in the time direction and a period corresponding to a shutter time is divided.

92 is a model diagram in which the pixel value is expanded in the time direction and the period corresponding to the shutter time is divided. FIG.

FIG. 93 is a model diagram in which pixel values are expanded in the time direction and a period corresponding to shutter time is divided.

FIG. 94 is a block diagram showing an example of a configuration of a separating unit 601.

[Fig. 95] Fig. 95 is a diagram illustrating an example of the separated foreground component image and background component image.

[Fig. 96] Fig. 96 is a flowchart illustrating a process of separating a foreground and a background.

97 is a block diagram showing an example of a configuration of a motion blur adjusting unit 106. FIG.

FIG. 98 is a diagram illustrating a processing unit.

[Fig. 99] A pixel value of a foreground component image is expanded in the time direction,
It is a model figure which divided the period corresponding to shutter time.

[Fig. 100] Fig. 100 is a model diagram in which the pixel values of the foreground component image are expanded in the time direction and the period corresponding to the shutter time is divided.

101 is a model diagram in which the pixel values of the foreground component image are expanded in the time direction and the period corresponding to the shutter time is divided. FIG.

FIG. 102 is a model diagram in which the pixel values of the foreground component image are expanded in the time direction and the period corresponding to the shutter time is divided.

103 is a diagram showing another configuration of the motion-blur adjusting unit 106. FIG.

FIG. 104 is a flowchart illustrating a process of adjusting the amount of motion blur included in the foreground component image by the motion blur adjustment unit.

FIG. 105 is a block diagram showing another example of the configuration of the motion-blur adjusting unit 106.

[Fig. 106] Fig. 106 is a diagram illustrating an example of a model that specifies correspondence between pixel values and foreground components.

FIG. 107 is a diagram illustrating calculation of a foreground component.

FIG. 108 is a diagram illustrating calculation of a foreground component.

[Fig. 109] Fig. 109 is a flowchart illustrating a process of removing motion blur of a foreground.

110 is a block diagram showing another configuration of the function of the separation processing server. FIG.

111 is a diagram showing a configuration of a combining unit 1001. FIG.

FIG. 112 is a block diagram showing still another configuration of the function of the separation processing server.

113 is a block diagram showing a configuration of a mixture ratio calculation unit 1101. FIG.

114 is a block diagram showing a configuration of a foreground / background separator 1102. FIG.

FIG. 115 is a block diagram showing still another configuration of the function of the separation processing server.

FIG. 116 is a diagram showing a structure of a combining unit 1201.

FIG. 117 is a flowchart illustrating a separation service.

FIG. 118 is a flowchart illustrating a charging process.

FIG. 119 is a diagram illustrating a billing process.

FIG. 120 is a flowchart illustrating another example of the charging process.

FIG. 121 is a flowchart illustrating a motion detection service.

FIG. 122 is a flowchart illustrating an area specifying service.

FIG. 123 is a flowchart illustrating a mixture ratio calculation service.

FIG. 124 is a flowchart illustrating a foreground / background separation service.

FIG. 125 is a flowchart illustrating a motion blur adjustment service.

FIG. 126 is a diagram illustrating an encoding server.

FIG. 127 is a flowchart illustrating an encoding service.

[Fig. 128] Fig. 128 is a diagram for describing compression capacity during compression by encoding processing.

FIG. 129 is a diagram illustrating another example of the encoding server.

FIG. 130 is a flowchart illustrating a composition service.

[Fig. 131] Fig. 131 is a diagram for describing a cryptographic motion blur adding unit.

[Fig. 132] Fig. 132 is a diagram for describing a cryptographic motion blur removal unit.

[Fig. 133] Fig. 133 is a diagram for describing a process of adding motion blur for encryption.

[Fig. 134] Fig. 134 is a diagram for describing a process of adding motion blur for encryption.

[Fig. 135] Fig. 135 is a diagram for describing a process of adding motion blur for encryption.

FIG. 136 is a flowchart illustrating an encryption process.

[FIG. 137] FIG. 137 is a diagram illustrating a process of adding motion blur for encryption.

[FIG. 138] FIG. 138 is a diagram for describing processing for adding encryption motion blur.

[FIG. 139] FIG. 139 is a diagram illustrating a process of adding encryption motion blur.

[Fig. 140] Fig. 140 is a diagram for describing processing for adding motion blur for encryption.

[Fig. 141] Fig. 141 is a diagram for describing processing for adding encryption motion blur.

[Fig. 142] Fig. 142 is a diagram for describing a process of adding motion blur for encryption.

[FIG. 143] FIG. 143 is a diagram illustrating a process of adding encryption motion blur.

[Fig. 144] Fig. 144 is a diagram for describing a process of adding motion blur for encryption.

[FIG. 145] FIG. 145 is a diagram illustrating a process of adding encryption motion blur.

[Fig. 146] Fig. 146 is a diagram for describing processing for adding encryption motion blur.

[FIG. 147] FIG. 147 is a diagram illustrating a process of adding encryption motion blur.

[FIG. 148] FIG. 148 is a diagram for describing processing for adding encryption motion blur.

[FIG. 149] FIG. 149 is a diagram illustrating a process of adding encryption motion blur.

[Fig. 150] Fig. 150 is a diagram for describing a process of adding motion blur for encryption.

FIG. 151 is a flowchart illustrating a cryptographic service.

FIG. 152 is a diagram illustrating a correction server.

FIG. 153 is a diagram illustrating a correction process.

FIG. 154 is a flowchart illustrating a correction service.

FIG. 155 is a flowchart illustrating a purchase service.

FIG. 156 is a flowchart illustrating a sale service.

FIG. 157 is a flowchart illustrating a sale charge process.

FIG. 158 is a diagram illustrating a search server.

FIG. 159 is a flowchart illustrating a search service.

FIG. 160 is a diagram illustrating a search service.

[Explanation of symbols]

11 separation processing server, 12 motion detection server, 1
3 area identification server, 14 mixture ratio calculation server, 15
Foreground / background separation server, 16 motion blur adjustment server,
17 encoding server, 18 storage server, 19
Synthesis server, 20 modification server, 21 purchasing server,
22 sale server, 23 search server, 24 billing server, 25, 26 financial server, 27 client computer, 28 camera terminal device, 41
CPU, 42 ROM, 43 RAM, 46 input section,
47 output unit, 48 storage unit, 49 communication unit, 71
CPU, 72 ROM, 73 RAM, 76 input section,
76a sensor, 76b GPS, 77 output section,
78 storage unit, 79 communication unit, 91 magnetic disk,
92 optical disk, 93 magneto-optical disk, 94
Semiconductor memory, 101 object extraction unit, 1
02 motion detection unit, 103 area identification unit, 104
Mixing ratio calculator, 105 Foreground / background separator, 106
Motion blur adjusting unit, 107 selecting unit, 201 frame memory, 202-1 to 202-4 static / moving determining unit,
203-1 to 203-3 area determination unit, 204
Determination flag storage frame memory, 205 combining unit,
206 determination flag storage frame memory, 301 background image generation unit, 302 binary object image extraction unit, 303 temporal change detection unit, 321 correlation value calculation unit, 322 threshold processing unit, 341 frame memory, 342 region determination unit, 361 Robustization department,
381 motion compensation unit, 382 switch, 383-
1 to 383-N frame memory, 384-1 to 384-N weighting unit, 385 integrating unit, 401
Estimated mixing ratio processing unit, 402 Estimated mixing ratio processing unit, 40
3 Mixing ratio determination unit, 421 frame memory, 42
2 frame memory, 423 mixture ratio calculation unit, 44
1 selection unit, 442 estimated mixture ratio processing unit, 443
Estimated mixture ratio processing unit, 444 selection unit, 501 delay circuit, 502 addition unit, 503 arithmetic unit, 6
01 separating unit, 602 switch, 603 combining unit, 604 switch, 605 combining unit, 621
Frame memory, 622 separation processing block, 6
23 frame memory, 631 uncovered area processing section, 632 covered area processing section, 633 combining section, 634 combining section, 801 processing unit determining section,
802 Modeling unit, 803 Equation generating unit, 804
Addition unit, 805 operation unit, 806 motion blur addition unit, 807 selection unit, 821 selection unit, 901
Processing unit determination unit, 902 modeling unit, 903
Equation generation unit, 904 calculation unit, 905 correction unit,
906 motion blur adding section, 907 selecting section, 100
1 synthesis unit, 1021 background component generation unit, 1022
Mixed area image synthesis unit, 1023 image synthesis unit, 11
01 mixture ratio calculation unit, 1102 foreground / background separation unit,
1121 selecting unit, 1201 combining unit, 1221
Selection unit, 2001 encoding unit, 2002 separation processing unit,
2021 motion blur addition unit for encryption, 2031 input information processing unit, 2032 imaging unit, 2033 motion blur addition unit, 2041 motion blur removal unit for encryption, 205
1 input information processing unit, 2052 motion blur removing unit, 2
053 signal conversion unit, 2101 motion blur adjustment unit, 2
102 synthesis part, 2161 control part, 2162 database, 2163 storage part, 2164 request information generation part, 2165 communication part, 2166 time calculation part, 2167 separation processing part

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ identification code FI theme code (reference) H04N 7/18 H04N 7/18 V (72) Inventor Junichi Ishibashi 6-735 Kitashinagawa, Shinagawa-ku, Tokyo No. Sony Corporation (72) Inventor Takashi Sawawa 6-735 Kita-Shinagawa, Shinagawa-ku, Tokyo Sony Corporation (72) Takahiro Nagano 6-35 Kita-Shinagawa, Shinagawa-ku, Tokyo Sony Corporation (72) Inventor Toru Miyake 6-735 Kitashinagawa, Shinagawa-ku, Tokyo Sony Corporation (72) Inventor Seiji Wada 6-735 Kitashinagawa, Shinagawa-ku, Tokyo Sony -In-house F-term (reference) 5B057 AA20 BA02 CA08 CA16 CB12 CB16 CC01 CE03 CE08 5C023 AA06 AA07 AA11 AA26 BA01 BA11 BA17 CA01 DA04 5C054 AA02 AA05 CA04 CC03 DA09 EA01 EA05 ED11 EJ07 FC13 GA14 G02 FE12 GA01 B01 HA14 5C064 BB01 BB05 BC01 BC16 BC23 BC25 BD02 BD08

Claims (30)

[Claims] 1. Request information input means for inputting request information of a user, and it is determined for each pixel on the basis of the request information and according to an amount of light forming an image which is temporally integrated. Mixing for estimating a mixing ratio indicating a mixing ratio of the foreground component image including the foreground object component forming the foreground object and the background component image including the background object component forming the background object, of the image data including the pixel value A ratio estimating means, a separating means for separating image data into the foreground component image and the background component image based on the mixture ratio, the foreground component image separated by the separating means, and an arbitrary background component. An image or a synthesizing unit for synthesizing the background component image separated by the separating unit and an arbitrary foreground component image at a predetermined mixing ratio to generate a combined image. And an output unit that outputs the composite image combined by the combining unit. 2. The request information input means further inputs, in addition to the user's request information, a motion blur adjustment amount for adding motion blur to the foreground component image, and the motion information is added to the foreground component image. The image forming apparatus further includes a motion blur adding unit that adds motion blur corresponding to the blur adjustment amount, and the combining unit combines the foreground component image to which motion blur is added by the motion blur adding unit and the arbitrary background component image. The communication device according to claim 1, wherein the communication device generates a composite image by synthesizing at a predetermined mixing ratio. 3. The request information input unit further inputs the arbitrary background component image in addition to the user's request information, and the combining unit includes the foreground component image and the arbitrary background component image. The communication apparatus according to claim 1, wherein the communication ratio is combined with the mixture ratio estimated by the mixture ratio estimating unit to generate a combined image. 4. The charging device according to claim 1, further comprising a charging unit that executes a charging process according to the request information.
The communication device according to 1. 5. The billing means, in accordance with the request information, an identifier of the user, an identifier of the communication device, and
The communication device according to claim 4, wherein billing information including money amount information corresponding to the request information is generated, and billing processing is executed for the financial account of the user based on the billing information. . 6. A point storage unit for storing points for each user used for the billing process, wherein the billing unit, in accordance with the request information, the identifier of the user, the identifier of the communication device, And charging processing is executed by generating charging information including money amount information corresponding to the composite image, and subtracting the number of points stored in the point storage means by an amount corresponding to the money amount information. The communication device according to claim 4. 7. The communication according to claim 4, wherein the output unit outputs the composite image to a communication device of a user who has completed the billing process after the billing process is completed by the billing unit. apparatus. 8. The request information input means inputs the predetermined image data in addition to the user's request information, and the output means charges the combined image combined by the combining means with the charge. After the billing process is completed by means,
The communication device according to claim 4, wherein the communication device outputs the billing process to the communication device of the user. 9. A request information input step of inputting request information of a user, and it is determined for each pixel based on the request information and according to the amount of light forming an image integrated temporally. Mixing for estimating a mixing ratio indicating a mixing ratio of the foreground component image including the foreground object component forming the foreground object and the background component image including the background object component forming the background object, of the image data including the pixel value Ratio estimation step, based on the mixture ratio, the image data, the separation step of separating the foreground component image, the background component image, the foreground component image separated by the process of the separation step, any A background component image, or the background component image separated in the process of the separating step and an arbitrary foreground component image are combined at a predetermined mixing ratio, and combined. A communication method comprising: a synthesizing step of generating an image; and an output step of outputting the synthesized image synthesized by the processing of the synthesizing step. 10. A request information input control step of controlling input of request information from a user, and based on the request information, for each pixel and according to an amount of light forming an image integrated temporally. Of the foreground component image of the foreground object component of the foreground object and the background component image of the background object component of the background object of the image data of the pixel values determined by A mixture ratio estimation control step of controlling estimation, a separation control step of controlling separation of the image data into the foreground component image and the background component image based on the mixture ratio, and a process of the separation control step. The separated foreground component image and any background component image, or the background component image separated in the separation control step, and any foreground component A combination with an image at a predetermined mixture ratio, a combination control step for controlling the generation of the combined image, and an output control step for controlling the output of the combined image combined in the processing of the combining control step. A recording medium having a computer-readable program recorded thereon. 11. A request information input control step of controlling input of request information of a user, and based on the request information, for each pixel and according to an amount of light forming an image integrated temporally. Of the foreground component image of the foreground object component of the foreground object and the background component image of the background object component of the background object of the image data of the pixel values determined by A mixture ratio estimation control step of controlling estimation, a separation control step of controlling separation of the image data into the foreground component image and the background component image based on the mixture ratio, and a process of the separation control step. The separated foreground component image and any background component image, or the background component image separated in the separation control step, and any foreground component A combination of an image with a predetermined mixing ratio, a combination control step of controlling the generation of the combined image, and an output control step of controlling the output of the combined image combined in the processing of the combination control step are performed by a computer. The program to run. 12. A communication system comprising a first communication device and a second communication device, wherein the first communication device comprises request information input means for inputting request information of a user, and the request information input means. The second communication device includes: a request information transmitting unit that transmits the input request information to the second transmitting device; and a receiving unit that receives a composite image transmitted from the second communication device. Is a request information receiving unit that receives the request information transmitted from the first transmission device, and a light forming an image that is temporally integrated for each pixel based on the request information. Foreground component image of the foreground object component of the foreground object of the image data of the pixel value determined according to the amount, and the background object component of the background object A mixture ratio estimating means for estimating a mixture ratio indicating a mixture ratio with a background component image; a separation means for separating image data into the foreground component image and the background component image based on the mixture ratio; The foreground component image separated by the separating unit and an arbitrary background component image, or the background component image separated by the separating unit and an arbitrary foreground component image are combined at a predetermined mixing ratio and combined. A synthesizing unit for generating an image; and the synthetic image synthesized by the synthesizing unit as the first
And a composite image transmitting means for transmitting to the communication device. 13. The request information input means further inputs, in addition to the user's request information, a motion blur adjustment amount for adding motion blur to the foreground component image, and the second communication device sets the foreground The component image further comprises a motion blur adding means for adding motion blur corresponding to the motion blur adjustment amount, the combining means, the foreground component image to which motion blur is added by the motion blur adding means, The communication system according to claim 12, wherein the arbitrary background component image is combined with a predetermined mixing ratio to generate a combined image. 14. The request information input unit further inputs the arbitrary background component image in addition to the user's request information, and the combining unit includes the foreground component image and the arbitrary background component image. 13. The communication system according to claim 12, wherein is combined with the mixture ratio estimated by the mixture ratio estimating unit to generate a combined image. 15. The communication system according to claim 12, wherein the second communication device further includes a charging unit that executes a charging process according to the request information. 16. The billing means, according to the request information, an identifier of the user, an identifier of the second communication device,
The billing information including the amount information corresponding to the request information is generated, and the billing process is executed for the financial account of the user based on the billing information. Communications system. 17. The second communication device further comprises point storage means for storing points for each user used for the billing process, wherein the billing means responds to the request information of the user. Billing information including an identifier, an identifier of the second communication device, and amount information corresponding to the composite image is generated, and the number of points stored in the point storage unit is subtracted by an amount corresponding to the amount information. 16. The communication system according to claim 15, wherein the billing process is executed by the above. 18. The composite image transmitting means transmits the composite image to the first communication device of the user who has completed the charging processing after the charging processing is completed by the charging means. Item 16. The communication system according to Item 15. 19. The request information input means inputs the predetermined image data in addition to the user's request information, and the output means charges the combined image combined by the combining means with the charge. After the billing process is completed by means,
16. The communication system according to claim 15, wherein the billing process is output to the first communication device of the user. 20. A communication method of a communication system comprising a first communication device and a second communication device, wherein the communication method of the first communication device comprises a request information input step of inputting request information of a user. A request information transmitting step of transmitting the request information input in the processing of the request information inputting step to the second transmitting device, and a receiving step of receiving a composite image transmitted from the second communication device. Including a request information receiving step of receiving the request information transmitted from the first transmitting device, and a communication method of the second communication device, for each pixel, and a time based on the request information. Of the foreground component image of the foreground object component of the foreground object in the image data of the pixel value determined according to the amount of light that forms the image integrated A mixture ratio estimating step of estimating a mixture ratio indicating a mixture ratio with a background component image composed of a background object component forming an object, and image data based on the mixture ratio, the foreground component image and the background component. A separation step of separating into images, the foreground component image separated by the processing of the separation step, an arbitrary background component image, or the background component image separated by the processing of the separation step, and an arbitrary foreground component A combining step of generating a combined image by combining the image with a predetermined mixture ratio; and a combined image transmitting step of transmitting the combined image combined in the processing of the combining step to the first communication device. A communication method for a communication system, comprising: 21. A program for controlling a communication system including a first communication device and a second communication device, wherein the program controlling the first communication device controls input of user request information. A request information input control step; a request information transmission control step of controlling transmission of the request information input in the processing of the request information input step to the second transmission device; and a transmission from the second communication device. A reception control step of controlling reception of an incoming composite image, the program controlling the second communication device is a request information reception controlling reception of the request information transmitted from the first transmission device. A control step, based on the request information, for each pixel, and of image data consisting of pixel values determined according to the amount of light constituting the image that is temporally integrated, A mixture ratio estimation control step of controlling estimation of a mixture ratio indicating a mixture ratio of a foreground component image including a foreground object component forming a background object and a background component image including a background object component forming a background object; A separation control step of controlling separation of the image data into the foreground component image and the background component image based on a mixture ratio, the foreground component image separated by the processing of the separation control step, and an arbitrary background A component image, or a background component image separated by the process of the separation step, and a combination of a foreground component image with a predetermined mixture ratio, a combination control step for controlling the generation of a composite image, A composite image transmission control step of controlling transmission of the composite image composited in the processing of the composite control step to the first communication device. A recording medium having a computer-readable program recorded thereon. 22. A request for controlling input of request information of a user to a computer controlling a communication system including a first communication device and a second communication device, the computer controlling the first communication device. An information input control step; a request information transmission control step of controlling transmission of the request information input in the processing of the request information input step to the second transmission device; A reception control step for controlling the reception of the composite image, and a request information reception control for controlling the reception of the request information transmitted from the first transmission device by a computer controlling the second communication device. A control step, and an image data consisting of pixel values determined for each pixel based on the request information and according to the amount of light forming the image integrated temporally. A mixture ratio estimation control step for controlling the estimation of the mixture ratio indicating the mixture ratio of the foreground component image including the foreground object components forming the foreground object and the background component image forming the background object component forming the background object. And, based on the mixing ratio, the image data, the foreground component image, a separation control step for controlling the separation into the background component image, the foreground component image separated by the process of the separation control step, Synthesis of an arbitrary background component image or the background component image separated by the process of the separation step and an arbitrary foreground component image at a predetermined mixing ratio, and a synthesis control step for controlling generation of a synthetic image And a composite image transmission control step for controlling transmission of the composite image combined in the processing of the combination control step to the first communication device. A program that executes and. 23. Request information input means for inputting user request information, request information transmitting means for transmitting the request information input by the request information input means to another communication device, and the other communication device A communication device comprising: a reception unit that receives a composite image transmitted from the communication device. 24. A request information input step of inputting request information of a user, a request information transmitting step of transmitting the request information input in the processing of the request information input step to another communication device, and A receiving step of receiving a composite image transmitted from the communication device. 25. A request information input control step of controlling input of request information of a user, and controlling transmission of the request information whose input is controlled in the processing of the request information input control step to another communication device. A recording medium having a computer-readable program recorded thereon, comprising: a request information transmission control step; and a reception control step of controlling reception of a composite image transmitted from the other communication device. 26. A request information input control step of controlling input of request information of a user, and controlling transmission of the request information whose input is controlled in the processing of the request information input control step to another communication device. A program that causes a computer to execute a request information transmission control step and a reception control step that controls reception of a composite image transmitted from the other communication device. 27. Request information receiving means for receiving request information transmitted from another communication device, and light for forming an image integrated pixel by pixel and temporally based on the request information. The ratio of the mixture of the foreground component image composed of the foreground object component forming the foreground object and the background component image composed of the background object component forming the background object of the image data composed of pixel values determined according to the amount is shown. A mixture ratio estimation unit that estimates a mixture ratio, a separation unit that separates image data into the foreground component image and the background component image based on the mixture ratio, and the foreground component image that is separated by the separation unit. And an arbitrary background component image, or the background component image separated by the separating means and an arbitrary foreground component image are combined at a predetermined mixing ratio, and combined. A communication device comprising: a synthesizing unit for generating an image; and a synthetic image transmitting unit for transmitting the synthesized image synthesized by the synthesizing unit to the other communication device. 28. A request information receiving step of receiving the request information transmitted from another communication device, and a light forming an image integrated pixel by pixel and temporally based on the request information. The ratio of the mixture of the foreground component image consisting of the foreground object component forming the foreground object and the background component image forming the background object component forming the background object of the image data consisting of pixel values determined according to the A mixture ratio estimating step of estimating a mixture ratio shown, based on the mixture ratio, the image data, the separation step of separating the foreground component image, the background component image, the separation by the process of the separation step A foreground component image, an arbitrary background component image, or the background component image separated by the process of the separating step, and an arbitrary foreground component image , A combining step of generating a combined image by combining at a predetermined mixture ratio, and a combined image transmitting step of transmitting the combined image combined in the processing of the combining step to the other communication device. Communication method to do. 29. A request information reception control step of controlling reception of the request information transmitted from another communication device, and an image integrated temporally for each pixel based on the request information. Mixing of the foreground component image composed of the foreground object components forming the foreground object and the background component image composed of the background object components forming the background object, of the image data composed of pixel values determined according to the amount of light A mixture ratio estimation control step of controlling the estimation of a mixture ratio indicating the ratio of the image data, based on the mixture ratio, image data, a separation control step of controlling the separation of the foreground component image and the background component image, The foreground component image separated by the process of the separation control step and an arbitrary background component image, or the background image separated by the process of the separation control step. The background component image and an arbitrary foreground component image are combined at a predetermined mixing ratio, a combination control step of controlling the generation of the combined image, and the combination image combined in the processing of the combination control step. And a composite image transmission control step for controlling transmission to another communication device. 30. A request information reception control step of controlling reception of the request information transmitted from another communication device, and an image integrated temporally for each pixel based on the request information. Mixing of the foreground component image composed of the foreground object component forming the foreground object and the background component image composed of the background object component forming the background object, of the image data composed of pixel values determined according to the amount of light A mixture ratio estimation control step for controlling the estimation of a mixture ratio indicating the ratio of the image data, based on the mixture ratio, the image data, the separation control step for controlling the separation of the foreground component image and the background component image, The foreground component image separated by the process of the separation control step and an arbitrary background component image, or the background image separated by the process of the separation control step. A background component image and an arbitrary foreground component image are combined at a predetermined mixing ratio, a combination control step of controlling generation of a combined image, and the combined image combined in the processing of the combining control step. A program that causes a computer to execute a composite image transmission control step for controlling transmission to another communication device.