JP2000197044A - Image processing unit, its method and computer-readable memory - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing unit that can easily execute compositing of plural images and generate a composite image with satisfactory image quality, and to provide its method and a computer-readable memory. SOLUTION: This image processing unit extracts a characteristic of a background from coded data of at least one background image, extracts a characteristic of an object including statistic information of the image information from coded data of at least one object image, decodes the coded data of the background image to generate a background reproduction image, and generates the object reproduction image to decode the coded data of the object image. A correction device 216 corrects the object reproduction image, on the basis of the characteristics of the background and the object. An image synthesizer 217 composites the background reproduction image and the corrected object reproduction image.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to an image processing apparatus and method for synthesizing a plurality of images, and a computer readable memory.

[0002]

2. Description of the Related Art Conventionally, h.
H.261, MPEG-1, and MPEG-2 are known. These encoding systems are internationally standardized by the ITU and ISO, and are respectively defined by h. H.264 Recommendation, IS
It is documented as O11172, 13818. In addition, a motion image is encoded by adapting still image encoding (for example, JPEG encoding) to each frame.
n JPEG encoding is also known.

[0003] Hereinafter, referring to FIG.
An encoding system that encodes a moving image using EG-1 will be described.

FIG. 19 is a diagram showing the configuration of a conventional encoding system.

[0005] A video signal input from a TV camera 1001 is input to an input terminal 1003 of a moving picture coding apparatus 1002 and output to an A / D converter 1004. The video signal converted into a digital signal by the A / D converter 1004 is input to the block former 1005,
16 × 16 macroblocks are formed in order from the upper left to the lower right of the image based on the video signal. In MPEG-1, there are an I-frame for performing intra-frame coding, a P-frame for performing inter-frame coding from past frames, and a B-frame for performing inter-frame coding from past and future frames. The frame mode unit 1015 determines the modes of these frames. The frame mode is determined in consideration of the encoding bit rate, prevention of image quality deterioration due to accumulation of DCT calculation errors, image editing, and scene changes.

In an I-frame, the motion compensator 1006 does not operate and outputs 0. The subtractor 1007 subtracts the output of the motion compensator 1006 from the output of the block former 1005 and inputs the result to the DCT transformer 1008. The DCT transformer 1008 performs DCT transform on the input signal in units of 8 × 8 blocks, and converts the DCT-transformed signal into the quantizer 1.
In step 009, quantization is performed. Next, the quantized signal is one-dimensionally rearranged by the encoder 1010, and the code is determined based on the 0 run length and the value. The coded signal is output from the terminal 1011 and recorded on a storage medium or transmitted via a network or a line. Also, the quantizer 10
09 is inversely quantized by an inverse quantizer 1012, inverse DCT-transformed by an inverse DCT transformer 1013, and
At 14, the output is added to the output of the motion compensator 1006 and stored in the frame memory 1015 or 1016.

In the case of a P-frame, the motion compensator 1006 is operated, and the output of the block former 1005 is input to the motion compensator 1006, from the frame memory 1015 or 1016 to which the image of the immediately preceding frame is input. Perform motion compensation and output a motion vector and a predicted macroblock. A differentiator 1007 is a block former 1005
Of the prediction macroblock and the input from the
Input to converter 1008. DCT converter 1008 is:
The input signal is subjected to DCT transform, and the DCT-transformed signal is quantized by a quantizer 1009. The sign of the quantized signal is determined together with the motion vector by the encoder 1010 and output from the terminal 1011. Quantizer 1
The output of 009 is inversely quantized by an inverse quantizer 1012,
The inverse DCT transform is performed by the inverse DCT transformer 1013 and the adder 10
At 14, the output is added to the output of the motion compensator 1006 and stored in the frame memory 1015 or 1016.

In the B-frame, motion compensation is performed in the same manner as in the P frame. However, the motion compensator 1006 performs motion compensation from both the frame memories 1015 and 1016, generates a predicted macroblock, and performs encoding.

[0009]

However, in the above-mentioned conventional method of encoding an entire image, it is necessary to repeatedly transmit an image of a background portion or the like having no motion, and the code length is wasted. For example, only a person actually moves in a videophone or a video conference, and the background does not move. In an I-frame sent at regular intervals, the background image that is not moving is also sent, resulting in wasted code. FIG. 20 shows an example.

FIG. 20 shows a frame in which a person is facing a television camera in a room. This person 105
1 and the background 1050 are encoded the same in the same frame. Since there is no motion in the background 1050, little code is generated if motion compensation is performed, but it is coded when an I-frame is sent. For this reason, a code is repeatedly sent to a portion where there is no motion, which is wasteful. In addition, a sufficient code length cannot be obtained in an I-frame after the motion of the person 1051 is large and a large code length is generated in encoding. For this reason, in the I-frame, it is necessary to make the quantization coefficient rough, and there is a problem that the image quality of a background with no motion is reduced.

Therefore, it is conceivable to improve the coding efficiency by separately coding the background and the object as in MPEG-4. In this case, it is also possible to combine an object photographed in another place, so that, for example, as shown in FIG. 21, a frame in which another person 1052 is further combined with the frame in FIG. 20 can be configured.

[0012] However, an unnatural image may remain in an image (a person 1052) obtained by synthesis due to a color cast generated due to the characteristics of the photographing equipment, and the viewer may feel uncomfortable. For example, a person 1052 shoots with equipment showing a green cast,
On the other hand, when the person 1051 is photographed with a device exhibiting a tendency of red fogging, an image obtained by combining the two becomes remarkably conspicuous in color fog, and becomes very unnatural.

[0013] Also, an image obtained by combining images due to a difference in contrast caused by a difference in environment such as lighting conditions and characteristics of photographing equipment may also remain unnatural, giving the viewer a sense of discomfort. For example, when the person 1052 is photographed in sunlight and the person 1051 is photographed with artificial light, the contrast between the two is greatly different, which is very unnatural.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and has an image processing apparatus and method capable of easily executing a synthesis of a plurality of images and generating a synthesized image having a good image quality. To provide a computer readable memory.

[0015]

An image processing apparatus according to the present invention for achieving the above object has the following arrangement.
That is, an image processing apparatus for synthesizing a plurality of images, a background feature extracting means for extracting a background feature from encoded data of at least one background image, and a statistical information of image information from encoded data of at least one target image. Target feature extraction means for extracting a target feature including target information, background decoding means for decoding encoded data of the background image to generate a background reproduced image, and a target reproduced image for decoding encoded data of the target image A correction means for correcting the target reproduction image based on the background feature and the target characteristic, and a synthesis means for synthesizing the background reproduction image and the target reproduction image corrected by the correction means. And

Preferably, the target feature extraction means includes a calculation means for calculating a histogram based on the statistical information of the image information, and the correction means includes a correction method for the target image based on the histogram. To determine.

Preferably, the target feature extracting means extracts DC information of a block image included in the encoded data as statistical information of the image information.

Preferably, the target feature extracting means extracts low-frequency information of a block image included in the encoded data as statistical information of the image information.

Preferably, one or both of the background decoding means and the target decoding means are
Decoding means for decoding quantized data from the encoded data, inverse quantization means for calculating frequency domain data from the quantized data, and high-speed inverse discrete cosine transform means for calculating spatial domain data from the frequency domain data. The high-speed inverse discrete cosine transform unit includes an output unit that outputs a radix butterfly operation result of an arbitrary number of stages, and the target feature extraction unit uses the radix butterfly operation result of the arbitrary number of stages as low-frequency information of image information. Extract.

Preferably, the correction means includes time series adaptation means for gradually changing an input / output relationship between a signal input to the correction means and a signal output from the correction means according to a time series.

Preferably, the target feature extracting means calculates the maximum value and the minimum value of a pixel value from DC information or low frequency information of a block image included in the encoded data as statistical information of the image information. Extract.

Preferably, the target feature extracting means extracts, as statistical information of the image information, a variance and an average of pixel values from DC information or low frequency information of a block image included in the encoded data. I do. Preferably, the correction unit performs a linear transformation on the target image. Preferably, the correction unit performs a piecewise spline transformation on the target image.

Preferably, the correction means detects a presence or absence of a significant color bias from the target feature extracted by the target feature extraction means, and detects a color deviation based on a detection result of the detection means. Color correction means for correcting the bias. Preferably, the detecting means satisfies a condition that a difference absolute value of a variance from a difference absolute value of an average value between each color signal is equal to or less than a threshold value from statistical information included in the extracted target feature. And if the above condition is satisfied, the presence or absence of significant color bias in a specific area of the histogram based on the statistical information is detected. Preferably, the color correction means performs linear correction so that the maximum value of each color signal is equal. Also, preferably, the color correction means is:
No correction is performed for the blue signal.

Preferably, the correction means detects a significant difference in contrast between the target feature extracted by the target feature extraction means and the background feature extracted by the background feature extraction means. And a contrast correction means for correcting contrast based on the detection result of the detection means. Preferably, the detecting unit extracts a maximum value and a minimum value of a pixel value obtained from the target feature and the background feature, respectively, and the contrast correction unit determines an object having a different maximum value or the minimum value. In the image and the background image, the correction is performed so that the difference absolute value of the maximum pixel value and the difference absolute value of the minimum pixel value are reduced. Preferably, the detecting unit extracts a maximum value and a minimum value of a pixel value obtained from the target feature and the background feature, respectively, and the contrast correction unit determines whether the maximum value or the minimum value is The correction is performed so that the difference absolute value of the variance is reduced between the target image and the background image that are substantially equal.

An image processing method according to the present invention for achieving the above object has the following arrangement. That is, an image processing method for synthesizing a plurality of images, a background feature extracting step of extracting a background feature from encoded data of at least one background image, and a statistical process of image information from the encoded data of at least one target image. Feature extraction step of extracting a feature containing target information, a background decoding step of decoding encoded data of the background image to generate a background reproduction image, and a target reproduction image of decoding the encoded data of the target image , A correction step of correcting the target reproduction image based on the background feature and the target characteristic, and a synthesis step of synthesizing the background reproduction image and the target reproduction image corrected in the correction step. And

A computer readable memory according to the present invention for achieving the above object has the following configuration. That is,
A computer readable memory storing a program code of an image processing for synthesizing a plurality of images, the program code of a background feature extracting step of extracting a background feature from encoded data of at least one background image;
Program code for a target feature extraction step of extracting a target feature including statistical information of image information from encoded data of two target images, and a background decoding step of decoding the encoded data of the background image to generate a background reproduced image A program code of a target decoding step of generating a target playback image for decoding encoded data of the target image, and a correction step of correcting the target playback image based on the background feature and the target feature. And a program code for a combining step of combining the background reproduced image and the target reproduced image corrected in the correction step.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will be described below in detail with reference to the drawings. <First Embodiment> FIG. 1 is a block diagram showing the configuration of a moving image transmission system according to a first embodiment of the present invention.

In the first embodiment, encoded images that are encoded and transmitted at a plurality of locations in different photographing environments and encoded data stored in a storage medium such as a pre-database are stored in a database. Will be described by taking as an example a case where a host that manages the information is decrypted and synthesized, and transmitted to another terminal or network.

In FIG. 1, reference numeral 101 denotes a TV camera for shooting a moving image with a blue background (blue background). T
The V camera 101 may be any moving image input means such as a TV camera or another storage medium as a moving image input means. Here, it is assumed that the person 1052 in FIG. 13 is imaged. A TV camera 102 captures a moving image. This may be any moving image input means. Here, it is assumed that the background image 1050 and the person 1051 in FIG. 20 are imaged. A target extractor 103 extracts a person 1052 as a target image from the blue background. 1
05 is a target encoding unit 1 for encoding the extracted target image
05. Here, it is assumed that encoding by MPEG-4 is performed.

Reference numeral 104 denotes an encoder for encoding a moving image captured by the TV camera 102. Although the encoding method is not particularly limited, an example of encoding by MPEG-1 will be described here. Reference numerals 106 and 107 are transmitters for transmitting encoded data. 108 and 109 are communication lines. 11
Numerals 0 and 111 are receivers for receiving encoded data. 1
Reference numeral 16 denotes a storage device that stores encoded data that has been encoded in advance. Reference numeral 112 denotes a moving image editing apparatus according to the present invention. Reference numeral 113 denotes an encoder for encoding the result of editing performed by the moving image editing apparatus 112, and here, MP
A description will be given by taking coding by EG-1 as an example. Note that the encoding method used by the encoder 113 is not limited to this, but may be MPEG-4, MPEG-2, h. 263, etc., as long as it is a coding method for coding moving images. Reference numeral 114 denotes a transmitter for transmitting the encoded data obtained from the encoder 113. Reference numeral 115 denotes a communication network such as a public line or a broadcast wave.

In such a configuration, the TV camera 10
1 is a person 105 to be imaged against a blue background
2 is imaged. The target extractor 103 extracts a person 1052, which is a target image, from the input moving image. This is shown in FIGS.

FIG. 2 cuts out a person 1052 to be imaged in a rectangular area 1200. Subsequently, the image data of an area 1200 for extracting the blue back portion and generating the mask information 1201 as shown in FIG.
Is input to the target encoding unit 105. FIG. 4 shows an image obtained by the processing by the target encoding unit 105, the details of which will be described later.

The storage device 116 is, for example, a CD-ROM
And magnetic disk and tape storage devices, etc.
The data can be stored without being limited to a specific coding method. here,
In particular, it is assumed that the storage device is a storage device that stores encoded data composed of a sequence encoded by MPEG-4.
3 is stored.

Next, a detailed configuration of the target encoding unit 105 according to the first embodiment will be described with reference to FIG.

FIG. 5 is a block diagram showing a detailed configuration of the target encoding unit according to the first embodiment of the present invention.

Reference numerals 121 and 122 denote terminals. The terminal extractor 103 shown in FIG. 1 outputs terminal image data of a region 1200 of the target image to be encoded.
Enter 201. Reference numeral 123 denotes a mask memory for storing mask information 1201. 124 is mask information 1201
Is a mask encoder for encoding. 125 is the area 12
00 is a target memory for storing image data of 00. 126
Is an average value calculator for calculating the average value of the pixel values of the target image. A block forming unit 127 divides the target image into blocks in units of coding. Reference numeral 128 denotes a frame mode setting unit that appropriately selects an I, P, or B frame mode according to a predetermined cycle of a frame encoding mode.

Reference numeral 129 denotes a differentiator. 130 is DCT
(Discrete Cosine Transform) This is a DCT transformer for performing a transform. 131 is a DCT converter 1
30 is a quantizer which quantizes the output of 30. Reference numeral 132 denotes an encoder that arranges the quantization results one-dimensionally, assigns a code to the 0 run length and the value, and performs encoding. 133 synthesizes the encoded data generated by the mask encoder 124 and the encoder 132. Reference numeral 134 denotes a terminal, which finally outputs the generated encoded data. 135 is an inverse quantizer for performing inverse quantization. 136 is an inverse DCT converter that performs an inverse DCT transform. 137 is an adder, 138 and 139 are target memories for storing reproduced image data. 1
40 is an input from the block former 127 and the target memory 1
This is a motion compensator that performs motion compensation based on the contents of 38 and 139.

In the above configuration, each memory is cleared at the start of encoding, and each component is initialized.
The frame mode setting unit 128 indicates an I-frame when encoding the first frame. At this time, the motion compensator 1
40 does not operate and outputs 0 as the motion compensation prediction value.
The image data of the area 1200 is read from the terminal 122 in synchronization with the mask information 1201 from the terminal 121, and stored in the target memory 125 and the mask memory 123, respectively.

When one frame is stored, the mask encoder 124 encodes the mask information 1201 and
Output to 3. The average value calculator 127 outputs the mask information 120
It is determined according to 1 whether the input pixel is the background or the target image, and the average value m of the person 1052 as the target image is calculated.
The block forming unit 127 reads the image data of the area 1200 and the mask information 1201 in synchronization with each other on a block basis.
If the mask information 1201 of the input pixel indicates a background pixel, the pixel value is replaced with the average value m. Otherwise, the input pixel value is output as it is to form an 8 × 8 block. That is, in the entire image, the background portion is replaced with the average value m as shown in FIG. Since the motion compensation prediction value is 0, the differentiator 129 outputs the input as it is. This output is DCT-transformed by the DCT transformer 130, and its coefficient is quantized by the quantizer 131. The quantization result is the encoder 1
A code is assigned at 32 and output to the combiner 133. The combiner 133 includes a mask encoder 124 and an encoder 132
A header necessary for the encoded data generated in step (1) is added and the data is aligned, and output from the terminal 134. Further, the quantization result is inversely quantized by the inverse quantizer 135, a reproduction pixel value is obtained by the inverse DCT transformer 136, and stored in the target memory 138 or 139 via the adder 137.

Further, the frame mode setting unit 128 sets the P-
If a frame or B-frame has been designated, the motion compensator 140 is operated, image data necessary for motion compensation is read from the target memories 138 and 139, and it is determined whether or not to perform motion compensation. When performing motion compensation, the motion compensation prediction value is output to a differentiator 129 and an adder 137, and a motion vector used for motion compensation is input to an encoder 132. When motion compensation is not performed, the motion compensation prediction value is set to 0.
And

The encoded data obtained by the encoding by the target encoding unit 105 is output to the communication line 108 via the transmitter 106.

On the other hand, the image picked up by the TV camera 102 is coded by an encoder 104 into MPEG-1 with the same configuration as that of the moving picture coding apparatus 1002 shown in FIG.
Output to the communication line 109 via the transmitter 107.

The receivers 110 and 111 are connected to the communication line 108,
The coded data is received via the control unit 109 and transmitted to the moving image editing apparatus 112. Further, the storage device 116 stores the MPEG-
The encoded sequence read in step 4 is read, and the encoded data is transmitted to the moving image editing apparatus 112.

Next, the moving image editing apparatus 112 according to the first embodiment
The detailed configuration will be described with reference to FIG.

FIG. 6 is a block diagram showing a detailed configuration of the moving picture editing apparatus according to the first embodiment of the present invention.

Reference numerals 200, 201, and 202 denote terminals. The terminal 200 is connected to the receiver 110, and the terminal 201 is connected to the receiver 11
From 1, encoded data from the storage device 116 is input to the terminal 202. These encoded data are supplied to respective terminals 2 of the target decoders 203 and 204 and the decoder 205.
19, 220 and 221. Terminals 206, 20
9, 225 output image data in RGB format. From the terminals 208, 211, and 212, color fog correction image information 22 necessary for calculating a color fog correction value.
2, 223 and 224 are output. Terminals 207, 210
Output mask information. A correction value calculator 213 calculates a correction value from the color cast correction image information.

Reference numerals 214, 215 and 216 denote correctors for correcting the color cast of the image data from the correction values. An image synthesizer 217 synthesizes image data from the image data and the mask information. 218 is a terminal, which is a synthesized RG
The image data in the B format is output to the encoder 113.

Next, the target decoders 203, 2
The detailed configuration of No. 04 will be described with reference to FIG. still,
In FIG. 7, a detailed configuration of the target decoder 203 will be described.
The detailed configuration of the target decoder 204 having the same configuration is omitted here.

FIG. 7 is a block diagram showing a detailed configuration of the target decoder according to the first embodiment of the present invention. Reference numeral 219 denotes a terminal to which encoded data from the receiver 110 is input. 2
Reference numeral 41 denotes a separator, which separates the encoded data of the mask information and the encoded data of the area of the target image from the input encoded data. Reference numeral 242 denotes a mask decoder that decodes mask information. A mask memory 243 stores mask information. The mask information in the mask memory 243 is
07. Reference numeral 244 denotes a code memory for storing the encoded data of the area of the target image. A decoder 245 decodes the encoded data of the area of the target image. 24
6, 248 are demultiplexers. 248, 25
5, 262 are inverse quantizers. 249, 256, 26
3 is an inverse DCT converter.

Reference numerals 250, 257, and 264 denote adders.
Reference numerals 251, 252 and 253 are target memories for storing luminance Y data of the reproduced target image area. 258, 25
Reference numerals 9 and 260 are target memories for storing the color difference Cb data of the reproduced target image area. 265, 266, 267
Is a target memory for storing the color difference Cr data of the area of the reproduced target image. 254, 261, and 268 are motion compensators. Reference numerals 269 and 273 denote color signal converters for performing color signal conversion from YCbCr format image data to RGB format image data. 270, 271, and 272 are buffers. Reference numeral 206 denotes a terminal from which image data in RGB format is output. A terminal 207 outputs mask information. Reference numeral 208 denotes a terminal from which color fog correction image information is output.

In the above configuration, the coded data of the mask information and the coded data of the target image area are separated by the separator 214 from the input coded data, and are input to the mask decoder 242 and the code memory 244, respectively. The mask decoder 242 decodes the encoded data of the mask information to reproduce the mask information, and stores the reproduced mask information in the mask memory 243. The encoded data stored in the code memory 244 is decoded by the decoder 245, and the quantized value is reproduced. After that, the demultiplexer 247 decodes the luminance Y data, the chrominance Cb data, and the chrominance Cb.
Demultiplexed into r data. The luminance Y data is supplied to the inverse quantizer 248, and the chrominance Cb data is supplied to the inverse quantizer 255.
In addition, the chrominance Cr data is input to the inverse quantizer 262.

The luminance Y data is inversely quantized by an inverse quantizer 248 and inverse DCT transformed by an inverse DCT transformer 249.
In the case of an I-frame, the motion compensator 254 does not operate and outputs 0. At the time of a P-frame and a B-frame, the motion compensator 254 operates and outputs a motion compensation prediction value. The adder 250 outputs the output of the inverse DCT transformer 249 and the motion compensator 25.
4 are added and stored in the target memory 251 and the target memory 252 or 253. On the other hand, only during the I-frame, the luminance Y out of the output from the inverse quantizer 248
Only the DC component information representing the average value of the data is stored in the buffer 2
72.

The chrominance Cb data is inversely quantized by the inverse quantizer 255 and inverse DCT-transformed by the inverse DCT converter 256. In the case of an I-frame, the motion compensator 261 does not operate,
Outputs 0. At the time of a P-frame and a B-frame, the motion compensator 261 operates and outputs a motion compensation prediction value. The adder 257 adds the output of the inverse DCT transformer 256 and the output of the motion compensator 261 and stores the result in the target memory 258 and the target memory 259 or 260. On the other hand, only in the case of an I-frame, only the DC component information representing the average value of the color difference Cb data among the outputs from the inverse quantizer 255 is stored in the buffer 271.

The chrominance Cr data is inversely quantized by the inverse quantizer 262 and inverse DCT transformed by the inverse DCT transformer 263. In the case of an I-frame, the motion compensator 268 does not operate,
Outputs 0. At the time of a P-frame and a B-frame, the motion compensator 268 operates and outputs a motion compensation prediction value. The adder 264 adds the output of the inverse DCT transformer 263 and the output of the motion compensator 268, and stores the sum in the target memory 265 and the target memory 266 or 267. On the other hand, only in the case of the I-frame, only the DC component information representing the average value of the color difference Cr data among the outputs from the inverse quantizer 262 is stored in the buffer 270.

When the processing of the macro block is completed, the luminance Y DC component information from the buffers 270, 271, 272,
The color difference Cb DC component information and the color difference Cr DC component information are read out, converted into RGB format by the color signal converter 273, and output from the terminal 208 as color cast correction image information.

The target memories 251, 258, 265
When the image data in the YCbCr format is read, the image data is converted into image data in the RGB format by the color signal converter 269 and then output from the terminal 206.

Next, the detailed configuration of the decoder 205 according to the first embodiment will be described with reference to FIG.

FIG. 8 is a block diagram showing a detailed configuration of the decoder according to the first embodiment of the present invention.

Reference numeral 221 denotes a terminal to which encoded data from the storage device 116 is input. Reference numeral 301 denotes a code memory for storing encoded data. Reference numeral 302 denotes a decoder that decodes encoded data. 303 and 304 are demultiplexers. 305, 312 and 319 are inverse quantizers. 306, 313, and 320 are inverse DCT converters.
307, 314 and 321 are adders. 308, 30
Reference numerals 9 and 310 are memories for storing luminance Y data of an image obtained by decoding encoded data. 315, 316,
317 is a color difference C of an image obtained by decoding the encoded data.
b is a memory for storing data. 322, 323, 3
24 is the color difference Cr of the image obtained by decoding the encoded data.
This is a memory for storing data. 311, 318, 32
5 is a motion compensator. 326 and 330 are color signal converters for performing color signal conversion from YCbCr format image data to RGB format image data. 327, 328, 329
Is a buffer. Reference numeral 225 denotes a terminal from which image data in RGB format is output. Reference numeral 212 denotes a terminal from which color cast image information is output.

In the above configuration, the coded data stored in the code memory 301 is decoded by the decoder 262 and then decoded by the demultiplexer 303 to obtain the luminance Y data and the color difference C.
The data is demultiplexed into b data and color difference Cr data.
The luminance Y data is supplied to the inverse quantizer 305, the color difference Cb data is supplied to the inverse quantizer 312, and the color difference Cr data is supplied to the inverse quantizer 31.
9 is input.

The luminance Y data is inversely quantized by the inverse quantizer 305 and inverse DCT transformed by the inverse DCT transformer 306.
In the case of an I-frame, the motion compensator 311 does not operate and outputs 0. At the time of a P-frame and a B-frame, the motion compensator 311 operates and outputs a motion compensation prediction value. The adder 307 outputs the output of the inverse DCT converter 306 and the motion compensator 31
One output is added and stored in the memory 308 and the memory 309 or 310. On the other hand, only in the case of the I-frame, only the DC component information representing the average value of the luminance Y data among the outputs from the inverse quantizer 305 is stored in the buffer 329.

The chrominance Cb data is inversely quantized by the inverse quantizer 312 and inverse DCT transformed by the inverse DCT transformer 313. In the case of an I-frame, the motion compensator 318 does not operate,
Outputs 0. At the time of a P-frame and a B-frame, the motion compensator 318 operates and outputs a motion compensation prediction value. The adder 314 adds the output of the inverse DCT transformer 313 and the output of the motion compensator 318, and
6 or 317. On the other hand, the color difference Cb of the output from the inverse quantizer 312 is limited only to the I-frame.
Only the DC component information representing the average value of the data is stored in the buffer 3
28.

The chrominance Cr data is inversely quantized by the inverse quantizer 319 and inverse DCT transformed by the inverse DCT transformer 320. In the case of an I-frame, the motion compensator 325 does not operate,
Outputs 0. At the time of a P-frame and a B-frame, the motion compensator 325 operates and outputs a motion compensation prediction value. The adder 321 adds the output of the inverse DCT converter 320 and the output of the motion compensator 325, and
3 or 324. On the other hand, only during the I-frame, the chrominance Cr
Only the DC component information representing the average value of the data is stored in the buffer 3
27.

When the processing of the macro block is completed, the luminance Y DC component information from the buffers 327, 328, and 329,
The chrominance Cb DC component information and the chrominance Cr DC component information are read out, converted into the RGB format by the color signal converter 330, and output from the terminal 212 as color cast image information.

When image data in the YCbCr format is read from the memories 308, 315, and 322, the image data is converted into image data in the RGB format by the color signal converter 326 and output from the terminal 224. In the configuration of the video editing device 112, decoding of one frame is completed, and the target memories 251 and 25 in the target decoder 203 are finished.
8, 265 and the target memory 25 in the target decoder 204
1, 258, 265 and the memory 30 in the encoder 205.
When the image data is stored in 8, 315, and 322, the correction value calculator 213 obtains the following correction formula from the correction formula calculation algorithm described later using the color cast correction image information. That is, the R pixel value correction formula f1R (x) for the corrector 214,
G pixel value correction formula f1G (x) for corrector 214, corrector 214
B pixel value correction formula f1B (x), R pixel value correction formula f2R (x) for corrector 215, G pixel value correction formula f2G for corrector 215
(x), B pixel value correction formula f2B (x) for corrector 215, R pixel value correction formula f3R (x) for corrector 216, G for corrector 216
A pixel value correction formula f3G (x) and a B pixel value correction formula f3B (x) for the corrector 216 are obtained.

Thereafter, RGB pixel values are read from the decoder 205 by raster scan in the order of the pixels of the scanning line, corrected by the corrector 216, and input to the image synthesizer 217. In the corrector 216, the input R pixel value r, G pixel value g, B
Correction formulas f3R (x), f3G (x), f3B (x) for pixel value b
Is performed according to the following equation to obtain and output corrected R pixel values R, G pixel values G, and B pixel values B.

R = f3R (r), G = f3G (g), B = f3b (b) (1) On the other hand, when the scan position reaches the position where the target image data of the target decoder 203 is synthesized, the target decoding is performed. The mask information and the RGB pixel values are read from the device 203, corrected by the corrector 214, and input to the image synthesizer 217. Corrector 21
In step 4, the input R pixel value r, G pixel value g, and B pixel value b are corrected by the correction formulas f1R (x), f1G (x), and f1B (x) according to the following equations. R pixel value R,
A G pixel value G and a B pixel value B are obtained and output.

R = f1R (r), G = f1G (g), B = f1b (b) (2) When the scan position reaches the position where the target image data of the target decoder 204 is synthesized, the target decoding is performed. The mask information and the RGB pixel values are read from the device 204, corrected by the corrector 215, and input to the image synthesizer 217. Corrector 21
In 5, the input R pixel value r, G pixel value g, and B pixel value b are corrected by the correction formulas f2R (x), f2G (x), and f2B (x) according to the following formulas. R pixel value R,
A G pixel value G and a B pixel value B are obtained and output.

R = f2R (r), G = f2G (g), B = f2b (b) (3) The image synthesizer 217 determines whether the mask information indicates the target image data of the target decoder 203. The pixel value from the corrector 214, and the pixel value from the corrector 215 when the mask information indicates the target image data of the target decoder 204,
If none of the above cases is satisfied, the image is synthesized by outputting the pixel value from the corrector 216 and the terminal 2
18 to the encoder 113. FIG. 9 shows the background 10
A background 1160 and a person 1061, which are images obtained by correcting 50 and the person 1051, a person 1062 which is an image obtained by correcting the person 1052, and an image obtained by correcting the person 1053. This shows a state in which a person 1063 is combined. The encoder 113 encodes the output image using MPEG-1, and sends the image to the communication network 115 via the transmitter 114.

In the above operation, the correction formula calculating algorithm of the correction value calculator 213 operates according to the following.

Correction formulas f3R (r), f3G for corrector 216
(g) and f3b (b) are calculated as follows.

The human eye is relatively dull to blue, and the correction effect does not appear so much. Therefore, f3b (b) for correcting the B pixel value is given by f3B (b) = b (4).

Next, the maximum value RMax1, the average value RE1, and the variance RR1 of the R information in the color fog correction image information 224 from the decoder 205 are obtained.

Next, the maximum value GMax1, the average value GE1, and the variance GR1 of the G information in the color fog correction image information 224 from the decoder 205 are obtained.

Then, a two-dimensional histogram showing the distribution of the value of the R information and the value of the G information is calculated.

| RE1-GE1 | is less than a threshold and | R
When R1−GR1 | is equal to or smaller than a certain threshold value, RMax1 ≧ GMax1, and the diagonal line in the two-dimensional histogram is represented by (RMax1, RMax1) − (GMax1−
If there is a significant bias toward the R axis in a square area defined as (T, GMax1-T), f3B (r) = r, f3G (g) = g × RMax1 / GMax1 (5)

GMax1 ≧ RMax1, and the diagonal line in the two-dimensional histogram is represented by (GMax1, GMax1) −
If there is a significant bias toward the G axis in the square area defined as (RMax1−T, RMax1−T), f3G (g) = g, f3R (r) = r × GMax1 / RMax1 (6) I do.

If none of the above cases applies, f3R (r) = r, f3G (g) = g (7) Here, T is a certain positive number.

Otherwise, f3r for correcting the R pixel value
(r), f3g (g) for correcting the G pixel value: f3R (r) = r, f3G (g) = g (8)

From the above, the correction equations f3R (r), f3G (g), f
The calculation of 3b (b) ends.

Similarly, the correction formula f1R for the corrector 214
(r), f1G (g), f1b (b), correction formula f2R for corrector 215
(r), f2G (g) and f2b (b) are calculated.

As described above, according to the first embodiment, when the background image and the target image are separated, and when the coded images are combined, the feature amount of each image data is extracted and the object to be combined is extracted. By correcting the pixel values of the image, it is possible to perform image synthesis without a sense of discomfort. Further, high-speed processing can be performed by using a DC component in block units for calculating a correction value.

In the first embodiment, MPEG-4 is used for encoding a target image, and MPEG-
1 has been described, but the present invention is not limited to this, and any device that performs the same function as those described above may be used.

The memory configuration is not limited to this, and it goes without saying that processing may be performed by a line memory or the like.
Other configurations may be used.

Further, a part or all of each component may be replaced with a CP.
Of course, it may be realized by software operating on U or the like. Embodiment 2 Embodiment 2 is a modification of Embodiment 1 in which the target decoders 203 and 204, the decoder 205, and the correction value calculator 213 are changed. Therefore, the first embodiment
The description of the parts overlapping with the above is omitted, and only the changed part will be described.

The moving image transmission system uses the configuration shown in FIG. 1 as in the first embodiment. In addition, the moving image editing device 112
The configuration of FIG. 5 is used as in the first embodiment.

Next, the target decoders 203 and 2 in the second embodiment
The detailed configuration of No. 04 will be described with reference to FIG.
In FIG. 10, the detailed configuration of the target decoder 203 will be described, and the detailed configuration of the target decoder 204 having the same configuration will be omitted.

FIG. 10 is a block diagram showing a detailed configuration of a target decoder according to the second embodiment of the present invention.

Reference numeral 219 denotes a terminal to which encoded data from the receiver 110 is input. Reference numeral 401 denotes a separator, which separates encoded data of mask information and encoded data of a target image area from the encoded data. Reference numeral 402 denotes a mask decoder that decodes mask information. A mask memory 403 stores mask information. The mask information in the mask memory 403 is output from a terminal 207. A code memory 404 stores the encoded data of the target area. 4
Reference numeral 05 denotes a decoder for decoding the encoded data of the area of the target image. 406 and 407 are demultiplexers. 4
08, 415, and 422 are inverse quantizers. 409, 4
16 and 423 are high-speed inverse DCT converters.

Here, the detailed configuration of the high-speed inverse DCT converters 409, 416, and 423 of the first embodiment will be described with reference to FIG.

FIG. 11 shows a high-speed inverse DC according to the second embodiment of the present invention.
It is a block diagram which shows the detailed structure of a T converter.

The output of each Radix butterfly operation unit 1101 to 1104 in FIG. 11 has a path for multiplexing and outputting the output of each stage by the multiplexer 1105 in addition to the path of the normal Radix butterfly operation. However, the first stage Radix butterfly operation unit 1
Only the direct current component from the multiplexer
5 is input. From the rear of the second-stage Radix butterfly operation unit 1102, a Radix butterfly operation result of 2 × 2 low frequency components is input to the multiplexer 1105. Also, from the rear of the third-stage Radix butterfly operation unit 1103, a Radix butterfly operation result of 4 × 4 low frequency components is input to the multiplexer 1105. In addition, the fourth stage Radix butterfly operation unit 1
An 8 × 8 inverse DCT result is input to the multiplexer 1105 from the back of the 104.

Returning to the description of FIG.

Reference numerals 410, 417 and 424 denote adders.
411, 412, and 413 are target memories for storing luminance Y data of the reproduced target image area. 418, 41
Reference numerals 9 and 420 are target memories for storing the color difference Cb data of the reproduced target region image. 425, 426, 427
Is a target memory for storing the color difference Cr data of the area of the reproduced target image. 414, 421, 428 are motion compensators. Reference numerals 429 and 433 denote color signal converters for performing color signal conversion from YCbCr format image data to RGB format image data. 430, 431 and 432 are buffers. Reference numeral 206 denotes a terminal from which image data in RGB format is output. Reference numeral 208 denotes a terminal from which color fog correction image information is output.

In the above configuration, the coded data of the mask information and the coded data of the target image area are separated by the separator 401 from the input coded data, and are input to the mask decoder 402 and the code memory 404, respectively. The mask decoder 402 decodes the encoded data to reproduce mask information,
It is stored in the mask memory 403. The encoded data stored in the code memory 404 is decoded by the decoder 405 and the quantized value is reproduced, and then demultiplexed by the demultiplexer 407 into luminance Y data, color difference Cb data, and color difference Cr data. . The luminance Y data is converted to an inverse quantizer 408.
The chrominance Cb data is input to the inverse quantizer 415, and the chrominance Cr data is input to the inverse quantizer 422.

The luminance Y data is inversely quantized by the inverse quantizer 408 and inverse DCT-transformed by the high-speed inverse DCT converter 409 by a radix butterfly operation. In the case of an I-frame, the motion compensator 414 does not operate and outputs 0. P-
At the time of a frame and a B-frame, the motion compensator 414 operates and outputs a motion compensation prediction value. The adder 410 adds the output of the high-speed inverse DCT transformer 409 and the output of the motion compensator 414, and stores the sum in the target memory 411 and the target memory 412 or 413. On the other hand, only in the case of the I-frame, the result of the n-th stage Radix butterfly operation is multiplexed and output from the high-speed inverse DCT converter 409, and the image data consisting only of the low-frequency component of the luminance Y data is stored in the buffer 432. Is stored.

The color difference Cb data is inversely quantized by an inverse quantizer 415 and inverse DCT-transformed by a high-speed inverse DCT converter 416 by a radix butterfly operation. In the case of an I-frame, the motion compensator 421 does not operate and outputs 0. P
The motion compensator 421 operates at the time of a -frame and a B-frame, and outputs a motion compensation prediction value. The adder 417 adds the output of the high-speed inverse DCT converter 416 and the output of the motion compensator 421, and stores the result in the target memory 418 and the target memory 419 or 420. On the other hand, only in the case of the I-frame, the high-speed inverse DCT converter 416 multiplexes the n-th stage Radix butterfly operation result and outputs the image data consisting of only the low-frequency component of the color difference Cb data and stores the image data in the buffer 431. Is done.

The chrominance Cr data is inversely quantized by an inverse quantizer 422 and inverse DCT-transformed by a high-speed inverse DCT converter 423 by a radix butterfly operation. In the case of an I-frame, the motion compensator 428 does not operate and outputs 0. P
The motion compensator 428 operates at the time of a -frame and a B-frame, and outputs a motion compensation prediction value. The adder 424 adds the output of the high-speed inverse DCT transformer 423 and the output of the motion compensator 428, and stores them in the target memory 425 and the target memory 426 or 427. On the other hand, only in the case of an I-frame, the n-th stage Radix butterfly operation result is multiplexed from the high-speed inverse DCT converter 423 and output and stored in the buffer 430. Is done.

When the processing of the macro block is completed, the luminance Y data and the chrominance C are output from the buffers 430, 431, and 432.
The b data and the color difference Cr data are read out, converted into the RGB format by the color signal converter 273, and then output from the terminal 208 as color fog correction image information.

Also, the target memories 411, 418, 425
When image data in the YCbCr format is read, the image data is converted into image data in the RGB format by the color signal converter 429 and then output from the terminal 206.

Next, the detailed configuration of the decoder 205 according to the second embodiment will be described with reference to FIG.

FIG. 12 is a block diagram showing a detailed configuration of the decoder according to the second embodiment of the present invention.

Reference numeral 202 denotes a terminal to which encoded data from the storage device 116 is input. Reference numeral 452 denotes a code memory for storing encoded data. A decoder 453 decodes the encoded data. 454 and 455 are demultiplexers. Reference numerals 456, 463, and 470 denote inverse quantizers. 457, 464 and 471 are high-speed inverse DCT converters. The detailed configuration of the high-speed inverse DCT converters 457, 464, and 471 is the same as that shown in FIG. 458, 46
5, 472 are adders. Reference numerals 459, 460, and 461 denote memories for storing luminance Y data of an image obtained by decoding encoded data. Reference numerals 466, 467, and 468 denote memories for storing color difference Cb data of an image obtained by decoding encoded data. Reference numerals 473, 474, and 475 are memories for storing color difference Cr data of an image obtained by decoding encoded data. 462, 469 and 476 are motion compensators. Reference numerals 477 and 481 denote color signal converters for performing color signal conversion from YCbCr format image data to RGB format image data. 478, 479, and 480 are buffers. Reference numeral 225 denotes a terminal from which image data in RGB format is output. Reference numeral 212 denotes a terminal from which color cast image information is output.

In the above configuration, the coded data stored in the code memory 452 is decoded by the decoder 453 and the quantized value is reproduced, and then the luminance data Y, the chrominance Cb data, and the chrominance Cr data are decoded by the demultiplexer 455. Is demultiplexed. The luminance Y data is converted to an inverse quantizer 456.
The chrominance Cb data is input to the inverse quantizer 463, and the chrominance Cr data is input to the inverse quantizer 470.

The luminance Y data is inversely quantized by an inverse quantizer 456 and inverse DCT-transformed by a high-speed inverse DCT converter 457 by a Radix butterfly operation. During an I-frame, the motion compensator 462 does not operate and outputs 0. P-
At the time of a frame and a B-frame, the motion compensator 462 operates and outputs a motion compensation prediction value. The adder 458 adds the output of the high-speed inverse DCT converter 457 and the output of the motion compensator 462, and outputs the result from the memory 459 and the memory 460 or 461.
To be stored. On the other hand, only during the I-frame, the high-speed inverse DCT converter 457 outputs the result of the nth-stage Radix butterfly operation after being multiplexed, and outputs the image data consisting only of the low-frequency component of the luminance Y data to the buffer 480. Is stored.

The chrominance Cr data is inversely quantized by an inverse quantizer 463 and inverse DCT-transformed by a high-speed inverse DCT converter 464 by a radix butterfly operation. In the case of an I-frame, the motion compensator 469 does not operate and outputs 0. P
The motion compensator 469 operates at the time of a -frame and a B-frame, and outputs a motion compensation prediction value. The adder 465 adds the output of the high-speed inverse DCT converter 464 and the output of the motion compensator 469, and outputs the result to the memory 466 and the memory 467 or 46.
8 is stored. On the other hand, only in the case of the I-frame, the n-th stage Radix butterfly operation result is multiplexed and output from the high-speed inverse DCT converter 464,
Image data consisting of only low-frequency components of the color difference Cr data is stored in the buffer 479.

The chrominance Cb data is inversely quantized by an inverse quantizer 470 and inverse DCT-transformed by a high-speed inverse DCT converter 471 by a radix butterfly operation. In the case of an I-frame, the motion compensator 476 does not operate and outputs 0. P
The motion compensator 476 operates at the time of a -frame and a B-frame, and outputs a motion compensation prediction value. An adder 472 adds the output of the high-speed inverse DCT converter 471 and the output of the motion compensator 476, and outputs the result to the memory 473 and the memory 474 or 47.
5 is stored. On the other hand, only in the case of an I-frame, the n-th stage Radix butterfly operation result is multiplexed from the high-speed inverse DCT converter 471 and output.
Image data including only the low-frequency component of the color difference Cb data is stored in the buffer 478.

When the processing of the macro block is completed, the luminance Y data and the color difference C are output from the buffers 478, 479 and 480.
The b data and the color difference Cr data are read out, converted into the RGB format by the color signal converter 481, and then output from the terminal 212 as color fog correction image information.

When image data in the YCbCr format is read from the memories 459, 466, and 473, the image data is converted into image data in the RGB format by the color signal converter 429 and then output from the terminal 225.

In the configuration of the moving picture editing apparatus 112 described above, decoding of one frame is completed,
03; target memories 411, 418, and 425 in the target encoder 204;
When the image data is stored in the memories 459, 466, and 473 in the encoder 205, the correction value calculator 213 obtains the following correction formula from the correction formula calculation algorithm described later using the color cast correction image information. That is, the corrector 214
R pixel value correction formula f1R (x), G pixel value correction formula f1G (x) for corrector 214, B pixel value correction formula f1B (x) for corrector 214
And R pixel value correction formula f2R (x) for corrector 215, corrector 2
15 G pixel value correction formula f2G (x), corrector 215 B pixel value correction formula f2B (x), and corrector 216 R pixel value correction formula f
3R (x), a G pixel value correction formula f3G (x) for the corrector 216, and a B pixel value correction formula f3B (x) for the corrector 216 are obtained.

After that, the RGB pixel values are read from the decoder 205 by raster scan in the order of the pixels of the scanning line, corrected by the corrector 216, and input to the image synthesizer 217. In the corrector 216, the input R pixel value r, G pixel value g, B
Correction formulas f3R (x), f3G (x), f3B (x) for pixel value b
Is performed according to the following equation to obtain and output corrected R pixel values R, G pixel values G, and B pixel values B.

R = f3R (r), G = f3G (g), B = f3b (b) (9) On the other hand, when the scan position reaches the position where the target decoder 203 combines the target image data, the target decoding is performed. The mask information and the RGB pixel values are read from the device 203, corrected by the corrector 214, and input to the image synthesizer 217. Corrector 21
In step 4, the input R pixel value r, G pixel value g, and B pixel value b are corrected by the correction formulas f1R (x), f1G (x), and f1B (x) according to the following equations. R pixel value R,
A G pixel value G and a B pixel value B are obtained and output.

R = f1R (r), G = f1G (g), B = f1b (b) (10) When the scan position reaches the position where the target image data of the target decoder 204 is synthesized, the target decoding is performed. The mask information and the RGB pixel values are read from the device 204, corrected by the corrector 215, and input to the image synthesizer 217. Corrector 21
In step 4, the input R pixel value r, G pixel value g, and B pixel value b are corrected by the correction formulas f2R (x), f2G (x), and f2B (x) according to the following equations. The calculated R pixel value R, G pixel value G, and B pixel value B are output.

R = f2R (r), G = f2G (g), B = f2b (b) (11) The image synthesizer 217 determines whether the mask information indicates the target image data of the target decoder 203. The pixel value from the corrector 214, and the pixel value from the corrector 215 when the mask information indicates the target image data of the target decoder 204,
If none of the above cases is satisfied, the image is synthesized by outputting the pixel value from the corrector 216 and the terminal 2
18 to the encoder 113. FIG. 9 shows the background 10
A background 1160 and a person 1061, which are images obtained by correcting 50 and the person 1051, a person 1062 which is an image obtained by correcting the person 1052, and an image obtained by correcting the person 1053. This shows a state in which a person 1063 is combined. The encoder 113 encodes the output image using the MPEG-1 encoding method, and sends the image to the communication network 115 via the transmitter 114.

In the above operation, the correction formula calculating algorithm of the correction value calculator 213 operates according to the following.

Correction formulas f3R (r), f3G for corrector 216
(g) and f3b (b) are calculated as follows.

The human eye is relatively dull for blue, and does not show much correction effect. Therefore, f3b (b) for correcting the B pixel value is given by f3B (b) = b (12).

Next, the maximum value RMax1, average value RE1, and variance RR1 of the R information in the color fog correction image information 224 from the decoder 205 are obtained.

Next, the maximum value GMax1, average value GE1, and variance GR1 of the G information in the color fog correction image information 224 from the decoder 205 are obtained.

Then, a two-dimensional histogram showing the distribution of the value of the R information and the value of the G information is calculated.

| RE1-GE1 | is equal to or less than a threshold and | R
When R1−GR1 | is equal to or smaller than a certain threshold value, RMax1 ≧ GMax1, and the diagonal line in the two-dimensional histogram is represented by (RMax1, RMax1) − (GMax1−
T, GMax1−T), if there is a significant bias toward the R axis from the center of gravity and variance of the histogram and the area where the histogram value is 0 in this area, / f3R ( x) and / f3G (x) are defined as / f3R (x) = r, / f3G (g) = g × RMax1 / GMax1 (13).

GMax1 ≧ RMax1, and the diagonal line in the two-dimensional histogram is represented by (GMax1, GMax1) −
In a square area defined as (RMax1−T, RMax1−T), when a significant bias toward the G axis is recognized from the center of gravity and variance of the histogram and the area where the value of the histogram is 0 in this area, / f3R (x) and / f3G
(x) and / f3G (g) = g, / f3R (r) = r × GMax1 / RMax1 (14)

If none of the above cases applies, / f3R (x) and / f3G (x) are determined as / f3R (r) = r, / f3G (g) = g (15). Here, T is a certain positive number.

Otherwise, / f3R (x) and / f3G (x) are determined as / f3R (r) = r, / f3G (g) = g (16).

The above is | RE1-GE1 | and | RR1-GR1.
|.

Based on the correction formula one frame before, the current correction formula is determined as follows.

F3R (x) = f3R (x) + γ (/ f3R (x) −f3R (x)) f2G (x) = f3G (x) + γ (/ f3G (x) −f3G (x)) (17) Here, γ is a time-variant tracking weight variable of the correction formula.

As described above, the correction equations f3R (r), f3G (g), f
The calculation of 3b (b) ends.

Similarly, the correction equation f1R for the corrector 214
(r), f1G (g), f1b (b), correction formula f2R for corrector 215
(r), f2G (g) and f2b (b) are calculated.

As described above, according to the second embodiment, when separating the background image and the target image and synthesizing the coded images, the characteristic amount of each image data is extracted and the target image is synthesized. By correcting the pixel values of the image, it is possible to perform image synthesis without a sense of discomfort. Further, in consideration of the size of the target image and the calculation speed, a correction value is calculated by selectively selecting a DC component or an inverse DCT result of a 2 × 2 or 4 × 4 low frequency component or an 8 × 8 inverse DCT result. In this case, flexible and highly accurate processing can be performed. Furthermore, by causing the color fogging correction process to slowly follow the time change, it is possible to perform image synthesis without discomfort even for an image that changes rapidly.

In the second embodiment, MPEG-4 is used for encoding a target image, and MPEG-
1 has been described, but the present invention is not limited to this, and any device that performs the same function as those described above may be used.

The memory configuration is not limited to this, and it goes without saying that processing may be performed by a line memory or the like.
Other configurations may be used.

A part or all of the constituent elements may be replaced by a CP.
Of course, it may be realized by software operating on U or the like. <Third Embodiment> A third embodiment is a modification of the first embodiment in which the moving image editing apparatus 112 is modified. Therefore, the first embodiment
The description of the parts overlapping with the above is omitted, and only the changed part will be described.

The moving image transmission system uses the configuration shown in FIG. 1 as in the first embodiment.

Next, the moving picture editing apparatus 112 according to the third embodiment
Will be described with reference to FIG.

FIG. 13 is a block diagram showing a detailed configuration of the moving picture editing apparatus according to the third embodiment of the present invention.

Reference numerals 1200, 1201, and 1202 denote terminals. The terminal 1200 receives encoded data from the receiver 110, the terminal 1201 receives encoded data from the receiver 111, and the terminal 1202 receives encoded data from the storage device. These encoded data are supplied to the target decoders 1203 and 1204 and the decoders 1203 and 1204.
5 is input. Image data is output from the terminals 1206, 1209, and 1225. Terminals 1207, 121
From 0 and 1212, contrast correction image information 1222 and 122 necessary for calculating a contrast correction value.
3 and 1224 are output. Terminals 1207, 1
210 outputs mask information. A correction value calculator 1213 calculates a correction value from the image information for contrast correction. 1214, 1215 and 1216 are correctors for correcting the contrast of the image data from the correction values.
Reference numeral 1217 denotes an image synthesizer that synthesizes image data from the image data and the mask information. 1218 is a terminal,
The combined image data is output to the encoder 113.

Next, the target decoder 1203 of the third embodiment,
The detailed configuration of 1204 will be described with reference to FIG. In FIG. 14, the detailed configuration of the target decoder 1203 will be described, and the detailed configuration of the target decoder 1204 is the same.

FIG. 14 is a block diagram showing a detailed configuration of a target decoder according to the third embodiment of the present invention.

Reference numeral 219 denotes a terminal to which encoded data from the receiver 110 is input. 1241 is a separator;
The encoded data of the mask information and the encoded data of the area of the target image are separated from the encoded data. Reference numeral 1242 denotes a mask decoder that decodes mask information. Reference numeral 1243 denotes a mask memory for storing mask information. Mask memory 124
The mask information in 3 is output from the terminal 1207. 1
Reference numeral 244 denotes a code memory for storing the encoded data of the area of the target image. Reference numeral 1245 denotes a decoder for decoding the encoded data of the area of the target image. Reference numeral 1246 denotes an inverse quantizer. The DC information in the dequantized image data is output from the terminal 1208 as image information for contrast correction. Reference numeral 1247 denotes an inverse DCT converter. 1248 is an adder. Reference numerals 1249, 1250, and 1251 denote target memories for storing image data of a reproduced target image area. Reference numeral 1252 denotes a motion compensator. Target memory 1249
Are output from the terminal 1206.

In the above configuration, the coded data of the mask information and the coded data of the target image area are separated from the input coded data by the separator 1241 and input to the mask decoder 1242 and the code memory 1244, respectively. The mask decoder 1242 decodes the encoded data to reproduce the mask information, and stores the mask information in the mask memory 1243. The encoded data stored in the code memory 1244 is decoded by the decoder 1245, and the quantized value is reproduced. This value is inversely quantized by the inverse quantizer 1246 and the inverse DCT transformer 124
7 is subjected to inverse DCT. In the case of an I-frame, the motion compensator 1252 does not operate and outputs 0. At the time of a P-frame and a B-frame, the motion compensator 1252 operates and outputs a motion compensation prediction value. The adder 1248 adds the output of the inverse DCT transformer 1247 and the output of the motion compensator 1252, and stores the result in the target memory 1249 and the target memory 1250 or 1251. On the other hand, a DC component representing the average value of the luminance data is output from the terminal 1208 from the inverse quantizer 1246 as image information for contrast correction.

Next, the detailed configuration of the decoder 1205 according to the third embodiment will be described with reference to FIG.

FIG. 15 is a block diagram showing a detailed configuration of the decoder according to the third embodiment of the present invention.

Reference numeral 1221 denotes a terminal to which encoded data from the storage device 116 is input. Reference numeral 1261 denotes a code memory for storing encoded data. Reference numeral 1262 denotes a decoder for decoding encoded data. Reference numeral 1263 denotes an inverse quantizer. The DC information in the dequantized image data is output from the terminal 1212 as image information for contrast correction. Reference numeral 1264 denotes an inverse DCT converter. 1265 is an adder. Reference numerals 1266, 1267, and 1268 denote memories for storing the decoded image data. Reference numeral 1269 denotes a motion compensator. The image data in the memory 1266 is
225.

In the above configuration, the encoded data stored in the code memory 1261 is decoded by the decoder 1262,
Regenerate the quantized value. This value is calculated by the inverse quantizer 12
The inverse DCT is performed by the inverse DCT converter 1264.
T conversion is performed. Motion compensator 1269 for I-frame
Does not operate and outputs 0. At the time of the P-frame and the B-frame, the motion compensator 1269 operates and outputs a motion compensation prediction value. The adder 1265 is an inverse DCT converter 1264
Of the motion compensator 1269 and the output of the memory
66 and the memory 1267 or 1268.
On the other hand, a DC component representing the average value of the luminance data is output from the terminal 1212 from the inverse quantizer 1263 as contrast correction image information.

In the structure of the moving picture editing apparatus 112 described above, decoding of one frame is completed.
The target memory 1249 in the target 203 and the target encoder 1204
When the image data is stored in the target memory 1249 in the memory and the memory 1266 in the encoder 1205, the correction value calculator 1
213 obtains the following correction formula from the correction formula calculation algorithm described below using the image information for contrast correction.
That is, the correction formula f1 (x) for the corrector 1214 and the corrector 1
The correction formula f2 (x) for the 215 and the correction formula f3 for the corrector 1216
(x).

After that, the memory 126 in the decoder 1205
6, pixel values are read out by raster scan in the order of the pixels of the scanning line, corrected by the corrector 1216, and
Enter 17. The corrector 1216 corrects the input pixel value p by the correction formula f3 (x) according to the following formula, obtains a corrected pixel value P, and outputs the corrected pixel value P.

P = f3 (p) (1a) On the other hand, when the scan position reaches the position where the target image data of the target decoder 1203 is synthesized, the target decoder 1203
The mask information and the image data are read out from the mask memory 1243 and the object memory 1249 in the above, corrected by the corrector 1214, and input to the image synthesizer 1217. Corrector 12
At 14, the input pixel value p is corrected by the correction formula f1 (x) according to the following formula, and the corrected pixel value P
And output.

P = f1 (p) (2a) When the scan position reaches the position where the target image data of the target decoder 1204 is synthesized, the target decoder 1204
The mask information and the image data are read out from the mask memory 1243 and the object memory 1249, corrected by the corrector 1215, and input to the image synthesizer 1217. Corrector 12
In step 15, the input pixel value p is corrected by the correction formula f2 (x) according to the following formula, and the corrected pixel value P
And output.

P = f 2 (p) (3a) The image synthesizer 1217 outputs the mask information to the target decoder 120.
In the case of indicating the target image data of No. 3, the corrector 1214
, And outputs the pixel value from the corrector 1215 when the mask information indicates the target image data of the target decoder 1204, and outputs the pixel value from the corrector 1216 when none of these values is met. Then, the images are synthesized and output from the terminal 1218 to the encoder 1113.
FIG. 9 shows a background 1160, a person 1061, and a person 1052, which are images obtained by correcting the background 1050 and the person 1051.
1062, which is an image obtained by correcting
3 shows a state in which an image obtained by correcting 3 is combined with a person 1063 which is an image obtained. The encoder 113 converts the output image to MP
Encoded by EG-1 and transmitted to the communication network 11 via the transmitter 114
5 is sent.

In the above operation, the correction value calculator 1213
Operates according to the following.

First, the maximum value Max1, Max1 in the contrast correction image information 1222 from the target decoder 1203,
A minimum value Min1, an average value E1, and a variance R1 are obtained.

Next, the maximum value Max2 in the contrast correction image information 1223 from the target decoder 1204,
The minimum value Min2, the average value E2, and the variance R2 are obtained.

Next, the maximum value Max3, the minimum value Min3, the average value E3, and the variance R3 in the contrast correction image information 1224 from the decoder 1205 are obtained.

Then, the image information 12 for contrast correction is obtained.
22, 1223, and 1224, when the maximum value is 255 and the minimum value is 0 at most, if the maximum value is Max and the minimum value is Min, f 1 (x), f 2 (x),
f3 (x) is determined as follows.

F1 (x) = [{α (Max−Max1) + Max1} − {β (Min−Min1) + Min1}] / (Max1−Min1) × (x−Min1) + {α (Max−Max1) + Max1 }… (4a) f2 (x) = [{α (Max−Max2) + Max2} − {β (Min−Min2) + Min2}] / (Max2−Min2) × (x−Min2) + {α (Max−Max2) ) + Max2} (5a) f3 (x) = [{α (Max−Max3) + Max3} − {β (Min−Min3) + Min3}] / (Max3−Min3) × (x−Min3) + {α (Max) −Max3) + Max3} (6a) where α and β are weighting variables or weighting functions.

Otherwise, the maximum value of the contrast correction image information 1222, 1223, and 1224 is 255.
If the minimum value is 255 and the minimum value is 255. For example, if the maximum value is not 255 or the minimum value is not 0, the contrast correction image information 1222 is obtained.
1 (x), f2 (x) and f3 (x) are determined as follows.

F1 (x) = [{α (255−Max1) + Max1} − {β (0−Min1) + Min1}] / (Max1−Min1) × (x−Min1) + {α (255−Max1) + Max1 } (7a) f2 (x) and f3 (x) determine a function such that the variance difference | R2-R3 | As an example of the determination method, an example using a cubic spline including the following three contact points will be described.

For example, when R2> R3, f2 (x) = x (8a) f3 (x) = f31 (x); x ≦ E3 f32 (x); x> E3 (9a) where f31 (0) = 0; f31 (E3) = E3; f32 (255) =
255; f32 (E3) = E3; f (2) 31 (E3) = f (2) 32 (E
3); f (1) 31 (E3) = φ; f (1) 32 (E3) = ψ.

Further, α, β, φ and ψ are weighting variables or weighting functions.

Otherwise, f2 (x), f2 (x) and f3 (x)
Means the difference | R1-R2 |, | R1-R3 |, | R2-
A function is determined so that R3 | is small.

As an example of the determination method, an example using a cubic spline having the following three contact points will be described.

For example, when R1>R2> R3, f1 (x) = x (10a) f2 (x) = f21 (x); x ≦ E2 f22 (x); x> E2 (11a) f3 (x) = f31 (x); x ≦ E3 f32 (x); x> E3 (12a) where f21 (0) = 0; f21 (E2) = E2; f22 (255) =
255; f22 (E2) = E2; f (2) 21 (E2) = f (2) 22 (E
2); f (1) 21 (E3) = φ2; f (1) 22 (E2) = ψ2 and f
31 (0) = 0; f31 (E3) = E3; f32 (255) = 255;
f32 (E3) = E3; f (2) 31 (E3) = f (2) 32 (E3); f (1) 3
1 (E3) = φ3; f (1) 32 (E3) = ψ3.

Φ2, ψ2, φ3 and ψ3 are weighting variables or weighting functions.

As described above, according to the third embodiment, when the background image and the target image are separated, and when the coded images are combined, the feature amount of each image data is extracted and the object to be combined is extracted. By correcting the pixel values of the image, it is possible to perform image synthesis without a sense of incongruity, and it is possible to perform high-speed processing by using a DC component in block units for calculating a correction value.

In the third embodiment, MPEG-4 is used for the target encoding and MPEG-1 is used for the other encoding. However, the present invention is not limited to this. It does not matter anything that fulfills.

The memory configuration is not limited to this, and it goes without saying that processing may be performed by a line memory or the like.
Other configurations may be used.

Further, a part or all of the constituent elements may be replaced by a CP.
Of course, it may be realized by software operating on U or the like. Fourth Embodiment A fourth embodiment is a modification of the third embodiment in which the target decoders 1203 and 1204, the decoder 1205, and the correction value calculator 1213 are changed. Therefore, portions that are the same as those in the third embodiment are omitted, and only the changed portion will be described.

The moving image transmission system uses the configuration shown in FIG. 1 as in the first embodiment. The detailed configuration of the moving image editing apparatus 112 uses the configuration shown in FIG. 13 as in the third embodiment.

Next, the object decoder 1203 of the fourth embodiment,
The detailed configuration of 1204 will be described with reference to FIG. In FIG. 16, the detailed configuration of the target decoder 1203 is described, and the detailed configuration of the target decoder 1204 having the same configuration is omitted here.

FIG. 16 is a block diagram showing a detailed configuration of a target decoder according to Embodiment 4 of the present invention.

Reference numeral 1219 denotes a terminal to which encoded data from the receiver 110 is input. Reference numeral 1302 denotes a separator, which separates the encoded data of the mask information and the encoded data of the image data of the target image area from the encoded data. 1
Reference numeral 303 denotes a mask decoder that decodes mask information. 1
Reference numeral 304 denotes a mask memory for storing mask information. The mask information in the mask memory 1304 is output from a terminal 1207. Reference numeral 1305 denotes a code memory for storing the encoded data of the area of the target image. Reference numeral 1306 denotes a decoder for decoding the image data of the area of the target image. 1307 is an inverse quantizer. Reference numeral 1308 denotes a high-speed inverse DCT converter. The detailed configuration of the high-speed inverse DCT converter 1308 is shown in FIG.
1 is configured. 1309 is an adder. 13
Reference numerals 10, 1311 and 1312 are target memories for storing the areas of the reproduced target images. 1313 is a motion compensator. The image data in the target memory 1310 is
6 is output.

In the above configuration, the coded data of the mask information and the coded data of the area of the target image are separated from the input coded data by the separator 1302 and input to the mask decoder 1303 and the code memory 1305, respectively. The mask decoder 1303 decodes the encoded data to reproduce the mask information, and stores it in the mask memory 1304. The coded data stored in the code memory 1305 is decoded by the decoder 1306, and the quantized value is reproduced. This value is inversely quantized by an inverse quantizer 1307, and the high-speed inverse DCT converter 1
In step 308, inverse DCT is performed by Radix butterfly operation. In the case of an I-frame, the motion compensator 1313 does not operate and outputs 0. At the time of a P-frame and a B-frame, the motion compensator 1313 operates and outputs a motion compensation prediction value. The adder 1309 is a high-speed inverse DCT converter 1308
And the output of the motion compensator 1313 are added and stored in the target memory 1310 and the target memory 1311 or 1312. On the other hand, from the high-speed inverse DCT converter 1308,
The n-th stage Radix butterfly operation result is multiplexed, and is output to terminal 12 as image information for contrast correction.
08 is output.

Next, the detailed configuration of the decoder 1205 of the fourth embodiment will be described with reference to FIG.

FIG. 17 is a block diagram showing a detailed configuration of the decoder according to the fourth embodiment of the present invention.

Reference numeral 1221 denotes a terminal to which encoded data from the storage device 116 is input. Reference numeral 1322 denotes a code memory for storing encoded data. Reference numeral 1323 denotes a decoder that decodes encoded data. Reference numeral 1324 denotes an inverse quantizer. Reference numeral 1325 denotes a high-speed inverse DCT converter. The detailed configuration of the high-speed inverse DCT converter 1325 is the same as that shown in FIG. 1326 is an adder. 1327, 132
Reference numerals 8 and 1329 denote memories for storing image data obtained by decoding encoded data. 1330 is a motion compensator. The image data in the memory 1327 is
Output.

In the above configuration, the encoded data stored in the code memory 1322 is decoded by the decoder 1323,
Regenerate the quantized value. This value is calculated by the inverse quantizer 13
24, and is inversely quantized by a high-speed inverse DCT transformer 1325. Motion compensator 13 for I-frame
30 does not operate and outputs 0. At the time of a P-frame and a B-frame, the motion compensator 1330 operates and outputs a motion compensation prediction value. The adder 1326 is an inverse DCT converter 13
25 and the output of the motion compensator 1330 are added together and stored in the memory 1327 and the memory 1328 or 1329. On the other hand, from the high-speed inverse DCT converter 1325, the DC component or the n-th stage Radix butterfly operation result is multiplexed and output from the terminal 1212 as image information for contrast correction.

In the structure of the moving picture editing apparatus 112 described above, decoding of one frame is completed.
The target memory 1310 in the target 203 and the target encoder 1204
When the image data is stored in the target memory 1310 in the memory and the memory 1327 in the encoder 1205, the correction value calculator 1
213 obtains the following correction formula from the correction formula calculation algorithm described below using the image information for contrast correction.
That is, the correction formula f1 (x) for the corrector 1214 and the corrector 1
The correction formula f2 (x) for the 215 and the correction formula f3 for the corrector 1216
(x).

Thereafter, the memory 132 in the decoder 1205
7, the pixel values are read out by raster scan in the order of the pixels of the scanning line, corrected by the corrector 1216, and
Enter 17. The corrector 1216 corrects the input pixel value p by the correction formula f3 (x) according to the following formula, obtains a corrected pixel value P, and outputs the corrected pixel value P.

P = f3 (p) (13a) On the other hand, when the scan position reaches the position where the target image data of the target decoder 1203 is synthesized, the target decoder 1203
The mask information and the image data are read out from the mask memory 1304 and the target memory 1310, corrected by the corrector 1214, and input to the image synthesizer 217. Corrector 121
In step 5, the input pixel value p is corrected by the correction formula f1 (x) according to the following formula, and the corrected pixel value P
And output.

P = f1 (p) (14a) When the scan position reaches the position where the target image data of the target decoder 1204 is synthesized, the target decoder 1204
The mask information and the image data are read out from the mask memory 1304 and the target memory 1310, corrected by the corrector 1215, and input to the image synthesizer 1217. Corrector 12
At 14, the correction equation f2 (x) is applied to the input pixel value p.
Is performed according to the following equation, and a corrected pixel value P is obtained and output.

P = f 2 (p) (15a) The image synthesizer 1217 outputs the mask information to the target decoder 120.
In the case of indicating the target image data of No. 3, the corrector 1214
, And outputs the pixel value from the corrector 1215 when the mask information indicates the target image data of the target decoder 1204, and outputs the pixel value from the corrector 1216 when none of these values is met. By doing so, the images are synthesized and output from the terminal 1218 to the encoder 113. A background 1060 and a person 1061, which are images obtained by correcting the background 1050 and the person 1051, a person 1062 which is an image obtained by correcting the person 1052, and an image obtained by correcting the person 1053. The state of combining with a certain person 1063 is almost the same as FIG. 9 used in the third embodiment. However, the contrast is strictly different from that in the third embodiment.
The encoder 113 encodes the output image using MPEG-1, and sends the image to the communication network 115 via the transmitter 114.

In the above operation, the correction value calculator 1213
Works according to the following.

First, the maximum value Max1 in the contrast correction image information 1222 from the target decoder 1203,
A minimum value Min1, an average value E1, and a variance R1 are obtained.

Next, the maximum value Max2 in the contrast correction image information 1223 from the target decoder 1204,
The minimum value Min2, the average value E2, and the variance R2 are obtained.

Next, the maximum value Max3 in the contrast correction image information 1224 from the target decoder 1205,
The minimum value Min3, the average value E3, and the variance R3 are obtained.

Image information for contrast correction 1222, 1
When the maximum value is 255 and the minimum value is 0 among at most one of 223 and 1224, the maximum value is Ma
x, the minimum is Min, / f1 (x), / f2 (x), / f3
(x) is defined as follows.

/ F1 (x) = [{α (Max−Max1) + Max1} − {β (Min−Min1) + Min1}] / (Max1−Min1) × (x−Min1) + {α (Max−Max1) + Max1} (16a) / f2 (x) = [{α (Max−Max2) + Max2} − {β (Min−Min2) + Min2}] / (Max2−Min2) × (x−Min2) + {α (Max −Max2) + Max2} (17a) / f3 (x) = [{α (Max−Max3) + Max3} − {β (Min−Min3) + Min3}] / (Max3−Min3) × (x−Min3) + { α (Max−Max3) + Max3} (18a) where α and β are weighting variables or weighting functions.

Otherwise, of the contrast correction image information 1222, 1223, and 1224, the maximum value is 255.
In addition, when the minimum value is 255 and the minimum value is 255, for example, if the maximum value is not 255 or the minimum value is not 0, the image information 1222 for contrast correction is obtained.
f1 (x), / f2 (x), and / f3 (x) are determined as follows.

/ F1 (x) = [{α (255−Max1) + Max1} − {β (0−Min1) + Min1}] / (Max1−Min1) × (x−Min1) + {α (255−Max1) + Max1} (19a) / f2 (x) and / f3 (x) determine a function such that the variance difference | R2-R3 | is small. As an example of the determination method, an example using a cubic spline including the following three contact points will be described.

For example, when R2> R3, / f2 (x) = x (20a) / f3 (x) = / f31 (x); x ≦ E3 / f32 (x); x> E3 (21a However, / f31 (0) = 0; / f31 (E3) = E3; / f32 (25
5) = 255; / f32 (E3) = E3; / f ⁽²⁾ 31 (E3) = / f
⁽²⁾ 32 (E3); / f ⁽¹⁾ 31 (E3) = φ; / f (1) 32 (E3) = ψ.

Α, β, φ and ψ are weighting variables or weighting functions.

Otherwise, / f2 (x), / f2 (x) and / f3
(x) is the variance difference | R1-R2 |, | R1-R3 |, | R
A function that reduces 2-R3 | is determined.

As an example of the determination method, an example using a cubic spline having the following three contact points will be described.

For example, when R1>R2> R3, / f1 (x) = x (22a) / f2 (x) = / f21 (x); x ≦ E2 / f22 (x); x> E2 ... (23a) / f3 (x) = / f31 (x); x ≦ E3 / f32 (x); x> E3 (24a) where / f21 (0) = 0; / f21 (E2) = E2; f22 (255)
= 255; f22 (E2) = E2; / f ⁽²⁾ 21 (E2) = / f ⁽²⁾ 22
(E2); / f ⁽¹⁾ 21 (E3) = φ2; / f ⁽¹⁾ 22 (E2) = ψ2 and / f31 (0) = 0; / f31 (E3) = E3; / f32 (255)
= 255; / f32 (E3) = E3; / f ⁽²⁾ 31 (E3) = f ⁽²⁾ 32
(E3); / f ⁽¹⁾ 31 (E3) = φ3; / f ⁽¹⁾ 32 (E3) = ψ3.

Φ2, ψ2, φ3 and ψ3 are weighting variables or weighting functions.

The current correction equation is determined as follows based on the correction equation one frame before.

F1 (x) = f1 (x) + γ (/ f1 (x) −f1 (x)) (25a) f2 (x) = f2 (x) + γ (/ f2 (x) −f2 (x) ) (26a) f3 (x) = f3 (x) + γ (/ f3 (x) −f3 (x)) (27a) Here, γ is a time-variation tracking weight variable of the correction formula.

As described above, according to the fourth embodiment, when the background image and the target image are separated, and when the coded images are combined, the feature amount of each image data is extracted and the object to be combined is extracted. By correcting the pixel values of the image, it is possible to perform image synthesis without a sense of discomfort. Further, in consideration of the size of the target image and the calculation speed, a correction value is calculated by selectively selecting a DC component or an inverse DCT result of a 2 × 2 or 4 × 4 low frequency component or an 8 × 8 inverse DCT result. In this case, flexible and highly accurate processing can be performed. Furthermore, by causing the contrast correction processing to slowly follow the time change, it is possible to perform image synthesis without discomfort even for an image that changes rapidly.

In the fourth embodiment, the description has been made using MPEG-4 for the target encoding and MPEG-1 for the other encoding. However, the present invention is not limited to this. It does not matter anything that fulfills.

Further, the memory configuration is not limited to this, and it goes without saying that processing may be performed by a line memory or the like.
Other configurations may be used.

A part or all of the constituent elements may be replaced by a CP.
Of course, it may be realized by software operating on U or the like.

Finally, the processing flow of the processing executed in the first to fourth embodiments will be described with reference to FIG.

FIG. 18 is a flowchart showing the processing flow of the processing executed in the present invention. First, step S1
At 01, the input encoded data is separated into encoded data of a background image and encoded data of a target image. In step S102, a background feature of the encoded data of the background image is extracted. In step S103, a target feature of the encoded data of the target image is extracted. In step S104, the encoded data of the background image is decoded to generate a background reproduced image. In step S105, the coded data of the target image is decoded to generate a target reproduced image. In step S106, the target playback image is corrected based on the extracted background features and target features. The details of this correction are as described in each embodiment. In step S107, the background reproduced image and the corrected target reproduced image are combined.

Even if the present invention is applied to a system including a plurality of devices (for example, a host computer, an interface device, a reader, a printer, etc.), a device including one device (for example, a copying machine, a facsimile machine) Etc.).

Further, an object of the present invention is to supply a storage medium storing a program code of software for realizing the functions of the above-described embodiments to a system or apparatus, and to provide a computer (or CPU) of the system or apparatus.
And MPU) read and execute the program code stored in the storage medium.

In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiment, and the storage medium storing the program code constitutes the present invention.

As a storage medium for supplying the program code, for example, a floppy disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD
-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

When the computer executes the readout program codes, not only the functions of the above-described embodiment are realized, but also the OS (Operating System) running on the computer based on the instructions of the program codes. ) May perform some or all of the actual processing, and the processing may realize the functions of the above-described embodiments.

Further, after the program code read from the storage medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, based on the instruction of the program code, It goes without saying that the CPU included in the function expansion board or the function expansion unit performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments.

[0210]

As described above, according to the present invention,
It is possible to provide an image processing apparatus, a method thereof, and a computer-readable memory capable of easily executing the synthesis of a plurality of images and generating a synthesized image with good image quality.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a moving image transmission system according to a first embodiment of the present invention.

FIG. 2 is a diagram illustrating an example of a region of a target image according to the first embodiment of the present invention.

FIG. 3 is a diagram illustrating an example of mask information according to the first embodiment of the present invention.

FIG. 4 is a diagram illustrating an example of an encoded image according to the first embodiment of the present invention.

FIG. 5 is a block diagram illustrating a detailed configuration of a target encoding unit according to the first embodiment of the present invention.

FIG. 6 is a block diagram illustrating a detailed configuration of the moving image editing apparatus according to the first embodiment of the present invention.

FIG. 7 is a block diagram illustrating a detailed configuration of a target decoder according to the first embodiment of the present invention.

FIG. 8 is a block diagram illustrating a detailed configuration of a decoder according to the first embodiment of the present invention.

FIG. 9 is a diagram illustrating an example of combining target images according to the first embodiment of the present invention.

FIG. 10 is a block diagram illustrating a detailed configuration of a target decoder according to Embodiment 2 of the present invention.

FIG. 11 is a block diagram illustrating a detailed configuration of a high-speed inverse DCT converter according to a second embodiment of the present invention.

FIG. 12 is a block diagram illustrating a detailed configuration of a decoder according to Embodiment 2 of the present invention.

FIG. 13 is a block diagram illustrating a detailed configuration of a moving image editing device according to a third embodiment of the present invention.

FIG. 14 is a block diagram illustrating a detailed configuration of a target decoder according to Embodiment 3 of the present invention.

FIG. 15 is a block diagram illustrating a detailed configuration of a decoder according to Embodiment 3 of the present invention.

FIG. 16 is a block diagram illustrating a detailed configuration of a target decoder according to Embodiment 4 of the present invention.

FIG. 17 is a block diagram illustrating a detailed configuration of a decoder according to Embodiment 4 of the present invention.

FIG. 18 is a flowchart showing a processing flow of processing executed in the present invention.

FIG. 19 is a diagram illustrating a configuration of a conventional encoding system.

FIG. 20 is a diagram illustrating an example of an image according to the first embodiment of the present invention.

FIG. 21 is a diagram illustrating an example of an image according to the first embodiment of the present invention.

FIG. 22 is a diagram illustrating an example of encoded data according to the first embodiment of the present invention.

[Explanation of symbols]

101, 102 TV camera 103 Target extractor 104, 113 Encoder 105 Target encoder 106, 107 Transmitter 108, 109 Communication line 110, 111 Receiver 112 Video editing device 115 Communication network 116 Storage device 121, 122 Terminal 123 mask memory 124 mask encoder 125, 138, 139 target memory 126 average value calculator 127 block former 128 frame mode setter 129 differentiator 130 DCT converter 131 quantizer 132 encoder 133 synthesizer 135 inverse quantizer 136 Inverse DCT converter 137 Adder 140 Motion compensator 200, 201, 202, 206, 207, 208, 2
09, 210, 211, 212, 218, 225 Terminal 203, 204 Object decoder 205 Decoder 213 Correction value calculator 214, 215, 216 Corrector 217 Image synthesizer 241 Separator 242 Mask decoder 243 Mask memory 244, 301 Code memories 245, 302 Decoders 246, 247, 303, 304 Demultiplexers 248, 255, 262, 305, 312, 319 Inverse quantizers 249, 256, 263, 306, 313, 320 Inverse DCT transformers 250, 257 264, 307, 314, 321 Adders 251, 252, 253, 258, 259, 260, 2
65, 266, 267 Target memory 254, 261, 268, 311, 318, 325 Motion compensator 308, 309, 310, 315, 316, 317, 3
22, 323, 324 memory 270, 271, 272, 327, 328, 329 Buffer 269, 273, 326, 330 Color signal converter 401 Separator 402 Mask decoder 403 Mask memory 404, 452 Code memory 405, 453 Decoder 406 , 407, 454, 455 Demultiplexer 408, 415, 422, 456, 463, 470 Dequantizer 409, 416, 423, 457, 464, 471 High-speed inverse DCT converter 410, 417, 424, 458, 465, 472 Adders 411, 412, 413, 418, 419, 420, 4
25, 426, 427 Target memory 414, 421, 428, 462, 469, 476 Motion compensator 459, 460, 461, 466, 467, 468, 4
73, 474, 475 memories 430, 431, 432, 478, 479, 480 buffers 429, 433, 477, 481 color signal converters 1200, 1201, 1202, 1206, 1207,
1208, 1209, 1210, 1211, 1212,
1218, 1225 Terminals 1203, 1204 Target decoder 1205 Decoder 1213 Correction value calculator 1214, 1215, 1216 Corrector 1217 Image synthesizer 1241, 1302 Separator 1242, 1303 Mask decoder 1243, 1304 Mask memory 1244, 1261, 1305 , 1322 Code memory 1245, 1262, 1306, 1323 Decoder 1246, 1263, 1307, 1324 Inverse quantizer 1247, 1264 Inverse DCT transformer 1248, 1265, 1309, 1326 Adder 1249, 1250, 1251, 1310, 1311,
1312 Target memory 1252, 1269, 1313, 1330 Motion compensator 1266, 1267, 1268, 1327, 1328,
1329 Memory 1308, 1325 High-speed inverse DCT converter

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5C059 KK37 MA00 MA05 MA09 MA23 MB12 MB22 MC23 PP05 PP06 PP07 PP15 PP16 PP26 PP27 PP29 RC19 SS07 SS20 SS26 TA01 TA44 TB07 TC02 TC04 TC24 TD01 TD02 TD03 TD04 TD10 TD12 UA05 UA25 066 AA13 CA09 CA17 EC06 ED04 EE01 GA01 GA02 GA05 GA22 GA32 HA01 JA02 KE09 KE16

Claims (35)

[Claims] 1. An image processing apparatus for synthesizing a plurality of images, comprising: a background feature extracting unit configured to extract a background feature from encoded data of at least one background image; and an image processing unit configured to extract an image from encoded data of at least one target image. A target feature extracting unit that extracts a target feature including statistical information of information; a background decoding unit that decodes encoded data of the background image to generate a background reproduction image; and decodes encoded data of the target image. A target decoding unit for generating a target reproduction image; a correction unit for correcting the target reproduction image based on the background feature and the target characteristic; and combining the background reproduction image and the target reproduction image corrected by the correction unit. An image processing apparatus comprising: 2. The target feature extraction unit includes a calculation unit that calculates a histogram based on statistical information of the image information, and the correction unit determines a correction method of the target image based on the histogram. 2. The method according to claim 1, wherein
An image processing apparatus according to claim 1. 3. The image processing apparatus according to claim 1, wherein the target feature extracting unit extracts DC information of a block image included in the encoded data as statistical information of the image information.
An image processing apparatus according to claim 1. 4. The image processing apparatus according to claim 1, wherein the target feature extracting unit extracts low-frequency information of a block image included in the encoded data as statistical information of the image information. apparatus. 5. One or both of said background decoding means and said target decoding means are decoding means for decoding quantized data from said encoded data, and inverse quantization for calculating frequency band data from said quantized data. And a high-speed inverse discrete cosine transform unit for calculating spatial domain data from the frequency domain data.The high-speed inverse discrete cosine transform unit includes an output unit that outputs a result of a Radix butterfly operation of an arbitrary number of stages. The image processing apparatus according to claim 4, wherein the target feature extracting unit extracts the Radix butterfly operation result of the arbitrary number of stages as low-frequency information of image information. 6. The apparatus according to claim 1, wherein the correction unit includes a time series adaptation unit that gradually changes an input / output relationship between a signal input to the correction unit and an output signal according to a time series. Image processing device. 7. The target feature extracting means extracts a maximum value and a minimum value of a pixel value from DC information or low frequency information of a block image included in the encoded data as statistical information of the image information. The image processing apparatus according to claim 1, wherein: 8. The target feature extracting means extracts, as statistical information of the image information, a variance and an average value of pixel values from DC information or low frequency information of a block image included in the encoded data. The image processing apparatus according to claim 1, wherein: 9. The image processing apparatus according to claim 1, wherein the correction unit performs a linear transformation on the target image. 10. The apparatus according to claim 1, wherein the correction unit performs a piecewise spline transformation on the target image. 11. A detecting means for detecting presence or absence of a significant color bias from the target feature extracted by the target feature extracting means, and correcting the color bias based on a detection result of the detecting means. 5. A color correction means comprising:
The image processing device according to any one of the above. 12. The detecting means satisfies a condition that, based on statistical information included in the extracted target feature, a difference absolute value of a variance from a difference absolute value of an average value between color signals is equal to or less than a threshold. Detecting whether or not there is significant color bias in a specific area of the histogram based on the statistical information when the condition is satisfied. An image processing apparatus according to claim 1. 13. The image processing apparatus according to claim 11, wherein the color correction unit performs linear correction so that the maximum value of each color signal is equal. 14. The image processing apparatus according to claim 11, wherein the color correction unit does not correct a blue signal. 15. The detecting means for detecting a significant contrast difference from the target feature extracted by the target feature extracting means and the background feature extracted by the background feature extracting means, The image processing apparatus according to claim 1, further comprising: a contrast correction unit configured to correct a contrast based on a detection result of the detection unit. 16. The detecting means extracts a maximum value and a minimum value of a pixel value obtained from the target feature and the background feature, respectively, and the contrast correcting means extracts an object having a different maximum value or the minimum value. 16. The image processing according to claim 15, wherein the correction is performed so that the difference absolute value between the maximum pixel value and the minimum pixel value is reduced between the image and the background image. apparatus. 17. The detection unit extracts a maximum value and a minimum value of a pixel value obtained from the target feature and the background feature, respectively, and the contrast correction unit determines whether the maximum value or the minimum value is 16. The image processing apparatus according to claim 15, wherein the correction is performed such that the difference absolute value of the variance is reduced between the substantially equal target image and the background image. 18. An image processing method for synthesizing a plurality of images, comprising: extracting a background feature from encoded data of at least one background image; and extracting an image from encoded data of at least one target image. A target feature extraction step of extracting a target feature including statistical information of information; a background decoding step of decoding encoded data of the background image to generate a background reproduction image; and decoding encoded data of the target image. A target decoding step of generating a target playback image; a correction step of correcting the target playback image based on the background feature and the target feature; and synthesizing the background playback image and the target playback image corrected in the correction step. An image processing method comprising: 19. The target feature extracting step includes a calculating step of calculating a histogram based on the statistical information of the image information, and the correcting step determines a correction method of the target image based on the histogram. 2. The method according to claim 1, wherein
9. The image processing method according to 8. 20. The image processing method according to claim 18, wherein the target feature extracting step extracts DC information of a block image included in the encoded data as statistical information of the image information. 21. The image processing apparatus according to claim 18, wherein the target feature extracting step extracts low-frequency information of a block image included in the encoded data as statistical information of the image information. Method. 22. One or both of the background decoding step and the target decoding step include: a decoding step of decoding quantized data from the encoded data; and an inverse quantization step of calculating frequency band data from the quantized data. A high-speed inverse discrete cosine transform step of calculating spatial domain data from the frequency domain data, and the high-speed inverse discrete cosine transform step includes an output step of outputting a Radix butterfly operation result of an arbitrary number of stages, 22. The image processing method according to claim 21, wherein the target feature extracting step extracts the Radix butterfly operation result of the arbitrary number of stages as low-frequency information of image information. 23. The correction method according to claim 18, wherein the correction step includes a time-series adaptation step of gradually changing an input / output relationship between a signal input to the correction step and an output signal according to a time series. Image processing method. 24. The target feature extracting step includes extracting a maximum value and a minimum value of a pixel value from DC information or low frequency information of a block image included in the encoded data as statistical information of the image information. 19. The method according to claim 18, wherein
The image processing method according to 1. 25. The target feature extracting step includes extracting, as statistical information of the image information, a variance and an average value of pixel values from DC information or low frequency information of a block image included in the encoded data. 19. The image processing method according to claim 18, wherein: 26. The image processing method according to claim 18, wherein the correcting step performs a linear transformation on the target image. 27. The image processing method according to claim 18, wherein the correcting step performs a piecewise spline transformation on the target image. 28. The correcting step includes detecting a presence or absence of a significant color bias from the target feature extracted in the target feature extracting step, and correcting the color bias from the detection result of the detecting step. 22. The image processing method according to claim 18, further comprising: 29. The detecting step satisfies a condition that, based on statistical information included in the extracted target feature, a difference absolute value of a variance from a difference absolute value of an average value between color signals is equal to or less than a threshold. 29. The method according to claim 28, further comprising: detecting whether or not there is a significant color bias in a specific area of the histogram based on the statistical information when the condition is satisfied. The image processing method according to 1. 30. The image processing method according to claim 28, wherein in the color correction step, linear correction is performed so that the maximum value of each color signal becomes equal. 31. The image processing method according to claim 28, wherein in the color correction step, no correction is performed on a blue signal. 32. The detecting step, comprising: detecting a significant contrast difference between the target feature extracted in the target feature extracting step and the background feature extracted in the background feature extracting step. 22. The image processing method according to claim 18, further comprising: a contrast correction step of correcting contrast based on a detection result of the detection step. 33. The detection step extracts a maximum value and a minimum value of a pixel value obtained from the target feature and the background feature, respectively. The contrast correction step includes a step of extracting a target having a different maximum value or the minimum value. 33. The image processing according to claim 32, wherein the image and the background image are corrected such that the absolute value of the difference between the maximum pixel value and the absolute value of the minimum pixel value decreases. Method. 34. The detecting step extracts a maximum value and a minimum value of pixel values obtained from the target feature and the background feature, respectively, and the contrast correction step determines whether the maximum value or the minimum value is 33. The image processing method according to claim 32, wherein the correction is performed so that the difference absolute value of the variance is reduced between the substantially equal target image and the background image. 35. A computer readable memory storing a program code for image processing for synthesizing a plurality of images, the program code for a background feature extracting step of extracting a background feature from encoded data of at least one background image; A program code for a target feature extraction step of extracting a target feature including statistical information of image information from encoded data of at least one target image; and decoding the encoded data of the background image to generate a background reproduced image. A program code for a background decoding step, a program code for a target decoding step for generating a target playback image for decoding the encoded data of the target image, and correcting the target playback image based on the background feature and the target feature. The program code of the correction step, the background reproduced image and the target reproduced image corrected in the correction step are combined. Computer-readable memory comprising: a program code for a synthesis process to be performed.