JP2001189931A - Image processing method, image processor, and data recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor, that encodes a moving image signal of an object and can reduce the quantity of phase compensation processing of pixel values and arithmetic processing of filter arithmetic operation, without losing the image quality of a composited image, when compositing the object as a foreground with a composited image as a background. SOLUTION: The image processor is provided with a shape coder U1, that applies coding processing including local decoding processing to the shape signal Sin of an object (foreground), a texture coder U3 that applies coding processing including the local decoding processing to a texture signal Tin of the object, and a pixel compensation device U9a, that applies compensation processing to a local decoding texture signal Ltout by replacing pixel values of pixels at the outside of the object with pixel values of the pixels in the object, and a compensated texture Pa obtained by the compensation processing is composited with the texture signal Vin of the composited image (background).

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing method, an image processing apparatus, and a data recording medium, and more particularly to an encoding including a local decoding process for a moving image signal corresponding to an object (object image) constituting a scene. The present invention relates to a decoding process for a processed or encoded moving image signal, and a combining process for combining a moving image signal obtained by the local decoding process or the decoding process with an image to be combined.

[0002]

2. Description of the Related Art In recent years, the multimedia era in which voices, images, and other data are handled in an integrated manner has been entered, and conventional information media, that is, means for transmitting information such as newspapers, magazines, televisions, radios, and telephones to humans. Has been taken up as an object of multimedia. Generally, multimedia means not only characters, but also figures, sounds, and especially images, etc., are simultaneously associated with each other. However, in order to use conventional information media as multimedia, the information must be converted to digital format. Is an essential condition.

However, when the amount of information handled by each of the above information media is estimated as the amount of digital information, the amount of information per character is 1 to 2 bytes, whereas the amount of information per character is 1 to 2 bytes per second. 64Kbits (telephone quality), plus 100Mbi per second for video
An information amount equal to or greater than ts (current television broadcast quality) is required.
Therefore, in most of the information media mentioned above,
It is not realistic to handle the vast amount of information in digital form. For example, a videophone is 64Kbps ~
Integrated Services Digital Network (ISDN) with 1.5Mbps transmission speed
Has already been put into practical use, but it is impossible to send video information of a television camera by ISDN as it is.

Therefore, what is needed is an information compression technique. For example, for videophones, ITU-
T (International Telecommunication Union, Telecommunication Standardization Sector) 261 and H.E. H.263 video compression technology is used. Further, according to the information compression technique of the MPEG1 standard, it is possible to store image information together with audio information in a normal music CD (compact disc).

Here, MPEG (Moving Picture Exper)
ts Group) is an international standard related to a technology for compressing moving image data (image signals of moving images). MPEG1 compresses moving image data to 1.5 Mbps, that is, TV signal information to about 1/100. This is a technical standard.
In addition, since the transmission speed for the MPEG1 standard is mainly limited to about 1.5 Mbps, the moving picture data is compressed to 2 to 15 Mbps in MPEG2 standardized to meet the demand for higher image quality. .

Further, at present, MPEG1, MPEG2
And the working group that has promoted standardization (ISO / IEC JTC1 / SC2
9 / WG11), a moving image data compression technology that enables encoding processing and signal operation on an object basis and realizes new functions required in the multimedia age is being standardized as MPEG4. At first, MPEG4 aimed to standardize a low bit rate encoding method.
The standardization target has been extended to a more general-purpose encoding process of a high bit rate corresponding to an interlaced image.

One of the features of MPEG4 is a mechanism for simultaneously encoding and transmitting moving picture signals corresponding to a plurality of picture sequences (moving pictures). This mechanism enables one scene (an image of one display screen) to be composed of a plurality of images (objects).

For example, in MPEG4, the foreground (object) and the background (object) constituting one scene are separated as separate image sequences, and the frame frequency,
Image quality, bit rate, etc. can be changed.
In MPEG4, images of a plurality of image sequences are arranged in a horizontal or vertical direction so as to form a multi-display screen on one display screen, and a user extracts or enlarges only images of a desired image sequence. And can be displayed.

In MPEG4, the background is
In general, encoding processing is performed to encode only a texture signal indicating a picture (luminance and color) as a moving image signal in the same way as MPEG2. A process of simultaneously encoding a shape signal indicating the shape of the object is performed.
Generally, the encoding process for the foreground is known as an encoding process for each object which encodes a texture signal and a shape signal corresponding to each object constituting a scene.

FIG. 13 is a diagram for explaining an encoding process for each object. FIG. 13A shows a pattern Tin obtained from a texture signal of an object placed on the background as a foreground during image synthesis. FIG. 13B shows a shape Sin obtained from a shape signal of the object as the foreground.
In (b), the oblique lattice pattern portion represents the inside of the object. FIG. 13C shows a pattern (combined image) Vin obtained from a texture signal of an object placed behind the foreground as a background during image composition, and FIG. The synthesized image Vout is obtained by the synthesis processing with the foreground, that is, obtained by superimposing the foreground on the background.

Originally, an object has a pixel having a predetermined pixel value only inside the object, and no pixel exists outside the object. Therefore, the pixel value outside the object is undefined. Therefore,
In the composite image Vout obtained by the above-described composite processing, FIG.
In the area corresponding to the inside of the foreground object (the oblique lattice pattern portion) shown in FIG. 3B, a foreground pattern Tin (see FIG. 13A) is displayed, and the foreground object shown in FIG. The area corresponding to (the part other than the diagonal lattice pattern part) has a background pattern Vin.
(See FIG. 13C) will be displayed.

Now, in the image processing of MPEG4,
The compression rate can be improved by performing the encoding process using the pixel value correlation between the screens on the moving image signal corresponding to each object in the same manner as the MPEG2 inter-screen encoding process. In the encoding process for each object, since the size of the object changes depending on the time, the compression ratio cannot be improved by a simple inter-picture encoding process such as MPEG2.

Hereinafter, this point will be briefly described. FIG. 14 is a diagram for describing inter-frame encoding processing for a moving image signal corresponding to one object on a per-object basis, and the size of the object to be encoded increases with time. FIG. FIGS. 14 (a), (b),
(c) shows time t1, time t2, and time t of the object, respectively.
3 (t1 <t2 <t3).

For example, a large object (time t2)
In the inter-screen encoding process of encoding the texture signal of (i) with reference to the texture signal (time t1) of the small object, the pixel whose pixel value is referred to is located outside the small object. Sometimes.

That is, as described with reference to FIG. 13, since the pixel values of the non-object pixels constituting the texture signal corresponding to each object are undefined, the pixel values of the non-object pixels (undefined values) are referred to. When the inter-picture encoding process of the texture signal is performed, it is not always possible to correctly decode the encoded texture signal. In addition, if the range in which the pixel value can be referred to is limited to the area within the object, the problem of referring to an indefinite value as the pixel value during the inter-screen encoding process can be solved. May be very small, and the compression efficiency by inter-picture coding may hardly improve.

Therefore, in MPEG4, the pixel value of the pixel outside the object is generated from the pixel value of the pixel inside the object, and the generated pixel value is replaced with the pixel value of the pixel outside the object. A mechanism has been introduced to make it possible to refer to the pixel values of the image data in the inter-screen encoding process. This mechanism is called pixel value compensation (padding) processing, and has been recognized to have a significant effect on improving the compression ratio.

In MPEG4, encoding processing of a moving image signal corresponding to each object is performed in units of an image space called, for example, 16 × 16 pixels called a macroblock. Pixel value compensation processing is also performed in units of macroblocks. Done. The macro block forms an image space (rectangular area) corresponding to the object including one object. A texture rectangular area is formed by a texture signal corresponding to the foreground, and a shape is determined by a shape signal corresponding to the foreground. A rectangular area is formed. FIG. 15 is a diagram for explaining the pixel value compensation processing.

As shown in FIG. 15A, the texture rectangular area TR of the object treated as the foreground when synthesizing the image is
The texture rectangular area TR is obtained from the texture signal corresponding to the foreground, and includes an object as a picture Tin. Further, as shown in FIG. 15B, the shape rectangular region SR of the object treated as the foreground is obtained from the shape signal corresponding to the foreground, and the shape rectangular region SR has the shape Sin
As an object.

Here, the texture rectangular area TR
Includes a boundary macroblock Bti including the boundary of the object and an extra-object macroblock Bto located outside the object, and the shape rectangular region SR includes a boundary macroblock Bsi including the boundary of the object and a boundary macroblock Bsi. An extra-object macroblock Bso located on the outside is included.

The pixel value compensation processing for the texture signal is performed as follows. First, a macroblock including a part of an object (boundary macroblock) is subjected to a first pixel padding process called simplified padding. In the first compensation process, as shown in FIG. 15C, the pixel value of the pixel located at the object boundary in the macroblock becomes a continuous value with respect to the boundary macroblock Bti. The pixel value of the extra-object pixel is supplemented by the pixel value of the intra-object pixel so that the pixel value between the in-vivo pixel and the extra-object pixel is the same or close to each other. Here, the macro block Bpt
Reference numeral 1 denotes a macroblock subjected to the first pixel compensation processing, and a texture rectangular area TRpad1 is a rectangular area obtained by a texture signal subjected to the first pixel compensation processing.

Next, for a macroblock that does not include an object (macroblock outside the object), extended compensation is performed.
A second pixel padding process called a padding process is performed. In this extended compensation processing, as shown in FIG.
Pixel values that are continuous, that is, the same as or close to the pixel values of the pixels in the macroblock Bpt1 that have been subjected to the simple interpolation processing, are replaced with the pixel values of the pixels of the macroblock outside the object. Here, the macroblock Bpt2 is a macroblock that has been subjected to the second pixel compensation processing, and the texture rectangular area TRpad2 is a texture that has been subjected to the second pixel compensation processing in addition to the first pixel compensation processing. This is a rectangular area obtained by a signal.

As described above, the pixel value of the pixel outside the object is compensated so that the pixel has a continuous pixel value at the object boundary portion. Has a pixel value substantially the same as the pixel value of the pixel of the macroblock in the object,
When the inter-screen encoding process is performed on an object having a small size or an object having a complicated shape, the motion compensation residual is reduced as a reference image corresponding to a macroblock to be encoded. Such a reference image can be obtained from any position on the reference screen, and the compression ratio can be improved.

FIG. 16 is a block diagram of a conventional moving picture coding apparatus. The configuration of this video encoding device is M
It is obtained by mounting functional blocks as described in the PEG4 standard. This video encoding device 2
00a refers to the processed shape signal Sref for the shape signal (encoded shape signal) Sin to be processed,
A shape encoder U1 that performs a shape encoding process including a local decoding process for each object, outputs a locally decoded shape signal Sout, and outputs a shape encoded stream Sstr
And temporarily stores the locally decoded shape signal Sout,
A shape memory U2a which outputs the processed shape signal Sref to the shape encoder U1.

Also, the moving picture coding apparatus 200a
For the texture signal (encoded texture signal) Tin to be processed, the processed texture signal Tref
A pixel code for performing a pixel encoding process including a local decoding process for each object, outputting a locally decoded texture signal Tout obtained by the local decoding process, and outputting a texture encoded stream Tstr It is determined whether each pixel constituting the texture rectangular area is an intra-object pixel located inside the object or an extra-object pixel located outside the object based on the transform unit U3 and the local decoded shape signal Sout. Then, the local decoding texture signal LTout is subjected to compensation processing, and a pixel compensator U9a that outputs the compensation texture signal Pad1 and the compensation texture signal Pad1 are temporarily stored, and the texture is processed as the processed texture Tref. And a texture memory U5a for outputting to the encoder U3. Here, the pixel compensator U9a performs a simple compensating process for the boundary macroblock shown in FIG. 15C as a compensating process for the local decoded texture signal LTout, and FIG.
This is configured to perform the extended compensation processing on the extra-object macroblock shown in FIG. 5 (d).

Further, the moving picture coding apparatus 200a determines whether each pixel constituting the rectangular area is a pixel inside the object or a pixel outside the object based on the locally decoded shape signal Sout. Based on the synthesized position indicating signal Pos, the synthesized image signal Vin corresponding to the synthesized image as the background
And a pixel synthesizer U7 that performs a synthesis process of synthesizing the locally decoded texture signal Tout so that the image of the foreground rectangular area overlaps the synthesized image, and outputs an image signal Vout corresponding to the synthesized image. are doing.

In the moving picture coding apparatus as described above, the coding processing of the moving picture signal corresponding to the object is performed while monitoring what kind of synthesized picture is generated on the decoding side. Done. For this reason, a conventional moving picture coding apparatus generally has a pixel synthesizer, and the output is input to a display device. Here, the combined position indicating signal Pos is information such as a position at which the object is combined and a size at which the object is combined, and is designated by a sender or a receiver (user).

FIG. 17 is a diagram for explaining the operation of a conventional moving picture coding apparatus, and shows a flow of coding processing in the apparatus. First, the video encoding device 20
When the encoded shape signal Sin and the encoded texture signal Tin corresponding to the object as the foreground are input to 0a, the encoded shape signal Sin is locally decoded by the shape encoding unit U1. Is performed, and the locally decoded shape signal Sout and the shape encoded stream Sstr
Is output (step S1). At this time, the locally decoded shape signal Sout is stored in the shape memory U2a.

The encoded texture signal Tin is subjected to a texture encoding process including a local decoding process in the texture encoding unit U3, so that the locally decoded texture signal Tout and the texture encoded stream Ts are processed.
tr is output (step S2). Next, the locally decoded shape signal Sout and the locally decoded texture signal Tout
Is input to the pixel synthesizer U7, a synthesizing process of synthesizing the locally decoded texture signal Tout with the image signal Vin corresponding to the image to be synthesized is performed based on the synthesizing position indication signal Pos from the outside. The signal Vout is output (Step S3).

The locally decoded texture signal T
out is subjected to a supplementing process in the pixel compensator U9a (step S4), and is stored in the texture memory U5a as the compensated texture signal Pad1 (step S4).
5). Here, in the pixel compensator U9a, as a compensation process for the locally decoded texture signal LTout, a simple compensation process for the boundary macroblock shown in FIG. 15C and a compensation process for the extra-object macroblock shown in FIG. Extended compensation processing is performed.

Thereafter, it is determined whether or not the encoding process has been completed for all frames of one object (step S6). If the processing is completed and the encoding processing for all the frames has not been completed, the processing in steps S1 to S6 is performed again.

FIG. 18 is a block diagram for explaining a conventional moving picture decoding apparatus. This video decoding device 20
0b receives the shape-encoded stream Sstr from the video encoding device 200a, performs a shape-decoding process on the shape-encoded stream Sstr with reference to the processed shape signal Sref for each object, and performs decoding. It has a shape decoder U10 that outputs a shape signal Sout, and a shape memo U2 that temporarily stores the decoded shape signal Sout and outputs the processed shape signal Sref to the shape decoder U10.

Further, the moving picture decoding apparatus 200b performs a pixel decoding process on the texture coded stream Tstr from the moving picture coding apparatus 200a with reference to the processed texture signal, thereby obtaining the decoded texture signal Tstr.
out, and based on the decoded shape signal Sout, a determination is made as to whether each pixel of the texture rectangular area is a pixel inside the object or a pixel outside the object. A pixel compensator U9b that performs a compensation process on the decoded texture signal Tout to be formed and outputs a compensated texture signal Pb, and temporarily stores the compensated texture signal Pb, and texture-decodes the processed texture signal Tref. Chemical U1
1 and a texture memory U5b for outputting the data to the memory 1.

Further, based on the decoded shape signal Sout, the moving picture decoding apparatus 200b determines whether each pixel in the texture rectangular area of the corresponding object is a pixel inside the object or a pixel outside the object. Then, the locally decoded pixel signal Tout is synthesized based on the synthesized position indicating signal Pos from the outside so that the image of the texture rectangular area overlaps the synthesized image signal Vin corresponding to the synthesized image on the synthesized image. And a pixel synthesizer U9b that outputs an image signal Vout corresponding to the synthesized image. Here, the pixel compensator U9b performs a simple compensation process on the boundary macroblock shown in FIG. 15C and an extension process on the extra-object macroblock shown in FIG. It is configured to perform compensation processing.

FIG. 19 is a diagram for explaining the operation of the conventional moving picture decoding apparatus, and shows the flow of decoding processing in the apparatus. First, the video decoding device 20
When 0b receives the shape coded stream Sstr and the texture coded stream Tstr corresponding to the foreground from the image coding device 200a, the shape coded stream Sstr is decoded by the shape decoder U10. A shape decoded signal Sout is output (step S11).
Further, the texture encoded stream Tstr is decoded by the pixel decoder U11, and the decoded pixel value signal Tout
Is output (step S12).

Next, when the decoded shape signal Sout and the decoded pixel value signal Tout are input to the pixel synthesizer U7b, the image signal corresponding to the image to be synthesized is generated based on the synthesis position indication signal Pos from outside. Decoded pixel value signal Tou to Vin
A combining process for combining t is performed, and a combined image signal Vout is output (step S13).

The decoded texture signal Tout
Are subjected to a compensation process by the pixel compensator U9b (step S14), and are stored in the texture memory U5b as the compensated texture signal Pb (step S15).
Here, in the pixel compensator U9b, as a compensation process for the decoded texture signal Tout, a simple compensation process for the boundary macroblock shown in FIG.
An extended compensation process is performed on the extra-object macroblock shown in FIG.

Thereafter, it is determined whether or not decoding processing has been completed for all frames of one object (step S16). Are completed, and if the decoding process for all the frames has not been completed, the processes in steps S11 to S16 are performed again.

Next, a specific configuration of the pixel synthesizers U7a and U7b will be briefly described. FIG. 20 is a block diagram of the pixel synthesizer U7a. This pixel synthesizer U7
a is a shape enlargement / reduction unit U50 that performs enlargement / reduction processing on the locally decoded shape signal LSout in accordance with the synthesis position indication signal Pos and outputs a scaled shape signal;
A texture enlargement / reduction unit U52 that performs a scaling process on the locally decoded texture signal LTout according to the synthesis position indication signal Pos to output a scaled texture signal T1, and a decoded texture scaled according to the synthesis position indication signal Pos. A phase compensator U53 that performs a phase compensation process on the signal T1 and outputs a texture signal T2 that has undergone phase compensation.

In general, the spatial resolution of the shape signal is higher than that of the texture signal (particularly the color difference signal), and
Depending on the position of the object image to be combined with the pixel image, the pixel position in the image to be combined Vin does not match the pixel position in the object region formed by the scaled texture signal T2.

Therefore, the phase compensator U53 shifts the spatial phase of the scaled texture signal (color difference signal) T2, thereby changing the pixel position of the composite image Vin and the object area obtained from the scaled texture signal T2. The pixel positions are matched.

The pixel synthesizer U7a sets the ratio (synthesis ratio) when synthesizing pixel values between the image Vin to be synthesized and the rectangular phase compensation texture region at each pixel position from 0% to 100%. Composition ratio between
a pixel signal synthesis ratio deriving unit U51 that outputs r, a multiplier U54 that outputs a synthesized texture signal V1 obtained by multiplying the texture signal Vin of the synthesized image by (100−r) / 100, and a phase compensated signal. Texture signal T
A multiplier U55 that outputs a synthesized texture signal T3 obtained by multiplying 2 by r / 100, the synthesized texture signal V1 and the synthesized texture signal T3 are added, and a synthesized image signal Vout corresponding to the synthesized image is output. Adder U
56.

Further, the pixel synthesizer U7b in the video decoding device 200b is connected to the video encoding device 200b.
a has the same circuit configuration as that of the pixel synthesizer 7a in FIG.
A decoding texture signal is input in place of the local decoding texture signal in 7a.

It should be noted that the position at which the object is synthesized with the image to be synthesized and the rotation angle of the object can also be indicated by the synthesized position instruction signal Pos. Device U50 and pixel scaling device U52
May be configured to perform not only the object enlargement / reduction processing but also the object's linear movement processing and rotational movement processing.

Next, another configuration example of the conventional moving picture coding apparatus will be described. In the moving picture coding apparatus 200a shown in FIG. 16, a supplementary texture signal obtained by subjecting the locally decoded texture signal LTout from the texture encoder U3 to compensation processing is stored in the texture memory U5.
a, the locally decoded texture signal LTout from the texture encoder U3.
Is stored in the texture memory U5a without performing the compensation processing, the compensation processing is applied to the texture signal output from the texture memory U5a, and the texture signal subjected to the compensation processing is referred to in the encoding processing. You may make it supply to the texture encoder U3 as a processed texture signal.

FIG. 21 is a block diagram for explaining a moving picture coding apparatus having such a configuration. This moving picture coding device 210a is different from the moving picture coding device 200a shown in FIG.
5a, a texture memory U8a for storing the local decoded texture signal LTout from the texture encoder U3, and a texture signal Ta output from the texture memory U8a, and the supplemented texture signal is referred to as a reference texture signal. A pixel compensator U20a that outputs the texture encoder U3 as a Trefa to the texture encoder U3. To insert a specific value α. Here, the pixel compensator U20a performs the compensation process on the decoded texture signal Tout as shown in FIG.
The simple compensation process for the boundary macroblock shown in FIG. 15C and the extended compensation process for the extra-object macroblock shown in FIG. The other configuration of the moving picture coding apparatus 210a is the same as that of the moving picture coding apparatus 200a shown in FIG.

FIG. 22 is a diagram for explaining the operation of the moving picture coding apparatus 210a, and shows the flow of coding processing in the moving picture coding apparatus 210a. First, when the encoded shape signal Sin and the encoded pixel value signal Tin are input to the video encoding device 210a, the shape encoding unit U1 performs Similarly,
Shape encoding processing including local decoding processing is performed on the encoded shape signal Sin, and at this time, the locally decoded shape signal Sout is stored in the shape memory U2a (step S1).

In the texture encoding unit U3a, texture encoding including local decoding is performed on the encoded texture signal Tin, and the local encoded texture signal Tout and the texture encoded signal Tstr are processed. Is output (step S2). At this time, the specific pixel value insertion processing is performed simultaneously with the local decoding processing, and the local decoding texture signal LTout
Is a texture signal in which a specific value α is inserted as a pixel value of a pixel outside the object.

Next, the locally decoded shape signal LSout and the locally decoded texture signal LTout are converted by the pixel synthesizer U7.
a, the pixel synthesizer U7a synthesizes the locally decoded texture signal LTout with the image signal (texture signal) Vin corresponding to the image to be synthesized, based on the synthesis position indication signal Pos from outside. Is performed, and the combined image signal Vout is output (step S3). The locally decoded texture signal LTout is stored in the texture memory U8a (Step S4a).

Next, in the pixel compensator U20a, the texture signal Ta output from the texture memory U8a is subjected to compensation processing (step S5a), and the compensated texture signal obtained by this compensation processing is referred to as the reference texture. The signal is supplied to the texture encoder U3 as a signal Trefa. Here, the pixel compensator U20a
As in the pixel compensator U9a in the moving picture coding apparatus 200a shown in FIG. 16, the simple compensation processing for the boundary macroblock shown in FIG. 15 (c) is performed as the compensation processing for the local decoded texture signal LTout.
An extended compensation process for the extra-object macroblock shown in (d) is performed.

Thereafter, it is determined whether or not the encoding process for all the frames of one object is completed (step S6). Is completed, and if the encoding process for all the frames has not been completed, the above-described step S
The processing in 1, S2, S3, S4a, and S5a is performed again.

The moving picture coding apparatus 210 having such a configuration
a, the moving picture coding apparatus 200a shown in FIG.
Similarly to the above, the encoding process and the synthesizing process of the image signal corresponding to the object can be performed. In addition, in the moving picture coding apparatus 210a having this configuration, when performing coding processing on an intra-frame coded frame (I frame), the moving picture signal of the coded frame is not referred to, and therefore, no compensation processing is performed. . For this reason, the compensation processing can be efficiently performed as needed. More specifically, the compensation processing need only be performed at the time of encoding processing for an inter-screen encoded frame (P frame or B frame).

Next, another configuration example of the conventional moving picture decoding apparatus will be described. In the moving picture decoding apparatus 200b shown in FIG. 18, the compensation texture signal Pb obtained by performing the compensation process on the decoded texture signal Tout from the texture decoder U11 is stored in the pixel memory U5b. Texture decoder U1
1 is stored in the pixel memory U5b without performing the interpolation process, and the decoded texture signal Tout is stored in the pixel memory U5b.
b) applying a compensation process to the texture signal output from b,
The texture signal subjected to the compensation processing may be supplied to the texture decoder U11 as the reference texture signal Trefb.

FIG. 23 is a block diagram for explaining a moving picture decoding apparatus having such a configuration. The video decoding device 210b includes a texture memory U9b for storing a decoded texture signal Tout from the texture decoder U11 instead of the pixel compensator U9b and the pixel memory U5b in the video decoding device 200b.
And a pixel compensator U20b for performing a compensation process on the texture signal Tb output from the texture memory U9b and outputting the compensated texture signal Tb to the texture decoder U11.
Is configured to perform a process of inserting a specific value α as a pixel value of a pixel outside an object in a texture rectangular area in addition to a decoding process on a texture encoded stream. Here, the pixel compensator U20b performs a simple compensation process on the boundary macroblock shown in FIG. 15C and an extension on the extra-object macroblock shown in FIG. 15D as the compensation process for the decoded texture signal Tout. It is configured to perform compensation processing. Therefore, the other configuration of the video decoding device 210b is the same as that of the video decoding device 200b shown in FIG.

FIG. 24 is a diagram for explaining the operation of the moving picture decoding apparatus 210b, and shows the flow of decoding processing in the apparatus. First, when the shape coded stream Sstr and the texture coded stream Tstr from the video coding device 210a are input to the video decoding device 210b, the shape coded stream Sstr is input.
Is decoded by the shape decoder U10, and a shape decoded signal Sout is output (step S11). Further, the texture encoded stream Tstr is decoded by the texture decoder U11, and the decoded texture signal Tou is decoded.
t is output (step S12). At this time, the insertion process of the specific pixel value is performed simultaneously with the decoding process, and the decoded texture signal Tout is a texture signal into which the specific value α is inserted as the pixel value of the pixel outside the object.

Next, when the decoded shape signal Sout and the decoded texture signal Tout are input to the pixel synthesizer U7b, the pixel synthesizer U7b converts the image to be synthesized based on the synthesis position instruction signal Pos from the outside. The corresponding image signal (texture signal) Vin is decoded into a texture signal T.
Out is synthesized, and a synthesized image signal Vout is output (step S13). The decoded texture signal Tout is stored in the pixel memory U8b (Step S14a).

Next, in the pixel compensator U20b, compensation processing is performed on the decoded texture signal Tout output from the texture memory U8b (step S1).
5a), the compensated pixel value signal obtained by this compensation processing is supplied to the pixel decoder U11 as the reference pixel signal Tref. Here, in the pixel compensator U20b, as a compensation process for the decoded texture signal Tout, a simple compensation process for the boundary macroblock shown in FIG. 15C and an extension for the extra-object macroblock shown in FIG. Compensation processing is performed.

Thereafter, it is determined whether or not the decoding processing for all frames of one object has been completed (step S16). Is completed, and if the decoding process for all the frames has not been completed, the processes in steps S11, S12, S13, S14a, and S15a are performed again.

The conventional moving picture coding apparatus 210a
(See FIG. 21), the conventional video decoding device 210b (see FIG. 23) uses a special value α as the pixel value of the pixel outside the object in the locally decoded texture signal and the decoded texture signal, and In U20a and U20b, the inside / outside object determination processing for each pixel is performed based on the special value. Such a configuration is based on the conventional moving image encoding device 200a (see FIG. 16) and the conventional moving image decoding. The present invention is applicable to the conversion device 200b (see FIG. 18). In this case, the pixel compensator U9 in the video encoding device 200a
a, Pixel Compensator U9 in Video Decoding Device 200b
b, the local decoded shape signal LSout and the decoded shape signal S
Instead of the inside / outside object determination processing referring to out, the inside / outside object determination processing based on the value of the insertion value α is performed.

[0059]

FIG. 10 shows a position of a pixel Vp in an image space formed by a texture signal corresponding to an object (background) (FIG. 10A) and a shape signal corresponding to the object (foreground). Of the pixels Spo and Spi in the image space (rectangular region) formed by (FIG. (B))
Is shown.

In the texture signal, the value of the luminance or color difference (pixel value) of each pixel is usually represented by 8-bit information. When the object is opaque, the shape signal indicates whether each pixel is a pixel inside the object or a pixel outside the object by 1-bit information. For example, FIG.
0 (b) shows pixels Spo outside the object (oblique lattice pattern) and pixels Spi inside the object (dot pattern) in the shape rectangular area corresponding to the opaque object. Here, Obo is a region outside the object in the shape rectangular region, and Obi is a region inside the object in the shape rectangular region.

When the object is translucent, it is necessary to represent not only whether the pixel of the shape rectangular area is the pixel inside the object or the pixel outside the object, but also the transparency of each pixel. , The shape signal is a signal representing the transparency of each pixel in the shape rectangular area by 8-bit information.

By the way, an image corresponding to each object (foreground)
When synthesizing the image and the image to be synthesized (background), a weighted addition process is performed between the pixel value of the texture signal of the foreground and the pixel value of the texture signal of the background at the same pixel position. For example, if the weighting ratio of the foreground and the background to a predetermined pixel is 100: 0, the foreground is displayed, and if the ratio is 0: 100, the background is displayed. If the ratio is 50:50, the foreground and the background are mixed and displayed. In general, in the foreground and background synthesis processing, the pixel positions do not always match between the foreground and the background, and therefore processing for matching the spatial phase of the pixels is required.

FIG. 11 shows the pixel value of the texture signal.
FIG. 9 is a diagram for explaining a process of combining a foreground and a background. For example, pixel values X1 to X4 of pixels Tfp1 to Tfp4 in a texture rectangular area corresponding to an object (foreground)
Is combined with the pixel value Y of the pixel Tbp of the image to be combined (background) Vin, the pixel value of the position (corresponding position) in the texture rectangular area corresponding to the position of the pixel Tbp is calculated as the pixel Tfp1. It must be derived from the pixel values X1 to X4 of Tfp4.

However, since the pixels Tfp1 and Tfp2 are pixels outside the object, the pixel value of the corresponding position is derived only from the pixel values X3 and X4 of the pixels Tfp3 and Tfp4 that are the pixels inside the object. In other words, in the process of combining the background and the foreground, when deriving the pixel value in the foreground corresponding to the pixel position in the background, each neighborhood located around the position in the rectangular region of the foreground corresponding to the pixel position in the background For a pixel, it is necessary to perform a process of determining whether a neighboring pixel is a pixel inside the object or a pixel outside the object.
Therefore, in the synthesis processing, the calculation load required for the above-mentioned inside / outside object determination processing becomes very heavy, and a heavy load is imposed on hardware as a moving image encoding device and a moving image decoding device. In addition, in the synthesis process, a scaling process may be performed on a foreground rectangular area. In this scaling process, a filter operation process is performed.

FIG. 12 is a diagram for explaining a filter operation for a pixel value of a pixel located at an object boundary in a texture rectangular area. When an image is enlarged or reduced, a low-pass filter is generally used to prevent image quality deterioration due to flipping distortion. Number of filter coefficients (that is, the number of pixel values used for filter operation)
It is also known that the higher the number, the better the image quality. However, at the object boundary, if the number of filter coefficients is large, the load of the judgment processing inside and outside the object becomes heavy.

For example, as shown in FIG. 12, when performing a filter operation using five filter coefficients, that is, pixels Tp1 to Tp5 in a texture rectangular area of an object, pixels Tp4 and Tp5 need to be pixels outside the object. For example, it is necessary to perform the filter operation using only the pixel values of the remaining three pixels (pixels Tp1 to Tp3 in the object). In this case, the load of processing for determining whether a pixel (neighboring pixel) located in the vicinity of a pixel to be subjected to the filtering process is an in-object pixel or an out-of-object pixel is very heavy, and a heavy load is imposed on hardware. Become.

The present invention has been made in order to solve the above-mentioned problems.
In the process of determining whether a plurality of neighboring pixels located around the position in the image space of the foreground corresponding to the pixel position of the background are inside or outside the object, or at the time of image reduction / enlargement in the synthesis process, the filtering process is performed. Image that eliminates the need to determine whether a pixel (neighboring pixel) located in the vicinity of the target pixel is an in-object pixel or an out-of-object pixel, and can greatly reduce the amount of computation without deteriorating image quality An object of the present invention is to provide a processing method, an image processing apparatus, and a data storage medium storing a program for realizing processing by the image processing method by software.

[0068]

An image processing method according to the present invention (Claim 1) corresponds to an object comprising a shape signal representing a shape of an object constituting an image and a texture signal representing a picture of the object. An image processing method that performs encoding processing including local decoding processing for each object on a moving image signal to be processed and synthesizes a decoded moving image signal obtained by the local decoding processing with an image signal to be synthesized. A shape code for generating a shape-encoded stream and a locally decoded shape signal by performing an encoding process including a local decoding process on each of the objects with reference to the processed shape signal with respect to the shape signal. Coding processing including a local decoding process for the texture signal with reference to the processed texture signal for each object to obtain a texture coded stream and a local decoding texture. A texture encoding process for generating a texture signal, and a supplementation process for supplementing pixel values of pixels outside the object, which constitute the locally decoded texture signal and located outside the object, to generate a compensated texture signal. Synthesizing the compensated texture signal with a texture signal constituting the image signal to be synthesized based on the locally decoded shape signal, and using the compensated texture signal as the processed texture signal. .

According to a second aspect of the present invention, in the image processing method according to the first aspect, the pixel value of a pixel outside the object located outside the object is replaced by a pixel value inside the object located inside the object. Is performed to replace the pixel value.

According to a third aspect of the present invention, in the image processing method according to the first aspect, the texture signal and the locally decoded texture signal form an object area which is an image space including the object. The texture encoding process is performed on a macroblock consisting of a predetermined number of pixels constituting the object region, and the compensation process is performed in a boundary macroblock including the boundary of the object.
A first supplementation process for replacing a pixel value of a pixel outside the object located outside the object with a pixel value of a pixel inside the object located inside the object; A second compensation process for replacing a pixel value of a pixel with a pixel value of a pixel in the object located in the object.

According to the image processing method of the present invention (claim 4), an image coded stream obtained by coding a moving image signal corresponding to an object constituting an image for each object is provided.
An image processing method for performing a decoding process for each object and synthesizing a decoded moving image signal obtained by the decoding process with an image signal to be synthesized, wherein a shape signal representing the shape of the object is encoded for each object. The shape encoded stream obtained by
With reference to the decoded shape signal, a shape decoding process of decoding for each object to generate a decoded shape signal, and a texture encoded stream obtained by encoding a texture signal representing a picture of the object for each object, With reference to the decoded texture signal, a texture decoding process of decoding for each object to generate a decoded texture signal, and a pixel value of a pixel outside the object located outside the object, constituting the decoded texture signal And a compensation process for generating a compensated texture signal; and
Based on the decoded shape signal, including a synthesis process for synthesizing the texture signal constituting the image signal to be synthesized,
The compensated texture signal is used as the processed texture signal.

According to a fifth aspect of the present invention, in the image processing method according to the fourth aspect, the pixel value of the pixel outside the object located outside the object is replaced by the pixel value inside the object located inside the object. Is performed to replace the pixel value.

According to a sixth aspect of the present invention, in the image processing method according to the fourth aspect, the decoded texture signal forms an object area which is an image space including the object. The processing is performed on a macroblock consisting of a predetermined number of pixels constituting the object area, and the interpolation processing is performed on the boundary macroblock including the boundary of the object, on the outside of the object located outside the object. A first supplementation process for replacing a pixel value with a pixel value of a pixel in an object located in the object;
A second interpolation process for replacing the pixel value of the pixel of the macroblock located outside the object with the pixel value of the pixel inside the object located inside the object.

An image processing apparatus according to the present invention (claim 7) is capable of processing a moving image signal corresponding to an object, comprising a shape signal representing the shape of an object constituting an image and a texture signal representing a picture of the object. An image processing apparatus that performs an encoding process including a local decoding process for each object, and synthesizes a decoded moving image signal obtained by the local decoding process with an image signal to be synthesized. A shape encoder for performing a coding process including a local decoding process for each object with reference to the processed shape signal to generate a shape-coded stream and a locally decoded shape signal; A shape memory for storing the decoded shape signal as the processed shape signal, and applying an encoding process including a local decoding process to the texture signal for each object with reference to the processed texture signal. A texture encoder that generates a texture-encoded stream and a locally decoded texture signal; and a compensated texture signal that compensates for pixel values of pixels outside the object that are outside the object and that constitute the locally decoded texture signal. A pixel compensator for generating
A pixel synthesizer for synthesizing the compensated texture signal with a texture signal constituting the image signal to be synthesized based on the locally decoded shape signal; and a texture memory for storing the compensated texture signal as the processed texture signal. It is provided with.

The image processing apparatus according to the present invention (claim 8) is capable of processing a moving image signal corresponding to an object, comprising a shape signal representing the shape of an object constituting an image and a texture signal representing a picture of the object. An image processing apparatus that performs an encoding process including a local decoding process for each object, and synthesizes a decoded moving image signal obtained by the local decoding process with an image signal to be synthesized. For a shape signal that forms a shape object region that is an image space that includes a shape encoding process that includes a local decoding process with reference to the processed shape signal, a predetermined number of texture objects that constitute the texture object region for each object are processed. A shape encoder that applies a macroblock composed of pixels as a unit to generate a shape-encoded stream and a locally decoded shape signal, and the local decoded shape signal is the processed shape signal. A shape memory for storing, and a texture signal forming a texture object region which is an image space including the object, refer to a processed texture signal, perform an encoding process including a local decoding process, for each object, A texture encoder for applying a macroblock consisting of a predetermined number of pixels constituting an object area as a unit to generate a texture encoded stream and a locally decoded texture signal; and for the local decoded texture signal, In a boundary macroblock including a boundary, a first complementation process is performed to replace a pixel value of a pixel outside the object located outside the object with a pixel value of a pixel inside the object located inside the object, and a first compensation texture signal is generated. And a first complementing means for outputting the first compensated texture signal to the local decoding shape signal. A pixel synthesizer for synthesizing the texture signal forming the image signal to be synthesized based on the texture signal, a texture memory for storing the first compensated texture signal, and a first supplementary texture signal recorded in the texture memory. for,
Performing a second interpolation process of replacing the pixel value of a pixel of an extra-object macroblock located outside the object in the object region with a pixel value of an intra-object pixel located inside the object; And a second compensation means for outputting the second compensation texture signal obtained by the above to the texture encoder as the processed texture signal.

An image processing apparatus according to the present invention (claim 9) performs a decoding process for each object on an image encoded stream obtained by encoding a moving image signal corresponding to an object constituting an image. An image processing apparatus for combining a decoded moving image signal obtained by the decoding processing with a to-be-synthesized image signal, wherein a shape encoded stream obtained by encoding a shape signal representing a shape of an object for each object With reference to the processed shape signal, a shape decoder that decodes for each object to generate a decoded shape signal, a shape memory that stores the decoded shape signal as the processed shape signal, A texture encoded stream obtained by encoding a texture signal representing a picture for each object is decoded for each object with reference to the decoded texture signal to generate a decoded texture signal. A texture decoder, a pixel compensator that composes the decoded texture signal, and compensates for a pixel value of an extra-object pixel located outside the object to generate a compensated texture signal; and a compensated texture signal. A pixel synthesizer that synthesizes the texture signal constituting the image signal to be synthesized based on the decoded shape signal, and a texture memory that records the compensated texture signal as the processed texture signal. is there.

The image processing device according to the present invention (claim 10) performs a decoding process for each object on an image encoded stream obtained by encoding a moving image signal corresponding to an object constituting an image. An image processing apparatus for combining a decoded moving image signal obtained by the decoding processing with a to-be-synthesized image signal, wherein a shape encoded stream obtained by encoding a shape signal representing a shape of an object for each object is provided. Is decoded for each object with reference to the processed shape signal, a shape decoder that generates a decoded shape signal that forms a shape object region that is an image space including the object, and the decoded shape signal A shape memory for storing as a processed shape signal, and a texture coded stream obtained by coding a texture signal representing a picture of an object for each object with reference to the decoded texture signal. A texture decoder for decoding each body to generate a decoded texture signal that forms a texture object region that is an image space including an object; and a boundary macro including the object boundary with respect to the decoded texture signal. A first compensation processing for replacing a pixel value of a pixel outside the object located outside the object in the block with a pixel value of a pixel inside the object located inside the object to output a first compensation texture signal; A pixel synthesizer for synthesizing the first compensated texture signal with a texture signal constituting the composited image signal based on the decoded shape signal; and In response to the texture memory to be stored and the first supplementary texture signal recorded in the texture memory, A second compensation process is performed to replace the pixel value of the pixel of the macroblock outside the object to be placed with the pixel value of the pixel inside the object located in the object, and a second compensation texture signal obtained by the second compensation process As the processed texture signal to the texture decoder.

In the data recording medium according to the present invention (claim 11), an encoding process including a local decoding process for a moving image signal, and a locally decoded moving image signal obtained by the local decoding process are converted to a synthesized image signal. A data storage medium that stores a program for performing a combining process by a computer using a computer, wherein the computer performs a local decoding process on the shape signal with reference to a processed shape signal. A shape encoding process for generating a shape-encoded stream and a locally decoded shape signal by performing an encoding process for each object, and a local decoding process for the texture signal with reference to the processed texture signal. Which performs encoding processing including for each object to generate a texture encoded stream and a locally decoded texture signal Encoding processing, which constitutes the locally decoded texture signal, compensates the pixel value of the pixel outside the object located outside the object, the compensation processing to generate a compensated texture signal used as the processed texture signal, An image encoding program for performing a synthesizing process of synthesizing the compensated texture signal with a texture signal constituting the image signal to be synthesized based on the locally decoded shape signal is stored.

The data recording medium according to the present invention (claim 12) performs a decoding process on an image coded stream obtained by coding a moving image signal corresponding to an object constituting an image for each object, and the decoding A data storage medium storing a program for performing, by a computer, a combining process of combining a decoded moving image signal obtained by the process with a to-be-combined image signal. A shape-encoded stream obtained by encoding each object, by referring to the processed shape signal, decoding each object to generate a decoded shape signal, and a texture signal representing a picture of the object. Is encoded for each object, and a texture encoded stream obtained by encoding the object is decoded for each object with reference to the processed texture signal. A texture decoding process for generating an encoded texture signal, and a compensated texture signal used as the processed texture signal by compensating for the pixel value of an extra-object pixel located outside the object, which constitutes the decoded texture signal. , And a combining process of combining the compensated texture signal with a texture signal constituting the combined image signal based on the decoded shape signal. Things.

[0080]

BEST MODE FOR CARRYING OUT THE INVENTION The present inventor has conducted intensive studies on the above-mentioned conventional problems, and as a result, in a conventional moving picture coding apparatus and moving picture decoding apparatus, encoding and decoding processing for a texture signal has been performed. Attention is paid to the fact that the process of compensating for the pixel value of the pixel outside the object is performed, and when synthesizing the texture signal of the object image with the texture signal of the image to be synthesized, the pixel value of the pixel outside the object in the decoded texture signal is used. Instead of using the pixel value of the decoded texture signal that has been subjected to the interpolation process, the pixel located near the target pixel (neighboring pixel) can be either an in-object pixel or an out-of-object pixel during the synthesis process. It has been found that the process of determining whether or not is unnecessary, and the amount of calculation can be greatly reduced without deteriorating the image quality.

Therefore, according to the present invention, a supplementary process for replacing the pixel value of the pixel outside the object with a significant pixel value is performed on the decoded texture signal obtained by the local decoding process or the decoding process, and the decoded texture signal corresponding to the object is processed. The supplemented texture signal and the texture signal corresponding to the combined image are combined so that the object image is arranged on the combined image.

Hereinafter, embodiments of the present invention will be described with reference to FIGS. 1 to 9. (Embodiment 1) FIG. 1 is a block diagram for explaining a moving picture coding apparatus according to Embodiment 1 of the present invention. The moving picture coding apparatus 101 according to the first embodiment includes a pixel compensator U9 instead of the output (locally decoded texture signal) LTout of the texture signal coding unit U3 in the conventional moving picture coding apparatus 200a.
a (compensation texture signal) Pa from the pixel synthesizer U
7a. The other configuration of the moving picture coding apparatus 101 is the same as that of the conventional moving picture coding apparatus 200a.

Next, the function and effect will be described. FIG. 2 shows a flowchart of the moving picture coding process by the moving picture coding apparatus 101 according to the first embodiment. First, an encoded shape signal Sin and an encoded texture signal Tin corresponding to an object (foreground) are provided to the video encoding device 101.
Is input, the shape encoding unit U1 performs encoding processing of the encoded shape signal Sin (step S1), and the texture encoding unit U3 performs encoding of the encoded shape signal U3, as in the conventional moving image encoding device 200a. The encoding process of the texture signal Tin is performed (step S2). As a result, the locally decoded shape signal LSout and the shape encoded stream Sstr are output from the shape encoding unit U1, and the locally decoded texture signal LTout is output from the texture encoding unit U3.
And a texture coded stream Tstr.

Next, in the first embodiment, in the pixel compensator U9a, the local decoding texture signal Tout is subjected to compensation processing for supplementing the pixel value of the pixel outside the object, and the compensated texture signal is output. Pa is output (step S30). That is, in the pixel compensator U9a, as the compensation processing for the locally decoded texture signal LTout, the simple compensation processing for the boundary macroblock shown in FIG. 15C and the extension for the extra-object macroblock shown in FIG. Compensation processing is performed.

Further, in the first embodiment, the compensated texture signal Pa corresponding to the object (foreground) is determined by the pixel synthesizer U7a based on the synthesized position indicating signal Pos and the locally decoded shape signal Sout from the outside. And the texture signal Vin of the image to be combined (background) are combined, and a combined image signal Vout is output (step S4).
0). On the other hand, the compensated texture signal Pa is stored in the pixel memory U5a as the reference texture signal Tref (step S5).

Thereafter, the conventional moving picture coding apparatus 200
As in the case of a, it is determined whether or not the encoding process has been completed for all frames of one object (step S).
6) The processes in steps S1, S2, S30, S40, S5, and S6 are repeatedly performed until the encoding process for the object is completed.

As described above, in the first embodiment, the local decoding texture signal LTout corresponding to the object (foreground) is subjected to interpolation processing for replacing the pixel value of the pixel outside the object with the pixel value of the pixel inside the object. The compensated texture Pa obtained by the compensation process is combined with the texture signal Vin of the image to be combined (background), so that an unnatural value is not generated as a combined pixel value at the contour (boundary) of the object. Can be Further, when the object image is synthesized with the image to be synthesized, for each of a plurality of neighboring pixels located around the pixel to be subjected to the filtering process, the neighboring pixel is an outside object pixel or an inside object pixel. This eliminates the need for processing to determine whether or not there is, and without compromising the image quality of the synthesized image, the computational load of the phase compensation processing of the pixel value for the texture signal of the foreground, and the computational load of the filter processing for the pixel value of the pixels constituting the foreground. Can be reduced.

For example, it is not necessary to switch the number of neighboring pixels to be referred to at the time of a filter operation between a boundary pixel located at a boundary in the object and other pixels in the object. The load can be reduced. As a result, in the pixel synthesizing process when encoding a moving image signal corresponding to an object, various arithmetic processes can be performed well.

(Embodiment 2) FIG. 3 is a block diagram for explaining a moving picture decoding apparatus according to Embodiment 2 of the present invention. The moving picture decoding apparatus 102 according to the second embodiment.
Inputs the output (compensated texture signal) Pb of the pixel compensator U9b to the pixel synthesizer U7b instead of the output (decoded texture signal) Tout of the texture signal decoder U11 in the conventional video decoding device 200b. It is like that. Note that this video decoding device 10
2 is a conventional moving picture decoding apparatus 2
Same as 00b. Note that this video decoding device 10
2 is a conventional moving picture decoding apparatus 2
Same as 00b.

Next, the function and effect will be described. FIG. 4 shows a flowchart of a moving picture decoding process by the moving picture decoding apparatus 102 according to the second embodiment. First, the video encoding device 101 is added to the video decoding device 102.
, The shape-encoded stream Sstr and the pixel-value-encoded stream Tstr corresponding to the object (foreground) are input to the shape decoding unit U10, as in the conventional video decoding device 200b. The decoding process of Sstr is performed (step S11), and the texture decoding unit U11 performs the decoding process of the texture coded stream Tstr (step S12). As a result, the decoded shape signal Sout is output from the shape decoding unit U10, and the decoded texture signal Tout is output from the texture decoding unit U11.

Next, in the second embodiment, in the pixel compensator U9b, the above-mentioned decoded texture signal Tout is subjected to compensation processing for supplementing the pixel value of the pixel outside the object.
The compensated texture signal Pb is output (step S
30a). That is, in the pixel compensator U9b, as a compensation process for the decoded texture signal Tout,
The simple compensation process for the boundary macroblock shown in FIG. 15C and the extended compensation process for the extra-object macroblock shown in FIG. 15D are performed.

Further, in the second embodiment, the compensated texture signal Pb corresponding to the object (foreground) is obtained by the pixel synthesizer U7b based on the synthesized position indicating signal Pos and the decoded shape signal Sout from the outside. , Composite image (background)
Is synthesized with the texture signal Vin, and a combined image signal Vout is output (step S40a). On the other hand, the compensated texture signal Pb is stored in the pixel memory U5b as the reference texture signal Trefb (step S15).

Thereafter, the conventional moving picture decoding apparatus 200
Similarly to b, it is determined whether or not the decoding process for all frames of one object is completed (step S).
16), steps S11, S12, S30a, S40a, S1 until the decoding process for the object is completed.
5, the processing in S16 is repeatedly performed.

As described above, in the second embodiment, the decoded texture signal Tout corresponding to the object (foreground) is subjected to the interpolation process for replacing the pixel value of the pixel outside the object with the pixel value of the pixel inside the object. Since the compensated texture Pb obtained by the compensation process is combined with the texture signal Vin of the image to be combined, it is possible to prevent an unnatural value as a combined pixel value from appearing at the contour (boundary) of the object. it can. Further, when the object image is synthesized with the image to be synthesized, for each of a plurality of neighboring pixels located around the pixel to be subjected to the filtering process, the neighboring pixel is an outside object pixel or an inside object pixel. This eliminates the need to perform the process of determining whether or not there is, and without compromising the image quality of the synthesized image, the calculation load of the phase compensation processing of the pixel value with respect to the texture signal of the foreground, and the calculation in the filter processing with respect to the pixel value of the pixel constituting the foreground The load can be reduced.

For example, it is not necessary to switch the number of neighboring pixels to be referred to when performing a filter operation between a boundary pixel located at a boundary in the object and other pixels in the object. The load can be reduced. As a result, in the pixel synthesizing process when decoding the moving picture coded signal corresponding to the object, various arithmetic processes can be favorably performed.

(Embodiment 3) FIG. 5 is a block diagram for explaining a moving picture coding apparatus according to Embodiment 3 of the present invention. Moving picture coding apparatus 103 according to the third embodiment
Is the local decoding texture signal L from the texture signal encoder U3 instead of the pixel compensator U9a and the pixel memory U5a of the video encoding device 101 of the first embodiment.
A first pixel compensator U4a that performs a first pixel compensation process on Tout and outputs a compensation texture signal Pa;
The compensation texture signal Pa is referred to as the texture signal T.
A texture memory U8a for recording as ra and a second pixel compensator U6a for performing a second pixel compensation process on the reference texture signal Tra and outputting a compensated reference texture signal PTrefa.

In other words, the moving picture coding apparatus 103 according to the third embodiment performs the filling processing in the moving picture coding apparatus 101 according to the first embodiment with the pixel filling processing for the boundary macroblock (the first pixel filling processing). Process) and a pixel compensation process (second pixel compensation process) for the macroblock outside the object, and the first and second pixel compensation processes are divided into the first and second pixel compensation processes.
This is performed by the second pixel compensators U4a and U6a.

Hereinafter, the supplementary processing in the third embodiment will be briefly described. The interpolation process in the encoding process corresponding to MPEG4 is divided into two processes: a simple interpolation process for an object boundary macroblock (see FIG. 15C) and an extended interpolation process for an extra-object macroblock (see FIG. 15D). Can be separated.

In the encoding system corresponding to MPEG4, various kinds of signal processing are basically performed on a macroblock basis. For this reason, the moving picture coding apparatus that performs the pixel compensation processing can be realized by a minimum-scale hardware, if it has a sequential processing configuration that basically repeats various kinds of signal processing in macroblock units.

However, the pixel value of the pixel of the macroblock outside the object is compensated by the pixel value of the pixel of the macroblock in the vicinity (see FIG. 15D) (extended compensation process).
Is a process that refers to the pixel values of pixels of neighboring macroblocks adjacent to the macroblock to be compensated in the upper, lower, left, and right directions. Therefore, it is difficult to implement the extended compensation by sequential processing in units of macroblocks. .

Therefore, in the third embodiment, the simple compensating process (see FIG. 15 (c)) for the boundary macroblock that can be sequentially processed in macroblock units is performed by the pixel compensator U4 as the first pixel compensating process. The extended interpolation processing (refer to FIG. 15D) for the non-object macroblock in which it is difficult to perform the sequential processing in units of macroblocks is performed by the pixel interpolation unit U6 as the second pixel interpolation processing.

As a result, it is possible to perform the supplementary processing on the locally decoded texture signal LTout without breaking the configuration for the sequential processing in macroblock units in the texture encoder U3 and the pixel synthesizer U7a. Note that the first pixel compensator U4a does not perform the compensation process on the pixel value of the pixel of the macroblock outside the object, and therefore, in the third embodiment, the pixel compositor U7a
Determines whether each pixel is located inside the object or outside the object in macroblock units, and performs a filter operation or the like on the pixel value so that the synthesis processing is not performed on the pixels of the macroblock outside the object. Is prohibited from being used.

Next, the function and effect will be described. FIG. 6 shows a flowchart of a moving picture coding process by the moving picture coding apparatus 103 according to the third embodiment. First, when the encoded shape signal Sin and the encoded texture signal Tin corresponding to the object are input to the video encoding device 103, an encoding process on the encoded shape signal Sin (step S1), and The encoding process (step S2) for the texture signal to be encoded Tin is performed in exactly the same way as in the moving image encoding device 101 of the first embodiment.

Next, in the third embodiment, the first pixel compensator U4a performs the local decoding texture signal LTou.
The first compensation processing for supplementing the pixel value of the pixel outside the object in the boundary macroblock is performed on t, and the compensated texture signal Pa is output to the pixel synthesizer U7a and the texture memory U8a (step S31). Subsequently, in the pixel synthesizer U7a, a synthesizing position indicating signal P
based on the os and the locally decoded shape signal LSout, a compensated texture signal Pa corresponding to the object (foreground);
A process of synthesizing the texture signal Vin of the synthesized image (background) is performed, and a synthesized image signal Vout is output (step S41). In addition, the compensated texture signal Pa
Is stored in the texture memory U8a as the reference texture signal Tra (step S5).

Further, the reference texture signal Tra read out from the texture memory U8a is subjected to a second pixel compensation process for supplementing the pixel value of the pixel with respect to the macroblock outside the object by the second pixel compensation unit U6a. (Step S32), and the compensated reference texture signal PTrefa
Is output to the texture encoder U3. Thereafter, as in the case of the moving picture coding apparatus 101 according to the first embodiment, it is determined whether or not the coding processing for all frames of one object is completed (step S6), and the code for the object is determined. Step S until the conversion process is completed.
The processing in 1, S2, S31, S40, S5, S32, and S6 is repeatedly performed.

In the third embodiment having such a configuration, the first interpolation processing for interpolating the pixel value of the pixel outside the object in the boundary macroblock is performed on the locally decoded texture signal from the texture encoder U3. A first pixel compensator U4a that outputs a compensated texture signal Pa;
A second pixel compensator U6a for performing a second compensation process for compensating for the pixel value of the pixel in the extra-object macroblock on the compensated texture signal Pa as the reference texture signal Tra; Since the compensated texture signal Pa from the compensator U4a is combined with the texture signal Vin of the to-be-combined image, the combining process is performed in the combining process of the compensated texture signal Pa and the texture signal Vin of the to-be-combined image. The determination whether the pixel is located inside the object or outside the object is performed on a macroblock basis, and the amount of calculation in the combining process can be significantly reduced without deteriorating the image quality of the combined image.

In the third embodiment, an example has been described in which pixel compensation processing is divided into two processes, a first compensation process for a boundary macroblock and a second compensation process for a non-object macroblock. However, in the case where the second complementing process for the extra-object macroblock is, for example, to supplement the pixel value corresponding to the extra-object macroblock with the pixel value corresponding to the left macroblock, the extra-object macroblock In the second supplementary processing for, it is easy to perform sequential processing for each macroblock, and thus it is not always necessary to divide the supplementary processing into two for boundary macroblocks and one for non-object macroblocks as in the third embodiment.

(Embodiment 4) FIG. 7 is a block diagram for explaining a moving picture decoding apparatus according to Embodiment 4 of the present invention. Moving picture coding apparatus 104 according to Embodiment 4
Is a moving picture decoding apparatus 102 according to the second embodiment shown in FIG.
A first pixel compensator that performs a first pixel compensation process on the decoded texture signal Tout from the texture decoder U11 in place of the pixel compensator U9b and the pixel memory U5b, and outputs a compensated texture signal Pb. U
4b and a texture memory U8b for recording the supplementary texture signal Pb as a reference texture signal Trb.
And a second pixel compensator U6b that performs a second pixel compensation process on the reference texture signal Trb and outputs a compensated reference texture signal PTrefb.

In other words, the moving picture coding apparatus 104 according to the fourth embodiment performs the filling processing in the moving picture coding apparatus 102 according to the second embodiment with the pixel filling processing for the boundary macroblock (the first pixel filling processing). Process) and a pixel compensation process (second pixel compensation process) for the macroblock outside the object, and the first and second pixel compensation processes are divided into the first and second pixel compensation processes.
This is performed by the second pixel compensators U4b and U6b.

The first and second pixel compensators U4
b and U6b are the first and second pixel compensators U4a and U4a, respectively, in the moving picture coding apparatus 103 according to the third embodiment.
It has exactly the same configuration as U6a. In addition, the first
In the pixel compensator U4b, the pixel compensator U7b does not perform compensation processing on the pixel values of the pixels of the macroblock outside the object, so that the pixel compositor U7b locates each pixel within the object or outside the object in macroblock units. Is determined, and the pixel value of the macroblock outside the object is prohibited from being used by a filter operation or the like so that synthesis processing is not performed on the pixel.

Next, the function and effect will be described. FIG. 8 shows a flowchart of a moving picture decoding process by the moving picture coding apparatus 104 according to the fourth embodiment. First, the video encoding device 103 is added to the video decoding device 104.
When the shape-encoded stream Sstr and the texture-encoded stream Tstr are input, the decoding process on the shape-encoded stream Sstr (step S11) and the decoding process on the texture-encoded stream Tstr (step S12) are performed. This is performed in exactly the same way as in the moving picture decoding apparatus 102 according to the second embodiment.

Next, in the fourth embodiment, the first pixel compensator U4b compensates the decoded texture signal Tout for the pixel value of the pixel outside the object in the boundary macroblock. And the compensated texture signal Pb is output to the pixel synthesizer U7b and the texture memory U8b (step S31a). Subsequently, in the pixel synthesizer U7b, the object (foreground) is determined based on the synthesis position indication signal Pos and the decoded shape signal Sout from outside.
Is performed, and the texture signal Vin of the image to be combined (background) is combined with the compensated texture signal Pb, and the combined image signal Vout is output (step S14).
a). The compensated texture signal Pb is used as the reference texture signal Trb as the texture memory U5b.
(Step S15a).

Further, the reference texture signal Trb read from the texture memory U5b is subjected to a second pixel compensation process for supplementing the pixel value of the pixel to the macroblock outside the object by the second pixel compensation unit U6b. (Step S32a), and the compensated reference texture signal is output to the texture decoder U11. After that,
Similarly to the moving picture coding apparatus 102 according to the second embodiment,
It is determined whether or not the encoding process for all frames of one object is completed (step S16), and the above-described step S1 is performed until the encoding process for the object is completed.
1, S12, S31a, S14a, S15a, S32
a, The processing in S6 is repeatedly performed.

In the fourth embodiment having such a structure, the decoded texture signal from the texture decoder U11 is subjected to the first compensation processing for supplementing the pixel value of the pixel outside the object in the boundary macroblock. A first pixel compensator U4b that outputs the compensated texture signal Pb and a second compensation process that compensates for the compensated texture signal Pb as the reference texture signal Trb with the pixel value of the pixel in the extra-object macroblock. A second pixel compensator U6b to be applied, and the compensated texture signal Pb from the first pixel compensator U4b is combined with the texture signal Vin of the image to be combined. In the synthesis process of the texture signal Vin of the synthesized image, the pixel to be synthesized is located inside the object or outside the object. Determination of Luke will be performed in units of macroblocks, without impairing the quality of the composite image, the calculation amount in the synthesis process can be significantly reduced.

In the fourth embodiment, an example has been described in which the pixel compensation processing is divided into two processes, a first compensation process for a boundary macroblock and a second compensation process for a non-object macroblock. However, in the case where the second complementing process for the extra-object macroblock is, for example, to supplement the pixel value corresponding to the extra-object macroblock with the pixel value corresponding to the left macroblock, the extra-object macroblock In the second compensation processing for the macroblock, it is easy to successively perform the processing for each macroblock. Therefore, it is not always necessary to divide the compensation processing into two for the boundary macroblock and for the extra-object macroblock as in the fourth embodiment.

Further, a program for realizing the configuration of the moving picture coding apparatus and the moving picture decoding apparatus shown in each of the above embodiments by software is recorded on a storage medium such as a floppy disk. Accordingly, the moving picture coding processing or the moving picture decoding processing described in each of the above embodiments can be easily performed by an independent computer system.

FIG. 9 is a data recording medium (FIGS. 9 (a) and 9 (b)) storing a program for realizing a moving picture coding apparatus or a moving picture decoding apparatus according to each of the above-described embodiments by a computer system. And FIG. 9 is a diagram for explaining the computer system (FIG. 9C). FIG. 9A shows an external appearance, a sectional structure, and a floppy disk of the floppy disk as viewed from the front, and FIG. 9B shows an example of a physical format of the floppy disk as a recording medium body.

The floppy disk FD is housed in the case F. A plurality of tracks Tr are formed concentrically from the outer circumference toward the inner circumference on the surface of the disk, and each track has 16 sectors Se in the angular direction. Is divided into
Therefore, in the floppy disk storing the above-mentioned program, the computer executes the moving image encoding method or the moving image decoding method according to each embodiment in the area allocated on the floppy disk FD as the above program. Is recorded.

FIG. 9C shows a floppy disk F
D shows a configuration for recording and reproducing the program. When recording the program on the floppy disk FD, the image processing program is written from the computer system Cs via the floppy disk drive. When the moving picture encoding method and the moving picture decoding method are to be constructed in a computer system by a program in a floppy disk, the image processing program is read from the floppy disk by a floppy disk drive and transferred to the computer system.

In the above description of the computer system, a floppy disk is taken as an example of a data recording medium. However, even if an optical disk is used instead of a floppy disk, each of the above embodiments can be used in the same manner as when a floppy disk is used. The moving picture coding apparatus or the moving picture decoding apparatus according to the embodiment can be realized by a computer system. Further, the data recording medium is not limited to the one described above,
Any device such as an IC card or a ROM cassette that can record a program may be used.

[0121]

As described above, the present invention (Claims 1, 2 and 2)
According to the image processing method according to 3), an encoding process including a local decoding process on a shape signal corresponding to the object, an encoding process including a local decoding process on a texture signal corresponding to the object, and A supplementation process for compensating for the pixel value of an extra-object pixel located outside the object, which constitutes a locally decoded texture signal obtained by the local decoding process, and generating a supplemented texture signal; And a synthesis process for synthesizing with a texture signal constituting a synthesized image signal based on the locally decoded shape signal obtained by the above local decoding process. Unnatural values can be prevented.

When the object image is synthesized with the image to be synthesized, for each of a plurality of neighboring pixels located around the pixel to be filtered, it is determined whether the neighboring pixel is a pixel outside the object. The process of determining whether the pixel is an in-vivo pixel is unnecessary, and the calculation load of the phase compensation process of the pixel value for the texture signal of the object (foreground) and the pixel value of the pixel constituting the foreground are maintained without deteriorating the image quality of the synthesized image. Can reduce the calculation load in the filter processing for. As a result, in the pixel synthesizing process when encoding a moving image signal corresponding to an object, various arithmetic processes can be performed well.

According to the image processing method of the present invention (claims 4, 5, and 6), a decoding process for a shape-coded stream corresponding to an object and a decoding process for a texture-coded stream corresponding to an object And constitute a decoded texture signal obtained by the decoding process,
Compensating the pixel value of the pixel outside the object located outside the object,
Compensation processing for generating a compensated texture signal, and combining processing for combining the compensated texture signal with a texture signal constituting an image signal to be combined based on the decoded shape signal obtained by the decoding processing. Therefore, it is possible to prevent an unnatural value from appearing as a composite pixel value in the contour portion (boundary portion) of the object.

When the object image is synthesized with the image to be synthesized, for each of a plurality of neighboring pixels located around the pixel to be subjected to the filtering process, it is determined whether the neighboring pixel is a pixel outside the object. The process of determining whether the pixel is an in-vivo pixel becomes unnecessary, and the calculation load of the phase compensation process of the pixel value with respect to the texture signal of the object (foreground) and the pixel of the pixel constituting the foreground can be obtained without deteriorating the image quality of the synthesized image. The calculation load in the filtering process for the value can be reduced. As a result, in the pixel synthesizing process when decoding the moving picture coded signal corresponding to the object, various arithmetic processes can be favorably performed.

According to the image processing apparatus of the present invention (claim 7), a shape encoder that performs an encoding process including a local decoding process on a shape signal corresponding to an object for each object, A texture encoder that performs encoding processing including local decoding processing for each object with respect to the texture signal, and supplements pixel values of pixels outside the object located outside the object, which constitute the locally decoded texture signal. A pixel compensator for generating a compensated texture signal, and compensating the compensated texture signal with a texture signal constituting an image signal to be combined based on the locally decoded shape signal. It is possible to prevent an unnatural value from appearing as a composite pixel value at the contour (boundary) of
In addition, the calculation load of the phase compensation processing of the pixel value with respect to the texture signal of the object (foreground) and the calculation load of the filter processing with respect to the pixel values of the pixels constituting the foreground can be reduced without deteriorating the image quality of the synthesized image. As a result, in the pixel synthesizing process when encoding a moving image signal corresponding to an object, various arithmetic processes can be performed well.

According to the image processing apparatus of the present invention (claim 8), a shape code for performing a shape coding process including a local decoding process on a shape signal corresponding to an object in macroblock units for each object. A texture encoder for performing a coding process including a local decoding process on a texture signal corresponding to an object on a macroblock basis for each object; and a boundary macro for the locally decoded texture signal. A first compensation means for performing a first compensation process for replacing a pixel value of a pixel outside the object in the block with a pixel value of a pixel inside the object, and a first compensated texture signal obtained by the first compensation process. A pixel synthesizer for synthesizing a texture signal constituting the image signal to be synthesized, and a pixel value of a pixel of an extra-object macroblock with respect to the first compensated texture signal. Second to replace the pixel values in the body pixels
And the second compensation means for performing the compensation processing of (1), the combination processing of the compensated texture signal and the image signal to be combined includes:
The determination as to whether the pixel to be subjected to the combining process is located inside the object or outside the object is performed on a macroblock basis. Can be reduced.

According to the image processing apparatus of the present invention (claim 9), a shape decoder that decodes a shape-encoded stream corresponding to an object for each object, and a texture-encoded stream corresponding to the object is converted to an object A texture decoder for decoding each time, a pixel compensator for producing a compensated texture signal by supplementing the pixel values of the pixels outside the object constituting the decoded texture signal, and a compensated texture signal.
Since a pixel synthesizer for synthesizing the texture signal constituting the synthesized image signal and the image signal is provided, it is possible to prevent an unnatural value from appearing as a synthesized pixel value at the contour (boundary) of the object. The computational load of the phase compensation processing of the pixel value for the texture signal of the foreground and the computational load of the filter processing for the pixel values of the pixels constituting the object (foreground) can be reduced without deteriorating the image quality of the synthesized image. As a result,
In the pixel synthesizing process when encoding a moving image signal corresponding to an object, various arithmetic processes can be performed well.

According to the image processing apparatus of the present invention (claim 10), a shape decoder for decoding a shape-encoded stream corresponding to an object for each object, and a texture-encoded stream corresponding to the object for an object A texture decoder for decoding each time, and a first compensation process for replacing the pixel value of the pixel outside the object in the boundary macroblock with the pixel value of the pixel inside the object for the decoded texture signal. A compensating means, a pixel composer for composing a first compensated texture signal obtained by the first compensating process with a texture signal constituting a combined image signal, and a first compensating texture signal. And a second compensation means for performing a second compensation process for replacing the pixel value of the pixel of the macroblock outside the object with the pixel value of the pixel inside the object. In the process of synthesizing the image signal and the image to be synthesized, it is determined whether the pixel to be subjected to the synthesis process is located inside the object or outside the object in units of macroblocks, thereby deteriorating the image quality of the synthesized image. Therefore, the amount of calculation in the combining process can be significantly reduced.

According to the data recording medium of the present invention (claim 11), a shape encoding process in which a computer performs an encoding process including a local decoding process on a shape signal corresponding to an object for each object And a texture encoding process of performing an encoding process including a local decoding process on a texture signal corresponding to an object for each object, and supplementing a pixel value of a pixel outside the object constituting the locally decoded texture signal. Since an image encoding program for performing a compensation process for generating a compensated texture signal and a combining process for combining the compensated texture signal with a texture signal constituting a combined image signal is stored, the image encoding program corresponding to the object is stored. An image encoding device that can reduce the processing load of combining a moving image signal with a combined image signal can be simplified by a computer system. It can be realized.

The data recording medium according to the present invention (claim 12) is characterized in that a shape decoding process for decoding a shape-encoded stream corresponding to an object for each object by a computer;
A texture decoding process for decoding a texture-encoded stream corresponding to the object for each object, and a compensation process for supplementing the pixel values of pixels outside the object constituting the decoded texture signal to generate a compensated texture signal; Since an image decoding program for performing a synthesizing process of synthesizing the compensated texture signal with a texture signal constituting the image signal to be synthesized is stored, a process of synthesizing a moving image signal corresponding to the object with the image signal to be synthesized. Can be easily realized by a computer system.

[Brief description of the drawings]

FIG. 1 is a block diagram for explaining a video encoding device according to a first embodiment of the present invention.

FIG. 2 is a diagram showing a flow of a moving image encoding process by the moving image encoding device of the first embodiment.

FIG. 3 is a block diagram for explaining a video decoding device according to a second embodiment of the present invention.

FIG. 4 is a diagram showing a flow of a moving picture decoding process by the moving picture decoding apparatus according to the second embodiment.

FIG. 5 is a block diagram for explaining a moving picture coding device according to a third embodiment of the present invention.

FIG. 6 is a diagram showing a flow of a moving image encoding process by the moving image encoding device of the third embodiment.

FIG. 7 is a block diagram for explaining a video decoding device according to a fourth embodiment of the present invention.

FIG. 8 is a diagram showing a flow of a moving picture decoding process by the moving picture decoding apparatus according to the fourth embodiment.

FIG. 9 is a data recording medium (FIG. 9) storing a program for realizing a moving image encoding process or a moving image decoding process of each of the above-described embodiments using a computer system.
(a), (b)) and the above computer system (Fig. (c))
FIG.

FIG. 10 shows a pixel position in an image space formed by a texture signal corresponding to an object (FIG. 10A), and a pixel position in an image space formed by a shape signal corresponding to an object (FIG. 10B). FIG.

FIG. 11 is a diagram illustrating a process of combining a pixel value of a texture signal between a foreground and a background.

FIG. 12 is a diagram illustrating a process of synthesizing a pixel value of a texture signal between a foreground and a background.

FIG. 13 is a diagram for explaining the encoding process for each object, and is a pattern Tin obtained from a texture signal of a foreground.
(Figure (a)), the shape Sin obtained from the foreground shape signal (Figure
(b)), a picture (combined image) Vin (FIG. (c)) obtained from a background texture signal, and a synthesized image Vout (FIG. (d)) obtained by synthesizing the background and foreground. .

FIG. 14 is a diagram for describing inter-frame encoding processing for a texture signal in object units, and an object whose size increases with the passage of time (time t1, time t2, and time t3) (FIG. , FIG. (B), FIG. (C)).

FIG. 15 is a diagram for explaining a pixel value compensation process for a texture signal, wherein a foreground texture rectangular region TR (FIG. (A)), a foreground shape rectangular region SR (FIG. (B)), and a simple supplementation process. Applied texture rectangular area TRpad1
(FIG. (C)) shows the texture rectangular area TRpad2 (FIG. (D)) that has been subjected to the expansion compensation processing.

FIG. 16 is a block diagram for explaining a conventional moving picture encoding device.

FIG. 17 is a diagram illustrating a flow of an encoding process performed by a conventional moving image encoding device.

FIG. 18 is a block diagram for explaining a conventional video decoding device.

FIG. 19 is a diagram showing a flow of a decoding process performed by a conventional video decoding device.

FIG. 20 is a block diagram for explaining a pixel synthesizer included in the conventional moving picture encoding apparatus or moving picture decoding apparatus.

FIG. 21 is a block diagram illustrating another configuration example of a conventional video encoding device.

22 is a diagram showing a flow of an encoding process by the conventional video encoding device shown in FIG. 21.

FIG. 23 is a block diagram illustrating another configuration example of a conventional video decoding device.

24 is a diagram showing a flow of a decoding process by the conventional video decoding device shown in FIG. 23.

[Explanation of symbols]

 101, 103 Moving picture encoding device (image processing device) 102, 104 Moving picture decoding device (image processing device) U1 Shape encoder U2a, U2b Shape memory U3 Texture encoder U5a, U5b, U8a, U8b Texture memory U7a, U7b Pixel compositor U4a, U4b First pixel compensator U6a, U6b Second pixel compensator U9a, U9b Pixel compensator U10 Shape decoder U11 Texture decoder Pa, Pb Complement texture signal Sin Shape signal Sout Decoded shape signal Srefa, Srefb Reference shape signal Sstr Shape coded stream Tin texture signal Tra, Trb, Trefa, Trefb Reference texture signal PTrefa, PTrefb Supplementary reference texture signal Tstr Texture coded stream Tout Decoded texture signal LSout Local decoding Shape signal LTout locally decoded texture signal Vin be-combined image signal (to be synthesized texture signal) Vout combined image signal (synthesized texture signals) Pos combined position indication signal Cs computer system FD floppy disk FDD floppy disk drive

Claims (12)

[Claims] 1. A shape signal representing a shape of an object constituting an image and a texture signal representing a picture of the object.
An encoding process including a local decoding process is performed on the moving image signal corresponding to the object for each object, and the decoded moving image signal obtained by the local decoding process is combined with the to-be-synthesized image signal. An image processing method, wherein the shape signal is subjected to an encoding process including a local decoding process for each object with reference to a processed shape signal, and a shape encoded stream and a locally decoded shape signal are obtained. A shape encoding process to be generated, and a texture encoding stream and a local decoding texture signal are applied to each texture object by performing an encoding process including a local decoding process on the texture signal with reference to the processed texture signal. A texture encoding process to be generated, and a pixel value of an outside-of-object pixel located outside the object, which constitutes the locally decoded texture signal, is compensated for. And a synthesizing process of synthesizing the compensated texture signal with a texture signal constituting the synthesized image signal based on the locally decoded shape signal. An image processing method characterized by being used as a processed texture signal. 2. The image processing method according to claim 1, wherein the compensation process is a process of replacing a pixel value of a pixel outside the object located outside the object with a pixel value of a pixel inside the object located inside the object. An image processing method comprising: 3. The image processing method according to claim 1, wherein the texture signal and the locally decoded texture signal form an object area which is an image space including the object. The object area is formed by performing a macroblock consisting of a predetermined number of pixels as a unit, and the above-described compensation processing is performed by calculating a pixel value of a pixel outside the object located outside the object in the boundary macroblock including the boundary of the object. A first supplementation process for replacing the pixel value of a pixel in an object located in the object with a pixel value of a pixel of an extra-object macroblock located outside the object in the object region. A second supplementation process for replacing the pixel value with a pixel value of a pixel in the object. 4. A decoding process for each object is performed on an image-encoded stream obtained by encoding a moving image signal corresponding to an object constituting an image for each object, and obtained by the decoding process. An image processing method for synthesizing the decoded moving image signal and the image signal to be synthesized, wherein a shape encoded stream obtained by encoding a shape signal representing the shape of the object for each object is referred to a decoded shape signal. A shape decoding process for decoding the object and generating a decoded shape signal for each object, and a texture coded stream obtained by coding a texture signal representing a picture of the object for each object, with reference to the decoded texture signal. And a texture decoding process of decoding for each object to generate a decoded texture signal; and an object outside image located outside the object, constituting the decoded texture signal. A complementing process for producing a compensated texture signal by supplementing the elementary pixel values; and a combining process for combining the compensated texture signal with a texture signal constituting the composite image signal based on the decoded shape signal. An image processing method comprising: using the compensated texture signal as the processed texture signal. 5. The image processing method according to claim 4, wherein the compensation process is a process of replacing a pixel value of a pixel outside the object located outside the object with a pixel value of a pixel inside the object located inside the object. An image processing method comprising: 6. The image processing method according to claim 4, wherein the decoded texture signal forms an object region that is an image space including the object, and the texture decoding process converts the object region. Is performed in units of a macroblock composed of a predetermined number of pixels, and the above-described compensation processing is performed in the boundary macroblock including the boundary of the object, by calculating a pixel value of a pixel outside the object located outside the object within the object. A first supplementation process for replacing the pixel value of the pixel of the macroblock located outside the object in the object area with the pixel value of the macroblock located outside the object in the object region. An image processing method comprising: a second supplementation process for replacing a pixel value with a pixel value. 7. A shape signal representing a shape of an object constituting an image and a texture signal representing a picture of the object.
An encoding process including a local decoding process is performed on the moving image signal corresponding to the object for each object, and the decoded moving image signal obtained by the local decoding process is combined with the to-be-synthesized image signal. An image processing device, wherein the shape signal is subjected to an encoding process including a local decoding process for each object with reference to a processed shape signal, and a shape encoded stream and a locally decoded shape signal are obtained. A shape encoder for generating, a shape memory for storing the locally decoded shape signal as the processed shape signal, and a
With reference to the processed texture signal, a texture coder that performs a coding process including a local decoding process for each object to generate a texture coded stream and a locally decoded texture signal, and the local decoding texture signal A pixel compensator configured to compensate for a pixel value of an extra-object pixel located outside the object to generate an compensated texture signal; and to calculate the compensated texture signal based on the locally decoded shape signal. An image processing apparatus, comprising: a pixel synthesizer that synthesizes a texture signal that forms a synthesized image signal; and a texture memory that stores the compensated texture signal as the processed texture signal. 8. A shape signal representing a shape of an object forming an image and a texture signal representing a picture of the object.
An encoding process including a local decoding process is performed on the moving image signal corresponding to the object for each object, and the decoded moving image signal obtained by the local decoding process is combined with the to-be-synthesized image signal. An image processing apparatus, for a shape signal forming a shape object region that is an image space including the object, referencing a processed shape signal, performing shape encoding processing including local decoding processing for each object. A shape encoder that applies a macroblock consisting of a predetermined number of pixels constituting the texture object region as a unit to generate a shape-encoded stream and a locally decoded shape signal; and A processed texture signal is referred to for a shape memory for storing as a shape signal and a texture signal for forming a textured object area which is an image space including the object. A texture code which performs a coding process including a local decoding process for each object in units of a macroblock consisting of a predetermined number of pixels constituting the texture object region to generate a texture coded stream and a locally decoded texture signal. For the local decoded texture signal, in the boundary macroblock including the boundary of the object, the pixel value of the pixel outside the object located outside the object, the pixel of the pixel inside the object located inside the object First compensating means for performing a first compensating process to replace the value with a value and outputting a first compensated texture signal; and converting the first compensated texture signal into the synthesized image based on the locally decoded shape signal. A pixel synthesizer for synthesizing with a texture signal constituting the signal; and a texture memo for storing the first compensated texture signal. In response to the first supplementary texture signal recorded in the texture memory, the pixel value of the pixel of the extra-object macroblock located outside the object in the object area is stored in the object located in the object. A second compensation means for performing a second compensation process for replacing the pixel value of the pixel with a pixel value, and outputting a second compensation texture signal obtained by the second compensation process to the texture encoder as the processed texture signal; An image processing apparatus comprising: 9. A decoding process for each object is performed on an image coded stream obtained by coding a moving image signal corresponding to an object constituting an image, and the decoding obtained by the decoding process is performed. An image processing apparatus for synthesizing a moving image signal with an image signal to be synthesized, comprising: encoding a shape signal representing a shape of an object for each object; A shape decoder that decodes each time to generate a decoded shape signal, a shape memory that stores the decoded shape signal as the processed shape signal, and a texture signal that represents a picture of the object is encoded for each object. A texture decoder for decoding the obtained texture encoded stream for each object with reference to the decoded texture signal to generate a decoded texture signal; A pixel compensator that composes a texture signal and compensates for a pixel value of a pixel outside the object located outside the object to generate a compensated texture signal; and, based on the decoded shape signal, the compensated texture signal. An image processing apparatus, comprising: a pixel synthesizer for synthesizing a texture signal constituting the image signal to be synthesized; and a texture memory for recording the compensated texture signal as the processed texture signal. 10. An image encoding stream obtained by encoding a moving image signal corresponding to an object constituting an image is subjected to a decoding process for each object, and a decoding obtained by the decoding process is performed. An image processing apparatus for synthesizing a moving image signal with an image signal to be synthesized, comprising: encoding a shape signal representing a shape of an object for each object; A shape decoder that generates a decoded shape signal that forms a shape object region that is an image space including an object; a shape memory that stores the decoded shape signal as the processed shape signal; A texture-encoded stream obtained by encoding a texture signal representing a pattern of each object for each object is decoded for each object with reference to the decoded texture signal, and in an image space including the object. A texture decoder that generates a decoded texture signal that forms a certain texture object region; and, for the decoded texture signal, a pixel outside the object located outside the object in a boundary macroblock including the boundary of the object. First compensation means for performing a first compensation process for replacing a pixel value with a pixel value of a pixel in an object located in the object and outputting a first compensation texture signal; and the first compensated texture signal A pixel synthesizer that synthesizes the texture signal constituting the image signal to be synthesized based on the decoded shape signal, a texture memory that stores the first compensated texture signal, and a texture memory that is recorded in the texture memory. Of the pixel of the extra-object macroblock located outside the object in the object area in response to the first supplementary texture signal A second interpolation process is performed to replace the prime value with a pixel value of a pixel in the object located in the object, and a second interpolation texture signal obtained by the second interpolation process is used as the processed texture signal. An image processing apparatus comprising: a second compensation unit that outputs to a texture decoder. 11. A program for performing, by a computer, an encoding process including a local decoding process on a moving image signal and a combining process of combining a locally decoded moving image signal obtained by the local decoding process with an image signal to be combined. A data storage medium storing, by the computer, the shape signal by performing an encoding process including a local decoding process for each object with reference to a processed shape signal, A shape encoding process for generating an encoded stream and a locally decoded shape signal; and performing a texture process including a local decoding process on the texture signal with reference to the processed texture signal for each object. A texture encoding process for generating an encoded stream and a locally decoded texture signal, and the local decoded texture signal A compensating process for compensating a pixel value of a pixel outside the object located outside the object to generate a compensated texture signal used as the processed texture signal; and forming the compensated texture signal into the locally decoded shape. A data storage medium, which is an image encoding program for performing a combining process for combining a texture signal constituting the combined image signal based on the signal. 12. A decoding process for an encoded image stream obtained by encoding a moving image signal corresponding to an object constituting an image for each object, and a decoded moving image signal obtained by the decoding process is synthesized. What is claimed is: 1. A data storage medium storing a program for performing, by a computer, a combining process for combining with an image signal, the program comprising: a shape coding obtained by coding, by a computer, a shape signal representing a shape of an object for each object; A shape decoding process for decoding a stream for each object with reference to the processed shape signal to generate a decoded shape signal, and a texture encoding obtained by encoding a texture signal representing a picture of the object for each object. The texture decoding unit decodes the stream for each object with reference to the processed texture signal to generate a decoded texture signal. Encoding processing, which constitutes the decoded texture signal, supplements the pixel values of pixels outside the object located outside the object, and generates a compensated texture signal to be used as the processed texture signal. A data storage medium, which is an image decoding program for performing a synthesizing process of synthesizing a compensated texture signal with a texture signal constituting the synthesized image signal based on the decoded shape signal.