JPH11346329A - Image processor, its method and image processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To synthesize images obtained by applying suitable image processing to an image and a slave object attended with a master object in the image such as a shadow. SOLUTION: A shadow object is extracted in the steps S107, S108 from a closed area including a master object that is extracted in the step S106 and the master/slave objects are registered in a table. Then the master/slave objects are suitably synthesized in the S113 with a background image received in the S111 according to a synthesis method designated in the S110.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to an image processing apparatus and method, and more particularly, to an image processing apparatus and method for extracting and processing objects in an image.

[0002]

2. Description of the Related Art Conventionally, as a moving image encoding method, an encoding method such as MPEG-1, MPEG-2, or H.261 for encoding in a frame unit has been known. In these encoding methods, a moving image can be encoded with high efficiency by using the correlation between frames.

On the other hand, as a more efficient encoding method, a method of extracting an object from an image, encoding the object separately from the background, and transmitting only the object, such as MPEG-4, has been studied. If such an encoding method is used, the area of an image to be transmitted can be reduced, so that a moving image can be transmitted over a low bit rate line. In addition, the receiving side can select and display the object,
By changing the arrangement, size, and the like, it has become possible to perform suitable display on the receiving side. In addition, editing that can be combined with another background can now be easily performed.

[0004] Also, a JPEG encoding method is known as a still image encoding method.
For example, MMR coding is used for character images, and J is used for photographic images.
A method of assigning a PEG encoding method has also been proposed.

Here, as a method for extracting an object from a moving image, an extraction technique using a chroma key generally used in a broadcast station or the like is known. This is a technique in which an object such as a person is photographed in front of a blue background, and the person object is cut out by an electric signal.
Note that, in the case of object photographing using a chroma key, the photographing is generally performed in a photographing studio or the like, and since the lighting conditions are adjusted, no shadow of the object is generated.

In still images, an image area is automatically separated. When an object is extracted from an image, an area is manually designated by a user, or areas of similar colors are integrated. As a result, a desired image area is cut out.

[0007]

However, the objects that can be extracted by the above-mentioned chroma key are limited to relatively small objects within a range where the background can be photographed with a blue color. Therefore, a relatively large object may be extracted from a moving image of a natural image. As one of the methods, a method of inputting a background image in advance, extracting an object from a difference image between the background image and the input image, and acquiring color information and the like constituting the background in advance are provided. In addition, a method of extracting an area of a color different from the color information in an input image (image coding symposium PSCJ97I-3.15) and the like are known.

Hereinafter, an example in which the picture of the helicopter 1051 is cut out as an object from an image 1050 as shown in FIG. 21 will be described. That is, the difference between the image 1050 in FIG. 21 and the background image 1052 in FIG. 22 is obtained, and processing such as noise removal is performed, thereby extracting the object 1053 in FIG. By combining this object 1053 with the background image 1054 shown in FIG. 24, a new image shown in FIG. 25 can be edited.

However, objects in a moving image of a natural image often have shadows unlike natural shooting for chroma key synthesis because shooting is performed in natural light. Therefore, in the image of FIG. 25, a helicopter shadow is generated in the air, which results in an unnatural synthesis result.

[0010] In the image 1055 shown in FIG. 26, a cow 1056 eating grass on the lawn is similarly cut out, and as a result, 1057 shown in FIG. 27 is obtained as an object. When this object 1057 is combined with, for example, the background of an urban area as shown in FIG. 28, the shadow of the cow included in the object 1057 is also combined as it is. The original image 10
Since the image of the grass on the lawn at 55 is included, when combined with the asphalt background as shown in FIG. 28, the image becomes quite unnatural.

Further, considering a case where an object having no shadow is obtained by chroma key or the like and the object is synthesized, in this case, the three-dimensional positional relationship between the object and the synthesized background and the setting of a light source and the like are determined by a computer. It can be performed by processing or the like. However, this setting is effective, for example, in a limited environment such as a studio, but it is not possible to generate a shadow or the like for an object once input two-dimensionally as a moving image. There is a problem in that an unnatural composite image without the image is generated. FIG. 29 shows an image obtained by combining the cow 1056 without a shadow, and it can be seen that considerable unnaturalness has occurred. Such unnaturalness due to the combination without shadows becomes more remarkable when the image is combined with a background image in which many shadows of other objects are generated.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problem, and has been made in consideration of the above-described problems. It is an object of the present invention to provide an image processing apparatus capable of performing suitable image processing on an object including a main object and a subsidiary object such as a shadow, for example. It is an object to provide an apparatus and a method thereof.

[0013]

As one means for achieving the above object, the image processing apparatus of the present invention has the following arrangement.

That is, extracting means for extracting at least one object from an image, main object setting means for setting a main object which is a main object among the extracted objects, and Slave object setting means for setting a slave object attached to the main object; main object processing means for performing image processing on the main object; and an image corresponding to the image processing in the main object processing means for the slave object. And subordinate object processing means for performing processing.

[0015] Further, extraction means for extracting at least one object from an image, classification means for classifying the extracted object into a main object and its subordinate objects, and a main object code for encoding the main object Encoding means, and slave object encoding means for encoding the slave object.

Object extracting means for extracting at least one main object from an image; sub-object generating means for generating a sub-object attached to the main object; main object processing for performing image processing on the main object Means, a slave object processing means for performing image processing on the slave object in accordance with image processing in the main object processing means,
It is characterized by having.

[0017]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment according to the present invention will be described below in detail with reference to the drawings.

<First Embodiment> FIG. 1 is a block diagram showing a configuration of a moving picture editing apparatus according to the present embodiment. In the present embodiment, a case where software processing using a CPU is performed will be described. In FIG. 1, reference numeral 101 denotes a CPU that executes software and controls the entire apparatus. 1
02 is a terminal to which a user (not shown) instructs an editing process. A storage device 103 stores image data used for editing. Reference numeral 104 denotes a memory, which stores an OS for controlling the entire apparatus, software executed by the CPU 101, a work memory required for various processes, and the like. An image memory 105 stores image data required for each operation. Reference numeral 106 denotes a synthesizer that performs image synthesis processing, 107 denotes a frame memory that stores a synthesis result, and 108 denotes a frame memory.
This is a monitor that displays the contents of. A bus 109 exchanges various data, control commands, and image data between the components.

In such a configuration, the user first activates the entire apparatus from the terminal 102. Further, an image including a background image used for image synthesis and an object to be synthesized therein is selected. For example, the image 1054 of FIG. 24 as a background image to be synthesized, as an image including an object,
Assume that an image 1050 including the helicopter 1051 in FIG. 21 is selected. At this time, the background image 1052 in FIG. 22 is simultaneously selected as the original background image of the image 1050 including the object. Subsequently, an image combining process is performed. The procedure is shown in a flowchart of FIG.

First, an original background image 10 shown in FIG.
The image data of 52 is read from the storage device 103 and written in a predetermined area of the image memory 105 (step S101). Then, an image 1050 including the helicopter 1051 as an object
Is read from the storage device 103 and written in a predetermined area (Object 1) of the image memory 105 (step S10).
2). At this time, the frame counter n is set to 0.
Note that each of the background image and the image including the object needs to be designated in advance by the user, but the designation method is not limited.

Hereinafter, the object extraction processing of each frame is performed according to the frame counter n. First, a difference image between the image of the n-th frame and the background image 1052 is obtained (step S104). Then, noise is removed by setting the pixel value of the difference image equal to or smaller than a predetermined threshold value to 0 (step S1).
05), after dividing a set of pixels having a pixel value of 0 or more as a closed area, extracting only a closed area having a size equal to or less than a predetermined size, and writing the same in a predetermined area (Object 1) of the image memory 105
(Step S106).

Then, in step S107, the image memory
From the closed area held in Object 1 of 105, an area having a similar difference value is extracted, written to the frame memory 107, displayed on the monitor 108, and the user selects a shadow area based on the display. And specify it on the terminal 102. With this operation, a shadow area is extracted. Then, of the pixels in the closed area held in Object1, the pixels included in the designated shadow area are written to a predetermined area (Object2) on the image memory 105. Further, it is assumed that this shadow area is a shadow of an object, the position and size of the shadow area in the screen are stored in a position table of the memory 104, and that this shadow is also one object (subordinate object). For example, it is registered as No. 1. The shadow image as the slave object obtained in this way is the shadow 1060 shown in FIG.
It is.

Subsequently, in step S108, of the pixels in the closed region held in Object1, other pixels are removed while leaving only pixels included in the region as an object (main object) to be synthesized. That is, a shadow object, which is a slave object, is removed from the closed area of the object, and a master object is generated. Then, the main object is registered in, for example, No. 2 of the object table in the memory 104, and the position, size, and the like in the screen are described in the position table. At this time, the number “1” of the object table in which the shadow, which is the slave object related to the main object, is registered in the second LINK column of the object table (step S108). The image of the main object obtained in this manner is the helicopter 1 shown in FIG.
061.

Then, 1 is added to the frame counter n, and processing for the next frame is performed (step S109). In the first and subsequent frames, by selecting an area close to the average difference value of the shadow difference image obtained in the previous frame, the shadow area can be extracted without the user's confirmation. While the frame counter n is smaller than the total number of frames to be processed (step S103), the processing of steps S104 to S109 is repeated.

FIG. 5 shows an example of data written to the object table of the memory 104 as a result of the above processing.
In the figure, “Shade” is the first object table.
The slave object showing the shadow "Helicopter" in the second
Is registered, and it can be seen from the LINK column that the main object is linked to the first registered slave object.

Returning to FIG. 2, when the frame counter n exceeds the total number of frames (step S103), the CPU 101 prompts the user to input a shadow processing method, and the user inputs the processing method from the terminal 102 (step S110). ). Here, the shadow processing method specifies either to leave the shadow as it is at the time of the synthesis or to delete the shadow because it is inconvenient to leave the shadow in the synthesized image. Thereafter, the image data of the image 1054 (FIG. 24) serving as a new background is read out from the storage device 103 by image synthesis, and a predetermined area (B
ackground) (step S111). At this time, the frame counter n is set to 0. Then, the combining process is performed sequentially from the first frame, and the combined image is displayed on the monitor 108 (step S113).

Here, the details of the combining / displaying process shown in step S113 will be described with reference to the flowchart of FIG.

First, the n-th frame of the background image 1054 is written into the frame memory 107 (step S120). Subsequently, the image of the helicopter 1061 (FIG. 4), which is the main object registered in the object table, is stored in Object1 of the image memory 105.
(Step S121), and overwrites the position indicated by the position table on the frame memory 107 or a new position designated by the user (step S122). In the object table in the memory 104, for example, an object having "1" or some description in its LINK column is recognized as a main object. In the example shown in FIG. 5, “Shade” registered in No. 1 of the object table is a slave object, and “Helicopter” registered in No. 2 is recognized as a main object to be combined.

Then, the shadow processing method specified in step S110, that is, shadow processing according to whether to overwrite or delete, is performed. (Step S123). When the shadow is used as it is, the corresponding slave object shadow 1060 is read from Object2 of the image memory 105 based on the LINK column of the main object in the object table, and the position of the shadow indicated by the position table and the helicopter object According to the positional relationship of 1061, the frame memory 10
After overwriting on step 7 (step S124), the process proceeds to step S125.

On the other hand, when the shadow is to be deleted, the process proceeds to step S125 ignoring the slave object. For example, figure
When combining with the background image 1054 shown in FIG.
Since it is inconvenient that the shadow 1061 remains as shown in FIG. 25, shadow deletion is selected. Then, in step S125, the contents of the frame memory 107 are displayed on the monitor. FIG. 7 shows an image synthesized and displayed in this manner.

Referring back to FIG. 2, after the synthesizing / displaying process of S113 described above is completed, 1 is sequentially added to the frame counter n.
(Step S114), the synthesis / display processing for all frames is repeated. When the frame counter n exceeds the total number of frames (step S112), the CPU 101 ends all the processing.

As described above, according to the present embodiment,
By extracting a shadow region from an object to be synthesized and selectively performing the shadow processing according to the situation of the image to be synthesized, a synthesized image without a sense of incongruity can be easily created.

In this embodiment, a method of extracting a shadow area or an object based on a difference image from a background has been described. However, the present invention is not limited to this. For example, a 1997 video media processing symposium (IMPS97 ) I-3.15
The extraction method described in “Morphological segmentation using prior knowledge information in sports programs” (Oct. 8, 1997, Naemura et al.) May be used. This is a method in which information such as a ground where no player is an object is acquired in advance as a background, and an object (player) is extracted from a moving image based on a color histogram or the like of the background.

The image data stored in the storage device 103 may be encoded. In this case, the object may be divided and encoded for each object as in the MPEG-4 encoding method, or the shadow and the object may be stored separately.

The processing for only one frame of a moving image is the same as the processing for a still image. Therefore, by performing the same operation as in the present embodiment on a still image, it is needless to say that the same effect as in the present embodiment can be obtained in a still image.

In this embodiment, the case in which the number of objects is one has been described. However, the number of objects is not limited to one. Depending on the capacity of each table in the memory 104 and the storage area of the image memory 105, It can handle multiple objects.

<Second Embodiment> Hereinafter, a second embodiment according to the present invention will be described.
An embodiment will be described.

FIG. 8 is a block diagram showing the configuration of a moving picture editing system according to the second embodiment. In the figure,
TV cameras 201 and 202 capture moving images, convert them into digital signals, and output the digital signals. Hereinafter, a case will be described in which the image 1055 in FIG. 26 is captured from the TV camera 201 and the image 1070 illustrated in FIG. 9 is captured from the TV camera 202. An object extractor 203 extracts an object from a moving image. A shadow extractor 204 extracts a region representing the shadow from the extracted object as a slave object (shadow object), and further outputs a main object obtained by removing the shadow object from the object. A shadow encoder 205 encodes the extracted shadow object.
Is an object encoder for encoding the main object. The object encoder 206 encodes the main object as a moving image. Here, an example in which encoding is performed according to the MPEG-4 encoding method will be described. Of course, the encoding method is not limited to this.

Reference numeral 207 denotes an encoder for encoding a moving image captured by the TV camera 202. The encoding method is not particularly limited. Here, an example in which encoding is performed by the MPEG-1 encoding method will be described. I do. Reference numerals 208 and 209 denote transmitters, which transmit the generated encoded data to the communication lines 210 and 211. twenty one
2, 213 are receivers for receiving the encoded data. 214
Is an editing instruction device for the user to specify editing contents. twenty one
Reference numeral 5 denotes a moving image editing apparatus which is a feature of the present embodiment. Reference numeral 216 denotes an encoder, which performs encoding according to the MPEG-1 encoding method, but the encoding method is not particularly limited. twenty one
Reference numeral 7 denotes a transmitter for transmitting encoded data to the communication network 218. A storage device 219 stores encoded data.

The operation of the moving picture editing system according to the second embodiment having such a configuration will be described below. First, moving images are captured by the TV cameras 201 and 202, and are encoded. From the moving image input from the TV camera 201, an object area including the main object and its shadow object is extracted by the object extractor 203, but the extraction method is not particularly limited. For example, the method of storing the previously captured background image and extracting the object area from the difference image described in the first embodiment (FIG. 2)
Steps S104 to S109) may be used, or another method may be used. In the second embodiment, an example will be described in which a cow 1056 and its shadow are extracted as an object area from an image 1055 based on a background image captured in advance. The extracted object area is input to the shadow extractor 204,
Although it is separated into a main object and its shadow object, the method of this separation is not particularly limited. For example, as described in the first embodiment, separation may be performed based on the uniformity of the difference value from the background, or manual extraction by the user may be performed. Here, the cow 1071, which is the main object thus extracted, is shown in FIG. 10, and the shadow object 1072 is shown in FIG. It is needless to say that each object extracted here is substantially cut out as a rectangular area including the object.

The cow 1071 as the extracted main object is input to the object encoder 206, and is encoded by the MPEG-4 encoding method. According to the MPEG-4 encoding method, mask information representing the shape of an object and pixel values of an image are encoded. On the other hand, the extracted shadow object 10
72 is input to the shadow encoder 205. The shadow encoder 205 performs shape encoding and shadow intensity encoding, as in the MPEG-4 encoding method. The shadow extractor 204 outputs, to the shadow encoder 205, an image corresponding to the position of the shadow object among the previously captured background images referred to by the object extractor 203 in the preceding stage.

FIG. 12 shows a block diagram of the shadow encoder 205. As shown in FIG. In the figure, reference numeral 230 denotes a terminal for inputting an image of a rectangular area including a shadow object from the shadow extractor 204. Reference numeral 231 denotes a terminal for inputting a background image of a shadow object area from the shadow extractor 204. A mask generator 232 generates a mask indicating the shape information of the shadow object by expressing whether or not the pixel is included in the shadow object by a 1/0 binary image. 233 is a mask memory for storing the mask. An encoder 234 arithmetically encodes the contents of the mask memory 233. Reference numeral 235 denotes a differentiator for calculating the difference between the background pixel at the same position and the pixel value of the shadow object for the pixel determined to be present in the shadow area by the mask. An average value calculator 236 divides the sum of the obtained difference values for the pixels in the area by the number of pixels having a pixel value of 1 in the mask, that is, the number of pixels existing in the shadow object, and obtains an average value. An encoder 237 assigns a code to the calculated average value. Reference numeral 238 denotes a combiner for combining the codes generated by the encoder 234 and the encoder 237 and adjusting the codes. Reference numeral 239 denotes a terminal that outputs the combined encoded data.

The operation of the shadow encoder 205 having such a configuration will be described below. The image including the shadow object (shadow 1072) input from the terminal 230 is
2 and input to the differentiator 252. The mask generator 232 determines whether the pixel is included in the shadow object depending on whether the pixel is inside or outside the closed area of the shadow, and outputs the result as a mask of a 1/0 binary image. . That is, the mask indicates shadow shape information. Here, FIG. 13 shows the shadow 10 of FIG.
72 shows the mask information in 72. This mask is stored in the mask memory 233. The mask of the binary image stored in the mask memory 233 is encoded by the encoder 234. Also,
In the differentiator 235, the difference between the pixel value of the shadow object and the pixel value of the background image is obtained, and the difference is input to the average calculator 236. The average value calculator 236 calculates the total sum of these differences, and divides the pixel value in the mask stored in the mask memory 233 by “1”, that is, the number of pixels in the shadow area to obtain the average value. This average value is encoded by the encoder 237. The encoded data generated by the encoder 234 and the encoder 237 are combined by the combiner 238, adjusted, and output from the terminal 239.

Returning to FIG. 8, the coded data generated by the object coder 206 and the coded data generated by the shadow coder 205 are input to a transmitter 208, where the coded data is prepared and a communication line is prepared. Sent to 210.

On the other hand, the image data picked up by the TV camera 202 is encoded by the encoder 207 according to the MPEG-1 encoding method, and transmitted to the communication line 211 via the transmitter 209. TV
The image data captured by the camera 202 becomes a background image of the composite image (background image 1070 in FIG. 9).

The receivers 212 and 213 receive these encoded data and input them to the moving picture editing device 215. The moving image editing device 215 edits a moving image according to an instruction input by the user from the editing instruction device 214. Hereinafter, a case where the position of the object to be combined in the background image 1070, the size of the object, and the intensity of the shadow are input to the moving image editing device 215 as an editing instruction by the user will be described as an example.

FIG. 14 shows a block configuration of the moving image editing device 215. In the figure, reference numeral 250 denotes a terminal for inputting encoded data of an object to be synthesized from the receiver 212. Reference numeral 251 denotes a terminal for inputting encoded data of a background image from the receiver 213. Reference numeral 252 denotes a terminal for inputting an editing instruction from the editing instruction device 214. Reference numeral 253 denotes a separator, from encoded data input from the receiver 212, encoded data representing the shape of the main object, encoded data of the pixel value of the main object, encoded data indicating the intensity of the shadow, and shadow data. The coded data representing the shape is separated. Reference numeral 254 denotes a mask decoder that decodes mask information representing the shape of the main object, and 255 denotes a mask memory that stores the result. 256 is an object decoder for decoding the pixel value of the main object, and 257 is an object memory for storing the result. 258 is a decoder for decoding the intensity of the shadow, and 259 is a latch for storing the result. 260
Is a mask decoder for decoding mask information representing the shape of the shadow, and 261 is a mask memory for storing the result.

Reference numeral 262 denotes a decoder for decoding the encoded data of the background image input from the receiver 213, and performs a decoding process of the MPEG-1 encoding method. A frame memory 263 stores the result. 264, 265, 266, and 267 enlarge / enlarge the input image in accordance with the instruction from the editing instruction device 214.
This is a deformer that performs various deformations such as reduction / magnification processing, rotation processing, and affine transformation processing. 268 is a shadow synthesizer,
According to an instruction from the editing instruction device 214, a shadow object is synthesized with the background image. Reference numeral 269 denotes an image synthesizer which synthesizes the output of the shadow synthesizer 268 with the main object deformed by the deformers 264 and 265. Reference numeral 270 denotes a terminal for outputting a result of the synthesis.

The moving picture editing apparatus 21 having such a configuration
The operation in 5 will be described below. Terminal 250 is
The encoded data received by receiver 212 is input to separator 253. The demultiplexer 253 divides the coded data with reference to the header of each code and outputs the coded data to the respective subsequent decoders. That is, the encoded data representing the shape of the main object is supplied to the mask decoder 254, the encoded data of the pixel value of the main object is supplied to the object decoder 256, the encoded data of the shadow intensity is supplied to the decoder 258, and the The encoded data of the mask is output to decoder 260.

The mask decoder 254 decodes the mask data of the main object, reproduces the mask information, and
Store in 255. The object decoder 256 decodes the encoded data of the image of the main object, reproduces the pixel value of the main object, and stores it in the object memory 257. Decoder 258 decodes the shadow intensity of the shadow object and latch 2
Store in 59. The mask decoder 260 decodes the mask data of the shadow object, reproduces the mask information, and stores the mask information in the mask memory 261. The decoder 262 decodes the encoded data of the background image, reproduces the pixel value of the background image, and
Store in 263.

The parameters of transformation processing such as scaling processing, rotation processing, affine transformation processing and the like input from the editing instruction device 214 are inputted to the transformers 264, 265, 266 and 267. Hereinafter, a case will be described as an example where the composition is performed after the object is reduced according to the size of the background image.
The transform units 264, 265, and 266 receive the reduction ratio as a parameter from the editing instruction device 214. The deformer 264 reduces the content of the mask memory 255 and outputs it. The deformer 265 reduces the contents of the object memory 257 and outputs the reduced contents. The deformer 266 reduces the contents of the mask memory 261 and outputs it. In this case, since only the object needs to be reduced, the content of the frame memory 263 is output as it is in the deformer 267 for deforming the background image.

The shadow synthesizer 268 actually synthesizes the shadow in accordance with the following equation based on the average value m representing the shadow intensity stored in the latch 259 and the shadow intensity change parameter p input from the editing instruction device 214. The intensity S of the shadow to be calculated is calculated.

S = m × p (1) Further, the position at which the shadow object is synthesized is determined by the position at which the main object input from the editing instruction device 214 is synthesized,
And the relative positional relationship between the main object and the shadow object. Then, a shadow object is synthesized at the calculated position with respect to the deformation result of the background image output from the deformer 267. Specifically, the mask memory
According to the deformation result of the mask information held in 261, that is, the pixel at the position where the shadow object is synthesized according to the shape information of the shadow to be synthesized, the pixel value b of the background image and the intensity S of the shadow are calculated by the following equation. Convert to the new pixel value C obtained.

C = b−S (2) The background image in which the shadow object is synthesized in this manner is input to the image synthesizer 269. In the image synthesizer 269, the main object is synthesized according to the position where the main object input from the editing instruction device 214 is to be synthesized and the mask data (shape information) of the main object stored in the mask memory 255. As for the pixel, the pixel value of the background image is replaced with the pixel value of the main object.
Then, the result of the synthesis is sent from the terminal 270 to the encoder shown in FIG.
Output to 216. Here, FIG. 15 shows the background image 10 shown in FIG.
Shown at 70 is an image obtained by reducing a cow 1056 as an object and combining it with a shadow. According to FIG. 15, since the image of the asphalt road surface, which is the background, is reflected in the shadow portion of the cow, it can be seen that the sense of incongruity was considerably suppressed as compared with the image of FIG. 28 shown in the conventional example. In other words, it can be seen that natural composition is also performed for the shadow portion of the main object.

Returning to FIG. 8, the encoder 216 encodes the combined image data by the MPEG-1 encoding method and sends the encoded image data to the communication network 218 via the transmitter 217. To be stored.

As described above, according to the second embodiment, the number of pixels to be coded can be reduced by extracting the shadow area from the object to be synthesized and coding only the main object. By displaying and encoding the extracted shadow region as a shadow object using only its shape and intensity, the amount of information can be significantly reduced, and the communication cost and the capacity of the storage medium can be reduced.

Further, by adjusting the intensity of the deformation and the shadow according to the situation of the image to be synthesized, it is possible to easily generate a synthesized image without a sense of incongruity.

In the second embodiment, by designating the shadow intensity to be 0, it is also possible to perform synthesis without adding a shadow as in the first embodiment.

Further, in the second embodiment, an example in which the shadow encoder 205 encodes the representative value of the shape and the intensity of the shadow object has been described. However, the present invention is not limited to this, and the shadow intensity is not limited to this. Alternatively, the difference value from the background image may be used, or the transmittance when the black pixel is superimposed on the black image may be used. Also, multi-value encoding for the difference value may be performed.

Further, the modification processing in the second embodiment is not limited to the above-described example, and may be another modification processing. Also, it has been described that the encoder 216 performs MPEG-1 which is encoding on a frame basis.
Encoding may be performed again while each object is separated as in the EG-4 encoding method.

Since the processing for only one frame of a moving image is the same as the processing for a still image,
It is needless to say that by performing the same operation on the still image as in the second embodiment, the same effect as in the second embodiment can be obtained on the still image.

Third Embodiment Hereinafter, a third embodiment according to the present invention will be described.
An embodiment will be described.

FIG. 16 is a block diagram showing the configuration of the moving picture editing system according to the third embodiment. In the figure, the same components as those in FIG. 8 shown in the above-described second embodiment are denoted by the same reference numerals, and description thereof will be omitted. That is, in the configuration shown in FIG. 16, when transmitting the image of the TV camera 201,
It is characterized in that shadow extraction and encoding are not performed.

In such a configuration, the TV cameras 201 and 2
In 02, a moving image is captured and encoded. TV camera 20
The main object is extracted by the object extractor 303 from the moving image input from 1 in the same manner as the processing in the object extractor 203 and the shadow extractor 204 in the second embodiment. Here, it is assumed that the cow 1056 is extracted from the image 1055 shown in FIG. 26 and the main object 1071 shown in FIG. 10 is obtained. The cow 1071 as the extracted main object is input to the object encoder 206, and is encoded by the MPEG-4 encoding method. According to the MPEG-4 encoding method, mask information representing the shape of an object and pixel values of an image are encoded. On the other hand, image data picked up by the TV camera 202 is encoded by the encoder 207 according to the MPEG-1 encoding method, and transmitted to the communication line 211 via the transmitter 209. TV camera 20
The image data captured in step 2 becomes the background image of the composite image
(Background image 1070 in FIG. 9).

The receivers 212 and 213 receive these encoded data and input them to the moving picture editing apparatus 300. The moving image editing device 300 edits a moving image in accordance with an instruction input by the user from the editing instruction device 214. Hereinafter, a case where the position of the object to be combined in the background image 1070, the size of the object, the strength of the shadow, and the direction of the shadow are input to the moving image editing device 215 as an editing instruction by the user will be described as an example.

FIG. 17 shows a block configuration of the moving image editing apparatus 300. In the figure, components having the same functions as those in FIG. 14 shown in the above-described second embodiment are denoted by the same reference numerals, and detailed description will be omitted. Reference numeral 301 denotes a shadow generator which generates a shadow as a slave object from mask information indicating the shape of the cow 1071 as a main object. Reference numeral 302 denotes a shadow synthesizer that synthesizes the generated shadow.

The moving picture editing apparatus 300 having such a configuration
, The terminal 250 inputs the encoded data received by the receiver 212 to the separator 253. The demultiplexer 253 divides the coded data with reference to the header of each code and outputs the coded data to the respective subsequent decoders. That is, encoded data representing the shape of the object is output to the mask decoder 254, and encoded data of the pixel value of the object is output to the object decoder 256.

The mask decoder 254 decodes the mask data of the object, reproduces the mask information, and reproduces the mask information.
Store in 5. The object decoder 256 decodes the encoded data of the image of the object, reproduces the pixel value of the object, and stores it in the object memory 257. Decoder 262
Decodes the encoded data of the background image, reproduces the pixel value of the background image, and stores it in the frame memory 263.

The parameters of the transformation processing such as scaling processing, rotation processing, affine transformation processing and the like input from the editing instruction device 214 are input to the deformers 264 and 265 and the shadow generator 301. Hereinafter, a case where the object is reduced in accordance with the size of the background image will be described as an example. Transformers 264, 2
Numeral 65 inputs the reduction ratio as a parameter from the editing instruction device 214. The deformer 264 reduces the content of the mask memory 255 and outputs it. The deformer 265 reduces the contents of the object memory 257 and outputs the reduced contents. Note that the deformer 267 outputs the contents of the frame memory 263 holding the background image as it is.

The shadow generator 301 reads the binary image representing the shape information from the mask memory 255 and inverts the binary image to indicate the shadow (sub-object) of the cow 1071 as the main object.
Create the basic shape of. FIG. 18 shows the basic shape 1080 of the generated shadow. The editing instruction device 21
Enter the parameters to change the shape of the shadow from 4. The deformation parameter of this shape is determined in consideration of the direction of the light ray in addition to the deformation parameters given to the deformers 264 and 265. It should be noted that these adjustments may be made again after actually synthesizing the images. Also, the once determined deformation parameters can be used for editing another frame. Further, if there are a plurality of objects to be combined, it is possible to easily use the deformation parameters of another object, or to obtain the deformation parameters from the arrangement of objects having a shadow object. Further, it is also possible to detect a shadow existing in the background image to be synthesized and determine the most appropriate ray direction in the background from the positional relationship with the object forming the shadow. FIG. 19 shows an example in which the basic shape 1080 of the shadow is deformed by these deformation parameters.

In the shadow synthesizer 302, the image of the shadow, which is the slave object generated by the shadow generator 301, is represented by its shape, the intensity of the shadow output from the editing instruction device 214, and the position where the shadow is synthesized. In accordance with the transformation result of the background image. The background image in which the shadow as the slave object is synthesized in this way is input to the image synthesizer 269. In the image synthesizer 269, the pixels on which the main object is to be synthesized have a background according to the position at which the main object input from the editing instruction device 214 is to be synthesized and the mask data of the main object held in the mask memory 255. Replace the pixel values of the image with the pixel values of the main object. Then, the synthesis result is sent from the terminal 270 to the encoder 2 shown in FIG.
Output to 16. Here, FIG. 20 shows the background image 107 shown in FIG.
An image obtained by reducing the cow 1056 as an object to 0 and adding a shadow to the image is shown. As can be easily understood from the comparison with the image of FIG. 15 shown in the second embodiment, the third
In the embodiment, the direction of the shadow can be arbitrarily changed.

Returning to FIG. 16, the encoder 216 encodes the combined image data by the MPEG-1 encoding method and sends it out to the communication network 218 via the transmitter 217 or to a predetermined position in the storage device 219. To be stored.

As described above, according to the third embodiment, a shadow, which is an object according to an arbitrary direction, is generated and added based on the shape information of the main object to be synthesized. It is possible to easily suppress discomfort in the synthesized image, for example, by matching the direction of the shadow being cast with the direction of the shadow of the main object.

Note that the modification processing in the third embodiment is not limited to the above-described example, and may be another modification processing. Also, it has been described that the encoder 216 performs MPEG-1 which is encoding on a frame basis.
Encoding may be performed again with each object separated as in the -4 encoding method.

In the third embodiment, an example in which an image obtained by combining a shadow with the main object is encoded by the MPEG-1 encoding method has been described. Of course, coding using the strength of the shadow or coding using the MPEG-4 coding method may be performed.

Since the processing for only one frame of a moving image is the same as the processing for a still image,
By performing the same operation on the still image as in the third embodiment, it goes without saying that the same effect as in the third embodiment can be obtained on the still image.

<Other Embodiments> Even if the present invention is applied to a system including a plurality of devices (for example, a host computer, an interface device, a reader, a printer, etc.), an apparatus including one device (for example, For example, the present invention may be applied to a copying machine, a facsimile machine, and the like.

Further, an object of the present invention is to supply a storage medium storing a program code of software for realizing the functions of the above-described embodiments to a system or an apparatus, and to provide a computer (or CPU) of the system or the apparatus.
And MPU) read and execute the program code stored in the storage medium.

In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiment, and the storage medium storing the program code constitutes the present invention.

As a storage medium for supplying the program code, for example, a floppy disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD
-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

When the computer executes the readout program code, not only the functions of the above-described embodiment are realized, but also the OS (Operating System) running on the computer based on the instruction of the program code. ) May perform some or all of the actual processing, and the processing may realize the functions of the above-described embodiments.

Further, after the program code read from the storage medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, based on the instructions of the program code, It goes without saying that the CPU included in the function expansion board or the function expansion unit performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments.

[0083]

As described above, according to the present invention, it is possible to perform suitable image processing on an object including a main object and a subsidiary object attached thereto, such as a shadow, for example.

For example, by associating a master object with a slave object and reflecting the image processing result of the master object on the image processing result of the slave object, it is possible to realize a natural image synthesis without a sense of discomfort with a small amount of information. it can.

Further, a shadow which is a slave object can be generated and added to an object having no slave object based on its shape, so that a more natural image can be synthesized.

[0086]

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a moving image editing device according to an embodiment of the present invention.

FIG. 2 is a flowchart illustrating an image editing processing procedure according to the embodiment.

FIG. 3 is a diagram illustrating a state of a shadow that is a slave object in the present embodiment.

FIG. 4 is a diagram illustrating a state of a main object in the present embodiment.

FIG. 5 is a diagram for describing the contents of an object table in the present embodiment.

FIG. 6 is a flowchart illustrating a synthesis / display processing procedure according to the present embodiment.

FIG. 7 is a diagram illustrating a result of image composition in the present embodiment.

FIG. 8 is a block diagram illustrating a configuration of a moving image editing system according to a second embodiment of the present invention.

FIG. 9 is a diagram illustrating a background image according to the second embodiment.

FIG. 10 is a diagram illustrating a state of a main object according to the second embodiment.

FIG. 11 is a diagram illustrating a state of a shadow object according to the second embodiment.

FIG. 12 is a block diagram illustrating a configuration of a shadow encoder according to the second embodiment.

FIG. 13 is a diagram illustrating a mask of a shadow object according to the second embodiment.

FIG. 14 is a block diagram illustrating a configuration of a moving image editing device according to a second embodiment.

FIG. 15 is a diagram illustrating a result of image composition in the second embodiment.

FIG. 16 is a block diagram illustrating a configuration of a moving image editing system according to a third embodiment of the present invention.

FIG. 17 is a block diagram illustrating a configuration of a moving image editing device according to a third embodiment.

FIG. 18 is a diagram illustrating a basic shape of a shadow object according to the third embodiment.

FIG. 19 is a diagram illustrating a modification of the basic shape of the shadow object according to the third embodiment.

FIG. 20 is a diagram illustrating a result of image composition in the third embodiment.

FIG. 21 is a diagram illustrating an example of a frame image in a moving image including an object.

FIG. 22 is a diagram illustrating an example of a background image that does not include an object.

FIG. 23 is a diagram illustrating extracted objects.

FIG. 24 is a diagram illustrating an example of a background image to be combined.

FIG. 25 is a diagram illustrating a result of conventional image synthesis.

FIG. 26 is a diagram illustrating an example of a frame image in a moving image including an object.

FIG. 27 is a diagram illustrating an extracted object.

FIG. 28 is a diagram illustrating a result of conventional image synthesis.

FIG. 29 is a diagram illustrating a result of conventional image synthesis.

[Description of Signs] 101 CPU 102 Terminal 103, 219 Storage 104 Memory 105 Image memory 106 Combiner 107, 263 Frame memory 108 Monitor 109 Bus 201, 202 TV camera 203, 303 Object extractor 204 Shadow extractor 205 Shadow coding Unit 206 object encoder 207, 216, 234, 237 encoder 208, 209, 217 transmitter 210, 211 communication line 212, 213 receiver 214 editing instruction unit 215, 300 video editing unit 230, 231, 239, 250, 251, 252, 270 Terminals 232, 301 Mask generator 233, 255, 261 Mask memory 235 Differentiator 236 Average calculator 238 Combiner 253 Separator 254, 260 Mask decoder 256 Object decoder 257 Object memory 258, 262 Decoder 259 Latch 264,265,266,267 Deformer 268,302 Shadow synthesizer 269 Image synthesizer

Claims (39)

[Claims] An extracting unit configured to extract at least one object from an image; a main object setting unit configured to set a main object which is a main object among the extracted objects; A slave object setting means for setting a slave object attached to the master object; a main object processing means for performing image processing on the main object; and an image according to the image processing in the main object processing means for the slave object. An image processing apparatus comprising: a slave object processing unit that performs processing. 2. The image processing apparatus according to claim 1, further comprising an instruction unit for instructing an image processing method in said sub-object processing unit, wherein said sub-object processing unit performs image processing in said main object processing unit on said sub-object, and said instruction. 2. The image processing apparatus according to claim 1, wherein the image processing is performed in accordance with an instruction from the means. 3. The main object processing means includes main object synthesizing means for synthesizing the main object with another image, and the sub-object processing means is configured to generate the sub-object in response to an instruction from the instruction means. 3. The image processing apparatus according to claim 2, further comprising a slave object synthesizing unit for synthesizing the image with the image. 4. The image processing apparatus according to claim 3, wherein the instructing unit instructs whether to compose the slave object with the another image. 5. The image processing apparatus according to claim 1, wherein the extracting unit extracts an object from the image based on a background image of the image acquired in advance. 6. An extraction unit for extracting at least one object from an image, a classification unit for classifying the extracted object into a main object and a subsidiary object attached thereto, and a main object code for encoding the main object An image processing apparatus comprising: an encoding unit; and a slave object encoding unit that encodes the slave object. 7. The image according to claim 6, wherein said main object encoding means encodes an image of the main object, and said slave object encoding means encodes information representing a shape of the slave object. Processing equipment. 8. The image processing apparatus according to claim 7, wherein said slave object coding means further codes information indicating the strength of the slave object. 9. The image processing apparatus according to claim 6, further comprising a transfer unit that transfers the encoded main object and the encoded slave object. 10. The method according to claim 10, wherein the main object encoding means is an MP.
11. The image processing apparatus according to claim 6, wherein encoding is performed by an EG-4 encoding method. 11. An object extracting means for extracting at least one main object from an image, a slave object generating means for generating a slave object attached to the main object, and a main object processing for performing image processing on the main object An image processing apparatus comprising: a slave object processing unit that performs image processing on the slave object according to image processing performed by the main object processing unit. 12. The apparatus according to claim 11, further comprising main object encoding means for encoding said main object, wherein said sub object generation means generates a sub object based on said encoded main object. The image processing apparatus according to claim 11. 13. The image processing apparatus according to claim 12, wherein said slave object generating means generates shape information of a slave object based on the encoded shape information of the main object. 14. The main object encoding means includes an MP
14. The image processing apparatus according to claim 12, wherein encoding is performed by an EG-4 encoding method. 15. The image processing apparatus according to claim 1, wherein the slave object represents a shadow of the master object. 16. An object encoding device for extracting and encoding an object from an image, a background encoding device for inputting and encoding a background image, an object encoded by the object encoding device, and the background An image processing system comprising: an image synthesizing device that inputs, decodes, and synthesizes a background image encoded by the encoding device, wherein the object encoding device extracts at least one object from the image. Means, a classifying means for classifying the extracted object into a main object and a subsidiary object attached thereto, a main object coding means for coding the main object, and a slave object coding for coding the slave object And an image processing system. 17. The apparatus according to claim 16, wherein said main object coding means codes an image of the main object, and said slave object coding means codes information representing a shape of the slave object.
The image processing system described in the above. 18. The image processing system according to claim 17, wherein said slave object coding means further codes information indicating the strength of the slave object. 19. The object encoding device further comprises:
An object transmitting unit that transmits an encoded main object and an encoded sub-object; the background encoding device includes a background transmitting unit that transmits an encoded background image; 17. The image processing system according to claim 16, further comprising a receiving unit that receives the encoded main object, sub object, and background image. 20. The image processing system according to claim 16, wherein said image synthesizing device includes: an instructing unit that issues an instruction relating to image combining; and an image combining unit that performs image combining in accordance with the instruction. . 21. The image processing system according to claim 20, wherein said instructing means instructs a shape at the time of synthesizing said master object and slave object. 22. The image processing system according to claim 20, wherein said instructing means instructs an intensity at which said slave object is synthesized. 23. The image processing system according to claim 16, wherein said image synthesizing device has a synthesized image transmitting unit for transmitting a synthesized image. 24. An object encoding device for extracting and encoding an object from an image, a background encoding device for inputting and encoding a background image, an object encoded by the object encoding device and the background An image processing system comprising: an image synthesizing device that inputs, decodes, and synthesizes a background image encoded by an encoding device, wherein the object encoding device extracts at least one object from the image. Means, and a main object encoding means for encoding a main object among the extracted objects, wherein the image synthesizing apparatus, based on the encoded main object, Slave object generating means for generating, the encoded main object and the generated An image combining means for combining said background image object,
An image processing system comprising: 25. The image processing system according to claim 24, wherein said slave object generating means generates shape information of the slave object based on the encoded shape information of the main object. 26. The object encoding device, comprising: object transmitting means for transmitting an encoded main object; the background encoding device having background transmitting means for transmitting an encoded background image; 25. The image processing system according to claim 24, wherein the image synthesizing device has a receiving unit that receives the encoded main object and the background image. 27. The image processing apparatus according to claim 24, wherein the image synthesizing device has an instruction unit for giving an instruction regarding image synthesis, and the image synthesizing unit performs image synthesis in accordance with the instruction. system. 28. The image processing system according to claim 27, wherein said instructing means instructs a shape at the time of synthesizing said master object and slave object. 29. The image processing system according to claim 27, wherein said instructing means instructs an intensity at which said slave object is combined. 30. The image processing system according to claim 24, wherein the image synthesizing device has a synthesized image transmitting unit that transmits a synthesized image. 31. The main object encoding means, comprising:
14. The image processing apparatus according to claim 12, wherein encoding is performed by an EG-4 encoding method. 32. The slave object according to claim 16, wherein the slave object represents a shadow of the master object.
The image processing system according to any one of the above. 33. The slave object according to claim 16, wherein the slave object represents a shadow of the master object.
The image processing system according to any one of the above. 34. An extracting step of extracting at least one object from an image; a main object setting step of setting a main object which is a main object among the extracted objects; A sub-object setting step of setting a sub-object attached to the main object; a main object processing step of performing image processing on the main object; And a slave object processing step of performing processing. 35. An extraction step of extracting at least one object from an image; a classification step of classifying the extracted object into a main object and a subsidiary object attached thereto; and a main object code for encoding the main object. An image processing method comprising: an encoding step; and a slave object encoding step of encoding the slave object. 36. An object extraction step of extracting at least one main object from an image; a sub-object generation step of generating a sub-object attached to the main object; and a main object processing of performing image processing on the main object. An image processing method comprising: a sub-object processing step of performing image processing on the sub-object in accordance with image processing in the main object processing unit. 37. A recording medium in which a program code for image processing is stored, wherein: a code for an extraction step of extracting at least one object from an image; and a main object which is a main object among the extracted objects. Code for a main object setting step to be set; code for a sub object setting step for setting a sub object associated with the main object among the extracted objects; and main object processing for performing image processing on the main object A recording medium comprising: a code of a process; and a code of a sub-object processing step of performing image processing on the sub-object in accordance with image processing in the main object processing means. 38. A recording medium storing a program code for image processing, comprising: a code for an extraction step of extracting at least one object from an image; a main object and a subsidiary object attached to the extracted object; A recording medium comprising: a code of a classification step of classifying the sub object; a code of a main object encoding step of encoding the main object; and a code of a sub object encoding step of encoding the sub object. . 39. A recording medium storing a program code for image processing, comprising: a code for an object extracting step of extracting at least one main object from an image; and a slave object for generating a slave object attached to the main object. A code of a generation step, a code of a main object processing step of performing image processing on the main object, and a sub-object processing step of performing image processing on the sub-object in accordance with image processing in the main object processing means. And a code.