JPH11346363A - Image processing unit and its method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To extract objects of a moving image as pluralities of different objects in the case of extracting the objects from the moving image. SOLUTION: An area extract device 123 divides shape information of an object extracted from a moving image into areas. An area comparator 126 and an area selector 127 select to which area each of divided shape information sets belongs based on comparison with past shape information. An area data extract device 133 divides the object based on the result of selection.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to an image processing apparatus and method, and more particularly, to an image processing apparatus and method for extracting and processing objects in a moving image.

[0002]

2. Description of the Related Art In recent years, an object coding technique for separating a moving image into its background and objects and coding each of them has been developed. The MPEG-4 coding method is known as a representative example, and standardization work is in progress. MPEG
The -4 coding method is a method in which a moving image is separated into a background and a coding target called an object, and each is coded. As a result, the conventional MPEG-1, MPEG-2, h.261,
Unlike encoding in units of frames represented by h.263 or the like, encoding of a background portion with no (or little) motion is not performed twice or more, so that encoding at a low bit rate becomes possible. Furthermore, since it is possible to easily perform an editing operation such as selection of an object, enlargement / reduction, rotation, and the like on the decoding side, decoding that reflects the user's preference has become possible.

[0003] An example of encoding in the MPEG-4 encoding method will be described below. It should be noted that the method of extracting the background and the object in the image is outside the scope of the MPEG-4 standard and is free. For example, "Morphological Segmentation Using Prior Knowledge Information in Sports Programs" (1997 Visual Media Processing Symposium (IMPS97) I-
3,15 1997.10.8 Naemura et al.). This is a method in which information such as a ground without a player as an object is acquired in advance as a background, and an object (player) is extracted from a moving image based on the background information.

FIG. 18 is a block diagram showing a configuration of a conventional image input device. In the figure, first, the TV camera 1001
Is stored in the background memory 1002. Hereinafter, an image of a yacht and a battleship navigating at sea as shown in FIG. 19 will be described as an example. If you create a color histogram from this image and extract the background area,
As shown in FIG. 20, the upper half of the sky and the lower half of the sea can be detected. Since the background image shown in FIG. 20 is a still image, the still image is encoded by the background image encoder 1003.

Subsequently, an image containing the yacht and the battleship, which are objects, is captured as a moving image, and the object is extracted by taking a difference from the background image in the object extractor 1004 or extracting regions having different colors. Can be. The extracted objects are as shown in FIG. This is encoded by the object encoder 1005.

FIG. 22 shows a detailed block configuration of the object encoder 1005. Shape information is input from the terminal 1020 and stored in the shape memory 1022. This shape information is represented by a binary image in which the pixel representing the object is white and the others are black. The boundary extractor 1023 extracts a pixel where black and white are inverted from this image as a boundary, and a motion compensator 1024
To enter. If the frame mode is the I frame, the motion compensator 1024 does not operate. If the frame mode is the P frame or the B frame, the boundary memory 1025 storing the boundary state of the past frame.
And performs motion compensation, and arithmetically encodes the result. After that, the boundary information of the input shape information is stored in the boundary memory 1025.

On the other hand, the image data of the object shown in FIG.
Stored in 1027. The image data in the object memory 1027 and the shape memory 1022 are input to the compensator 1028. Compensator
Reference numeral 1028 compensates for pixels outside the object, that is, pixels whose shape information is black, in macroblock units using pixel values in nearby objects. The result of the compensation is shown in FIG. Here, the compensation is performed by repeating the vertical direction and the horizontal direction. A supplementary pixel is input from a compensator 1028 to a macroblock away from the object.

[0008] The image data thus supplemented is
A difference from the output of the motion compensator 1037 is obtained by a differentiator 1029, DCT-transformed by a DCT transformer 1030, and quantized by a predetermined quantization matrix by a quantizer 1031.
It is Huffman coded at 32. Quantization result is inverse quantizer 10
The pixel value is inversely quantized at 33, returned to the prediction difference value at the inverse DCT transformer 1034 and added to the output of the motion compensator 1037 to decode the pixel value,
The data is stored in the object memory 1036 and used for the next motion compensation. The motion compensator 1037 compares the contents of the object memory 1036 and the object memory 1027 for the P frame and the B frame, performs motion compensation, and calculates a predicted value and a motion vector. The motion vector is encoded and input to the synthesizer 1038. The outputs of the arithmetic encoder 1026, the motion compensator 1037, and the coefficient encoder 1032 are added with a header and the like by a synthesizer 1038.
The data is arranged into MPEG-4 encoded data and output from the terminal 1039.

Returning to FIG. 18, the synthesizer 1006 is a background encoder.
After synthesizing the output of the object encoder 1005 with the output of the object encoder 1005 and adding headers and the like to prepare encoded data of MPEG-4, it is transmitted to the communication line 1008 via the communication interface 1007,
It is stored in the storage device 1009.

[0010]

However, in the above-described object extraction method, since objects other than the background are used as objects, a plurality of objects that should be originally different, for example, both the yacht and the battleship in FIG. Therefore, it was very difficult to handle them as individual objects.

Also, as shown in FIG. 24, when objects are separated from each other in the next frame, etc.
The size of the entire object collected as an object increases. Therefore, on the decoding side, the decoded shape information and the image of the object become large and unnecessarily occupy a large amount of memory, resulting in poor processing efficiency.

Further, since objects are extracted in units of frames, even if the objects are extracted for each object, the relationship between frames is not known, so that encoding using inter-frame correlation is difficult.

In view of the above problems, it is an object of the present invention to provide an image processing apparatus and method for dividing and extracting an object in a moving image and enabling efficient encoding processing. .

[0014]

As one means for achieving the above object, the image processing apparatus of the present invention has the following arrangement.

[0015] That is, input means for inputting moving picture data, object extracting means for extracting at least one object from the moving picture and outputting it together with its shape information, shape dividing means for dividing the shape information, An object dividing unit configured to divide the object based on a result of the division by the shape dividing unit.

Input means for inputting encoded moving image data; separating means for separating the moving image data into background encoded data and object encoded data;
Extracting means for extracting shape information from the encoded data of the separated object; shape dividing means for dividing the shape information; and a dividing result in the shape dividing means.
Object dividing means for dividing the encoded data of the object.

According to the present invention, an operation of dividing an object using shape information can be easily and reliably performed.

[0018]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment according to the present invention will be described below in detail with reference to the drawings.

<First Embodiment> FIG. 1 is a block diagram showing the configuration of a moving image input apparatus according to this embodiment. In the present embodiment, a case will be described in which the aforementioned “morphological segmentation using prior knowledge information in a sports program” is used as a method for extracting objects in an image, and an MPEG-4 encoding method is used as an encoding method. .

In FIG. 1, reference numeral 101 denotes a TV camera. The TV camera 101 may be any type as long as it captures a moving image and outputs it as a digital signal. Ten
A background memory 2 stores a background image captured in advance. Reference numeral 103 denotes a background encoder that encodes a background image. Reference numeral 104 denotes an object extractor that extracts an object by the above-described morphological segmentation. Reference numeral 105 denotes an object divider that divides an object, which is a feature of the present embodiment. An object encoder 106 encodes each of the divided objects. Reference numeral 107 denotes a synthesizer that adds a header or the like to the obtained encoded data and synthesizes the encoded data so as to adapt to the MPEG-4 encoding format. 108 is a communication interface, and 109 is a communication line. 110 is a storage device for storing encoded data.

Hereinafter, a process of capturing a moving image, encoding the image, and outputting the image in the present embodiment will be described in detail. In order to make the description clearer, the case of capturing an image shown in FIG. This will be described as an example.

In the configuration shown in FIG. 1, the user first initializes each section and captures an image without an object, that is, an image of only the background from the TV camera 101. The captured background image as shown in FIG. 20, for example, is input to the background memory 102. The content of the background memory 102 is encoded by a background encoder 103 and prepared as MPEG-4 encoded data by a synthesizer 107, and then transmitted via a communication interface 108 via a communication interface 108.
Or stored in the storage device 110.

Subsequently, the moving image including the object is captured by the TV camera 101 and input to the object extractor 104. The object extractor 104 extracts a region having a different characteristic from the characteristics of the background image as an object by the above-described morphological segmentation, and inputs the shape information and the image data to the object divider 105.

FIG. 2 shows a detailed block configuration of the object divider 105. In the figure, a terminal 120 inputs a binary image as shape information from the object extractor 104. A terminal 121 inputs image data of the object from the object extractor 104. A shape memory 122 stores the input shape information for each frame. Reference numeral 132 denotes an object memory which stores input image data of the object for each frame. Reference numeral 123 denotes an area extractor that extracts a closed area from the shape information, and cuts out and extracts a rectangle for each closed area. 124 is an area memory for storing the extracted area, and 125 is a deformer for scaling and deforming the data. Reference numeral 126 denotes an area comparator that compares the areas by calculating the exclusive OR of the data of each area output from the area extractor 123 and the output of the deformer 125. 127 is the most based on the area comparison result. It is an area selector for selecting a similar area. An area information memory 128 stores information on the extracted area, and a terminal 129 outputs information on the extracted area. Reference numeral 130 denotes a shape memory for storing each area from the area extractor 123 as new shape information according to the area information. Reference numeral 131 denotes a terminal for outputting the shape information of each area stored in the shape memory 130. An area data extractor 133 reads out image data from the object memory 132, determines which closed area belongs to which area according to the area information in the area information memory 128 and the shape information in the shape memory 130, and determines each of them. The image data is stored in the object memory 134 for each closed area.

In such a configuration, for the first frame including the object, the frame shape information (FIG. 3) and the object image data (FIG. 21) are input from the terminals 120 and 121, respectively. Each is stored in the object memory 132. Region extractor 123
Reads a binary image from the shape memory 122 and extracts a closed region included in the shape information. For example, in FIG.
It can be seen that there are two closed regions. One is a yacht,
The other is a battleship. In the area extractor 123, first, the closed area (yacht) on the left side is cut out as a circumscribed rectangle as shown in FIG. 4, and stored in the area memory 124 and the shape memory 130. When the processing frame is the first frame, the area comparator 126 and the area selector 127 do not operate, and the area information memory 12
Numeral 8 stores information such as the position and size of the extracted closed region as new shape information. Subsequently, the closed area (battleship) on the right side is similarly cut out as a circumscribed rectangle as shown in FIG. 5, and stored in the area memory 124, the shape memory 130, and the area information memory 128.

Thereafter, the image data is read from the object memory 132 by the area data extractor 133, and it is determined which closed area belongs to which area according to the area information in the area information memory 128 and the shape information in the shape memory 130. Then, the image data is stored in the object memory 134 for each closed area. In the present embodiment, the image data stored in the object memory 134 becomes a new divided object.

The above is the processing of dividing one frame by the object divider 105. Hereinafter, input to the object divider 105 is performed in the order of encoding frames. Here, the processing of the first and subsequent frames in the object divider 105 will be described with the image of the subsequent frame as FIG.

According to FIG. 24, it can be seen that the yacht is moving while turning the rudder to the left of the screen, and the battleship is moving toward this. At this time, frame shape information and object image data are input from the terminals 120 and 121 of the object divider 105, and stored in the shape memory 122 and the object memory 132. As in the case of the first frame, the area extractor 123 reads the binary image from the shape memory 122 and extracts a closed area included in the shape information. The extracted closed area of the yacht is processed first, wherein the extracted closed area is shown in FIG. The deformer 125 first reads a plurality of areas stored in the area memory 124, and performs scaling so as to have the same size as the shape information (corresponding to FIG. 6) input to the area comparator 126. Will be

The area comparator 126 performs a comparison by obtaining a difference between the scaled result and the input area. Here, FIG. 7 shows the result of transforming the shape information of the yacht of the previous frame and performing an exclusive OR operation with the newly input closed area image, and FIG. 8 shows the shape information of the battleship of the previous frame. This shows the result of exclusive OR operation after transformation. That is, FIGS. 7 and 8 show that the pixels shown in black are different pixels between both closed regions. Fig. 7
It is clear from FIG. 8 that there are more black pixels in FIG. That is, in the area comparator 126, the number of black pixels is counted, and the closed area having the smaller count value is
Indicates that the similarity with the input area is higher.

The area comparator 126 inputs the count result to the area selector 127. The area selector 127 detects that the closed area input from the terminal 120 represents the same object as the closed area of the yacht detected in the previous frame, based on the count result from the area comparator 126, and newly detects the closed area. The area information such as the fact that the closed area is the same object as the yacht and its position, size, etc. is written to the area information memory 128.

Subsequently, an exclusive OR operation with a plurality of closed areas is similarly performed on the right battleship area shown in FIG. 24, and based on the comparison result, the area information memory 128 stores the newly detected closed area. Is the same object as the battleship, and the area information such as its position and size is stored in the area information memory 128.
To be stored. The contents of the area information memory 128 are assigned a code by the object encoder 106 shown in FIG. 1 as object information at the time of encoding, and are recorded in a header or the like by the synthesizer 107.

The closed area detected as described above is
Each is stored in the shape memory 130. The area data extractor 133 cuts out each object data from the object memory 132 based on the contents of the area information memory 128 and the contents of the shape memory 130, and stores them separately in the object memory 134. Here, FIG. 9 shows a cutout result of a yacht, and FIG. 10 shows a cutout result of a battleship. The contents of the shape memory 130 and the object memory 134 are read from the terminals 131 and 135 at the time of encoding.

The above is the description of the dividing process in the object divider 105.

Returning to FIG. 1, the object encoder 106
Area information memory 1 output from object divider 105
The content of 28, the content of the shape memory 130, and the content of the object memory 134 are encoded according to the MPEG-4 encoding format, and output to the synthesizer 107. Encoding of a shape and encoding of image data of an object can be performed by the configuration shown in the block diagram of FIG. 22 described above. A combiner 107 combines the outputs of the background encoder 103 and the object encoder 106, prepares MPEG-4 encoded data by adding a header and the like, and sends the data to the communication line 109 via the communication interface 108. Or store it in the storage device 110.

As described above, according to this embodiment,
By dividing the object from the shape information by a series of selection operations and measuring the similarity for each frame, a continuous object can be easily determined. As a result, the encoding efficiency can be improved, and editing can be performed for each object, or the object can be handled as different encoded data.

Further, since the size of one object can be minimized on the encoding side and the decoding side, the processing can be reduced by reducing the area to be compensated. It is also possible to reduce the memory capacity for decoding the object.

Although the present embodiment has been described with reference to the example in which the number of unmatched pixels of the shape information is referred to in order to measure the similarity of the regions, the present invention is not necessarily limited to this example. Alternatively, the shape itself may be used. For example, as a method of comparing the shapes themselves, it is only necessary to use characteristics such as how to bend or branch by contour tracing.

The encoding method is not limited to the MPEG-4 method. For example, another coding method such as chain coding may be used as shape coding, or coding without using inter-frame correlation may be performed as coding of image data of an object.

The method of outputting the obtained encoded data is not limited to the present embodiment. Further, after the object is divided, the image can be directly input to a device that edits an image without encoding, for example, to change the arrangement of the object, or to apply an image editing function such as rotation / enlargement / reduction to the object itself. .

<Second Embodiment> Hereinafter, a second embodiment according to the present invention will be described.
An embodiment will be described. In the second embodiment,
The image data to be processed is input as encoded data.

FIG. 11 is a block diagram showing a configuration of an image processing apparatus according to the second embodiment. Here, the first
The same components as those in FIG. 1 shown in the embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted. In addition,
In the second embodiment, the input data is coded data according to the MPEG-4 coding method. However, the encoding method in the second embodiment is not limited to this.

In FIG. 11, reference numeral 201 denotes a storage device, which is an MPE.
Image data encoded by the G-4 encoding method is stored. Reference numeral 202 denotes a separator, which separates the input MPEG-4 encoded data into encoded data relating to a background image and encoded data relating to an object. Reference numeral 203 denotes a background code memory, which stores encoded data relating to the separated background image. Reference numeral 204 denotes an object division unit which is a feature of the second embodiment, and details thereof will be described later.

In the configuration shown in FIG.
, The data encoded by the MPEG-4 encoding method is read and input to the separator 202. Here, the configuration of the input encoded data is shown in FIG. According to the figure, the encoded data first has a header representing the information of the image and the entire code at the beginning. Then, encoded data on the background image follows, which includes a background header including information such as the size of the background image, and data of the background image encoded in the I frame mode. After that, encoded data on the objects continues for the number of included objects. The encoded data of each object has an object header including information such as the size of the object, followed by the encoded data of the object encoded by the MPEG-4 encoding method. As encoded data of each object, encoded data of each frame is linked,
The encoded data of each frame is composed of a frame header including a frame mode and a synchronization code, and a main body of the encoded data. Note that the encoded data includes encoded data of shape information representing the shape of the object and encoded data of image data.

In the separator 202, the header of the encoded data shown in FIG. 12 is analyzed, each header is sent to the synthesizer 107, and the encoded data relating to the background image is sent to the background code memory 203.
The encoded data of each object is stored in the object division unit 20.
Separate into 4. The background code memory 203 stores coded data of the background image.

FIG. 13 shows a detailed block configuration of the object division unit 204. In the same drawing, the same reference numerals are given to those which fulfill the same functions as those in FIG. 2 described above, and detailed description will be omitted.

In FIG. 13, reference numeral 210 denotes a terminal for inputting encoded data relating to an object from the separator 202. twenty one
Reference numeral 1 denotes a separator for separating the input encoded data into encoded data of shape information and encoded data of image data.
Reference numeral 212 denotes a shape information decoder that decodes shape information of the MPEG-4 encoding method. A shape memory 213 stores a binary image as decoded shape information. Reference numeral 214 denotes a histogram processor, which generates a histogram for each area output from the area extractor 123. A histogram memory 215 stores the generated histogram for each area. Reference numeral 216 denotes a histogram comparator for calculating the similarity between the regions based on the data of each region output from the region extractor 123 and the output of the histogram processor 215, and 217 determines the most similar region based on the histogram comparison result. An area selector to be selected. A terminal 223 outputs information on the extracted area.

Reference numeral 220 denotes a shape information encoder for encoding each area as new shape information from the area extractor 123 in accordance with the area information in the area information memory 128.
Reference numeral 224 denotes a terminal that outputs encoded data of the shape information of each area encoded by the shape information encoder 220. An object code memory 219 stores encoded data of each object separated by the separator 211. 221 is an object code memory 2 according to the area information of the area information memory 128.
This is a code divider that divides 19 encoded data into extracted encoded data for each object. Reference numeral 222 denotes a memory that stores the divided encoded data, and reference numeral 225 denotes a terminal that outputs encoded data of an image of each object stored in the object code memory 222.

In the object dividing unit 204 having such a configuration, first, the encoded data of the first frame including the object is input from the terminal 210. The first
The encoded data of the frame is subjected to intra-frame encoding, and no motion compensation is performed. The input encoded data is separated into encoded data of shape information and encoded data of image data by a separator 211, the encoded data of shape information is sent to a shape information decoder 212, and the image data is sent to an object code memory 219. Is entered. The shape information decoder 212 decodes the encoded data, reproduces a binary image, and
213 is stored. The region extractor 123 extracts a closed region as in the first embodiment. The extracted shape information of each area is input to the histogram processor 214 for each area.

The histogram processor 214 generates a histogram by first counting pixel values in each main scanning direction with black (1) inside the closed area and white (0) outside the closed area. Subsequently, a histogram is generated by counting the pixel values in the sub-scanning direction. Here, considering the case where the histogram processing is performed on the image shown in FIG. 19, the result of the histogram processing on the yacht which is the closed area on the left side is as shown in the graph of FIG. FIG. 14A shows a histogram in the main scanning direction, and FIG. 14B shows a histogram in the sub scanning direction.
These histogram data are stored in the histogram memory 215 for each region. Similarly, by performing the histogram processing on the battleship that is the closed area on the right side, the histogram in the main scanning direction shown in FIG.
The histogram in the sub-scanning direction shown in (b) is obtained. These histogram data are stored in the histogram memory 215 for each area. Note that the histogram comparator 216 does not operate in the first frame. The area selector 217 sets each of the two extracted areas as a new object. The area information memory 128 stores information such as the position and size of new shape information of the extracted object.

After that, the shape information is read from the shape memory 213, divided into the respective objects by the area extractor 123, and input to the shape information encoder 220. The shape information encoder 220 converts the shape information of each object into an MPEG-4
Encoding is performed according to the shape information encoding of the encoding method.
Further, encoded data of image data is read from the object code memory 219, and the code divider 221 belongs to any closed area according to the information of the area information memory 128 and the shape information of the new object indicated by the area extractor 123. Is determined, and the encoded data is divided for each closed region.

Here, the input coded data is coded in macroblock units. FIG. 16 illustrates an example in which an image obtained by extracting an object from the image of FIG. 19 is divided into macroblocks. According to the drawing, it can be seen that the extracted objects include a yacht area 250 and a battleship area 251 which are objects to be divided. FIG.
Indicates one macroblock. In the second embodiment, encoded data is read out in units of macroblocks.

If the macro block is not included in any of the objects to be divided shown by the regions 250 and 251, the encoded data is discarded. On the other hand, when the macro block is completely included in the object, the encoded data is used as it is. In addition, for a macroblock in a portion where regions overlap, a not_coded code indicating whether or not the macroblock is coded is added in macroblock units. That is, in the region 250, the region 251
The 56 macroblocks that overlap with are outside the object (yacht pattern), so 56 not_co
A ded code is respectively added.

Here, regarding the macro block (blocks 252 to 254 in FIG. 16) which is in contact with the other area, there is a possibility of replacing the code. If the macroblock immediately before or immediately above the macroblock is a macroblock in another area, code replacement occurs. That is, in these macroblocks, prediction is performed from surrounding macroblocks, and decoding may not be possible due to a change in the contents of the referenced macroblock. For this reason, for a macroblock that has been encoded with reference to the surrounding macroblock, the encoded data of the macroblock is temporarily decoded, and the surrounding macroblock is compared with a macroblock that is a block in another area. Encoding is performed again with the value of the area information memory 128 inserted. In this way, the encoded data is divided for each object and stored in the object code memory 222.

The above is the process of dividing one frame by the object dividing unit 204. Hereinafter, input to the object dividing unit 204 is performed in the order of the subsequent frames. here,
The image of the subsequent frame is shown in FIG.
The processing of the first and subsequent frames in 204 will be described.
Similarly to the first frame, the input coded data is separated into coded data of shape information and coded data of image data by a separator 211, and a shape information decoder 212 decodes the coded data to form a shape memory. 213 is stored. Region extractor 123
Performs extraction of a closed region, shape information of each region is input to the histogram processor 214, and a histogram is generated for each region. In FIG. 24, a graph showing the result of performing the histogram processing on the yacht that is the closed area on the left is
It is 17. FIG. 17A shows a histogram in the main scanning direction, and FIG. 17B shows a histogram in the sub scanning direction. These histogram data are compared by the histogram comparator 216 with the histogram for each area stored in the histogram memory 215, and the similarity is calculated. Here, the method of comparison is not limited, but for example, the correlation of the position of the local maximum value, the distribution of frequency, or the like may be used.

In the area selector 217, the object having the highest similarity is regarded as the same object, and information such as the position and size is added to the information of the corresponding object in the previous frame and stored in the area information memory 128. . Similarly, the same processing is performed for the battleship that is the closed area on the right side in FIG. After that, these histogram data are stored in the histogram memory 215 for each area.

After that, the shape information is read from the shape memory 213, divided into respective objects by the area extractor 123, and encoded by the shape information encoder 220.

Further, encoded data of image data is read from the object code memory 219, and is stored in the area information memory.
According to the information of 128 and the shape information of the new object indicated by the region extractor 123, it is determined which closed region belongs to, and the code divider 221 divides the coded data for each closed region.

Note that motion compensation may be performed for the second and subsequent frames. For macroblocks without motion compensation, code division is performed in macroblock units, as described above. On the other hand, a macroblock for which motion compensation has been performed may not be able to be decoded if the content of the macroblock to be referenced changes when the motion vector refers to an area outside the area. Therefore, only when the motion vector is decoded, and only when a part overlapping with another area outside or within the area is referred to, the macroblock is decoded once and is intra-frame coded or the immediately preceding macroblock is temporarily decoded. After decoding, motion compensation is performed using this as a reference macroblock, and encoding is performed. By the above processing, the code is divided in units of macroblocks, divided for each object, and stored in the object code memory 222.

Finally, the synthesizer 107 shown in FIG. 11 combines the outputs of the background code memory 203 and the object dividing unit 204, adds a header and the like, and prepares MPEG-4 encoded data. Via the communication line 109 or stored in the storage device 110.

As described above, according to the second embodiment, it is possible to divide an object without decoding all encoded data. For this reason, the object can be divided at high speed, and it is not necessary to store the decoded image, so that the memory can be reduced.

Further, since the segmentation of the object is performed by using the feature amount extracted from the shape information, the past shape information is not represented by an image but by a numerical value. Therefore, the amount of memory for storing the past shape information can be reduced. It is now possible. Therefore, it is understood that the past shape information need not be image information.

Although the storage device 110 and the storage device 201 are described independently in the second embodiment, for example, they may be configured by the same device and written in different memory areas.

In the second embodiment, an example in which the similarity is obtained from the histogram has been described. However, the present invention is not limited to this example, and the same means as in the first embodiment may be used. Using OCR technology,
It is also possible to select a region having the highest similarity to the stored shape.

The configuration and the like of the memory in the second embodiment are not limited to this. In addition, some or all of the functions may be realized by software using a CPU or the like.

The encoding method is not limited to MPEG-4. For example, JBIG which is a binary image encoding method
Encoding may be performed by an encoding method or an MMR encoding method, or image data of an object may be compressed in frame units,
Of course, encoding with MPEG-1, 2, h.261 or the like is also acceptable.

<Other Embodiments> Even if the present invention is applied to a system including a plurality of devices (for example, a host computer, an interface device, a reader, a printer, etc.), an apparatus including one device (for example, For example, the present invention may be applied to a copying machine, a facsimile machine, and the like.

Another object of the present invention is to provide a storage medium storing a program code of software for realizing the functions of the above-described embodiments to a system or an apparatus, and to provide a computer (or CPU) of the system or apparatus.
And MPU) read and execute the program code stored in the storage medium.

In this case, the program code itself read from the storage medium implements the functions of the above-described embodiment, and the storage medium storing the program code constitutes the present invention.

As a storage medium for supplying the program code, for example, a floppy disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD
-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

When the computer executes the readout program code, not only the functions of the above-described embodiment are realized, but also the OS (Operating System) running on the computer based on the instruction of the program code. ) May perform some or all of the actual processing, and the processing may realize the functions of the above-described embodiments.

Further, after the program code read from the storage medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, based on the instruction of the program code, It goes without saying that the CPU included in the function expansion board or the function expansion unit performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments.

[0072]

As described above, according to the present invention,
A plurality of objects in a moving image can be divided and extracted. Furthermore, since the phases between frames of the divided object can be grasped, efficient coding processing can be performed.

As a result, an editing operation or the like for each object in an image can be efficiently performed, and by performing further encoding, efficient data transfer and accumulation can be performed.

[0074]

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a moving image input device according to an embodiment of the present invention.

FIG. 2 is a block diagram illustrating a detailed configuration of an object divider according to the embodiment.

FIG. 3 is a diagram illustrating an example of an object area according to the present embodiment.

FIG. 4 is a diagram illustrating an example of an object area according to the present embodiment.

FIG. 5 is a diagram illustrating an example of an object area according to the present embodiment.

FIG. 6 is a diagram illustrating an example of an object area according to the present embodiment.

FIG. 7 is a diagram illustrating a state of a difference in shape information according to the present embodiment.

FIG. 8 is a diagram illustrating a state of a difference in shape information according to the present embodiment.

FIG. 9 is a diagram for describing a state of dividing an object in the present embodiment.

FIG. 10 is a diagram for describing a state of dividing an object in the present embodiment.

FIG. 11 is a block diagram illustrating a configuration of an image processing device according to a second embodiment of the present invention.

FIG. 12 is a diagram for describing a configuration of encoded data according to the second embodiment.

FIG. 13 is a block diagram illustrating a detailed configuration of an object division unit according to the second embodiment.

FIG. 14 is a diagram for explaining a state of a histogram in the second embodiment.

FIG. 15 is a diagram for explaining a state of a histogram in the second embodiment.

FIG. 16 is a diagram for explaining a state of division of a macroblock in the second embodiment.

FIG. 17 is a diagram for explaining a state of a histogram according to the second embodiment.

FIG. 18 is a block diagram illustrating a configuration of a conventional image input device.

FIG. 19 is a diagram illustrating the contents of one frame of a moving image as an example.

FIG. 20 is a diagram illustrating an example of a background image.

FIG. 21 is a diagram illustrating an example of an extracted object.

[Fig. 22] Fig. 22 is a diagram for describing the MPEG-4 encoding method.

FIG. 23 is a diagram illustrating a state of supplementing an object in a conventional example.

FIG. 24 is a diagram illustrating the contents of one frame of a moving image as an example.

[Explanation of symbols]

 101, 1001 TV camera 102, 1002 Background memory 103, 1003 Background encoder 104, 1004 Object extractor 105 Object divider 106, 1005 Object encoder 107, 1006 Compositor 108, 1007 Communication interface 109, 1008 Communication line 110 , 201, 1009 Storage devices 120, 121, 129, 131, 135, 210, 223, 224, 225 Terminals 122, 130, 213 Shape memory 123 Area extractor 133 Area data extractor 124 Area memory 125 Transformer 126 Area comparator 127, 217 region selector 128 region information memory 132, 134 object memory 202, 211 separator 203 background code memory 204 object division unit 212 shape information decoder 214 histogram processor 215 histogram memory 216 histogram comparator 219, 222 object code memory 220 Shape information encoder 221 Code divider 250,251 Area 252,253,254 Macro block

Claims (22)

[Claims] An input unit for inputting moving image data; extracting at least one object from the moving image;
An image, comprising: an object extracting unit that outputs the shape information together with the shape information; a shape dividing unit that divides the shape information; and an object dividing unit that divides the object based on a division result in the shape dividing unit. Processing equipment. 2. The image processing apparatus according to claim 1, wherein the shape dividing unit divides the shape information for each closed area. 3. The method according to claim 1, wherein the shape dividing means includes: comparing means for comparing the shape information with past shape information; and selecting means for selecting an area to which the shape information belongs based on the comparison result. The image processing device according to claim 2. 4. The image processing apparatus according to claim 3, wherein the comparing unit compares similarities based on a difference between the shape information and past shape information. 5. The image processing apparatus according to claim 3, further comprising a shape information updating unit that updates the past shape information with the shape information output by the object extracting unit. 6. The image processing apparatus according to claim 1, further comprising encoding means for encoding each of the objects divided by said object dividing means. 7. An image processing apparatus according to claim 6, wherein said encoding means encodes each of the shape information divided by said shape dividing means together with a corresponding object. 8. The image processing apparatus according to claim 6, wherein said encoding means performs encoding according to an MPEG-4 encoding method. 9. The image processing apparatus according to claim 1, further comprising a background holding unit that holds a background image of the moving image data, wherein the object extracting unit extracts an object from the moving image data based on the background image held by the background holding unit. The image processing apparatus according to claim 1, wherein the image is extracted. 10. A background encoding unit that encodes the background image, an object encoding unit that encodes each of the objects divided by the object division unit, and a background image that is encoded by the background encoding unit. The image processing apparatus according to claim 9, further comprising: a combining unit configured to combine the background image and the object encoded by the object encoding unit. 11. The image processing apparatus according to claim 10, wherein said background encoding unit and said object encoding unit perform encoding according to an MPEG-4 encoding method. 12. Input means for inputting encoded moving image data, separating means for separating the moving image data into background encoded data and object encoded data, and code of the separated object Extraction means for extracting shape information from the encoded data; shape division means for dividing the shape information; and object division means for dividing the encoded data of the object based on the division result in the shape division means. An image processing apparatus characterized by the above-mentioned. 13. The image processing apparatus according to claim 12, wherein the shape dividing unit divides the shape information for each closed area. 14. The apparatus according to claim 1, wherein the shape dividing means includes: comparing means for comparing the shape information with past shape information; and selecting means for selecting an area to which the shape information belongs based on the comparison result. Claim 13
The image processing apparatus according to any one of the preceding claims. 15. The image processing apparatus according to claim 14, wherein the comparing unit compares the histogram of the shape information with a histogram of past shape information. 16. The image processing apparatus according to claim 14, further comprising shape information updating means for updating said past shape information with the shape information extracted by said extracting means. 17. The apparatus according to claim 12, further comprising combining means for combining the encoded data of the background separated by said separating means and the encoded data of the object divided by said object dividing means. Image processing device. 18. An image processing apparatus according to claim 12, wherein said input means inputs moving image data encoded by an MPEG-4 encoding method. 19. An inputting step of inputting moving image data, extracting at least one object from the moving image,
An image extracting step of outputting an object together with the shape information; a shape dividing step of dividing the shape information; and an object dividing step of dividing the object based on a division result in the shape dividing step. Processing method. 20. An input step of inputting encoded moving image data, a separating step of separating the moving image data into background encoded data and object encoded data, and a code of the separated object. An extraction step of extracting shape information from encoded data; a shape division step of dividing the shape information; and an object division step of dividing encoded data of the object based on a division result in the shape division step. An image processing method characterized by the following. 21. A recording medium on which a program code for image processing is recorded, wherein a code of an input step of inputting moving image data, and at least one object is extracted from the moving image,
A code of an object extraction step to be output together with the shape information, a code of a shape division step of dividing the shape information, and a code of an object division step of dividing the object based on a division result in the shape division step.
A recording medium comprising: 22. A recording medium on which a program code for image processing is recorded, wherein: a code of an input step of inputting encoded moving image data; and said moving image data are encoded with background encoded data and an object code. Code for a separation step for separating the data into coded data; code for an extraction step for extracting shape information from the coded data of the separated object; code for a shape division step for dividing the shape information; A code for an object division step of dividing the encoded data of the object based on the division result in the above.