JP2000082145A - Object extraction device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To precisely extract/track a target object from a moving image without being affected by an extra surrounding movement other than the target object. SOLUTION: Rectangles R(i-1), R(i) and R(i+1) surrounding an object are set to each of temporally continuous frames f(i-1), f(i) and f(i+1). By an interframe difference between a current frame f(i) and a first reference frame f(f-1) and an interframe difference between the current frame f(i) and a second reference frame f(i+1), difference pictures fd(i-1, i) and fd(i, i+1) can be obtained. Regarding each of a polygon Rd(i-1, i)=R(i-1) or R(i) and a polygon Rd(i, i+1)=R(i) or R(i+1), a background area is decided and remaining areas are selected as an object are candidate. By taking intersection of these object area candidates, an object area O(i) of the current frame f(i) can be extracted.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an object extracting apparatus for an image, and more particularly, to an object extracting apparatus which detects a position of a target object from an input moving image to track / extract a moving object.

[0002]

2. Description of the Related Art Conventionally, algorithms for tracking / extracting an object in a moving image have been considered. This is a technique for extracting only a certain object from an image in which various objects and a background are mixed. This technique is useful for processing and editing moving images, and for example, can combine a person extracted from a moving image with another background.

As a method used for object extraction, spatio-temporal image area division (Echigo and Iisaku, “spatio-temporal image area division for video mosaic”, 1997 IEICE Information and System Society Conference, D. 12-81,
p. 273, September 1997) is known.

In the area dividing method using the spatio-temporal image area dividing, a small area is divided by a color texture in one frame of a moving image, and the areas are merged using a motion relation between frames. When dividing an image in a frame, it is necessary to give an initial division, which has a problem that the division result is largely affected. Therefore, by utilizing this in reverse, in this region segmentation method using spatiotemporal image segmentation, the initial segmentation is changed in another frame, resulting in a different segmentation result, and contradiction due to the movement between frames. The method of merging the divisions is performed.

However, if this technique is directly applied to tracking and extraction of an object in a moving image, the motion vector is affected by extra motion other than the intended moving object,
In many cases, the reliability is not sufficient, and the problem is that erroneous merging is performed.

Further, Japanese Patent Application Laid-Open No. Hei 8-241414 discloses a moving object detecting / tracking device using a plurality of moving object detecting devices in combination. This conventional moving object detection and tracking device is used, for example, in a monitoring system using a monitoring camera, and detects and tracks a moving object from an input moving image. In this moving object detection / tracking apparatus, an input moving image is input to an image dividing unit, an inter-frame difference type moving object detection unit, a background difference type moving object detection unit, and a moving object tracking unit. In the image dividing unit, the input moving image is divided into blocks of a predetermined size. The division result is sent to the inter-frame difference type moving object detection unit and the background difference type moving object detection unit. The inter-frame difference type moving object detection unit detects the moving object in the input moving image using the inter-frame difference for each division result. In this case, the frame interval at the time of obtaining the inter-frame difference is set based on the detection result of the background difference type moving object detection unit so that the moving object can be detected without being affected by the moving speed of the moving object. Is done. The background difference type moving object detection unit detects the moving object by taking the difference between the background image created for each division result and the moving object using the moving image input so far. The integration processing unit integrates the detection results of the inter-frame difference type moving object detection unit and the background difference type moving object detection unit, and extracts motion information of the moving object. After extracting the object in each frame, the moving object tracking unit associates the corresponding moving objects between the frames.

In this configuration, since the moving object is detected using not only the inter-frame difference but also the background difference, the detection accuracy is higher than when only the inter-frame difference is used. However, since the moving object is detected from the entire input moving image by using the inter-frame difference and the background difference, the detection results of the inter-frame difference and the background difference are different from each other. However, there is a problem that a target moving object cannot be extracted and tracked well in an image having a complicated movement in the background.

As another object extraction technique, there is also known a method of first generating a background image using a plurality of frames, and extracting a region where the pixel value difference between the background image and the input image is large as an object. I have.

An example of an existing technique for object extraction using this background image is disclosed, for example, in Japanese Patent Application Laid-Open No. 8-55222, entitled "Moving Object Detection Device, Background Extraction Device, and Unattended Object Detection Device".

[0010] The image signal of the currently processed frame is input to a frame memory for storing an image of one frame, first motion detecting means, second motion detecting means, and a switch. The image signal of the previous frame is read out from the frame memory and input to the first motion detecting means. On the other hand, the background image signal generated up to that point is read from the frame memory prepared for holding the background image, and is input to the second motion detecting means and the switch. The first motion detecting means and the second motion detecting means extract an object area and an object area using a difference value between two input image signals, and send them to the logic operation circuit. In the logical operation circuit, the logical product of the two input images is calculated, and the logical product is output as a final object region. The object area is also sent to the switch. In the switch, a background pixel signal is selected for a pixel belonging to the object region, and conversely, for a pixel not belonging to the object region, an image signal of the current processing frame is selected, and the image signal is stored in the frame memory as an overwrite signal. Sent to overwrite the pixel values in the frame memory.

In this method, as shown in Japanese Patent Application Laid-Open No. 8-55222, the background image gradually becomes correct as the processing proceeds, and the object is eventually correctly extracted. However, at the beginning of the moving image sequence, the object is mixed with the background image, and the object extraction accuracy is poor. Further, when the movement of the object is small, the image of the object remains in the background image forever, and the extraction accuracy does not increase.

[0012]

As described above, the conventional object extraction / tracking method has a mechanism for detecting a moving object from an entire input moving image from the image. There has been a problem that the target moving object cannot be accurately extracted and tracked because it is affected by extra motion other than the moving object.

In the object extraction method using a background image,
If the extraction accuracy is low at the beginning of the moving image sequence and the movement of the object is small, there is a problem that the extraction accuracy is not improved because the background image is not completed forever.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a moving image object extracting apparatus capable of accurately extracting / tracking an object without being affected by extra movement around the object other than the object. I do.

Further, the present invention makes it possible to determine a background image with high accuracy, and to obtain high extraction accuracy regardless of the size of the motion of an object and at the beginning of a moving image sequence as well as at the end. It is an object of the present invention to provide a device for extracting an object.

[0016]

According to the present invention, the current frame and the first frame are extracted based on a difference between a current frame from which an object is to be extracted and a first reference frame temporally different from the current frame. A first background region common to the reference frame of the current frame, and based on a difference between the current frame and a second reference frame temporally different from the current frame, the current frame and the second reference frame are determined. Background area determining means for determining a second background area common to the frame; and an area not belonging to either the first background area or the second background area in the in-graphic image of the current frame. And an object extraction device comprising: means for extracting an object area; and object stillness detection means for detecting a stationary object area.

In this object extraction device, two reference frames are prepared for each current frame from which an object is to be extracted, and the current frame and the first frame are obtained by a first difference image between the current frame and the first reference frame. A first common background area commonly used for the first and second reference frames is determined, and a second difference image between the current frame and the second reference frame is used to determine a current frame and a second reference frame. And a second common background area commonly used is determined. First
Since both the second difference image and the second difference image commonly include the object region on the current frame, in the region that does not belong to either the first common background region or the second common background region,
By detecting a region included in the in-figure image of the current frame, an object region on the current frame is extracted. When this object region corresponds to a stationary object, a stationary object region is detected when there is no difference between the previous object region and the current object region.

In this way, by determining an area that does not belong to any of the plurality of common background areas determined based on temporally different reference frames as an object to be extracted and tracking the object, The target object can be accurately extracted and tracked without being affected by the extra surrounding movement other than the object.

The background movement of each of the reference frames or the current frame is corrected so that the background movement between the first and second reference frames and the current frame is relatively zero. It is preferable to further include a background correction unit. By providing this background correction means in either the input stage of the graphic setting device or the input stage of the background region determining device, the background image gradually changes between successive frames, for example, when the camera is panned. Even in such a case, the background video can be pseudo-constant between the frames. Therefore, the current frame and the first
Alternatively, by taking the difference from the second reference frame, the background can be canceled between those frames, and the common background area detection processing and the object area extraction processing that are not affected by the background change can be performed. The background correction means can be realized by a motion compensation process.

The background region determining means may include, in a difference image between the current frame and the first or second reference frame, an image in a figure of the current frame or the first image.
Or means for detecting a difference value of each pixel in the vicinity of an outline of a region belonging to the image in the figure of the second reference frame;
Means for determining a difference value to be determined as the common background area using a difference value of each pixel in the vicinity of the contour line, and determining the determined difference value for a background / object area determination. Preferably, the common background area is determined from the difference image by using the threshold value as a threshold value.
By focusing on the difference value of each pixel in the vicinity of the contour as described above, it is possible to easily determine the threshold value without checking the entire difference image.

The figure setting means includes means for dividing the image in the figure of the reference frame into a plurality of blocks;
Means for searching for an area on the input frame that minimizes an error with respect to the input frame for each of the plurality of blocks; and means for setting a figure surrounding the searched areas to the input frame. Is preferred. This makes it possible to set a new graphic that is optimal for the target input frame, regardless of the initially set shape and size of the graphic.

Also, the present invention provides a predicting means for predicting the position or shape of an object on a current frame from which an object is to be extracted from a frame from which an object area has already been extracted, Means for selecting the first and second reference frames to be used by the background area determining means based on the position or shape of the object.

As described above, by selecting an appropriate frame as a reference frame to be used, it is possible to always obtain a good extraction result.

Here, O _i , O _j , and O _curr are respectively referred to as reference frames f _i , f _j and a current frame f _curr to be extracted.
, The optimal reference frames f _i and f _j for correctly extracting the shape of the object are frames satisfying (O _i ∩O _j ) ⊆O _curr , that is, the intersection of O _i and O _j frame f _i but as belonging to the O _curr, is f _j.

Further, the present invention is characterized in that a plurality of object extracting means for extracting objects by different methods are provided, and the object is extracted while selectively switching these object extracting means. In this case, first object extraction means for extracting an object using a difference between the current frame and at least two reference frames which are temporally shifted from the current frame, and object extraction using inter-frame prediction. It is desirable to use in combination with a second object extracting means for extracting an object by estimating an object area of the current frame from the frame on which the image processing has been performed. Thereby, even when the object is partially stationary and a difference from the reference frame cannot be detected, the difference can be compensated for by the object extracting means using inter-frame prediction.

In the case where a plurality of object extracting means are provided, the apparatus further comprises means for extracting, from the current frame from which the object is to be extracted, an image feature of at least a part of the area. It is preferable that the plurality of object extracting means be switched based on the characteristic amount.

For example, if it is known in advance whether there is a background movement, it is better to use that property. If there is background motion, background motion compensation is performed, but it cannot always be completely compensated. In some cases, frames that have complicated movements can hardly compensate. Since such a frame can be selected in advance by the amount of compensation error of the background motion compensation,
It is possible to devise, for example, not to use as a reference frame candidate. However, if there is no background movement, this process is unnecessary. If another object is moving, wrong frame motion compensation may be performed, or the frame may be deviated from the reference frame candidate, so that the optimal frame for the reference frame selection condition may not be selected, and the extraction accuracy may be reduced. is there. Also, various properties may be mixed in one image. The movement and texture of the object are also partially different, and the same tracking / extracting method, apparatus, and parameters may not be able to extract well. Therefore, the user designates a part having a special property in the image, or automatically detects a difference in the image as a feature amount, and based on the feature amount, for example, a partial block unit in a frame or the like. It is better to switch the tracking / extraction method to extract or change the parameters.

In this way, if a plurality of object extracting means are switched based on the feature amount of an image, it becomes possible to extract the shapes of objects in various images with high accuracy.

A first object extracting means using a difference between the current frame and at least two reference frames temporally shifted from the current frame, and a second object extracting means using inter-frame prediction. When the prediction error by the second object extraction unit is within a predetermined range, the extraction result by the second object extraction unit is used as an object region, and the prediction error is Is exceeded, the first and second object extraction means are selectively selected on a block-by-block basis based on the prediction error amount so that the result of extraction by the first object extraction means is used as an object area. It is desirable to use it by switching.

The second object extracting means performs inter-frame prediction in a different order from the input frame order so that a frame interval between the reference frame and the current frame from which the object is to be extracted is longer than a predetermined frame. It is characterized by. As a result, the amount of motion between frames is larger than when inter-frame prediction is sequentially performed in the order of input frames, so that prediction accuracy can be improved, and as a result, extraction accuracy can be improved.

That is, depending on the frame interval, the motion is too small or too complicated, and the shape prediction method based on inter-frame prediction may not be compatible. Therefore, for example, if the error of the shape prediction does not fall below the threshold, by spacing the extracted frame used for prediction,
The prediction accuracy increases, and as a result, the extraction accuracy improves. If there is motion in the background, the reference frame candidate calculates and compensates for the background motion with the extracted frame, but the background motion is too small or too complicated depending on the frame interval, and the background motion compensation is performed with high accuracy. It may not be possible. Also in this case, the motion compensation accuracy can be increased by providing a frame interval. By adaptively controlling the order of the extraction frames in this way, it is possible to more reliably extract the shape of the object.

Further, according to the present invention, moving image data and shape data representing an object region on a predetermined frame among a plurality of frames constituting the moving image data are input, and the moving image data is input using the shape data. In an object extraction device that extracts an object region from, by reading the moving image data from the storage device in which the moving image data is recorded, by motion-compensating the shape data,
Means for generating shape data for each frame constituting the read moving image data, and by sequentially overwriting the image data of the background area of each frame determined by the generated shape data on the background memory, Means for generating a background image of the moving image data, and reading the moving image data again from a storage device in which the moving image data is recorded, and reading the moving image data on the background memory for each frame constituting the read moving image data Means for determining a difference between a corresponding pixel of the background image stored in the image data and a pixel having an absolute value of the difference larger than a predetermined threshold value as an object area.

In this object extraction device, a background image is generated on a background memory in the first scanning process for reading out moving image data from a storage device. Then 2
A second scan process is performed, and an object region is extracted using the background image completed in the first scan. In this manner, by scanning the moving image data twice using the fact that the moving image data is stored in the storage device, the object region can be extracted with sufficiently high accuracy from the beginning of the moving image sequence. It becomes possible.

Further, according to the present invention, one of an object area determined based on the shape data of each frame and an object area determined based on an absolute value of a difference between the background image and the background image is selectively obtained as an object extraction result. Means for outputting to For some images, the object area determined by the shape data obtained in the first scan may have higher extraction accuracy in the second scan than the object area obtained by using the difference from the background image in the second scan. Therefore, by selectively outputting the object region obtained in the first scan and the object region obtained in the second scan, the extraction accuracy can be further improved.

Further, according to the present invention, moving image data and shape data representing an object region on a predetermined frame among a plurality of frames constituting the moving image data are input, and a frame or a frame given the shape data is input. An object extraction device that sequentially obtains shape data of each frame by using a frame for which shape data has already been obtained as a reference frame, and a unit that divides a current processing frame into blocks, and for each of the blocks, Means for searching the reference frame for a similar block having a pattern similar to that of the image data and having a larger area than the current processing block; and extracting and reducing the shape data of the similar block from the reference frame to the current processing. Means for attaching to each block of the frame, and the attached shape data And means for outputting a shape data of the current processing frame.

In this object extracting apparatus, a search process for a similar block whose image data (texture) pattern is similar and whose area is larger than the current processing block is performed for each block of the current frame from which the object is to be extracted. A process of cutting out the reduced shape data of the searched similar block and pasting it to the block of the current processing frame is performed. In this way, by shrinking and pasting the shape data of a similar block larger than the current processing block, even if the contour line of the object area given by the shape data is shifted, it can be corrected to the correct position. . Therefore, for example, by simply giving, as shape data, a user roughly tracing the outline of the object region on the first frame with a mouse or the like, the object region can be extracted with high accuracy in all subsequent input frames. .

According to the present invention, there is provided an object extracting apparatus for inputting image data and shape data representing an object region of the image, and extracting the object region from the image data using the shape data. Means for setting blocks in the outline portion, for each block, searching for a similar block in which the pattern of the image data is similar and larger than the block from the same image; and Means for replacing the shape data of the similar block with a reduced one;
Means for repeating the replacement for a predetermined number of times; and means for outputting the shape data, for which the replacement has been repeated, as corrected shape data.

As described above, by performing the replacement process using the similar book by the block matching in the frame, it becomes possible to correct the contour line given by the shape data to a correct position. In addition, since block matching is performed within a frame, similar book search and replacement can be repeatedly performed for the same block, thereby further improving the correction accuracy.

[0039]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 shows an overall configuration of a moving picture object tracking / extracting apparatus according to a first embodiment of the present invention. This object tracking / extracting device is for tracking the movement of a target object from an input moving image signal, and includes an initial figure setting unit 1 and an object tracking / extracting unit 2. The initial figure setting unit 1 is used for initializing a figure surrounding a target object to be tracked / extracted with respect to the input moving image signal a1, based on an external figure setting instruction signal a0 from outside. The input moving image signal a1 is generated by the initial figure setting instruction signal a0 such that a figure having an arbitrary shape such as a rectangle, a circle, and an ellipse surrounds a target object.
Is set on the initial frame. As an input method of the initial figure setting instruction signal a0, for example, a user directly writes a figure itself using a pointing device such as a pen or a mouse on a screen displaying the input moving image signal a1, or uses these pointing devices. For example, a method of designating the position and size of a figure to be input can be used. As a result, it is possible to easily specify the object to be tracked / extracted from the outside on the initial frame image where the target object appears.

Instead of inputting a figure by the user, the figure is initially set by, for example, detecting a face or body contour of a person or an animal by processing for analyzing a normal frame image, and automatically setting the figure so as to surround it. It can also be realized by setting.

The object tracking / extracting section 2 includes an initial figure setting section 1
The tracking and extraction of the object are performed based on the image in the figure included in the figure set in the step. In this case, in the moving object tracking / extraction process, the movement of the object is tracked by focusing on the object specified by the figure. Therefore, the target moving object can be extracted / tracked without being affected by extra surrounding movement other than the target moving object.

FIG. 2 shows an example of a preferred configuration of the object tracking / extracting section 2.

This object tracking / extracting unit, as shown in FIG.
It comprises memories (M) 11 and 14, a figure setting unit 11, a background area determination unit 12, and an object extraction unit 13.

The graphic setting section 11 is used to sequentially set a graphic for an input frame while using an arbitrary frame which has been input and set as a reference frame. The figure setting unit 11 includes a current frame image 101, an image in a figure of a reference frame, and its position 10
3 and an object extraction result 106 of the reference frame are input, and an image 10 representing the inside of an area surrounded by an arbitrary figure of the current frame is input.
2 is output. That is, in the graphic setting process by the graphic setting unit 11, based on the correlation between the image 103 in the graphic of the reference frame and the current frame image 101, the error in the image 103 in the graphic of the reference frame is minimized. An area is searched, and a figure surrounding the area is set for the current frame image 101. The shape to set is
Anything may be used, such as a rectangle, a circle, an ellipse, and an area surrounded by an edge. Hereinafter, a case of a rectangle will be described for simplicity. The specific configuration of the graphic setting unit 11 will be described later with reference to FIG. When a figure surrounding an object is not used, the image in the figure is an entire image, and there is no need to input and output positions.

The memory 10 holds at least about three frames which have been input and set as graphics. The information held includes the image of the frame in which the figure is set, the position and shape of the set figure, the image in the figure, and the like. Further, not the entire image of the input frame but only the image in the figure may be held.

The background area determination unit 12 uses at least two arbitrary frames among frames temporally different from the current frame for each current frame from which an object is to be extracted as reference frames, and uses the current frame for each reference frame. The common background area between each of these reference frames and the current frame is determined by taking the difference from the frame. The background region determination unit 12 stores an arbitrary in-graphic image of the current frame and its position 102 and an arbitrary in-graphic image of at least two frames and its position 103 held in the memory 10;
The object extraction result 106 of the at least two frames is input, and a common background area 104 of the current frame and the in-graphic image of each of the at least two frames is output.
That is, when the first and second two frames are used as reference frames, the first difference image obtained by taking an inter-frame difference between the current frame and the first reference frame or the like is used. A common first background area used as a background area is determined in both the current frame and the first reference frame, and an inter-frame difference between the current frame and the second reference frame is calculated. The common second background area used as the background area in both the current frame and the second reference frame is determined based on the second difference image obtained as a result. The specific configuration of the background area determination unit 12 will be described later with reference to FIG. There is also a method of using a background memory to obtain a common background.

When a figure surrounding an object is not used,
The images in the figure are all images, and there is no need to input and output positions.

The object extracting unit 13 is used to extract only the object region from the image in the figure of the current frame by using the common background region determined by the background region determining unit 12. A common background area 104 for each of the three frames is input, and an object extraction result 106 of the current frame is output. Since the first and second difference images both include the object region on the current frame in common, a region that does not belong to either the first common background region or the second common background region. , An object region on the current frame is extracted by detecting a region included in the in-figure image of the current frame. This utilizes the fact that areas other than the common background area are object area candidates. That is,
On the first difference image, regions other than the first common background region are object region candidates, and on the second difference image, regions other than the second common background region are object region candidates. The overlapping region of the object region candidates can be determined as the object region of the current frame. Object extraction result 1
As 06, information indicating the position and shape of the object area can be used. Further, an image of the object area may be actually extracted from the current frame using the information.

The memory 14 holds at least two object extraction results, and is used to feed back already extracted results to improve extraction accuracy.

Here, with reference to FIG. 8, a method of object extraction / tracking processing used in the present embodiment will be described.

Here, a case where an object is extracted from the current frame f (i) using three temporally continuous frames f (i-1), f (i), f (i + 1) will be exemplified. explain.

First, the figure setting section 11 performs a figure setting process. Three frames f (i-1), f
For each of (i) and f (i + 1), a figure setting process is performed by using an arbitrary reference frame, and a rectangle R (i−1),
R (i) and R (i + 1) are set. Note that rectangular figures R (i-1), R (i), and R (i + 1) are information on positions and shapes, and do not exist as images.

Next, a common background area is determined by the background area determination unit 12.

In this case, first, an inter-frame difference between the current frame f (i) and the first reference frame f (i-1) is obtained, and the first difference image fd (i-1, i) is obtained. Is required. Similarly, the inter-frame difference between the current frame f (i) and the second reference frame f (i + 1) is also taken,
A second difference image fd (i, i + 1) is obtained.

By obtaining the first difference image fd (i-1, i), the portion having a common pixel value between the current frame f (i) and the first reference frame f (i-1) is obtained. Since the pixel values are canceled, the difference value of the pixel becomes zero. Therefore, if the backgrounds of the frames f (i-1) and f (i) are almost the same, basically, the first difference image fd (i)
-1, i) include an image in the figure of the rectangle R (i-1),
An image corresponding to the OR of the rectangle R (i) with the image in the figure remains. As shown in the figure, the figure surrounding the remaining image is a polygon Rd (i-1, i) = R (i-1) OR
R (i). The common background area of the current frame f (i) and the first reference frame f (i-1) is a polygon Rd
Actual object area in (i-1, i) (here, an area in the shape of a figure 8 obtained by partially overlapping two circles)
All areas other than.

The second difference image fd (i, i + 1)
Also, the image in the figure of the rectangle R (i) and the rectangle R
An image corresponding to the OR with the (i + 1) in-graphic image remains. As shown in the figure, the figure surrounding the remaining image is a polygon Rd (i, i + 1) = R (i) OR R
(I + 1). The common background area of the current frame f (i) and the second reference frame f (i + 1) is a polygon Rd
The actual object area in (i, i + 1) (here, the figure of 8 is obtained by partially overlapping two circles)
All areas other than.

Thereafter, the first difference image fd (i−1,
i) to the current frame f (i) and the first reference frame f
The processing of determining the common background area of (i-1) is performed.

A difference value is required as a threshold value for determining a common background area / object area. This may be provided by the user, or may be automatically set by detecting noise or properties of the image. In this case, the threshold value may not be one for one screen but may be partially determined according to a partial property in the image. The properties of the image may include the strength of the edge and the variance of the difference pixels. Further, it can also be obtained by using a figure for tracking an object.

In this case, a difference value serving as a threshold value for judging the common background area / object area is obtained, and a pixel area having a difference value equal to or smaller than the threshold value is determined as the common background area. This threshold value is equal to the first difference image fd (i−
The determination is made using the histogram of the difference value of each pixel along one line outside the polygon Rd (i-1, i) of (1, i), that is, the contour of the polygon Rd (i-1, i). be able to. The horizontal axis of the histogram is the pixel value (difference value), and the vertical axis is the number of pixels having the difference value. For example, a difference value such that the number of pixels is half of the total number of pixels existing on the frame formed by the polygon Rd (i-1, i) is determined as the above-described threshold. By determining the threshold value in this manner, it is possible to easily determine the threshold value without checking the distribution of pixel values over the entire first difference image fd (i-1, i).

Next, a common background area in the polygon Rd (i-1, i) of the first difference image fd (i-1, i) is determined using the threshold value. Areas other than the common background area are object areas including occlusion. As a result, the region in the polygon Rd (i-1, i) is divided into a background region and an object region, and the pixel value of the background region is "0" and the pixel value of the object region is "1". Is converted to

Similarly, for the second difference image fd (i, i + 1), a process of determining a common background area between the current frame f (i) and the second reference frame f (i + 1) is performed. The area within the polygon Rd (i, i + 1) is converted into a background area with a pixel value of “0” and an object area with a pixel value of “1”.

After that, the object extracting unit 13 performs the object extracting power.

Here, between the first and second difference images, the binary image in polygon Rd (i-1, i) and the binary image in polygon Rd (i, i + 1) are compared. An arithmetic operation for performing an AND operation for each pixel is performed, whereby the intersection of the objects containing the occlusion is determined, and the current frame f (i)
The upper object O (i) is extracted.

Here, a case has been described where all the regions other than the object region in the frame difference image are obtained as the common background region. However, only the in-figure image is extracted from each frame and each figure on the frame is extracted. The difference calculation between the images in the figure may be performed in consideration of the position of the image in the figure. In this case, the polygon Rd (i−1, i) can be calculated without being conscious of the background area outside the figure. And only the common background region within polygon Rd (i, i + 1).

As described above, in the present embodiment, 1) calculating the difference image between the current frame and the first and second at least two reference frames that are temporally different from the current frame, First
The OR of the image in the figure between the reference frames and the OR of the image in the figure between the current frame and the second reference frame are obtained. 2) The difference image obtained by the OR processing of the images in the figure is AND-processed. Thus, the object extraction is performed by the ORAND method focusing on the image in the figure, that is, extracting the target object region from the image in the figure of the current frame.

The time relationship between the current frame and the two reference frames is not limited to the above example.
Two frames f (i) that temporally precede (i)
-M), f (in) as reference frames, or two temporally continuous frames f (i + m), f
(I + n) can also be used as a reference frame.

For example, in FIG. 8, the frame f (i-
1) and f (i) are used as reference frames, the difference between each of these reference frames and the frame f (i + 1) is calculated, and the same processing is performed on the difference images.
An object can be extracted from the frame f (i + 1).

FIG. 3 shows a second configuration example of the object tracking / extracting section 2.

The main difference from the configuration of FIG. 2 is that a background motion deletion unit 21 is provided. The background motion deletion unit 21 is used to correct the background motion so that the background motion is relatively zero between each reference frame and the current frame.

Hereinafter, the apparatus shown in FIG. 3 will be specifically described.

The background motion deleting unit 21 outputs the current frame 201
And an arbitrary in-graphic image of at least two frames shifted in time and its position 206 are input, and an image 202 in which background motion of at least two frames shifted in time is deleted is output. A specific configuration example of the background motion deletion unit 21 will be described later with reference to FIG.

The graphic setting unit 22 is provided with the graphic setting unit 11 shown in FIG.
And inputs a current frame 201, at least two images 202 from which the background motion has been deleted, and a rest extraction result 206 of the image 202, and converts the current frame and the at least two images 202 into arbitrary figures. An image 203 representing the inside of the enclosed area is output.

The memory 26 holds an image in an arbitrary figure and its position.

The background area determining section 23 corresponds to the background area determining section 12 in FIG. 2, and inputs the arbitrary figure image and its position 203 and the object extraction result 206 of the image 202.
Output a common background area 204 between the current frame and the at least two images 202. The object extracting unit 24 corresponds to the object extracting unit 13 in FIG.
A common background area 204 with one image is input, and an object extraction result 205 of the current frame is output. The memory 25 holds at least two object extraction results. This is shown in FIG.
Of the memory 14.

By providing the background motion removing unit 21 in this manner, even if the background image gradually changes between consecutive frames, such as when the camera is panned, the background motion is deleted between those frames. The image can be made pseudo-constant. Therefore, when the difference between the current frame and the reference frame is obtained, the background can be canceled between the frames, and the common background area detection processing and the object area extraction processing that are not affected by the background change can be performed.

Note that the background motion deleting unit 21 may be provided at the input stage of the background area determining unit 23 so that the background motion of the reference frame is deleted according to the current frame.

FIG. 4A shows an example of a specific configuration of the background area determination unit 12 (or 23).

The change amount detecting section 31 is used to obtain a difference between the current frame and the above-mentioned first and second reference frames. The position 302 and the object extraction result 301 of the frame shifted in time are input, and the amount of change 303 between any in-figure images of the frame shifted in time from the current frame is output. As the amount of change, for example, a luminance difference between frames, a change in color, an optical flow, or the like can be used. If an object extraction result of a frame shifted in time is used, an object can be extracted even if the object does not change between frames. For example, assuming that the amount of change is an inter-frame difference, the portion of the inter-frame difference of zero belonging to the object is the same as the object extraction result of a temporally shifted frame because the object is stationary.

The representative area determining section 32 inputs an arbitrary graphic image of the current frame and its position 302, and outputs the background of the arbitrary graphic image as a representative area 304. As the representative area, an area that is expected to have the most background in any figure is selected. For example, a band-like region is set on the outermost side in the figure, such as the outline of the figure on the difference image described in FIG. Since the shape is set to surround the object,
It is likely to be a background.

In the background change amount determining section 33, the representative area 30
4 and the change amount 303, and outputs a change amount 305 for determining the background. The background change amount is determined by taking a histogram of the change amount of the difference value of the representative area as described with reference to FIG. 8 and, for example, a change amount corresponding to a number of pixels equal to or more than half (the majority) of both prime numbers, ie, the difference amount. An area having a value is determined as a background area.

The representative area background determination unit 34 inputs the amount of change 305 of the background and determines the background 306 of the representative area.
Output. The background region of the representative region is determined based on the background change amount determined previously. The background area determination unit 35 inputs the amount of change 303, the threshold value 305 for background determination, and the background 306 of the representative area, and outputs the background 307 of an area other than the representative area. The background area other than the representative area is determined from the representative area by a growth method. For example, if an undetermined pixel adjacent to the determined pixel in the internal direction of the figure matches the background change amount, the background is determined. Pixels that are not adjacent to the background or pixels that do not match the background change amount are determined to be other than the background. Alternatively, the determination may be made simply based on whether or not the background change amount is determined beforehand. In this way, by narrowing down from the outline of the figure on the difference image toward the inner periphery, it is possible to determine how far the background area is in the image within the figure.

On the other hand, an object area which protrudes from the figure outward from the contour of the figure is detected. For example,
If an unsettled pixel adjacent to the pixel determined to be other than the background in the outward direction of the figure does not match the background change amount, it is determined to be other than the background. Pixels that are not adjacent to pixels other than the background or pixels that match the background change amount are determined to be background. In this way, by expanding outward from the outline of the figure on the difference image, it is possible to determine how far the image outside the figure is the area outside the background. In this case, since it is necessary to obtain the change amount even outside the figure, a figure which does not protrude sufficiently from the object is newly set by thickening an arbitrary figure by several pixels, and
The change amount may be obtained inside the frame, or the change amount may be obtained simply for the entire frame. Alternatively, the amount of change may be determined only inside the figure, and the above-described processing may be performed while determining the amount of change at any time when determining the outside of the figure. Of course, when the object does not protrude from the figure, for example, when there is no pixel other than the background on the contour, there is no need to perform processing outside the figure.

When the object or a part of the object is stationary between the current frame and the reference frame, the difference between the current frame and the reference frame may not be detected, and the shape of the object may not be correctly extracted. . Therefore, a method of detecting an object in the current frame using the already extracted reference frame will be described with reference to FIG.

FIG. 4B shows the background area determining section 12 (or 23) including the stationary object area detecting section 37. According to this, the change amount detection unit 31 compares the image in the figure of the current frame and its position with at least two
An arbitrary in-figure image of one frame and its position 311 are input, and a change amount 313 between the in-figure image of the current frame and the reference frame is detected.

The shape estimating unit 36 calculates the image in the figure of the current frame and its position, the image in the figure and the position 311 of at least two frames which are temporally shifted, the image and the object of the frame already extracted. With the shape 317 as an input, an object shape 312 is predicted and output for a frame in which an object has not yet been extracted among frames temporally shifted from the current frame.

The stationary object region determining unit 37 calculates the predicted object shape 312 and the change amount 31 between the reference frame and the current frame.
3 and the object shape 317 of the already extracted frame, and an object region 314 that is stationary with respect to the current frame is determined from at least two frames.

The background area determination unit 35 determines the object area 314 that is stationary with respect to the current frame and the amount of change 3 between the reference frame and the current frame for at least two frames.
13 as input, a common background area 316 for each of at least two frames and the current frame is determined and output.

First, when the object of the reference frame has been extracted, if the inter-frame difference between the reference frame and the current frame is zero, if the region at the same position in the reference frame is a part of the object, the current frame is extracted. That region in the frame can be extracted as part of a stationary object. Conversely, if the region is a part of the background in the reference frame, the region in the current frame becomes the background.

However, if the object of the reference frame has not been extracted yet, a stationary object or a part of the object cannot be extracted by the above method. In that case, the object shape of the reference frame that has not been extracted can be predicted using another frame from which the object has already been extracted, and it can be determined whether or not the frame is a part of the object. As a prediction method, a block matching method, an affine transformation method, or the like often used in image coding is used.

As an example, a block matching method as shown in FIG. 13 can be considered. If the shape of the object is predicted in this manner, it is possible to determine whether the region where no inter-frame difference is detected is a part of the stationary object or the background.

When a figure surrounding an object is not used, the image in the figure is an entire image, and there is no need to input and output positions. For this shape prediction, the same shape prediction as that used when a reference frame is selected can be used. Further, in the embodiment in which switching is performed with another object extraction method, an object shape obtained by another object extraction method can be used.

FIG. 5 shows the figure setting section 11 (or 22).
Is shown as an example of the specific configuration of the above.

The division unit 41 inputs an arbitrary figure image of a frame temporally shifted from the current frame and its position 402, and outputs the divided image 403. The division of an arbitrary in-figure image may be divided into two equal parts, four equal parts, or an edge may be detected and division along the edge may be performed. Hereinafter, simply, the figure is divided into two, and the divided figure is called a block. The motion detector 42 receives the divided arbitrary intra-graphic image and its position 403, the arbitrary intra-graphic image of the current frame and its position 401, and outputs the motion and error 404 of the divided image. . Here, the position of the block corresponding to the current frame is searched so as to minimize the error, and the motion and the error are obtained. The division determining unit 43 receives the motion and the error 404 and the object extraction result 407 of the temporally shifted frame, and determines whether or not to divide an arbitrary in-figure image of the temporally shifted frame.
If no division is performed, a motion 405 is output. Here, if the object extraction result of the frame shifted in time is not included in the divided block, the block is deleted from the figure. Otherwise, from the error found,
If the error is equal to or larger than the threshold, the motion is further divided and the motion is obtained again. Otherwise, determine the motion of the block. The figure determining unit 44 receives the movement 405 as an input, and outputs an image in the figure of the current frame and its position 407. Here, the figure determining unit 44 obtains the position correspondence of each block to the current frame, and determines a new figure so as to include all the blocks at the corresponding positions. The new figure may be a rectangle or a circle that includes all the blocks and has a good connection between all the blocks.

As described above, the image in the figure of the reference frame is divided into a plurality of blocks, a region where the error from the current frame is minimized for each of the plurality of blocks, and the plurality of searched regions are surrounded. By setting a graphic as the current frame, it is possible to set a new graphic having an optimal shape for the input frame to be set, regardless of the initially set shape and size of the graphic.

The reference frame used for setting the figure may be a frame in which a figure has already been set and which is temporally shifted from the current frame, and the forward prediction and the backward prediction can be performed by ordinary coding techniques. Similarly, a frame that is temporally later than the current frame can be used as a reference frame for setting a figure.

FIG. 6 shows an example of a specific configuration of the background motion deletion section 21.

The representative background area setting section 51 receives an arbitrary image in a figure shifted in time and its position 501 as an input, and outputs a representative background area 503. The representative background region is a region that represents the global movement in an arbitrary figure, that is, the movement of the background in the figure. For example, when an arbitrary figure is made into a rectangle, a rectangle as shown in FIG. A band-like frame area of several pixels width is set. Also, several pixels outside the figure may be used. The motion detection unit 52 receives the current frame 502 and the representative background area 503 and
4 is output. Using the above example, the motion of the band-shaped frame area around the rectangle with respect to the current frame is detected. The frame area may be detected as one area. Further, as shown in FIG. 7, the motion may be obtained by dividing the motion into a plurality of blocks, and the average motion of each motion may be output, or the motion with the largest motion may be output.

The motion compensator 53 receives the temporally shifted frame 501 and the motion 504, and outputs a motion compensated image 505. Using the motion obtained previously, the motion of the frame shifted in time is deleted according to the current frame. The motion compensation may be block matching motion compensation or motion compensation using affine transformation.

Note that the motion may be deleted not only for the background in the figure but also for the entire frame.

As described above, in the present embodiment,
(1) Tracking an object using a figure that roughly surrounds the object instead of the contour of the object; (2) Setting an arbitrary figure in the current frame, and setting a figure for each of the current frame and at least two frames Determining a common background area with the inner image and extracting the object of the current frame; (3) removing background motion of at least two frames shifted in time; The amount of change between images is detected, a representative region is determined, and the amount of change in the image in the figure of the current frame and at least two frames and its position corresponding to the background is determined. Judging whether it is the background, (5) dividing the image in the figure, detecting the movement of a part of the image in the figure or the divided image in the figure, and detecting the image in the figure or the divided figure Whether to split part of the image And (6) setting an area representing the background, detecting the background motion, and detecting the background motion of the temporally shifted frame. By creating an image in which is deleted, the target object can be accurately extracted and tracked by relatively simple processing without being affected by extra movement around the target object.

Further, the procedure of the object extraction / tracking processing of the present embodiment can be realized by software control. Even in this case, the basic procedure is exactly the same, and after performing the initial setting of the figure, the figure setting processing is sequentially performed on the input frame, and in parallel with this figure setting processing or after the completion of the figure setting processing, The background area determination processing and the object extraction processing may be performed.

Next, a second embodiment of the present invention will be described.

In the above-described first embodiment, only one object extracting means based on the ORAND method is provided. However, depending on an input image, sufficient extracting performance may not be obtained by only that means. In the ORAND method according to the first embodiment, a common background is set based on a difference between a current frame from which an object is to be extracted and a first reference frame that is temporally different from the current frame. A common background is set based on a difference between the current frame and a second reference frame that is temporally different. However, there is no specific method for selecting the first and second reference frames. Depending on the selection of the first and second reference frames, there may be a large difference in the object extraction result, and good results may not be obtained.

Therefore, in the second embodiment, the first embodiment is improved so that an object can be extracted with high accuracy regardless of the input image.

First, a first configuration example of the object tracking / extracting apparatus according to the second embodiment will be described with reference to the block diagram of FIG.

In the following, only the configuration corresponding to the object tracking / extracting section 2 of the first embodiment will be described.

The figure setting section 60 is the same as the figure setting section 11 of the first embodiment described with reference to FIG. 2, and includes a frame image 601 and an initial frame or a figure 602 already set for another input frame. Is input, and a figure is set in the frame image 601 and output. Switch section SW6
1 receives the result 605 of the object extraction already performed as an input,
Based on this, a signal 604 for switching the object extraction unit to be used is output.

The object tracking / extracting section 62 is composed of first to K-th plural object tracking / extracting sections as shown in the figure. These object tracking / extracting units perform object extraction by different methods. The object tracking / extracting unit 62 includes at least one using the ORAND method described in the first embodiment. Further, as the object tracking / extracting unit using another method, for example, a unit using a shape prediction method by block matching, an object shape prediction by affine transformation, or the like can be used. In these shape predictions, the position or shape of the object region on the current frame is predicted by inter-frame prediction between the frame from which the object has already been extracted and the current frame, and based on the result of the prediction, the image region 603 of the current frame is used to extract the object region. Is extracted.

FIG. 13 shows an example of shape prediction by block matching. The in-graphic image of the current frame is divided into blocks of the same size as shown. For each block, a block having the most similar pattern (texture) is searched from the reference frame from which the shape and position of the object have already been extracted. For this reference frame, shape data representing the object region has already been generated. In the shape data, the pixel values of the pixels belonging to the object area are represented by “255”, and the other pixel values are represented by “0”. The shape data corresponding to the searched block is attached to the corresponding block position in the current frame. By performing such a texture search and shape data pasting process on all the blocks constituting the graphic image in the current frame, the graphic image in the current frame is filled with shape data for distinguishing the object region from the background region. . Therefore, by using this shape data, an image (texture) corresponding to the object area can be extracted.

The switch SW61 performs, for example, the same operation as the first object tracking / extracting unit. When the extraction accuracy is good, the switch SW61 is switched to select the first object tracking / extracting unit. Switch to selecting a copy. For example, if the first object tracking extraction unit is an object shape prediction unit based on block matching, the switching of the object tracking extraction unit may be controlled according to the magnitude of the matching error. In the case of object shape prediction by affine transformation, the object tracking extraction unit can be switched according to the magnitude of the estimation error of the affine transformation coefficient. Switch SW
The switching unit at 61 is not a frame unit but a small area in a frame, for example, a block or an area divided based on luminance or color. As a result, the object extraction method to be used can be selected more finely, and the extraction accuracy can be increased.

FIG. 10 shows a second example of the moving image object tracking / extracting apparatus according to the second embodiment.

The figure setting section 70 is the same as the figure setting section 11 of the first embodiment described with reference to FIG.
And the figure 702 already set for the initial frame or another input frame, and the figure is set to the frame image 701 and output.

The second object tracking / extracting section 71 is used to extract an object area by shape prediction such as a block matching method or affine transformation. And the shape and position 7 of the object on another reference frame already extracted
07 is input, and the shape and position of the object are predicted from the in-figure image 703 of the current frame.

The reference frame selecting section 72 inputs the predicted shape and position 704 of the object of the current frame predicted by the second object tracking / extracting section 71 and the shape and position 707 of the object already extracted. , Select at least two reference frames. Here, a method of selecting a reference frame will be described.

O_i, O_j, O_currAre frames i, j
And the object of the frame curr being extracted. Two temporal
Different reference frame f_i, F_jAnd the difference d_i, D_jTo
Then, these differences are AND-processed and the current frame f_curr
Is extracted, the object O to be extracted is extracted._currOther than things
Body O_i , O_jOverlapped parts of frames that differ in time
It is extracted by AND processing. Of course, O_i∩ O_j=
φ, that is, the object O_i , O_j There is no overlapping part of
Body O_i , O_jIs a problem if the overlap of
No.

However, the object O_i , O_jOverlap part of
(O_i∩ O_j≠ φ) and the overlapping part is extracted
If it exists outside the object you want to_currAnd O_i∩
 O_j Are left as extraction results.

[0117] In this case, as shown in FIG. 14 (a), O _curr
{O _curr } when there is no common region between the background region (O _curr ￣) and the objects Oi and O _j ∩ (O _i ∩
If O _j ) = φ｝, there is no problem. However, FIG.
As in (b), O _curr the background region and (O _curr ¯), the object O _i and, when all of the common area between the O _j exists {O _c
urr ￣ ∩ (O _i ∩ O _j ) ≠ φ ≠ is extracted in an erroneous shape such as O _curr indicated by oblique lines.

Therefore, the optimal reference frames f _i and f _j for correctly extracting the shape of the object are frames satisfying (O _i ∩O _j ) ∩O _curr (1), that is, the frames of O _i and O _j Frames f _i and f _j whose overlapping portions belong to O _curr (FIG. 14A).

When two or more reference frames are selected, (O _i ∩O _j ∩... ∩O _k ) ∩O _curr (2)

Therefore, based on the prediction result of the position or shape of the object on the current frame from which the object is to be extracted,
By selecting a reference frame that satisfies the expression (1) or the expression (2), the shape of the object can be reliably extracted.

The first object tracking / extracting section 73 receives at least two reference frames 705 selected by the reference frame selecting section 72 and the current image 701, extracts an object by the ORAND method, and extracts its shape and position. 706 is output.

The memory 74 holds the extracted object shape and position 706.

FIG. 11 shows a third configuration example of the object tracking / extracting apparatus according to the second embodiment.

As shown, the object tracking / extracting apparatus comprises a figure setting unit 80, a second object tracking / extracting unit 81, a switch unit SW82, and a first object extracting unit 83. The figure setting unit 80, the second object tracking / extracting unit 81, and the first object extracting unit 83 are each configured as shown in FIG.
0 corresponds to the figure setting unit 70, the second object tracking / extracting unit 71, and the first object extracting unit 73. In this example,
The extraction result of the second object tracking / extraction unit 81 and the extraction result of the first object extraction unit 83 are selectively used by the switch unit SW82.

That is, in the graphic setting section 80, the image 80
1 and the shape and position 802 of the initial figure are input, and the shape and position 803 of the figure are output. The second object tracking / extracting unit 81 inputs the shape and position 803 of the figure and the shape and position 806 of the object that has already been extracted, and predicts the predicted shape and position 804 of the object that has not yet been extracted. Output. The switch unit SW82 inputs the shape and position 804 of the object predicted by the second object extraction unit, and outputs a signal 8 for switching whether or not to perform the first object tracking / extraction unit.
05 is output. The object tracking / extracting unit 83 inputs the shape and position 806 of the object that has already been extracted and the predicted shape and position 804 of the object that has not yet been extracted, and determines and outputs the shape and position 805 of the object.

The unit of switching in the switch unit SW82 may be switched for each block as in the above-described example, or may be switched for each area divided based on luminance or color. As a method of determining switching, for example, a prediction error at the time of predicting an object can be used. That is, when the prediction error in the second object tracking / extracting unit 81 that performs object extraction using inter-frame prediction is equal to or smaller than a predetermined threshold, the prediction obtained by the second object tracking / extracting unit 81 Switch unit S so that the shape is used as the extraction result
Switching by W82 is performed, and the second object tracking / extracting unit 8
When the prediction error at 1 exceeds a predetermined threshold, the first object tracking / extracting unit 83 performs switching by the switch unit SW82 so that the object is extracted by the ORAND method, and the extraction result is obtained. Is output to the outside.

FIG. 15 shows an example of an extraction result when the extraction unit to be used is switched based on the matching error for each block serving as a prediction unit.

Here, the area shown by the mesh is the object shape obtained by the prediction by the second object tracking / extracting section 81,
The shaded area is the object shape obtained by the first object tracking / extracting unit 83.

FIG. 12 shows a fourth configuration example of the moving image object tracking / extracting apparatus according to the second embodiment.

This object tracking / extracting apparatus is obtained by adding a reference frame selecting unit shown in FIG. 10 to the configuration shown in FIG.

The graphic setting section 90 inputs the image 901 and the shape and position 902 of the initial figure, and outputs the shape and position 903 of the figure. In the second object tracking / extracting unit 91,
The shape and position 903 of the figure and the shape and position 908 of the object already extracted are input, and the predicted shape and position 904 of the object not yet extracted are predicted and output. The switch unit SW92 receives the predicted shape and position 904 of the object, determines whether the accuracy of the predicted object is high, and outputs a signal 905 for switching the output of the object extracted by the second object extraction unit. I do. Reference frame selection unit 93
Is the predicted shape and position 90 of the object that has not yet been extracted.
4 and the shape and position 908 of the object already extracted, and select and output the shape and position 906 of at least two reference frame objects or predicted objects. The object tracking / extracting unit 94 calculates the current image 901 and the shape and position 90 of the object or the predicted object of at least two reference frames.
6, an object is extracted, and its shape and position 90 are extracted.
7 is output. The memory 95 stores the extracted shape and position 907 of the object and the predicted shape and position 904 of the object.
Hold one.

Hereinafter, the procedure of the object tracking / extracting method in this example will be described with reference to FIG.

(Step S1) As a reference frame candidate, a frame temporally shifted from the current frame is set in advance. This may be all frames other than the current frame, or may be limited to several frames before and after the current frame.
For example, the total number of frames is limited to an initial frame, three frames before the current frame, and one frame after the current frame. However, if there are not three previous frames, the number of candidates for the subsequent frame is increased, and if there is no subsequent frame, four previous frames are used as candidates.

(Step S2) First, a figure in which the user writes an object to be extracted in the initial frame is set, for example, as a rectangle. For the graphics in the subsequent frames, the default graphics are divided into blocks, matching is performed, and the blocks are pasted at corresponding positions. The object is tracked by setting a new rectangle to include all the pasted blocks. An object tracking figure is set for all reference frame candidates. Whenever an object is extracted, it is used to recalculate the object tracking figure of the previous frame, thereby preventing the extraction error. Further, the user inputs the object shape of the initial frame.

Hereinafter, it is assumed that the frame to be extracted is the current frame, that the object before the current frame has already been extracted, and that the previous frame has not been extracted.

(Step S3) An appropriate area is set around the reference frame candidate figure, the background movement with the current frame is detected, and the background in the reference frame figure is deleted.
As a method of detecting the background motion, a region having a width of several pixels around the figure is set, this region is matched with the current frame, and a motion vector that minimizes the matching error is set as the background motion.

(Step S4) Reference frames having a large motion vector detection error at the time of background motion deletion are excluded from candidates, thereby preventing an extraction error when background motion deletion is not appropriate. When the number of reference frame candidates decreases, a new reference frame candidate may be selected. If the figure setting and background motion deletion of the newly added reference frame candidate have not been performed, it is necessary to newly perform the figure setting and background motion deletion.

(Step S5) Next, the current frame from which no object has been extracted and the object shape of a reference frame candidate preceding the current frame are predicted. The rectangle set as a candidate for the current frame or the previous reference frame is divided into blocks, for example, and is matched with a frame (an earlier frame) from which an object has already been extracted, and the corresponding object shape is pasted to predict the object shape. I do. Whenever an object is extracted, it can be used to redo the object prediction of the previous frame to prevent an extraction error.

(Step S6) At this time, for the block with a small prediction error, the predicted shape is output as it is as the extraction result. Further, if the prediction of the object shape is performed on a block basis, a block error may occur due to a matching error. Therefore, a filter that eliminates the distortion may be applied to smooth the entire object shape.

The rectangle division performed at the time of object tracking and object shape prediction may be performed with a fixed block size, or may be performed by hierarchical block matching using a matching threshold.

For a block having a large prediction error, the following processing is performed.

(Step S7) A provisional reference frame is set from the reference frame candidates, and a set of reference frames satisfying Expression (1) or Expression (2) is selected for each combination. If none of the combinations of all reference frame candidates satisfies the expression (1) or the expression (2), it is preferable to select the one having the minimum number of pixels in O _{i ∩} O _j . Also, it is better to consider the combination of reference frame candidates so as to select a frame in which the motion vector detection error at the time of background motion deletion is as small as possible. Specifically, when there is a reference frame set having the same condition according to the expression (1) or the expression (2), there is a method of selecting a smaller motion vector detection error when deleting background motion. Hereinafter, it is assumed that two reference frames are selected.

(Step S8) When a reference frame is selected, an inter-frame difference from the current frame is obtained, and attention is paid to the inter-frame difference in the set graphic. A histogram of the absolute value of the difference of one line pixel outside the set figure is obtained, and the absolute value of the difference that frequently appears is set as the difference value of the background area.
The background pixel of one line pixel outside the set figure is determined. From the background pixel of one line pixel outside the set figure to the inside, a pixel having a difference value of an adjacent background area is determined as a background pixel, and is sequentially continued until it is determined that the pixel is not a background pixel. This background pixel is a common background area for the current frame and one reference frame. At this time, since the boundary between the background region and the other portion may be unnatural due to the influence of noise, a filter for smoothing the boundary or a filter for deleting extra or noise regions may be applied.

(Step S9) When a common background area is obtained for each reference frame, an area not included in the two common background areas is detected and extracted as an object area. For parts that do not use the shape of the object predicted earlier,
The result here is output, and the shape of the entire object is output.

When the part using the shape obtained from the common background and the part using the object shape predicted earlier cannot be matched, it is possible to obtain a good output result by applying a filter at the end.

As described above, according to the second embodiment, an object can be accurately extracted regardless of an input image.
Alternatively, a reference frame suitable for object extraction can be selected.

Next, a third embodiment of the present invention will be described.

First, using the block diagram of FIG.
A first example of the object tracking / extracting device according to the embodiment will be described.

Here, a configuration is adopted in which a feature amount of an image of at least a part of the region is extracted from a current frame from which an object is to be extracted, and a plurality of object extraction means are switched based on the feature amount. .

That is, as shown in the figure, the object tracking / extracting apparatus comprises a figure setting unit 110, a feature amount extracting unit 111, a switch SW 112, a plurality of object tracking / extracting units 113,
And a memory 114. Figure setting unit 1
10, the switch unit SW112, and the plurality of object tracking / extracting units 113 are the same as the graphic setting unit 60, the switch unit SW61, and the plurality of object tracking / extracting units 62 of FIG. 9 described in the second embodiment, respectively. Based on the feature amount of the image of the current frame extracted by the feature amount extraction unit 111,
The difference is that the object tracking / extracting unit to be used is switched.

The figure setting unit 110 is used to
1, an initial figure 1102 set by the user, and an extraction result 1106 of an already extracted frame are input, a figure is set in the extracted frame, and the figure is output. The figure may be a geometric figure such as a rectangle, a circle, an ellipse, or the user may input an object shape to the figure setting unit 110. In that case, the graphic need not be a precise shape but may be a rough shape. The feature amount detection unit 111 includes an extraction frame 1103 in which a figure is set and an extraction result 1 of the already extracted frame.
106, and outputs a feature quantity 1104. The switch unit SW112 receives the feature amount 1104 and the extraction result 1106 of the already extracted frame, and controls the input of the extracted frame 1103 in which the figure is set to the object tracking / extracting unit.

When the feature amount has been obtained for the entire image, the switch unit SW112 detects the characteristics of the image and can be used for controlling the input to an appropriate object tracking / extracting unit for the image. The inside of the figure is divided into an appropriate size, and the feature amount may be given for each divided figure. The feature amount is a variance, a luminance gradient, an edge strength, or the like. In this case, these can be automatically calculated. In addition, the property of the object visually recognized by the human is changed by the user through the switch unit W112.
May be given to For example, if the target object is a person, the parameters at the time of extraction may be specially selected by designating hair with unclear edges, and may be extracted after edge correction in preprocessing.

The feature amount may be not only the inside of the set figure (object and its surroundings) but also the outside of the figure (background part).

A plurality of (first to k) object tracking / extracting units 11
3, the extracted frame 1103 in which the figure is set
And an extraction result 1106 of the already extracted frame, and outputs a result 1105 of tracking and extracting the object.

The plurality of object tracking / extracting sections 113
Includes one that extracts objects using the ND method, one that extracts objects using chroma keying, one that extracts objects by block matching and affine transformation, and the like.

In the first embodiment, the background pixel is determined using the histogram of the inter-frame difference of the pixel values around the set figure. However, the pixel whose inter-frame difference is equal to or smaller than the threshold value is simply determined. It may be determined as a background pixel.
Further, in the first embodiment, the background pixel (the difference value is equal to or less than a certain value) is determined from the set figure toward the inside of the figure. ) May be determined or may be in any order of operation.

The memory 114 receives the result 1105 obtained by tracking and extracting the object, and holds the result.

In the following, the characteristic amount indicating the characteristic of the image
The reason why a better extraction result is obtained by switching the tracking / extraction method will be described.

For example, if it is known in advance whether or not there is background movement, it is better to use that property. If there is background motion, background motion compensation is performed, but it is not known whether it can be completely compensated. Almost no motion can be compensated for frames with complex motion. Since such a frame can be known in advance by a compensation error of the background motion compensation, it is possible to devise a method such as not using the frame as a reference frame candidate. However, if there is no background movement, this process is unnecessary. If another object is moving, erroneous background motion compensation is performed, or the frame deviates from the reference frame candidate, so that even if it is the best frame for the reference frame selection condition, it is not selected, and the extraction accuracy decreases. There is.

Also, various properties are mixed in one image. The movement and texture of the object are also partially different,
The same tracking / extraction method, device, and parameters may not be able to extract an object well. Therefore, the user can specify a part having a special property in the image, automatically detect the difference in the image as a feature, extract the object by partially switching the tracking / extraction method, It is better to change the parameters.

By switching the means for tracking / extracting a plurality of objects as described above, it becomes possible to accurately extract the shapes of objects in various images.

Next, referring to the block diagram of FIG.
A second configuration example of the moving image object tracking / extracting device according to the embodiment will be described.

The figure setting unit 120 is used to
1, an initial figure 1202 set by the user, and an extraction result 1207 of the already extracted frame are input, and a figure is set to the extracted frame and output. The second object tracking / extracting unit 121 is used for extracting an object region by shape prediction such as a block matching method or an affine transformation, and includes an extraction frame 1203 in which a figure is set and an extraction result of an already extracted frame. 1207 is input and an object tracking / extraction result 1204 is output.

The feature amount extraction unit 122 receives the object tracking / extraction result 1204 and outputs the object feature amount 1205 to the switch unit SW123. Switch section SW123
Controls the input of the object tracking / extraction result 1204 to the first object tracking / extracting unit with the feature amount 1205 of the object as an input. For example, when the object shape is tracked and extracted by the second object tracking / extracting unit 121 by the block matching method, the feature amount is set as a matching error, and a portion where the matching error is small is determined by the second object tracking / extracting unit 121. It is output as the extraction result of the predicted shape. Further, as other feature amounts, there are parameters (such as fractal dimensions) representing the brightness gradient, variance, and texture complexity for each block. In the case where the luminance gradient is used, the input to the first object tracking / extracting unit is performed for a block having almost no luminance gradient so that the result of the first object tracking / extracting unit 124 using the ORAND method is used. Controlled. When the edge detection is performed and the presence / absence or strength of the edge is used as a feature amount, the first object tracking / extracting unit 124 uses the result of the first object tracking / extracting unit 124 at a position where there is no edge or a weak position. The input to the extraction unit is controlled. In this way, the switching control is changed in units of blocks or regions which are part of the image. Adaptive control can be performed by increasing or decreasing the switching threshold.

The first object tracking / extracting section 124 includes: an extraction frame 1201; an object tracking / extraction result 1204;
The extraction result 1207 of the already extracted frame is input, and the tracking / extraction result 1206 of the extracted frame is stored in the memory 1.
25. The memory 125 receives and outputs the tracking / extraction result 1206 of the extracted frame.

Next, a third configuration example of the object tracking / extracting apparatus according to the third embodiment will be described with reference to the block diagram of FIG.

This object tracking / extracting apparatus is obtained by adding the reference frame selecting unit described in the second embodiment to the configuration shown in FIG. That is, this object tracking / extracting device includes a figure setting unit 130, a second object tracking / extracting unit 131, a feature amount extracting unit 132, and a switch unit S, as shown in the figure.
W 133, a reference frame selecting unit 134, a first object tracking / extracting unit 135, and a memory 136.

In the figure setting unit 130, the extracted frame 13
01, an initial figure 1302 set by the user, and an extraction result 1308 of a frame already extracted, and a figure is set as an extracted frame and output. Second object tracking
The extraction unit 131 is for extracting an object region by shape prediction such as a block matching method or an affine transformation, and receives an extraction frame 1303 in which a figure is set and an extraction result 1308 of an already extracted frame as inputs.
The object tracking / extraction result 1304 is output.

The feature extraction unit 132 receives the object tracking / extraction result 1304 as input and outputs the object feature 1305. The switch unit SW133 includes a feature amount 1305 of the object.
Is input, and the input of the object tracking / extraction result 1304 to the first object tracking / extracting unit 135 is controlled.

The reference frame selecting unit 134 outputs the object tracking / extracting result 1304 to the first object tracking / extracting unit 135.
, The extraction result 1308 of the already extracted frame is input, and the reference frame 1306 is output.

One example of a feature of an object is the complexity of movement. When the second object tracking / extracting unit 131 tracks and extracts an object by the block matching method, a first object extraction result is output for a portion having a large matching error. If there is a partially complicated motion, the matching error increases in that portion, and the first object tracking / extracting unit 135
Will be extracted. Therefore, the selection method of the reference frame used by the first object tracking / extracting unit 135 using this matching error as a feature amount is switched. Specifically, not the entire object shape, but only the portion extracted by the first object tracking / extracting unit 135 so as to satisfy the selection condition of the equation (1) or (2) described in the second embodiment. It is better to choose a selection method.

Examples of the feature amount of the background are 1) an image where the background is stationary, 2) there is a zoom, and 3) information that there is a pan. This feature may be input by the user, or a parameter obtained from the camera may be input as the feature. Background feature values include a background motion vector, accuracy of a background motion corrected image, background luminance distribution, texture, and edges. For example, the reference frame selection method can be controlled using the accuracy of the background motion corrected image as the feature amount of the difference between the background motion corrected image and the image before correction. As a control example, when the difference average is very large, the frame may not be selected as a reference frame candidate, or a frame may be selected by lowering the selection order of the frame. If the background is stationary or if the background motion correction is complete for all frames, the difference will be zero. As the reference frame selection method, the same method as in the second embodiment can be used.

The first object tracking / extracting unit 135 receives the extracted frame 1301, the reference frame 1306, and the extraction result 1308 of the already extracted frame as inputs, and
The tracking / extraction result 1307 of the extracted frame is output to the memory 135 by the ND method. The memory 135 inputs and holds the tracking / extraction result 1307 of the extracted frame.

Next, referring to FIG. 22, an example in which the feature amount is obtained from the output from the second object tracking / extracting unit and the plurality of reference frame selecting units are switched by using the output from the output from the second object tracking / extracting unit This will be described as a fourth configuration example.

The figure setting section 160 is provided with the extraction frame 160
1, an initial figure 1602 set by the user, and a frame 1608 from which an object has already been extracted,
603 is output. The second object tracking / extracting unit 161
It is used to extract an object region by shape prediction such as block matching method or affine transformation.
603 and a frame 1608 from which an object has already been extracted are input, and an object tracking / extraction result 1604 is output. The feature amount detection unit 163 receives the object tracking / extraction result 1604 as an input and outputs the feature amount 1605 to the switch unit SW164. The switch unit SW 164 receives the feature amount 1605 and outputs the object tracking / extraction result 160 to the reference frame selection unit.
4 is controlled.

The plurality of reference frame selection units 165 receive an object tracking / extraction result 1604 and a frame 1608 from which an object has already been extracted, and output at least two reference frames 1606.

The first object tracking / extracting section 166
It is used to extract an object by the ND method. The reference frame 1696 and the extracted frame 1601 are input, and the tracking / extraction result 1607 of the object is output to the memory 167. The memory 167 receives an object tracking / extraction result 1607 as an input and holds the result.

Next, with reference to the block diagram 23, an example in which the background information is obtained and the input of a plurality of reference frame selection units is controlled by an error in the background motion correction will be described.

[0179] The graphic setting section 170
1, an initial graphic 1702 set by the user, and a frame 1710 in which an object has already been extracted,
703 is output. The second object tracking / extracting unit 171
Setting graphic 1703 and frame 1 from which an object has already been extracted
710 is input and an object tracking / extraction result 1704 is output. The switch unit SW172 inputs background information 1705 specified by the user and controls the input of the extracted frame 1701 to the background motion correction unit 173.

[0180] The background motion compensating unit 173 extracts the extracted frame 1
701 and a frame 1710 from which an object has already been extracted are input, and a frame 1706 in which background motion has been corrected is output.

The background feature detection unit 174 receives as input the extracted frame 1701 and the frame 1706 in which the background motion has been corrected, and outputs the background feature 1707 to the switch SW175.
Output to The switch SW 175 receives the feature amount 1707 of the background, and controls the input of the tracking / extraction result 1704 of the object to the reference frame selection unit 176. The reference frame selection unit 176 obtains an object tracking / extraction result 1704.
And input the frame 1710 from which the object has already been extracted,
Output at least two reference frames 1708.

The first object tracking / extracting section 177 includes at least two reference frames 1708 and an extracted frame 17
01 is input, and an object tracking / extraction result 1709 is output to the memory 178. The memory 178 receives and holds the tracking / extraction result 1709 of the object.

Next, a fifth configuration example of the object tracking / extracting apparatus according to the third embodiment will be described with reference to the block diagram of FIG.

The extraction frame output control unit 140
401 and the order 1405 of frames to be extracted,
The extracted frame 1402 is output. The frame order control unit 141 stores information 1 on the frame order given by the user.
405 is input, and a frame order 1406 is output.
The object tracking / extracting device 142 is an object tracking / extracting method and device for extracting / tracking a target object from a moving image signal. The object tracking / extracting device 142 receives an extraction frame 1402 as an input, and outputs a tracking / extraction result 1403 as a tracking / extraction result output. Output to the control unit 143. The tracking / extraction result output control unit 143 receives the tracking / extraction result 1403 and the frame order 1406 as inputs,
The frame order is rearranged to the order of the image 1401 and output.

The order of the frames may be given by the user or may be determined adaptively according to the motion of the object. A frame interval at which the motion of the object is easily detected is determined, and the object is extracted. That is, the frame order is controlled so that the object extraction processing is performed in an order different from the input frame order so that the frame interval between the reference frame and the current frame as the object extraction target is at least two frames or more. . Thereby, compared with the case where the shape prediction by the inter-frame prediction or the ORAND operation is performed in the input frame order,
The prediction accuracy can be improved, and as a result, the extraction accuracy can be improved. In the case of the ORAND method, since the extraction accuracy can be improved by selecting an appropriate reference frame, the shape prediction method based on inter-frame prediction by block matching or the like is particularly effective.

That is, depending on the frame interval, the motion may be too small or too complicated, and the shape prediction method based on inter-frame prediction may not be able to cope with it. Therefore, for example, if the error of the shape prediction does not fall below the threshold, by spacing the extracted frame used for prediction,
The prediction accuracy increases, and as a result, the extraction accuracy improves. If there is motion in the background, the reference frame candidate calculates and compensates for the background motion with the extracted frame, but the background motion is too small or too complicated depending on the frame interval, and the background motion compensation is performed with high accuracy. It may not be possible. Also in this case, the motion compensation accuracy can be increased by providing a frame interval. By adaptively controlling the order of the extraction frames in this way, it is possible to more reliably extract the shape of the object.

Next, a sixth example of the object tracking / extracting apparatus according to the third embodiment will be described with reference to the block diagram of FIG.

The extraction frame output control unit 150
501 and the order 1505 of frames to be extracted are input,
The extracted frame 1502 is output. The frame order control unit 151 stores information 1 on the frame order given by the user.
505 is input, and a frame order 1506 is output.
That is, the frame order control unit 151 is given a frame interval and determines the frame extraction order. The plurality of object tracking / extracting devices 152 are object tracking / extracting methods and devices for extracting / tracking a target object from a moving image signal. The input of the extracted frame 1502 is controlled according to the frame order 1506, and the tracking / extraction is performed. Extraction result 1503
Is output. The tracking / extraction result output control unit 153 receives the tracking / extraction result 1503 and the frame order 1506 as inputs, and rearranges them in the order of the image 1501 and outputs them.

The skipped frames may be interpolated from already extracted frames, or may be extracted by the same algorithm by changing the method of selecting reference frame candidates.

Here, an example of the processing of the object tracking / extracting apparatus of FIG. 21 will be described with reference to FIG.

In FIG. 25, the frame indicated by oblique lines is a frame to be extracted first with an interval of two frames. The skipped frame is extracted by the second object tracking / extracting device. After the frames on both sides are extracted as shown in FIG. 25,
The object shape may be obtained by interpolation from the extraction results of the frames on both sides. Alternatively, parameters such as a threshold value may be changed, or the frames on both sides may be added to the reference frame candidates and extracted in the same manner as the frames on both sides.

Next, another configuration example of the object tracking / extracting apparatus will be described with reference to the block diagram of FIG.

The switch SW182 receives the background information 1805 specified by the user as an input, and
3 is controlled. The background motion correction unit 183 receives the extracted frame 1801 and the frame 1811 from which an object has been extracted as an input, and outputs a frame 1806 in which the background motion has been corrected. The background feature detection unit 184 receives the extracted frame 1801 and the frame 1806 in which the background motion has been corrected, and receives the background feature 180
7 is output. The switch unit SW187 receives the feature amount 1807 of the background and controls the input of the tracking / extraction result 1804 of the object to the reference frame selection unit 188. The figure setting unit 180 receives the extracted frame 1801, the frame 1811 in which the object has already been extracted, and the initial figure 1802 set by the user, and sets the extracted frame 1 in which the figure is set.
803 is output. The second object tracking / extracting unit 185
An extraction frame 1803 in which a figure is set and a frame 1811 in which an object has already been extracted are input, and an object tracking / extraction result 1804 is output. The feature detection unit 185 receives the object tracking / extraction result 1804 as an input and outputs a feature 1808. The switch unit SW186 receives the feature amount 1808 as an input, and outputs the object tracking / extraction result 1 to the reference frame selection unit.
804 input is controlled. Reference frame selection unit 188
Receives an object tracking / extraction result 1804 and a frame 1811 from which an object has already been extracted, and outputs at least two reference frames 1809.

The first object tracking / extracting section 189 comprises at least two reference frames 1809 and an extracted frame 18
01 is input and an object tracking / extraction result 1810 is output to the memory 190. The memory 190 holds an object tracking / extraction result 1810.

The processing flow is as follows.

The user roughly encloses the object to be extracted in the initial frame. The rectangles of the subsequent frames are set by expanding the rectangle surrounding the already extracted object by several pixels in the vertical and horizontal directions. This rectangle is divided into blocks, the extracted frames are matched, and the shape of the extracted object is attached to the corresponding positions. The object shape (predicted object shape) obtained by this processing represents a rough object. If the accuracy of the prediction does not fall below a certain threshold value, a process may be performed to correct the prediction from another frame to further increase the prediction accuracy.

When the prediction accuracy is good, the whole or a part of the predicted shape is output as an extraction result as it is. This method can extract an object while tracking the object.

Blocking performed at the time of object tracking and object shape prediction may be performed by dividing a rectangle by a fixed block size, or by performing hierarchical block matching using a matching threshold. The frame may be divided by a fixed size, and only the block including the object may be used.

Considering the case where the prediction is bad, the object prediction shape is extended by several pixels, and irregularities and holes due to prediction errors are corrected. With this method, the predicted object shapes are set for all the reference frame candidates. Each time an object is extracted, the object tracking figure of the previous frame is re-determined using the object, thereby preventing an extraction error. This tracking figure is a tracking figure set to surround the object.

Hereinafter, it is assumed that an object has already been extracted for a frame before the extracted frame, and no object has been extracted for the previous frame.

The reference frame candidates are five frames before and after, which are temporally shifted at regular intervals from frames to be extracted at regular intervals. The reference frame candidate is, for example,
The frame is limited to a total of five frames including the initial frame, three frames before the current frame, and one frame after the current frame. However, if there are not three previous frames, the number of candidates for the subsequent frame is increased, and if there is no subsequent frame, four previous frames are used as candidates.

An appropriate area is set around the reference frame candidate object, and the background movement between this area and the current frame is detected, whereby the background in the figure of the reference frame is deleted. As a method of detecting background motion, matching is performed on the current frame in all regions except for the object, and a motion vector that minimizes a matching error is determined as background motion.

By removing reference frames having a large motion vector detection error at the time of background motion removal from candidates, it is possible to prevent an extraction error when background motion removal is not appropriate. When the number of reference frame candidates decreases, a new reference frame candidate may be selected. If the figure setting and the background motion deletion of the newly added reference frame candidate have not been performed, it is necessary to newly perform the figure setting and the background motion deletion.

If it is known in advance that there is no background movement, this processing is not performed.

A temporary reference frame is set from the reference frame candidates, and a set of reference frames satisfying Expression (1) or Expression (2) of the second embodiment is selected for the combination of these frames. If none of the combinations of all the reference frame candidates satisfy Expression (1) or Expression (2), O _{i ∩} O
_It is better to select a frame with the minimum number of pixels in _j .

Further, it is better to consider the combination of reference frame candidates so that a frame having a motion vector detection error at the time of background motion deletion is as small as possible. In particular,
When there is a reference frame set having the same condition according to the expression (1) or the expression (2), there is a method of selecting a frame having a smaller motion vector detection error when deleting background motion. If there is no background movement, a frame from which a difference between frames can be sufficiently detected can be preferentially selected.

When the object prediction accuracy is high and a part of the object is output as it is, the frame satisfying the condition of the expression (1) or (2) is targeted only for the region where the object prediction result is not the extraction result. Choose

The processing when two reference frames are selected will be described below.

When a reference frame is selected, an inter-frame difference from the extracted frame is obtained, and attention is paid to the inter-frame difference in the set figure.

The difference between frames is binarized using the set threshold value. The threshold value used for binarization may be constant for an image, or may be changed for each frame according to the accuracy of background motion compensation. As an example of control, if the accuracy of background motion compensation is poor, there are many extra differences in the background. Further, it may be changed according to a partial luminance gradient, texture, or edge strength of the object. As an example of this control, a relatively flat region such as a region having a small luminance gradient or a region having a small edge intensity has a small threshold value for binarization. Further, the user may give a threshold from the property of the object.

[0211] With respect to the pixels outside the object tracking figure, the pixel having the difference value of the adjacent background area is determined as the background pixel. At the same time, with respect to the pixels inside the object tracking graphic, it is determined that the pixels having no difference value between the adjacent background areas are not the background pixels.

An inter-frame difference cannot be detected in a stationary area of an object. Therefore, if the difference between frames from the frame used for prediction is zero, and the frame used for prediction is a pixel inside the object, it is not added to the background pixel as a still area pixel.

This background pixel is a common background area for the current frame and one reference frame. At this time, since the boundary between the background region and other portions may be unnatural due to the influence of noise, a filter for smoothing the boundary or a filter for removing an extra noise region may be applied to the image signal. .

When a common background area is obtained for each reference frame, an area that is not included in the two common background areas is detected and extracted as an object area. An extraction result is output for a portion that does not use the shape of the object predicted earlier, and the shape of the entire object is extracted. If the part using the shape obtained from the common background and the part using the object shape predicted earlier cannot be matched, the output result obtained by applying the filter at the end can be made good for viewing.

[0215] Finally, the extracted object area is output by replacing the extraction order with the input frame order.

A method and apparatus for extracting the shape of an object according to the present invention are MPEG-4, which is currently being standardized.
Can be used as input means for object encoding. As an application example of the MPEG-4 and the object extraction, there is a display system in which the object shape is a window format. Such a display system is effective for a multipoint conference system. Rather than displaying text material and a person participating in a meeting at each point in a rectangular window on a display having a limited size, the person can be displayed in the shape of a person as shown in FIG. 26, thereby saving space. MPEG-4
By using the function, only the person who is speaking can be enlarged, or the person who is not speaking can be made translucent, and the usability of the system is improved.

As described above, according to the third embodiment, by selecting an object by a method and an apparatus according to the properties of an image, unnecessary processing can be omitted and stable extraction accuracy can be obtained. Further, by removing the constraint of the time order, sufficient extraction accuracy can be obtained regardless of the motion of the object.

The third embodiment is to improve the performance of the first and second embodiments, and has been described in the first and second embodiments and the third embodiment. It can be used in combination with the configuration as appropriate.

FIG. 27 shows a first configuration example of an object extraction device according to the fourth embodiment of the present invention.

[0220] The texture image 221 input to the object extracting apparatus after being picked up by an external camera or read from a storage medium such as a video tape or a video disk is stored in the recording apparatus 222, the switch section 223, The signal is input to the compensation-based object extraction circuit 224. Recording device 2
Reference numeral 22 holds the input texture image 221. For example, a hard disk or a magneto-optical disk used in a personal computer or the like is used. Recording device 22
2 is necessary for using the texture image 221 again later. If the texture image 221 is an image recorded on an external storage medium, there is no need to prepare a separate recording device 222, and the storage medium is used for recording. Used as device 222. In this case, there is no need to input the texture image 221 to the recording device 222 again. The texture image is obtained by, for example, arranging pixels in which the luminance (Y) of each pixel is represented by a value of 0 to 255 in a raster order (from the upper left pixel of the image to the right, from the upper line to the lower line). And is generally called an image signal. In order to distinguish it from a shape image described later, it is referred to as a texture image here. As the texture image, in addition to the luminance, a color difference (U, V, etc.) and a color (R, G, B, etc.) may be used.

On the other hand, in the first frame, a shape image 22 from which the operator has separately extracted an object to be extracted.
5 is input to the motion-compensated object extraction circuit 224. The shape image is generated, for example, by arranging pixels in which the pixel values of the pixels belonging to the object are represented by “255” and the pixel values of the other pixels by “0” in the raster order in the same manner as the texture image.

Here, the shape image 2 of the first frame
An example for generating 5 will be described in detail with reference to FIG.

Although not shown in FIG. 34, the background and the foreground also have a pattern, of which a house-shaped object 226 is formed.
Suppose you want to extract The operator traces the outline of the object 226 with the mouse or the pen on the image 227 displayed on the monitor. An image obtained by substituting “255” for a pixel inside the contour and “0” for a pixel outside the contour is defined as a shape image. If the operator draws the outline with great care, the accuracy of the shape image will be high. However, even when the accuracy is somewhat low, the accuracy can be increased by using the following method.

FIG. 35 shows a line 2 drawn by the operator.
28 and the outline 229 of the object 226 are shown. At this stage, the correct position of the outline 229 has not been extracted, of course, but the outline 229 is shown to indicate the positional relationship with the line 228.

First, a block is set so as to include the outline 228. More specifically, the screen is scanned in raster order, and when there is an outline 228, that is, when there is a difference between adjacent pixel values in the shape image of the outline 228, a predetermined value is set around that pixel. Provide a block of size. At this time, if the block already set and the current block overlap, the current block setting is not performed, and if the scan is advanced, the blocks are set so that they do not overlap each other as shown in FIG. it can. However, this alone is not sufficient for parts 230, 231, 23
2 is not in the block. Therefore, scanning is performed again, and when there is a contour line not included in the block, a block is provided around the pixel. However, at the time of the second scan, even if there is a portion where the current block overlaps with the already set block, the current block is set unless the center pixel is included in the already set block. In FIG. 37, hatched blocks 233, 234, 235, and 236 are blocks set in the second scan. The block size may be fixed, but is large when the number of pixels surrounded by the contour 228 is large, small when the number of pixels is small, large when the contour 228 has few irregularities, It may be set to be small when there are many, or large when the design of the image is flat, and small when the design is fine.

At the edge of the screen, if a block is set normally, it may run off the screen. In such a case, the end of the block is cut into a rectangular block so that only the block does not protrude from the screen. In this case, the similar blocks are also rectangular.

The above is the method of setting blocks in a shape image.

Next, for each block, a similar block is searched using a texture image. Here, “similar” means that, when blocks having different block sizes are enlarged or reduced so that one block size is the same as the other, the pixel values of the corresponding pixels are substantially equal. For example, with respect to the block 237 in FIG. 38, the block 238 has a similar texture image pattern. Similarly, for block 239, block 2
40, the block 241 is the block 242,
It is similar. In the present embodiment, the similar block is made larger than the block set on the contour line. The similar block does not search the entire screen, but, for example, as shown in FIG.
It is sufficient to search within a certain range with four corners at 4,245,246,247. FIG. 39 shows a case where the starting point of the block 243 is used to move the starting points of the blocks 244, 245, 246, and 247 in the vertical and horizontal directions by a predetermined pixel width using the starting point of each block.
FIG. 40 shows a case where the starting point is located at the upper left corner of the block.

[0229] Even within the search range, similar blocks partially protruding from the screen are excluded from the search target. However, if the block is at the edge of the screen, all similar blocks within the search range are excluded from the search target. Sometimes. In such a case, for the block at the end of the screen, the search range is shifted to the inside of the screen.

When searching for similar blocks, a multi-step search can reduce the amount of calculation. The multi-step search is, for example, 1
Instead of searching the entire search range while shifting the starting point by a pixel or a half pixel, first, an error is checked at the starting point of the discrete position. Next, a method of narrowing down the position of the similar block by repeatedly moving the starting point slightly around the starting point where the error is small and checking the error is repeated.

In the search for similar blocks, if processing for reducing similar blocks is performed every time, a long processing time is required. Therefore, if a reduced image of the entire image is generated in advance and stored in another memory, it is only necessary to read out the data corresponding to the similar block from the memory.

In FIG. 38, three blocks 237, 23
Similar blocks are shown only for 9, 241. However, similar blocks are actually obtained for all the blocks shown in FIG. The above is the similar block search method. Searching for similar blocks is not a shape image,
It is important to use a texture image. When considering a primary transformation that maps similar blocks to blocks in a screen, the contour line of the texture image is invariant in the primary transformation.

Next, a method of correcting the contour of the shape image so as to match the contour of the texture image using the positional relationship between each block and its similar blocks will be described.

In FIG. 41, the outline 228 is a line drawn by the operator. This line is the correct contour line 22
It should be close to 9. For this purpose, the similarity block 238 of the shape image is read out, and is read out of the block 237.
And the block size 237 of the shape image is replaced. This operation has the property of bringing the contour line closer to an invariant set including the fixed point of the linear transformation from the similar block to the block.
Approaches the contour 229. When one side of the similar block is twice as long as one side of the block, the gap between the contour 228 and the correct contour 229 is generally １ ／ by one replacement.
become. The result of performing this replacement once for all blocks is the contour 248 in FIG. If this block replacement is repeated, the outline 248 comes closer to the correct outline, and eventually coincides with the correct outline as shown in FIG. In practice, it is meaningless that the displacement between the two contour lines is smaller than the pixel-to-pixel distance. This method is effective when a block of N × N pixels set in the shape image includes a contour line in the texture image. In this case, the distance between the contour line of the shape image and the contour line of the texture image is maximum. It is approximately N / 2. When the length of one side of the similar block is A times the length of one side of the block, the distance between two contour lines is 1 / A for each replacement. When the number of replacements is x, (N / 2) × (1 / A) ＾ x <1. Here, ＾ represents a power, and in the above equation, (1 / A)
Is multiplied x times. From the above equation, x> log (2 / N) / log (1 / A). For example, when N = 8 and A = 2, x> 2, and three replacements are sufficient.

FIG. 30 is a block diagram of the object extracting apparatus. First, the shape image 2 input by the operator
49 are recorded in the shape memory 250. Blocks are set in the shape memory 250 as described above with reference to FIGS. On the other hand, texture image 2
51 is recorded in the texture memory 252. The texture memory 252 sends a block texture image 254 to the search circuit 255 with reference to the block position information 253 sent from the shape memory 250. At the same time, similar block candidates are also sent from the texture memory 252 to the search circuit 255 as described with reference to FIGS. The search circuit 255 calculates the error with respect to the block after reducing each similar block candidate, and determines the one with the minimum error as the similar block. The error may be a sum of absolute values of differences between luminance values or a sum of absolute values of differences of color differences. Compared to the luminance alone, using the color difference also increases the amount of computation, but even if the luminance step at the contour of the object is small,
Since the similar block can be correctly determined when the step of the color difference is large, the accuracy is improved. Information on similar block position 2
56 is sent to the reduction conversion circuit 257. Reduction conversion circuit 2
57, a shape image 258 of a similar block is also sent from the shape memory 250. In the reduction conversion circuit 257, the shape image of the similar block is reduced, and the reduced similar block is returned to the shape memory 250 as a shape image 259 whose contour is corrected, and the shape image of the corresponding block is overwritten. . When the replacement of the shape memory 250 reaches a predetermined number,
The corrected shape image 259 is output to the outside. The rewriting of the shape memory 250 may be sequentially overwritten for each block. Alternatively, two screens of memory are prepared, and the shape image of the entire screen is first copied from one to the other. The block may be replaced with a reduced one.

This object extraction method will be described with reference to the flowchart in FIG.

(Method of Extracting Object Using Reduced Block Matching in Frame) In step S31, a block is set at the outline of the shape data. Step S32
In, a similar block in which the design of the current processing block is similar to that of the image data is found from the same image data. In step S33, the shape data of the current processing block is replaced with data obtained by reducing the shape data of the similar block.

If the number of processed blocks has reached the predetermined number in step S34, the flow advances to step S35. Otherwise, the processing target advances to the next block and the flow returns to step S32.

In step S35, if the number of repetitions of replacement reaches a predetermined number, the flow advances to step S36. Otherwise, the flow returns to step S31 with the replaced shape data as a processing target. In step S36, the shape data subjected to repeated replacement is output as an object area.

This method is effective when the edge of a block matches the edge of a similar block. Therefore, if a block has a plurality of edges, the edges may not match correctly. Therefore, for such a block, the edge that has been input is retained without being replaced. Specifically, the shape image of the block is scanned in each line in the horizontal direction and the vertical direction, and two points at which one line changes from "0" to "255" or from "255" to "0". A block having a predetermined number of lines or more is not replaced. Further, even at the boundary between the object and the background, the luminance may be flat depending on the portion. Even in such a case, since the effect of the edge correction cannot be expected, the edge as input is held without replacing the block in which the variance of the texture image is equal to or less than the predetermined value.

If the error of the similar block does not become smaller than a predetermined value, the similar block may be obtained with the same size by giving up the reduction. At this time, choose similar blocks so as to avoid overlapping with yourself. The block that is not reduced alone does not have the effect of correcting the edge.However, by copying the corrected edge from the block whose edge has been corrected by performing the reduction, the block that is not reduced is also Edges are indirectly corrected.

The flowchart shown in FIG. 48 is an example in which a similar image is replaced immediately after a similar block is found. However, by retaining the position information of similar blocks of all blocks, the similarity is firstly obtained. A method of performing block search for all blocks and then replacing the shape image for all blocks will be described with reference to the flowchart of FIG.

In this example, the replacement of the shape image can be repeated a plurality of times for one similar block search.

In step S41, a block is set at the outline of the shape data. In step S42,
A similar block in which the pattern of the current processing block is similar to that of the image data is found in the same image data. In step S43, when the process of finding similar blocks has been completed for all blocks, that is, when the number of processed blocks has reached a predetermined number, the process proceeds to step S44.
If not, the process returns to step S42. Step S4
In step 4, the shape data of the current processing block is replaced with a reduced version of the shape data of the similar block.

In step S45, when the process of replacing all blocks is completed, that is, when the number of processed blocks is different from the predetermined number, step S46 is executed.
Proceed to. If not, the process returns to step S44. In step S46, if the number of replacements of all blocks has reached the predetermined number, the process proceeds to S47. Otherwise, S44
Return to In step S47, the shape data on which the replacement conversion has been repeated is output as an object region.

Next, a description will be given of a method of setting a block capable of improving the accuracy of edge correction.

As described above, in the method of setting a block around the contour of the shaved image, as shown in FIG. 51A, a part of the correct contour 301 may not be included in the block. Here, the outline 302 of the shape image is indicated by a thick line. Assuming that the lower right side of the contour is an object and the upper left side is a background, a part 303 that is actually a background is erroneously set as an object,
It cannot be modified because it is not included in the block.
If there is a gap between the block and the correct contour line, the correction is not performed correctly.

In order to reduce the gap between the block and the correct outline, there is a method of overlapping the blocks to some extent as shown in FIG. In this case, the number of blocks increases, so that the calculation amount increases, but the gap 304 becomes smaller. Therefore, the accuracy of extraction is improved. However, in this example, the gap is not completely eliminated yet.

To reduce the gap, it is effective to increase the block size as shown in FIG. In this example, the above-described superposition of blocks is used together. This completely eliminates the gap in this example.

As described above, it is effective to increase the block size in order to expand the range in which the contour can be corrected. However, if the block size is too large, the shape of the contour line included in the block becomes complicated, and it becomes difficult to find similar blocks. An example is shown in FIG.

In FIG. 52A, a hatched portion 305 represents an object region, and a white portion 306 represents a background region. The contour line 307 of the given shape image is indicated by a black line.
As described above, the contour 307 of the shape image is largely separated from the correct contour, and the correct contour has irregularities. On the other hand, the result of arranging blocks by a method different from the method described above is shown in FIG. Here, first, the image is divided into rectangular blocks that do not overlap each other and have no gap. The variance in the texture image is calculated for each block, and the setting is canceled for a block whose variance is smaller than a predetermined value. Therefore, in FIG. 52 (b), only the blocks whose variance is equal to or more than the predetermined value remain. A similar block is obtained for each of these blocks. For example, there is no pattern in which this is doubled vertically and horizontally near the block 308, and the same applies to many other blocks. Therefore, although the portion having the smallest error is selected as a similar block, even if the replacement transformation of the sheave image is repeated using the positional relationship, it is possible to obtain the data shown in FIG.
Does not match the correct contour as shown in FIG. However, FIG.
Compared with the contour 307 of the shape image of FIG. 52A, the contour 309 of the shape image of FIG.
Reflects the rough unevenness of the contour line of the texture image (a level having a mountain on the left and right and a valley between them). If the block size is reduced in this example, even this rough correction is not performed.

As described above, if the block size is increased in order to widen the range of correction, the shape of the outline included in the block becomes complicated, and it may be difficult to find similar blocks. As a result, the edge is only roughly corrected. In such a case, if edge correction is performed with a large block size at first, and the result is subjected to edge correction with the block size being reduced again, the accuracy of correction is improved. 52 (c), the block size is reduced to 縦 in height and width, and the correction is performed again.
It is shown in (d). As described above, the accuracy of the correction can be improved by repeating the correction while gradually reducing the block size.

A method for gradually reducing the block size will be described with reference to the flowchart in FIG.

In the step S51, the block size b = A
Set as In step S52, FIG.
Edge correction similar to the edge correction shown in FIG. In step S53, b is observed, and when b becomes smaller than Z (<A), this processing ends. If b is greater than or equal to z, the process proceeds to step S54. In step S54, the block size b is halved, and the process proceeds to S52.

As described above, an example has been described in which the correction accuracy is improved by increasing the block size at first and repeating the correction while gradually reducing the block size.

FIG. 54 (a) shows an example in which a block is inclined by 45 degrees to make it difficult to form a gap between the block and a correct contour line. As described above, when the contour is oblique, the correct contour can be covered by inclining the block without increasing the block size as in FIG. 51C. Further, in this example, even if the blocks do not overlap, the correct outline can be covered as shown in FIG. In this manner, by inclining the sides of the block in the same direction as the outline of the shape image, it is possible to make it difficult to generate a gap between the block and the correct outline. Specifically, the inclination of the contour line of the alpha image is detected, and if it is horizontal or nearly vertical, the block orientation is as shown in FIG. 51 (c). Otherwise, the block inclination is as shown in FIG. b). The determination that it is close to horizontal or vertical is made by comparing with a threshold value.

The above is the object extraction processing of the first frame. This is a technique that can be used not only for the first frame of a moving image but also for still images in general. In addition, a block is set again for the shape image that has been replaced once, a similar block is obtained again, and the second replacement is performed. If this is performed, the amount of calculation increases, but the effect of correction can be obtained more.

It is preferable that the similar block is selected from a portion as close as possible to the block. Therefore, the range in which the similar block is searched may be switched according to the block size. That is, when the block size is large, the range for searching for similar blocks is widened, and when the block size is small, the range for searching for similar blocks is narrowed.

In the present method, small holes or isolated small areas may appear as errors in the shape data replacement process. Therefore, steps S34, S35, S36, S45, S46, S4
7, before S53, etc., the accuracy of correction can be improved by removing small holes and isolated small areas from the shape data. To remove small holes and isolated small areas, for example, an image analysis handbook (supervised by Takagi and Shimoda, University of Tokyo Press, First Edition, January 1991) 575-5
Processing combining the expansion and contraction described on page 76, a majority filter described on page 677, and the like are used.

Also, the blocks may be set more simply as shown in FIG. That is, the screen is simply divided into blocks, and similar blocks are searched or replaced only for blocks including the outline 228, such as the block 2200.

If the given texture image has been compressed in advance by fractal coding (Japanese Patent Publication No. 08-329255, “Method and Apparatus for Image Region Division”), the compressed data is similar to the compressed data of each block. Contains block information. Therefore, if compressed data is used as a similar block of the block including the outline 228, there is no need to search for a similar block again.

Returning to FIG. 27, description of the object extracting apparatus for extracting an object from a moving image will be continued.

In the object extraction circuit 242 using motion compensation,
Using the motion vector detected from the texture image 221, based on the shape image 25 of the first frame, the shape image 260 of the second and subsequent frames.
Generate

FIG. 29 shows an object extraction circuit 22 using motion compensation.
Four examples are shown. Shape image 22 of the first frame
5 is recorded in the shape memory 261. In the shape memory 261, blocks are set on the entire screen as shown in a frame 262 in FIG. On the other hand, the texture image 221 is sent to the motion estimation circuit 264 and is recorded in the texture memory 263. From the texture memory 263, the texture image 2 one frame before
65 is sent to the motion estimation circuit 264. Motion estimation circuit 2
In 64, a reference block with the minimum error is found from within the frame one frame before for each block of the current processing frame. FIG. 45 shows an example of a block 267 and a reference block 268 selected from the frame 266 one frame before. Here, if the error is smaller than a predetermined threshold, the reference block is made larger than the block. FIG. 45 also shows an example of a block 269 and a reference block 70 having a size twice as large in length and width.

Returning to FIG. 29, the information 271 on the position of the reference block is sent to the motion compensation circuit 272. The shape image 273 of the reference block is also sent from the shape memory 261 to the motion compensation circuit 272. Motion compensation circuit 272
So, if the size of the reference block is the same as the block,
If the size of the reference block is larger than the block, the shape image of the reference block is reduced, and the shape image of the reference block is output as the shape image 260 of the current processing frame. In addition, in preparation for the next frame, the shape image 260 of the current processing frame is sent to the shape memory 261 to overwrite the shape image of the entire screen.

In the case where the reference block is larger than the block, there is an effect of correcting when the outline is shifted from the correct position, as described with reference to FIGS. Therefore, in every frame of the video sequence following the given first frame shape image,
Objects are extracted with high accuracy. Unlike the conventional method, there is no problem that accuracy is poor at the beginning of the moving image sequence or when the motion of the object is small.

Object extraction by motion compensation between frames will be described with reference to the flowchart in FIG.

At step S21, the current processing frame is divided into blocks. In step S22, a reference block having a similar pattern between the current processing block and the image data and larger than the current processing block is found from each frame or from within the frame for which the shape data has already been obtained. In step S23, the shape data of the reference block is cut out and the reduced sub-block is pasted on the current processing block.

In step S24, when the number of processed blocks reaches a predetermined number, the flow advances to step 25. Otherwise, the processing target advances to the next block and the flow returns to step S22. In step S25, the combined shape data is output as an object area.

Here, each frame is the first frame in this embodiment, and is a frame to which a shape image is given in advance. The reference block is not necessarily 1
Even if it is not the frame before the frame, as described here, it is sufficient that the frame has a shape image already obtained.

The above is the description of the object extraction using the motion compensation. As the object extraction circuit 224, in addition to the above-described method, Japanese Patent Application Laid-Open No.
There is also a method using an inter-frame difference image in the “moving image object tracking / extracting device”.

Returning to FIG. 27, description of the embodiment of the object extracting apparatus for extracting an object from a moving image will be continued.

The shape image 260 is sent to the switch section 223 and the switch section 281. In the switch unit 223,
When the shape image 260 is “0” (background), the texture image 221 is sent to the background memory 274 and recorded. When the shape image 260 is “255” (object), the texture image 221 is not sent to the background memory 274. This is performed for some frames, and if the shape image 260 is accurate to some extent, an image of only the background portion that does not include the object is generated in the background memory 274.

Next, the texture image 275 is read again from the recording device 222 in order from the first frame, or only the frame from which the operator specifies an object to be extracted is read and input to the difference circuit 276. at the same time,
The background image 277 is read from the background memory 274,
The signal is input to the difference circuit 276. The difference circuit 276 calculates a difference value 278 between pixels of the texture image 275 and the background image 277 which are located at the same position in the screen.
It is input to an object extraction circuit 279 using a background image. In the object extraction circuit 279, a shape image 280 is generated. This is because a pixel whose absolute value of the difference value 278 is larger than a predetermined threshold value is regarded as belonging to the object and the pixel value is set to “255”. Other pixels are generated by setting the pixel value to “0” as belonging to the background. When not only luminance but also color difference or color is used as the texture image, the sum of the absolute values of the differences between the signals is compared with a threshold to determine whether the object or the background. Alternatively, a threshold value is separately set for each of the luminance and the color difference, and if the absolute value of the difference is larger than the threshold value in any of the luminance and the color difference, the object is determined, and if not, the background is determined. The shape image 280 generated in this way is
Sent to 1. In addition, a selection signal 282 determined by the operator is externally input to the switch unit 281, and one of the shape image 260 and the shape image 280 is selected by the selection signal 282 and output to the outside as the shape image 283. Is done. The operator displays each of the shape image 260 and the shape image 280 on a display or the like, and selects the correct one. Alternatively, when the shape image 260 is generated and displayed, and the accuracy is not satisfactory, the shape image 28
0, and if the accuracy of the shape image 260 is satisfactory, the shape image 280 is not generated,
If the shape image 260 is output to the outside as the shape image 283, processing time can be saved. The selection may be performed for each frame or for each moving image sequence.

An object extraction method corresponding to the object extraction device of FIG. 27 will be described with reference to the flowchart of FIG.

(Object Extraction Method Using Background Image) In step S11, the shape data of each frame is generated by motion-compensating the shape data of the given frame. In step S12, the image data of the background area determined by the shape data is stored in the memory as a background image.

In step S13, when the number of processed frames reaches a predetermined number, the flow advances to step S14. Otherwise, the processing target advances to the next frame and the flow returns to step S11. In step S14, a pixel having a large absolute value of a difference between the image data and the background image is set as an object region, and a pixel other than that is set as a background region.

In the present embodiment, for example, the background moves when the camera that captures the image moves. In this case, the motion of the entire background from the previous frame (global motion vector)
And in the first scan, it is shifted from the previous frame by the amount of the global motion vector and recorded in the background memory.
In the second scan, a portion shifted from the previous frame by the amount of the global motion vector is read from the background memory. If the global motion vector detected in the first scan is recorded in a memory, and read out and used in the second scan, the time for obtaining the global motion vector can be saved. If it is known that the background is stationary because the camera is fixed, the operator does not detect the global motion vector, for example, by switching a switch. If is always set to zero, processing time can be further saved. When obtaining a global motion vector with half-pixel accuracy, the background memory has a pixel density twice as high in both the vertical and horizontal directions of the input image. That is, the pixel values of the input image are written to the background memory every other pixel. For example, in the next frame, if the background moves by 0.5 pixel in the horizontal direction, the pixel value is written every other pixel between the previously written pixels. In this case, there may be a pixel which is never written to the background image at the time when the first scan is completed. In that case, the gap is filled by interpolation from surrounding written pixels.

[0279] Regardless of whether a half-pixel motion vector is used or not, a pixel value that is never used as a background area throughout the entire moving image sequence is not substituted into the background memory even after the first scan. Such an undefined portion is always determined as an object in the second scan.
This is because a pixel value (Y, U) that is expected to appear only rarely in the background is prepared without preparing a memory for recording an undefined portion and determining each undefined portion. ,
V) = (0,0,0) or the like, the first scan may be started after the background memory is initialized in advance. Since the initial pixel value remains in the undefined pixel, the pixel is automatically determined as an object in the second scan.

In the above description, when the background memory is generated, the pixel value to which the background pixel value has already been assigned is overwritten by the pixel value in the background area.
In this case, the pixel value of the background at the end of the moving image sequence is recorded in the background memory in the portion that is the background at the beginning or the end of the moving image sequence. If the background is exactly the same pixel value at the beginning and end of the video sequence, there is no problem, but the pixel value may be very small between frames due to the camera moving very slowly or the background brightness changing little by little. , The difference in pixel value between the first background and the last background of the moving image sequence becomes large. The part is also erroneously detected as an object. Therefore, in the previous frames, the background area is not written at all, and only the pixels that have become the background area for the first time in the current processing frame are written to the background memory, and the background pixel value has already been assigned. If the pixel is not overwritten, the background is recorded in the background memory at the beginning of the moving image sequence, so that the object can be correctly extracted. Also, in the second scan, if the background area of the current processing frame is overwritten in the background memory according to the object extraction result, the correlation between the background of the frame immediately before the current processing frame and the background of the current processing frame is obtained. Are compared with each other, and that portion is less likely to be erroneously detected as an object. The overwriting in the second scan is effective when there is a slight change in the background. Therefore, if the operator switches the switch so that the background does not move, the overwriting is not performed. This switch may be common to the switch that switches between performing and not performing the previous global motion vector.

Since the purpose of the first scan is to generate a background image, it is not always necessary to use all frames. Even if frames are thinned out every other frame, every other frame, etc., almost the same background image is obtained, and the processing time is shortened.

If only pixels having an inter-frame difference equal to or smaller than the threshold value in the background area are recorded in the background memory, other objects entering the screen do not need to be recorded in the background memory. Further, when the object area of the first scan is erroneously detected on the object side relative to the actual, the pixel value of the object is recorded in the background memory. Therefore, pixels in the background area that are close to the object area are not input to the background memory.

When only a background image excluding a person in the foreground and the like is required in an image photographed at a tourist spot or the like, the background image recorded in the background memory is output to the outside.

The above is the description of the first configuration example of the present embodiment. According to this example, a high extraction accuracy can be obtained in the first part of the moving image sequence as in the last part. Even if the movement of the object is small or does not move at all, it is correctly extracted.

Next, an example of correcting the generated shape image 280 will be described with reference to FIG. Shape image 2
Until 80 is generated, it is the same as FIG.

The shape image 280 is input to an edge correction circuit 284 using a background palette. The texture image 275 is input to an edge correction circuit 284 using a background palette and an edge correction circuit 285 using reduced block matching. A detailed block diagram of the edge correction circuit 284 is shown in FIG.

In FIG. 31, the shape image 280 is
The texture image 275 of the same frame input to the correction circuit 286 is input to the comparison circuit 287. The background color 289 is read from the memory 288 holding the background palette, and is input to the comparison circuit 287. Here, the background palette includes the luminance (Y) and the color difference (U,
V), that is, a set of vectors, is prepared in advance as (Y1, U1, V1) (Y2, U2, V2) (Y3, U3, V3)... Specifically, the background palette is a collection of sets of pixels Y, U, and V belonging to the background area in the first frame. Here, for example, if Y, U, and V take 256 values, the number of combinations becomes enormous, and the number of combinations of (Y, U, V) in the background also increases. Since the amount of calculation increases, Y,
By quantizing the values of U and V, the number of combinations can be suppressed. This is because, when quantization is not performed, different vector values may have the same vector value due to quantization.

In the comparison circuit 287, the texture image 27
5, the Y, U, and V of each pixel are quantized, and it is checked whether the vector matches one of the vectors registered in the background palette sequentially transmitted from the memory 288, that is, one of the background colors 289. For each pixel, a comparison result 290 as to whether the color of the pixel is the background color is obtained by the comparison circuit 2
87 to the correction circuit 286. In the correction circuit 286, the pixel value of a certain pixel of the shape image 280 is “25”.
If the comparison result 290 is a background color despite the fact that it is 5 "(object), the pixel value of that pixel is replaced with" 0 "(background) and output as a corrected shape image 291. By this processing, when the object region protrudes into the background region in the shape image 280 and is erroneously extracted, the background region can be correctly separated, but the background and the object have a common color, and the background palette has the color of the object. If it is also registered, the color part of the object is also determined as the background, so in the first frame, the palette described above is set as a temporary palette for the background,
Create a palette of objects for the first frame in the same way.
Next, the colors included in the palette of the object in the temporary palette of the background are excluded from the temporary palette of the background, and the remaining colors are used as the background palette. Thus, a problem that a part of the object becomes the background can be avoided.

In consideration of a case where there is an error in the shape image given in the first frame, pixels near the edge of the shape image may not be used for generating a palette. Alternatively, the frequency of appearance of each vector may be counted, and a vector whose frequency is equal to or less than a predetermined value may not be registered in the palette. If you make the quantization step size too small,
If the processing time is long, or even a color very similar to the background color is not determined as the background because the vector value is slightly different, conversely if the quantization step size is too large, it will be common to the background and the object It's just a vector. Therefore, several quantization step sizes are tried for the first frame, and a quantization step size at which a background color and an object color are separated like a given shape image is selected.

Also, since a new color may appear on the background or object in the middle, the background palette may be recreated with the middle frame.

Returning to FIG. 28, the shape image 291 is input to the edge correction circuit 285. Edge correction circuit 28
5 is a circuit in which the shape image 249 is changed to the shape image 291 and the texture image 251 in the circuit of FIG.
Is the same as the circuit for setting the texture image 275 to
Although the description is omitted, the shape image is corrected so that the edge of the shape image matches the edge of the texture image.
The corrected shape image 292 is sent to the switch unit 281. From the switch unit 281, the shape image 292
And the shape image 293 selected from the shape images 260 is output.

In this example, the edge correction circuit is provided after the object extraction circuit 279. However, if it is provided after the object extraction circuit 224, the accuracy of the shape image 260 can be improved.

There is also a rare case where the extraction accuracy is rather deteriorated by the edge correction. At that time, in order not to output the deteriorated shape image 292,
In FIG. 28, shape image 280 and shape image 2
If 91 is also input to the switch unit 281, it is also possible to select a shape image 280 without edge correction or a shape image 291 with only edge correction using the background palette.

FIG. 44 shows pixels of the background color registered in the background palette by cross hatching. When searching for similar blocks described earlier with reference to FIGS. 30 and 29, FIG.
By using the information of No. 4, the accuracy of contour extraction can be further improved. When there is a pattern in the background, a similar block may be selected not along the edge of the object and the background but along the edge of the background pattern. In such a case,
When calculating the error between the block and the block obtained by reducing the similar block, if any of the corresponding pixels have the same background color, if the error of the pixel is not included in the calculation, the edge of the background pattern is shifted. Also, no error occurs, and therefore the similar blocks are correctly selected such that the edges of the object and the background match.

FIG. 32 shows an object extracting device 29 of this embodiment.
4 is an example of an image synthesizing apparatus in which the image synthesizing apparatus 4 is incorporated. The texture image 295 is input to the switch unit 296 and the object extraction device 294, and the shape image 2100 of the first frame is input to the object extraction device 294. The object extraction device 294
27 and 28. A shape image 297 of each frame is generated and sent to the switch unit 296. On the other hand, the recording circuit 298 stores the background image 2 for synthesis in advance.
99 is held, and the background image 29 of the current processing frame is stored.
9 is read from the recording circuit 298 and the switch unit 296 is read.
Sent to In the switch unit 296, the texture image 2 is set for a pixel whose pixel value of the shape image is “255” (object).
95 is selected and output as the composite image 2101, and the background image 299 is selected and output as the composite image 2101 for the pixel whose pixel value of the shape image is “0” (background). As a result, an image in which the object in the texture image 295 is synthesized with the foreground of the background image 299 is generated.

FIG. 33 shows another example of performing edge correction.
It is assumed that one of the blocks set as shown in FIG. 33 is a block 2102 in FIG. The block is divided into an object area and a background area with the outline as a boundary. Blocks 2103, 2104, 2105 and 2106 are obtained by shifting the outline in the left-right direction. The shifting width and direction are different. Literature: Fukui “Object Contour Extraction Based on Degree of Separation between Regions” (Transactions of the Institute of Electronics, Information and Communication Engineers, D-II, Vol. J80-D-II, No. 6, p.
p. 1406-1414, June 1997) 1408
The degree of separation described on the page is determined for each contour, and the contour having the highest degree of separation among the blocks 2102 to 2106 is adopted. Thereby, the contour of the shape image matches the edge of the texture image.

As described above, according to the fourth embodiment, a high extraction accuracy can be obtained in the first part of the moving image sequence as in the last part. Even if the movement of the object is small or does not move at all, it is correctly extracted. Furthermore, by shrinking and pasting the shape data of a similar block larger than the current processing block, even if the contour line of the object area given by the shape data is shifted, it can be corrected to the correct position, By simply giving a rough outline of the region as shape data, the object region can be extracted with high accuracy in all subsequent input frames.

Note that the above first to fourth embodiments can be used in appropriate combinations. Further, all the procedures of the object extraction method of the first to fourth embodiments can be realized by software. In this case, a computer program for executing the procedure is simply introduced into a normal computer via a recording medium. Thus, the same effect as in the first to fourth embodiments can be obtained.

[0299]

As described above, according to the present invention, by tracking a target object using a figure surrounding the target object, the target object is not affected by extra movement around the target object. In addition, it is possible to accurately extract and track a target object.

[0300] Also, high extraction accuracy can be obtained regardless of the input image. Further, a high extraction accuracy can be obtained in the first part of the moving image sequence as in the last part.
Even if the movement of the object is small or does not move at all, it is correctly extracted.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a basic configuration of a moving image object tracking / extracting apparatus according to a first embodiment of the present invention.

FIG. 2 is an exemplary block diagram showing a first configuration example of the object tracking / extracting apparatus according to the embodiment;

FIG. 3 is an exemplary block diagram showing a second configuration example of the object tracking / extracting apparatus according to the embodiment;

FIG. 4 is an exemplary block diagram showing an example of a specific configuration of a background area determining unit provided in the object tracking / extracting apparatus of the embodiment.

FIG. 5 is an exemplary block diagram showing an example of a specific configuration of a graphic setting unit provided in the object tracking / extracting apparatus of the embodiment.

FIG. 6 is an exemplary block diagram showing an example of a specific configuration of a background motion deleting unit provided in the object tracking / extracting apparatus of the embodiment.

FIG. 7 is an exemplary view showing an example of a representative background area used in a background motion deletion unit provided in the object tracking / extracting apparatus of the embodiment.

FIG. 8 is an exemplary view for explaining the operation of the object tracking / extracting device of the embodiment.

FIG. 9 is a block diagram showing a first moving image object tracking / extracting device according to a second embodiment of the present invention.

FIG. 10 is a block diagram showing a second moving image object tracking / extracting device according to the second embodiment.

FIG. 11 is a block diagram showing a third moving image object tracking / extracting apparatus according to the second embodiment.

FIG. 12 is a block diagram showing a fourth moving image object tracking / extracting apparatus according to the second embodiment.

FIG. 13 is a view for explaining an object prediction method used in the object tracking / extracting apparatus according to the second embodiment.

FIG. 14 is an exemplary view for explaining a reference frame selection method used in the object tracking / extracting apparatus according to the second embodiment.

FIG. 15 is a diagram showing an example of a result of extracting an object by switching between a first object tracking / extracting unit and a second object extracting unit in the object tracking / extracting apparatus of the second embodiment.

FIG. 16 is an exemplary view for explaining the flow of object tracking / extraction processing of a moving image using the object tracking / extracting apparatus of the second embodiment.

FIG. 17 is a block diagram illustrating a first moving image object tracking / extracting apparatus according to a third embodiment of the present invention.

FIG. 18 is a block diagram showing a second moving image object tracking / extracting device according to the third embodiment.

FIG. 19 is a block diagram showing a third moving image object tracking / extracting device according to the third embodiment.

FIG. 20 is a block diagram showing a fifth moving image object tracking / extracting device according to the third embodiment.

FIG. 21 is a block diagram showing a sixth moving image object tracking / extracting device according to the third embodiment.

FIG. 22 is a block diagram showing a fourth moving image object tracking / extracting device according to the third embodiment.

FIG. 23 is a block diagram showing another configuration example of the moving image object tracking / extracting apparatus according to the third embodiment.

FIG. 24 is a block diagram showing still another configuration example of the moving image object tracking / extracting device according to the third embodiment.

FIG. 25 is an exemplary view for explaining an example of an extracted frame order by frame order control applied to the moving image object tracking / extracting apparatus according to the third embodiment.

FIG. 26 is a diagram showing an application example of the moving image object tracking / extracting device according to the third embodiment.

FIG. 27 is a block diagram showing an object extraction device according to a fourth embodiment of the present invention.

FIG. 28 is a block diagram showing a configuration example in a case where edge correction processing is applied to the object extraction device according to the fourth embodiment.

FIG. 29 is a block diagram showing a configuration example of a motion compensation unit applied to the object extraction device according to the fourth embodiment.

FIG. 30 is a block diagram showing an example of the configuration of an object extraction unit based on reduced block matching applied to the object extraction device according to the fourth embodiment.

FIG. 31 is a diagram showing an edge correction circuit based on a background pallet used in the object extraction device according to the fourth embodiment.

FIG. 32 is a diagram showing an image synthesizing device applied to the object extracting device according to the fourth embodiment.

FIG. 33 is a view for explaining the principle of edge correction using the degree of separation used in the object extraction device according to the fourth embodiment.

FIG. 34 is a view showing the entire processed image processed by the object extracting apparatus according to the fourth embodiment.

FIG. 35 is a view showing an outline drawn by an operator used in the fourth embodiment.

FIG. 36 is a view showing a state of block setting (first scan) used in the fourth embodiment.

FIG. 37 is a view showing a state of block setting (second scan) used in the fourth embodiment.

FIG. 38 is a view for explaining similar blocks used in the fourth embodiment.

FIG. 39 is an exemplary view for explaining a similar block search range used in the fourth embodiment;

FIG. 40 is an exemplary view for explaining another example of a similar block search range used in the fourth embodiment;

FIG. 41 is an exemplary view showing a state before replacement conversion of a shape image used in the fourth embodiment;

FIG. 42 is a view showing a state after replacement conversion of a shape image used in the fourth embodiment.

FIG. 43 is a view showing an outline extracted in the fourth embodiment.

FIG. 44 is a diagram showing a background color portion extracted in the fourth embodiment.

FIG. 45 is a view for explaining motion compensation used in the fourth embodiment;

FIG. 46 is a flowchart of an object extraction method using a background image used in the fourth embodiment.

FIG. 47 is a flowchart of an object extraction method by motion compensation used in the fourth embodiment.

FIG. 48 is a flowchart of an object extraction method using reduced block matching in a frame used in the fourth embodiment.

FIG. 49 is a view showing another example of block setting used in the fourth embodiment.

FIG. 50 is a flowchart for explaining edge correction;

FIG. 51 is a diagram showing an example of block setting.

FIG. 52 is a view showing a process of searching for a contour line of an object area.

FIG. 53 is a flowchart illustrating a method of gradually reducing the block size.

FIG. 54 is a view showing another example of block setting.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Initial figure setting part 2 ... Object tracking / extraction part 11 ... Figure setting part 12 ... Background area determination part 13 ... Object extraction part 21 ... Background motion deletion part 22 ... Figure setting part 23 ... Background area determination part 24 ... Object extraction Unit 31 Change amount detection unit 32 Representative region determination unit 33 Background change amount determination unit 34 Background determination unit of representative region 35 Background region determination unit 36 Shape prediction unit 37 Static object region determination unit 41 Separation unit Reference numeral 42: motion detection unit 43: division determination unit 44: graphic determination unit 51: background representative area setting unit 52: motion detection unit 53: motion compensation unit 61: graphic setting unit 62: multiple object tracking / extraction units 70: graphic setting Unit 71: second object tracking / extracting unit 72: reference frame selecting unit 73: first object tracking / extracting unit 111: feature amount extracting unit 141: frame order control unit 224: object extracting unit 279: object extraction 284 ... edge correction unit 285 ... edge correction unit

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Takashi Ida 1-term, Komukai Toshiba-cho, Saiwai-ku, Kawasaki-shi, Kanagawa F-term in Toshiba R & D Center (reference) 5L096 BA18 FA18 FA34 FA46 GA19 GA51 HA03 HA08 JA03

Claims (20)

[Claims] 1. A method according to claim 1, wherein the current frame includes a target object extracted from a moving image signal and a first reference frame temporally different from the current frame. Determining a first background region common to the frames, and determining, based on a difference between the current frame and a second reference frame temporally different from the current frame, the current frame and the second reference frame; Background area determination means for determining a common second background area; and, in the image of the current frame, an area that does not belong to either the first background area or the second background area is set as an object area Means for extracting a moving image, comprising: 2. A pixel having a small difference between the current frame and the reference frame uses a given object shape of the reference frame, and when the pixel of the reference frame belongs to an object area, 2. The object extracting apparatus according to claim 1, further comprising: a stationary object determining unit that determines a pixel as an object region and, when a pixel of the reference frame belongs to a background region, determines a pixel of the current frame as a background region. 3. The object shape given in advance is used when the object shape of the reference frame has already been extracted, and when the object region has not been extracted yet in the object shape of the reference frame. 3. The object extraction method according to claim 2, further comprising: generating a shape of the object of the reference frame from the frame from which the object shape has already been extracted by a block matching method, and using the thus generated shape. apparatus. 4. The motion of the background of the reference frame or the current frame is corrected such that the motion of the background is relatively zero between each of the first and second reference frames and the current frame. 2. The apparatus according to claim 1, further comprising a background correction unit.
An object extraction device according to claim 1. 5. The object extracting apparatus according to claim 1, wherein the background area determining means includes means for determining the common background area using a predetermined threshold value. 6. The predetermined threshold value is obtained by measuring a magnitude of a difference between the current frames, and setting a large threshold value when the difference is large, and setting a threshold value when the difference is small. 6. The object extracting apparatus according to claim 5, further comprising: means for setting the size to be small. 7. The predetermined threshold value is obtained by dividing the current frame into a plurality of regions, measuring the magnitude of a difference in each of the divided regions, and setting a large threshold value when the difference is large. 6. The object extracting apparatus according to claim 5, further comprising means for setting a small threshold value when the difference is small. 8. From a frame in which an object region has already been extracted,
Prediction means for predicting the position or shape of the object on the current frame; and the first and the second regions to be used by the background area determination means based on the position or shape of the object on the current frame predicted by the prediction means. 2. The object extracting apparatus according to claim 1, further comprising: means for selecting a second reference frame. 9. An initial figure setting means for setting a figure surrounding the target object in an initial frame of the moving image signal, for each input frame of the moving image signal, Graphic setting means for setting, in the input frame, a graphic enclosing an area on the input frame corresponding to the graphic image in the reference frame, based on the correlation with the graphic image in the reference frame that is temporally different. 2. The method according to claim 1, further comprising: extracting an area that does not belong to any of the first background area and the second background area in the in-graphic image as an object area. An object extraction device according to claim 1. 10. The object extracting apparatus according to claim 9, wherein the initial figure setting means sets a figure surrounding the target moving object based on an external input. 11. An initial graphic setting means for setting a graphic surrounding a target object in an initial frame of a moving image signal, and for each input frame of the moving image signal, an input frame and a temporal relation to the input frame. A graphic setting means for setting, in the input frame, a graphic surrounding an area on the input frame corresponding to the graphic image in the reference frame, based on a correlation with the graphic image in a different reference frame. A first background area common to the current frame and the first reference frame is determined based on a difference between a current frame from which an object is to be extracted and a first reference frame temporally different from the current frame. Then, based on a difference between the current frame and a second reference frame temporally different from the current frame, a common And a background area determining means for determining a second background region, wherein in the figure in the image of the current frame, a region which does not belong to either of the said first background region second background region,
First object extraction means including means for extracting the object area as an object area; and a method different from the first object extraction means. An object extraction device comprising: a second object extraction unit to be extracted; and a unit for selectively switching between the first and second object extraction units. 12. From a current frame from which an object is to be extracted,
The image processing apparatus further includes means for extracting a feature amount of an image for at least a part of the area, wherein the switching means selectively selects the first and second object extracting means based on the extracted feature amount. The object extraction device according to claim 11, which switches. 13. A prediction method for predicting a position or a shape of an object on a current frame from which an object is to be extracted from a reference frame using a frame from which an object region has been extracted as a reference frame. The object extracting apparatus according to claim 11, further comprising means for extracting an object region from a current frame based on the prediction result. 14. When the prediction error by the second object extraction means is within a predetermined range, the result of the extraction by the second object extraction means is used as an object area, and the prediction error exceeds the predetermined range. In this case, the first and second object extracting means are selectively operated in block units within a frame based on the prediction error amount so that the result of the extraction by the first object extracting means is used as an object area. The object extraction device according to claim 13, which is used by switching. 15. The second object extracting means performs inter-frame prediction in an order different from the input frame order such that a frame interval between a reference frame and a current frame from which an object is to be extracted is at least a predetermined frame. The object extraction device according to claim 11, which performs the operation. 16. By reading out the moving image data from a storage device in which the moving image data is recorded, and performing motion compensation on shape data representing an object area on a predetermined frame in a plurality of frames constituting the moving image data. Means for generating shape data for each frame constituting the read moving image data; and by sequentially overwriting image data of a background area of each frame determined by the generated shape data on a background memory. Means for generating a background image of the moving image data; and reading the moving image data again from a storage device in which the moving image data is recorded, and the background memory for each frame constituting the read moving image data. The difference between the corresponding pixel of the background image accumulated above and the absolute value of the difference is calculated as a predetermined value. Means for determining the larger pixels than threshold as the object region, the constructed object extracting apparatus. 17. A means for selectively outputting, as an object extraction result, one of an object region determined based on shape data of each frame and an object region determined based on an absolute value of a difference from the background image. The object extraction device according to claim 16, further comprising: 18. An image input means for inputting moving image data and shape data representing an object area on a predetermined frame in a plurality of frames constituting the moving image data, and a current processing frame is divided into a plurality of blocks. Means for dividing, for each of the blocks, means for searching for a similar block in which the pattern of the image data is similar and the area is larger than the current processing block from the reference frame, and converting the shape data of the similar block from the reference frame. An object extraction device comprising: means for attaching cut-out and reduced shape data to each block of the current processing frame; and means for outputting the attached shape data as shape data of the current processing frame. 19. An image input means for inputting image data and shape data representing an object region of the image, a block is set in a contour portion of the shape data, and a pattern of the image data is set for each block. Means for searching the same image for similar blocks that are similar and larger than the block; means for replacing the shape data of each block with shape data obtained by reducing the shape data of each similar block; And a means for repeating the replacement by a predetermined number of times, and outputting the corrected shape data as corrected shape data. 20. An apparatus according to claim 19, further comprising an iterative means for performing the search for the similar block and replacing the shape data a predetermined number of times a plurality of times, wherein the block size is smaller at the end than at the beginning of the iteration. 20. The object extraction device according to 19.