JP4679216B2 - Object mosaic processing method and mosaic processing apparatus - Google Patents

Description

  The present invention relates to video processing, and more particularly to an object mosaic processing method and mosaic processing apparatus that generate mosaic processing at a specific part corresponding to a person's movement.

   Mosaic processing is difficult. This is because there are many situations in which video that requires mosaic processing can be recorded accidentally, and there are very few examples of recording to perform mosaic processing. In other words, it is difficult to prepare an environment on the photographing side for video processing. Under such circumstances, the existing chroma key device and mask generation device cannot be used. Therefore, the recorded video that was not premised on performing the mosaic processing is processed.

 In order to perform mosaic processing with higher quality, an operator extracts a necessary area for mosaic processing in each frame and creates a mask, which requires enormous work time.

There are mainly the following two methods for mosaic processing.
The first is a method in which a human creates a mask by handwriting frame by frame.
The area that must be subjected to mosaic processing one frame at a time is depicted. The work time and work process inevitably increase in order to work on 30 recorded images per second.

The second is a method of performing mosaic processing using video composition software.
An operator designates a mosaic correction area in the first frame (first point). The correction area is designated again at the second point. The first point and the second point are automatically complemented. However, since the moving speed of the object is not constant and the camera angle is not fixed, the object often goes out of the area by linear interpolation. The worker repeats the work of detecting and correcting the place where the object has moved out of the area.

  The present invention is to automatically extract a region that must be subjected to mosaic processing. Conventionally, tracking data is generated using a motion tracking function. The motion tracking function generates movement data of the object based on the color data of the video. However, movement along the shape of the object cannot be generated.

  In this regard, Patent Document 1 proposes a method and apparatus for tracking a video object. According to this previously proposed technique, an object is tracked with multiple image frames. In the first frame, the operator selects the object to be tracked. A background mask and a foreground mask are generated by distinguishing the selected object or a modified estimate of the selected object from the rest of the background portion of the image. The foreground mask corresponds to the object to be tracked. The background mask model is used and updated in subsequent frames, and the foreground mask model is used and updated in subsequent frames. Pixels in subsequent frames are classified as belonging to the background or foreground. And in each subsequent frame, a decision is made that includes: That is, which pixels do not belong to the background, which pixels in the foreground should be updated (based on the original image), which pixels in the background were incorrectly recognized in the current frame, and which A determination is made whether the background pixel is recognized for the first time. Some of these decisions are not necessarily mutually exclusive with other decisions.

  Furthermore, according to the previously proposed technique, in addition to pixel classification, mask filtering is performed to correct errors, remove small islands, and maintain the spatial and temporal consistency of the foreground mask. Object tracking is achieved with a small output delay. In one embodiment, a three frame latency is employed.

  The advantage of the proposed technique is that an object having a rapidly changing shape is accurately tracked against a non-moving background. The motion of the object is detected using information in the previous frame and the future frame. Objects can change internally and can even hide themselves. Another advantage is that previously unrecovered (ie hidden) background portions are identified, increasing the accuracy of the object estimate. Another advantage is that holes in the object are also accurately tracked.

However, even with the previously proposed technique, depending on the type of object to be processed (video), the reliability is insufficient.
JP2003-51983

  The present invention provides a highly reliable tracking function for creating a shape along an object on a specified frame with a polygon and automatically extracting an object on a subsequent frame in a series of movements. The present invention relates to a mask processing system provided.

The above-described problems can be solved by the following configuration.
1. The user designates the object area to be tracked on the first frame, tracks the designated area (object) for the subsequent frame, determines its position on the subsequent frame, and sets the new position of the obtained designated area. Based on this, the position of the designated area (object) is determined by tracking for the next frame, this tracking process is continuously executed for the subsequent frames, and the object on each frame obtained in this way is obtained. the specified mask processing a mosaic processing method of facilities to object against,
In the matching process that determines where the polygon points that are the points of the polygons generated by the points (control points) generated by mouse clicks are located in the next frame from the previous frame, both region-based tracking and boundary-based tracking It has a concept and uses two types of blocks: simple blocks for simple block matching and derivative blocks for derivative block matching. Simple block generation and simple block matching use the advantages of area tracking. The derivative block uses the advantage of boundary tracking. When a derivative block is generated, the object mosaic processing is characterized in that pixels on and near the boundary are not included in the block. Method.

2. "Read input AVI (video standard format) file" step, "Display first frame" step, "Create polygon in specified area" step, "Track specified area on subsequent frame" step, "Get by tracking" masking (mosaic generation, cutout) on the specified region in the "step, a Luo object mosaic processing method of having a step of" save AVI files to the mask process has been added ",
In the matching process that determines where the polygon points that are the points of the polygons generated by the points (control points) generated by mouse clicks are located in the next frame from the previous frame, both region-based tracking and boundary-based tracking It has a concept and uses two types of blocks: simple blocks for simple block matching and derivative blocks for derivative block matching. Simple block generation and simple block matching use the advantages of area tracking. The derivative block uses the advantage of boundary tracking, and when generating a derivative block, pixels on and near the boundary are not included in the block.
Mosaic processing method for objects.

3. The user designates the object area to be tracked on the first frame, tracks the designated area (object) for the subsequent frame, determines its position on the subsequent frame, and sets the new position of the obtained designated area. Based on this, the position of the designated area (object) is determined by tracking for the next frame, this tracking process is continuously executed for the subsequent frames, and the object on each frame obtained in this way is obtained. A mosaic processing method for an object to be subjected to the mask processing specified for
The object to be tracked may be slightly deformed in the area, and the shape change should be taken into account when tracking, and the block shape on the previous frame is not changed when matching with the current frame. The object mosaic processing method is characterized in that the concept of “deformation” and “deformation limit” defined below is introduced into the block matching algorithm.
[Deformation and deformation limit]
For each block pixel on the previous frame, in addition to matching with the corresponding pixel on the current frame, it also matches the pixel adjacent to that pixel, and the number of adjacent pixels depends on the applicable deformation limit. The block porosity of the block pixel used for matching is set to 2 or more, the round pixel on the block of the previous frame is matched with the round pixel of the corresponding block on the current frame, and the round pixel on the previous frame is In addition to the comparison with the corresponding block pixel, it is also matched with neighboring pixels.

4). "Read input AVI (video standard format) file" step, "Display first frame" step, "Create polygon in specified area" step, "Track specified area on subsequent frame" step, "Get by tracking" A mask processing on a designated area (mosaic generation, clipping) ”step and“ save AVI file with mask processing added ”steps,
The object to be tracked may be slightly deformed in the area, and the shape change should be taken into account when tracking, and the block shape on the previous frame is not changed when matching with the current frame. The object mosaic processing method is characterized in that the concept of “deformation” and “deformation limit” defined below is introduced into the block matching algorithm.
[Deformation and deformation limit]
For each block pixel on the previous frame, in addition to matching with the corresponding pixel on the current frame, it also matches the pixel adjacent to that pixel, and the number of adjacent pixels depends on the applicable deformation limit. The block porosity of the block pixel used for matching is set to 2 or more, the round pixel on the block of the previous frame is matched with the round pixel of the corresponding block on the current frame, and the round pixel on the previous frame is
In addition to comparison with the corresponding block pixel on the frame, it is also matched with neighboring pixels.

5. The user designates the object area to be tracked on the first frame, tracks the designated area (object) for the subsequent frame, determines its position on the subsequent frame, and sets the new position of the obtained designated area. Based on this, the position of the designated area (object) is determined by tracking for the next frame, this tracking process is continuously executed for the subsequent frames, and the object on each frame obtained in this way is obtained. A mosaic processing method for an object to be subjected to the mask processing specified for
When tracking the target area by following the frame, a small error that is difficult to recognize occurs between the previous and next frames, and this error is accumulated as the frame is followed, and finally there is a difference in level that can be clearly recognized. An object mosaic processing method that uses the concept of a key frame defined below as a method to deal with.
[Key frame]
The user defines objects to be tracked in advance on several frames (key frames), and sets frames that are likely to cause the following problems in tracking as key frames.
(1) When the object moves suddenly or very fast
(2) When camera movement is large
(3) When multiple objects overlap
(4) When lighting conditions change
When a key frame is reached in the middle of tracking, tracking will be performed using a defined object polygon on the key frame, and when it reaches the key frame, error accumulation will be canceled and all registered The current frame is tracked for the keyframes, and the polygon on the keyframe that produced the best result among them is selected as a polygon candidate on the current frame. As a final adjustment, the entire candidate is compared, and the polygon candidates obtained in the above process are translated, rotated, and offset vertically and horizontally on the current frame to minimize the color difference in the entire region comparison. Find the shape to make.

6). "Read input AVI (video standard format) file" step, "Display first frame" step, "Create polygon in specified area" step, "Track specified area on subsequent frame" step, "Get by tracking" A mask processing on a designated area (mosaic generation, clipping) ”step and“ save AVI file with mask processing added ”steps,
When tracking the target area by following the frame, a small error that is difficult to recognize occurs between the previous and next frames, and this error is accumulated as the frame is followed, and finally there is a difference in level that can be clearly recognized. As a way to deal with the key defined below
A method for mosaic processing objects that are processed using the concept of lemmes.
[Key frame]
The user defines objects to be tracked in advance on several frames (key frames), and sets frames that are likely to cause the following problems in tracking as key frames.
(1) When the object moves suddenly or very fast
(2) When camera movement is large
(3) When multiple objects overlap
(4) When lighting conditions change
When a key frame is reached in the middle of tracking, tracking will be performed using a defined object polygon on the key frame, and when it reaches the key frame, error accumulation will be canceled and all registered The current frame is tracked for the keyframes, and the polygon on the keyframe that produced the best result among them is selected as a polygon candidate on the current frame. As a final adjustment, the entire candidate is compared, and the polygon candidates obtained in the above process are translated, rotated, and offset vertically and horizontally on the current frame to minimize the color difference in the entire region comparison. Find the shape to make.

7). The tracking target area is defined by polygons generated by mouse click points (control points), and tracking in subsequent frames is performed for the area specified by the user. Tracking is performed for each point on the polygon. When the point on the previous frame polygon is tracked with respect to the current frame, the point coordinate value on the previous frame is used as the initial point coordinate value on the current frame. A block is generated in the vicinity of the polygon point for all the points of, and the same block is also created at the corresponding point candidate position on the current frame. Is determined, any of 1 to 6 above Mosaic processing method of the placement of objects.

8). 8. The object mosaic processing method according to any one of 1 to 7 above, wherein the point neighborhood block includes only pixels in the polygon area (in the designated object area), and avoids an influence outside the polygon area. .

9. A matching process that determines where the polygon point in the previous frame is located in the next frame is created for the point neighborhood block created for the point candidate position in the next frame and the corresponding point in the previous frame. This determination is made by comparing whether the neighboring blocks at the same point are the same, and this determination is made by comparing the RGB values of the pixels at the corresponding positions in both blocks, so that all the pixels in the block on the next frame are If the RGB value is the same as the corresponding pixel of the block on the frame, then both blocks are perfectly matched (similar determination is made for the derivative block), and the degree of this matching is the reference determination value. If you are not satisfied, shift the point candidate position on the next frame. Performed again similar determination, mosaicking method object of any one of the 1 to 8, characterized in that repeated until the process point position on the next frame determined.

10. The matching process has the concept of both region-based tracking and boundary-based tracking, and uses two types of blocks: simple block for simple block matching and derivative block for derivative block matching. Block matching uses the advantage of region tracking, derivative block uses the advantage of boundary tracking, and when generating a derivative block, the pixels on and near the boundary should not be included in the block 10. The method for mosaic processing an object according to any one of 3 to 9 above.

11. The size of the block is determined based on the size of the target area polygon, does not cover the entire target area, and the concept of the porous block defined below is introduced for the block near the point. The above-mentioned 1 to 10 are characterized in that only the remaining pixels obtained by thinning out some pixels at the following specific thinning rate (porosity) are processed as block pixels. An object mosaic processing method according to any one of the above.
[Thinning rate (porosity)]
The thinning rate (porosity) is determined in proportion to the average polygon width, and the porosity increases as the polygon width increases. For large polygons with a wide polygon width, a larger block size is required to improve tracking accuracy. Although necessary, since the porosity increases, the number of block pixels does not increase so much, and pixels outside the boundary are excluded during block generation, and pixels within the boundary are selected discretely.

12 The object to be tracked may be slightly deformed in the area, and the shape change should be taken into account when tracking, and the block shape on the previous frame is not changed when matching with the current frame. The object of any one of 1, 2 or 5 to 11, wherein the concept of “deformation” and “deformation limit” defined below is introduced into the block matching algorithm. Mosaic processing method.
[Deformation and deformation limit]
For each block pixel on the previous frame, in addition to matching with the corresponding pixel on the current frame, it also matches the pixel adjacent to that pixel, and the number of adjacent pixels depends on the applicable deformation limit. The block porosity of the block pixel used for matching is set to 2 or more, the round pixel on the block of the previous frame is matched with the round pixel of the corresponding block on the current frame, and the round pixel on the previous frame is In addition to the comparison with the corresponding block pixel, it is also matched with neighboring pixels.

13. A database tracking method has been introduced in which frames defined in advance for various shapes that can be taken by the target object are registered as reference frames, and tracking of the current frame is executed for all registered reference frames. The polygon on the reference frame that has produced the best result in is selected as a polygon candidate on the current frame, and tracking processing is further performed on this polygon candidate. A method for mosaic processing of the described object.

14 When tracking the target area by following the frame, a small error that is difficult to recognize occurs between the previous and next frames, and this error is accumulated as the frame is followed, and finally there is a difference in level that can be clearly recognized. The object mosaic processing method according to any one of 1 to 4 or 7 to 13 described above, wherein processing is performed using a concept of key frames defined below as a method of coping.
[Key frame]
The user defines objects to be tracked in advance on several frames (key frames), and sets frames that are likely to cause the following problems in tracking as key frames.
(1) When the object moves suddenly or very fast
(2) When camera movement is large
(3) When multiple objects overlap
(4) When lighting conditions change
When a key frame is reached in the middle of tracking, tracking will be performed using a defined object polygon on the key frame, and when it reaches the key frame, error accumulation will be canceled and all registered Current key frame
Tracking is performed, and the polygon on the key frame that produced the best result among them is selected as the polygon candidate on the current frame. During this tracking process, the entire polygon internal area is compared, Further, as a final adjustment, a shape that minimizes the color difference in the entire region comparison is obtained by executing parallel movement, rotation, and offset on the current frame in the vertical and horizontal directions with respect to the polygon candidate obtained by the above processing.

15. In a mosaic processing apparatus for an object having a processor, a user designates an object area to be tracked on the first frame and tracks the designated area (object) for the subsequent frame to determine its position on the subsequent frame. Based on the new position of the designated area obtained, the position of the designated area (object) is determined by tracking for the next frame, and this tracking process is continuously executed for the subsequent frames. An object mosaic processing apparatus having a processor configured to perform a specified mask process on an object on each frame obtained as described above,
In the matching process that determines where the polygon points that are the points of the polygons generated by the points (control points) generated by mouse clicks are located in the next frame from the previous frame, both region-based tracking and boundary-based tracking It has a concept and uses two types of blocks: simple blocks for simple block matching and derivative blocks for derivative block matching. Simple block generation and simple block matching use the advantages of area tracking. The derivative block uses the advantage of boundary tracking. When generating a derivative block, it is necessary to have a processor configured to perform processing that does not include pixels on and near the boundary. Mosaic processing device object to symptoms.

16. In the mosaic processing apparatus for an object having a processor, the processor performs an “input AVI (video standard format) file reading” step, a “display first frame” step, a “polygon creation in a designated area” step, and a “following frame on” The object is configured to perform the steps of “tracking specified area”, “mask processing (mosaic generation, clipping) on specified area obtained by tracking”, and “save AVI file with mask processing”. A mosaic processing apparatus,
In the matching process that determines where the polygon points that are the points of the polygons generated by the points (control points) generated by mouse clicks are located in the next frame from the previous frame, both region-based tracking and boundary-based tracking It has a concept and uses two types of blocks: simple blocks for simple block matching and derivative blocks for derivative block matching. Simple block generation and simple block matching use the advantages of area tracking. The derivative block uses the advantage of boundary tracking, and when generating a derivative block, the pixels on and near the boundary are not included in the block.
An object mosaic processing apparatus comprising: a processor.

17. In a mosaic processing apparatus for an object having a processor, a user designates an object area to be tracked on the first frame and tracks the designated area (object) for the subsequent frame to determine its position on the subsequent frame. Based on the new position of the designated area obtained, the position of the designated area (object) is determined by tracking for the next frame, and this tracking process is continuously executed for the subsequent frames. An object mosaic processing apparatus having a processor configured to perform a specified mask process on an object on each frame obtained as described above,
The object to be tracked may be slightly deformed in the area, and the shape change should be taken into account when tracking, and the block shape on the previous frame is not changed when matching with the current frame. An object mosaic processing apparatus comprising a processor having a configuration in which the concept of “deformation” and “deformation limit” defined below is introduced into a block matching algorithm.
[Deformation and deformation limit]
For each block pixel on the previous frame, in addition to matching with the corresponding pixel on the current frame, it also matches the pixel adjacent to that pixel, and the number of adjacent pixels depends on the applicable deformation limit. The block porosity of the block pixel used for matching is set to 2 or more, the round pixel on the block of the previous frame is matched with the round pixel of the corresponding block on the current frame, and the round pixel on the previous frame is In addition to the comparison with the corresponding block pixel, it is also matched with neighboring pixels.

18. In the mosaic processing apparatus for an object having a processor, the processor performs an “input AVI (video standard format) file reading” step, a “display first frame” step, a “polygon creation in a designated area” step, and a “following frame on” The object is configured to perform the steps of “tracking specified area”, “mask processing (mosaic generation, clipping) on specified area obtained by tracking”, and “save AVI file with mask processing”. A mosaic processing apparatus,
The object to be tracked may be slightly deformed in the area, and the shape change should be taken into account when tracking, and the block shape on the previous frame is not changed when matching with the current frame. An object mosaic processing apparatus comprising a processor having a configuration in which the concept of “deformation” and “deformation limit” defined below is introduced into a block matching algorithm.
[Deformation and deformation limit]
For each block pixel on the previous frame, in addition to matching with the corresponding pixel on the current frame, it also matches with the pixel adjacent to that pixel.
The number of contact pixels is determined by the deformation limit to be applied, the block porosity of the block pixel used for matching is 2 or more, and the round pixel on the block of the previous frame is matched with the round pixel of the corresponding block on the current frame. A round pixel on a frame must be matched with a neighboring pixel in addition to a comparison with the corresponding block pixel on the current frame.

19. In a mosaic processing apparatus for an object having a processor, a user designates an object area to be tracked on the first frame and tracks the designated area (object) for the subsequent frame to determine its position on the subsequent frame. Based on the new position of the designated area obtained, the position of the designated area (object) is determined by tracking for the next frame, and this tracking process is continuously executed for the subsequent frames. An object mosaic processing apparatus having a processor configured to perform a specified mask process on an object on each frame obtained as described above,
When tracking the target area by following the frame, a small error that is difficult to recognize occurs between the previous and next frames, and this error is accumulated as the frame is followed, and finally there is a difference in level that can be clearly recognized. An object mosaic processing apparatus comprising: a processor configured to perform processing using a concept of a key frame defined below as a method for coping.
[Key frame]
The user defines objects to be tracked in advance on several frames (key frames), and sets frames that are likely to cause the following problems in tracking as key frames.
(1) When the object moves suddenly or very fast
(2) When camera movement is large
(3) When multiple objects overlap
(4) When lighting conditions change
When a key frame is reached in the middle of tracking, tracking will be performed using a defined object polygon on the key frame, and when it reaches the key frame, error accumulation will be canceled and all registered The current frame is tracked for the keyframes, and the polygon on the keyframe that produced the best result among them is selected as a polygon candidate on the current frame. As a final adjustment, the entire candidate is compared, and the polygon candidates obtained in the above process are translated, rotated, and offset vertically and horizontally on the current frame to minimize the color difference in the entire region comparison. Find the shape to make.

20. In the mosaic processing apparatus of an object having a processor, the processor reads the “input AVI (video standard format) file” step, “displays the first frame.
Step ”,“ Create Polygon in Specified Area ”Step,“ Tracking Specified Area on Subsequent Frames ”Step,“ Mask Processing on Specified Area Obtained by Tracking (Mosaic Generation, Crop) ”Step,“ Mask An object mosaic processing apparatus configured to perform the step of “save AVI file with processing added”,
When tracking the target area by following the frame, a small error that is difficult to recognize occurs between the previous and next frames, and this error is accumulated as the frame is followed, and finally there is a difference in level that can be clearly recognized. An object mosaic processing apparatus comprising: a processor configured to perform processing using a concept of a key frame defined below as a method for coping.
[Key frame]
The user defines objects to be tracked in advance on several frames (key frames), and sets frames that are likely to cause the following problems in tracking as key frames.
(1) When the object moves suddenly or very fast
(2) When camera movement is large
(3) When multiple objects overlap
(4) When lighting conditions change
When a key frame is reached in the middle of tracking, tracking will be performed using a defined object polygon on the key frame, and when it reaches the key frame, error accumulation will be canceled and all registered The current frame is tracked for the keyframes, and the polygon on the keyframe that produced the best result among them is selected as a polygon candidate on the current frame. As a final adjustment, the entire candidate is compared, and the polygon candidates obtained in the above process are translated, rotated, and offset vertically and horizontally on the current frame to minimize the color difference in the entire region comparison. Find the shape to make.

21. The tracking target area is defined by polygons generated by mouse click points (control points), and tracking in subsequent frames is performed for the area specified by the user. Tracking is performed for each point on the polygon. When the point on the previous frame polygon is tracked with respect to the current frame, the point coordinate value on the previous frame is used as the initial point coordinate value on the current frame. A block is generated in the vicinity of the polygon point for all the points of, the same block is also created at the corresponding point candidate position on the current frame, and the position of the corresponding point on the current frame by matching comparison on both blocks Characterized by having a processor configured to determine Mosaic processing device object according to the 15-20.

22. The block near the point is a pixel in the polygon area (in the specified object area)
The object mosaic processing device according to any one of 15 to 21, further comprising a processor configured to avoid an influence outside the polygon region.

23. A matching process that determines where the polygon point in the previous frame is located in the next frame is created for the point neighborhood block created for the point candidate position in the next frame and the corresponding point in the previous frame. This determination is made by comparing whether the neighboring blocks at the same point are the same, and this determination is made by comparing the RGB values of the pixels at the corresponding positions in both blocks, so that all the pixels in the block on the next frame are If the RGB value is the same as the corresponding pixel of the block on the frame, then both blocks are perfectly matched (similar determination is made for the derivative block), and the degree of this matching is the reference determination value. If you are not satisfied, shift the point candidate position on the next frame. Performed again similar determination, mosaic processing apparatus object according to any one of 15 to 22, characterized in that it comprises a processor for repeated arrangement until the process point position on the next frame determined.

24. The matching process has the concept of both region-based tracking and boundary-based tracking, and uses two types of blocks: simple block for simple block matching and derivative block for derivative block matching. Block matching uses the advantage of region tracking, and derivative blocks use the advantage of boundary tracking. When generating a derivative block, the pixels on and near the boundary are not included in the block. 24. The object mosaic processing device according to any one of 17 to 23, further including a processor configured to perform the processing.

25. The size of the block is determined based on the size of the target area polygon, does not cover the entire target area, and the concept of the porous block defined below is introduced for the block near the point. It is characterized by having a processor configured to process only the remaining pixels obtained by thinning out some pixels with the following specific thinning-out rate (porosity) as block pixels, instead of treating all the pixels as block pixels. The object mosaic processing apparatus according to any one of 15 to 24.
[Thinning rate (porosity)]
The thinning rate (porosity) is determined in proportion to the average polygon width, and as the polygon width increases.
For example, a large polygon with a wide polygon width needs a larger block size to increase tracking accuracy, but the number of block pixels does not increase so much because the porosity increases. When generating a block, pixels outside the boundary are excluded, and pixels within the boundary are selected discretely.

26. The object to be tracked may be slightly deformed in the area, and the shape change should be taken into account when tracking, and the block shape on the previous frame is not changed when matching with the current frame. Any one of 15, 16 and 19-25, characterized in that it has a processor configured to introduce the concept of “deformation” and “deformation limit” defined below in the block matching algorithm. The object mosaic processing device described in 1.
[Deformation and deformation limit]
For each block pixel on the previous frame, in addition to matching with the corresponding pixel on the current frame, it also matches the pixel adjacent to that pixel, and the number of adjacent pixels depends on the applicable deformation limit. The block porosity of the block pixel used for matching is set to 2 or more, the round pixel on the block of the previous frame is matched with the round pixel of the corresponding block on the current frame, and the round pixel on the previous frame is In addition to the comparison with the corresponding block pixel, it is also matched with neighboring pixels.

27. A database tracking method has been introduced in which frames defined in advance for various shapes that can be taken by the target object are registered as reference frames, and tracking of the current frame is executed for all registered reference frames. The polygons on the reference frame that yielded the best result in (5) are selected as polygon candidates on the current frame, and the processor has a configuration in which tracking processing is further performed on the polygon candidates. 27. The mosaic processing apparatus for objects according to any one of 26.

28. When tracking the target area by following the frame, a small error that is difficult to recognize occurs between the previous and next frames, and this error is accumulated as the frame is followed, and finally there is a difference in level that can be clearly recognized. The object mosaic processing device according to any one of 15 to 18 or 21 to 27, further comprising a processor configured to perform processing using a concept of a key frame defined below as a method for coping.
[Key frame]
The user defines objects to be tracked in advance on several frames (key frames), and sets frames that are likely to cause the following problems in tracking as key frames.
(1) When the object moves suddenly or very fast
(2) When camera movement is large
(3) When multiple objects overlap
(4) When lighting conditions change
When a key frame is reached in the middle of tracking, tracking will be performed using a defined object polygon on the key frame, and when it reaches the key frame, error accumulation will be canceled and all registered The current frame is tracked for the keyframes, and the polygon on the keyframe that produced the best result among them is selected as a polygon candidate on the current frame. As a final adjustment, the entire candidate is compared, and the polygon candidates obtained in the above process are translated, rotated, and offset vertically and horizontally on the current frame to minimize the color difference in the entire region comparison. Find the shape to make.

  The present invention relates to a digital image object tracking process and apparatus. A conceptual diagram of a system according to the present invention is shown in FIG.

  A user designates a specific area (object) on the first frame of an AVI (Video Standard Format) file, and the area (object) is tracked for successive subsequent frames. The user designates an object area to be tracked on the first frame, and for the subsequent frame, the designated area (object) is tracked to determine its position on the subsequent frame. Based on the obtained new position of the designated area, the position of the designated area (object) is determined by tracking for the next frame. This tracking process is continuously performed on subsequent frames. The designated mask process is performed on the objects on each frame thus obtained. An overall flowchart of this process is shown in FIG.

  That is, "Read input AVI file" step, "Display first frame" step, "Create polygon in specified area" step, "Track specified area on subsequent frame" step, "Specified area obtained by tracking" The above processing is performed in the steps of “mask processing (mosaic generation, clipping)” and “save AVI file with mask processing”.

  The tracking target area is defined by a polygon generated by a point (control point) group by mouse click. Tracking in subsequent frames is executed for the area specified by the user. Tracking is performed for each point on the polygon. When tracking the point on the polygon of the previous frame with respect to the current frame, the point coordinate value on the previous frame is used as the initial point coordinate value on the current frame in order to increase the tracking speed. A block is generated near the polygon point for all points of each polygon on the previous frame, the same block is also created at the corresponding point candidate position on the current frame, and matching on both blocks is compared on the current frame. The position of the corresponding point at is fixed.

  The concept of generating a point neighborhood block is used in many research papers, but the point neighborhood block adopted in the present invention is composed only of pixels in the polygon area (in the specified object area), and the influence outside the polygon area Therefore, a good tracking result can be expected. In the matching process, region-based tracking and boundary-based tracking methods are usually used. However, in the present invention, a unique algorithm incorporating the concepts of both region-based tracking and boundary-based tracking is used. In the present invention, two types of blocks are used: a simple block for simple block matching and a derivative block for derivative block matching. Simple block generation and simple block matching take advantage of the advantages of region tracking. Derivative blocks take advantage of boundary tracking, and the method of generating and matching derivative blocks was originally developed by the inventors. The method adopted for the block generation and matching method was derived from a great deal of examination and research by the present inventors. When generating a derivative block, pixels on and near the boundary are not included in the block. The reason is that these pixels are greatly affected by the color of the out-of-boundary pixels, thereby adversely affecting the tracking result.

  In order to achieve the conflicting purpose of improving the tracking processing speed and maintaining the tracking accuracy, the block size is determined based on the size of the target region polygon and does not cover the entire target region. Furthermore, a new concept of a porous block is introduced to the block near the point, and not all pixels in the block are treated as block pixels, but only the remaining pixels obtained by thinning out some pixels are processed as block pixels.

  Small errors accumulate as tracking on each frame is repeated. This error is not a problem as long as the number of frames to be tracked is small, but the shape collapse due to error accumulation cannot be ignored as many frames are processed. It is also important to maintain the object's geometric shape while performing accurate tracking. In the present invention, a technique called a shape correction algorithm is introduced to solve this problem.

  The tracking target object does not always move only without rigid deformation. The object area may be slightly deformed. Therefore, it is necessary to consider a change in shape when tracking. When matching with the current frame, the block shape on the previous frame should not be compared as invariant. In order to solve this problem, a new concept of “deformation” and “deformation limit” was introduced into the block matching algorithm.

  Repeated extremely fast movements exist in practice. In such a case, there arises a problem that cannot be followed by the sequential tracking method of tracking the current frame based only on the information of the previous frame. As a countermeasure for such a case, a database tracking method has been introduced in which a frame in which various shapes that can be taken by the target object are defined in advance is registered as a reference frame. Current frame tracking is performed on all registered reference frames, and the polygon on the reference frame that produced the best result among them is selected as a candidate polygon on the current frame. Further tracking processing is executed for the polygon candidate.

  FIG. 1 shows an example of a configuration diagram of the Mask Tracker system. Video data is composed of two types of audio data and video data. Video data is converted to AVI format. A video portion to be subjected to tracking processing is cut out using video editing software, and tracking processing is executed by the Mask Tracker application via the Mask Tracker plug-in. The AVI file on which the tracking process for the tracking object has been performed is written into the original video AVI file using video processing software via the Mask Tracker plug-in, and further converted into video tape by conversion hardware. The Mask Tracker plug-in plays an intermediary role between the video editing software and the Mask Tracker application. Input data to the Mask Tracker application is an AVI file received from the Mask Tracker plug-in. The user can track the target object using the Mask Tracker application. The tracked AVI file is returned to the video editing software again via the Mask Tracker plug-in.

  The user designates a specific area on the first frame of the AVI file and tracks that area for successive subsequent frames. The user designates an object area to be tracked on the first frame. The user designates an object area to be tracked on the first frame, and for the subsequent frame, the designated area (object) is tracked to determine its position on the subsequent frame. The entire flowchart of the tracking process is as shown in FIG.

  The Mask Tracker handler accesses the first frame having a region-designated polygon, and then reads the second frame. Polygon tracking is performed on this frame using a frame tracking algorithm. After the polygon position on the second frame is determined, tracking on the third frame is executed based on this polygon. Similar processing is performed successively on subsequent frames. A flowchart of the Mask Tracker handler algorithm is shown in FIG.

  That is, until the last frame from the first frame having the tracking area designated polygon to the last frame, tracking is executed one after another. Unless the current frame is the last frame, the next frame to be tracked is selected, the frame tracking algorithm is used, and the tracking is repeated with reference to the polygon on the previous frame. Then, when the current frame becomes the last frame, tracking of the AVI file ends.

  In the present invention, the tracking target area is defined by a polygon defined by a point (control point) group by mouse click. Since a spline is created through the points created on the frame, the polygon is displayed as a smooth curve with no corners.

  When a point on a polygon existing on the previous frame is tracked with respect to the current frame, the point coordinate value on the previous frame is used as the initial point coordinate value on the current frame in order to increase the tracking speed. All points of each polygon on the previous frame are tracked using a block matching algorithm on the current frame. A block is generated near the polygon point for all points of each polygon on the previous frame, the same block is also created at the corresponding point candidate position on the current frame, and matching on both blocks is compared on the current frame. The position of the corresponding point at is determined. When the tracking for all the polygons on the previous frame is completed, the tracking process for the current frame ends. A flowchart of this frame tracking algorithm is shown in FIG. The block matching algorithm will be described later.

  As shown in the flowchart of FIG. 4 (A), the frame tracking algorithm includes a “previous frame having a polygon and a current frame to be tracked” step, an “access target polygon on current frame” step, a “selected polygon”. "Access the next point above" step, "Tracking the polygon point using the block matching handler algorithm" step, and the step "(A)" below. The step “access to the polygon to be processed” and the subsequent “(C)” go through the step “access the next point on the selected polygon”. As shown in the flowchart of FIG. 4B, from the step “(A)”, “set X and Y coordinate values of the point of the previous frame to the initial tracking value in the current frame” step, “whether the current point is the last ”, If“ NO ”in this step, go to“ (C) ”, if YES, go to“ Is the current polygon the last polygon on the previous frame ”step, if NO,“ (B ) ", And in the case of YES, the process ends through the step" End tracking on current frame ".

  The concept of generating a point neighborhood block is used in many research papers, but the point neighborhood block adopted in the present invention is composed of only some pixels in the polygon area (in the specified object area) and is outside the polygon area. Therefore, good quality tracking results can be expected. The tracking processing speed is proportional to the number of processing target pixels in the block. Therefore, instead of treating all the pixels in the block as block pixels, the tracking speed can be improved by making the remaining pixels obtained by thinning out some pixels as block pixels to be processed. If the block size is simply reduced by improving the tracking speed, a good tracking result cannot be obtained. A new concept of porous blocks was introduced as a way to achieve both the advantages of large block size and improved tracking speed. The thinning rate (porosity) is determined based on the polygon width. As the porosity increases, the tracking speed increases but the tracking accuracy deteriorates. The porosity is determined in proportion to the average polygon width, and the porosity increases as the polygon width increases. In the case of a large polygon having a wide polygon width, a larger block size is required to improve tracking accuracy. However, since the porosity increases, the number of block pixels does not increase so much. FIG. 7 shows an example of a porous block. Line 1 shows the polygon boundary. A rectangle 2 is a point on the polygon. The figure shows an example where the porosity is 3, with every third pixel being selected. The selected pixel is indicated by circle 3. When generating a block, pixels outside the boundary are excluded, and pixels within the boundary are selected discretely.

  In the matching process, region-based tracking and boundary-based tracking methods are usually used, but in the present invention, a unique algorithm that combines the concepts of both region-based tracking and boundary-based tracking is used. In the present invention, two types of blocks are used: a simple block for simple block matching and a derivative block for derivative block matching.

  FIG. 5 shows a simple block generated near a point on the polygon. The gray area 5 is a block area. A circle 4 is a point (control point) on the polygon. Points 6 that are discretely present within the gray area indicate the selected pixel. This indicates the porosity of the block.

  The idea of generating a block near a point is used in many research papers, but the method adopted by the present inventors is based on a unique idea. The block near the point in the present invention is composed only of pixels in the polygon area (in the designated object area). Since the influence outside the polygon area can be avoided, a good tracking result can be expected. In order to increase the tracking processing speed, a new concept of porous block was introduced. Instead of treating all pixels in the block as block pixels, the remaining pixels obtained by thinning out some pixels are processed as block pixels. The thinning rate (porosity) is determined based on the polygon width.

  As the porosity increases, the tracking speed increases but the tracking accuracy deteriorates. The porosity is determined in proportion to the average polygon width, and the porosity increases as the polygon width increases.

  In the case of a large polygon having a wide polygon width, a larger block size is required to improve tracking accuracy. However, since the porosity increases, the number of block pixels does not increase so much.

  The block in Fig. 5 schematically shows a block near the point on the polygon. There are two types of blocks near the point: a simple block and a derivative block. This figure shows the case of a simple block. In this figure, the lines are surrounded by lines, each square represents one pixel area, and the rectangular part (on the black thick line) is a point on the polygon.

  The matching process determines where this polygon point in the previous frame is located in the next frame. The matching point block created for the point candidate position in the next frame and the corresponding point in the previous frame It is determined whether the point neighboring blocks created for the same are the same. This determination is made by comparing the RGB values of the pixels at the corresponding positions in both blocks. If all the pixels in the block on the next frame are the same RGB values as the corresponding pixels in the block on the previous frame, then both blocks are perfectly matched (the same decision is made for the derivative block). ). If the degree of matching does not satisfy the reference determination value, the same determination is performed again by shifting the point candidate position on the next frame. This process is repeated until the point position on the next frame is determined.

  The process described above requires enormous operations. A porous block was introduced for the purpose of reducing this calculation amount. In this case, the comparison is not performed for all the pixels on the block, but only a part of the pixels is picked up and the comparison operation is performed. The pixel at the position of the circle in FIG. 5 is a pixel that has been picked up. There are three blank grids between each round pixel. Pixels at the positions of these blank squares are excluded from the calculation target. That is, this figure shows an example in which the porosity is 3.

  When generating a derivative block, pixels on and near the boundary are not included in the block. The reason is that these pixels are greatly affected by the color of the out-of-boundary pixels, thereby adversely affecting the tracking result. Pixels far from the boundary do not have boundary characteristics, so these pixels are not included in the derivative block. The shortest distance between the polygon boundary and the boundary pixel of the derivative block is at least 4 pixels or more, and the longest distance is determined based on the polygon width (longest distance = polygon width / 2.5). However, this maximum distance is not at least 8 pixels or less. FIG. 6 shows an example of a derivative block. Derivative blocks have porosity similar to simple blocks.

  Simple block generation and simple block matching take advantage of the advantages of region tracking. Derivative blocks take advantage of boundary tracking, and the method of generating and matching derivative blocks was developed by the inventors. The method adopted for the block generation and matching method was derived from a great deal of research by the inventors. When generating a derivative block, pixels on and near the boundary are not included in the block. The reason is that these pixels are greatly affected by the color of the out-of-boundary pixels, thereby adversely affecting the tracking result.

A flowchart of the block matching handler algorithm is shown in FIG.
In other words, the “obtain the point to be tracked from the previous frame” step, the “generate neighboring block” step, the “use block matching algorithm to determine the optimal point on the current frame” step, the “block matching algorithm” Search in all directions, the search limit for the first point is 20 pixels, the search limit for the other points is 2 pixels, ”step,“ if the point displacement of the tracking result is more than 18 pixels (with respect to the first point If the answer is YES, the “continue matching with new position correction” step and the “search for all directions using the block matching algorithm. Search limit for points is 20 pixels, etc. Search limits against to point to step into 2 pixels. "When NO, the the" tracking end of the target point ".

  The block matching handler algorithm generates a block composed of pixels near the point on the tracking target polygon of the previous frame. The block near the point is matched on the current frame using a block matching algorithm. If the target point is the first point on the polygon of the previous frame, this point is also used on the current frame without position correction. Point neighboring blocks are subjected to matching processing within a maximum of 20 pixels in all four directions starting from the corresponding point position on the current frame. If the target point is not the first point on the polygon of the previous frame, the point neighboring block is subjected to matching processing within a maximum of 2 pixels in all four directions starting from the corresponding point position on the current frame.

  This algorithm is the core algorithm of the matching process. Tracking on the current frame is performed for simple blocks and derivative blocks at each point (control point) of the polygon on the previous frame. The input in the block matching process is the block pixel of the previous frame that is referred to in the matching on the current frame, and the output is the coordinates of the optimum matching point position on the current frame. The tracking search area on the current frame is wide for the first polygon point. In order to shorten the tracking processing time, in the tracking of the second and subsequent polygon points, the displacement of the first polygon point is corrected and used. FIG. 9 shows block matching on the current frame.

  In the two frames shown in FIG. 9, frame 1 (A diagram) is a previous frame in which a tracking target object is defined, and frame 2 (B diagram) is a current frame in which the target object is generated by tracking processing. In frame 1, a part of the polygon indicating the target object is displayed. Circles P1, P2, and P3 indicate polygon points. P1 is the first polygon point to be tracked. P1 is matched in a wide area (20 pixels square) on the current frame. For polygon points other than P1, the position of the displacement of P1 is used after being corrected, so that matching is performed in a narrower region (two pixels square). Matching is performed for all four directions in the search area. This search area is shown as a rectangular area near the polygon point in frame 2. The new P1 of frame 2 is the position obtained by the tracking result for the first polygon point P1. Candidates P2 and P3 are candidate positions on the current frame corresponding to P2 and P3 on the previous frame after correcting the displacement of P1. For these polygon points, the search area for positioning is narrow.

  The simple block and derivative block formed on the previous frame are placed on the current frame, and the difference between the block pixel on the previous frame and the block pixel on the current frame is calculated. The sum of the color difference between the simple blocks and the color difference between the derivative blocks is obtained, and the position where the sum is minimum is set as the position of the corresponding polygon point on the current frame. The following equation for calculating the sum was determined based on empirical rules.

Sum of color differences = color difference between derivative blocks +
(Color difference between simple blocks x Color difference between simple blocks)

  In simple block matching, a color difference between a block pixel on the previous frame and a corresponding pixel on the current frame is calculated. Derivative block matching algorithms are designed to take advantage of the fact that color values vary greatly across boundaries. Since the object region and the adjacent region are often separated by an identifiable boundary, the boundary provides extremely important information regarding tracking.

  FIG. 10 shows the X coefficient (A) and the Y coefficient (B) of the derivative mask used in the Mask Tracker application.

  Small errors occur as tracking on each frame is repeated. This error is not a problem as long as the number of frames to be tracked is small, but the shape collapse due to error accumulation cannot be ignored as many frames are processed. It is also important to maintain the object's geometric shape while performing accurate tracking. In the present invention, a technique called a shape correction algorithm is introduced to solve this problem. The shape correction algorithm improves the tracking result. Find the interior angle of the polygon on the previous frame. The interior angle of the polygon on the current frame obtained as a result of tracking is also obtained by calculation. Calculate whether there is a difference between the interior angle of the polygon on the previous frame and the interior angle of the polygon on the current frame. The difference is minimized by moving the polygon point on the current frame to a position that minimizes the difference in the inner angle with respect to the point having the largest difference in the inner angle. The polygon point to be moved in order to minimize the difference in interior angle is the amount of movement within a range that is consistent with the block matching result. This corrects the polygon shape on the current frame.

  The tracking target object does not always move only without rigid deformation. The object area may be slightly deformed. Therefore, when matching with the current frame, the block shape on the previous frame should not be compared as invariant. In order to solve this problem, a new concept of “deformation” and “deformation limit” was introduced into the block matching algorithm. The deformation process is performed as follows. For each block pixel on the previous frame, in addition to matching with a corresponding pixel on the current frame, matching is also performed with a pixel adjacent to that pixel. The number of adjacent pixels is determined by the deformation limit to be applied. A circle 7 in FIG. 11 is a block pixel used for matching. In this example, the porosity of this block is 3. The round pixel 7 on the block (A) of the previous frame is matched with the round pixel 7 of the corresponding block (B) on the current frame. The round pixel 7 on the previous frame is further matched with the neighboring pixel 8 in addition to the comparison with the corresponding block pixel 7 on the current frame. That is, each block pixel on the previous frame is matched with the 9th pixel on the current frame.

  Ideally, if the object to be tracked is defined on the first frame, the Mask Tracker application should track to the last frame. But in practice it is impossible for a number of reasons. For example, the Mask Tracker application tracks a target region by following a frame, but a small error that is difficult to recognize occurs between frames before and after. This error is accumulated as the frame is followed, and finally there is a level difference that can be clearly recognized. The keyframe concept was introduced as a way to deal with this problem.

  The user defines the tracking target object on several frames in advance. These frames are key frames. Frames that are likely to cause tracking problems are set as key frames. The following cases are likely to cause problems.

(1) When the object moves suddenly or very fast (2) When the camera moves greatly (3) When multiple objects overlap (4) When the lighting conditions change When a key frame is reached during tracking, The Mask Tracker application performs tracking using a predefined object polygon on a key frame. When the key frame is reached, the error accumulation is cancelled. This approach saves processing time and allows tracking batch processing because it is not necessary to monitor whether tracking is progressing without problems.

  Repeated extremely fast movements exist in practice. In such a case, there is a problem in tracking ability in sequential tracking in which the current frame is tracked based only on the information of the previous frame. As a countermeasure for such a case, a method has been developed in which a frame indicating a position that can be taken by the target object is registered in advance as a key frame (reference frame). Current frame tracking is performed on all registered reference frames, and the polygon on the reference frame that produced the best result among them is selected as a candidate polygon on the current frame. In this tracking process, the entire polygon internal area is to be compared. Further, as a final adjustment, a shape that minimizes the color difference in the entire region comparison is obtained by executing parallel movement, rotation, and offset on the current frame in the vertical and horizontal directions with respect to the polygon candidate obtained by the above processing.

Block diagram of the Mask Mask Tracker system Flow chart of Mask Mask Tracker system Mask Tracker handler algorithm flowchart Frame tracking algorithm flowchart Illustration of simple block generated near polygon point Illustration of derivative block generated near polygon point Explanatory drawing of porous block Block matching handler algorithm flowchart Illustration of block matching Illustration of X and Y coefficients of derivative mask Explanatory drawing of deformation limit

Claims (28)

The user designates the object area to be tracked on the first frame, tracks the designated area (object) for the subsequent frame, determines its position on the subsequent frame, and sets the new position of the obtained designated area. Based on this, the position of the designated area (object) is determined by tracking for the next frame, this tracking process is continuously executed for the subsequent frames, and the object on each frame obtained in this way is obtained. A mosaic processing method for an object to be subjected to the mask processing specified for
In the matching process that determines where the polygon points that are the points of the polygons generated by the points (control points) generated by mouse clicks are located in the next frame from the previous frame , both region-based tracking and boundary-based tracking It has a concept and uses two types of blocks: simple blocks for simple block matching and derivative blocks for derivative block matching. Simple block generation and simple block matching use the advantages of area tracking. derivative block utilizes the advantages of the boundary tracking, in generating the derivative block, mosaic processing characteristics and to Luo objects that pixel boundaries on and near the boundary is not included in the block Method. "Read input AVI (video standard format) file" step, "Display first frame" step, "Create polygon in specified area" step, "Track specified area on subsequent frame" step, "Get by tracking" A mask processing on a designated area (mosaic generation, clipping) ”step and“ save AVI file with mask processing added ”steps,
In the matching process that determines where the polygon points that are the points of the polygons generated by the points (control points) generated by mouse clicks are located in the next frame from the previous frame , both region-based tracking and boundary-based tracking It has a concept and uses two types of blocks: simple blocks for simple block matching and derivative blocks for derivative block matching. Simple block generation and simple block matching use the advantages of area tracking. The derivative block uses the advantage of boundary tracking, and when generating a derivative block, pixels on and near the boundary are not included in the block.
Mosaic processing method of Luo object. The user designates the object area to be tracked on the first frame, tracks the designated area (object) for the subsequent frame, determines its position on the subsequent frame, and sets the new position of the obtained designated area. Based on this, the position of the designated area (object) is determined by tracking for the next frame, this tracking process is continuously executed for the subsequent frames, and the object on each frame obtained in this way is obtained. A mosaic processing method for an object to be subjected to the mask processing specified for
The object to be tracked may be slightly deformed in the area, and the shape change should be taken into account when tracking, and the block shape on the previous frame is not changed when matching with the current frame. without comparing as, being defined, "variant" and mosaic processing method features and to Luo objects that the concept of "deformation limit" was introduced to the block matching algorithm below.
[Deformation and deformation limit]
For each block pixel on the previous frame, in addition to matching with the corresponding pixel on the current frame, it also matches the pixel adjacent to that pixel, and the number of adjacent pixels depends on the applicable deformation limit. The block porosity of the block pixel used for matching is set to 2 or more, the round pixel on the block of the previous frame is matched with the round pixel of the corresponding block on the current frame, and the round pixel on the previous frame is In addition to the comparison with the corresponding block pixel, it is also matched with neighboring pixels. "Read input AVI (video standard format) file" step, "Display first frame" step, "Create polygon in specified area" step, "Track specified area on subsequent frame" step, "Get by tracking" A mask processing on a designated area (mosaic generation, clipping) ”step and“ save AVI file with mask processing added ”steps,
The object to be tracked may be slightly deformed in the area, and the shape change should be taken into account when tracking, and the block shape on the previous frame is not changed when matching with the current frame. without comparing as, being defined, "variant" and mosaic processing method features and to Luo objects that the concept of "deformation limit" was introduced to the block matching algorithm below.
[Deformation and deformation limit]
For each block pixel on the previous frame, in addition to matching with the corresponding pixel on the current frame, it also matches the pixel adjacent to that pixel, and the number of adjacent pixels depends on the applicable deformation limit. The block porosity of the block pixel used for matching is set to 2 or more, the round pixel on the block of the previous frame is matched with the round pixel of the corresponding block on the current frame, and the round pixel on the previous frame is In addition to the comparison with the corresponding block pixel, it is also matched with neighboring pixels. The user designates the object area to be tracked on the first frame, tracks the designated area (object) for the subsequent frame, determines its position on the subsequent frame, and sets the new position of the obtained designated area. Based on this, the position of the designated area (object) is determined by tracking for the next frame, this tracking process is continuously executed for the subsequent frames, and the object on each frame obtained in this way is obtained. A mosaic processing method for an object to be subjected to the mask processing specified for
When tracking the target area by following the frame, a small error that is difficult to recognize occurs between the previous and next frames, and this error is accumulated as the frame is followed, and finally there is a difference in level that can be clearly recognized. mosaic processing method Luo objects to process using the concept of key frame defined below as a method to deal with.
[Key frame]
The user defines objects to be tracked in advance on several frames (key frames), and sets frames that are likely to cause the following problems in tracking as key frames.
(1) When the object moves rapidly or very fast (2) When the camera moves greatly (3) When multiple objects overlap (4) When the lighting situation changes When a key frame is reached during tracking, Tracking is performed using a defined object polygon on the key frame, and when it reaches the key frame, the error accumulation is canceled and the current frame is tracked for all registered key frames. Is executed, and the polygon on the key frame that produced the best result is selected as a candidate for the polygon on the current frame. , Up, down, left, and right on the current frame for the polygon candidate obtained by the above process Translation, rotation, determine the shape that minimizes the differences in color in comparison entire area running offset. "Read input AVI (video standard format) file" step, "Display first frame" step, "Create polygon in specified area" step, "Track specified area on subsequent frame" step, "Get by tracking" A mask processing on a designated area (mosaic generation, clipping) ”step and“ save AVI file with mask processing added ”steps,
When tracking the target area by following the frame, a small error that is difficult to recognize occurs between the previous and next frames, and this error is accumulated as the frame is followed, and finally there is a difference in level that can be clearly recognized. mosaic processing method Luo objects to process using the concept of key frame defined below as a method to deal with.
[Key frame]
The user defines objects to be tracked in advance on several frames (key frames), and sets frames that are likely to cause the following problems in tracking as key frames.
(1) When the object moves rapidly or very fast (2) When the camera moves greatly (3) When multiple objects overlap (4) When the lighting situation changes When a key frame is reached during tracking, Tracking is performed using a defined object polygon on the key frame, and when it reaches the key frame, the error accumulation is canceled and the current frame is tracked for all registered key frames. Is executed, and the polygon on the key frame that produced the best result is selected as a candidate for the polygon on the current frame. , Up, down, left, and right on the current frame for the polygon candidate obtained by the above process Translation, rotation, determine the shape that minimizes the differences in color in comparison entire area running offset. The tracking target area is defined by polygons generated by mouse click points (control points), and tracking in subsequent frames is performed for the area specified by the user. Tracking is performed for each point on the polygon. When the point on the previous frame polygon is tracked with respect to the current frame, the point coordinate value on the previous frame is used as the initial point coordinate value on the current frame. Generate a block near the polygon point for all points in
The same block also creates a corresponding point candidate positions on the current frame, any claim 1-6, characterized in that the position of the corresponding points on the current frame by matching comparison on both blocks are determined mosaic processing method of the object according to any. The object mosaic processing according to any one of claims 1 to 7 , wherein the point neighborhood block includes only pixels in the polygon area (in the designated object area), and avoids an influence outside the polygon area. Method. A matching process that determines where the polygon point in the previous frame is located in the next frame is created for the point neighborhood block created for the point candidate position in the next frame and the corresponding point in the previous frame. This determination is made by comparing whether the neighboring blocks at the same point are the same, and this determination is made by comparing the RGB values of the pixels at the corresponding positions in both blocks, so that all the pixels in the block on the next frame are If the RGB value is the same as the corresponding pixel of the block on the frame, then both blocks are perfectly matched (similar determination is made for the derivative block), and the degree of this matching is the reference determination value. If you are not satisfied, shift the point candidate position on the next frame. Performed again similar determination, mosaic processing method of an object according to any one of claims 1 to 8, characterized in that repeated until the process point position on the next frame determined. The matching process has the concept of both region-based tracking and boundary-based tracking, and uses two types of blocks: simple block for simple block matching and derivative block for derivative block matching. Block matching uses the advantage of region tracking, derivative block uses the advantage of boundary tracking, and when generating a derivative block, the pixels on and near the boundary should not be included in the block The method for mosaic processing an object according to any one of claims 3 to 9 . The size of the block is determined based on the size of the target area polygon, does not cover the entire target area, and the concept of the porous block defined below is introduced for the block near the point. The entire pixels of the above are not treated as block pixels, but only the remaining pixels obtained by thinning out some pixels with the following specific thinning rate (porosity) are processed as block pixels. The method for mosaic processing an object according to any one of 10 .
[Thinning rate (porosity)]
The thinning rate (porosity) is determined in proportion to the average polygon width, and the porosity increases as the polygon width increases. For large polygons with a wide polygon width, a larger block size is required to improve tracking accuracy. Although necessary, since the porosity increases, the number of block pixels does not increase so much, and pixels outside the boundary are excluded during block generation, and pixels within the boundary are selected discretely. The object to be tracked may be slightly deformed in the area, and the shape change should be taken into account when tracking, and the block shape on the previous frame is not changed when matching with the current frame. The object according to claim 1 , 2 or 5 to 11 , wherein the concept of “deformation” and “deformation limit” defined below is introduced into the block matching algorithm. Mosaic processing method.
[Deformation and deformation limit]
For each block pixel on the previous frame, in addition to matching with the corresponding pixel on the current frame, it also matches the pixel adjacent to that pixel, and the number of adjacent pixels depends on the applicable deformation limit. The block porosity of the block pixel used for matching is set to 2 or more, the round pixel on the block of the previous frame is matched with the round pixel of the corresponding block on the current frame, and the round pixel on the previous frame is In addition to the comparison with the corresponding block pixel, it is also matched with neighboring pixels. A database tracking method has been introduced in which frames defined in advance for various shapes that can be taken by the target object are registered as reference frames, and tracking of the current frame is executed for all registered reference frames. in the polygon on the reference frame produced the best results is selected as the polygon candidate on the current frame, claim 1-12, characterized in that it further tracking process with respect to the polygon candidate is executed The mosaic processing method of the object described in 1. When tracking the target area by following the frame, a small error that is difficult to recognize occurs between the previous and next frames, and this error is accumulated as the frame is followed, and finally there is a difference in level that can be clearly recognized. mosaic processing method of an object according to claim 1 or 7 to 13, characterized in that treatment with the concept of key frame defined by the following as a way of handling.
[Key frame]
The user defines objects to be tracked in advance on several frames (key frames), and sets frames that are likely to cause the following problems in tracking as key frames.
(1) When the object moves rapidly or very fast (2) When the camera moves greatly (3) When multiple objects overlap (4) When the lighting situation changes When a key frame is reached during tracking, Tracking is performed using a defined object polygon on the key frame, and when it reaches the key frame, the error accumulation is canceled and the current frame is tracked for all registered key frames. Is executed, and the polygon on the key frame that produced the best result is selected as a candidate for the polygon on the current frame. , Up, down, left, and right on the current frame for the polygon candidate obtained by the above process Translation, rotation, determine the shape that minimizes the differences in color in comparison entire area running offset. In a mosaic processing apparatus for an object having a processor, a user designates an object area to be tracked on the first frame and tracks the designated area (object) for the subsequent frame to determine its position on the subsequent frame. Based on the new position of the designated area obtained, the position of the designated area (object) is determined by tracking for the next frame, and this tracking process is continuously executed for the subsequent frames. An object mosaic processing apparatus having a processor configured to perform a specified mask process on an object on each frame obtained as described above,
In the matching process that determines where the polygon points that are the points of the polygons generated by the points (control points) generated by mouse clicks are located in the next frame from the previous frame , both region-based tracking and boundary-based tracking It has a concept and uses two types of blocks: simple blocks for simple block matching and derivative blocks for derivative block matching. Simple block generation and simple block matching use the advantages of area tracking. The derivative block uses the advantage of boundary tracking. When generating a derivative block, it is necessary to have a processor configured to perform processing that does not include pixels on and near the boundary. Mosaic processing unit of butterflies and to Luo object. In the mosaic processing apparatus for an object having a processor, the processor performs an “input AVI (video standard format) file reading” step, a “display first frame” step, a “polygon creation in a designated area” step, and a “following frame on” The object is configured to perform the steps of “tracking specified area”, “mask processing (mosaic generation, clipping) on specified area obtained by tracking”, and “save AVI file with mask processing”. A mosaic processing apparatus,
In the matching process that determines where the polygon points that are the points of the polygons generated by the points (control points) generated by mouse clicks are located in the next frame from the previous frame , both region-based tracking and boundary-based tracking It has a concept and uses two types of blocks: simple blocks for simple block matching and derivative blocks for derivative block matching. Simple block generation and simple block matching use the advantages of area tracking. The derivative block uses the advantage of boundary tracking. When generating a derivative block, it is necessary to have a processor configured to perform processing that does not include pixels on and near the boundary. Mosaic processing unit of butterflies and to Luo object. In a mosaic processing apparatus for an object having a processor, a user designates an object area to be tracked on the first frame and tracks the designated area (object) for the subsequent frame to determine its position on the subsequent frame. Based on the new position of the designated area obtained, the position of the designated area (object) is determined by tracking for the next frame, and this tracking process is continuously executed for the subsequent frames. An object mosaic processing apparatus having a processor configured to perform a specified mask process on an object on each frame obtained as described above,
The object to be tracked may be slightly deformed in the area, and the shape change should be taken into account when tracking, and the block shape on the previous frame is not changed when matching with the current frame. without comparing as defined below, "variant" and mosaic processing device characteristics and to Luo object that has a processor configuration in which introduced the concept of "deformation limit" to block matching algorithm.
[Deformation and deformation limit]
For each block pixel on the previous frame, in addition to matching with the corresponding pixel on the current frame, it also matches the pixel adjacent to that pixel, and the number of adjacent pixels depends on the applicable deformation limit. The block porosity of the block pixel used for matching is set to 2 or more, the round pixel on the block of the previous frame is matched with the round pixel of the corresponding block on the current frame, and the round pixel on the previous frame is In addition to the comparison with the corresponding block pixel, it is also matched with neighboring pixels. In the mosaic processing apparatus for an object having a processor, the processor performs an “input AVI (video standard format) file reading” step, a “display first frame” step, a “polygon creation in a designated area” step, and a “following frame on” The object is configured to perform the steps of “tracking specified area”, “mask processing (mosaic generation, clipping) on specified area obtained by tracking”, and “save AVI file with mask processing”. A mosaic processing apparatus,
The object to be tracked may be slightly deformed in the area, and the shape change should be taken into account when tracking, and the block shape on the previous frame is not changed when matching with the current frame. without comparing as defined below, "variant" and mosaic processing device characteristics and to Luo object that has a processor configuration in which introduced the concept of "deformation limit" to block matching algorithm.
[Deformation and deformation limit]
For each block pixel on the previous frame, in addition to matching with the corresponding pixel on the current frame, it also matches the pixel adjacent to that pixel, and the number of adjacent pixels depends on the applicable deformation limit. The block porosity of the block pixel used for matching is set to 2 or more, the round pixel on the block of the previous frame is matched with the round pixel of the corresponding block on the current frame, and the round pixel on the previous frame is In addition to the comparison with the corresponding block pixel, it is also matched with neighboring pixels. In a mosaic processing apparatus for an object having a processor, a user designates an object area to be tracked on the first frame and tracks the designated area (object) for the subsequent frame to determine its position on the subsequent frame. Based on the new position of the designated area obtained, the position of the designated area (object) is determined by tracking for the next frame, and this tracking process is continuously executed for the subsequent frames. An object mosaic processing apparatus having a processor configured to perform a specified mask process on an object on each frame obtained as described above,
When tracking the target area by following the frame, a small error that is difficult to recognize occurs between the previous and next frames, and this error is accumulated as the frame is followed, and finally there is a difference in level that can be clearly recognized. mosaic processing device characteristics and to Luo objects that have a processor configured to process using the concept of key frame defined by the following as a way of handling.
[Key frame]
The user defines objects to be tracked in advance on several frames (key frames), and sets frames that are likely to cause the following problems in tracking as key frames.
(1) When the object moves rapidly or very fast (2) When the camera moves greatly (3) When multiple objects overlap (4) When the lighting situation changes When a key frame is reached during tracking, Tracking is performed using a defined object polygon on the key frame, and when it reaches the key frame, the error accumulation is canceled and the current frame is tracked for all registered key frames. Is executed, and the polygon on the key frame that produced the best result is selected as a candidate for the polygon on the current frame. , Up, down, left, and right on the current frame for the polygon candidate obtained by the above process Translation, rotation, determine the shape that minimizes the differences in color in comparison entire area running offset. In the mosaic processing apparatus for an object having a processor, the processor performs an “input AVI (video standard format) file reading” step, a “display first frame” step, a “polygon creation in a designated area” step, and a “following frame on” The object is configured to perform the steps of “tracking specified area”, “mask processing (mosaic generation, clipping) on specified area obtained by tracking”, and “save AVI file with mask processing”. A mosaic processing apparatus,
When tracking the target area by following the frame, a small error that is difficult to recognize occurs between the previous and next frames, and this error is accumulated as the frame is followed, and finally there is a difference in level that can be clearly recognized. mosaic processing device characteristics and to Luo objects that have a processor configured to process using the concept of key frame defined by the following as a way of handling.
[Key frame]
The user defines the tracked object on several frames in advance (
Key frames), frames that are likely to cause the following problems in tracking are set as key frames.
(1) When the object moves rapidly or very fast (2) When the camera moves greatly (3) When multiple objects overlap (4) When the lighting situation changes When a key frame is reached during tracking, Tracking is performed using a defined object polygon on the key frame, and when it reaches the key frame, the error accumulation is canceled and the current frame is tracked for all registered key frames. Is executed, and the polygon on the key frame that produced the best result is selected as a candidate for the polygon on the current frame. , Up, down, left, and right on the current frame for the polygon candidate obtained by the above process Translation, rotation, determine the shape that minimizes the differences in color in comparison entire area running offset. The tracking target area is defined by polygons generated by mouse click points (control points), and tracking in subsequent frames is performed for the area specified by the user. Tracking is performed for each point on the polygon. When the point on the previous frame polygon is tracked with respect to the current frame, the point coordinate value on the previous frame is used as the initial point coordinate value on the current frame. A block is generated in the vicinity of the polygon point for all the points of, and the same block is also created at the corresponding point candidate position on the current frame. Characterized by having a processor configured to determine Mosaic processing apparatus object according to claim 1 5 to 20. Point near the block, a polygon area and consists only pixel (specified object region), in any one of claims 1 5 to 21, characterized in that it comprises a processor arrangement to avoid the influence of outside polygonal area The object mosaic processing apparatus. A matching process that determines where the polygon point in the previous frame is located in the next frame is created for the point neighborhood block created for the point candidate position in the next frame and the corresponding point in the previous frame. This determination is made by comparing whether the neighboring blocks at the same point are the same, and this determination is made by comparing the RGB values of the pixels at the corresponding positions in both blocks, so that all the pixels in the block on the next frame are If the RGB value is the same as the corresponding pixel of the block on the frame, then both blocks are perfectly matched (similar determination is made for the derivative block), and the degree of this matching is the reference determination value. If you are not satisfied, shift the point candidate position on the next frame. Performed again similar determination, mosaic processing apparatus object according to any of claims 1 5-22, characterized in that a processor of the repeated configuration until the process point position on the next frame determined. The matching process has the concept of both region-based tracking and boundary-based tracking, and uses two types of blocks: simple block for simple block matching and derivative block for derivative block matching. Block matching uses the advantage of region tracking, and derivative blocks use the advantage of boundary tracking. When generating a derivative block, the pixels on and near the boundary are not included in the block. mosaic processing apparatus object according to any of claims 1 7-23, characterized in that it has a processor configured to perform. The size of the block is determined based on the size of the target area polygon, does not cover the entire target area, and the concept of the porous block defined below is introduced for the block near the point. It is characterized by having a processor configured to process only the remaining pixels obtained by thinning out some pixels with the following specific thinning-out rate (porosity) as block pixels, instead of treating all the pixels as block pixels. mosaic processing apparatus object according to any of claims 1 5 to 24 to.
[Thinning rate (porosity)]
The thinning rate (porosity) is determined in proportion to the average polygon width, and the porosity increases as the polygon width increases. For large polygons with a wide polygon width, a larger block size is required to improve tracking accuracy. Although necessary, since the porosity increases, the number of block pixels does not increase so much, and pixels outside the boundary are excluded during block generation, and pixels within the boundary are selected discretely. The object to be tracked may be slightly deformed in the area, and the shape change should be taken into account when tracking, and the block shape on the previous frame is not changed when matching with the current frame. without comparing as defined below, "deformation" and according to claim 1, 5, 16 or 19 to 25, characterized in that it comprises a processor configuration introduced into "deformation limit" block matching algorithm the concept of An object mosaic processing apparatus according to any one of the above.
[Deformation and deformation limit]
For each block pixel on the previous frame, in addition to matching with the corresponding pixel on the current frame, it also matches the pixel adjacent to that pixel, and the number of adjacent pixels depends on the applicable deformation limit. The block porosity of the block pixel used for matching is set to 2 or more, the round pixel on the block of the previous frame is matched with the round pixel of the corresponding block on the current frame, and the round pixel on the previous frame is In addition to the comparison with the corresponding block pixel, it is also matched with neighboring pixels. A database tracking method has been introduced in which frames defined in advance for various shapes that can be taken by the target object are registered as reference frames, and tracking of the current frame is executed for all registered reference frames. 2. A processor according to claim 1, further comprising a processor configured to select a polygon on a reference frame that yields the best result as a polygon candidate on the current frame, and to perform tracking processing on the polygon candidate. The mosaic processing apparatus for objects according to any one of 5 to 26 . When tracking the target area by following the frame, a small error that is difficult to recognize occurs between the previous and next frames, and this error is accumulated as the frame is followed, and finally there is a difference in level that can be clearly recognized. The object mosaic processing device according to any one of claims 15 to 18, or 21 to 27 , comprising a processor configured to perform processing using a concept of a key frame defined below as a method of coping with the object. .
[Key frame]
The user defines objects to be tracked in advance on several frames (key frames), and sets frames that are likely to cause the following problems in tracking as key frames.
(1) When the object moves rapidly or very fast (2) When the camera moves greatly (3) When multiple objects overlap (4) When the lighting situation changes When a key frame is reached during tracking, Tracking is performed using a defined object polygon on the key frame, and when it reaches the key frame, the error accumulation is canceled and the current frame is tracked for all registered key frames. Is executed, and the polygon on the key frame that produced the best result is selected as a candidate for the polygon on the current frame. , Up, down, left, and right on the current frame for the polygon candidate obtained by the above process Translation, rotation, determine the shape that minimizes the differences in color in comparison entire area running offset.

Images