TW460813B - Apparatus and method for context-based indexing and retrieval of image sequences - Google Patents

Abstract

An apparatus (150) and method for implementing object motion segmentation and object trajectory segmentation for an image sequence. Specficially, block-based motion vectors for a pair of adjacent frames are used to derive optical flow, e.g., affine, motion parameters. Such optical flow motion parameters are employed to determine key objects where their motion and trajectory within a sequence of frames are calculated and stored. Such object motion information is used to improve or offer image processing functions such as context-based indexing of the input image sequence by using motion-based information.

Description

460813 Five 'Invention Description (1) This application claims US 60/1 1 7, 629 filed on January 28, 999, which is listed here for reference only. The present invention relates to the sequential indexing of images such as images and the processing of images. To be more specific, the present invention is about emphasizing efficient structure, background of the invention, and indexing of indexing and manipulation of moving sequences based on this article. With the increase of available multimedia content such as audiovisual content, the need for long and complex information organization and management becomes important. =, When the library of media content continues to grow, it becomes difficult ^ This highly complex information is indexed in order to help effectively in later time by taking the most descriptive symbols describing multimedia content ^ The content that appears in the database You can find out, so that the search interface is this interface standard *, and this sector = (索 = 2 international standard provides will be used with filtering media. This new member of the _G standard 丨: 丄 丨 Engine, content description interface, Moreover, the code name is "MpEG_7", and the nickname is "Multimedia. For example, a typical sequence of an image" is obtained by "shooting several images". "In-picture :: Continuous images in image trimming by dividing the sequence into pieces Frame, image and photo, meaning is to describe a sudden scene change in one thing, wipe, or fade away ^ such as a mix, before and after dissolving. Detection of the boundary of image shooting 2 ^ special effect scene change is like clipping, so it constitutes A random access to the event towards the internal event. As long as an image capture is detected, it is called " the first step of choosing to browse to capture the evolution of the event, such as the representative frame of the frame " Yes, 'The main frame can be identified to indicate a 460813. V. Description of the invention (2) Add scenes, and process an image along some of the main frames. The scenes can be manipulated without mentioning (for example, that is, the organization can describe It is regarded as supportable from a single content low-main frame for a simple movement. In addition to "" with a symbol search engine, a description is provided from a variety of perspectives to support a wide range of applications. Therefore, in the image sequence The complex problem of a moving segmentation device based on the present invention and a motion-based text-based motion tracking scene scene, a romantic scene sequence of many image frames, such as the color and level from a still image Existing knowledge subject's meta-data in abstraction. Single solution to extract meta-data when meta-data is retrieved. Mobile information can be quite extended. In addition, queries can have verb-based and mobile information. Known additional information about interrelated colors is beneficial. Uses scene dynamics to more intelligently describe symbols about object movement information. The same perspective Scenery Leap Quickly analyze the scenery of the map and store only the corresponding post-design materials unexpectedly. For example, this is very important in the example, where a need that is not always present is a device and method for capturing mobile information. In this way, specific embodiments of indexing and manipulating and various editing functions are used to implement the segmentation of an image object track—the information on the device side to improve or provide other image processing functions such as input indexing, and so on. This simplifies, but as long as the shape, organization and then can provide, the above big inquiry ", the shape, to pass. Favorable descriptors to obtain information, if the description can be expected, the content asks the scope, and is related to sending It is the last-generation analysis of the supplement number to take this capability and monitor the inquiry = and describe the improvement in the film such as the image processing of the book. Sequential Object Shift ’which allows you to use the image sequence

ί 460813 i. Description of the invention (3) ---. Block-based motion vectors are used to obtain, for example, affine motion parameters for flow motion parameters. To be clear, the first rows of optical flow (for example, affine) object moving segments are for a pair of contiguous frames. The affine movement parameters can then be used to determine the main objects identified in each frame. These main objects can then be supervised over certain intervals in the sequence of images (also known as "images" with many input image sequence frames), and their movement information; at intervals of time Acquisition and tracking. 5 Then, optical flow (eg, affine) orbit segmentation is performed on the image sequence. Specifically, the affine movement parameters generated by each adjoining pair of main objects identified by the frame will be It is processed in a section of the image sequence, and it takes effect in segments of object orbits. That is, for example, the movement of direction, velocity, and acceleration, the orbit can infer each major object in some frame intervals, thereby providing the month b δ Xunwen uses another aspect of mobile information. Schematic description of the invention The description of the present invention can be considered by considering the following detailed description and drawings, of which: Figure 1 depicts a text server of the present invention. Block diagram; Figure 2 depicts a text-based indexing program block diagram of the present invention; Figure 3 depicts a text-based indexing method that implements an image sequence by using mobile-based information Flow diagram; Figure 4 depicts a method flow diagram to implement optical flow (eg, affine) object movement; Figure 5 depicts a method to implement optical flow (eg, affine) object movement segmentation

460813 V. Description of the invention (4) A more detailed method flow diagram; and FIG. 6 depicts one of a plurality of connected regions. To help understand, use a photo frame. The numbers are the same in the drawings:: Reference numerals, where these references are shown in Figure 1. In one embodiment of the present invention, the feeder 100 herein is illustrated by means of a block diagram. On a concrete level. Therefore, the server described in this article is a general purpose computer and implemented) 140, such as a random access memory-processor (Central Processing Unit Qi Yegan a) a memory 120, a RAM, Indexing program 150 based on this article, a selective encoder ⑴ different input / output devices 130, (e.g. each kind of sound recorder,-camera,-hand-held camera, a video monitor-any number of video devices Or storage devices, including but not limited to a tape drive, a floppy disk drive, a hard disk drive, or a laser drive.) It should be understood that the encoder 110 and the indexing program 15 based on this article 〇 Can be implemented jointly or individually. That is, the "encoder 110" and the index-based program 15 based on this article can be coupled to the actual device of the central processing unit 1 through a communication channel. Or, encoding The device 110 and the indexing program 150 based on this text can be represented by one or more software applications (or even, for example, by using a combination of software and hardware of a special application integrated circuit (ASIC)), which makes the software from a The storage medium is loaded (for example, a magnetic or optical disk drive or a magnetic disk drive), and the parallelism is operated by the central processing unit in the electric memory 120. Similarly, the encoder 110 and the present invention are based on this document. Index program 15〇 (package

(Including combined data structures) can be stored on a computer-readable medium, such as a machine access memory, magnetic or optical drive, or magnetic disk. In terms of implementation, various multimedia information is received on the path 105 'and stored in a storage device 13 of the server 100 of this document. Multi-body information may include, but is not limited to, a variety of different video sequences such as complete movies, film cuts, shootings, advertising clips, music videos, etc. = sequence may or may not include audio or data streams, such as completed plus ^ screen Due to the increase in available multimedia content and their larger size, the input information on Route 05 can be interpolated as one or more selective encoders. Ι = These encoders U0 contain, for example, similar MPEG encoders. Scene J and 耷 s encoders, and these similar liver 跖 encoders in the leaf < reduce " and temporary redundancy '; however, any number of compression methods are clear and not limited to any special method. Since the input The various types of compression that have been performed outside the server of this document are optional, where the input stream is already in a contracted format. In this implementation, the 'encoder U0 can be omitted., == ί = Πΐί150 is a large and often complex multimedia used to analyze the information lost in the eighth; in: ίί: i3 ° stored basis of the code! 丨 program 丨 5G is to provide a method that allows this article to be used as a reference and a combined data structure. // 2 ·, The method of efficiency method is quickly accessed on path 195. It is more clear that the miscellaneous / media content can be hindered by the index program 15. It is confirmed by the use of mobile knives, and this article is currently based on Kirigon ' To allow the use of "verbs" (eg.

4 6〇813-: ----- V. Description of the invention (6) More complicated inquiry about mobile information) to replace only the gang (for example, the color of the object). r ^ t ′ contains, for example, the image sequence of a blue background with a blue sky. The second door is as follows: ▲ Doming's inquiry, thereby reducing the effectiveness of the inquiry function. Right ", where-Wushe County modified the response in the order of" view ": can = speed and move to the left in the foreground, then an image m of the speed-moving plane inquired crosses the blue sky background. And the program 15 ° includes an object moving segmenter 16〇θ object track cutter 170. In short, the object moving segmentation cry 1Rn 疋 is used to widely determine the object moving segmenter in each frame] The 6 ° object track segmentation is used to broadly determine the objects 许多 in the many frames of phase two shift two =. * Scene / image order 陴 = 2 description includes object moving segmenter 16 and object 7: ^ One is an indexing program based on this article. Fifteen: The main object separator 216, and an optical stream (for example, a simulator: 18. The object track segmenter 17 includes a main object track segmentation knife. 220 and a sub-object implement the segmenter 222. The functions rewarded by these modules are briefly described in Figure 2. A detailed description of these functions is provided in the broad flow chart and other diagrams of Figure 3-6. On the one hand, an image sequence is received in the block-based device 210, of which As to the block based motion vector as the movement of the shift

460813

5. Description of the invention (7) Information is calculated from the order of the images of each frame. However, if the server 100 in this paper has an external encoder 110, or the input image sequence already contains motion information, that is, where the motion vector is encoded in the image sequence, then the block-based motion estimator 21 can be used. Omitted. That is, block-based movement information can be retrieved from the compressed bit stream itself or provided by other modules in this article "server" 100, thereby reducing the need to calculate motion vector objects Move the segmenter 1 6 0. In turn, an 'optical flow (e.g.,' affine ') segmenter 212 can provide motion vector information to generate " affine motion parameters ". Although the present invention is described below by using an affine motion model, it should be understood that other optical flow models can also be used. The affine motion model was developed by j. Nieweglówski et al. In & August 1993 " A Novel Video Coding Scheme Based 〇n

Temporal Prediction Using Digital Image Warping ", IEEE Trans. Consumer Electronics, Vol. 39, Vol. 3, page 141-15, Rev. ", which is listed here for reference. The affine motion model constructs a predicted image or frame from a previous image by applying a geometric transformation known as " Image Distortion ". The transformation specifies a spatial relationship between the previous and each point of the predicted image. The parent's use of block-matched motion compensation usually provides the ability to switch movements—good overall efficiency. However, when the motion contains a rotation or scaling element (such as' zoom or image-rotation), the block conforming to the motion estimation is a bad actuator. In contrast, the affine movement model (affine transformation) is defined by 6 parameters to 8, and is expressed as follows: 6

4 608 13 V. Description of the invention (8) Ί a4 〇 I a5 〇 I (1) a6 0 I j Guang la! [X, y, 1] = [u, v, 1] I a2-1 a3

L where (x'y) is the pixel coordinate in the previous frame, and (u, v) is the coordinate of a specific pixel in the prediction frame. A detailed discussion of determining the six parameters appears in the reference by J. Nieweglowski et al. The feature of the affine relationship is represented by 6 parameters. Therefore, affine movement models can be more effective in predictive movements such as transformation, scaling, and rotation, and these predictive movements can be observed not only from time to time in a natural order, but also by using digital effects to observe the shooting score in a comprehensive scene. The trajectory tracking of the segmenter affine parameters is important enough to identify the object. The meaning of the object in the background. As long as the main object is located at the image of each master, imitating "main objects" considers that their movement is important. The large and small objects are not the main objects. However, the movements determined by a qualified master user can then be determined by or For example, the purpose or other work of 21 2 is the identification, segmentation, and generation of each frame of the image sequence. Primary objects can be objects for indexing or other image processing functions. Typically, the main objects are based on them. That is, large objects are typically the main objects, but small. Therefore, a mobile vehicle is typically a major small moving insect and not a major object. Of course, the requirements "special applications" are also defined by the main objects of the present invention, which are tracked by the movement tracker 214. The movement information of the required object components is also very important for the editing function of the household index. The additional processing is 5. Explanation of the invention (9) is performed by the main object separator 216. Specifically, a main object can be a child object, and the movement information of these child objects can be tracked individually. For example, a main object of a human being can be divided into 6 sub-objects, which include a head, a body, and four limbs. So 'a query can now be a technique to search for " actions " related to sub-objects within a main object, such as searching for an image sequence, where a person's four limbs will rise above the head and so on. Although these main objects can be quickly divided into defined sub-objects, other main objects need further processing to identify the boundaries of the sub-objects. Therefore, the main object information can be directly transmitted from the main object tracker 214 to one of the " sub-objects " identification and segmentation of each main object, the affine segmenter 218. Therefore, the affine segmenter 218 can also interact with the affine generating sub-objects. It should be noted that although indexers based on this document describe the applicators 212 and 218, it should be understood that a single affine segmenter can perform affine processing (ie, the main object and the sub-object are rare). ^ The movement information of the main object 2 object tracker 214 will be transmitted to the main object 1 orbit segmenter 220. Although it can maintain and track the movement, for example, the affine movement parameters of each main object are required to store this movement information-substance Storage requirements. :: Information: The track information and interval (graph frame interval) that will be sent to the main object track segmenter can be produced for each main object. The gas track movement information can be summarized as " Information on object orbits; it is better to define the direction and speed of the interval (on many frames) ^ Some are based on the efficiency of multimedia content and move as fast as possible ^ This allows soil deformation or other images

1M P. 13 4 60813 V. A separate catch and storage of the invention description (ίο) processing). The mobile data interval (frame interval) transmitted to the sub-object orbital molecular segment device can be generated for each sub-object. ㈣Orbital Besson and Figure 3 describe one method used to implement affine segmentation. Flow-based assets are used in this paper. The 2 = e.g., The outline of an input image sequence is in Step 30 ST function. More specifically, the object movement segmentation is performed in step 310 'of the many frames of affine!', And one of the image capture ") :: Take :: 2: The movement information of f can be changed in these intervals. The main object is generated in step 310. The affine movement parameter can be generated for each identified t ^ ^ 320 'for each identified main object of the frame for each identified ^ shot ^ motion parameter is processed in the image sequence interval time. In order to make the object in 芊: ^ 52: Yes, for example, the direction of speed and acceleration of the moving orbit can infer each major object in time, thereby providing another view of the movement information that can be used in 3 ° 25 knot . The method 300 will then be described in step ^ 4 to implement the affine object movement segmentation-method 3 to 10 flow diagram. I7. Method 310 is a more detailed description of step 31 () of FIG. 3. Method 31〇 疋 starts at step 405, and proceeds to step a ?, where ^ a generates an affine movement parameter from the block-based movement information. A random number of blocks can be selected, and their motion vectors with blocks as the soil can be used to obtain the affine mobile parameters discussed below.

460813 V. Description of invention (11). At step 410, the method 310 attempts to trace or identify the main item from a previous frame, that is, to obtain a main object label from a previous frame. That is, as long as the main object has been identified as a pair of her connected frames as discussed in step 42, it will need to provide the same detection steps for the next image in the image sequence again. That is to say, considering & direction ", for a new frame, the moving direction based on the block = can quickly determine whether these blocks point to the main objects marked. If you want to ..., there is no way to protect it. If so, this block will retain the same label as the previous frame. For example, if there are four (4) consecutive frames, and if the next frame n is required: the item has been decided-to be late / is used to determine whether the 4 commands in the 5 connected areas of the current frame are identified with 4 Mainly against the left knee of the dagger, the * 1 I objects are related to each other. If so, the,,, and access fields are tested according to step 420 to determine if it is a -main object. The tracking method can significantly reduce the second decision, that is, the main object is the negative effect of the image sequence, for example, the object has become too small, the census is no longer tracked, and the item can be identified, step 41. It will skip, s this situation is the method of a new image capture. The time "1" will be processed in step 4 2 0, method 31 ○ · τ money: the main object in the frame. In one: the order-like basis determines whether or not the block is identified within the affine object stream. If the object is identified by the method 310 in step 430, the object may be too small to be used obviously; Π ′, that is, the index of the special application, for example: 1 = 460813 V. Description of the invention (12) The frame is merged with other birds to form the main object of a group of birds. At step 440, the method 310 may selectively identify child objects within the main object. In contrast to step 430, the identified main object combines important movement information with components (sub-objects) used to index in a particular application. For example, a main object identified by a human being includes sub-objects, that is, human limbs, and the movement information of the sub-objects is important for indexing the image capture. In step 4 50, method 3 1 0 will ask whether the extra frame is combined with the current π image capture ". If the query is answered negatively, method 3 1 0 will end at step 4 5 5. If the query is answered affirmatively, then method 3 10 proceeds to step 407, where the affine movement parameters are generated in the next pair of consecutive frames. That is, affine segmentation must be performed between successive pairs of frames. The reason is that the affine movement parameters are needed at every instant of modeled execution, 1 and also deal with new objects / occlusions. When all the frames have been processed, method 3 1 0 then ends at steps 4 5 5. FIG. 5 depicts a 500 flow diagram of a method for implementing affine object moving segmentation. More specifically, method 5 0 0 is a more detailed description of steps 4 1 0 and 4 2 0 of FIG. 4. Method 5 0 0 starts at step 5 0 5 and proceeds to step 5 1 0 where the main object will be determined. It should be noted that various methods exist to determine the area movement information within a frame, for example, optical flow techniques used to calculate each pixel movement, where these techniques are usually computationally complex. In addition, if the order is already coded (for example, as u 1 MP PEG-2 bit stream), the optical stream calculation will require full decoding of the bit stream. Therefore, in this book, affine

O: \ 62 \ 62535.ptd Page 16 4 6 0 813 V. Description of the invention (13) The movement parameters can be used in the compression field by using simple blocks that are summed up in absolute difference (on hardware or software platforms) Calculated, or obtained from the motion vector of the P frame. In a specific embodiment, the affine movement parameters can be obtained by using a Random Sampling Consistency (RANSAC) method that provides a rapid method to identify potential primary object candidates compared to a very complex thorough search. This method uses the affine stream (in other words, an affine moving segment) between every two consecutive frames of an image sequence or part of the image capture to calculate the area. That is, in step 5 1 0, the method 5 0 0 will first try to find all the "connection areas" with a frame. Any number of methods can be used to determine, for example, whether a block pixel of a giant block is part of a " connection area ". An example describing the connectivity of a block has been applied for the US project number " SAR 1 3 0 7 6 '' Name " Method And Apparatus For Generic Shape Coding " for _____ — Zhonglulu, which is only listed here For reference. For example, a giant block within a shape boundary can be seen as a connected area. In step 515, three random giant blocks are selected from this set of connected giant blocks, that is, from each connection Figure 6 depicts a frame with a plurality of connected regions 6 1 0-6 4 0. For example, the connected regions 6 1 0 and 6 3 0 are the background and foreground respectively, while the connected regions 6 2 0 and 6 4 0 Are a sailing boat and a flag respectively. In order to describe, in step 5 1 5, the three blocks described by some X are arbitrarily selected from the connection area 610, and the three blocks described by 0 are the connection area. Arbitrary selection of 620, etc. At step 5 2 0, method 5 0 0 can calculate the affine movement parameters of 3 randomly selected blocks. Specifically, these 3 selected blocks are block-based

O: \ 62 \ 62535.ptd Page 17 460813 V. Description of the invention (14) The motion vector is sufficient to infer 6 affine motion parameters. Since 3 blocks are the minimum number of blocks required to generate 6 affine motion parameters, it should be possible to use motion vectors that exceed 3 blocks from the connection area. It should be noted that this assumption is that the affine movement parameters are calculated from 3 giant blocks. At a particular movement of a triangle vertex formed by 3 blocks, 6 affine movement parameters can be calculated. That is, the affine movement parameters provide a transposition at each point within a triangle segment formed by three giant blocks. In step 5 25, the method 5 0 0 can ask whether the movement of a block is within the affine stream defined by the affine movement parameter on a block-by-block basis for each block movement in the entire frame. In other words, an affine movement parameter is provided to each giant block in the frame, and the distance between this position and the position provided by the block's movement vector is calculated. If the distance is less than a critical n e T h r e s hπ, then the giant block is added to the list of blocks as part of the connection area, that is, the part of the giant block labeled as the main object. Therefore, if the question is answered affirmatively, then the block is considered to fall in an affine stream object at step 5 35. If the query is answered in the negative, the block is considered to fall outside an affine stream object at step 530. That is, a major object may be identified or marked by all macroblocks exceeding the threshold. At step 540, the method 50 asks whether the included block is larger than a threshold of many blocks " Reg MBs ize " (e.g., 10 giant blocks). This critical " Reg Μ B size " is, for example, a special application for determining the size and resolution of the input frame. If the query is affirmative, then method 5 0 0 proceeds to step 5 5 0, where the selected or included area A block would be considered a potential "main item". If the question is answered in the negative, the selected or included block is considered too small and

O: \ 62 \ 62535.ptd Page 18 4 60813 V. Description of the invention (15) Untraceable. That is, 'if the total number of giant blocks with the same affine parameter is greater than a certain preset number of RegMBsize, then all giant blocks that fall within these parameters can provide a common label, such as the main object 1 and so on. All giant blocks with the same label constitute a new mobile area, that is, aggregated into a main object. This step ensures that the 3 vertices really belong to an actual object and that the object is not too small. > At f ϊ I! 5, method 50 asks if there are additional connected areas. If, ask; ^ Qingding answer, then the method 500 returns to step 5 丨 5 of which 3 blocks are arbitrarily selected in the connection area. Steps 515_55 are then repeated in the next connection area. If the query is answered negatively, then method 5 00 will begin the merge operation at step 5 6 0. It should be noted that previously marked as a major, the block of Ministry = is no longer evaluated in the subsequent execution of step 5 丨 5 _ 5 5 〇. That is, as long as a block in a frame is considered to be a multiplier of a main object, this block is removed from subsequent detection and consideration of other main objects, and the number of blocks can be reduced for subsequent evaluation. . Therefore, if possible, it is better to start the main object segmentation process by selecting 3 blocks from the largest connected area. At step 56 0 'a merge operation will proceed, where the two actual connections = the main object can be selected, the purpose is to use a test to determine whether the movement of the two selected areas is close, so that they can be considered as one Integral main object 'to replace two separate main objects. π Ϊ ί The three blocks of the new group can be combined from two selected connection regions, and the domains are selected. For example, if the connection area 62 and the sum are combined and estimated, then three arbitrary selected blocks can be referenced by the reference number 65.

O: \ 62 \ 62535.ptd

Page 19 460813 V. Description of Invention (16) Description. Similar to step 5 2 0, a new set of affine motion parameters is generated from the motion vectors of the newly selected set of 3 blocks. Similar to step 5 2 5, at step 5 7 0, method 5 0 0 can query whether the block movement is based on the newly generated affine movement parameters on a block-by-block basis for each block movement in the two main objects. Within the defined affine stream. In other words, the two connected main objects are compared to a generally appropriate affine, and it is determined that some percentage of the giant blocks in the two objects have error distances below eThresh. If the percentage is exceeded, the two main objects are merged into a single connected main object. That is, if the query is affirmative (reduced error), then most of the blocks are considered to fall within the newly formed affine stream object. Therefore, the two regions are merged in step 575. If the query is answered negatively (increased error), then most blocks are considered to fall outside the newly formed affine stream object. Therefore, the two regions will not merge in step 580. At step 585, method 5 0 0 asks if there are additional primary items considered to be merged. If the question is answered affirmatively, then method 5 0 returns to step 5 6 0, in which the two connected main objects can be selected, that is, steps 5 6 0-5 8 0 are then repeated until all the objects consider it necessary. Until the merger. If the query is answered negatively, then method 5 0 0 ends at step 5 9 0. In addition, it should be noted that the relevant steps of FIG. 5 are suitable for obtaining the affine movement parameters of the sub-objects identifying the main object as described above with reference to FIG. 2. Specifically, a main object can be divided into sub-objects, where method 500 can then generate affine flow sub-objects. That is, the method 5 0 0 can be provided at the level of the sub-objects that identify each of the main objects, that is, the "sub-objects" in Fig. 5 are replaced with "quote-sub-objects"

O: \ 62 \ 62535.ptd Page 20 4 60813 V. Description of the invention (17) Terms. Hierarchical organization within each main object, zone separation can be done based on zone movement between sub-zones. For example, after segmenting the entire human body as a main object, human body parts such as faces, limbs, etc. can be divided into sub-regions or a connected group of sub-objects. Affine model methods from block motion vectors, tracking, and moving orbit model programs can also be applied to these sub-regions. Therefore, each sub-region is also segment-modeled in interval time. • If two or more sub-areas belong to the same main object, their orbits can be used to identify the actions performed by that main object. Specifically, 'if further text on a main object and his subarea is provided by a manual device or analysis of their shape, color, organization, etc., then this knowledge of an organizational relationship between subareas Conditions can develop. Therefore, by analyzing the movement of the sub-area, the special action performed by the main object can be speculated. For example, if the result determines that the main object is a person (for example, through skin color detection and shape), and the sub-regions match the person's limbs, then track from different joints of the limbs, such as running, walking, lifting, etc. (see in This is a detailed list of actions below). The correspondence between objects in different frames is performed by tracking objects of obvious size in the first frame until it is no longer tracked, for example, until the object moves out of view or is permanently closed. Objects that did not have a corresponding size in the previous frame in the previous frame are the new object tags provided in the cutout, and are tracked from the frame to the future. Now that block matching has been performed, tracking can be performed by assigning an area in the previous frame to the area in the previous frame.

O: \ 62 \ 62535.ptd Page 21 4 60813 V. Invention description (18) One of the §11 areas in the group. Earlier method was achieved ', and this exercise frame is where most of the giant block movement vectors are Pointing. The above method is mainly used to capture large moving objects within an "Image Shooting". In other words, the criticality at this stage is chosen so that the movement of an object can be captured as a whole, rather than a small change within it. Since all objects are trackable, the initial giant block members and affine movement parameters between the frames are sufficient to obtain a crude composition of the movement track of each main object. Smooth segments of shifted * tracks can be modeled by using time polynomials. The method used to model the moving trajectory is performed by the object track segmenter 170, which is described briefly below. In fact, the detailed description of the "new object execution segmenter 1 70" has been described in the name " Apparatus And Method For Describing The Motion Parameters Of An Object In An Image Sequence " of U.S. Patent No. SAR 1 342 7. It is listed here for reference only and is also applied here. Object track segmenter 1 70 can be used to reduce the need for affine parameters in each frame of the image sequence. It can be fragmented by using the appropriate Ξ ΐ coefficient table ΐ 2 e. Poor objects. It is very useful to answer questions about the overall object movement. 2 For example, move quickly, up, left, right, diagonal. , Rotation, enter the z direction, and shift. Typically, the main object of the largest connection will be the back. Dynamic analysis can automatically provide for example panning, zooming, etc. ^ Wai sigh = some can be caused by either camera movement or external , * The main objects tracked by the camera can be identified by this: 2: Put on the fact that has the least change. Between & Time

V. Description of the Invention (19) In a specific embodiment, the main object can be modeled by using an oval shape. Recognized by estimating the elliptical overlap from orbital information, occlusion, and conversation with the main object. Occlusion can be distinguished from the conversation by the fact that the movement continues smoothly through the occlusion point, but object conversations often cause a change in movement. Object conversations can be situations such as connection, collision, etc., which are inherently important meta data (as a temporary point of interest). Occlusion can be used in the selection of frames, and these frames can be used for spatial analysis (to capture color, shape, and post-weave data). Typically, the smallest overlapping frame between the main objects (except the background) is ideal for this purpose, so the data can be retrieved by setting the data after an appropriate space on each main object. It should be noted that the tracking step need not be performed on segmented images, such as an MPEG-4 image object. However, the procedure of a huge block moving vector transformed by an affine model and the method of estimating a fragment's moving trajectory of the main object are still applicable. When inferences can be drawn or how many details of inferences will determine the field of application. For example, to facilitate fast search and retrieval, a search engine calculates "actions π" and sorts them according to their order. To allow searches in different applications and interoperability, if the moving track description symbol is accompanied by some The start position and size descriptors are standardized for each main object / area / sub-area, and it will be sufficient. These descriptors provide a small but powerful search / filter based on the action. These descriptors also work smoothly To supplement the descriptive symbols of other spaces and the knowledge of the influence from these descriptive symbols to improve the accuracy of extraction. The structural nature can lead to a one-level description method. It should be noted that although the current object segmentation has used affine

O: \ 62 \ 62535.ptd Page 23 4 60813 V. Description of the invention (20) model to describe, but the current new structure can also be easily extended to other mobile models. For example, the current segmentation of object movement can be modified by using the "transformation movement" model, "isotropic movement" model (certain proportions, X, y + transformation rotation), and "perspective movement" model. In a specific embodiment, the object track segmenter 170 uses a method of splitting and merging a quadratic equation (relevant time) movement model of a region's geometric meaning, and the geometric meaning of the region can be summarized as follows Random moving orbit model: Suppose that one area of frame-by-frame movement information of an area develops an expression to predict the position of all points on the object at the interval time, and at each time instant predicted based on the comparison with the original position, the area At each point, an error metric is selected that deviates from the sum of the measurement distance. Then if the error metric exceeds a certain experimental threshold T 1 and the instant at which the offset is the smallest, this interval can be separated. If a common prediction representation on both intervals results An error less than T 1 or another experimental criticality, the merge procedure will merge two contiguous intervals. An example of an affine movement model is shown below: (a) Obtain affine movement parameters between adjacent frames in order to describe the position of each point on a region at each time instant. (b) Determined by, for example, a quadratic equation that requires at least two data sets in a suitable order. Obtain affine movement parameters between the frames of a sub-sampling group in the selected interval. (C) Decompose the affine movement parameters into its components, namely scale, rotation, clipping, and transformation. Assume that each component has a different temporal The model is determined by his essence. For example, transformations can be modeled by using a quadratic equation of time. Proportions can be modeled linearly with time. Rotation can be modeled by using

O: \ 62 \ 62535.ptd Page 24 < 80813 V. Description of the invention (21) A fixed angular velocity is assumed and modeled. (d) Obtain the coefficients of the selected prediction model with the affine movement required by the sub-fetch. (e) Apply the prediction model on each frame and calculate the sum of the deviations on all points of each frame. Sum the frame errors and compare them to a normal threshold T1. If the error is> T1, divide the interval into 2 at the position of the maximum frame error. (f) Repeat steps (b)-(e) Repeat 5 for each interval until the error in all intervals is lower than TΓ. (g) Now if the common prediction model of two contiguous intervals causes a normal error below the critical T 2, the intervals can now be merged. (h) Repeat (g) for all recursive intervals before adding normalization errors to any of the further T 2 merged intervals described above. -Data Structure For the purpose of this disclosure, it should be understood that an image sequence can be divided into 11 image captures ". Each image capture is represented by a linked list (ie, a new data structure) using effective image capture description methods at different intervals in the image capture. The image capture data structure (DS) has sub-description methods or description symbols according to color, shape, organization, and movement. A mobile data with the same mobile description symbol is disclosed below. The data structure has a hierarchical structure described by global_motion, object_motion, and region_niotion. A linked list can be used to point to the next interval. Moving track description method DS description method

O: \ 62 \ 62535.ptd Page 25 46〇8i3

V. Description of the invention (22) A link list data structure can be assumed (for example, the main object description method in an image capture can be accessed by starting from the initial indicator and accessing the next indicator.) . The description in the class is shown in a zigzag pattern. ° (start time, interval time) is always used to indicate time support. ° Time can be media time (time reference in this article) or actual time. shot_DS {-start_time, time—interval (Time and duration in media / actual time)

Number of key objects (First level key objects in a shot)

Global—motion—DS (Background key object)

Pointer to Key-object JDS array (Pointer to first entry in a linked list of key object DSs)

Key_object_interaction_times DS (DS containing the instants within the shot when key objects interacted)-} GIobal_motion_DS {start_time, time_interval

Pointer to first link of Global_motion_subinterval a trajectory JDS}

Sub_interval_global_motion_trajectory_DS {start ”ime, time” nterval

Model_class ”d for motion_trajectory—descriptor (Trajectory description may use a predefined set of models (e.g., quadratic in position of object centroid)-the model—clas-id is the index of the chosen model in the set))

Global motion_trajectory_descriptor '(descriptor parameters for global motion trajectory based on the model_class_id)

Pointer to next_interval_

Sub—interval_globaljmotion_trajectory—DS

Key_obj ect—interaction one times one DS {

Pointer to first like of key_object—interaction time descriptor

O: \ 62 \ 62535.ptd Page 26 460813 V. Description of the invention (23)

Key „obj ect_DS {

Key_object_ID (Label or index of key object) start_time, time_interval (start time and time interval in terms

of frames, i.e., at what frame is object detected and for how long) Key_object a first__frame „geometry_DS

(Used to define the initial coarse shape) Key_object_motion_trajectory_DS

Number of sub_regions (Number of sub-objects within the object)

Pointer to sub_region „DS array

Key_object_spatial_DS (Spatial description of key objects (such as color, shape, texture, etc.))}

Key „object_motion_trajectory_DS {

Pointer to first link of key_object_sub_interval_motion_trajectory_DS (Linked_list of key_object_sub_interval_motion_trajectory_DS)}

Key_object_sub_interval_ motion_trajectory-DS {start_tim6, timejnterval Model_class-id for motion_trajectory-descriptor Key-object motion_trajectory_descriptor Pointer to sub-region-DS array Pointer to next_interval key-object-sub-interval-motion-trajectory —DS}

Sub_Region_DS {

Sub_Region_ID — start_tixne, time_interval

SuB_region_motion_trajectory_DS Sub_region_spatial JDS}

Sub „region_motion_trajectory_DS {

Pointer to first link of sub „region_sub_interval_motion_trajectory_DS (Linked_list of sub_region_sub_ interval_motion_trajectory_DS) ΙΗΠ page 27 460813 V. Description of the invention (24)

Sub_region_sub_interval_ motion_trajectory_DS {start_time, time "nterval

Model_class_id for motion_trajectory_descriptor Sub_region motion_trajectory—descriptor Pointer to next_interval

sub_re ^ ion_sub_interval._m〇tion_trajectory_DS Insert ifi · 1 '------ .........., ....... .... .... _____ „________... .............................................. ---------- motion_trajectory „descriptor {float ax, ay? vx, vy, px, py; (for quadratic model))

Model_class_ID descriptor for motion trajectory {

Model class ID of the motion trajectory model used

Key_object_interaction_time_descriptor {time reference (Time with respect to start of shot) number of key_objects (involved in the interaction) key_object_id of the objects interacting at that time (indices of key objects) The moving orbit description symbol of the quadratic equation model of the time geometric mean position is provided as a description only. In view of this moving segment and orbit segment, a plurality of new queries can be implemented as follows: Only possible queries based on movement a) Universal motion: i) An image capture in which the camera is tilting, tilting, rolling, zooming , Or tracking (background moving). ii) The range of general movement (indicating the range of action in image shooting) b) Area object / area movement: i) The number of obvious objects in image shooting

O: \ 62 \ 62535.ptd Page 28 4 6 0 81 3 V. Description of the invention (25) ii) Range of movement (fast / slow) iii) Objects tracked by the camera iv) Sudden discontinuous positions v) Vigorous movements vi) Recognition of traversing sequence and similar tracking in time vii) Comparison of internal and external contact movements Possible human inquiries based on conditions on other descriptors: walking, running, jumping, bending, sitting Standing, standing, falling, spinning, dancing, stance, shaking hands, throwing, catching, raising, hitting, skates, kneeling, pedaling, punching, cutting, fenced, nodding, holding, writing, type Vehicle movements: fly, move, stop, sail, cruise, turn, dive, spin-other living objects. Action: animal (bird, etc.) running, leap, swimming, diving. Any action: leaves, river objects. Talking: in action / A similar metric-like general solution to human actions on non-action objects to find each query above is to create a feasible model for each action in the moving model parameter space by using sequence data. Energy density function (p d f). Then, a specific set of sampled moving model parameters can be divided into one of these actions by using the maximum possible (or recorded possible) test of the metric to test the M-a y hypothesis. The model parameters must be normally suitable for the establishment of p d f and subsequent classification, so that the size and position of the objects are independent of the frame rate of the analysis.

O: \ 62 \ 62535.ptd Page 29 4 60 81 3 V. Description of the Invention (26) Although the object movement segmentation and object movement instruction segmentation are used to improve the order and retrieval of the image sequence, but It should be understood that the present invention can be used to improve or provide other image processing functions. For example, the present invention can be used in a coding function, for example, the main object identified can be considered as a particularly important area, or general motion parameters can be retrieved. In addition, although the invention is described above as an object, it should be understood that an object is a specific area that is broadly defined as a variable size depending on the application. Similarly, although the present invention is described above as a block such as a giant block, it should be understood that a block is a block that is broadly defined to determine a change in size for a particular application. Although various specific embodiments described in connection with the present invention have been shown and described in detail herein, those skilled in the art can still combine these descriptions to form other different specific embodiments. One

O: \ 62 \ 62535.ptd Page 30

Claims (1)

/ 、, Patent application scope h Methods for performing a method with plural figure pivot knife slaves, the method includes the following steps: image sequence of object movement a) decide in the image sequence of ~ ~ field; at least in the picture frame; A connection area: and b) applying block-based movements to generate optical flow movement parameters c) using the generated optical flow movements 2. The number of methods such as the scope of the patent application to identify a main object. The at least three areas that are more than the connection area ', wherein the step b) can provide a vector' to generate the block-based movement of the optical flow movement parameter 3. As the method step of the scope of patent application: 'It further includes the following step d) to decide on any of the items found in step--a single primary object. 4 Identifies whether the main object is 4. If the method of the scope of the patent application is the first, *, step: 'It further includes the main object identified by one of the following steps; and-the current vector of the frame is tracked in one Regional fish and apricot ^ = Γ are not the main objects, so that in the current frame, °, the main objects of the rigid frame are related to each other. 5. The method according to item 丨 of the scope of patent application, which further includes a series of steps: d) indexing the image sequence according to any optical flow movement parameter identifying the main object from step c). r 46081 3 VI. Scope of Patent Application Step 6: If the method of the scope of patent application is applied, it further includes the following steps d) separating the identified main object into plural sub-objects; and e) providing from these sub-objects Up to ^ = = = Π vector of each of the sub-objects to generate such sub-objects: a new set of optical flow movement parameters for its sub-objects. , And track: two: with: frame-moving segmentation method of image segment retreat segment method The method includes the following steps: Frame 2: Use optical flow objects to move segments' to obtain the image sequence-frame-frame Movement information of at least one main object; and Figure t) (i) Use optical flow object track segmentation to obtain track information of the at least one main object between-& of the image sequence. And rail: a device (15) for performing moving segmentation and segmentation of an image sequence with a plurality of frames. The device includes: a knife-sequential mobile segmenter (16) to obtain the The moving information of at least one main object of a frame of the image along the frame; and an optical flow object track segmenter (17) to obtain the orbit information of the at least one main object of the image along the frame of the Q frame. Refers to a person—a computer-readable medium (130) with a plurality of stored instructions. The plural includes when executed by a processor, causing the processor to perform the following steps: a) Decide at least in one of the frames of the image sequence. A connected area b) applying block-based motion vectors to generate optical flow movement parameters 4 80813_ VI. Patent application scope; and C) using the generated optical flow movement parameters to identify a major object A computer-readable medium (130) having a plurality of instructions stored therein. When executed by a processor, the processor causes the processor to perform the following steps: a) Applying an optical stream object to move segments to obtain a picture of the image sequence Movement information of at least one main object of the frame; and b) applying optical flow object track segmentation to obtain track information of the at least one main object in a frame interval of the image sequence. Page 33