JP4998787B2 - Object region extraction processing program, object region extraction device, and object region extraction method - Google Patents

Description

  The present invention relates to a technical field such as a system and method for tracking an object region in a moving image.

  Conventionally, a technique for extracting an object region in a moving image has been actively studied as an elemental technique for processing and editing a moving image. In such a technique, for example, a target object region is extracted from the first frame image (or several key frame images) in a moving image, and the object region in another frame image is tracked based on the target object region. An object region is extracted from each frame image. As a technique for extracting an object region in a frame image (still image), for example, techniques disclosed in Non-Patent Documents 1 to 3 are known.

By the way, in the research related to the tracking of the object region so far, as disclosed in Non-Patent Documents 4 and 5 and Patent Document 1, for example, the purpose is to improve the accuracy of automatic tracking. For example, with the technique disclosed in Patent Document 1, it is possible to track an object region with fewer failures by characterizing a tracking target (object region) that is not affected by a non-tracking target in a moving image. The effort to reset the tracking frame can be reduced.
Y. Li, J. Sun, CK Tang and HY Shum, “Lazy Snapping,” ACM Transactions on Graphics (TOG), Vol.23, Issue.3, pp.303-308, Aug.2004. YY Boykov and MP Jolly, “Interactive graph cuts for optimal boundary & region segmentation of objects in ND images,” Proc. ICCV'01, Vol.1, pp.105-112, Jul.2001. C. Rother, V. Kolmogorov and A. Blake, “GrabCut: interactive foreground extraction using iterated graph cuts,” ACM Transactions on Graphics (TOG), Vol.23, Issue.3, pp.309-314, Aug.2004. Ryuzo Okada, Yoshiaki Shirai, Jun Miura, Yoshinori Kuno, “Animal tracking based on optical flow and distance information,” IEICE theory (D-II), Vol.J80-D-II, No.6, pp.1530- 1538, Jun. 1997. MJ Black and AD Jepson, “EigenTracking: Robust Matching and Tracking of Articulated Objects Using a View-Based Representation,” International Jounal of Computer Vision, Vol.26, No.1, pp.63-84, Jan.1998. JP 2003-6654 A

  However, it is difficult to eliminate tracking errors (failures) through all frame images. Therefore, it is necessary to manually perform an object extraction operation (correction operation) again on the frame image in which the tracking error has occurred. In order to reduce the burden of manual work on the assumption that tracking errors will occur at least, the approach of improving tracking accuracy and reducing the number of tracking errors and the burden of correction work itself will be reduced from the interface side. Two approaches are conceivable. Conventionally, there has been no known technique that makes it possible to reduce the manual work load from both approaches.

  The present invention has been made in view of the above points, and one example of the problem is an object region extraction processing program, an object region extraction device, and an object region extraction that can reduce the burden of manual work. It aims to provide a method.

In order to solve the above-mentioned problem, the object region extraction processing program according to claim 1, wherein the display control means for causing the computer to display at least one frame image among a plurality of frame images constituting the moving image, Accepting from a user designation of line segments that serve as clues for extracting an object area included in the displayed frame image and are separately drawn on the object area and an area other than the object area means, based on the line segment to be the designated cue, the object area extracting means for extracting the object region from the frame image, and, in other frame images other than frame image segment serving as the cue is specified , to function as a track means Ri or hand to track the movement of a line clues said specified, the object area extracting means, is , Based on the line segment to be the tracked clues, and extracts an object area from the frame image.

According to the present invention, instead of tracking an object region in a frame image other than a frame image in which a line segment serving as a clue is designated , a line segment serving as a clue is tracked, and the clue tracked in each frame image is tracked. Since the object area is extracted based on the line segment , the object area can be extracted with higher accuracy and the object area can be extracted even if the object area extraction fails in a certain frame image. This failure has an effect that the object image extraction in the next frame image is not affected. Furthermore, as compared with the case where the extracted object region itself is tracked, it is only necessary to perform a correction work to add or delete a line segment that becomes a clue in the failed frame image, thereby reducing the work load on the user (manpower). Can do.

In addition, since the line segment as a clue is tracked, tracking can be performed with a smaller amount of data when tracking the object region, so that the tracking process can be speeded up.

According to a second aspect of the present invention, in the object region extraction processing program according to the first aspect, the object region extraction means moves the line segment serving as the designated clue in the plurality of other frame images. It is characterized by continuous tracking.

According to the present invention, an object region can be extracted from each frame image based on a line segment that is a clue that is continuously tracked.

According to a third aspect of the present invention, in the object region extraction processing program according to the first or second aspect, the computer causes the computer to detect a frame image in which the extraction of the object region based on the tracked line segment has failed. It further functions as a failure detection means for detecting.

According to the present invention, it is possible to increase the efficiency and speed of the correction work for adding and deleting line segments that are clues in a frame image in which extraction of the object region has failed.

  According to a fourth aspect of the present invention, in the object region extraction processing program according to the third aspect, the computer is further caused to function as information presenting means for presenting information indicating the detected frame image. .

According to the present invention, it is possible to increase the efficiency and speed of the correction work for adding and deleting line segments that are clues in a frame image in which extraction of the object region has failed.

According to a fifth aspect of the present invention, in the object region extraction processing program according to the third aspect, the display control means displays the detected frame image, and the reception means displays the displayed frame image. On the other hand, re-designation of the line segment serving as the clue is received from the user, and the object region extracting means extracts the object region from the frame image in which the line segment serving as the clue is re-designated.

According to the present invention, re-designation is realized simply by performing a correction operation for adding or deleting a line segment as a clue in a frame image in which extraction of the object region has failed, and re-extraction of the object region is performed, so that there are few corrections. The object region extraction failure can be corrected by the amount, and the work burden on the user can be reduced.

  According to a sixth aspect of the present invention, in the object region extraction processing program according to any one of the third to fifth aspects, the failure detection means includes an object region extracted from a reference frame image and the frame. A frame image in which extraction of the object region has failed is detected by comparing object regions extracted in other frame images other than the image and calculating a degree of coincidence thereof.

  According to the present invention, it is possible to easily detect a frame image in which object region extraction has failed.

According to a seventh aspect of the present invention, in the object region extraction processing program according to the first aspect, the object region extraction unit is configured to use a plurality of frame images other than the frame image in which the line segment serving as the clue is specified. The movement of the line segment serving as the designated clue is continuously tracked, and the object area extracting means extracts the object area from each frame image based on the tracked line segment serving as the clue. An object region extracted in a frame image in which a line segment serving as a clue is designated, and an object region extracted in each of a plurality of other frame images other than the frame image, Are further functioned as coincidence degree calculating means for calculating the respective coincidence degrees, and the display control means is configured such that the calculated coincidence degree is a predetermined threshold value. And wherein the displaying the frame image is lower.

  According to this invention, the user can grasp the frame image to be corrected at a glance, and the correction work by the user can be made efficient and quick.

  According to an eighth aspect of the present invention, in the object region extraction processing program according to the seventh aspect, the display control means determines that the degree of coincidence of the frame image in which the calculated degree of coincidence is equal to or less than the predetermined threshold value. A display object that can be visually grasped by the user is displayed in correspondence with the frame image.

  According to this invention, the user can grasp the frame image to be corrected at a glance, and the correction work by the user can be made efficient and quick.

  According to a ninth aspect of the present invention, in the object region extraction processing program according to the eighth aspect, the display control means includes the frame image in which the calculated degree of coincidence is equal to or less than a failure detection threshold smaller than the predetermined threshold. And the display object corresponding to the frame image in which the calculated degree of coincidence is larger than the failure detection threshold value are displayed in different display modes.

  According to the present invention, the frame image in which the extraction of the object region has failed and the frame image that is not so (for example, the user can grasp at a glance the frame image close to this, and the correction work by the user can be made efficient and quick. .

According to a tenth aspect of the present invention, in the object region extraction processing program according to the first aspect, the object region extracting means is configured to use a plurality of frame images other than the frame image in which the line segment serving as the clue is specified. The movement of the line segment serving as the designated clue is continuously tracked, and the object area extracting means extracts the object area from each frame image based on the tracked line segment serving as the clue. An object region extracted in a frame image in which a line segment serving as a clue is designated, and an object region extracted in each of a plurality of other frame images other than the frame image, Are further functioned as a degree of coincidence calculating means for calculating the degree of coincidence of the two, and the display control means has the smallest degree of coincidence calculated. And wherein the displaying a predetermined number of the frame image selected from the side.

  According to this invention, the user can grasp the frame image to be corrected at a glance, and the correction work by the user can be made efficient and quick.

  According to an eleventh aspect of the present invention, in the object region extraction processing program according to the tenth aspect, the display control means uses the predetermined number of display objects whose calculated degree of coincidence can be visually grasped by a user. The frame images are displayed in correspondence with each of the frame images.

  According to this invention, the user can grasp the frame image to be corrected at a glance, and the correction work by the user can be made efficient and quick.

  According to a twelfth aspect of the present invention, in the object region extraction processing program according to the eleventh aspect, the display control unit is configured to display the display object corresponding to the frame image in which the calculated degree of coincidence is a failure detection threshold value or less. And a display object corresponding to the frame image in which the calculated degree of coincidence is larger than the failure detection threshold value are displayed in different display modes.

  According to the present invention, the user can grasp at a glance the frame image in which the extraction of the object region has failed and the frame image that is not (for example, close to this), and the correction work by the user can be made efficient and quick. it can.

According to a thirteenth aspect of the invention, in the object region extraction processing program according to the first aspect of the invention, the object region extraction means is provided in a plurality of frame images other than the frame image in which the line segment serving as the clue is specified. The movement of the line segment serving as the designated clue is continuously tracked, and the object area extracting means extracts the object area from each frame image based on the tracked line segment serving as the clue. An object region extracted in a frame image in which a line segment serving as a clue is designated, and an object region extracted in each of a plurality of other frame images other than the frame image, Are further functioned as a degree of coincidence calculating means for calculating the degree of coincidence of the frames, and the display control means Together to display an image, the calculated degree of matching visually prehensible displayed object by a user, and wherein the displaying the in correspondence with each frame image.

  According to this invention, the user can grasp the frame image to be corrected at a glance, and the correction work by the user can be made efficient and quick.

  According to a fourteenth aspect of the present invention, in the object region extraction processing program according to the thirteenth aspect, the display control means is configured to display the display object corresponding to the frame image in which the calculated degree of coincidence is equal to or less than a failure detection threshold. And a display object corresponding to the frame image in which the calculated degree of coincidence is larger than the failure detection threshold value are displayed in different display modes.

  According to the present invention, the user can grasp at a glance the frame image in which the extraction of the object region has failed and the frame image that is not (for example, close to this), and the correction work by the user can be made efficient and quick. it can.

  According to a fifteenth aspect of the present invention, in the object region extraction processing program according to any one of the sixth to ninth aspects, the comparison includes a color histogram of the object region, an area of the object region, and the object region. At least one of the following shape, the texture of the object region, and the center of gravity of the object region is used.

  According to the present invention, it is possible to easily detect a frame image in which object region extraction has failed.

The invention according to claim 16 is the object region extraction processing program according to any one of claims 1 to 15, wherein the computer is extracted based on the tracked line segment. On the other hand, it is further characterized in that it further functions as a metadata providing means for assigning metadata including information on the object represented in the object area.

  According to the present invention, metadata can be directly attached to an object region included in a moving image.

  According to a seventeenth aspect of the present invention, in the object region extraction processing program according to the sixteenth aspect, the metadata attaching unit assigns the metadata to each of the plurality of extracted object regions. And

  According to the present invention, even when there are a plurality of object areas in a moving image, metadata can be directly assigned to each object area.

  According to an eighteenth aspect of the present invention, in the object region extraction processing program according to the sixteenth or seventeenth aspect, the metadata includes position information indicating a position of the extracted object region on the frame image. It is characterized by that.

  According to the present invention, the position of the object region to which metadata is added can be easily specified later.

  According to a nineteenth aspect of the present invention, in the object region extraction processing program according to the eighteenth aspect, the position information is coordinates of at least one point on a line segment drawn on or in the vicinity of the extracted object region. It is characterized by that.

  According to the present invention, the amount of data can be reduced compared to the case where the coordinates of the outer frame of the object region (or the coordinates of the entire object region) are used as position information.

The invention according to claim 20 is a display control means for displaying at least one frame image among a plurality of frame images constituting a moving image, and a clue for extracting an object region included in the displayed frame image. Based on the line segment that serves as the designated clue, and a receiving unit that accepts designation of the line segment that is drawn separately on the object region and the region other than the object region , hand to track the object area extracting means for extracting the object region from the frame image, in other frame images other than frame image line is designated to be the clue, the movement of the segment to be the designated cue comprising a painter Ri tracking means, wherein the object area extracting means further on the basis of the line segment to be the tracked cues, the object from the frame image And extracting the range.

The invention according to claim 21 is an object region extraction method performed by a computer, the step of displaying at least one frame image among a plurality of frame images constituting a moving image, and the displayed frame image A step of receiving from a user a designation of a line segment that serves as a clue to extract an included object area and is separately drawn on the object area and an area other than the object area; and In the object region extraction step of extracting an object region from the frame image based on the line segment serving as a clue, and in the frame image other than the frame image in which the line segment serving as the clue is specified, the specified clue and the higher engineering move you track line segments, comprises, in the object area extraction step, further line serving as the tracked clues Based on, and extracting the object region from the frame image.

According to the present invention, clues segment clues instead of following the object region in another frame images other than the specified frame images, which tracks the segment as a cue, have been tracked in each frame image Since the object area is extracted based on the line segment , the object area can be extracted with higher accuracy and the object area can be extracted even if the object area extraction fails in a certain frame image. This failure has an effect that the object image extraction in the next frame image is not affected. Furthermore, as compared with the case where the extracted object region itself is tracked, it is only necessary to perform a correction work to add or delete a line segment that becomes a clue in the failed frame image, thereby reducing the work load on the user (manpower). Can do. In addition, since the line segment as a clue is tracked, tracking can be performed with a smaller amount of data when tracking the object region, so that the tracking process can be speeded up.

Hereinafter, the best embodiment of the present invention will be described in detail with reference to the drawings. The embodiment described below is a clue region (hereinafter referred to as “clue”) that serves as a clue for extracting an object region in a moving image, and is an object region based on the clue specified by the user. This is an embodiment when the present invention is applied to an image editing apparatus that performs extraction, tracking of an object region by tracking the clue, and adding metadata to the extracted object region.
[I. Configuration and function of image editing apparatus]
First, the configuration and function of the image editing apparatus S according to the present embodiment will be described with reference to FIG.

  FIG. 1 is a diagram illustrating a schematic configuration example of an image editing apparatus S according to the present embodiment.

  As shown in FIG. 1, the image editing apparatus S includes an operation unit 1, a display unit 2, a communication unit 3, a drive unit 4, a storage unit 5, an input / output interface unit 6, a system control unit 7, and the like. The system control unit 7 and the input / output interface unit 6 are connected via a system bus 8. A personal computer can be applied as the image editing apparatus S.

  The operation unit 1 includes, for example, a keyboard and a mouse, receives an operation instruction from the user, and outputs the instruction content to the system control unit 7 as an instruction signal.

  The display unit 2 includes, for example, a CRT (Cathode Ray Tube) display, a liquid crystal display, and the like, and displays information such as characters and images.

  The communication unit 3 is connected to a network NW such as the Internet or an intranet (configured by a communication line and a relay device such as a router) and controls the communication state with the information providing server, for example.

  For example, the drive unit 4 reads data from a disk DK (recording medium) such as a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disc), and the like, while reading data from the disk DK (recording medium). Record.

  The input / output interface unit 6 performs interface processing between the operation unit 1 to the storage unit 5 and the system control unit 7.

  The storage unit 5 includes, for example, a hard disk drive and stores an operating system (O / S), various programs, data, and the like. Here, the programs stored in the storage unit 5 include a moving image editing application program (having the object region extraction processing program of the present invention) and the like. The storage unit 5 stores moving image files in a format such as MPEG (Moving Picture Expert Group).

  The moving image editing application program is provided by being recorded on a disk DK such as a CD-ROM or provided by downloading from an information providing server connected to the network NW, and is used after being installed.

  The system control unit 7 includes a CPU (Central Processing Unit) 7a, a ROM (Read Only Memory) 7b, a RAM (Random Access Memory) 7c used as a main memory and an image memory, and the like. Then, the system control unit 7 executes the moving image editing application program to display the display control unit, the receiving unit, the object region extracting unit, the clue region tracking unit, the coincidence degree calculating unit, the failure detecting unit, the information in the present invention. The image editing apparatus S functions as a presentation unit, a metadata providing unit, and the like. The image editing apparatus S includes (1) an object region search support function, (2) an object region extraction function, (3) an object region tracking function by cue tracking, (4) It has an object area extraction failure detection function and (5) a metadata addition support function.

  Each of these functions will be specifically described below with reference to FIGS.

  2 to 9 are diagrams showing examples of display screens displayed on the display in the display unit 2 in the execution process of the moving image editing application program.

  (1) The object region search support function divides a moving image into scene units composed of a plurality of frame images, displays a list of the first frame images of each scene of the moving image, and searches for object region display sections by the user It is a function to support.

  In the object area search support function, a moving image file is read from the storage unit 5 by the system control unit 7 and developed on the image memory, and then a scene division process and a scene list display process are sequentially executed. The display example in FIG. 2A shows an example in which the first frame image of each scene divided by the scene division process is displayed as a list (frame image surrounded by a broken line 51 in the figure) by the scene list display process. Yes. In the display state of FIG. 2A, the user selects one of the head frame images (for example, the head frame image F4 of FIG. 2A) (for example, clicks with the mouse), that is, the head frame image is selected. When a scene to be included is selected, a plurality of frame images included in the selected scene are displayed as a list (frame images surrounded by a broken line 52 in the figure), as shown in FIG. At this time, for example, the number of frames from the frame image F4 to the frame image F5 varies depending on the number of frames of the moving image to be read. Therefore, all the frames from the frame image F4 to the frame image F5 are shown in the broken line 52 in the figure. In some cases, some of the frame images are displayed every predetermined number of frames instead of the images.

  In FIG. 2B, the frame image surrounded by the broken line 52 is a frame image sandwiched between the top frame image F4 and the top frame image F5. The enlarged frame image (reference numeral 50) at the top of the display screen is a frame image selected by the user from the frame image surrounded by the broken line 51 and the frame image surrounded by the broken line 52. In addition, when the user designates the new creation button 53a and the edit button 53b, the screen is changed to the display screen shown in FIG.

  (2) The object region extraction function is a function for extracting an object region from a frame image selected from the list display based on a clue designated by the user on the frame image selected by the user.

  In the object region extraction function, the user can start by specifying a range to be tracked. The display screen shown in FIG. 3A serves as an object area display section selection interface used for the user to specify a section of a frame image in which an object to be extracted is displayed. By designating the designation button 54a and the end frame image designation button 54b (for example, clicking with the mouse), it is possible to select the start and end frame images of the object area display section (that is, the tracking section). That is, it is possible to specify the range to be tracked. At this time, in order to find the start and end frame images, the user may use the frame search slide bar 55a for efficiently searching for an arbitrary frame image in the scene or the frame forward / back frame button 55b. it can. Thus, after the range to be tracked is specified, when the user designates the decision button 54c (for example, clicks with the mouse), an area designation button 56 is displayed as shown in FIG.

  When the user designates the area designation button 56 (for example, clicks with the mouse), as shown in FIG. 4A, an area extraction screen 61 on which the start frame image 61a selected by the user is displayed. Is displayed. The area extraction screen 61 serves as a clue input interface. When a clue is designated (input) by the user on the start frame image 61a on the region extraction screen 61, the system control unit 7 that has accepted the designation executes object region extraction processing, and uses the designated clue. The object region is automatically extracted based on (for example, using position information). The user only needs to specify at least two cues, one on the desired object area (broken line 61b in FIG. 4B), and the other on the background area (an area other than the object area, FIG. It only needs to be on the alternate long and short dash line 61c). In the display example of FIG. 4B, a straight line segment is used as a clue, but a curved line, a circular line, a rectangular line segment, or a point may be used. Good. In addition, the line segments that specify the object region and the background region can be specified in any number and in any order.

  In order to specify a clue, for example, the user operates the mouse to move the pointer to a desired position on the frame image 61a in the area extraction screen 61, and drags while holding down the left button of the mouse. The clue 61b for designating the background area may be pulled by dragging the clue 61b for designating the area while holding down the right button of the mouse. In addition, for example, when the user draws an incorrect line segment, regardless of the line segment that specifies the object area or the line segment that specifies the background area, the user can hold down the middle button of the mouse and point the line segment to be deleted with the pointer. The line segment can be deleted by tracing. Instead of specifying the object region and the background region with the left and right buttons of the mouse, the user can select the object (for example, “foreground” in FIG. 4A) and the background (for example, FIG. It is also possible to specify by dragging while holding down any mouse button after first specifying “background” in A) and deleting (for example, “delete” in FIG. 4A).

  Further, in the display example of FIG. 4B, the region surrounded by the solid line 61d is the extracted object region. The extraction of the object area is performed when one clue 61b specifying the object area and one clue 61c specifying the background area are drawn, and the extraction result is displayed as indicated by a solid line 61d. Thereafter, each time the user designates a clue for designating the object region or a clue for designating the background region, the object region is re-extracted and the extraction result is displayed. In this way, after the user confirms the final extraction result and designates the completion button 61e (for example, clicks with the mouse), the extraction result is reflected on the display screen shown in FIG.

  By the way, the object area extraction process using the clue is a technique for extracting the object area from the clue drawn on the object area and the clue drawn on the background area, for example, a known technique such as Lazy Snapping logic. Can be used. Lazy Snapping consists of two steps: separation of the object region from the background and correction of the boundary line after separation in units of pixels. In the present invention, when Lazy Snapping is used, Only the separation step from the background is employed. The feature of Lazy Snapping is that object region extraction is realized by simply drawing several lines in the region, and every time a line is added, the object region is re-extracted sequentially. The user can add and delete lines while looking at the extraction result, and interactive object region extraction is possible.

In addition, object region extraction methods based on graph cuts such as Lazy Snapping are characterized by replacing the image with a graph G = (V, E) and reducing the object region extraction problem to a labeling problem for each vertex. is there. Here, V is all vertex sets, and E is all edge sets. When an image is replaced with a graph, each pixel is made to correspond to a vertex, and the adjacency relationship in four directions or eight directions around each pixel is made to correspond to an edge. The labeling problem is a problem of assigning a label x _i ∈ {object region (= 1), background (= 0)} to each vertex i∈V of the graph G. The optimal solution X = {x _i } for this problem minimizes the energy function C (X) defined by

Here, C ₁ (x _i ) represents the likelihood that the vertex i is x _i , and C ₂ (x _i , x _j ) represents the cost when the vertices i and j are x _i and x _j , respectively. A graph cut algorithm is used as an energy minimization method. Refer to Non-Patent Document 1 for the definition of the energy function used in Lazy Snapping.

  The principle of object region extraction by such energy minimization is used not only by Lazy Snapping but also by the methods of Non-Patent Document 2 and Non-Patent Document 3. As long as the object region is extracted by energy calculation based on the clue specified by the user, the object region extraction method applicable in the present invention is not limited to Lazy Snapping, and other methods can be applied. Also good.

  (3) The object area tracking function based on the cue tracking is a function for tracking the movement of the designated cue in other frame images other than the frame image for which the cue is designated by the user (that is, the designated cue is other This is a function for tracking to which position in the frame image of the movement.

  In the object area tracking function based on the cue tracking, the cue tracking process is executed by the system control unit 7 to track the movement position of the cue, and the object area extraction process is performed based on the cue in each frame image. Extraction of object areas in all frame images within the area display section is automated. For example, when the user operates the mouse to specify the tracking direction (playback direction or reverse (rewind) direction) on the display screen shown in FIG. For example, when the start frame image in the object area display section is over a predetermined (for example, one) number of frame images (ie, the movement of the clue is over) in a plurality of consecutive frame images. Tracking may be included between tracked frame images (which may include frame images that do not track the movement of cues))) tracking is initiated, and object regions are extracted from each frame image based on the cues tracked in each frame image. The tracking of the clue can also be started from an intermediate frame image positioned between the start frame image and the end frame image in the object region display section.

  In this way, by tracking the movement of the cue and extracting the object region from each frame image based on the tracked cue, if tracking fails, that is, the object region extraction fails in a certain frame image. However, since the object region extraction failure does not affect the object image extraction in the next frame image, the clue is added or deleted in the failed frame image as compared with the case where the extracted object region itself is tracked. The correction work may be performed, and the correction work can be performed quickly and efficiently.

  By the way, for the clue tracking process, a known technique such as an optical flow logic representing the velocity field of each point on the screen can be used. Refer to Non-Patent Documents 4 and 5 for details of the optical flow calculation method.

  (4) The object region extraction failure detection function detects a frame image in which tracking has failed (tracking error) (that is, extraction of an object region based on the tracked cue region has failed), and indicates the detected frame image, etc. This is a function for presenting information to the user (for example, as shown in FIG. 6 or voice output).

  In the object region extraction failure detection function, failure detection processing is executed by the system control unit 7, and the object region extracted in the reference frame image (for example, the start frame image in which the clue is specified) is extracted from the frame image. Object regions extracted by comparing (extracting based on tracked cues) in consecutive frame images (that is, other frame images other than the frame images) and calculating the degree of coincidence thereof A frame image in which the extraction of the image has failed is detected.

  That is, the degree of coincidence is calculated for each frame image excluding the reference frame image, and the calculated degree of coincidence is, for example, equal to or less than a failure detection threshold preset by the user. A frame image is detected as a failed frame image in which extraction of the object region has failed. The comparison uses, for example, at least one of a color histogram of the object region, an area of the object region, a shape of the object region, a texture of the object region, and a barycentric position of the object region. May be applied.

  Further, in the object region extraction failure detection function, a tracking result display process is executed by the system control unit 7, and a predetermined number selected from the one with the smallest (low) calculated degree of coincidence (for example, the degree of coincidence is low). 10 frame images are displayed in order. For example, if the number of failed frame images is less than a predetermined number, a frame image that is not a failed frame image is also displayed (of course, only the failed frame image may be displayed). In addition, the object region extraction failure detection function displays the predetermined number of frame images, and displays a bar graph (an example of a display object) with which the degree of coincidence can be visually grasped by the user for each of the predetermined number of frame images. It is made to display correspondingly.

  The display example of FIG. 6 shows an example in which bar graphs (displayed in the graph display area 59) corresponding to 10 frame images selected from the one with the lowest degree of coincidence are displayed corresponding to the frame images. Yes. The association between the bar graph and the frame image is made by a straight line 60 that connects the frame image and the bar graph, so that the user can grasp at a glance which position of the frame image is highly consistent. . Furthermore, in the display example of FIG. 6, the bar graph 59a corresponding to the frame image whose matching degree is equal to or lower than the failure detection threshold and the bar graph 59b corresponding to the frame image whose matching degree is larger than the failure detection threshold are different from each other. For example, colors and patterns are different).

  For example, the bar graph 59a is displayed so as to have a conspicuous color (for example, red), and the bar graph 59b has a conspicuous color (for example, green). Thereby, the user can grasp the failed frame image at a glance.

  In the above display example, a bar graph 59a (for example, red) corresponding to a frame image whose matching degree is equal to or less than the failure detection threshold, a bar graph 59b (for example, green) corresponding to a frame image having a matching degree greater than the failure detection threshold, Are configured to be displayed in different display modes. However, the frame images having a matching degree larger than the failure detection threshold and the matching degree being close to the failure frame image (for example, the matching degree is “failure detection threshold + α The bar graph corresponding to “the frame image below” may also be displayed in the same display mode (for example, red) as the bar graph corresponding to the frame image whose coincidence is equal to or lower than the failure detection threshold. Accordingly, the user can grasp the failed frame image at a glance and can grasp a frame image close to the failed frame image at a glance.

  The frame image displayed together with the bar graph indicating the degree of coincidence can be selected by the user, and the selected frame image is enlarged and displayed on the upper part of the display screen (reference numeral 50). In such a display state, for example, when the user selects a failure frame image and designates the region designation button 56, the region in which the failure frame image 71a selected by the user is displayed as shown in FIG. An extraction screen 71 is displayed. In the display example of FIG. 7A, since the extracted object region 71b is not the region intended by the user, when the user re-specifies the clue on the failed frame image 71a (addition or deletion of the clue), The object region extraction process is executed by the system control unit 7 that has received the designation, the object region is extracted based on the redesignated clue, and is displayed as an extraction result as indicated by a solid line 71c in FIG. 7B. become. At this time, the clue added to the frame image can be reflected in a frame image continuous thereto by re-tracking from there.

  The frame image displayed by the system control unit 7 in the tracking result display process (for example, displayed in the graph display area 59 shown in FIG. 6) is equal to or less than a predetermined threshold value in which the calculated matching degree is larger than the failure detection threshold value. It is good also as a frame image. In this case, the number of frame images to be displayed is within the maximum number of displayable frame images. On the other hand, if there is no frame image that is equal to or less than the predetermined threshold value, there is no frame image to be displayed. Also in this case, a bar graph corresponding to a frame image that is equal to or less than the predetermined threshold is displayed in association with the frame image. Therefore, it is not necessary to display a bar graph (for example, green color) corresponding to a frame image whose matching degree is larger than a predetermined threshold value (matching considerably).

  Further, regardless of the threshold value, all frame images within the maximum number of displayable frame images and corresponding bar graphs (graphs proportional to the degree of coincidence) may be displayed.

  Then, after the user confirms the final extraction result and designates the completion button 71d, the display state shown in FIG. 6 is returned while the extraction result is reflected. Note that extraction of the object region by such re-designation can be performed for each frame image regardless of whether or not it is a failed frame image.

  Thereafter, when the user operates the mouse to specify the tracking direction (playback direction or reverse (rewind) direction) on the display screen shown in FIG. Continuous re-tracking of the movement of the cues in a plurality of consecutive frame images from the frame image from which the object region has been extracted by the re-designation is started, and each frame image based on the cues re-tracked in each frame image The object region is extracted from

  In this way, by displaying and correcting a frame image with a low (low) degree of coincidence including a failed frame image that has failed to be tracked, the object region can be extracted more accurately and the correction load on the user can be reduced. Can be reduced.

  (5) The metadata addition support function is a function for supporting the input of metadata including information on the object represented in the object area to be added to the extracted object area.

  In the metadata addition support function, for example, when the user designates the text editing button 60a on the display screen shown in FIG. 6, a text information input screen 81 is displayed as shown in FIG. The text information input screen 81 serves as a metadata input interface. In such a text information input screen 81, the user tags, for example, text data (for example, “jacket”) for identifying an object region as tag name data, as shown in FIG. Arbitrary text data (for example, “brand name”, “price”, and “URL of shopping site” where a jacket can be purchased) is input as a tag content data in the name input field 81a and the tag content input field 81b. And the completion button 81c is specified, the system control unit 7 executes the metadata adding process, and the input tag name data and tag content data are added to the object region as metadata. The For example, the assignment is performed on object area position data that is position information indicating the position of the object area on the frame image (for example, the number of the frame image and the coordinates of the object area on the frame image are included). This is done by associating the metadata.

  Here, in order to express the object area position data with a smaller amount of data, the coordinates of the outer frame of the object area (for example, the solid line 71c shown in FIG. 7B) are used instead of the coordinates of the extracted object, for example. It is desirable to be defined by the coordinates and radius of the center of the circle surrounding the region and the circumscribed circle. For example, when the user designates the result display button 60b on the display screen shown in FIG. 6, a circle 50a surrounding the object area is displayed in the frame image 50 as shown in FIG. The coordinates and radius of the center of the circle 50a are the object area position data. As a result, the data amount can be reduced as compared with the case where the coordinates of the outer frame of the object region (or the coordinates of the entire object region) are used as the object region position data. In addition to the circle and circumscribed circle surrounding the extracted object area, for example, on the line segment (straight line, curve, rectangle, etc.) drawn on or near the extracted object area as the representation of the object area position data The coordinates of at least one point (or all points) of can be expressed with a smaller amount of data.

  Further, the display example of FIG. 9B is an example that is displayed after the addition of metadata, but the added metadata is displayed on the metadata display unit 50b. When the save button 60c shown in FIG. 9B is designated by the user, the metadata is stored in, for example, an XML file together with the object area position data, and the XML file is saved in a directory where the moving image file is stored. Will be.

Thus, the stored XML file is associated with the moving image file and registered in, for example, a Web server constituting a predetermined site on the Internet. Thus, the Web server provides the moving image data and metadata to the client in response to a request from the client (Web browser), so that the client displays the moving image, and the object region in the moving image is displayed by the user. For example, when specified with the mouse, metadata corresponding to the object area is displayed.
[II. Operation of image editing apparatus S]
Next, the operation of the image editing apparatus S according to the present embodiment will be described with reference to FIGS.

  FIG. 10 is a flowchart showing processing in the system control unit 7 of the image editing apparatus S according to the present embodiment, and FIG. 11 is a flowchart showing details of the metadata providing processing shown in FIG. Note that the processing illustrated in FIGS. 10 and 11 is an example in which the flow of the operation of the image editing apparatus S is simplified, and is not limited to the processing.

  The processing shown in FIG. 10 is started when a desired moving image file is designated by the user after the moving image editing application program is started.

  In FIG. 10, first, the system control unit 7 reads a moving image file designated by the user from the storage unit 5 (step S1) and develops it on the image memory.

  Next, the system control unit 7 performs scene division processing and divides the read moving image for each scene (step S2).

  Next, the system control unit 7 performs a scene list display process to display a list of the first frame images of each divided scene as shown in FIG. 2A, for example (step S3).

  Next, the system control unit 7 performs a metadata input acceptance preparation process (step S4). In this metadata input acceptance preparation process, for example, a process corresponding to the selection of various buttons displayed on the display screen shown in FIG.

  Next, the system control unit 7 designates the start frame image and end frame image of the object area display section (for example, the start frame image designation button 54a displayed on the display screen shown in FIG. 3A is designated, and further ends. Is received (by designating the frame image designation button 54b) (step S5).

  Next, the system control unit 7 designates a clue for extracting the object region on the designated start frame image (for example, a line on the frame image displayed on the region extraction screen 61 shown in FIG. 4B). (By subtracting the minutes) is accepted (step S6).

  Next, the system control unit 7 performs an object region extraction process, and extracts an object region as described above based on the designated clue (step S7).

  Next, the system control unit 7 displays an object region extraction result (for example, a solid line 61d displayed on the region extraction screen 61 shown in FIG. 4B) (step S8).

  Next, the system control unit 7 determines whether or not there has been a tracking start instruction from the user (step S9). If there is no tracking start instruction (step S9: NO), the system control unit 7 returns to step S6 and receives the tracking start instruction. (For example, the completion button 61e displayed on the area extraction screen 61 shown in FIG. 4B, for example, is designated, and then the tracking start button 57 displayed on the display screen shown in FIG. 5 is designated. (Depending on the situation) (step S9: YES), the process proceeds to step S10.

  In step S10, the system control unit 7 specifies one frame image to be tracked by the clue. For example, in the first processing of step S10, the next frame image of the frame image from which the object region has been extracted by the processing of steps S6 to S8 is specified.

  Next, the system control unit 7 performs a clue tracking process to track the movement of the clue specified by the start frame image in the specified frame image (step S11).

  Next, the system control unit 7 performs an object region extraction process, and extracts an object region as described above based on the tracked clue (step S12).

  Next, the system control unit 7 determines whether or not the frame image from which the object region has been extracted in step S12 is an end frame image (step S13). If the frame image is not the end frame image (step S13: NO) Returning to step S10, the next frame image is identified and the same processing as described above is performed. If it is not the end frame image (step S13: YES), the process proceeds to step S14. In this way, the start frame image is specified in order toward the end frame image, and the object region is extracted in each frame image.

  In step S14, the system control unit 7 performs failure detection processing, and as described above, the object region extracted in the reference frame image, the object region extracted in each frame image in which the clue is tracked, Are compared with each other to calculate the degree of coincidence, and a frame image whose calculated degree of coincidence is equal to or less than the failure detection threshold is detected as a failure frame image.

  Next, the system control unit 7 performs a tracking result display process to display a list of a predetermined number (for example, 10) of frame images selected from the one with the smallest calculated degree of coincidence, for example, as shown in FIG. At the same time, the object region extraction result (outer frame of the object region) is displayed on each frame image, and a bar graph indicating the matching degree of each frame image is displayed corresponding to each frame image (step S15).

  Next, the system control unit 7 determines whether or not there has been a clue correction (re-designation) instruction from the user (step S16), and if there is a correction instruction (the user selects a frame image to be corrected and designates a region). (By designating the button 56) (step S16: YES), the process proceeds to step S6 and the same processing as described above is performed. In this case, in steps S10 to S13, the clue is re-tracked from the next frame image to the end frame image of the frame image in which the clue is corrected and the object region is extracted. The user can also newly designate an end frame image (for example, for narrowing the tracking section) different from the end frame image designated in step S5.

  On the other hand, when there is no correction instruction (step S16: NO), the system control unit 7 determines whether or not there is an instruction to give metadata (step S17), and when there is an instruction to give metadata. For example (for example, when the text editing button 60a displayed on the display screen shown in FIG. 6 is designated) (step S17: YES), the process proceeds to step S18. Step S17: NO), it returns to step S16.

  In step S18, the system control unit 7 performs a metadata providing process. In this metadata providing process, the system control unit 7 displays a metadata input interface (for example, the text information input screen 81 shown in FIG. 8A) as shown in FIG. 11 (step S181), and the tag The name data and the tag content data are input as metadata (step S182), the metadata is temporarily stored in the RAM 7c (step S183) and displayed on the screen (step S183), and the process returns to the process of FIG. The metadata providing process can be executed at an arbitrary timing after the metadata input reception preparation process.

  In step S19 shown in FIG. 10, when the save button 60c shown in FIG. 9B is designated by the user, the system control unit 7 executes a metadata writing process, and the temporarily stored metadata. And the object region position data described above are associated with each other, and an XML file including these data is stored in a directory having a moving image file (may be stored in an arbitrary directory designated by the user).

  The metadata and the object area position data are not limited to being stored in the XML file format, but may be stored in other file formats. Further, the metadata writing process can be executed at an arbitrary timing after the metadata input reception preparation process.

Next, in step S20, the system control unit 7 determines whether or not there is an instruction to extract another object region in the moving image from the user, and if there is an instruction to extract (for example, on the display screen shown in FIG. 6). (By specifying the new creation button 53a displayed on the display screen shown in FIG. 2B displayed after the save button 60c displayed in FIG. 2 is specified) (step S20: YES), step The process proceeds to S4 and the same process as described above is performed. When there is no extraction instruction (step S20: NO), the process ends.
[III. Evaluation experiment example]
Next, as an object region tracking performance evaluation, an evaluation experiment example in which the above-described method according to the present invention is compared with the conventional method will be described with reference to FIG. Note that the conventional method uses a basic method of tracking all pixels in the object region by optical flow (that is, tracking the object region itself). In this performance evaluation, the accuracy (in other words, the same object region in the method according to the present invention and the conventional method) tracked in each frame image in the moving image (in the present invention, tracking the object region by cue tracking). , Accuracy) is evaluated by precision and recall. Incidentally, precision S _P and recall S _R is a following equation example definition.

Here, N (R) is the number of pixels in the region R, R _h is the true object region, R _c is the extracted object region, and R _h ∧R _c is the common region of R _h and R _c . The precision S _P represents the ratio of the true object area in the tracked object area, and the recall S _R represents the ratio of the object area that can be tracked in the true object area. A true object region is an object region that has been manually extracted from each frame image in advance.

  FIG. 12A is a diagram showing a certain frame image in a moving image, FIG. 12B is a graph showing an evaluation experiment result of the method according to the present invention, and FIG. 12C is a conventional method. It is a graph showing the evaluation experiment result of. In the frame image of FIG. 12A, an object region (a hat in this example) represented by a bold line 101 is a tracking (extraction) target. In the graphs shown in FIGS. 12B and 12C, the horizontal axis is the number of frames (the number of frame images, 50 in this example), and the vertical axis is precision (solid line in the figure). Represents recall (dotted line in the figure).

  In the conventional method, as shown in FIG. 12C, it has been confirmed that precision and recall gradually decrease as the number of frames increases. In the conventional method, recall is gradually and greatly reduced because when tracking all the pixels in the object region, the number of pixels indicating the object region is decreased by tracking different pixels to the same location. It is thought that the cause is that it decreases.

  On the other hand, in the method according to the present invention, as shown in FIG. 12B, it is not seen that the precision gradually decreases, but only in the 47th frame image, the precision greatly decreases (tracking fails). It was confirmed that The precision does not decrease gradually because the tracking of the clue and then extracting the object area based on it is performed in two steps, so the tracking error of the clue does not directly affect the precision. This is considered as a cause. Also, the reason why the precision is greatly reduced in the 47th frame image is considered to be that the cue specifying the object region is near the boundary and the edge strength of the boundary is small. Although it is the same clue, it is considered that the reason for being divided into success and failure is that there are some differences in lightness and saturation between the frame images, and the edge strength changes accordingly.

  From the above evaluation experiment, it can be said that the method according to the present invention enables tracking with higher accuracy than the method of tracking the object region itself according to the conventional method because the tracking error does not directly affect the error of object region extraction. In addition, when tracking failed, it was confirmed that there was a sharp decrease in precision compared to the conventional method, but when tracking failed, an extreme decrease in precision occurred rapidly in the area of the object region. It shows that there was a change. Therefore, as described above, by measuring the area change of the object region (calculating the degree of coincidence), the computer (system control unit 7) can automatically detect the frame image that has failed to be tracked and should be corrected for the user. It is also possible to easily present a frame image, and a system with less burden of correction work can be realized.

  Further, in the method according to the present invention, it is considered that the reason why the tracking error occurs locally is due to a small amount of information of, for example, a line as a clue given in the first frame image. Therefore, as a result of the verification, after adding several lines in the first frame image and tracking again, the extraction result of the object area does not change in the first frame image, but when adding lines, the precision and recall It was confirmed that the tracking results were improved in accuracy without any local degradation. The ability to improve tracking results by adding a few lines allows automatic correction of tracking results by a computer. If it is possible to automatically detect a frame image that has failed to be tracked as described above, it can be determined by a computer whether or not the frame image has been corrected. For example, a line within a distance set in advance from a line designated by the user. Automatic correction can be performed by automatically adding and the like.

  Further, as another example of the automatic correction described above, it may be configured to correct and change an erroneous clue.

  FIG. 13 is a diagram illustrating an example in which an erroneous clue is automatically corrected. For example, when it is detected that the extraction of the object region in the n + 1 frame image has failed based on the clue in the n + 1 frame image, which is the result of tracking the clue in the n frame image, the clue automatic correction is executed. Specific points and regions are discriminated based on curvature, coordinate distribution, etc. in the clues in the n + 1 frame image, and singular points or singular regions are deleted, added, and corrected so as to fall within a predetermined threshold value. For example, as shown in FIG. 13 (right side), when the clue is a polyline composed of a plurality of straight lines, the system control unit 7 determines a singular point based on the curvature and the like, deletes the point, and deletes the point. Connect the connected points with a straight line, and fix the point by placing it in the middle.

  After such automatic correction, as shown in the figure, the cue tracking process may be continued in the frame image after the n + 1 frame image.

Note that the above experimental results are merely examples, and the results may vary depending on the size of the object region, the edge strength of the boundary between the object region and the background region, etc. This shows an example in which the difference between the results of the method and the conventional method appears more prominently.
[IV. effect]
As described above, according to the above embodiment, the object area is not tracked in the frame image other than the frame image in which the clue is specified, but the clue is tracked and the clue tracked in each frame image is used. Since the object area is extracted based on the above-mentioned evaluation experiment, the tracking error does not directly affect the object area extraction error and is higher than the conventional method of tracking the object area itself. It is possible to extract an object region with high accuracy, and on the assumption that tracking failure will occur, even if object region extraction fails (tracking fails) in a certain frame image, the failure will be detected in the next frame image. This has the effect of not affecting the object image extraction. Furthermore, compared with the case where the extracted object region itself is tracked, it is only necessary to perform a correction work for adding or deleting a clue in the failed frame image, and the work burden on the user (manual operation) can be reduced (tracking). This makes it easier for the user to make corrections.

  In addition, since the clue is tracked, tracking can be performed with a smaller amount of data when tracking the object region, so that the tracking process can be speeded up.

  Also, when tracking fails, the accuracy can be drastically reduced compared to the conventional method.For example, if the degree of coincidence is calculated by measuring the area of the object region, the failure frame image can be easily detected. It is also possible to easily present a frame image to be corrected to the user, making it possible to improve and speed up the correction work for adding and deleting clues by the user and reducing the burden of the correction work. It can be further reduced.

  In addition, it is possible to re-specify the object area by re-extracting the object area simply by adding or deleting clues in the frame image that has failed to be tracked, so that the object area extraction failure can be corrected with a small amount of correction. The work burden on the user can be further reduced.

  Further, as shown in FIG. 6, a bar graph indicating the degree of coincidence of each frame image is associated with each frame image together with the frame image having a small degree of coincidence including the failure frame image whose coincidence degree is equal to or less than the failure detection threshold. Since the image is attached and displayed, the user can grasp at a glance at which position the matching degree of the frame image is low, and the correction work by the user can be made efficient and quick.

  Further, the bar graph corresponding to the failure frame image whose coincidence degree is equal to or lower than the failure detection threshold and the bar graph corresponding to the frame image having a coincidence degree greater than the failure detection threshold are different from each other in display mode (for example, different in color, pattern, etc.) The user can grasp the failed frame image at a glance.

  In addition, metadata is not assigned to the entire scene in the moving image, but metadata can be directly attached to the object area included in the scene, even when there are multiple object areas in the scene. The metadata corresponding to each of the plurality of object regions can be directly given.

  Furthermore, the object area position data added to the metadata is not defined by the coordinates of the outer frame of the object area but, for example, by the coordinates and radius of the center of the circle or circumscribed circle surrounding the extracted object area. If configured, the amount of data stored can be reduced.

It is a figure which shows the example of a schematic structure of the image editing apparatus S which concerns on this embodiment. It is a figure which shows the example of a display screen displayed on the display in the display part in the execution process of a moving image editing application program. It is a figure which shows the example of a display screen displayed on the display in the display part in the execution process of a moving image editing application program. It is a figure which shows the example of a display screen displayed on the display in the display part in the execution process of a moving image editing application program. It is a figure which shows the example of a display screen displayed on the display in the display part in the execution process of a moving image editing application program. It is a figure which shows the example of a display screen displayed on the display in the display part in the execution process of a moving image editing application program. It is a figure which shows the example of a display screen displayed on the display in the display part in the execution process of a moving image editing application program. It is a figure which shows the example of a display screen displayed on the display in the display part in the execution process of a moving image editing application program. It is a figure which shows the example of a display screen displayed on the display in the display part in the execution process of a moving image editing application program. It is a flowchart which shows the process in the system control part 7 of the image editing apparatus S which concerns on this embodiment. It is a flowchart which shows the detail of the metadata provision process shown in FIG. (A) is a figure which shows a certain frame image in a moving image, (B) is a graph showing the evaluation experiment result of the method by this invention, (C) represents the evaluation experiment result of the conventional method. It is a graph. It is a figure which shows the example by which the clue which is incorrect is corrected automatically.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Operation part 2 Display part 3 Communication part 4 Drive part 5 Memory | storage part 6 Input / output interface part 7 System control part 7a CPU
7b ROM
7c RAM
8 System bus S Image editing device

Claims (21)

Computer
Display control means for displaying at least one of the plurality of frame images constituting the moving image;
A line segment serving as a clue to extract an object area included in the displayed frame image, which is separately drawn on the object area and an area other than the object area, is received from a user. Reception means,
An object region extracting means for extracting an object region from the frame image based on the line segment serving as the designated clue; and
In another frame image other than the frame image segment serving as the cue is specified, to function as a track means Ri hand or to track the movement of the segment to be the designated cue,
The object region extraction processing program further extracts an object region from the frame image based on the tracked line segment as a clue. In the object region extraction processing program according to claim 1,
The object region extraction processing program that continuously tracks the movement of the designated line segment in the plurality of other frame images. In the object region extraction processing program according to claim 1 or 2,
An object region extraction processing program that causes the computer to further function as failure detection means for detecting a frame image in which extraction of the object region based on the tracked line segment has failed. In the object region extraction processing program according to claim 3,
An object area extraction processing program causing the computer to further function as information presenting means for presenting information indicating the detected frame image. In the object region extraction processing program according to claim 3,
The display control means displays the detected frame image,
The accepting unit accepts re-designation of a line segment as a clue to the displayed frame image from a user,
The object region extraction processing program, wherein the object region extraction unit extracts an object region from a frame image in which a line segment as a clue is redesignated. In the object region extraction processing program according to any one of claims 3 to 5,
The failure detection means compares the object region extracted in the reference frame image with the object region extracted in another frame image other than the frame image, and calculates the degree of coincidence thereof. An object area extraction processing program for detecting a frame image in which extraction of an object area has failed. In the object region extraction processing program according to claim 1,
The object area extracting means, in other plurality of frame images other than the frame image segment serving as the cue is specified, continuously tracks the movement of the segment to be the designated cue,
The object area extracting means extracts an object area from each frame image based on the tracked line segments ,
The computer compares the object region extracted in the frame image in which the line segment serving as the clue is designated with the object region extracted in each of a plurality of frame images other than the frame image. Further function as a degree of coincidence calculation means for calculating the degree of coincidence of each,
The display control means displays the frame image in which the calculated degree of coincidence is equal to or less than a predetermined threshold value. In the object region extraction processing program according to claim 7,
The display control means displays a display object in which the degree of coincidence of the frame image in which the calculated degree of coincidence is equal to or less than the predetermined threshold can be visually grasped by a user in correspondence with the frame image. An object region extraction processing program characterized by In the object region extraction processing program according to claim 8,
The display control means includes the display object corresponding to the frame image in which the calculated coincidence is less than or equal to a failure detection threshold smaller than the predetermined threshold, and the calculated coincidence is greater than the failure detection threshold. An object region extraction processing program that displays a display object corresponding to a frame image in different display modes. In the object region extraction processing program according to claim 1,
The object area extracting means, in other plurality of frame images other than the frame image segment serving as the cue is specified, continuously tracks the movement of the segment to be the designated cue,
The object area extracting means extracts an object area from each frame image based on the tracked line segments ,
The computer compares the object region extracted in the frame image in which the line segment serving as the clue is designated with the object region extracted in each of a plurality of frame images other than the frame image. Further function as a degree of coincidence calculation means for calculating the degree of coincidence of each,
The display control means displays a predetermined number of the frame images selected from the one with the smallest calculated degree of coincidence. In the object region extraction processing program according to claim 10,
The display control means displays a display object whose calculated degree of coincidence can be visually grasped by a user in correspondence with each of the predetermined number of the frame images. . In the object region extraction processing program according to claim 11,
The display control means includes the display object corresponding to the frame image in which the calculated matching degree is equal to or less than a failure detection threshold, and a display corresponding to the frame image in which the calculated matching degree is greater than the failure detection threshold. An object region extraction processing program that displays objects in different display modes. In the object region extraction processing program according to claim 1,
The object area extracting means, in other plurality of frame images other than the frame image segment serving as the cue is specified, continuously tracks the movement of the segment to be the designated cue,
The object area extracting means extracts an object area from each frame image based on the tracked line segments ,
The computer compares the object region extracted in the frame image in which the line segment serving as the clue is designated with the object region extracted in each of a plurality of frame images other than the frame image. Further function as a degree of coincidence calculation means for calculating the degree of coincidence of each,
The display control means displays each frame image for which the degree of coincidence is calculated, and displays a display object in which the calculated degree of coincidence can be visually grasped in correspondence with the frame image. An object region extraction processing program characterized by that. In the object region extraction processing program according to claim 13,
The display control means includes the display object corresponding to the frame image in which the calculated matching degree is equal to or less than a failure detection threshold, and a display corresponding to the frame image in which the calculated matching degree is greater than the failure detection threshold. An object region extraction processing program that displays objects in different display modes. In the object region extraction processing program according to any one of claims 6 to 14,
The comparison uses at least one of a color histogram of the object region, an area of the object region, a shape of the object region, a texture of the object region, and a barycentric position of the object region. Object area extraction processing program. In the object region extraction processing program according to any one of claims 1 to 15,
The computer is further caused to function as a metadata adding unit that adds metadata including information on an object represented in the object area to the object area extracted based on the tracked line segment . An object region extraction processing program characterized by that. In the object region extraction processing program according to claim 16,
The object region extraction processing program, wherein the metadata attaching unit assigns the metadata to each of the plurality of extracted object regions. In the object region extraction processing program according to claim 16 or 17,
An object area extraction processing program characterized in that the metadata includes position information indicating a position of the extracted object area on the frame image. The object region extraction processing program according to claim 18,
The position information is a coordinate of at least one point on a line segment drawn on or near the extracted object area. Display control means for displaying at least one frame image among a plurality of frame images constituting the moving image;
A line segment serving as a clue to extract an object area included in the displayed frame image, which is separately drawn on the object area and an area other than the object area, is received from a user. Receiving means;
An object region extracting means for extracting an object region from the frame image based on the line segment serving as the designated clue;
In another frame image other than the frame image segment serving as the cue is specified, and the tracking means Ri hand or to track the movement of the segment to be the designated cue,
With
The object area extracting device further extracts an object area from the frame image based on the tracked line segment as a clue. An object region extraction method performed by a computer,
Displaying at least one frame image of a plurality of frame images constituting the moving image;
A line segment serving as a clue to extract an object area included in the displayed frame image, which is separately drawn on the object area and an area other than the object area, is received from a user. Process,
An object region extracting step of extracting an object region from the frame image based on the line segment serving as the designated clue;
In another frame image other than the frame image segment serving as the cue is specified, and as engineering you track the movement of the segment to be the designated cue,
Including
In the object area extraction step, an object area is further extracted from the frame image based on the tracked line segment .