JP2001092849A - Video retrieval method, computer-readable recording medium having program for causing computer to execute the method recorded thereon, video retrieval processor, method for imparting video index, and computer-readable recording medium having program for causing computer to execute the method recorded thereon, method for creating introductory document of image contents and a computer readable medium having program for causing computer to execute the method recorded thereon - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make an inquiry about the meaning of the contents of a video by using a highly abstract term or concept and to perform retrieval at a high speed. SOLUTION: In a video retrieval method, a structural unit which is suitable as a unit to be retrieved at the time of retrieving a video is set as a retrieval granularity with respect to a term which can be expressed by the combination of a plurality of events, and the state transition pattern corresponding to the term is defined as the input column of a plurality of event indexes which are continuously generated in a retrieval granularity on the basis of generation pattern of the event expressing the term. When a desired video scene is retrieved from the video, the term expressing the desired video scene is inputted and whether the state transition pattern corresponding to the inputted term coincides with the generation pattern of an event index in the structural unit is discriminated at every structural unit which is coincident with the retrieval granularity by using the retrieval granularity corresponding to the inputted word as a retrieval object unit. The coincident structural unit is outputted as a retrieved result.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a video search method and a video search processing apparatus for performing a video search process using terms having a high degree of abstraction on a video structured using a structure index and an event index. A video index assigning method for performing a video index assigning process using a term having a high degree of abstraction, a video content description generating method for generating a description to explain the video content of a video scene, and a method for causing a computer to execute the method The present invention relates to a computer-readable recording medium recording the program.

[0002]

2. Description of the Related Art In recent years, with the progress of computer hardware technology and information processing technology, and the spread of the Internet and digital satellite broadcasting, it has become possible to use various images on a daily basis. For this reason, the value, diversity, and recreational value of the information possessed by the video become even more important, and the video can be continuously displayed, such as in the case of television broadcasting and video playback that have been used in the past. In addition to playback and viewing, search and view the desired video scene from the structured video with an index added, collect information, create a digest version, etc.
Various usages have been proposed.

In order to efficiently search for video, a video, which is a set of video scenes that are temporally continuous, is usually divided into smaller units (sections) and used. Also, at this time, the video scene is not physically divided such as divided into predetermined unit times or divided according to the amount of the video scene, but is logically defined as a set of video scenes satisfying a predetermined condition. It is common to divide it into two. After performing this logical division, by adding an index to the logical division, the divided video scene can be handled as a reusable semantic group.

As a logical division method, for example,
As a method of visually dividing a video, dividing the video manually (manually), and assigning an index to the video, or a method of real-time authoring (real-time content description), the index at the starting point of the logical division is indexed. (Structure index), and a method of determining the start of the next logical division as a division section is considered.

On the other hand, in addition to the structure index representing the section to be divided, an event index representing an event that has occurred on the video has been considered as a video index. This is because the logical structure index is a logical section of the video (Duration: duration).
Is basically an index having no section. However, this event index can also set only fragmentary information with the current technology.

Conventionally, extraction of a logical section by a search using the event index is a video search process. In other words, a desired video portion can be searched by using a query that matches a term (fragmentary information) set as the event index.

[0007]

However, according to the above-mentioned prior art, a video can be searched for an inquiry that matches a term set as a structure index or an event index. There was a problem that it was not possible to search for the meaning of.

[0008] In other words, according to the above-described conventional technique, only a fragmentary index can be assigned to a video. Therefore, a search is performed in units of a logical structure based on the fragmentary index, and the video is searched. Although it is possible to extract a part, it is not always possible to search for a video part that the user really wants.

[0009] In order to clarify the explanation, the "meaning of the contents of the video" is defined as follows in this specification. The meaning of the content of the video is information generated for a specific section when a person views the video and understands the content of the video abstractly and conceptually. That is, it is information having a meaning that can be discovered only by using human subjectivity. In addition, the meaning of the content of the video means that the event index is a fragmentary representation of the event that occurred on the video, while the state of the change of the event that occurred on the video and the result that occurred due to the change of the event Based on the state of the event, the content of the video in a certain section is comprehensively judged by a person and given meaning. Therefore, the meaning of the content of the video is meaningful only after the content of the video in a certain section becomes clear.

[0010] Also, when searching for a video, a section expected as an answer to an inquiry from a user does not always correspond to one structural unit, and may be a part of a certain structural unit. May be, or a combination of multiple structural units, but based on the fragmentary index assigned to the logical structural unit,
Since one structural unit is selected as an answer (search result) to the contents of the inquiry, it has not always been possible to search for a video portion that satisfies the user.

[0011] In addition, since the video scene is interpreted in various ways, if the person who created the logical structure (that is, the person who defined it) is different from the person making the inquiry, the section set as a meaningful group may be different. They did not always match.

Further, searching for the meaning of the contents of a video also means "searching for a term having a high degree of abstraction". However, in the above-described conventional technique, only the index information that is fragmentarily added is used. The only way to do this is to extract logical structural units. Except for specific applications, in general, the meaning of video content is efficiently analyzed using high-level search conditions. Could not be processed.

Further, in the above-mentioned conventional technique, when an attempt is made to generate a character string describing the content of a video using an index, it is only possible to arrange in a fragmentary manner an index in parallel. There was a problem that it was not possible to generate sentences that were easy for people to understand. That is, it was not possible to convert a fragmentary index into a general meaningful character string and generate a sentence that is easy for the user to understand.

At the time of real-time authoring, the attribute (or event index) for a certain event
May be determined by successive events. For example, in a baseball video or the like, it is not possible to determine whether a player has hit two or three at the time of hitting. After that, it is not possible to make meaning (attachment of an attribute or an event index) only after watching the result of how many bases the batter has made.

As a technique for solving this problem, as a method for describing the contents in real time, a related study by Akasako, Iijima, Kakutani and Tanaka (“Real-time contents description method of video data and its implementation”, DEWS '99 Proceedings, 3A-2, 1
March 4-6, 999). According to this report, incomplete index information sequences that come in real time, such as narration of a radio, are interpreted and replaced with higher-level concepts. That is, the superordinate concept is represented in advance by a state transition diagram, and replaced with the superordinate concept in real time, thereby correcting the redundancy and incompleteness of the original input index sequence.

However, in the method proposed in the above related research by Akasako et al., As described in real-time content description, it is assumed that a state transition diagram is used to interpret a time-series index. And For this reason, if this method is applied to the search as it is, the index is interpreted in a time series, so that there is a problem that the processing takes time. Also, no solution to the problem of how to realize a search with a high degree of abstraction at high speed has not been proposed.

Further, there is a problem that the index which is always included in the time series must be interpreted by comparing it with all the state transition diagrams, which is not always efficient.

The present invention has been made in view of the above, and an object of the present invention is to make it possible to make an inquiry about the meaning of the contents of a video using terms or concepts with a high degree of abstraction and to perform a high-speed search. And

Further, the present invention has been made in view of the above, and interprets the meaning of the contents of a video, converts the meaning into a character string having a general meaning, and provides a description of the video contents which is easy for a user to understand. The purpose is to be able to generate character strings.

Further, the present invention has been made in view of the above, and it is an object of the present invention to efficiently and quickly assign an index using terms having a high degree of abstraction.

[0021]

In order to achieve the above object, a video search method according to claim 1 comprises at least a structure index for dividing a video into a semantic group and an event index in the video. An event index for specifying content and location is given, and a video scene in a section divided by the structure index is used as a video structure unit,
In a video search method for searching for a desired video scene from the video using the structure index and the event index for a video structured using a plurality of hierarchical structure units, a plurality of events are set in advance. For a term that can be expressed by a combination of, a suitable structural unit is set as a search granularity as a search target unit when searching the video, and based on an occurrence pattern of an event expressing the term, the search granularity is determined. A state transition pattern corresponding to the term is defined as an input sequence of a plurality of event indexes that occur consecutively in the video, and a desired video scene is expressed when a desired video scene is searched from the video. Enter a term and use the search granularity corresponding to the entered term as the search target unit. Occurrence pattern of an event index in the state transition patterns of structural units corresponding it is determined whether matching, and outputs the matching structural unit as a search result.

According to a second aspect of the present invention, there is provided the video search method according to the first aspect, further comprising, for each state transition pattern corresponding to the term, an event index existing in each state transition pattern. At least one of the event indexes is designated as a key index for starting a search, and when searching for a desired video scene from the video, a structure unit that matches a search granularity and the key index After searching for a structural unit with an event index that matches
For the corresponding structural unit, it is determined whether or not the state transition pattern matches the occurrence pattern of the event index in the structural unit.

According to a third aspect of the present invention, in the video search method according to the first or second aspect, a desired video scene is extracted from the video based on the structural unit output as the search result. In this case, the video scene specified in the structural unit is output.

According to a fourth aspect of the present invention, in the video search method according to the first or second aspect, a desired video scene is extracted from the video based on the structural unit output as the search result. At this time, an arbitrary upper or lower structural unit on the structured video can be designated.

According to a fifth aspect of the present invention, in the video search method according to the second aspect, a desired video scene is extracted from the video based on the structural unit output as the search result. The video scene is extracted by designating an offset for video clipping before and after the location where the key index is added.

A video search method according to a sixth aspect of the present invention is the video search method according to any one of the second to fifth aspects, wherein a state transition pattern corresponding to the input term and an event index in a structural unit. When it is determined whether or not the occurrence patterns match, an index representing the term is defined as an abstract index, newly added to the matching structural unit, and reused as a key index.

A video search method according to a seventh aspect of the present invention is the video search method according to any one of the first to sixth aspects, wherein each of the terms that can be expressed in advance by a combination of a plurality of events includes An abstract index is set as an index representing a term, and when defining the state transition pattern, a state corresponding to the term is used as an input sequence composed of an event index and an abstract index by using the abstract index in addition to the event index. It defines a transition pattern.

[0028] In a video search method according to claim 8, in the video search method according to any one of claims 2 to 7, a plurality of pieces of attribute information are added to the structure index and the event index. In the state transition pattern corresponding to the term, character string definition information for generating a description sentence related to the term using each attribute information of the structure index and the event index is added. A description sentence related to the term is generated by referring to the attribute information in the structural unit output as the search result based on the character string definition information of the transition pattern.

In the video search method according to the ninth aspect, at least a structure index for dividing the video into a semantic group and an event index for specifying the content and location of an event occurring in the video are provided. The video scene of the section divided by the structure index is used as the structure unit of the video, and the video structured using the plurality of hierarchical structure units is targeted. In a video search method for searching for a desired video scene from a video, as a process before performing a video search, a term that can be expressed in advance by a combination of a plurality of events is appropriately set as a search target unit when searching for the video. Is set as a search granularity, and the search granularity is set based on the occurrence pattern of an event expressing the term. A state transition pattern corresponding to the term is defined as an input sequence of a plurality of event indexes that occur consecutively in the form. Further, for each state transition pattern corresponding to the term, a state transition pattern exists in each state transition pattern. A state transition table generating step of specifying at least one event index among the event indexes as a key index for starting a search, and generating a state transition table including the term, search granularity, state transition pattern, and key index. A process of inputting a term representing a desired video scene as a process of performing a video search; and searching for a search granularity corresponding to the term input in the term input process with reference to the state transition table. Using the key index corresponding to the term as the target unit and A search step of searching for a structural unit having an event index that matches the index; and an event included in a state transition pattern corresponding to the term in the structural unit searched in the search step with reference to the state transition table. A first determination step of determining whether or not all the indexes are present; and a structural unit determined to be all present in the first determination step,
A second determining step of determining whether or not the state transition pattern corresponding to the input term matches the occurrence pattern of the event index in the structural unit; and determining whether the structural unit has been determined to match in the second determining step. A search result output step of cutting out a video scene from the video based on the search result and outputting it as a search result.

A computer-readable recording medium according to a tenth aspect records a program for causing a computer to execute the video search method according to any one of the first to ninth aspects.

In the video search processing device according to the present invention, at least a structure index for dividing the video into a semantic group and an event index for specifying the content and location of an event occurring in the video are added. The video scene of the section divided by the structure index is used as the structure unit of the video, and the video is structured using the plurality of hierarchical structure units, and the video is structured using the structure index and the event index. In a video search processing device for searching for a desired video scene from the inside, a video input means for inputting the structured video to be searched, a term for specifying a desired video scene to be searched, and a video A search granularity that specifies a structural unit as a search target unit when searching for, and the term is continuous in the search granularity A state transition pattern defined as an input sequence of a plurality of event indexes generated by the operation and a key index designating at least one event index among the event indexes existing in the state transition pattern are stored as a state transition table. Storage means, operation input means for inputting or specifying an inquiry term for searching for the desired video scene,
When a query term is input or specified via the operation input means, refer to the state transition table of the storage means, and use a key index corresponding to the query term, from the video input by the video input means. A search unit that matches the search granularity, and searches for a structural unit having an event index that matches the key index, and inputs the structural unit searched by the search unit,
Determining means for determining whether a state transition pattern corresponding to the query term matches an occurrence pattern of an event index in a structural unit; and And a search result output unit that cuts out a video scene from and outputs it as a search result.

According to a twelfth aspect of the present invention, in the video search processing device according to the eleventh aspect, the determining means includes:
First determining means for determining whether or not all the event indexes included in the state transition pattern corresponding to the query term exist; and for the structural unit determined to be all present by the first determining means. Determining whether the state transition pattern corresponding to the query term matches the occurrence pattern of the event index in the structural unit.
And determination means.

Further, in the video index assigning method according to the thirteenth aspect, at the time of structuring the video, at least the structure index for dividing the video into a semantic group and the content and location of an event occurring in the video are defined. In a video index assigning method for assigning a video index including an event index for specifying, in advance, a term whose meaning is established by a plurality of events occurring successively, and a plurality of event indexes expressing the term are input. A state transition pattern that defines the term using a sequence,
A state transition table is set in which a search granularity specified by associating the structural unit of an image defined by a predetermined structural index with the term is associated with each term, and a video index is assigned to the image. At this time, referring to the state transition table, for each structural unit specified by the structural index, the target structural unit and the search granularity match, and an event given in the target structural unit A search is made for a term in which the input sequence of the index and the input sequence of the plurality of event indexes of the state transition pattern match, and if a matching term exists, it is determined that the meaning of the matched term has occurred, An event index or attribute information indicating the establishment of the term is given.

According to a fourteenth aspect of the present invention, at the time of structuring a video, at least a structure index for dividing the video into a semantic group and the content and location of an event occurring in the video are defined. In a video index assigning method for assigning a video index including an event index for specifying, in advance, a term whose meaning is established by a plurality of events occurring successively, and a plurality of event indexes expressing the term are input. A state transition pattern that defines the term using a column, and a search granularity that specifies a structural unit of a video defined by a predetermined structure index in association with the term, an event index existing in the state transition pattern And a central index designating at least one event index. A state transition table corresponding to each image is set, and when an image index is assigned to the video, the event index matching the center index is assigned with reference to the state transition table. For each of the structural units specified by the structural index, the target structural unit and the search granularity match, and the assignment order of the event index assigned in the target structural unit and the plurality of events of the state transition pattern Search for a term that matches the input string of the index, and if there is a matching term, determine that the meaning of the matched term has occurred, and add an event index or attribute information indicating the establishment of the corresponding term Is what you do.

According to a fifteenth aspect of the present invention, there is provided a computer-readable recording medium on which a program for causing a computer to execute the video index assigning method according to the thirteenth or fourteenth aspect is recorded.

[0036] According to a still further aspect of the present invention, there is provided a method for generating a description of a video content, wherein at least a structure index for dividing the video into a semantic group and an event for specifying the content and location of an event occurring in the video. An index is provided, and a video scene of a section divided by the structure index is used as a structure unit of the video, and a description sentence describing the video content is provided for a video structured using a plurality of hierarchical structure units. In the method for generating a description of a video content to be generated, a state transition table in which information to be used for generating the description is set, and a character string or a character string previously assigned to the structure index and the event index can be converted. A description method for generating a description of a video content, wherein the description is generated using attribute information. For each term that can be expressed by a combination of the above events, the generation granularity is set based on a generation granularity in which an appropriate structural unit is set as a video unit when generating the description, and the generation granularity based on an occurrence pattern of an event expressing the term. At least one of a state transition pattern in which the term is defined as an input string of a plurality of event indexes that occur consecutively in the state transition pattern, and an event index that exists in each state transition pattern for each of the state transition patterns A key index set by selecting and, for each state transition pattern, a character string definition information that sets a syntax element used for generating the explanatory note and a character string input source used as a syntax element of the syntax, Is set, and when generating a description of the video content, a pair for generating the description with reference to the state transition table. Search for a generation granularity that matches the structural unit to be determined, and determine whether an event index that matches the key index of the state transition pattern corresponding to the relevant generation granularity exists in the target structural unit, If there is a matching event index, determine whether the corresponding state transition pattern and the occurrence pattern of the event index in the target structural unit match, if the occurrence pattern matches, the It is determined that the term defined in the corresponding state transition pattern is established, and a description of the video scene of the target structural unit is generated using the character string definition information of the state transition pattern of the established term. .

[0037] According to a seventeenth aspect of the present invention, in the method of generating a description of a video content according to the sixteenth aspect, the structure unit further includes a state transition defining a specific term. When a pattern is established, it is possible to add an index representing the term as an abstract index. In the state transition table, for each of the state transition patterns, an abstract index representing a corresponding term is included in the key index. When it is determined whether an event index that matches the key index of the state transition pattern corresponding to the corresponding generation granularity exists in the target structural unit, the event index is abstracted as the key index. The index is preferentially used, and an abstract index corresponding to the target structural unit is given. Is determined, if the abstract index is set, it is determined that the term defined in the corresponding state transition pattern has been established, the character string definition information of the state transition pattern of the established term Then, a description of the video scene of the target structural unit is generated.

[0038] A video content description generating method according to claim 18 is the video content description generating method according to claim 16 or 17, wherein the character string definition information is a character used as the syntax element. The attribute information of the structure index or the event index is set as a column input source.

According to a nineteenth aspect of the present invention, in the method for generating a description of a video content according to any one of the sixteenth to eighteenth aspects, the character string definition information may include a "what time" (When), where (Where), why (Why), who (Who), what (What), and how (How) it is.) The syntax element based on 5W1H is set. is there.

According to a twentieth aspect of the present invention, there is provided the video content description generating method according to the nineteenth aspect, wherein a plurality of state transition patterns corresponding to the corresponding generation granularity exist. It is determined whether or not a term is established for each state transition pattern, and when a plurality of terms are established, the target is obtained by combining the syntax elements of the character string definition information set in the state transition pattern of each term. This is for generating a description of a video scene in structural units.

According to a twenty-first aspect of the present invention, in the method of generating a description of a video content according to the twentieth aspect, the syntax of the character string definition information set in the state transition pattern of each term is provided. When generating the description of the video scene of the target structural unit by combining the elements, 5W
If there are duplicate syntax elements in the 1H syntax elements, the syntax element that refers to the event index that occurs later in time takes precedence.

According to a twenty-second aspect of the present invention, in the method of generating a description of a video content according to the twentieth or twenty-first aspect, the character string definition information set in the state transition pattern of each term is provided. When generating the description of the video scene of the target structural unit by combining the syntax elements of the above, if there are duplicate syntax elements among the syntax elements of 5W1H, the state transition patterns of each term are compared, and The priority is given to the syntax elements of the state transition pattern defined using many event indexes.

A video content description generating method according to claim 23 is the video content description generating method according to any one of claims 19 to 22, wherein the video content description generating method is set in the state transition pattern of each term. When generating the description of the video scene of the target structural unit by combining the syntax elements of the character string definition information, if there are duplicate syntax elements in the 5W1H syntax elements, Are arranged in parallel.

According to a twenty-fourth aspect of the present invention, a computer-readable recording medium records a program for causing a computer to execute the method of generating a description of a video content according to any one of the sixteenth to twenty-third aspects. Things.

[0045]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a video search method of the present invention, a computer-readable recording medium storing a program for causing a computer to execute the method, a video search processing device, a video index assigning method, and a computer-readable program And a computer-readable recording medium storing a program for causing a computer to execute the method, a method for generating a description of a video content, and a computer-readable recording medium storing a program for causing a computer to execute the method. This will be described in detail with reference to the drawings.

(Embodiment 1) A video search method, a video search processing device, and a video index assigning method according to Embodiment 1 are as follows: (1) Device configuration of video search processing device (2) Structure index and event index Relationship (3) Structure of State Transition Table (4) Example of Definition of State Transition Pattern (State Transition Diagram) (5) Example of Specific Video Search Processing Algorithm (6) Specific Operation Example

(1) Apparatus Configuration of Video Search Processing Apparatus FIG. 1 shows a configuration diagram of a video search processing apparatus according to the first embodiment, and FIG. 1A shows an example of a hardware configuration of the video search processing apparatus 100. FIG. 2B shows a functional block diagram of the video search processing device 100. Video search processing device 100
At least a structure index for dividing a video into a semantic group and an event index for specifying the content and location of an event that has occurred in the video are added, and the video scene of the section divided by the structure index is added to the video. With respect to a video structured using a plurality of hierarchically structured units, a desired video scene is searched from the video using the structure index and the event index.

The hardware configuration of the video search processing device 100 may be any device having at least a CPU (Central Processing Unit), a display, a keyboard, and a magnetic disk. For example, a personal computer as shown in FIG. Computer can be used.

As shown in FIG. 3B, the video search processing device 100 includes a video input unit 101 for inputting a structured video to be searched, and a term, search granularity, and state transition described later. A state transition table storage unit 102 storing a pattern and a key index as a state transition table, and an operation input unit 103 for inputting or specifying an inquiry term for searching for a desired video scene
When a query term is input or specified via the operation input unit 103, the state transition table storage unit 102
And using the key index corresponding to the query term to search for a structural unit that matches the search granularity and has an event index that matches the key index, using the key index corresponding to the query term. The video search unit 104 and the structural unit searched by the video search unit 104 are input, and the determination unit 105 determines whether the state transition pattern corresponding to the query term matches the occurrence pattern of the event index in the structural unit. And the structural unit determined to match by the determination unit 105,
A search result output unit 106 that cuts out a video scene from the video and outputs it as a search result.

It is to be noted that the judging section 105 includes the video searching section 104
The first determination unit 105a that determines whether or not all the event indexes included in the state transition pattern corresponding to the query term exist in the structural unit searched for by the first and second determination units 105a A second determination unit 105b that determines whether the state transition pattern corresponding to the query term matches the occurrence pattern of the event index in the structural unit with respect to the determined structural unit. .

(2) Relationship between structure index and event index Next, the video search processing device (video search method) of the present invention
The structured video to be searched will be described.
The structured video is provided with a structure index for dividing the video into a semantic group and an event index for specifying the content and location of an event that has occurred in the video.

FIGS. 2 and 3 are explanatory diagrams showing the relationship between the structure index and the event index. In FIG. 2, for example, if the video data (video scene) in the section indicated by the structure index of the structure 1 represents the entirety of one video, this video is provided below the structure 1 with the structure 2-
a, a structure 2-b, a structure 2-c... Here, the structure 2-a, the structure 2-b,
The section indicated by each structure index such as the structure 2-c is a section obtained by dividing the entire video shown by the structure 1,
In addition, when all of these divided sections are connected, they match the section of the entire video which is a higher section.

Further, below the structure 2-a, the structure 3-a
a, a structure 3-ab, a structure 3-ac, etc. are provided. Similarly, the section indicated by each structure index such as the structure 3-aa, the structure 3-ab, the structure 3-ac... Is a section obtained by dividing the section indicated by the structure 2-a, and When all the sections are connected, they match the section of the structure 2-a which is the upper section.

The section indicated by the structure index of the highest-order structure 1 is a structure unit indicating the entire video, and is defined by the following structure indexes such as structure 2-a, structure 2-b, structure 2-c, etc. The indicated section is a structural unit of the structure 2 level, and the section indicated by each structure index such as a lower structure 3-aa, structure 3-ab, structure 3-ac... Is a structural unit of the structure 3 level. .

As described above, the video (structure 1) is structured using the video scene of the section divided by the structure index as the video structural unit and using a plurality of hierarchical structural units.

In contrast to the structure index representing the structure of the video, the event index represents an event that has occurred on the video. As described above, the structure index indicates a logical section of a video, whereas the event index is basically an index having no section. This event index is basically added to a place where an event has occurred in a video as information indicating the content of the event.
For example, as shown in FIG. 3, the flow of the image (change in the time axis)
When an event occurs in the event index 3aa-
2 to 3aa-4, or the event indexes 3aa-1 to 3aa-4 may be added to the structural index 3aa where the event has occurred, as shown in FIG. Although a detailed description is omitted, it is also possible to separately store and manage a part of structured index information composed of a structure index and an event index and a part of an actual video scene, respectively. Of course you can.

(3) Structure of State Transition Table Next, the structure of the state transition table, which is an important element of the present invention, will be described in detail. According to the present invention, in order to search for a desired video scene using a meaning (term) having a high level of abstraction, a state transition pattern expressing the term is defined using video structure information (structure index and event index). Keep it. This processing corresponds to the state transition table generation step of the present invention.

The state transition table is as shown in FIG.
The following (3-1) to (3-4) are set correspondingly. (3-1) Term: A search term having a high degree of abstraction, which is set as a character string representing the meaning of the corresponding video scene. A term with a high degree of abstraction (a search term) is a term having a fragmentary meaning used as an event index that can be associated with one event on a video, but corresponds to only one event on a video It is a term that can not be expressed and can be expressed by a combination of a plurality of events. In other words, a term with a high degree of abstraction (search term) is meaningful only when the occurrence of a plurality of events in a certain section becomes clear.

(3-2) Search granularity: An appropriate structural unit is set as a search target unit when searching for a video. That is, by setting the minimum structural unit that satisfies the meaning of the term ((3-1)) as the search granularity, the search target unit can be narrowed down and the search can be performed efficiently. (3-3) State transition pattern: Term ((3-1) above)
Based on the occurrence pattern of the event expressing the event, an event corresponding to the term ((3-1)) as an input string of a plurality of event indexes that occur continuously in the search granularity ((3-2)) Defines a list. (3-4) Key index: At least one event index among the event indexes existing in the state transition pattern ((3-3)) is specified as a key (center event) for starting a search. .

In other words, the state transition table is used to efficiently define the meaning of the video contents by searching for a list of event indexes in a logical unit (search granularity) on a certain structure and a character string representing the meaning of the contents. And expressed as a pair with the key index of A search is to discover (parse) a state transition pattern on the state transition table for an image structured by an index.

To explain further with reference to FIG. 4, for a character string "xxx" representing a meaning, first, the term is represented by an input sequence of a plurality of events, and an event list " Event 1, Event 2, Event 3 "are set. This event list is
As shown in FIG. 5, event 1 occurs in state A, then event 2 occurs, then event 3 occurs, and the state transitions to state B.

When an event list for a term is created,
An event that symbolically represents the term among the events 1 to 3 existing in the event list or an event suitable for specifying the term is designated as a key index. Here, event 1 is specified as a key index. continue,
In the structural unit of the structured video, the smallest structural unit in which events 1 to 3 existing in the event list can occur as the meaning of the term is selected and set as the search granularity.
Here, it is described as particle size 1.

Similarly, the character string "0000 ...
For “0”, the term is first represented by an input sequence of a plurality of events, and an event list “event 1, event 2, event 4” is set. This event list indicates that event 1 occurs in state A, then event 2 occurs, then event 4 occurs, and the state transitions to state C, as shown in FIG.

When an event list for a term is created,
Among the events 1, 2, and 4 present in the event list, an event that symbolically represents a term or an event suitable for specifying the term is designated as a key index. Here, event 4 is specified as a key index. Subsequently, in the structured unit of the structured video, the smallest structural unit that can be generated as the meaning of the terms of events 1, 2, and 4 present in the event list is selected and set as the search granularity. Here, it is described as particle size 2.

As shown in FIG. 4, the granularity 1 and the granularity 2 set as the search granularity for different terms may be the same structural unit. An optimal search granularity may be selected for each term, and a plurality of structural units having the same search granularity may exist.

In the case of retrieving video content that means a term using the state transition table, for example, the search efficiency can be improved by narrowing down the search to a structural unit having a search granularity corresponding to the term. Furthermore, after the corresponding event is parsed using the key index, it is possible to perform a high-speed and efficient search by determining whether or not the event list is established in the structural unit of the specified search granularity.

(4) Example of Definition of State Transition Pattern (State Transition Diagram) In the present invention, as described above, terms (and concepts) having a high level of abstraction used by humans for retrieval are previously defined as state transition patterns. It is defined as a path regular expression pattern (state transition pattern).

As a prerequisite, * an event index is added to a video, and it is assumed that the event index is added and a new state is entered using the event index as an input symbol. * Also, the event index is defined as having no temporal width. * The video data between the two event indexes of interest is defined as a scene. * Scenes flow over time.

The flow of the scene, that is, the state of the state transition can be represented by a directed graph using the event index as a label. The nodes of the graph represent scenes. Various parameter values representing the situation are added to each scene as attributes. For example, when the video is a recorded baseball game, the attributes include a score, a position and a player name of a defensive player, a batter's name, an SBO (strike ball out), and the like.

The search is to find a state transition pattern in the indexed video. The found video part (scene or index) is the search result. If the relevant scene includes other relevant scenes, the shortest scene flow among them is used as the search result.

The path regular expression of the state transition pattern is expressed as "s0-I->s1" when "transition from scene s0 to scene s1 by event index I". “.” Represents an arbitrary scene and an arbitrary event index. Also,
“−.−>” is abbreviated as “→”. The transition between two temporally consecutive scenes is represented by “==>”.
“*” Indicates 0 or more repetitions, and “+” indicates 1
It represents the repetition of at least times.

For example, the flow of a scene starting from the scene s0 and reaching the scene s1 by the event index sequence "A. * B. * (C. *) * C" is represented by the following path regular expression. s0-A->. (→.) *-B->. (→.) * (-C-> (→.) *) *-C-> s1 This path expression is abbreviated as follows: I do. s0-ABC +-> s1

Strictly speaking, the regular expression “A. * B. * (C.
Although the expression contents of *) * C "and the regular expression" ABC + "are not the same, in the case of an event index, there are many ignorable event indexes, and it is troublesome to describe each of them. FIG. 6 shows a state transition diagram of the finite state automaton corresponding to the path regular expression.

It is also assumed that, as shown in FIG. 7, event indexes such as A, B, C, and D are added to a video (video scene). When the scenes s0 and s1 shown in FIG. 6 are searched for this video, s0 and s1 are obtained as shown in FIG.

By describing conditional expressions in this state transition pattern (state transition diagram), it is possible to improve the efficiency of retrieval by using structural information. For example, it is assumed that a conditional expression is written in [] after the state transition pattern. “S0-ABC +-> s1 [bats (s0) is the same as batting (s
1)] "

Example using structure information: Search for a reversal scene of a home team. However, the end of the scene sequence in the search results must be taken to the end of the inning.

Here, the definition after def is a simple replacement of a character string. When a scene environment such as x is set, value evaluation occurs.

(5) Example of Specific Video Search Algorithm Before explaining the specific video search algorithm, the processing before video search is confirmed. It is assumed that the state transition table has already been stored in the state transition table storage unit 102 of the video search processing device 100 through the above-described state transition table generation step.

FIG. 8 is a flowchart showing an algorithm of the video search processing according to the first embodiment. When searching for a desired video portion using the video search processing device 100 shown in FIG. 1, a user inputs a video to be searched to the video search processing device 100 via the video input unit 101. In addition,
When the magnetic disk of the apparatus is used as the video input unit 101, it is only necessary to specify the video to be searched.

As a process for performing a video search, first, when a user inputs a term expressing a desired video scene via the operation input unit 103, the user starts searching for a corresponding term from the state transition table in the state transition table. The obtained key index is obtained (S801).

Next, using the key index, an event index that matches the key index is searched, and as a result, a set of key indexes is obtained (S802).
With respect to the set of key indexes obtained in step S802, a structure instance (search granularity) including the key index is obtained from the constraint (state transition pattern) of the structure specified in the state transition table (S80).
3).

Subsequently, for one structure instance (structure unit that matches the search granularity) (S804), it is determined whether or not all event indexes included in the state transition pattern exist (S805). No processing is performed for the structure instance that has not been executed.

If a certain structural instance includes a plurality of key indexes, the corresponding key index is removed from the set of key indexes (S8).
06). Subsequently, it is determined whether or not a state transition is established for the structural units determined to be all present, in other words, the state transition pattern corresponding to the input term and the occurrence pattern of the event index in the structural unit. Are determined (S807), and if they are satisfied, it is determined that the meaning specified by the term with a high degree of abstraction is realized (S808). At this time, if necessary, a new event index corresponding to the established meaning is added to the obtained structure instance, or the information is added as an attribute of the existing event index.

Next, for all the key indices of the obtained set of key indices, the above-mentioned step S80 is performed.
Steps 3 to S808 are repeated (S809).

Thereafter, the obtained structure instance (structure unit) is returned as a return value (S810), and a video scene is cut out as a search result (S811). The extraction of the search result may be performed using the search granularity (structure unit) specified by default. In addition, an arbitrary structure unit including the extracted portion, or an offset specified before or after the key index. It is assumed that clipping can be specified.

As one of the applications of the search, a procedure for generating a commentary (explanatory sentence) which explains the meaning when a pattern matching the state transition pattern is found by parsing the search condition is defined. The commentary can be automatically generated by executing it. That is, it is possible to generate a character string describing a video by using the meaning definition (state transition pattern) of the video content specified in the state transition table. Simply arranging the indices that are fragmented simply means that the terms are simply arranged in parallel. However, if a state transition pattern is used, a user-friendly document can be generated.

(6) Specific Operation Example A specific operation example using the above configuration and the video search processing algorithm will be described. FIG. 9 is an explanatory diagram showing an example of a video index (structure index and event index) when a baseball video is taken as an example. According to the structure index, the game is divided into hierarchies such as times, innings, front / back, bats, and pitches from the start of the game, and is hierarchized. Such a structure is set in advance as a profile when defining a video index.

Further, event indexes such as hit, out, and home run are assigned as needed. As described above, a state transition pattern as shown in FIG. 6 is prepared for an image structured by the image index and in which the explanatory event is set in a fragmentary manner. For example, in the at-bat granularity (at-bat level structural unit), if there is at least one additional event after a hit, it means "timely". Further, if an additional event occurs after the out or fly event, it means "sacrifice hit".

At this time, the central event (key index)
The event defined as is an index that can be used as a search key. A video is searched by relying on this key index, and if a state transition pattern is found at a specified granularity, the granularity becomes a candidate for a search result.

Furthermore, a state transition can be defined for the effect that the event index has on the environmental parameters representing the state of the video. For example, in the case of baseball,
When an additional event has occurred for the at-bat granularity,
The change in the point addition status immediately before the start of the turn at bat and the change in the point addition status immediately after the end of the turn at bat can be designated by expressing the meaning such as "first open", "tie", "reverse", and the like.

Here, an example of the state transition pattern will be specifically described. * Timely hit s0 => bat in == s0'-hit / addition + → s1 =
> Bat in == s1 '* sacrifice bat s0 => pitch in = = s0'-out / addition + → s1 =
> At bat in => s1 '* Combat kill s0 => At bat in = = s0'-out / out → s1 =
> At-bat-in => s1 '* Reversal s0 => At-bat-in =>s0'-additional point + → s1 => At-bat in => s1 '[s0. Home run> s0. Away score && s1. 'Home score> s1. 'Away score || s0. Home run <s0. Away score && s1. 'Home score <s1. 'Away score]

By setting the state transition pattern as described above, it is possible to cope with a search of a meaning having a high degree of abstraction. Conversely, by simply defining this state transition pattern, a search based on the meaning of the content of the video can be performed without depending on the structure of the index information of the video or the event definition. That is, an environment for a unified search query for video index information as semi-structured data (a video index having the characteristic of semi-structured data that the structure of the index differs for each creator and is not fixed). Can be prepared.

In addition, as for the automatic generation of the commentary (explanatory text), information (character string definition information) as to where the subject data is to be taken from is added to the state transition table representing the meaning of the abstraction. By combining the subject, the semantic information, and the semantic information generated by the situation change with a connective in between, it is possible to generate a commentary that is comfortable for the user. For example, for the term “timely hit”, the conditions for generating the description (character string definition information) are specified as follows.

Timely hit s0 => Battery in == s0'-hit / addition → s1 =>
At-bat in => s1 'Character string: <times. Number of times><Inning, front or back><Battery.Batter'sname> [addition] point timely [tie in | reversal |
Additional points] # 1 <times. Number of times><Inning, front or back> ‥‥‥ When # 2 Batter's name> ‥‥‥ Who (subject) # 3 [addition] point timely ‥‥‥ What ’s the # 4 “same | reversal | addition” ‥‥‥ how

Here, <index. Attribute name> indicates the attribute of the specified video index. [Index] indicates the number of times an index has been generated at the specified granularity. "Term" indicates that a term having a high degree of abstract described therein is established.

The description sentence when the search result is obtained by the above designation is in the form of one time reverse XX two-point timely and reversed.

Also, in the case of real-time authoring,
When the key index (central event) of the above-mentioned state transition pattern occurs while the index is being assigned, it is verified whether the state transition pattern is satisfied within the specified granularity before and after the key index (central event). When events that satisfy the state transition pattern occur consecutively, it is assumed that the meaning defined there has occurred, and processing such as adding a new index or adding the information as an attribute of the key index is performed. Do.

For example, if the additional point index continues one or more times after the hit in the at-bat granularity, the hit is determined to be timely, and an index (abstract index) called a timely hit is newly added. Alternatively, processing such as adding “timely” as an attribute of the hit index (event index) is performed.

Further, an index called a timely hit (abstract index) may be set as a key index in the state transition table in advance.

The video search method and video index assigning method according to the first embodiment described above are realized by executing a program in advance by a computer according to the above-described description and the procedure shown in each flowchart. This program is recorded on a computer-readable recording medium such as a hard disk, a floppy (registered trademark) disk, a CD-ROM, an MO, and a DVD.
It is executed by being read from a recording medium by a computer. This program can be distributed via the recording medium or via a network.

(Embodiment 2) The video content explanation generating apparatus of Embodiment 2 basically parses search conditions as one of the applications of the search of the video search processing apparatus 100 of Embodiment 1. When a pattern that matches a state transition pattern is found in a video (in a structural unit), a procedure for generating a commentary (explanatory text) that describes the meaning of the pattern is defined and executed, and a description of the video content is executed. The generation is automatically generated. That is, a character string describing the video content is generated by using the semantic definition (state transition pattern) of the video content specified in the state transition table.

However, the generation of the explanation of the video content is not always executed as a post-processing or an additional function of the video search processing in the video search processing apparatus 100, and the structure index and the event index described above are added. If the video scene in the section divided by the structure index is used as the video structure unit and the video is structured using multiple hierarchical structure units, the video content is provided as an independent function separately from the video search processing. Can be performed. Therefore, in order to clearly express the video content description generation method of the present invention, the apparatus to which the present invention is applied will be referred to as a video content description generation device 200 here.

In the second embodiment, a part of structured index information constituted by a structure index, an event index and an abstract index and a part of an actual video scene are separately managed. When a description of the video content is generated, a part of the index information is input to generate the description. As a result, the processing can be performed separately from the video scene part having a large data amount, so that the load on the apparatus can be reduced and the processing for generating the description can be performed at high speed.

Hereinafter, the video content description generating apparatus 200 according to the second embodiment will be described. (7) Device configuration of the video content description generating apparatus (8) Relationship between event index and abstract index (9) Structure of State Transition Table (10) Example of Algorithm for Generating Description of Video Content (11) Specific Example of Description Generated Using Syntax and Character String Definition Information

(7) Apparatus Configuration of Video Content Description Generation Apparatus FIG. 10 shows a configuration diagram of a video content description generation apparatus according to the second embodiment, and FIG.
FIG. 2B is a functional block diagram of the video content description generating apparatus 200. The hardware configuration of the video content description generation device 200 includes at least a CPU (Central Processing Unit), a display,
Any device that has a keyboard and a magnetic disk may be used.
For example, a personal computer as shown in FIG.

Also, the video content description generating apparatus 200
As shown in FIG. 3B, a video input unit 201 for inputting a structured video to be generated for an explanatory note, and terms, generation granularity, state transition patterns, character string definition information, and key indexes described later. As a state transition table, a state transition table storage unit 202, an operation input unit 203 for inputting various designations required for a process of generating a description of video content, and a description sentence via the operation input unit 203. When the range (structural unit) of the video scene is specified, the state transition table in the state transition table storage unit 202 is referenced to search for a generation granularity that matches the structural unit for which the description is to be generated. A term search unit 204 that searches for terms that are satisfied in the structural unit using the generation granularity;
An explanation generating section 205 that generates an explanation of a video scene using character string definition information of a state transition pattern of a term (established term) searched by the term search section 204, and an explanation generating section 205 generates the explanation. Explanation display part 2 for displaying explanations
06.

Note that the video content explanation generating apparatus 200 according to the second embodiment includes a state transition table in which information used for generating an explanation is set, and a character string or a character string previously assigned to a structure index and an event index. A description is generated using attribute information that can be converted into a character string.

(8) Relationship between Event Index and Abstract Index Next, the video content explanation generating apparatus 200 according to the second embodiment
However, a structured video to be used for generating a description will be described. The structured video used in the second embodiment includes a structure index for dividing the video into a semantic group, an event index for specifying the content and location of an event that has occurred in the video, and When a state transition pattern that defines the term is established, an abstract index for indicating that the meaning of the term is established,
Is given.

Since the relationship between the structure index and the event index is the same as that described in “(2) Relationship between structure index and event index” in the first embodiment, the event index and the abstract are used here. The relationship with the index will be described. In addition, as a video unit used when generating a description, a structure unit defined by a structure index is a generation granularity.

An event index is an index for specifying the content and location of an event that has occurred in a video. In other words, it is provided in association with one event on the video and has fragmentary information indicating the content (meaning) of the event as attribute information.

An abstract index is a meaning represented by a combination of a plurality of events (that is, an abstract index according to the first embodiment).
This is an index that indicates a term with a high degree of abstraction described in (1). The abstract index has meaning only when the occurrence of a plurality of events in a certain section is clarified, and has a meaning that can be expressed by the occurrence pattern of the plurality of event indexes.

On the other hand, in the second embodiment, the definition of the state transition pattern is defined as an input sequence of a plurality of event indexes that occur continuously in the generation granularity based on the event (event index) occurrence pattern. Is defined.

In other words, since the abstract index has a meaning that can be represented by the occurrence pattern of a plurality of event indexes, and the state transition pattern defines a term as an input sequence of the plurality of event indexes, One abstract index and one state transition pattern exist in a one-to-one correspondence with the input sequence (or occurrence pattern) of the event index as a constituent requirement. Therefore, a state transition pattern (input sequence of a plurality of event indexes) corresponding to the abstract index on a one-to-one basis can be expressed using the abstract index. In this case, the state transition pattern can be represented by a smaller number of indexes by expressing the state transition pattern using the abstract index and the event index. Also, by expressing the state transition pattern using the abstract index and the event index, it is possible to easily define a term having a higher degree of abstraction.

Also, the corresponding abstract index can be added to the video only when the state transition pattern is established. When the abstract index is added to the video, the corresponding state transition pattern is added to the video. It means that it holds.

Next, referring to FIGS. 11 (a) to 11 (c),
An example in which a state transition pattern is defined using an abstract index will be described. For example, as shown in FIG. 11A, a state transition pattern of a certain term W is defined by an input column of an event index represented by event 1 (event index) → event 2 → event 3 → event 8 → event 12. Suppose you have

Further, as shown in FIG. 11B, the state transition pattern of a certain term Z is changed from event 1 → event 2 → event 3 →
It is assumed that the event index is defined by an input column of an event index indicated by event 8 → event 12 → event 13.

In such a case, it is defined that the state transition pattern of the term W (event 1 → event 2 → event 3 → event 8 → event 12) is expressed by using the abstract index W corresponding to the term W. In addition, the state transition pattern of the term Z can be described in a simplified manner as shown in FIG. 11C by using the abstract W (abstract index W) and the event 13 (event index).

In defining the state transition pattern, not only the event index but also the abstract index can be used, so that the state transition pattern of a more complex term (a term of a more abstract concept) can be easily defined. Become.

(9) Structure of State Transition Table of Second Embodiment FIG. 12 shows an example of the structure of a state transition table of the second embodiment. A term, a generation granularity, a state transition pattern, a key index, and character string definition information are set in the state transition table.

The term is a term that can be expressed by a combination of a plurality of events, and is the same as the content described in the first embodiment. The generation granularity is obtained by setting an appropriate structural unit as a video unit when generating a description. A structural unit defined by a structural index can be specified. That is, the generation granularity is the same as the search granularity in the first embodiment.

The state transition pattern defines a term as an input sequence of a plurality of event indexes that occur continuously in the generation granularity based on the occurrence pattern of an event expressing the term. In addition, as described above, the state transition pattern can be expressed using not only the event index but also the abstract index. For example, in FIG. 12, the state transition pattern of the term C can be expressed as an input sequence represented by “event 1 → event 4 → event 6 → event 7 → event 8” when expressed only by the event index. When expressed using an abstract index (abstract B), “abstract B
→ Event 7 → Event 8 ”.

The key index is set by selecting at least one of the event indexes existing in the state transition pattern. In the second embodiment, it is assumed that an abstract index representing a corresponding term, in other words, an abstract index representing the entire corresponding state transition pattern is set as one of the key indexes.

In the character string definition information, a syntax element used for generating an explanatory note and an input source of a character string used as a syntax element of the syntax are set. Also, attribute information of a structure index or an event index is set as an input source of a character string used as a syntax element. For example, a setting is made such that the content of the attribute information of １ of event 1 (event index) is a character string input source. As the input source of the character string, it is possible to simply specify the type of the attribute information as described later, or to use ““ ”as in the syntax element (What) of the term C.
A character string (xxxxxx) may be directly described between "and"", and this character string may be used as an input source.

As syntax elements, “when (When),
Where (Where), Why (Why), Whose (Wh)
o), what (what) and how (how) the syntax element based on 5W1H is set. In the character string definition information, all 5W1H syntax elements may be set, or only some of them may be set. It is also possible to set other syntax elements.

Here, the state transition table shown in FIG. 12 is regarded as a table in which an abstract index representing each term is defined, and an example of the definition of the abstract index in the case of a baseball video will be specifically described. I do.

For example, if the abstract index of the term "1 base hit" is defined, & ABSINDEX 1 base hit & RANGE pitch & PATTERN hit & K
EY hit & EXP <When: inning_time, Who: batter_name, What: "1
Base hit ">.

The above description has been created according to the following rules. "& ABSINDEX character string": & ABSINDEX is a declarator (identifier) of the abstract index, and the following "character string" indicates that the name (term) and meaning of the abstract index is "character string". “& RANGE character string”: & RANGE is a declarator of the structural unit (generation granularity), and the following “character string” specifies the structural unit (generation granularity).

"& PATTERN character string": & PATTERN is a state transition pattern declarator, and the following "character string" expresses the state transition pattern. For example, "& PATTERN hit" indicates that the state transition pattern is composed of one event (event index of hit). "& KEY character string": & KEY is a key index declarator, and the following "character string" specifies the key index. As the key index, in addition to the following “character string”, the abstract index itself declared in “& ABSINDEX character string” is automatically specified.

"& EXP <character string>": & EXP is a declarator of character string definition information, and the following "<character string>" defines a syntax element and its input source. For example, <When: inni
ng_time, Who: batter_name, What: "1 base hit">
The syntax element (When) has attribute information (inning_time) as an input source, and the syntax element (Who) has attribute information (batter_na).
me) as the input source and the syntax element (What) is “1 base hit”
(Indicating that a character between “,” is to be input).

Next, another example of the definition of an abstract index is shown below. Example: Abstract index of 2 run & ABSINDEX 2 run & RANGE pitch & PATTERN hit, 2
Base advance & KEY hit & EXP <When: inning_time, Who: batter_name, What: "2
Base hit "> Example: Abstract index of timely hit & ABSINDEX Timely hit & RANGE pitch & PATTER
N 1 base hit, additional point + & KEY additional point & EXP <When: inning_time, Who: batter_na
me, What: "Timely hit"> Example: Abstract index of timely two bass & ABSINDEX Timely two bass & RANGE pitch & PA
TTERN 2 base hits, additional points + & KEY additional points & EXP <When: inning_time, Who: batter_na
me, What: "Timely two base"> Example: Abstract index of reversal & ABSINDEX Reversal & RANGE pitch & PATTERN Additional point + ｛# o
ffense_score <#defense_score &&#result_offens
e_score>#defense_score｝& KEY additional point & EXP <How: "reverse">

(10) Example of Algorithm for Generating Explanation of Video Content Next, an algorithm for generating the description of video content will be described with reference to the flowchart in FIG. When generating a description of a video content using the video content description generation device 200 of the second embodiment, first, a video for which a description of the video content is to be generated, that is, a video of a target structural unit (only index information) Operation input unit 20).
3 and input via the video input unit 201 (S130).
1).

Here, as a method of inputting a video of a target structural unit, for example, the user specifies, via the operation input unit 203, a range of a video scene in which a description is desired to be generated. How to input video (index information) in structural units corresponding to the range of video scene,
Alternatively, a method may be used in which a specific structural unit is set in advance as a range for generating an explanatory note, and a video of the specific structural unit is automatically cut out and input. In the former method, the user can select a desired video scene and generate an explanatory note of only the desired video scene. In the latter method,
An explanatory note can be generated automatically and continuously for each specific structural unit.

Next, the term retrieval unit 204 refers to the state transition table in the state transition table storage unit 202 and searches for a generation granularity that matches the structural unit of the video for which the description is generated (S1302). As a result of this search, it is determined whether there is a matching generation granularity (S1303). If there is a matching generation granularity, the process proceeds to step S1304, and if there is no matching generation granularity, the process proceeds to step S1307. The term search unit 204 searches the state transition table in order from the top until there is no matching generation granularity, or every time a matching generation granularity is searched, the process proceeds to step S1302.
Then, control goes to a step S1303.

Subsequently, the term search unit 204 determines whether an event index that matches the key index of the state transition pattern corresponding to the corresponding generation granularity exists in the target structural unit (S1304). ), If there is no matching event index, step S
Returning to step 1302, the next matching generation granularity is searched.
On the other hand, if there is a matching event index,
It is determined whether or not the input sequence of the event index of the state transition pattern matches the occurrence pattern of the event index in the target structural unit (S1305).

If the occurrence patterns match in step S1305, it is determined that the term defined in the state transition pattern has been established, and the abstract index of the established term is added to the video (index information) input in step S1301. (S1306), and step S1302.
Return to By adding the abstract index of the established term to the video, for example, it becomes possible to use the abstract index when generating the description again from the same video.

The above steps S1302 to S13
The process of step 06 is executed until there is no matching generation granularity in the state transition table. In other words, it is determined whether or not a term of each generation granularity holds for all matching generation granularities in the state transition table, and a set of satisfied terms is extracted.

If it is determined in step S1303 that there is no matching generation granularity, the explanation generation unit 205
Generates an explanatory note using the character string definition information of the established term (S1307), displays the generated explanatory note on the explanatory note display unit 206, and ends the process. Note that a syntax based on 5W1H is prepared in advance in the explanatory sentence generating unit 205 for generating an explanatory sentence. When generating the description, the description generation unit 205 generates the description by arranging the syntax elements of the character string definition information in the syntax based on 5W1H.

In the above description sentence generation algorithm, the description has been made on the assumption that the key index is an event index. However, after step S1303,
By preferentially using an abstract index as a key index, a process of determining whether an abstract index (key index) corresponding to a target structural unit is added is added, and an abstract index is set. In this case, by adding a process of determining that the term defined in the corresponding state transition pattern has been established, it is possible to speed up the process of determining whether or not the term is established. After that, if there is no abstract index, step S1304 and subsequent steps are executed using the event index specified as the key index.

(11) Specific Example of Explanation Text Generated Using Syntax and Character String Definition Information Next, an example of a syntax based on 5W1H and an example of an explanation text generated using the syntax and character string definition information Are specifically mentioned. For the sake of simplicity, the following 5W1H syntax example is used as a syntax example.

Case where One Term is Satisfied (Example 1) First, the simplest example of generating a description will be described. For example, the established term is “timely hit” and its character string definition information is “& EXP <When: inning_time, Who: batter_name, What:”
If it is timely hit ">", three syntax elements "When", "Who", and "What" can be obtained from the character string definition information of the established term.

Assume that the content (character string) of the attribute information specified as the input source of the syntax element is “inning_time = 1 back”, “batter_name = xx”, “What:“ timely hit ”” , From the above syntax and string definition information,
The following description is generated. One time back (None) (None) XX Timely hit (None) When Where Why Who What How (None) indicates the part without the applicable information, and it is not necessary that all 5W1H syntax elements exist. Absent. As described above, an explanatory note can be generated using the syntax element obtained from the character string definition information. When a syntax element other than 5W1H is specified in the character string definition information, the syntax may be adjusted as appropriate to lengthen the description, or a plurality of syntaxes incorporating other syntax elements may be specified in advance. It may be prepared and used appropriately by selecting.

In the case where there are a plurality of established terms (Example 2) In the case where there are a plurality of terms which have a meaning, the explanatory sentence generating unit 205 sets the syntax element defined in the character string definition information of each term. Is extracted, and the extracted syntax elements are combined, arranged at the position of the corresponding syntax element of the 5W1H syntax, and an explanatory sentence is generated.

For example, the terms that have been established are “timely hit” and “reverse”, and the character string definition information is “& EXP <When: inning_time, Who: batter_name, What:”
In the case of "Timely hit"> and "& EXP <How:" Reverse ">", from the character string definition information of the established term, the syntax elements "When", "Who", "What", The four syntax elements “How” are obtained.

Also, the content (character string) of the attribute information specified as the input source of the syntax element is “inning_time = 1 back”, “batter_name = xx”, “What:“ timely hit ””, “How:” Inversion "], from the above syntax and character string definition information,
The following description is generated. One time back (None) (None) Reversal with timely hit of XX When Where Why Who What How

When there are a plurality of established terms (Example 3) When there are a plurality of terms that have a meaning and there are duplicate syntax elements (for example, when there are a plurality of "Who" syntax elements), Refers to the state transition pattern of each term that includes a duplicated syntax element, and selects the syntax element that refers to the later event index in the state transition pattern.

For example, two terms that have been established are “1 base hit” and “timely hit”, and the character string definition information is “& EXP <When: inning_time, Who: batter_name, What:”
"1 base hit">] "& EXP <When: inning_time, Who: batter_name, What:"
In the case of the timely hit ">", three syntax elements of "Wen", "Who", and "What" are obtained as syntax elements from the character string definition information of the established term. Syntax elements are all duplicated.

When the content (character string) of the attribute information specified as the input source of the syntax element is a single-base hit: “inning_time = 1 back”, “batter_name = xx”, “What:“ single-base hit ”” In the case of a timely hit: “inning_time = 1 back”, “batter_name = xx”, “What:“ timely hit ””. Here, since the contents of "inning_time" are the same for both "one time back", there is no need to particularly select them, but since the contents of "batter_name" and "What" are different, one must be selected.

In such a case, the state transition pattern of each term including the duplicated syntax element is referred to, and a syntax element which refers to a later event index in the state transition pattern is selected. For example, when the state transition pattern of the first base hit is & PATTERN hit, the state transition pattern of the timely hit is & PATTERN 1 base hit, additional point +, the event index (hit), the event index (1 base hit), and the event in the range of the video scene. The positions at which the index (addition point +) is assigned are compared, and the event index assigned at a later position in time is specified. Here, a description will be given assuming that the event index (addition point +) is assigned to a position that is later in time.

Since the term referring to the specified event index (addition point +) is a timely hit, the syntax element referring to the later event index in the state transition pattern is time Select the syntax element for the lead hit.

The following explanatory text is generated from the syntax and the character string definition information. Once back (none) (none) XX timely hit (none) When Where Why Who What How

Further, by adding a condition for generating an explanatory note when there are a plurality of established terms, an explanatory note can be generated by arranging overlapping syntax elements in the order of occurrence time as follows. 1 time back XX 1 base hit (following) XX timely hit When Who What Who What

In the case where there are a plurality of approved terms (Example 4) As another example of generating a description sentence in the case where there are a plurality of established terms, when there is a duplicate syntax element in the 5W1H syntax element, The state transition pattern of each term is compared, and a description is generated by giving priority to the syntax elements of the state transition pattern defined using more event indexes.

For example, the established terms are three: “timely hit”, “timely to base”, and “reverse”, and the character string definition information is “& EXP <When: inning_time, Who: batter_name, What: "
Timely hit ">""& EXP <When: inning_time, Who: batter_name, What:"
In the case of "Timely two base"> and "& EXP <How:" Reverse ">", from the character string definition information of the term that has been established, the syntax elements "When", "Who", "What", Although four syntax elements “How” are obtained, three syntax elements other than “How” are duplicated.

When the content (character string) of the attribute information specified as the input source of the syntax element is a timely hit: "inning_time = 1 back""batter_name = xxx""What:" timely hit "" Timely hit: "inning_time = back once""batter_name = xx""What:" timely to base "". Here, the contents of “inning_time” are the same for both “one time back” and “batter_name” is “×
×) is the same, so there is no need to select it.
at ”is different, so you have to choose one.

In such a case, the state transition patterns of the terms including the overlapping syntax elements are compared, and the syntax element of the state transition pattern defined using more event indexes is preferentially selected. For example, if the state transition pattern of the timely hit is & PATTERN 1 base hit, additional point + timely two base state transition pattern is & PATTERN hit, 2nd advance, additional point +, the event index referenced in the state transition pattern The number of "timely hits" is the event index (1 base hit) and the event index (additional point +), and "timely two bases" is the event index (hit) and the event index (second base). Event index (additional point +). Therefore, the number of references of the timely two base (3)> the number of references of the timely hit (2), and here, the syntax element of the timely two base is selected.

The following explanatory text is generated from the syntax and the character string definition information. Once back (None) (None) XX timely two-base reversing When Where Why Who What How

As another example of generating a description, as shown below, simply overlapping syntax elements may be listed in parallel. 1st back between 2nd and 3rd base XX hits and reverses Gusts of XX errors When Where Why Who What How

That is, the description is described as "Reverse due to a gust between the first and second and third bases, (error in xx, hit in xx)". As described above, depending on the situation, simply enumerating in parallel may be able to convey events occurring in the video scene as more accurate information.

[0159] The method of generating a description of a video content according to the second embodiment described above is realized by executing a program in advance by a computer according to the procedure described in the above description and each flowchart. This program is for hard disk, floppy disk, CD-R
The program is recorded on a computer-readable recording medium such as OM, MO, and DVD, and is executed by being read from the recording medium by the computer. This program can be distributed via the recording medium or via a network.

[0160]

As described above, according to the video search method of the present invention (claims 1 to 8), when a video is searched for a term that can be expressed by a combination of a plurality of events in advance. An appropriate structural unit as the search target unit is set as the search granularity, and the state corresponding to the term as an input string of multiple event indexes that occur consecutively in the search granularity based on the occurrence pattern of the event expressing the term When a transition pattern is defined and a desired video scene is searched from a video, a term expressing the desired video scene is input, and a search granularity corresponding to the input term is used as a search target unit, and a search granularity and For each matching structural unit, it is determined whether the state transition pattern corresponding to the input term matches the occurrence pattern of the event index in the structural unit, and the matching structural unit is detected. For outputting as a result, it can be accomplished using a high term or concept abstract queries for meanings of contents of the video, and can perform high-speed search.

Further, for each state transition pattern corresponding to a term, at least one of the event indexes existing in each state transition pattern is designated as a key index for starting a search, and is used in the video. When searching for a desired video scene from, a search is made for a structural unit having an event index that matches the key index, and a structural unit that matches the search granularity. By determining whether or not the occurrence pattern of the event index in the structural unit matches, the key index search (that is, keyword search) is performed at high speed to narrow down the structural unit to be targeted, and then the state transition pattern is determined. Uses terms or concepts with a higher level of abstraction because of the It is possible to increase the speed of the video search by Align.

When extracting a desired video scene from a video based on the structural unit output as a search result, the video scene specified by the structural unit is output.
That is, since the video scene is output in an appropriate structure unit set in advance as a search granularity when searching for a video, a video scene close to a video portion desired by the user can be easily output.

Further, when a desired video scene is extracted from a video based on the structural unit output as a search result, an arbitrary upper or lower structural unit on the structured video can be designated. By combining the input of the query using the abstract term and the specification of the structural unit to be output, the convenience in performing the search can be improved. For example, when there is a video scene that the user actually wants to see before and / or after the structural unit searched by the abstract term expressing the video scene, or the structural unit searched by the abstract term expressing the video scene This is convenient when there is a video scene that the user actually wants to see in a part (lower-order structural unit).

When extracting a desired video scene from a video based on the structural unit output as a search result, an offset for video clipping is designated before and after the place where the key index is added. Therefore, the convenience in performing the search can be improved. For example, it is convenient when the user wants to confirm the situation before and / or after the structural unit searched by using the abstract term representing the video scene.

When it is determined whether the state transition pattern corresponding to the input term matches the occurrence pattern of the event index in the structural unit, an index representing the term is added to the matching structural unit by the abstract index. When the video is searched again using the same abstract term, the key index (abstract index) is used to further increase the search speed. Can be achieved. When the structural unit output as a search result is confirmed again later, the same structural unit can be reliably searched using this key index (abstract index).

Further, a plurality of attribute information items are added to the structure index and the event index, and the state transition pattern corresponding to the term is explained using the attribute information of the structure index and the event index. Character string definition information for generating a sentence is added, and based on the character string definition information of the state transition pattern,
Interpret the meaning of the video content, convert it to a general meaningful character string, and use it to generate a description related to the term by referring to the attribute information in the structural unit output as the search result It is possible to generate an explanatory character string of the video content that is easy for the user to understand.

Further, since a state transition pattern corresponding to a term is defined as an input string composed of the event index and the abstract index using the abstract index in addition to the event index, the state transition pattern can be easily defined, and the search processing can be performed. It is also possible to increase the speed.

A video search method according to the present invention (claim 9)
According to, as a process before performing a video search, in advance, for a term that can be expressed by a combination of a plurality of events,
An input sequence of a plurality of event indexes that occur consecutively in the search granularity based on the occurrence pattern of the event expressing the term by setting the appropriate structural unit as the search target unit when searching for video, based on the occurrence pattern of the event expressing the term A state transition pattern corresponding to the term is defined as, and, for each state transition pattern corresponding to the term, at least one event index among the event indexes present in each state transition pattern is used as a key for starting a search. Including a state transition table generation step of generating a state transition table including a term, a search granularity, a state transition pattern, and a key index by designating as an index, as a process for performing a video search, a term expressing a desired video scene In the term input process, and the state transition table with reference to the state transition table. A search step of searching for a structural unit having an event index that matches the key index by using the search granularity corresponding to the input term as a search target unit and using the key index corresponding to the term, and referring to the state transition table. A first determining step of determining whether all the event indexes included in the state transition pattern corresponding to the term exist in the structural unit searched in the searching step; and determining that all of the event indexes exist in the first determining step. A second determination step of determining whether the state transition pattern corresponding to the input term matches the occurrence pattern of the event index in the structural unit with respect to the determined structural unit; A search result output step of cutting out a video scene from the video based on the structural unit determined to match and outputting the search result as a search result. Inquiries about the meaning of video content can be made using terms or concepts with a high degree of abstraction, and key index searches (ie, keyword searches) are performed at high speed to narrow down the target structural units, and then state Since the determination using the transition pattern is performed, the speed of the video search by the inquiry using the term or concept having a higher degree of abstraction can be increased.

Furthermore, according to the computer-readable recording medium of the present invention (claim 10), a program for causing a computer to execute the video search method according to any one of claims 1 to 9 is recorded. By causing the computer to execute this program, it is possible to make an inquiry about the meaning of the content of the video using terms or concepts with a high degree of abstraction, and to perform a key index search (that is, a keyword search) at high speed. Since the determination using the state transition pattern is performed after narrowing down the target structural unit, it is possible to speed up the video search by an inquiry using a term or concept with a higher degree of abstraction.

Further, according to the video search processing device of the present invention (claims 11 and 12), a video input means for inputting a structured video to be searched and a video input means for designating a desired video scene to be searched. A term, a search granularity that specifies a structural unit as a search target unit when searching for video using terms, and a state transition pattern that defines the term as an input sequence of multiple event indexes that occur consecutively in the search granularity And a key index designating at least one event index among the event indexes present in the state transition pattern, a storage means storing a state transition table, and an inquiry term for searching for a desired video scene. Or an operation input unit for specifying, and a storage unit when an inquiry term is input or specified via the operation input unit. Search means for referring to the state transition table and searching for a structural unit having an event index that matches the search granularity and matches the key index from the video input by the video input means using the key index corresponding to the query term And inputting the structural unit searched by the searching means, determining means for determining whether the state transition pattern corresponding to the query term matches the occurrence pattern of the event index in the structural unit, A search result output unit that cuts out a video scene from the video based on the determined structural unit and outputs the video scene as a search result, so that inquiries about the meaning of the video content are made using terms or concepts with a high degree of abstraction. Key index search (ie, keyword search) After narrowing down the structural unit comprising the elephants, since the determination using the state transition patterns, it is possible to increase the speed of video retrieval by query further with higher terms or concepts abstract.

Further, according to the video index assigning method of the present invention (claim 13), a term whose meaning is established by a plurality of successive occurrences of an event and a plurality of event indexes expressing the term are determined in advance. Set a state transition table that associates, for each term, a state transition pattern that defines terms using an input string, and a search granularity that specifies a structural unit of video defined by a predetermined structure index in association with the term In addition, when a video index is assigned to a video, the search granularity matches the target structural unit and the search granularity for each structural unit specified by the structural index with reference to the state transition table, and A search is made for a term in which the assignment order of the event indexes assigned in the structural unit matches the input sequence of the multiple event indexes of the state transition pattern, and a match is found. If a term exists, it is determined that the meaning of the matched term has occurred, and an event index or attribute information indicating the establishment of the corresponding term is added. Can be done at high speed.

Further, according to the video index assigning method of the present invention (claim 14), a term whose meaning is established by a plurality of consecutive occurrences of an event and a plurality of event indexes expressing the term are determined in advance. A state transition pattern in which terms are defined using an input sequence, and a search granularity in which a structural unit of a video defined by a predetermined structure index is specified in association with the term, are included in an event index existing in the state transition pattern. A state transition table in which at least one event index is designated and a center index corresponding to each term is set, and when a video index is assigned to a video, the state index is referred to by referring to the state transition table. When a matching event index is assigned, a target is set for each structural unit specified by the structural index. Search for terms whose structure unit and search granularity match, and in which the order of assignment of event indexes assigned in the target structural unit matches the input sequence of multiple event indexes of the state transition pattern, and match If a term exists, it is determined that the meaning of the matched term has occurred, and an event index or attribute information indicating the establishment of the corresponding term is added. Can be done at high speed.

According to the computer-readable recording medium of the present invention (claim 15), a program for causing a computer to execute the video index assigning method according to claim 13 or 14 is recorded. By causing the computer to execute, it is possible to efficiently and quickly assign an index using terms having a high degree of abstraction.

In addition, according to the method for generating a description of a video content according to the present invention (claims 16 to 22), a description is generated in the state transition table for each term that can be expressed by a combination of a plurality of events. The term is defined as an input sequence of multiple event indexes that occur continuously in the generation granularity based on the generation granularity that sets an appropriate structural unit as the video unit at the time and the occurrence pattern of the event expressing the term A state transition pattern, a key index that selects and sets at least one of the event indexes existing in each state transition pattern for each state transition pattern, and a description for each state transition pattern. And the character string definition information that sets the input source of the character string to be used as the syntax element of the syntax. At the time of generation, search for a generation granularity that matches the structural unit for which the description is to be generated with reference to the state transition table, and an event index that matches the key index of the state transition pattern corresponding to the relevant generation granularity Is determined in the target structural unit, and if there is a matching event index, whether the corresponding state transition pattern matches the occurrence pattern of the event index in the target structural unit Is determined, and when the occurrence patterns match, it is determined that the term defined in the corresponding state transition pattern is established, and the target structural unit is used using the character string definition information of the state transition pattern of the established term In order to generate a description of the video scene, the meaning of the video content is interpreted and converted into a general meaning character string, There video contents of description (description string) can be generated. In other words, for the target structural unit (video), it is possible to generate an optimal description (character string) according to the meaning of the content of the video generated in that part.

When a state transition pattern defining a specific term is established, an index representing the term can be assigned to the structural unit as an abstract index. An abstract index representing the corresponding term is set as one of the key indexes for each pattern, and an event index that matches the key index of the state transition pattern corresponding to the corresponding generation granularity exists in the target structural unit. When determining whether or not to perform an abstract index, it is determined whether or not the abstract index corresponding to the target structural unit is assigned by preferentially using the abstract index as the key index, and the abstract index is set. in case of,
It is determined that the term defined in the corresponding state transition pattern has been established, and a description of the target structural unit video scene is generated using the character string definition information of the state transition pattern of the established term. It is possible to efficiently specify a term corresponding to the meaning of the content (whether or not the term is satisfied) in a short time.

In the character string definition information, since the attribute information of the structure index or the event index is set as the input source of the character string used as the syntax element, the state transition pattern matches and the term is established. In such a case, since the attribute information is always present in the necessary structure index and state transition pattern, the description can be generated by reliably inputting the character string.

The character string definition information includes “what time (Wh
en), where (Where), why (Why), who (Who), what (What), how (How)
Since the syntax elements based on 5W1H of "Nita" have been set, it is possible to easily generate an explanatory sentence including information on 5W1H using these syntax elements.

When there are a plurality of state transition patterns corresponding to the corresponding generation granularity, it is determined whether or not a term is established for each state transition pattern. Since the description of the video scene of the target structural unit is generated by combining the syntax elements of the character string definition information set in the transition pattern, it is possible to generate the description in more detail.

When generating a description of a video scene of a target structural unit by combining syntax elements of character string definition information set in a state transition pattern of each term, 5W
If there is a duplicate syntax element in the 1H syntax element, a description element using newer information (attribute information) may be generated to give priority to a syntax element that refers to an event index that occurs later in time. it can.

Also, when generating a description of a video image of a target structural unit by combining syntax elements of character string definition information set in the state transition pattern of each term, 5W
If there is a duplicate syntax element in the 1H syntax element, newer information is used to compare the state transition pattern of each term and give priority to the state transition pattern syntax element defined using more event indexes. It is possible to generate a description using (attribute information) and express the meaning of the content of the video generated in that part with a more appropriate description (character string). That is, giving priority to the syntax element of the state transition pattern defined by using more event indexes means, for example, when there is a second state transition pattern that is configured by a part of the first state transition pattern. First
If the state transition pattern of
However, since the syntax element of the first state transition pattern having a large event index is always selected, the final result in the video of that portion can be selected as the syntax element.

When a syntactic element of character string definition information set in the state transition pattern of each term is combined to generate a description of a video scene of a target structural unit, 5 W
If there are duplicate syntax elements in the 1H syntax elements, the syntax elements are arranged in parallel as necessary, so that a more informative, in other words, an easy-to-understand explanation can be generated.

According to a twenty-fourth aspect of the present invention, a computer-readable recording medium records a program for causing a computer to execute the method for generating a description of a video content according to any one of the sixteenth to twenty-third aspects. By interpreting the meaning of the content of the video and converting it into a character string having a general meaning, it is possible to generate a description (description character string) of the video content that is easy for the user to understand.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of a video search processing device according to a first embodiment of the present invention.

FIG. 2 is an explanatory diagram showing a relationship between a structure index and an event index.

FIG. 3 is an explanatory diagram showing a relationship between a structure index and an event index.

FIG. 4 is an explanatory diagram illustrating a state transition table according to the first embodiment;

FIG. 5 is an explanatory diagram illustrating a state transition table according to the first embodiment;

FIG. 6 is a state transition diagram of a finite state automaton corresponding to a path regular expression according to the first embodiment.

FIG. 7 is a diagram illustrating a state transition of the finite state automaton corresponding to the path regular expression according to the first embodiment.

FIG. 8 is a flowchart illustrating an algorithm of a video search process according to the first embodiment.

FIG. 9 is an explanatory diagram showing an example of a video index (structure index and event index) when a baseball video is taken as an example.

FIG. 10 is a configuration diagram of a video content description generating device according to a second embodiment.

FIG. 11 is an explanatory diagram showing an example in which a state transition pattern is defined using an abstract index according to the second embodiment.

FIG. 12 is an explanatory diagram illustrating a state transition table according to the second embodiment;

FIG. 13 is a flowchart showing an algorithm of a process for generating a description of a video content according to the second embodiment.

[Explanation of symbols]

 REFERENCE SIGNS LIST 100 video search processing device 101 video input unit 102 state transition table storage unit 103 operation input unit 104 video search unit 105 determination unit 105 a first determination unit 105 b second determination unit 106 search result output unit 200 video content description generation device 201 video input unit 202 state transition table storage unit 203 operation input unit 204 term search unit 205 explanatory sentence generation unit 206 explanatory sentence display unit

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Yukari Shirata 1-3-6 Nakamagome, Ota-ku, Tokyo Inside Ricoh Company (72) Inventor Hiroko Mano 1-3-6 Nakamagome, Ota-ku, Tokyo Stock Ricoh Company (72) Inventor Atsushi Iizawa 1-3-6 Nakamagome, Ota-ku, Tokyo F-term in Ricoh Company (reference) 5B075 ND12 ND35 NK10 NS01 PP02 PQ02 PR06 5C052 AA01 AB03 AC08 CC06 CC11 DD04 (54) [ Patent application title: VIDEO SEARCH METHOD, COMPUTER-READABLE RECORDING MEDIUM CONTAINING PROGRAM FOR CAUSING COMPUTER TO EXECUTE THE METHOD, VIDEO SEARCH PROCESSOR, VIDEO INDEX ADDING METHOD, AND PROGRAM FOR CAUSING COMPUTER TO EXECUTE THE METHOD Computer readable record Body, computer-readable recording medium recording a program for executing a description generation method and method of the video content to the computer

Claims (24)

[Claims] At least a structure index for dividing a video into a semantic group and an event index for specifying the content and location of an event that has occurred in the video are provided, and a video of a section divided by the structure index is provided. A video search in which a scene is a structural unit of a video, and a video structured using a plurality of hierarchical structural units is searched, and a desired video scene is searched from the video using the structure index and the event index. In the method, in advance, for a term that can be expressed by a combination of a plurality of events, an appropriate structural unit is set as a search granularity as a search target unit when searching for the video, and an occurrence pattern of an event expressing the term Based on the above, as an input column of a plurality of event indexes that occur consecutively in the search granularity, A state transition pattern corresponding to a word is defined, and when searching for a desired video scene from the video, a term expressing the desired video scene is input, and a search granularity corresponding to the input term is searched. As a unit, for each structural unit that matches the search granularity, it is determined whether the state transition pattern corresponding to the input term matches the occurrence pattern of the event index in the structural unit, and the matched structural unit is used as a search result. A video search method characterized by outputting. Further, for each state transition pattern corresponding to the term, at least one of the event indexes present in each of the state transition patterns is designated as a key index for starting a search. When searching for a desired video scene from the video, after searching for a structural unit that matches the search granularity and that has an event index that matches the key index, 2. The video search method according to claim 1, wherein it is determined whether or not the state transition pattern matches an occurrence pattern of an event index in a structural unit. 3. The video scene specified by the structural unit is output when a desired video scene is extracted from the video based on the structural unit output as the search result. Or the video search method according to 2. 4. When extracting a desired video scene from the video based on the structural unit output as the search result, an arbitrary upper or lower structural unit on the structured video can be designated. 3. The video search method according to claim 1, wherein: 5. When extracting a desired video scene from the video based on the structural unit output as the search result, an offset for video clipping is designated before and after the location where the key index is added. 3. The video search method according to claim 2, wherein the video scene is extracted by using the video scene. 6. When it is determined whether a state transition pattern corresponding to the input term matches an occurrence pattern of an event index in a structural unit, an index representing the term is abstracted in the matching structural unit. The video search method according to any one of claims 2 to 5, wherein the video search method is defined as an index, newly added, and reused as a key index. 7. An abstract index is set in advance for each of the terms that can be expressed by a combination of a plurality of events as an index representing each term, and when defining the state transition pattern, an abstract index is set in addition to the event index. 7. The video search method according to claim 1, wherein a state transition pattern corresponding to the term is defined as an input string including an event index and an abstract index using the abstract index. 8. A plurality of attribute information items are added to the structure index and the event index, and a state transition pattern corresponding to the term is expressed by using each attribute information of the structure index and the event index. Character string definition information for generating a description sentence related to is added, and based on the character string definition information of the state transition pattern, referring to attribute information in the structural unit output as the search result The method according to any one of claims 2 to 7, wherein a description sentence related to the term is generated. 9. At least a structure index for dividing a video into a semantic group and an event index for specifying the content and location of an event that has occurred in the video are provided, and the video of a section divided by the structure index is provided. A video search in which a scene is a structural unit of a video, and a video structured using a plurality of hierarchical structural units is searched, and a desired video scene is searched from the video using the structure index and the event index. In the method, as a process before performing a video search, in advance, for a term that can be expressed by a combination of a plurality of events, an appropriate structural unit as a search target unit when searching for the video is set as a search granularity, A plurality of events that occur consecutively in the search granularity based on the occurrence pattern of the event expressing the term Define the state transition pattern corresponding to said term as input columns for index,
Further, for each state transition pattern corresponding to the term, at least one event index among event indexes present in each state transition pattern is designated as a key index for starting a search, and the term A state transition table generating step of generating a state transition table composed of a state transition pattern and a key index. As a process for performing a video search, a term input step of inputting a term representing a desired video scene; With reference to the transition table, the search granularity corresponding to the term input in the term input step as a search target unit,
And, using a key index corresponding to the term,
A search step of searching for a structural unit having an event index that matches the key index; and referring to the state transition table, the structural unit searched in the search step is included in a state transition pattern corresponding to the term. A first determination step of determining whether or not all event indexes are present; and a state transition pattern and a structural unit corresponding to the input term for the structural units determined to be all present in the first determination step. A second determining step of determining whether or not the occurrence patterns of the event indexes in the image pattern match; and extracting a video scene from the video based on the structural unit determined to match in the second determining step. And a search result output step of outputting as a search result. 10. A computer-readable recording medium on which a program for causing a computer to execute the video search method according to claim 1 is recorded. 11. A video of a section divided by the structure index, provided with at least a structure index for dividing the video into a semantic group and an event index for specifying the content and location of an event that has occurred in the video. A video search in which a scene is a structural unit of a video, and a video structured using a plurality of hierarchical structural units is searched, and a desired video scene is searched from the video using the structure index and the event index. In the processing device, a video input unit for inputting the structured video to be searched, a term for specifying a desired video scene to be searched, and a structure as a search target unit when searching for a video by the term A search granularity that specifies a unit and an input of a plurality of event indexes that continuously generate the term in the search granularity. Storage means for storing, as a state transition table, a state transition pattern defined as a column and a key index designating at least one event index among the event indexes present in the state transition pattern; Operation input means for inputting or specifying an inquiry term for searching for, and when the query term is input or specified via the operation input means, refer to a state transition table of the storage means, Using a key index corresponding to a term, a search unit that searches the video input by the video input unit for a structural unit that matches the search granularity and that has an event index that matches the key index; Enter the structural unit found in the Determining means for determining whether or not the generated state transition pattern and the occurrence pattern of the event index in the structural unit match; and extracting a video scene from the video based on the structural unit determined to match by the determining means. And a search result output means for outputting the result as a search result. 12. A first determining unit that determines whether all of the event indexes included in the state transition pattern corresponding to the query term exist in the structural unit searched by the searching unit. Determining means for determining whether the state transition pattern corresponding to the query term matches the occurrence pattern of the event index in the structural unit with respect to the structural units determined to be all present by the first determining means 12. The video search processing device according to claim 11, comprising: a second determination unit that performs the determination. 13. When structuring a video, a video index including at least a structure index for dividing the video into a semantic group and an event index for specifying the content and location of an event that has occurred in the video. In the video index adding method to be added, a term in which a meaning is established in advance by a plurality of events occurring consecutively and a state transition in which the term is defined using an input sequence of a plurality of event indexes expressing the term A state transition table is set in which a pattern and a search granularity specified by associating the structural unit of the video defined by the predetermined structural index with the term are associated with each term, and the video index is assigned to the video. At the time of assigning the structure unit, the structure unit specified by the structure index is referred to with reference to the state transition table. In each case, the target structural unit matches the search granularity, and the assignment order of the event indexes assigned in the target structural unit matches the input sequence of the plurality of event indexes of the state transition pattern. A video index characterized by searching for a term, determining that the meaning of the matched term has occurred when a matching term exists, and adding an event index or attribute information indicating the establishment of the corresponding term. Assignment method. 14. When structuring a video, a video index including at least a structure index for dividing the video into a semantic group and an event index for specifying the content and location of an event that has occurred in the video. In the video index adding method to be added, a term in which a meaning is established in advance by a plurality of events occurring consecutively and a state transition in which the term is defined using an input sequence of a plurality of event indexes expressing the term A pattern and a search granularity in which a structural unit of a video defined by a predetermined structural index is specified in association with the term, and at least one event index among event indexes existing in the state transition pattern is specified. A state transition table in which a central index is associated with each term is set in advance. When assigning a video index to the video, by referring to the state transition table, if an event index that matches the central index is assigned, each of the structural units specified by the structural index is targeted. Structural unit and the search granularity match, and search for a term in which the input sequence of the event index assigned in the target structural unit and the input sequence of the plurality of event indexes of the state transition pattern match, A video index assigning method, wherein when a matching term exists, it is determined that the meaning of the matched term has occurred, and an event index or attribute information indicating the establishment of the corresponding term is assigned. 15. A computer-readable recording medium on which a program for causing a computer to execute the video index assigning method according to claim 13 or 14 is recorded. 16. A video of a section divided by the structure index, to which at least a structure index for dividing the video into a semantic group and an event index for specifying the content and location of an event that has occurred in the video are provided. A video content description generating method for generating a description that describes a video content using a scene as a structural unit of a video and a video structured using a plurality of hierarchically structured units, A video for generating the explanatory note using a state transition table in which information used for generating a description is set, and a character string or attribute information that can be converted into a character string that is previously assigned to the structure index and the event index A method of generating a description of a content, wherein the state transition table includes, for each term that can be expressed by a combination of a plurality of events, ,
Based on a generation granularity in which an appropriate structural unit is set as a video unit when generating the explanatory sentence, and based on an occurrence pattern of an event expressing the term, a plurality of event indexes continuously occurring in the generation granularity A state transition pattern that defines the term as an input sequence of, and, for each of the state transition patterns, at least one of an event index existing in each state transition pattern.
And a key index set by selecting one of the following: for each of the state transition patterns, a character string definition information in which a syntax element used for generating the description and a character string input source used as a syntax element of the syntax are set. When generating the description of the video content, the state transition table is searched for a generation granularity that matches the structural unit for which the description is to be generated. It is determined whether or not an event index matching the key index of the state transition pattern corresponding to the generation granularity exists in the target structural unit. If the matching event index exists, the corresponding state transition is determined. It is determined whether the pattern and the occurrence pattern of the event index in the target structural unit match, and if the occurrence pattern matches, Determining that the term defined in the corresponding state transition pattern is established, and generating a description of the video scene of the target structural unit using character string definition information of the state transition pattern of the established term. The description generation method of the video content to be described. 17. When a state transition pattern defining a specific term is established, an index representing the term can be assigned to the structural unit as an abstract index. In each of the state transition patterns, an abstract index representing a corresponding term is set as one of the key indexes, and an event index that matches the key index of the state transition pattern corresponding to the corresponding generation granularity is set as the target index. When it is determined whether or not the structural unit exists in the target structural unit, the abstract index is preferentially used as the key index, and it is determined whether or not the abstract index corresponding to the target structural unit is given. If an abstract index has been set, it is defined by the corresponding state transition pattern. 17. The description according to claim 16, wherein it is determined that the established term is established, and a description of the video scene of the target structural unit is generated using the character string definition information of the state transition pattern of the established term. How to generate a description of the video content 18. The character string definition information according to claim 16, wherein attribute information of the structure index or the event index is set as an input source of a character string used as the syntax element. A method for generating a description of the video content described. 19. The character string definition information may include “what time (Wh
en), where (Where), why (Why), who (Who), what (What), how (How)
19. The method according to claim 16, wherein a syntax element based on 5W1H of "Nita" is set. 20. When there are a plurality of state transition patterns corresponding to the corresponding generation granularity, it is determined whether or not a term is established for each state transition pattern. 20. The method according to claim 19, wherein a syntactic element of the character string definition information set in the state transition pattern is combined to generate a description of the video scene of the target structural unit. . 21. When generating a description of a video scene of a target structural unit by combining syntax elements of character string definition information set in a state transition pattern of each term,
When there are duplicate syntax elements in the syntax elements of W1H,
21. The method according to claim 20, wherein a syntax element referring to an event index occurring later in time is prioritized. 22. When generating a description of a video scene of the target structural unit by combining syntax elements of character string definition information set in a state transition pattern of each term,
When there are duplicate syntax elements in the syntax elements of W1H,
22. The state transition pattern of each term is compared, and a syntax element of the state transition pattern defined using more event indexes is prioritized.
The method of generating the description of the video content described in. 23. When generating a description of a video scene of the target structural unit by combining syntax elements of character string definition information set in a state transition pattern of each term,
When there are duplicate syntax elements in the syntax elements of W1H,
The method according to any one of claims 19 to 22, wherein syntax elements are arranged in parallel as needed. 24. A computer-readable recording medium having recorded thereon a program for causing a computer to execute the method for generating a description of a video content according to any one of claims 16 to 23.