JP5437928B2 - METADATA ADDING DEVICE, VIDEO SEARCH DEVICE, METHOD, AND PROGRAM - Google Patents

Description

  The present invention provides a metadata adding apparatus that adds verbalized metadata to video using a sensor and sensor network technology, and a search key that expresses an event to be searched with a keyword or sensor data as an input. The present invention relates to a video search apparatus for searching for a video including the video.

  Various schemes have been devised so far for searching video. One of the typical methods is a method of searching for the same or similar video from another video set (accumulated signal) using the video section itself as a search key (target signal) (see Non-Patent Document 1). In the case of such a method, in order to realize high-speed processing, it is often the case that the feature is extracted from the target signal and the accumulated signal and the search is performed by comparing the feature amount, instead of directly comparing the videos. . In addition, by extracting features well, it is possible to have a certain degree of resistance against distortion and noise.

  Another possible method is a method of searching for a video by interpreting the content of the content using an image processing technique or the like. A simple method is to discriminate a scene's excitement from audio information. A method of searching by adding metadata for each scene to the content by manually checking the contents of the content can be considered.

Takayuki Kurosumi, Hidenao Nagano, Kunio Kanno, “Search for Matched Video Using Video Fragments Recorded in Real Environment”, IEICE Transactions D, Vol. J90-D, No. 8, pp. 2223-2231, 2007

In the method disclosed in Non-Patent Document 1 for searching for a video itself as a search key (target signal), the video is superficially compared and the content is not interpreted. For this reason, there has been a problem that it is not possible to perform a video search that takes into account what is happening in the video.
In addition, the method using audio has a problem that the video can be discriminated only in a specific scene (for example, sports broadcast). The method of manually assigning metadata is not realistic because it takes too much work.

The present invention has been made to solve the above problems, and provides a metadata adding apparatus, method, and program capable of giving metadata to video in order to realize video search based on the motion of a subject. The purpose is to do.
It is another object of the present invention to provide a video search apparatus, method, and program capable of realizing video search based on the motion of a subject.

  The metadata providing apparatus of the present invention includes a video acquisition unit that acquires a video of a subject photographed by a camera, a sensor data acquisition unit that acquires sensor data measured by a sensor attached to the subject at the time of the photographing, Corresponds to the event descriptor dictionary that stores in advance the correspondence between the event descriptor representing the event related to the event and the physical quantity representation of the event expressed by an expression using the sensor data, and the sensor data acquired by the sensor data acquisition means When an event having a physical quantity expression is registered in the event descriptor dictionary, the event descriptor representing the event is used as metadata, and metadata is added to the video acquired by the video acquisition unit. Means.

  Further, the video search apparatus of the present invention receives as input a storage means for storing in advance video provided with metadata representing an event that has occurred in a subject, and a search key that expresses the event to be searched with keywords or sensor data. A key type determining means for determining the type of the search key, an event descriptor dictionary for storing in advance a correspondence relationship between an event descriptor representing an event related to the subject and a physical quantity expression of the event expressed by an expression using sensor data; When the search key is a search key representing an event with sensor data, and an event having a physical quantity representation corresponding to the sensor data is registered in the event descriptor dictionary, an event description representing the event Metadata extraction means for extracting a child as metadata, and the search key is a search key expressing an event as a keyword, and the search key is registered in the event descriptor dictionary If the keyword is a keyword other than the event descriptor, the event descriptor corresponding to the search key is input metadata, and the search key is an event descriptor registered in the event descriptor dictionary. In this case, the event descriptor is input metadata, and when the search key is a search key representing an event by sensor data, the metadata extracted by the metadata extraction unit is input metadata and input. The image processing apparatus is characterized by further comprising search means for searching for a video to which metadata matching the metadata is given from the video stored in the storage means.

The present invention is also a metadata providing method for adding metadata to video in a computer having a CPU and a storage device, the video acquisition step for acquiring video of a subject shot by a camera, and at the time of shooting A correspondence relationship between a sensor data acquisition step of acquiring sensor data measured by a sensor attached to a subject and a physical quantity expression of the event expressed by an expression using an event descriptor and sensor data related to the subject in advance When an event having a physical quantity expression corresponding to the sensor data acquired in the sensor data acquisition step is registered in the event descriptor dictionary to be stored, the event descriptor representing the event is metadata, and the video acquisition step and metadata imparting step of imparting the metadata obtained video in, in the storage device It is characterized in that to be executed by the CPU in accordance憶program.

The present invention is also a video search method for searching for video in a computer having a CPU and a storage device, wherein a search key expressing an event to be searched as a keyword or sensor data is input, and the type of the search key is set. A key type determination step for determining, a search key in which the search key represents an event in sensor data, and an event descriptor representing an event related to a subject and a physical quantity representation of the event expressed in an expression using sensor data Metadata extraction that extracts an event descriptor representing this event as metadata when an event having a physical quantity expression corresponding to the sensor data of the search key is registered in an event descriptor dictionary that stores correspondence in advance And an event in which the search key is a search key expressing an event as a keyword, and the search key is registered in the event descriptor dictionary If it is a keyword other than a predicate, an event descriptor corresponding to the search key is input metadata, and if the search key is an event descriptor registered in the event descriptor dictionary, If the event descriptor is input metadata, and the search key is a search key expressing the event as sensor data, the metadata extracted in the metadata extraction step is input metadata, and matches the input metadata. A search step for searching for a video with metadata to be searched from videos with metadata stored in a storage unit according to a program stored in the storage device. It is.
The program of the present invention is characterized by causing a computer to execute each step of the metadata providing method or the video search method.

  According to the present invention, the physical quantity expression corresponding to the sensor data acquired by the sensor data acquisition unit is acquired by acquiring sensor data measured by the sensor attached to the subject at the same time as acquiring the video of the subject photographed by the camera. When an event has been registered in the event descriptor dictionary, the event descriptor representing this event is used as metadata, and metadata is added to the video acquired by the video acquisition means, so that the addition of metadata is manually performed. It can be realized automatically without spending time. As a result, according to the present invention, it is possible to realize video search based on the motion of the subject shown in the video.

  Further, in the present invention, when the search key is a search key expressing an event as a keyword, and the search key is a keyword other than the event descriptor registered in the event descriptor dictionary, the search key is If the corresponding event descriptor is the input metadata and the search key is an event descriptor registered in the event descriptor dictionary, the event descriptor is the input metadata, and the search key is the sensor data. If an event having a physical quantity representation corresponding to sensor data is registered in the event descriptor dictionary, the event descriptor representing the event is extracted as input metadata, and the input metadata By searching for videos with metadata that matches, from the video stored in the storage means, video search based on subject movement that could not be achieved with conventional methods was realized. Rukoto can.

It is a figure which shows the relationship between sensor data and an image | video. It is a block diagram which shows the structure of the metadata provision apparatus which concerns on embodiment of this invention. It is a flowchart which shows operation | movement of the metadata provision apparatus which concerns on embodiment of this invention. It is explanatory drawing which shows the specific example of the event descriptor dictionary of the metadata provision apparatus which concerns on embodiment of this invention. It is explanatory drawing which shows the specific example of the event descriptor dictionary of the metadata provision apparatus which concerns on embodiment of this invention. It is a block diagram which shows the structure of the video search apparatus which concerns on embodiment of this invention. It is a flowchart which shows operation | movement of the image | video search device which concerns on embodiment of this invention. It is a figure explaining the video search process which concerns on embodiment of this invention.

  The present invention uses a sensor and sensor network technology, a metadata providing device that provides verbalized metadata to content, and a content provided with metadata generated by the metadata providing device, Provided is a video search device for searching a section in an accumulated signal including an event using language information representing the desired event or an image including the desired event as a search key. Hereinafter, a certain time unit is referred to as a section, and the addition of metadata and video search are performed in this section unit.

  When shooting a video, a sensor is attached to the object or person shown in the video, and sensor data is acquired at the same time as the video. The associated items are stored in the storage unit.

  In the present invention, the mere time-series waveform itself acquired with sensor data is not added to the video as metadata, but the event indicated by the sensor data is interpreted as metadata by interpreting the contents of the sensor data as metadata. It is characterized in that a video is retrieved using metadata that is given to the video and verbalized on the video.

For example, if it is determined that the gravitational acceleration that is always applied is not applied in the normal state, it can be determined that the object is falling. Moreover, even if the operation is difficult to express in words, the search can be performed by expressing it in the operation.
FIG. 1 shows the relationship between sensor data and video. Reference numerals 100 to 103 in FIG. 1 indicate a series of images obtained by photographing a person dropping an object to which a sensor node described later is attached. Reference numeral 104 in FIG. 1 indicates sensor data output from the sensor node with respect to this series of movements. As a result of analyzing the sensor data, an event that occurred in the section where the video was shot is determined, and a word (for example, “fall-vertically”) representing this event is obtained as metadata.

[Metadata adding device]
Next, a metadata adding apparatus that extracts metadata obtained by verbalizing sensor data from video and sensor data (waveform) and applies the metadata to the video will be described in detail. FIG. 2 is a block diagram showing the configuration of the metadata providing apparatus according to the embodiment of the present invention. The metadata providing apparatus 1 includes a video acquisition unit 10, an accumulation signal storage unit 11, a sensor data acquisition unit 12, an accumulation sensor data storage unit 13, an event section detection unit 14, a data correction unit 15, and metadata. It has an assigning unit 16, an event descriptor dictionary 17, and an accumulated signal storage unit 18 with metadata.

  FIG. 3 is a flowchart showing the operation of the metadata providing apparatus 1. First, the video acquisition unit 10 acquires a video to be searched that is captured by the camera 2, and stores this video in the storage signal storage unit 11 as a storage signal (step S100 in FIG. 3). At this time, the sensor node 3 is attached to an object or person as a subject. The sensor node 3 includes sensors such as an acceleration sensor, an illuminance sensor, and an orientation sensor, a CPU, a memory, and a communication function unit that performs data communication wirelessly. The CPU of the sensor node 3 transmits sensor data measured by the sensor to the sensor network 4 through the communication function unit.

  The sensor data acquisition unit 12 acquires sensor data indicating the state of the subject from the sensor node 3 through the sensor network 4 simultaneously with the acquisition of the video by the video acquisition unit 10 and stores the sensor data in the storage sensor data storage unit 13. (FIG. 3, step S101).

  Considering an acceleration sensor or the like attached to a subject, the sensor reacts only when the subject moves, so the sensor is often not operating in many time zones. For this reason, it is lossy to determine what kind of event has occurred in all sections in which video is being shot. Therefore, in the metadata providing apparatus 1 of the present embodiment, the event section detection unit 14 analyzes the sensor data stored in the accumulated sensor data storage unit 13 and detects a section in which an event occurs as an event section ( FIG. 3 step S102).

  The event section detection unit 14 may determine that an event has occurred, for example, when the physical quantity value indicated by the sensor data is equal to or greater than a predetermined threshold value. For example, in the case of an acceleration sensor, it may be determined that an event has occurred when the acceleration is equal to or greater than a threshold value. Alternatively, it may be determined that an event has occurred when a change in physical quantity per unit time is equal to or greater than a predetermined threshold.

  The acceleration sensor has information on three axes (X axis, Y axis, and Z axis), but the direction in which the sensor node 3 is attached to the subject is unknown. Therefore, it is necessary to correct which direction the acceleration sensor is actually facing by using gravitational acceleration. Also, since sensor data output from the sensor often includes noise, noise is removed as much as possible. Specifically, the data correction unit 15 removes noise included in the triaxial acceleration data by, for example, filtering. Further, the data correction unit 15 detects the gravitational acceleration based on the triaxial acceleration data after noise removal, detects in which direction the acceleration sensor is tilted, and detects the direction of the detected acceleration sensor and the noise-removed direction. From the triaxial acceleration data, vertical acceleration data and acceleration data in a horizontal plane perpendicular to the vertical acceleration data are calculated (step S103 in FIG. 3). In this way, data can be corrected. In addition, what is necessary is just to perform noise removal about the sensor data of sensors other than an acceleration sensor.

  Next, the metadata adding unit 16 determines what event has actually occurred for the event section detected by the event section detecting unit 14 (step S104 in FIG. 3). A method for determining an event is disclosed in, for example, Japanese Patent Application Laid-Open No. 2007-304692. Hereinafter, a method for determining an event will be described based on the technique disclosed in Japanese Patent Application Laid-Open No. 2007-304692.

The event descriptor dictionary 17 includes a predicate including a verb indicating an event related to a person or an object, an event descriptor including a label indicating a group to which the predicate belongs, and a physical quantity expression of the event expressed by an expression using sensor data. Is a database that stores them in advance in association with each other.
The event descriptor dictionary 17 will be described in detail with reference to FIG. FIG. 4 is an explanatory diagram showing a specific example of the event descriptor dictionary 17.

The event descriptor dictionary 17 is generated in advance as follows. First, using the thesaurus system, the basic verbs related to “movement” and the basic verbs related to “scalar physical quantity” are used as seeds to follow the links related to the thesaurus, and to deal with physical phenomena. Collect as many predicates as possible.
Here, the verb “move, reach, pass,” related to “exercise” is utilized using the English electronic thesaurus system WordNet (see, for example, the document “C. Fellbaut, WordNet: an electronic lexical database, MIT Press, 1998”). `` exit, touch, enter '' and verbs `` rise, drop, increase, keep, remain '' related to `` scalar physical quantity '' as seeds, follow links in the thesaurus one after another and associate with physical phenomena Collect all possible predicates.

  At this time, a plurality of predicates having exactly the same meaning are labeled as one group by using a clause expressing the meaning described in WordNet. For example, the verb “shift, dislodge, reposition” forms one group with a common meaning “change direction” and labels the group “change-direction”. In this way, each labeled predicate group is created.

  Next, for each of these predicate groups, a distinction is made between transitive verbs and intransitive verbs, and auxiliary symbols are introduced into the labels. For example, for the label “change-location”, in the case of the group of the transitive verb “move, travel, locomote, go”, two arguments are added like “change-location (a, o)”, and the argument “a” in the transitive verb Represents the subject and allows the argument o to represent the object. On the other hand, in the case of an intransitive verb, only “change-location (o)” and the subject can be clearly indicated, or it is omitted and expressed as “change-location”.

  Furthermore, there is a word that includes information such as time and place as its meaning, and an expression that adds an argument specifying the information to a label for the word group is introduced. In other words, the intransitive word group “read, arrive at, gain” labeled “reach-destination” has the meaning of “location” or “object” to be reached. reach-destination (b: object) ", or more precisely" (reach-destination (o)) (b: object) ". A label having this argument is called an event descriptor, and an event descriptor dictionary 17 is created by arranging the event descriptors in alphabetical order and digitizing them. A group of words for the label is associated with the event descriptor.

  In the event descriptor dictionary 17, as shown in FIG. 4, an event descriptor 17B and a physical quantity expression 17C are associated with a predicate 17A indicating an event related to a target person or object. The physical quantity expression 17C includes a time constraint expression unit 17E including a time constraint expression indicating each section provided by dividing the event in time, and the event in the section such as three-dimensional coordinates, speed, temperature, humidity, and acceleration. It consists of a pair with a physical formula part 17D consisting of physical formulas (P1, P2, P3) expressed by an arbitrary physical quantity.

For example, in the example of FIG. 4A, for each predicate 17A “move, travel, locomote”, an event descriptor 17B “change-location (a, o)”;
P1: do not care (tO ≦ t <t1),
P2: | v |> 0 (t1 ≦ t ≦ t2),
P3: do not care (t2 <t ≦ t3).
Is associated with the physical quantity expression 17C. Here, in the physical quantity expression 17C, t0, t1, t2, and t3 in the time constraint expression unit 17E are parameters that represent real time values, the time t1 corresponds to the occurrence time of the event, and the time t2 corresponds to the time. Corresponds to the end time of the event.

Therefore, in the physical formula part 17D, the expression P1 indicates the state of the object in the section immediately before the event occurs, the expression P2 indicates the state of the object in the section in which the event occurs, and the expression P3 indicates the object in the section immediately after the event occurs. Indicates the state of the object.
In the example of FIG. 4A, as a condition in the event occurrence period from time t1 to time t2, a condition that the magnitude | v | of the velocity vector v of the object o is larger than 0, that is, the object o moves. This is defined by the physical formula part 17D.

In the example of FIG. 4B, an event descriptor “(go-from-region-to-region (o)) (r1: region, r2: region)” is associated with the predicate 17A “move”. 17B,
D (x, r1) = 0 (t0 ≦ t <t1),
| v |> 0 & D (x, r2)> 0 (t1 ≦ t ≦ t2),
D (x, r2) = 0 (t2 <t ≦ t3).
Is associated with the physical quantity expression 17C. Here, in the physical quantity expression 17C, x is the three-dimensional coordinate of the object o, D (r1, r2) represents the Euclidean distance between r1 and r2, and v represents the velocity vector of the object o. In this case, the object o is in the region region1 in the section before the time t1, starts moving from the time t1, reaches the region region2 at the time t2, and expresses that it is in the region 2 in the section after the time t2. . & Indicates a logical product.

Further, in the example of FIG. 4C, for the predicate 17A “drop”, an event descriptor 17B “fall-vertically (o)”,
do not care (tO ≦ t <t1),
(v / | v |) ・ (g / | g |) = 1 & a = 9 (t1 ≦ t ≦ t2),
do not care (t2 <t ≦ t3).
Is associated with the physical quantity expression 17C. Here, in the physical quantity expression 17C, g represents a gravity constant vector, “·” represents an inner product, and a represents the acceleration of the object o. In this case, the object o indicates that the object is moving vertically downward at an acceleration g in a section from time t1 to time t2. In this way, a physical quantity representation 17C for each event descriptor 17B representing each predicate 17A is formulated and stored in the event descriptor dictionary 17. FIG. 5 is an explanatory diagram showing a specific example of event descriptors and physical quantity expressions.

  The metadata adding unit 16 refers to the event descriptor dictionary 17 as described above, and an event having a physical quantity expression corresponding to the sensor data in the event section detected by the event section detecting unit 14 is registered in the event descriptor dictionary 17. If so, it is determined that the event has occurred within the event section (step S104 in FIG. 3).

For example, as shown in FIG. 5, for the predicate “drop”, an event descriptor “fall-vertically (o)”
do not care (tO ≦ t <t1); 0 <t1-t0,
(v / | v |) ・ (g / | g |) = 1 & a = g (t1 ≦ t ≦ t2); 200msec ≦ t2-t1 <800msec,
do not care (t2 <t ≦ t3); 0 <t3-t2.
Is associated with the physical quantity expression and registered in the event descriptor dictionary 17. Here, when the sensor data within the event section corrected by the data correction unit 15 includes sensor data indicating that the sensor data has moved downward vertically at an acceleration g in an elapsed time from 200 msec to less than 800 msec, Since it corresponds to the physical quantity expression shown in FIG. 5, the metadata adding unit 16 determines that an event expressed by an event descriptor “fall-vertically (o)”, that is, an event expressed by the word “fall” is an event. It is determined that it occurred in the section.

  In order to determine whether or not the condition “(v / | v |) · (g / | g |) = 1 & a = g” is satisfied, the velocity vector v is necessary. In order to calculate the velocity vector v, it is possible to calculate the velocity vector v at each time by numerically integrating the acceleration with the initial velocity set to “0 (zero)”.

Next, when the event that has occurred in the event section can be determined, the metadata adding unit 16 uses the word representing the event (for example, an event descriptor of “fall-vertically”) as metadata, and uses the metadata. The accumulated signal to which the metadata is added and accumulated in the accumulated signal in the time interval corresponding to the event interval is accumulated in the accumulated signal storage unit with metadata 18 (step S105 in FIG. 4).
Thus, the process of the metadata providing apparatus 1 ends.

[Video search device]
Next, a video search device that performs video search using the metadata-added accumulation signal generated by the metadata adding device 1 will be described. FIG. 6 is a block diagram showing the configuration of the video search apparatus according to the embodiment of the present invention. The video search device 5 includes a storage signal storage unit 50 with metadata, a video acquisition unit 51, a sensor data acquisition unit 52, a search key input unit 53, a key type determination unit 54, a metadata extraction unit 55, It has an event descriptor dictionary 56, an accumulated signal search unit 57, and an output unit 58.

FIG. 7 is a flowchart showing the operation of the video search device 5. The metadata-added accumulation signal storage unit 50 stores the metadata-added accumulation signal generated by the metadata providing apparatus 1.
A search key is input to the search key input unit 53 by the user of the video search device 5 (step S200 in FIG. 7). As the search key, (1) a motion that the person or object wants to search is expressed in words, and (2) a motion that the person or object wants to search is expressed by a set of video and sensor data.

  When using a keyword that expresses the movement of a person or object as a search key in words, for example, by searching for a keyword such as “drop” or “fall-vertically” It is possible to search for an interval in the accumulated signal that is operating.

  In addition, when using a search key that expresses a motion to search for a person or object as a set of video and sensor data, the user of the video search device 5 attaches a sensor node in front of the camera, for example. The operation to be searched is actually performed, and the acquired video and sensor data are used as search keys. In this case, at the same time that the video acquisition unit 51 acquires the video captured by the camera 2, the sensor data acquisition unit 52 acquires the sensor data from the sensor node 3 through the sensor network 4. Thereby, it is possible to search for a section in the accumulated signal including an operation similar to the operation performed by the user. In addition, you may utilize the accumulation signal and sensor data of one area memorize | stored in the accumulation signal memory | storage part 50 with metadata as a search key. In either of the cases (1) and (2), it is possible to search for a video including a motion similar to the motion specified by the search key, rather than searching for the exact same video. It is a feature.

  Subsequently, the key type determination unit 54 indicates that the type of the search key input from the search key input unit 53 expresses the above-described (1) movement of the person or object to be searched in words, (2) person or object It is discriminated whether the motion to be searched is represented by a set of video and sensor data (step S201 in FIG. 7). If the input search key is a keyword (for example, “drop”) that expresses a motion to search for a person or an object in words, the key type determination unit 54 uses this search key as a corresponding event descriptor (for example, “fall”). -vertically ") and then transmitted as metadata to the accumulated signal search unit 56 (step S202 in FIG. 7). If the search key is an event descriptor, no conversion is necessary, and the search key may be sent as it is to the stored signal search unit 56 as metadata. Also, the key type determination unit 54 transmits the search key to the metadata extraction unit 55 if the input search key represents a motion to search for a person or an object as a set of video and sensor data. (Step S203 in FIG. 7).

  The metadata extracting unit 55 performs the same processing as the data correcting unit 15 and the metadata adding unit 16 in the metadata adding apparatus 1 using the sensor data section that is the search key input from the key type determining unit 54 as an event section. Then, an event corresponding to the sensor data as the search key is determined, and a word representing this event is transmitted as metadata to the accumulated signal search unit 56 (step S204 in FIG. 7). Specifically, the metadata extraction unit 55 performs noise removal processing and correction processing similar to the data correction unit 15 on the sensor data, and then refers to the event descriptor dictionary 56 and corresponds to the corrected sensor data. When an event having a physical quantity expression to be registered is registered in the event descriptor dictionary 56, it is determined that the event has occurred within the event section, and a word representing this event is used as metadata.

The accumulated signal search unit 56 uses the metadata input from the key type determination unit 54 or the metadata extraction unit 55 to search the accumulated signal stored in the accumulated signal storage unit 50 with metadata (step S205 in FIG. 7). ). Specifically, the accumulation signal search unit 56 searches the accumulation signal storage unit with metadata 50 for an accumulation signal in a time interval to which metadata that matches the input metadata is assigned, and searches for the retrieved time interval. The accumulated signal is output to the output unit 58 as a search result image. The output unit 58 displays the video of the search result.
Thus, the processing of the video search device 5 ends.

  An image of the above search processing is shown in FIG. Reference numeral 800 in FIG. 8 indicates an image obtained by photographing a person dropping an object to which a sensor node is attached. Reference numeral 801 in FIG. 8 indicates sensor data output from the sensor node 3 with respect to this movement. As a result of analyzing the sensor data by the metadata adding device 1, an event that occurred in the section where the video was shot is determined, metadata “fall-vertically” representing this event is obtained, and this metadata is added to the video The The video to which the metadata is added is stored in the accumulated signal storage unit 50 with metadata of the video search device 5 through the metadata adding device 1.

  Next, reference numeral 802 in FIG. 8 shows an image of a situation where a user wearing a sensor node lowers his / her hand raised to input a search key. Reference numeral 803 in FIG. 8 indicates sensor data output from the sensor node 3 with respect to this movement. As a result of analyzing the sensor data by the video search device 5, an event that occurred in the section where the video was shot is determined, and metadata “fall-vertically” representing this event is obtained. The accumulated signal search unit 56 searches for the accumulated signal stored in the accumulated signal storage unit 50 with metadata using the metadata.

  As described above, in the present embodiment, metadata can be automatically assigned to a video, and by adding metadata to a video, a video based on the motion of a subject that could not be realized by a conventional method. Search can be realized.

Each of the metadata providing apparatus 1 and the video search apparatus 5 according to the present embodiment is realized by a computer having a CPU, a storage device, and an external interface, and a program for controlling these hardware resources. Can do. In such a computer, a program for realizing the present invention is provided in a state of being recorded on a recording medium such as a flexible disk, a CD-ROM, a DVD-ROM, or a memory card. The CPU of each device writes the program read from the recording medium to the storage device, and executes the processing described in this embodiment according to the program.
In the present embodiment, the metadata providing device 1 and the video search device 5 are separate devices, but the metadata providing device 1 and the video search device 5 may be integrated.

  The present invention provides a technique for adding verbalized metadata to video using the technology of sensors and sensor networks, and a search key that expresses an event to be searched with a keyword or sensor data as an input. It can be applied to search technology.

  DESCRIPTION OF SYMBOLS 1 ... Metadata provision apparatus, 2 ... Camera, 3 ... Sensor node, 4 ... Sensor network, 5 ... Image | video search apparatus, 10 ... Image | video acquisition part, 11 ... Accumulation signal storage part, 12 ... Sensor data acquisition part, 13 ... Accumulation Sensor data storage unit, 14 ... event section detection unit, 15 ... data correction unit, 16 ... metadata addition unit, 17 ... event descriptor dictionary, 18 ... accumulation signal storage unit with metadata, 50 ... accumulation signal storage with metadata , 51 ... Video acquisition unit, 52 ... Sensor data acquisition unit, 53 ... Search key input unit, 54 ... Key type determination unit, 55 ... Metadata extraction unit, 56 ... Event descriptor dictionary, 57 ... Accumulated signal search unit, 58: Output section.

Claims (5)

Video acquisition means for acquiring video of the subject photographed by the camera;
Sensor data acquisition means for acquiring sensor data measured by a sensor attached to the subject at the time of shooting;
An event descriptor dictionary that stores in advance a correspondence relationship between an event descriptor representing an event related to a subject and a physical quantity representation of the event expressed by an expression using sensor data;
When an event having a physical quantity expression corresponding to the sensor data acquired by the sensor data acquisition means is registered in the event descriptor dictionary, the event descriptor representing this event is set as metadata, and the video acquisition means A metadata providing apparatus, comprising: a metadata adding unit that adds the metadata to the acquired video. Storage means for storing in advance a video to which metadata representing an event that has occurred in a subject is attached;
A key type determination unit that inputs a search key expressing a phenomenon to be searched by a keyword or sensor data and determines the type of the search key;
An event descriptor dictionary that stores in advance a correspondence relationship between an event descriptor representing an event related to a subject and a physical quantity representation of the event expressed by an expression using sensor data;
When the search key is a search key expressing an event with sensor data, and an event having a physical quantity expression corresponding to the sensor data is registered in the event descriptor dictionary, an event descriptor representing this event Metadata extraction means for extracting as metadata,
When the search key is a search key expressing an event as a keyword, and the search key is a keyword other than the event descriptor registered in the event descriptor dictionary, the event corresponding to the search key If the descriptor is input metadata and the search key is an event descriptor registered in the event descriptor dictionary, the event descriptor is input metadata, and the search key uses the event as sensor data. In the case of the search key expressed in the above, the metadata extracted by the metadata extraction unit is used as input metadata, and a video provided with metadata that matches the input metadata is stored in the video stored in the storage unit. A video search apparatus comprising: search means for searching from among the above. A metadata providing method for adding metadata to video in a computer including a CPU and a storage device,
An image acquisition step for acquiring an image of a subject photographed by the camera;
A sensor data acquisition step of acquiring sensor data measured by a sensor attached to the subject at the time of shooting;
An event descriptor dictionary that stores in advance a correspondence relationship between an event descriptor representing an event related to a subject and a physical quantity expression of the event expressed by an expression using sensor data, and corresponds to the sensor data acquired in the sensor data acquisition step. When an event having a physical quantity expression is registered, an event descriptor representing the event is used as metadata, and a metadata adding step for adding the metadata to the video acquired in the video acquiring step includes the storage device. A method for assigning metadata, wherein the CPU executes the program according to a program stored in the program . A video search method for searching video in a computer having a CPU and a storage device,
A key type determination step for determining a type of the search key, using a search key expressing the event to be searched as a keyword or sensor data as an input,
An event description in which the search key is a search key that represents an event in sensor data, and stores in advance a correspondence relationship between an event descriptor that represents an event related to a subject and a physical quantity representation of the event represented by an expression using sensor data A metadata extraction step of extracting an event descriptor representing this event as metadata when an event having a physical quantity expression corresponding to the sensor data of the search key is registered in the child dictionary;
When the search key is a search key expressing an event as a keyword, and the search key is a keyword other than the event descriptor registered in the event descriptor dictionary, the event corresponding to the search key If the descriptor is input metadata and the search key is an event descriptor registered in the event descriptor dictionary, the event descriptor is input metadata, and the search key uses the event as sensor data. In the case of the search key expressed in (4), the metadata extracted in the metadata extraction step is used as input metadata, and the video to which metadata matching the input metadata is assigned is stored in the storage means. A video search method, comprising: causing the CPU to execute a search step for searching from an attached video according to a program stored in the storage device .   A program for causing a computer to execute each step according to claim 3 or 4.