JP2010074437A - Method, device and program for adding annotation, and computer readable recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform display for easily selecting a video image photographing a scene, addition of annotation to the scene being desired. <P>SOLUTION: When a user of a terminal who views a photographing target adds the annotation to a plurality of video images, a photographing instrument icon is displayed to suit a user coordinate system so as to obtain a layout so that a positional relationship of a center of gravity in the photographing target to be photographed by a plurality of photographing instruments viewed from the user who carries out the annotation is analogous to a layout of icons displayed in the user's terminal to indicate the plurality of photographing instruments. When this takes place, the annotation date and time are converted into a video image time (an integration time when a video image start point is 0) to register annotation information, a video image section is defined to the photographed video image, a digest video image is produced handling the video image section including the annotation as a target, and the annotation information is synchronously displayed. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an annotation assigning method and apparatus, a program, and a computer-readable recording medium. In particular, when a plurality of shooting targets are shot synchronously, a video target can be easily specified in a live manner, and the target video is set. The present invention relates to an annotation assigning method and apparatus, a program, and a computer-readable recording medium for constructing a digest for instantly viewing an important part based on an annotation result by assigning an annotation.

  Conventionally, there is a method for annotating video time and video time interval after shooting (see, for example, Patent Document 1). The method of annotating a video after shooting allows both a user who is not in the shooting hall and a user who is in the shooting hall to participate in the annotation, and is effective as asynchronous communication. On the other hand, there is a method in which users participating in a shooting venue annotate lively while feeling a sense of reality (see, for example, Non-Patent Document 1). By immediately annotating what the user felt at the shooting venue and the important points at the shooting venue, the user can leave the annotation with the sensitivity of the place without forgetting, Since the annotation annotation time period is shared with the user, the added annotation is easy to browse and the response to the browsed annotation is likely to be expected. Conceivable. In addition, even when viewed later, the described user can recognize the atmosphere of the spot, and it is meaningful for other users to browse the opinions and reference information felt on the spot.

Further, when annotating a plurality of videos, it is necessary to select a video to be annotated from the plurality of videos. As a method to support this processing, when the moving object is detected on the video under the assumption that there is a change in the video that is normally annotated, and the moving object moves across the shooting range of multiple cameras In addition, the camera icon on the map screen is connected and displayed with an arrow indicating the path of movement of the moving object, thereby supporting the selection of the camera that captures the annotation target (see, for example, Patent Document 2). .
JP 2003-283981 A JP 2006-67139 A Hitoshi Miyata (2002) Improvement of class communication in multi-person lectures using mobile phone compatible comment card system, Educational Information Research, Vol. 18, No. 3, pp. 11-19.

  However, in the conventional method of Patent Document 1, in the case where annotation is given to a plurality of video objects, the annotation is performed after shooting, so the user at the shooting venue forgets what he / she felt during shooting. In addition, the time for annotating is required separately from the time of shooting, and the annotation is not performed sufficiently because the timing of annotating with other users is not synchronized.

  In addition, even if annotation can be performed during a lecture as in the method of Non-Patent Document 1 above, if it is not known which target and which captured video the annotation was performed, The problem is that the content is not fully understood or misinterpreted.

  Also, unlike lecture videos, field sports such as soccer and baseball have many variations that can be taken, and the number of shots increases depending on the situation. Is more difficult to understand. In addition, when specifying the annotation target video, the annotation target video is not yet viewable at the timing of selecting the annotation target video, so it is necessary to specify without mistake the video to be specified as a target. Support is also an issue.

  Also, when annotating during shooting, if you want to add it in relation to the scene that was annotated a while ago, you want to check the status of the previous annotation assignment, but such a method can be used for live annotation or information that will be helpful in a short time You may want to read through. In order to make it easy to understand in a short time, it is good to follow the flow that has been noticed so far when the digest video is focused on the subject that has been noticed. On the other hand, when it is desired to read information that is useful for a short time, it is considered better to display a digest video centered on a subject to be photographed that is viewed by many viewing users.

  The present invention has been made in view of the above points. When annotating a plurality of captured images in a live manner, a display for easily selecting a selection of images capturing a scene to be annotated. An object of the present invention is to provide an annotation assigning method and apparatus, a program, and a computer-readable recording medium.

  FIG. 1 is a diagram for explaining the principle of the present invention.

The present invention (Claim 1) is a method for providing an annotation in a server that annotates a plurality of shooting target videos live during video shooting,
The shooting location ID, the shooting target absolute coordinate corresponding to the shooting device ID, the viewpoint reference point corresponding to the viewpoint ID, the shooting target range absolute coordinate, the shooting target representative image, and the shooting location management information including the shooting location are acquired from the user terminal. A location imaging environment management information acquisition step (step 1) stored in the location imaging environment management information storage means;
Referring to the location shooting environment information storage means, a shooting including a shooting device ID of a shooting device installed at a location corresponding to the shooting location ID and a shooting target representative coordinate that is a center of gravity of the shooting target shot by each shooting device. A reference step (step 2) for acquiring the target absolute coordinate value and the viewpoint reference point;
Using the imaging target representative coordinates of each imaging device corresponding to the imaging location ID and the conversion parameter indicated by the viewpoint reference point, each imaging device when the imaging target representative coordinates for the imaging device is viewed by the user is captured. A coordinate conversion step (step 3) for converting the coordinates to the user coordinate system that is the arrangement of the shooting target being performed and outputting the absolute coordinates of the shooting target to the location shooting environment management information storage means;
I do.

Further, according to the present invention (Claim 2), when a user facing a shooting target adds a live annotation to a plurality of videos of a user terminal that displays a video shot of the shooting target,
Based on the shooting location ID and viewpoint ID specified from the user terminal, the shooting target absolute coordinates are acquired from the location shooting environment management information storage means, and the terminal key of the user terminal used by the user based on the shooting target absolute coordinates Video terminal key associating step for associating the video captured by the photographing device with the video terminal key corresponding information storage means for each user / terminal,
When a shooting start request is received from the user terminal, the current date and time is held as the shooting start date and time. If the shooting device has not started recording, the recording start and shooting start video time offsets are set to 0, and the shooting device always When shooting, a video start point holding step for setting the current video time to a shooting start time offset and storing it in the video management information storage means together with the shooting ID and the shooting device ID;
An annotation time conversion step of obtaining an annotation giving video time by obtaining a shooting start date and time and a shooting start video time offset of the target photographing device ID from the video management information storage unit when acquiring the annotation giving request from the user terminal;
Annotation information registration step of registering the annotation giving video time and the annotation information acquired from the user terminal in the annotation information storage means;
Define the video section using the specified feature value for the captured video, generate a digest video for the video section including the annotation, acquire the video date and time and annotation information from the annotation information storage means, An annotated digest video reproduction step for synchronous display with the digest video is further performed.

  FIG. 2 is a principle configuration diagram of the present invention.

The present invention (Claim 3) is an annotation applying apparatus in a server for providing an annotation on a plurality of video images to be shot live during video shooting,
The shooting location ID, the shooting target absolute coordinate corresponding to the shooting device ID, the viewpoint reference point corresponding to the viewpoint ID, the shooting target range absolute coordinate, the shooting target representative image, and the shooting location management information including the shooting location are acquired from the user terminal. A location imaging environment management information acquisition unit 121 stored in the location imaging environment management information storage unit 130;
With reference to the location shooting environment management information storage unit 130, from the shooting device ID of the shooting device installed at the location corresponding to the shooting location ID and the shooting target representative coordinates that are the center of gravity of the shooting target shot by each shooting device. An imaging target absolute coordinate value and a reference means 122 for acquiring a viewpoint reference point;
Using the imaging target representative coordinates of each imaging device corresponding to the imaging location ID and the conversion parameter indicated by the viewpoint reference point, each imaging device when the imaging target representative coordinates for the imaging device is viewed by the user is captured. A coordinate conversion unit 123 that performs coordinate conversion into a user coordinate system that is an arrangement of the shooting target being performed and stores the absolute coordinates of the shooting target in the place shooting environment management information storage unit 130.

Further, according to the present invention (Claim 4), based on the shooting location ID and viewpoint ID specified from the user terminal, the shooting target absolute coordinates are acquired from the location shooting environment management information storage means 130, and the shooting target absolute coordinates are obtained. Based on the terminal key of the user terminal used by the user and the video terminal key associating means for associating the captured video captured by the photographing device with the video terminal key corresponding information storage means for each user / terminal,
When a shooting start request is received from the user terminal, the current date and time is held as the shooting start date and time. If the shooting device has not started recording, the recording start and shooting start video time offsets are set to 0, and the shooting device always When shooting, a video start point holding means for setting the current video time to a shooting start time offset and storing it in the video management information storage means together with the shooting ID and the shooting device ID;
An annotation time conversion unit that obtains the shooting start date and time and the shooting start video time offset of the target shooting device ID from the video management information storage unit when calculating the annotation addition request from the user terminal;
Annotation information registration means for registering annotated video time and annotation information acquired from the user terminal in the annotation information storage means;
Define the video section using the specified feature value for the captured video, generate a digest video for the video section including the annotation, acquire the video date and time and annotation information from the annotation information storage means, And an annotated digest video reproducing means for synchronously displaying the digest video.

  The present invention (Claim 5) is an annotation assignment program for causing a computer to function as each means constituting the annotation assignment apparatus according to Claim 3 or 4.

  The present invention (Claim 6) is a computer-readable recording medium storing the annotation adding program according to Claim 5.

  As described above, according to the present invention, it is possible to annotate live during shooting. In addition, it is possible to select an annotation target video without making a mistake by specifying a terminal key having a spatial sensation close to the actual shooting location by a user to whom annotation is given during live performance. Since the relation between the annotation target video and the annotation information is defined, the annotation and the target can be displayed in association with each other accurately, so that it is possible to perform video playback that is easy for the video viewer to understand. At the same time, since the video is digested by being limited to the portion to which the annotation is added, it is possible to perform a quick review to look back, and it is possible to support the content to be annotated in a short time.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  The present invention uses the positional relationship of the center of gravity of a subject to be photographed with a plurality of photographing equipment viewed from the viewpoint of a terminal user (browser) and the position of the terminal, and “a plurality of photographing equipments to be displayed on the terminal” The display position of the icon indicating the photographic equipment on the terminal is controlled so that the “arrangement of icons indicating“ is similar ”. As a result, the viewer can select the photographing equipment intuitively.

[First Embodiment]
FIG. 3 shows a system configuration in the first embodiment of the present invention.

  The system shown in the figure includes a server 100 and a user terminal 200, which communicate via respective communication units 110 and 210.

  The server 100 includes a communication unit 110, a processing unit 120, and a DB 130.

  In the DB 130, location shooting environment management information 131 is recorded. An example of the location shooting environment management information 131 is shown in FIG. The location shooting environment management information 131 manages a predetermined shooting target absolute coordinate, a predetermined viewpoint reference point, a shooting target range absolute coordinate, a shooting target representative image, a shooting location, and the like in association with the shooting location ID. Yes.

  The shooting location ID is an ID representing a shooting site such as “National Stadium”. It should be noted that the photographing target absolute coordinates (the photographing device ID associated with the photographing location ID and the centroid coordinate value of the photographing target photographed by the photographing device in the coordinate system predetermined for each photographing location) The target representative coordinate value) is determined in advance and managed as the location shooting environment management information 131 in association with the shooting location ID. That is, the number of photographing devices and where each photographing device is set to photograph are determined in advance and managed in association with the photographing location ID by the location photographing environment management information. The photographing target representative coordinate value is expressed using a coordinate system predetermined for each place ID.

  In addition, viewpoint reference points that can be selected by the user are determined in advance for each shooting location ID, and managed by the location shooting environment management information. The viewpoint reference point includes a viewpoint ID and a conversion parameter. Note that the viewpoint ID is an ID that indicates an approximate area in the place designated by the shooting place ID. For example, when the place indicated by the shooting place ID is “soccer stadium”, there is a method of providing viewpoint IDs indicating four places: the home goal back, the home zone, the away goal back, and the away zone. The home zone and the away zone may be further divided into the first floor and the second floor to provide a total of six shooting location IDs. By the way, when the user is at the position indicated by the viewpoint ID and the user faces the shooting target (for example, when the place indicated by the shooting place ID is “soccer stadium”, the user indicates the direction of the field. A user coordinate system based on the user's position and line-of-sight direction is assumed. Conversion parameters for converting the coordinate system determined for each shooting location ID into the user coordinate system are obtained in advance for each viewpoint ID, and are set in advance in the location shooting environment management information 131 of the DB 130 in association with the viewpoint ID. Leave. In the case of FIG. 4, parameters such as a rotation angle, translation, and enlargement / reduction associated with the viewpoint ID are conversion parameters.

  FIG. 5 is a flowchart of server processing according to the first embodiment of this invention.

  Step 100) The processing unit 120 of the server 100 acquires location shooting environment management information corresponding to the shooting location ID and viewpoint ID transmitted from the user terminal 200.

    Step 101) When the user specifies the shooting location ID and viewpoint ID, the communication unit 110 of the server 100 acquires the shooting location ID and viewpoint ID via the communication unit 210 of the user terminal 200.

    Step 102) The processing unit 120 refers to the location shooting environment management information 131 in the DB 130, and acquires the shooting target absolute coordinate value and the viewpoint reference point corresponding to the shooting location ID. The absolute coordinate value of the shooting target is obtained from the ID of the shooting device (shooting device ID) installed at the location indicated by the shooting location ID and the shooting target representative coordinates that are the center of gravity of the shooting target shot by each shooting device. Become.

  Step 200) The processing unit 120, when viewed from the user, arranges the objects captured by each imaging device (for example, the imaging targets of the three imaging devices are vertically arranged in a line at equal intervals). Ask for.

    Step 201) It is determined whether or not there is an unprocessed device ID among the photographing device IDs acquired in Step 102. If there is no unprocessed imaging device ID, the process proceeds to the end. If there is an unprocessed imaging device ID, the process proceeds to step 202, and the processing from step 202 onward is executed for one imaging device ID determined to be unprocessed.

    Step 202) The processing unit 120 refers to the location shooting environment management information 131 and uses the shooting target representative coordinates for each shooting device corresponding to the shooting location ID acquired in Step 102 and the viewpoint reference point to the shooting device. The image-capturing target representative coordinates are transformed into the user coordinate system. Specifically, the processing unit 120 applies the conversion parameter indicated by the viewpoint reference point to the photographing target absolute coordinate value for each device ID of the place photographing environment management information 131, and represents the photographing target representative for the photographing device. Coordinates the coordinate value to the user coordinate system. The result of expressing the photographing target absolute coordinate value in the user coordinate system is the arrangement of the images photographed by each photographing device when viewed from the user viewpoint. The processing unit 120 passes the coordinate conversion result of step 202 to the process of step 203. Note that the absolute coordinate value to be imaged is (x, y), the rotation angle indicated by the viewpoint ID is θ, the parallel movement amount in the x direction is α, the parallel movement amount in the y direction is β, the enlargement factor in the x direction is j, When the enlargement ratio in the y direction is k, the converted coordinate values (S, T) are calculated using the following equations.

ステップ２０３） 処理部１２０は、ユーザ座標系に変換された撮影対象絶対座標値を、撮影場所ＩＤ、撮影機器ＩＤ、視点ＩＤに対応付けたＤＢ１３０に記録する。必要に応じて、ステップ２０２以降の処理を実施した機器ＩＤが処理済であることを保持した後、ステップ２０１に戻る。

Step 203) The processing unit 120 records the shooting target absolute coordinate value converted into the user coordinate system in the DB 130 associated with the shooting location ID, the shooting device ID, and the viewpoint ID. If necessary, after holding that the device ID that has executed the processing from step 202 has been processed, the processing returns to step 201.

  In the flowchart of FIG. 5 described above, the process of recording the coordinate conversion result in the DB 130 is shown in Step 203. However, in Step 203, the DB 130 simply holds the coordinate conversion result, and in Step 201, all device IDs are stored. May be handed over to another process when it is determined that is processed.

  As a method of using the coordinate conversion result calculated in step 203, for example, when displaying icons of a plurality of photographing devices on a terminal held by the user, the arrangement of each photographing device viewed from the user's viewpoint is similar to the icon arrangement. There is a way to do that.

  Specifically, a situation is assumed in which an image is annotated while looking at sites photographed by a plurality of devices. At this time, the user looks at the site visually, but sometimes feels that he wants to annotate the video at some timing. The situation where I wanted to implement annotation is one of the support methods that can determine in short time which camera is being shot. Actual camera placement and the placement of camera icons to be displayed on the terminal A matching method is effective.

  For example, assume a situation where three shooting devices are installed in a soccer field, and two goals and the center of the field are shot with a camera. Further, assume that the user is behind the goal of the away team. At this time, on the terminal held by the user, three icons indicating the photographing devices are arranged vertically. In this situation, if the user feels that he wants to annotate the goal scene of the home team, select the camera that captures the away goal and Annotations need to be done. When selecting an imaging device, if the location of the location where each of the imaging devices is capturing is similar to the location of the location that indicates the shooting target displayed on the terminal, the user can intuitively annotate It is possible to select a photographing device to be performed. For example, in this example, the user selects a shooting device that is shooting near the goal of the away by selecting the icon displayed at the bottom of the three vertically arranged icons indicating the three shooting devices. It becomes possible.

[Second Embodiment]
In the present embodiment, an example will be described that includes processing for displaying an icon indicating a photographing device on a terminal held by a user using the coordinate value calculated in step 203 in the first embodiment.

  FIG. 6 is a system configuration diagram according to the second embodiment of the present invention. The system shown in the figure includes a server 300 and a user terminal 400.

  The server 300 includes a video terminal key association processing unit 310, a multiple video start point holding processing unit 320, an annotation target designation processing unit 330, an annotation assignment processing unit 340, an annotation time conversion processing unit 350, an annotation registration processing unit 360, a digest video. The reproduction processing unit 370, the location shooting environment management information DB 301, the user / terminal terminal key correspondence information DB 302, the video management information DB 303, and the annotation information DB 304 are configured.

  FIG. 7 is a flowchart of server processing according to the second embodiment of this invention.

  Step 301) The video terminal key association processing unit 310 takes out the photographing target absolute coordinate values managed by the place photographing environment management information stored in the place photographing environment management information DB 301. Also, a key pattern and key arrangement of user terminal information corresponding to the user terminal ID designated by the user as shown in FIG. Next, using the user ID, the shooting location ID, and the viewpoint ID specified from the user terminal, the user specified by the user ID by the terminal key and the shooting device corresponding process according to the method shown in FIGS. The association data of the terminal key used by the camera and the photographing device is stored in the video terminal key correspondence information DB 302 for each user / terminal. The user terminal-specific video terminal key correspondence information DB 302 stores a user terminal ID, a shooting location ID, a key character shooting video correspondence map {key arrangement, (key character, shooting device ID) set} as shown in FIG. Is done.

  In this association processing, the coordinate system is converted in accordance with the viewpoint viewed by the user who views the position of the shooting target, so that the positional relationship of the shooting target in the coordinate system of the user's line of sight is grasped and matched with the positional relationship. The purpose is to map the key arrangement of the user terminal 400. Specific processing details will be described with reference to FIGS. 10 and 11.

  10 shows the case of a fixed key arrangement, and FIG. 11 shows the case of a free key arrangement.

  Step 401) First, a photographing location ID, a viewpoint ID, and a user terminal ID are received from a user.

  Step 402) The photographing target coordinates are converted from the absolute coordinate system to the coordinate system from the viewpoint so that the user can grasp the positional relationship of the photographing target at the starting point of the photographing target. Here, the absolute coordinates of the shooting target are acquired from the location shooting environment management information DB 301 with the shooting location ID, and the rotation angle, parallel movement, and enlargement / reduction ratio data for conversion to the coordinate system at the user viewpoint are acquired with the viewpoint ID. . Assuming that the acquired absolute coordinates to be photographed before conversion are x, y, rotation is θ, and x magnification (j) and y magnification (k), coordinate transformation (S, T) can be performed by the following equations. .

これにより、撮影機器と対応した撮影対象の視点から座標系での座標値を計算できる。

Thereby, the coordinate value in a coordinate system can be calculated from the viewpoint of the imaging target corresponding to the imaging device.

  Step 403) Taking into consideration whether the subject is viewed side by side or vertically in the coordinate system from the viewpoint, depending on the appearance, the horizontal direction (x direction of the viewpoint coordinate system) or the vertical direction (viewpoint coordinate system) In the y direction), and is associated with the terminal key. Specifically, in step 403, the variance of the x coordinate and the variance of the y coordinate in the result coordinate value of step 402 are calculated, and the coordinate system having the larger variance is used to calculate the x coordinate value of the viewpoint coordinate system. Sort shooting equipment. The larger the variance is the axis of reference. In the processing of FIG. 10 and FIG. 11, the processing is almost the same so far.

  Step 404) The case of fixed key arrangement (FIG. 10) will be described.

  Using the user terminal ID and the result of the reference axis in step 403, the reference key of the key arrangement of the input terminal is acquired from the user terminal information, and the photographing device IDs are read in the order of the result sorted in step 403. The key characters are read in order, and the set and the key arrangement of the user terminal are output to the video terminal key correspondence information DB 302 for each user / terminal. In the case of a fixed key arrangement, the number of key arrangements in the x and y directions is fixed, so if the shooting device is larger than the number of terminal keys in each direction, information on which character key to map is specified in advance. This can be handled.

  Step 404 ′) In the case of a free key in this step (FIG. 11), keys for shooting targets are created in the direction determined in Step 403, and a table of correspondence between shooting devices and key characters is created in the order of sorting results. To do. In the case of the free key arrangement, it may be assumed that the number of photographing devices is displayed side by side in the direction on the determined reference axis. This completes the preparation for associating the user's terminal key with the annotation target.

  FIG. 12 shows an example when the viewpoint angles are different. In the figure, it is an example in which it is possible to cope with a live game other than a rectangle. In the example of the figure, it is assumed that the cameras A to C are shooting from the viewpoint 1 side. The processing in the example in the figure is basically the same as that in FIG. 10 or FIG. 11, but the coordinate system conversion in step 402 is different. An example of coordinate system conversion from each viewpoint in FIG. 12 is as follows.

Viewpoint 1: Same as global coordinate system (A (ax, ay), B (bx, by), C (cx, cy))
Viewpoint 2: Magnification rate 1 time, 45 degree rotation, (a, 0) parallel movement Viewpoint 3: Magnification ratio 1 time, 180 degree rotation, (a, a) Parallel movement Viewpoint 4: Magnification ratio 1 time, -45 degree rotation , (−a / √2, a / √2) parallel movement θ: degree of rotation, (α, β) is parallel movement, and if this is substituted into matrix calculation formula (2) in FIG. Desired. Then, the conversion arrangement to the key arrangement of FIG. 10 or FIG.

  In FIG. 12, even if it is not based on rectangles, such as soccer, tennis, and a general classroom, the process of said steps 401-404 can be performed similarly.

  Step 302) Next, the user makes a shooting start request at the shooting location, thereby associating actual video synchronization from each shooting device with the current date and time on the shot video. With the goal.

  As shown in FIG. 13, when the user makes a shooting start request, the multiple video start point holding processing unit 320 receives a shooting start request from the user (step 501), and the server 300 stores the current date and time as the shooting start date and time. (Not shown), and if the photographic device has not physically started recording, set the recording start and shoot start video time offset to 0, and if the photographic device is always shooting, Set the current video time to the shooting start video time offset. These results are held in the video management information DB 303 as video management information as shown in FIG. In the video management information shown in FIG. 14, a shooting ID, a shooting device ID, a shooting start date and time, a shooting start video time offset, and a recorded shot video file storage destination are set. This completes the preparation for selecting an annotation target and adding annotation information. This process may be set by one user existing at the shooting location.

  Step 303) Next, the user assigns annotation information to the video shot by which shooting device by selecting a terminal key that is associated with the live shooting target in visual order while watching live at the shooting location. Can be selected. Specifically, an interface example as shown in FIG. 15 can be considered. In the case of the fixed key arrangement (FIG. 15A), the representative images corresponding to the respective photographing devices are displayed in the same arrangement as the fixed key arrangement, thereby confirming which terminal key corresponds to which photographing device. An example of an interface that can be On the other hand, in the case of the free key arrangement (FIG. 15B), since the input keys can be freely arranged on the screen, the position where the representative image itself of each photographing device can actually be seen (the photographing target coordinates in the viewpoint coordinate system). The key characters can be transmitted to the server 300. Further, as a display form similar to the case of the fixed key, it may be possible to display the representative screen separately on one screen. In this way, it is possible to select a shooting target with an interface that is close in sensation to a live appearance from an actual user.

  Specifically, as shown in FIG. 16, when the user terminal ID and terminal key information are transmitted to the server 300 in response to the annotation target video designation request from the user terminal 400, the annotation target designation processing unit 330 of the server 300 The user terminal ID, terminal key, and shooting location ID shown in FIG. 9 are received (step 601), and based on these pieces of information, the user / terminal shooting as shown in FIG. The video terminal key correspondence information and the user terminal information are searched, and the user ID and the imaging device ID are transmitted to the annotation assignment processing unit 340 (step 602).

  Thus, the imaging device ID corresponding to the annotation target specified by the user can be passed to the annotation addition processing unit 340, and the annotation information received by the annotation addition processing unit 340 can be associated with the video captured by the imaging device ID.

  Step 304) Next, when the shooting ID, user ID, and annotation information (any one of a comment, a public flag, a vote, and a time mark) are transmitted from the user terminal 400 as an annotation grant request, the annotation grant processor 340 As shown in FIG. 17, the user ID and photographing device ID are received from the annotation target designation processing unit 330 (step 610), and the photographing ID, user ID, and annotation information which are user inputs are received, and the annotation reception date and time is as follows. The reception date and time are held (step 611). Finally, the user ID, annotation reception date / time, photographing ID, photographing device ID, and annotation information are set and output to the annotation time conversion processing unit 350 (step 612). Here, all of the input of the annotation target video designation processing unit 330 and the reception date and time at the input timing of the user and the user input information are merged and passed to the annotation time conversion processing unit 350.

  Step 305) Next, in the annotation time conversion processing unit 350, as shown in FIG. 18, transmission (set of user ID, annotation reception date / time, shooting ID, shooting device ID, annotation information) from the annotation assignment processing unit 340 is performed. Received (step 701), obtains the shooting start date and time and the shooting start video time offset of the target shooting device ID from the video management information DB 303 based on the shooting ID and the shooting device ID (step 702). (Step 703)

[Annotation video time] = [Annotation reception date and time]-[Shooting start date and time]
+ [Shooting start video time offset]
Finally, the user ID, the annotation addition video time calculated in step 703, the shooting ID, the shooting device ID, and the annotation information are set and output to the annotation registration processing unit 360 (step 704). Here, processing for associating an annotation to be given to the time of the video shot by each imaging device is performed.

  Step 306) Next, the annotation registration processing unit 360 stores the data received from the annotation time conversion processing unit 350 in the annotation information DB 304. A specific example of registration results in the annotation information DB 304 is shown in FIG.

As a result, the annotation information is held in a state where it can be associated with the video of the photographing device. As annotation information,
・ Comment: Comment text;
・ Public flag: whether to make a comment public or a personal note;
・ Voting: voting for annotation (annotation ID designation) (voting for comments);
・ Time mark: Mark to the video time of the target video (mark the target and time that seems to be important);
and so on.

  Step 307) Finally, digest video playback processing will be described.

  FIG. 20 is a flowchart of an outline of internal processing of the annotated digest video reproduction processing unit according to the second embodiment of the present invention.

  Step 810) First, the digest video reproduction processing unit 370 composes a digest video. Here, the digest video reproduction list is constructed while determining the important video section for the user who designates the annotation information or the entire user.

  Step 820) An annotation related to the video section is acquired and displayed and reproduced in synchronization with the digest video.

  Hereinafter, step 810 and step 820 will be sequentially described in detail.

  In step 810 above, the digest video playback processing unit 370 transmits the user ID, shooting, ID, digest video playback time or importance threshold in the digest video playback request from the user, as shown in FIG. 21 or FIG. Perform proper processing.

  First, the designated user pattern will be described.

  FIG. 21 is a flowchart of digest video processing (designated user pattern) of the digest video playback processing unit with annotation according to the second embodiment of the present invention.

  Step 811) First, the recorded video of each photographing device related to the received user ID and photographing ID is read. Since the location of the recorded video is managed in the video management information DB 303, the recorded video can be designated from the information.

  Step 812) The recorded video for each video device is divided into video sections, the start video time and the end video time of each divided part are set as division points, and the intervals between the division points are set as video sections. As a method of dividing the video into video sections, for example, a scene in which the same background scenery is continuously captured, a section in which a certain person is displayed, a section in which voice utterance is present, and a meaningful section are conceivable. For the semantic section, the video section viewpoint is set manually while checking the live video. Other than that, there is a method of automatically dividing a physical feature amount using image analysis technology. The video analysis techniques are described in JP-A-8-214210, JP-A-9-238298, JP-A-11-178007, and the like. In the present embodiment, as shown in FIG. 22, a method of setting a final video section from a video section using scene switching points is used.

  Step 813) The annotation information of the designated user is acquired from the annotation information DB 304 using the user ID and shooting ID as keys.

  Step 814) The video section information divided in Step 812 is acquired, and the following processing is performed for each captured video / video section for the annotation information in Step 813. The total number of annotations with comments is totaled for each divided video section. Set the amount of annotation (A) and total the number of annotations with “voting” for each divided video segment and set it as the annotation evaluation level (B). The total number of annotations "" is set as the video section attention level (C), the importance level (X) is calculated by the following method, and the importance level is held for each video section of each captured video.

Importance (X) = αA + βB + γC (0 ≦ α, β, γ ≦ 1)
Step 815) The following processing is performed in order from the first video section to the last video section. The photographed video ID having the highest importance in the same video section and its importance are stored in a memory (not shown) in descending order.

  Step 816a) If the importance threshold is specified, the results are read in the order of the results held in the memory (not shown) in Step 815. As long as the importance is equal to or higher than the importance threshold, the captured video ID and the importance By acquiring a set of video segments, it is possible to search only video segments that exceed the importance threshold.

Step 816b) If the digest playback time is specified, read in the order of the results held in the memory (not shown) in Step 815, add the video time of the video section cumulatively,
While the cumulative addition time ≦ the generation time is satisfied, a video digest within the reproduction time can be configured by acquiring a set of the captured video ID, importance, and video section.

  Step 817) The results of Step 816 are sorted again in the order of the start video time of the video section, and the set of the captured video ID and the video section (section start time, section end time) is transmitted to the next digest playback process as a digest video playback list. To do. By this process, a digest video reproduction list is constructed based on the user's own annotation information.

  Next, all user statistical patterns will be described.

  FIG. 23 is a flowchart of digest video composition processing (all user statistical patterns) of the digest video playback processing unit with annotation according to the second embodiment of the present invention.

  Step 811 ′) The recorded video of each photographing device related to the received photographing ID is read. Since the location of the recorded video is managed in the video management information DB 303, the recorded video can be designated from the information.

  Step 812 ′) Similar to the case of FIG. 21 described above, the recorded video for each photographing device is divided into video sections, and each divided point is video with each divided start video time and end video time as division points. Set as interval. The video section can be set by the method of FIG.

  Step 813 ′) The annotation information of all users is acquired from the annotation information DB 304 using the shooting ID as a key.

  Subsequent steps 814 ′ to 817 ′ are the same as steps 814 to 817 in FIG. 21. Thus, a digest video reproduction list can be generated based on the annotation information assignment for all users at the shooting location.

  Next, the process of step 820 in FIG. 20 will be described.

  FIG. 23 is a flowchart of synchronous playback processing of the annotated digest video playback processing unit in the second embodiment of the present invention.

  Step 821) First, the digest video playback processing unit 370 reads the digest video playback list created by the processing of FIGS.

  Step 822) The video sections of the video target are acquired in order from the beginning of the list.

  Step 823) Play the video segment of the video target. FIG. 25 shows an example of an interface to the user side in the display process of the step. In the figure, the video area and the comment area are displayed separately (FIG. 25, pattern 1), and the comment area is displayed superimposed on the video area (FIG. 25, pattern 2). The terminal has a video and comment area separately for each screen (FIG. 25, pattern 3). A digest video playback list can be listed together with thumbnails and comments, while thumbnails can be selected or only audio can be played back. In addition, an interface such as scrolling while synchronizing the screens of thumbnails and comments can be considered (FIG. 25, pattern 4).

  Step 824) Annotation information corresponding to the video object being reproduced and the video section is acquired, and display is performed in synchronization until the playback of the video section is completed.

  With the above, you can select the shooting target while watching a live sports stadium and add annotation information, or select the shooting target while attending a live presentation such as a lecture or lecture, and the annotation information Annotation information for video objects that match the user's point of view from a portable device such as a portable terminal, in a variety of live annotation information on the premise that the image is taken with a duplicate photographic device Easy to understand in a short time because digest video can be composed of multiple video with important video sections from usage statistics information of the entire user, which can be easily given to users from synchronized video from multiple cameras Also, by picking up important points and speeding up effectively, you won't get lost even in live situations. So as not to be a wrong understanding even over The, it is possible to provide the annotation information provided, given environment.

  In addition, the operation | movement of the component of the server 100 in said 1st Embodiment and the server 300 in 2nd Embodiment is constructed | assembled as a program, and it installs and runs on the computer utilized as the said server, or It is possible to distribute through a network.

  Further, the constructed program can be stored in a portable storage medium such as a hard disk, a flexible disk, or a CD-ROM, and can be installed or distributed in a computer.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

  The present invention is applicable to a technique for adding an annotation to a video.

It is a figure for demonstrating the principle of this invention. It is a principle block diagram of this invention. It is a system configuration figure in a 1st embodiment of the present invention. It is an example of the place imaging | photography environment management information in the 1st Embodiment of this invention. It is a flowchart of the process of the server in the 1st Embodiment of this invention. It is a system block diagram in the 2nd Embodiment of this invention. It is a flowchart of the process of the server in the 2nd Embodiment of this invention. It is an example of an input of the video terminal key matching process part in the 2nd Embodiment of this invention. It is an example of the result of the video terminal key matching process part in the 2nd Embodiment of this invention. It is a flowchart (fixed key arrangement | sequence) of the video terminal key matching process part in the 2nd Embodiment of this invention. It is a flowchart (free key arrangement | sequence) of the video terminal key matching process part in the 2nd Embodiment of this invention. It is a figure (example from which the angle of a viewpoint differs) for demonstrating the video terminal key matching process part in the 2nd Embodiment of this invention. It is a flowchart of the multiple image | video start point holding | maintenance process part in the 2nd Embodiment of this invention. It is a registration result (an example of video management information) of the multiple video start point holding processing unit in the second embodiment of the present invention. It is an example of the designation | designated interface at the time of annotation object designation | designated in the 2nd Embodiment of this invention. It is a flowchart of the annotation object designation | designated process part in the 2nd Embodiment of this invention. It is a flowchart of the annotation provision process part in the 2nd Embodiment of this invention. It is a flowchart of the annotation time conversion process part in the 2nd Embodiment of this invention. It is an example of the registration result to annotation information DB of the annotation registration process part in the 2nd Embodiment of this invention. It is a flowchart of outline | summary operation | movement of the digest video reproduction | regeneration processing part with an annotation in the 2nd Embodiment of this invention. It is a flowchart (designated user pattern) of the digest video structure process of the digest video reproduction process part with an annotation in the 2nd Embodiment of this invention. It is an example of the image | video division | segmentation method in the 2nd Embodiment of this invention. It is a flowchart (all user statistical pattern) of the digest video structure process of the digest video reproduction process part with an annotation in the 2nd Embodiment of this invention. It is a flowchart of the synchronous reproduction | regeneration processing of the digest video reproduction | regeneration processing part with an annotation in the 2nd Embodiment of this invention. It is a figure which shows the example of a display (synchronous reproduction) of the digest video reproduction | regeneration processing part with an annotation in the 2nd Embodiment of this invention.

Explanation of symbols

100 server 110 communication unit 120 processing unit 121 location shooting environment management information acquisition unit 122 reference unit 123 coordinate conversion unit 130 location shooting environment management information storage unit, DB
131 Location shooting environment management information 200 User terminal 210 Communication unit 300 Server 301 Location shooting environment management information DB
302 User / terminal video terminal key correspondence information DB
303 Video management information DB
304 Annotation information DB
310 Video terminal key association processing unit 320 Multiple video start point holding processing unit 330 Annotation target designation processing unit 340 Annotation giving processing unit 350 Annotation time conversion processing unit 360 Annotation registration processing unit 370 Digest video reproduction processing unit

Claims (6)

An annotation adding method in a server that annotates a plurality of shooting target videos live during video shooting,
The shooting location ID, the shooting target absolute coordinate corresponding to the shooting device ID, the viewpoint reference point corresponding to the viewpoint ID, the shooting target range absolute coordinate, the shooting target representative image, and the shooting location management information including the shooting location are acquired from the user terminal. A location shooting environment management information acquisition step stored in the location shooting environment management information storage means;
Referring to the location shooting environment information storage means, from the shooting device ID of the shooting device installed at the location corresponding to the shooting location ID and the shooting target representative coordinates that are the center of gravity of the shooting target shot by each shooting device. An imaging target absolute coordinate value and a reference step for obtaining a viewpoint reference point;
Each shooting when the shooting target representative coordinates for the shooting device are viewed from the user using the shooting target representative coordinates of the shooting device corresponding to the shooting location ID and the conversion parameter indicated by the viewpoint reference point. A coordinate conversion step of performing coordinate conversion to a user coordinate system, which is an arrangement of shooting targets that the device is shooting, and outputting the shooting target absolute coordinates to the location shooting environment management information storage means;
Annotation method characterized by performing When a user facing the shooting target adds a live annotation to a plurality of videos on the user terminal that displays the video shot of the shooting target,
Based on the shooting location ID and viewpoint ID specified from the user terminal, the shooting target absolute coordinates are acquired from the location shooting environment management information storage means, and the user used by the user based on the shooting target absolute coordinates A video terminal key associating step of associating a terminal key of the terminal with a captured video imaged by the photographing device and storing the video in the terminal key correspondence information storage unit for each user / terminal;
When a shooting start request is received from the user terminal, the current date and time is held as the shooting start date and time. If the shooting device has not started recording, the recording start and shooting start video time offsets are set to 0, and the shooting device A video start point holding step of setting the current video time to a shooting start time offset and storing it in the video management information storage means together with the shooting ID and the shooting device ID, when shooting constantly;
An annotation time conversion step of obtaining the annotation start video time by acquiring the shooting start date and time and the shooting start video time offset of the target shooting device ID from the video management information storage unit when the annotation grant request is acquired from the user terminal When,
Annotation information registration step of registering the annotation giving video time and the annotation information acquired from the user terminal in the annotation information storage unit;
Define the video segment using the specified feature value for the captured video, generate the digest video for the video segment including the annotation, and acquire the video date and time and the annotation information from the annotation information storage means And an annotated digest video playback step for synchronously displaying the digest video,
The annotation assigning method according to claim 1, further comprising: An annotation adding device in a server that annotates a plurality of shooting target videos live during video shooting,
The shooting location ID, the shooting target absolute coordinate corresponding to the shooting device ID, the viewpoint reference point corresponding to the viewpoint ID, the shooting target range absolute coordinate, the shooting target representative image, and the shooting location management information including the shooting location are acquired from the user terminal. Location shooting environment management information acquisition means stored in the location shooting environment management information storage means,
Referring to the location shooting environment information storage means, from the shooting device ID of the shooting device installed at the location corresponding to the shooting location ID and the shooting target representative coordinates that are the center of gravity of the shooting target shot by each shooting device. An imaging target absolute coordinate value and a reference means for acquiring a viewpoint reference point;
Each shooting when the shooting target representative coordinates for the shooting device are viewed from the user using the shooting target representative coordinates of the shooting device corresponding to the shooting location ID and the conversion parameter indicated by the viewpoint reference point. A coordinate conversion unit that converts the coordinates into a user coordinate system that is an arrangement of shooting targets that the device is shooting, and outputs the shooting target absolute coordinates to the location shooting environment management information storage unit;
An annotation giving device characterized by comprising: Based on the shooting location ID and viewpoint ID specified from the user terminal, the shooting target absolute coordinates are acquired from the location shooting environment management information storage means, and the user used by the user based on the shooting target absolute coordinates Video terminal key associating means for associating a terminal key of the terminal with a captured video imaged by the photographing device and holding the video terminal key corresponding information storage means for each user / terminal;
When a shooting start request is received from the user terminal, the current date and time is held as the shooting start date and time. If the shooting device has not started recording, the recording start and shooting start video time offsets are set to 0, and the shooting device In the case of continuous shooting, a video start point holding unit that sets the current video time to a shooting start time offset and stores it in the video management information storage unit together with the shooting ID and the shooting device ID;
Annotation time conversion means for obtaining the annotation start video time by acquiring the shooting start date and time and the shooting start video time offset of the target shooting device ID from the video management information storage means upon obtaining the annotation assignment request from the user terminal When,
Annotation information registration means for registering the annotation-added video time and the annotation information acquired from the user terminal in the annotation information storage means;
Define the video segment using the specified feature value for the captured video, generate the digest video for the video segment including the annotation, and acquire the video date and time and the annotation information from the annotation information storage means An annotated digest video reproduction means for synchronously displaying the digest video;
The annotation applying apparatus according to claim 3, further comprising:   An annotation assignment program for causing a computer to function as each means constituting the annotation assignment apparatus according to claim 3.   A computer-readable recording medium storing the annotation assigning program according to claim 5.

Images