JP2010061231A - Three-dimensional moving image reproduction device, three-dimensional moving image reproduction method, and program thereof - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a proper three-dimensional moving image reproduction device, suitable for performing indirect edition on a 3D image displayed on a 3D display by a space mouse. <P>SOLUTION: The three-dimensional moving image reproduction device is configured to detect a designated position by a space mouth in a 3D image, according to the change of the position of the space mouse, and to detect a plurality of display positions of 3D meta data in the 3D moving image among the designated positions, based on the position specification information received from the space mouse and the designated position by the space mouth at prescribed time intervals; to detect the time elapsed, from the start time of the 3D moving image displaying the 3D data in the 3D moving image, based on the start time of the 3D image and the time when the display position of the 3D data has been detected; and to record a plurality of display attribute information configured of the display position of the 3D data and the elapsed time, from the start time of the 3D moving image, when the 3D meta data are displayed at the display position in the 3D meta data DB at each display position of the 3D meta data. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a three-dimensional moving image reproduction device, a three-dimensional moving image reproduction method, and a program for displaying a 3D moving image and metadata on a 3D display.

There are many techniques for editing a target image displayed in a moving image. For example, editing using an input device such as a mouse for an object in a two-dimensional image displayed on a 2D (2Dimension) display is possible. Editing using an input device such as a spatial mouse for an object in a three-dimensional moving image displayed on a 2D display is applied to moving image editing software, a computer game application, or the like. The spatial mouse is an input device for specifically inputting a designated position on an image from a movement amount based on a change in the position of the real space. Patent Documents 1 and 2 disclose a technique for editing a moving image on a 2D image displayed by 3D (3Dimension) information displayed in the moving image.
“Overview of Solid Ray Laboratories Omega Space”, “online”, “Search July 7, 2008”, Internet <URL: http://www.solidray.co.jp/product/omega/index.html#1 > “Autodesk 3ds Max”, “online”, “July 7, 2008 search”, Internet <URL: http://usa.autodesk.com/adsk/servlet/index?id=5659302&siteID=123112>

  Here, in the editing of the moving image, direct editing for editing the image to be edited in the original moving image and indirect editing for additional information to be stored as information different from the original moving image and displayed in the moving image. Although there is editing, it is desired to develop an appropriate technique when performing indirect editing with a spatial mouse.

  Accordingly, an object of the present invention is to provide a three-dimensional moving image reproducing apparatus, a three-dimensional moving image reproducing method, and a program thereof suitable for performing indirect editing on a 3D moving image displayed on a 3D display with a spatial mouse. Yes.

  In order to achieve the above object, the present invention provides a three-dimensional moving image reproduction apparatus for reproducing and outputting a preliminarily stored three-dimensional moving image on a three-dimensional moving image display means, the position of the spatial input device existing in real space. And a designated position detecting means for detecting a designated position by the spatial input device in the three-dimensional moving image according to a change in the position, and among the designated positions by the spatial input device, in the three-dimensional moving image Additional information display position detecting means for detecting a plurality of additional information display positions based on either the position specifying information received from the spatial input device or the designated position by the spatial input device at a predetermined time interval; Based on the start time of the three-dimensional moving image and the time when the display position of the additional information is detected, an elapsed time from the start time for displaying the additional information in the three-dimensional moving image is calculated. Display attribute information including display time detection means to be output, and the display position of the additional information and the elapsed time from the start time of the three-dimensional moving image for displaying the additional information at the display position. A display attribute information recording unit that records a plurality of information to a display attribute information storage unit for each display position.

  According to the present invention, in the above-described three-dimensional moving image reproduction device, display trajectory calculation means for calculating display trajectory information for specifying a display trajectory connecting the plurality of display positions from the plurality of display positions of the additional information. The display attribute information recording means records a plurality of the display attribute information further including the calculated display trajectory information in the display attribute information storage means for each display position of the additional information.

  Further, the present invention is characterized in that, in the above-described three-dimensional moving image reproduction apparatus, the display trajectory information is each coefficient of an expression representing each straight line connecting the plurality of display positions of the additional information.

  Further, the present invention is characterized in that, in the above-described three-dimensional moving image reproduction device, the display trajectory information is each coefficient of an expression representing a spline curve connecting a plurality of the display positions of the additional information.

  According to the present invention, in the above-described three-dimensional moving image reproduction device, a plurality of continuous display positions among the display positions of the additional information in the plurality of detected three-dimensional moving images are start points of the plurality of continuous display positions. Whether or not it is located in a virtual cylinder with a predetermined radius centered on an extension line of a straight line connecting the display position of the next point and the display position of the next point, in order from the display position next to the display position of the next point, Display trajectory calculation means for calculating display trajectory information comprising coefficients of a formula representing a straight line connecting a display position immediately before a display position determined not to be located in the virtual cylinder for the first time and the display position of the start point; The display trajectory calculating means repeatedly calculates the display trajectory information starting from the display position immediately before the display position determined not to be located in the virtual cylinder for the first time, and the display attribute information recording means Calculated The display attribute information further including 示軌 trace information, characterized by a plurality of recording to the display attribute information storage unit for each display position of the additional information.

  According to the present invention, in the above-described 3D moving image reproduction device, a display trajectory connecting the plurality of display positions of the additional information represented by the display trajectory information is compared with a predetermined trajectory template. Trajectory template specifying means for specifying a matching trajectory template, wherein the display attribute information recording means displays the display attribute information further including the trajectory template for each display position of the additional information. A plurality of recordings are recorded.

  According to the present invention, in the above-described three-dimensional moving image reproduction device, two consecutive display positions among the display positions of the additional information in the plurality of detected three-dimensional moving images and display times of these display positions are based. The movement information calculation means for calculating the movement speed and movement direction of the additional information at each of the two consecutive display positions, and the display attribute information recording means further include the calculated movement speed and movement direction. A plurality of the display attribute information is recorded in a display attribute information storage unit for each display position of the additional information.

  According to the present invention, in the above-described three-dimensional moving image reproduction device, the three-dimensional moving image reproducing unit that reproduces and outputs the three-dimensional moving image to the three-dimensional moving image display unit, and the display attribute information of the additional information as the display attribute The additional information is read based on the display attribute information, and the additional information display means for displaying the additional information in the three-dimensional moving image based on the display attribute information, and the additional information is selected based on the designated position by the spatial input device. Additional information selection presence / absence determining means for determining whether or not the designated position movement detecting means for detecting movement of the designated position by the spatial input device, and movement of the designated position is detected while the additional information is selected. In this case, the display trajectory information after the movement of the additional information displayed in the three-dimensional moving image is reproduced based on the display trajectory information related to the additional information and the movement amount of the designated position. And a trajectory recalculation means for calculation, the additional information display means, based on the display locus information after the mobile, and displaying the additional information in the 3-dimensional moving image.

  The present invention also relates to a processing method in a three-dimensional moving image reproduction apparatus that reproduces and outputs a three-dimensional moving image stored in advance to a three-dimensional moving image display means, wherein the designated position detection means of the three-dimensional moving image reproduction apparatus comprises: The position of the spatial input device existing in the real space is detected, the designated position by the spatial input device in the three-dimensional moving image is detected according to the change of the position, and the additional information display of the three-dimensional moving image reproduction device The position detection means receives the display position of the additional information in the three-dimensional moving image among the specified positions by the spatial input device, the position specifying information received from the spatial input device, or the spatial input device at a predetermined time interval When the display time detection means of the three-dimensional moving image reproduction apparatus detects the start time of the three-dimensional moving image and the display position of the additional information. Based on the above, an elapsed time from the start time for displaying the additional information in the three-dimensional moving image is detected, and display attribute information recording means of the three-dimensional moving image reproducing apparatus is configured to display the display of the additional information. A plurality of display attribute information including a position and an elapsed time from the start time of the three-dimensional moving image for displaying the additional information at the display position are recorded in the display attribute information storage unit for each display position of the additional information. Is a processing method characterized by

  According to the present invention, in the processing method described above, the display trajectory calculating means of the three-dimensional moving image playback device specifies a display trajectory connecting the plurality of display positions from the plurality of display positions of the additional information. Trajectory information is calculated, and the display attribute information recording means of the three-dimensional moving image reproduction device displays the display attribute information further including the calculated display trajectory information for each display position of the additional information. A plurality of recordings are recorded.

  According to the present invention, in the processing method described above, the display trajectory calculation unit of the three-dimensional moving image reproduction device has a plurality of continuous display positions among the display positions of the additional information in the plurality of detected three-dimensional moving images. , Whether the position of the display point of the next point is within a virtual cylinder having a predetermined radius centered on an extension line of a straight line connecting the display position of the start point and the display point of the next point. Display trajectory information comprising coefficients of an expression representing a straight line connecting the display position immediately before the display position determined in order from the next display position and not first positioned in the virtual cylinder and the display position of the start point And the display trajectory calculation means of the three-dimensional moving image reproduction apparatus repeatedly calculates the display trajectory information starting from the display position immediately before the display position determined not to be positioned for the first time in the virtual cylinder. The third order The display attribute information recording unit of the moving image reproducing apparatus records a plurality of the display attribute information further including the calculated display trajectory information in a display attribute information storage unit for each display position of the additional information. .

  According to the present invention, in the processing method described above, a trajectory template specifying unit of the three-dimensional moving image playback device includes a display trajectory connecting the plurality of display positions of the additional information represented by the display trajectory information, The matching trajectory template is compared to identify a matching trajectory template, and the display attribute information recording means of the three-dimensional moving image reproduction device uses the display attribute information further including the trajectory template as the additional information. A plurality of display positions are recorded in the display attribute information storage means.

  According to the present invention, in the processing method described above, the movement information calculation unit of the three-dimensional moving image reproduction device includes two consecutive display positions among the display positions of the additional information in the plurality of detected three-dimensional moving images. And the display attribute information recording means of the three-dimensional moving image reproduction device, calculating the moving speed and moving direction of the additional information at each of the two consecutive display positions. A plurality of the display attribute information further including the calculated moving speed and moving direction is recorded in a display attribute information storage unit for each display position of the additional information.

  According to the present invention, in the processing method described above, the three-dimensional moving image reproducing unit of the three-dimensional moving image reproducing device reproduces and outputs the three-dimensional moving image to a three-dimensional moving image display unit, and the three-dimensional moving image reproducing unit The additional information display unit of the apparatus reads the display attribute information of the additional information from the display attribute information storage unit, displays the additional information in the three-dimensional moving image based on the display attribute information, and The additional information selection presence / absence determining means of the moving image reproduction device determines whether the additional information has been selected based on the designated position by the spatial input device, and the designated position movement detecting means of the three-dimensional moving image reproduction device includes: When the movement of the designated position is detected by the spatial input device, and the trajectory recalculation unit of the three-dimensional moving image reproduction device detects the movement of the designated position while the additional information is selected, The display trajectory information after the movement of the additional information displayed in the three-dimensional moving image is recalculated based on the display trajectory information related to the additional information and the movement amount of the designated position, and the three-dimensional moving image is reproduced. The additional information display means of the apparatus displays the additional information in the three-dimensional moving image based on the display trajectory information after the movement.

  According to the present invention, a computer of a three-dimensional moving image reproduction apparatus that reproduces and outputs a three-dimensional moving image stored in advance to a three-dimensional moving image display means detects the position of the spatial input device existing in the real space, and A designated position detecting means for detecting a designated position by the spatial input device in the three-dimensional moving image according to a change, and a display position of additional information in the three-dimensional moving image among the designated positions by the spatial input device, A plurality of additional information display position detection means for detecting a plurality of positions based on any one of the position specifying information received from the spatial input device or the designated position by the spatial input device at a predetermined time interval; and the start time of the three-dimensional moving image; Based on the time when the display position of the additional information is detected, a display time detection for detecting an elapsed time from the start time for displaying the additional information in the three-dimensional moving image. Means for displaying the display attribute information including the display position of the additional information and the elapsed time from the start time of the three-dimensional moving image for displaying the additional information at the display position for each display position of the additional information. This is a program for causing a plurality of display attribute information recording units to be recorded in the information storage unit.

  The present invention further causes the computer to function as display trajectory calculation means for calculating display trajectory information for specifying a display trajectory connecting the plurality of display positions from the plurality of display positions of the additional information, The display attribute information recording unit records a plurality of the display attribute information further including the calculated display trajectory information in a display attribute information storage unit for each display position of the additional information.

  According to the present invention, the computer further includes a three-dimensional moving image reproduction unit that reproduces and outputs the three-dimensional moving image to a three-dimensional moving image display unit, and reads the display attribute information of the additional information from the display attribute information storage unit. Additional information display means for displaying the additional information in the three-dimensional moving image based on the display attribute information, and additional information for determining whether the additional information is selected based on a designated position by the spatial input device A selection presence / absence determination unit; a specified position movement detection unit that detects movement of a specified position by the spatial input device; and when the movement of the specified position is detected while the additional information is selected, the three-dimensional moving image Trajectory re-calculation for recalculating the display trajectory information after the movement of the additional information displayed inside based on the display trajectory information related to the additional information and the amount of movement of the specified position Means, to function as the additional information display means, based on the display locus information after the mobile is a program and displaying said additional information in said three-dimensional moving image.

According to the present invention, 3D metadata to be synthesized and displayed on a 3D moving image can be created by an easy operation using a space mouse (only movement of the space mouse up and down, left and right and back and forth in space).
In addition, according to the present invention, at the time of creating 3D metadata, not all the positions on the 3D moving image designated by the spatial mouse are held as 3D metadata display trajectory information, and only a plurality of discrete sample points are stored. Is stored as trajectory information, so that data to be stored can be reduced.
In addition, according to the present invention, editing of the display trajectory of 3D metadata can be performed by an easy operation using a space mouse (only movement of the space mouse up and down, left and right and back and forth in space).
Further, according to the present invention, when a user selects a display track when editing a display track of 3D metadata, the 3D metadata is selected within a certain range without directly specifying the 3D metadata. Therefore, it is possible to easily select 3D metadata for editing.
In addition, a fixed template, a moving speed and a moving direction of each 3D metadata display trajectory can be stored as motion expression information, so that the 3D metadata displayed in the 3D moving image is searched. When the function is provided, it is possible to perform processing for searching for 3D metadata displayed on the 3D moving image by designating the template, the moving speed, and the moving direction as the data to be searched. Become.
Further, according to the present invention, when the user edits the 3D metadata displayed at the time of playback of the 3D moving image, the display trajectory after the editing is immediately calculated, and the composite display of the 3D metadata is performed based on the display trajectory. Therefore, 3D metadata can be edited in real time.
Further, according to the present invention, since the display position of the 3D metadata is automatically specified by moving the spatial mouse when the user creates the 3D metadata, the display locus of the 3D metadata is easily input. be able to.

Hereinafter, a three-dimensional moving image reproduction apparatus according to an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing a configuration of a three-dimensional moving image reproduction apparatus according to the embodiment.
From this figure, the three-dimensional moving image reproduction apparatus includes a spatial mouse 11, a spatial mouse control unit 12, a 3D moving image DB 13, a 3D metadata DB 14 (display attribute information storage means), a 3D display 15, a reproduction processing unit 16 (additional information display). Position detection means, display time detection means, three-dimensional moving image reproduction means, additional information display means), motion expression information generation unit 17 (display locus calculation means, locus template specification means, movement information calculation means, locus recalculation means), Each processing unit and storage unit of the sample point generation unit 18 (designated position detection means, additional information selection presence / absence judgment means, designated position movement detection means), 3D metadata generation editing unit 19 (display attribute information recording means) are provided. . Here, the space mouse 11 is used to input information to the three-dimensional moving image reproduction device by changing the position in the real space with the hand held by the user. The designated position on the image is detected from the amount of movement based on the variation. As a specific process, when the user moves the space mouse 11 in the real space, a detection unit connected to or attached to the three-dimensional moving image reproduction apparatus measures the distance and relative position with the space mouse by infrared rays or the like. Based on the measurement result, the spatial mouse control unit 12 performs processing for specifying the position on the 3D image. The spatial mouse control unit 12 is a processing unit that receives information from the spatial mouse 11 and detects a designated position of the spatial mouse. The 3D moving image DB 13 is a DB (database) that stores 3D moving image data. In addition, the 3D metadata DB 14 stores 3D metadata that is additional information to be combined and displayed on the 3D moving image. In addition, the 3D display 15 is a display dedicated to 3D image display so that a human who sees an image displayed on a plane can be seen three-dimensionally by a technique such as a parallax barrier method (parallax barrier method).

  The reproduction processing unit 16 is a processing unit that performs processing of outputting a 3D moving image to the 3D display 15 and outputting 3D metadata onto the 3D moving image. The motion expression information generating unit 17 also includes movement expression information including display position and display time of 3D metadata, display speed information for specifying a moving speed, a moving direction, and a trajectory of the 3D metadata on the 3D moving image. Is a processing unit for generating In addition, the sample point generation unit 18 performs a process of detecting the display position of 3D metadata in the 3D moving image among the designated positions on the 3D moving image by the spatial mouse control unit 12. The 3D metadata generation / editing unit 19 performs processing such as rewriting information stored in the 3D metadata DB 14 based on the motion expression information generated by the motion expression information generation unit 17.

  The 3D moving image playback apparatus according to the present embodiment plays back and outputs a 3D moving image stored in advance to the 3D display 15, detects the position of the space mouse 11 existing in the real space, and The specified position by the spatial mouse 11 in the 3D moving image is detected according to the change, and the position specifying information received from the spatial mouse 11 is the display position of the 3D metadata in the 3D moving image out of the specified position by the spatial mouse 11. Alternatively, a plurality of detections are made based on the designated position by the spatial mouse 11 at a predetermined time interval. Then, the 3D moving image reproduction device starts the 3D moving image for displaying 3D metadata in the 3D moving image based on the start time of the 3D moving image and the time when the display position of the 3D metadata is detected. The display attribute information including the display time of 3D metadata and the elapsed time from the start time of the 3D moving image for displaying the 3D metadata at the display position is detected as the display position of the 3D metadata. A process of recording a plurality of records in the 3D metadata DB 14 is performed every time.

  Further, the 3D moving image reproduction apparatus calculates display trajectory information for specifying a display trajectory connecting the plurality of display positions from the plurality of display positions of the 3D metadata, and further includes the calculated display trajectory information. A process of generating attribute information and recording a plurality of attribute information in the 3D metadata DB 14 for each display position of the 3D metadata is performed. Further, the 3D moving image playback apparatus plays back and outputs the 3D moving image to the 3D display 15, reads the display attribute information of the 3D metadata from the 3D metadata DB 14, and converts the 3D metadata to 3D based on the display attribute information. Display in the video. Then, the 3D moving image reproduction device determines whether 3D metadata is selected based on the designated position by the spatial mouse 11, detects the movement of the designated position by the spatial mouse 11, and selects the 3D metadata. When the movement of the designated position by the spatial mouse 11 is detected, the display trajectory information after the movement of the 3D metadata displayed in the 3D moving image is replaced with the original display trajectory information related to the 3D metadata. Then, recalculation is performed based on the movement amount of the designated position, and 3D metadata is redisplayed in the 3D moving image based on the display trajectory information after the movement.

FIG. 2 is a diagram showing a 3D metadata information table.
FIG. 3 shows a sample point information table.
Next, a data table stored in the 3D metadata DB 14 will be described with reference to FIGS. The 3D metadata information table shown in FIG. 2 is stored for each 3D metadata. The ID for specifying the 3D metadata, the 3D metadata type (character information, graphic information, etc.), and the 3D metadata Display start time on the 3D moving image, display time width from the display start time on the 3D moving image, spatial coordinates that are positions to be displayed on the 3D moving image at the display start time, according to the type of 3D metadata Individual point information and sample point information (display attribute information) used for displaying 3D metadata on a 3D moving image are stored in association with each other.

  The sample point information table shown in FIG. 3 is a data table that stores details of the sample point information in the 3D metadata information table shown in FIG. As shown in this figure, the sample point information table includes a sample point number for identifying each discrete display position on the 3D moving image of 3D metadata, and the 3D moving image of the display position indicated by the sample point number. Coordinates in virtual space, display position of two consecutive discrete sample points and 3D metadata movement direction, movement speed calculated based on each display time, 3D metadata trajectory template, and 3D metadata The coefficient of the mathematical expression representing the data locus line is stored in association with each other. Note that FIG. 3 shows an example in which the coefficient of the mathematical expression representing the trajectory line when the trajectory of the 3D metadata is expressed by a spline curve connecting sample points is shown.

FIG. 4 is a diagram showing a processing flow when creating 3D metadata.
Next, the processing flow of the three-dimensional moving image reproduction apparatus will be described with reference to FIG.
Creation of 3D metadata in the 3D moving image playback apparatus is performed in a state where a 3D moving image is displayed on the 3D display 15. At this time, first, the reproduction processing unit 16 reads the 3D moving image data designated by the user from the 3D moving image DB 13 and performs reproduction processing (step S101), and outputs it to the 3D display 15. Then, the user selects 3D metadata to be displayed on the 3D moving image output to the 3D display 15 using the spatial mouse 11 (for example, selects from a list or the like). Then, the reproduction processing unit 16 synthesizes and displays the selected 3D metadata information on the 3D moving image (step S102). Here, the user selects the 3D metadata displayed on the 3D moving image by using the space mouse 11, and moves the space mouse 11 in any direction of up, down, left, right, and back in the real space while selecting. The 3D metadata displayed on the 3D moving image can be moved. Specifically, in this process, the spatial mouse control unit 12 detects the movement direction and movement amount of the spatial mouse 11 and notifies the reproduction processing unit 16 of the movement direction and movement amount so that the reproduction processing unit 16 performs 3D. Processing for moving the display position of the metadata on the 3D moving image is performed.

  When the 3D metadata is moved and displayed on the 3D moving image, the user uses the space mouse 11 to instruct the start of metadata creation. Then, the sample point generation unit 18 receives an instruction to start creating metadata from the spatial mouse control unit 12 (step S103), and receives position specifying information from the spatial mouse 11 via the spatial mouse control unit 12 (step S103). S104). This position specifying information is for specifying a position specified by the user to display 3D metadata using the spatial mouse 11. The sample point generation unit 18 receives the position specifying information, the display position of the 3D metadata on the 3D moving image (the coordinates in the virtual space of the 3D moving image), and the 3D metadata at the display position. The playback processing unit 16 is inquired about the elapsed time from the playback start time of the 3D moving image at the displayed time. The reproduction processing unit 16 detects a display position (coordinates in the virtual space of the 3D moving image) of the 3D metadata at the reception time of the position specifying information (step S105), and also from the reproduction start time. Since the elapsed time is measured, the elapsed time at that time is detected (step S106), and the display including the detected display position of the 3D metadata, the elapsed time from the reproduction start time, and the ID of the 3D metadata. The attribute information is transmitted to the sample point generation unit 18. Then, the sample point generation unit 18 transmits the received display attribute information and 3D metadata creation instruction to the 3D metadata generation / editing unit 19.

  Next, the 3D metadata generation / editing unit 19 that has received the display attribute information and the 3D metadata creation instruction temporarily stores the display attribute information. In addition, when the user inputs the display position a plurality of times using the space mouse 11 or when the display position is automatically detected at a predetermined interval, the sample point generation unit 18 is positioned each time. The specific information is transmitted to the reproduction processing unit 16, and as a result, the sample point generation unit 18 receives the display attribute information from the reproduction processing unit 16 and transmits it to the 3D metadata generation / editing unit 19. As a result, the 3D metadata generation / editing unit 19 temporarily stores a plurality of display attribute information. Then, the spatial mouse control unit 12 determines whether the display end input of the 3D metadata has been received from the spatial mouse 11 based on the user's operation (step S107). The sample point generation unit 18 is notified to create the metadata up to. Then, the sample point generation unit 18 notifies the generation of 3D metadata to the 3D metadata generation / editing unit 19, and the 3D metadata generation / editing unit 19 converts the plurality of display attribute information temporarily stored into motion expression information. The data is transmitted to the generation unit 17 to instruct generation of search target data.

  Next, the motion expression information generation unit 17 starts generating search target data based on the received plurality of display attribute information (step S108). Here, the data to be searched is the moving direction and movement of the 3D metadata on the 3D moving image calculated based on the display position of the 3D metadata indicated by the display attribute information and the elapsed time from the reproduction start time. This is a shape template for speed and trajectory. The data to be searched such as the moving direction, moving speed, and trajectory shape template is information used for searching 3D metadata displayed in the 3D moving image. When the generation of the search target data is completed, the motion expression information generation unit 17 transmits the generated search target data to the 3D metadata generation / editing unit 19. The search target data is calculated for each display position (sample point) of the 3D metadata indicated by the display attribute information. Then, the 3D metadata generation / editing unit 19 associates each temporarily stored display attribute information with the searched data calculated for each display attribute information, and sample point information in the 3D metadata DB 14 Register in the table (step S109). That is, each display attribute information includes an ID of 3D metadata, a display position of the 3D metadata, an elapsed time from a reproduction start time displayed at the display position, a moving direction and a moving speed at the display position. The locus shape template is registered in the sample point information table. Note that, using the information recorded in the sample point information table as described above, 3D metadata is output when a 3D moving image is reproduced.

  Through the above processing, the moving direction, moving speed, trajectory shape template, display position, and display position of the 3D metadata displayed on the 3D moving image at the display position specified by the user or automatically detected. It is possible to register the elapsed time from the reproduction start time to be displayed on the screen. Since information on discrete display positions (sample points) is registered in this way, the information stored in the database can be reduced.

FIG. 5 is a diagram showing a processing flow of the composite display of 3D metadata during 3D moving image playback.
Hereinafter, 3D metadata composite display processing during 3D moving image playback will be described.
When instructed to reproduce the 3D moving image, the reproduction processing unit 16 reads the 3D moving image data instructed to reproduce from the 3D moving image DB 13 and reproduces it (step S201), and outputs it to the 3D display 15. Also, the ID of 3D metadata to be combined with the reproduced 3D moving image is stored in the 3D metadata DB 14 in association with the ID of the 3D moving image data, and the reproduction processing unit 16 sets the ID of the 3D metadata. In addition to reading, sample point information associated with the 3D metadata ID and registered in the 3D metadata DB 14 is acquired (step S202). Then, the reproduction processing unit 16 sets the display position of each sample point stored in the acquired sample point information, the set time elapsed from the reproduction start time at the display position, and the ID of the 3D metadata to be synthesized and displayed as motion expression information. It outputs to the production | generation part 17 and instruct | indicates the production | generation of display locus information (step S203).

  Next, when generating the display trajectory information, the motion expression information generating unit 17 displays the display trajectory information (sample point display position based on the set of the display position of each sample point and the elapsed time from the reproduction start time at the display position. Is calculated (step S204), and the display trajectory information and the information of the set of the display position of each sample point and the elapsed time from the playback start time at the display position are calculated in 3D. Corresponding to the metadata ID, it is stored in a memory or the like in its own processing unit (step S205). Then, the motion expression information generation unit 17 sets a set of the display position of each sample point of the 3D metadata to be synthesized and displayed and the elapsed time from the reproduction start time at the display position when the reproduction processing unit 16 reproduces the 3D moving image. Based on the display trajectory information of the 3D metadata, a display position corresponding to the elapsed time from the playback start time of the 3D moving image is calculated (step S206), and an instruction to display the 3D metadata at the display position is issued. Processing to be performed is performed on the reproduction processing unit 16 (step S207). Thereby, the reproduction processing unit 16 reproduces the 3D moving image and performs a process of combining and displaying the 3D metadata on the 3D moving image (step S208). The calculation of the display trajectory information may be performed in advance before the 3D moving image reproduction process or may be performed simultaneously with the reproduction of the 3D moving image. The display trajectory information may be recorded in advance when creating the 3D metadata, and recorded in the sample point information table (FIG. 3) as a part of the motion expression information. In addition, there are cases where the display trajectory is a trajectory connecting the display position of each sample point to a straight line, a display trajectory connecting the display position of each sample point by a spline curve, or a specific trajectory template. A calculation example of each display trajectory information when each of these display trajectories is used will be described later.

FIG. 6 is a first diagram illustrating a processing flow of display trajectory information calculation processing.
FIG. 7 is a first diagram showing an overview of the display trajectory information calculation process.
Next, a calculation example of display trajectory information when the display trajectory is a trajectory that connects the display positions of the sample points to a straight line will be described with reference to FIGS. 6 and 7.
First, the motion expression information generation unit 17 receives from the reproduction processing unit 16 information on the display position of each sample point, the set of elapsed time from the reproduction start time at the display position, and the ID of the 3D metadata to be synthesized and displayed. (Step S301). Then, the motion expression information generation unit 17 displays the display position of the sample point P1 having the earliest elapsed time from the playback start time and the sample point having the earliest elapsed time from the playback start time among the display positions of the respective sample points. Each coefficient of the linear equation (linear equation in the virtual space of the 3D moving image) between two points with the display position of P2 is obtained. Similarly, the motion expression information generation unit 17 obtains each coefficient of a linear equation between the display position of the sample point P2 and the display position of the sample point P3 that has elapsed from the playback start time next. Then, for each set of two display positions whose elapsed times are close, linear coefficients connecting the two points are obtained (step S302). Each linear coefficient is used as display trajectory information.

  Then, when the calculation of the display trajectory information is performed at the time of reproduction of the 3D moving image or immediately before the reproduction, the motion expression information generation unit 17 temporarily displays the display trajectory information indicating each coefficient of each generated linear expression. (Step S303), and notifies the reproduction processing unit 16 of completion of calculation of the display trajectory information (step S304). When the calculation of the display trajectory information is performed at the time of creating the 3D metadata, the motion expression information generation unit 17 transmits the generated display trajectory information to the 3D metadata generation / editing unit 19, and the 3D metadata generation / editing unit 19 registers the display trajectory information in the 3D metadata DB 14 in association with the 3D metadata ID. Then, the motion expression information generation unit 17 sets a set of the display position of each sample point of the 3D metadata to be synthesized and displayed and the elapsed time from the reproduction start time at the display position when the reproduction processing unit 16 reproduces the 3D moving image. Based on the display trajectory information of the 3D metadata, a display position corresponding to the elapsed time from the playback start time of the 3D moving image is calculated (step S305), and an instruction to display the 3D metadata at the display position is issued. The processing to be performed is performed on the reproduction processing unit 16 (step S306). As a result, the reproduction processing unit 16 reproduces the 3D moving image and performs a process of combining and displaying the 3D metadata at the designated display position on the 3D moving image (step S307).

FIG. 8 is a second diagram showing a processing flow of the display trajectory information calculation processing.
FIG. 9 is a second diagram showing an overview of the display trajectory information calculation process.
Next, an example of calculating the display trajectory information when the display trajectory is a trajectory connecting the display positions of the sample points by spline curves will be described with reference to FIGS.
The motion expression information generation unit 17 receives from the reproduction processing unit 16 information on the display position of each sample point and the set time elapsed from the reproduction start time at the display position and the ID of the 3D metadata to be displayed in combination (step S401). ). Then, the motion expression information generation unit 17 obtains a calculation formula for a spline curve that connects the display positions of all the sample points with a smooth curve, further obtains a coefficient for each calculation formula, and obtains each coefficient for the expression representing this spline curve. Display trajectory information is set (step S402). Here, in the present embodiment, the expression of the spline curve is expressed by a cubic function of the elapsed time t from the reproduction start time as shown below.
x = a ₁ t ³ + b ₁ t ² + c ₁ t + d ₁
y = a ₂ t ³ + b ₂ t ² + c ₂ t + d ₂
z = a ₃ t ³ + b ₃ t ² + c ₃ t + d ₃

  When the calculation of the display trajectory information is performed at the time of playback of the 3D moving image or immediately before the playback, the motion expression information generation unit 17 temporarily displays the display trajectory information indicated by each coefficient of each generated linear expression. (Step S403), and notifies the reproduction processing unit 16 of completion of calculation of the display trajectory information (step S404). When the calculation of the display trajectory information is performed at the time of creating the 3D metadata, the motion expression information generation unit 17 transmits the generated display trajectory information to the 3D metadata generation / editing unit 19, and the 3D metadata generation / editing unit 19 registers the display trajectory information in the 3D metadata DB 14 in association with the 3D metadata ID. Then, the motion expression information generation unit 17 sets a set of the display position of each sample point of the 3D metadata to be synthesized and displayed and the elapsed time from the reproduction start time at the display position when the reproduction processing unit 16 reproduces the 3D moving image. Based on the display trajectory information of the 3D metadata, a display position corresponding to the elapsed time from the playback start time of the 3D moving image is calculated (step S405), and an instruction to display the 3D metadata at the display position is given. Processing to be performed is performed on the reproduction processing unit 16 (step S406). Thereby, the reproduction processing unit 16 reproduces the 3D moving image and performs a process of combining and displaying the 3D metadata on the 3D moving image (step S407).

FIG. 10 is a third diagram illustrating the processing flow of the display trajectory information calculation processing.
FIG. 11 is a third diagram showing an outline of the calculation process of the display trajectory information.
Next, a calculation example of display trajectory information when the display trajectory is a fixed trajectory template specified based on the display position of each sample point will be described with reference to FIGS. 10 and 11.
In this case, it is assumed that the motion expression information generation unit 17 stores data of a fixed trajectory template in advance. First, the motion expression information generation unit 17 receives the display position of each sample point from the reproduction processing unit 16 (step S501). Then, the motion expression information generation unit 17 matches the center of gravity of the display position of each sample point with the center of gravity of the standard trajectory template, and whether the display position of each sample point is within a certain distance from the trajectory indicated by the standard trajectory template. It is determined whether or not (step S502). Then, it is determined whether or not the display position of each sample point is within a certain distance from the locus indicated by the fixed locus template using a plurality of fixed locus templates, and all the display positions of each sample point are within a certain distance. The coefficient of the formula of the standard trajectory template determined to be specified as display trajectory information calculated based on the display position of each sample point (step S503). The standard trajectory template includes a circle, an ellipse, a triangle represented by three straight lines, a quadrangle represented by four straight lines, a sine and cosine curve, etc. in the virtual space, and the trajectory is an elapsed time from the reproduction start time. It is represented by a function of t. When a plurality of standard trajectory templates can be specified, for example, the standard trajectory template when the average of the distances from the standard trajectory template of the display position of each sample point is the smallest is specified.

  When the calculation of the display trajectory information is calculated during or immediately before the playback of the 3D moving image, the motion expression information generation unit 17 temporarily stores the specified display trajectory information in a memory or the like (step S504). ), The reproduction processing unit 16 is notified of the completion of calculation of the display trajectory information (step S505). If the calculation of the display trajectory information is performed at the time of creating the 3D metadata, the motion expression information generating unit 17 transmits the specified display trajectory information to the 3D metadata generation / editing unit 19, and the 3D metadata generation / editing unit 19 registers the display trajectory information in the 3D metadata DB 14 in association with the 3D metadata ID. Then, the motion expression information generation unit 17 sets a set of the display position of each sample point of the 3D metadata to be synthesized and displayed and the elapsed time from the reproduction start time at the display position when the reproduction processing unit 16 reproduces the 3D moving image. Based on the display trajectory information of the 3D metadata, a display position corresponding to the elapsed time from the playback start time of the 3D moving image is calculated (step S506), and an instruction to display the 3D metadata at the display position is given. The processing to be performed is performed on the reproduction processing unit 16 (step S507). As a result, the reproduction processing unit 16 reproduces the 3D moving image and performs a process of combining and displaying the 3D metadata on the 3D moving image (step S508).

FIG. 12 is a fourth diagram illustrating a processing flow of the display trajectory information calculation processing.
FIG. 13 is a fourth diagram showing an overview of the display trajectory information calculation process.
Next, an example in which the display trajectory information is calculated based on the approximate position of each sample point based on the display position of each sample point will be described with reference to FIGS.
The motion expression information generation unit 17 receives from the reproduction processing unit 16 information on the display positions P1 to P17 of each sample point and the elapsed time from the reproduction start time at the display position and the ID of the 3D metadata to be synthesized and displayed. (Step S601). Then, the motion expression information generation unit 17 includes a plurality of continuous display positions P1 to P17 among the display positions of the sample points, and the start display position P1 and the start point P1 of the plurality of continuous display positions P1 to P17. It is determined in order from the display position P3 next to the display position P2 of the next point whether it is located within the virtual cylinder D1 having a predetermined radius centered on the straight line extending to the display position P2 of the next point (step) In step S602), display trajectory information including a coefficient of an expression representing a straight line T1 connecting the display position P6 immediately before the display position P7 determined to be not located in the virtual cylinder D1 and the display position P1 of the start point is calculated. (Step S603). Then, it is determined whether the display positions of all the sample points are included in the straight line indicated by the display trajectory information (step S604). If not included, the display position that is not included in the straight line of the display trajectory information is the most. The display position of the sample point indicating the point displayed at the elapsed time from the early reproduction start time is set as the start point (step S605), and the above-described display trajectory information calculation process in steps S602 and S603 is repeated.

That is, the motion expression information generation unit 17 repeats the calculation of the display trajectory information in the same manner, starting from the display position P6 immediately before the display position P7 determined not to be positioned for the first time in the virtual cylinder D1. That is, whether or not the position is within the virtual cylinder D2 having a predetermined radius centered on a straight extension line connecting the display position P6 of the start point and the display position P7 of the next point of the start point P6. A coefficient of an expression representing a straight line T2 that is determined in order from the next display position P8 and connects the display position P7 immediately before the display position P8 that is first determined not to be located in the virtual cylinder D2 and the display position P6 of the start point. Display trajectory information consisting of
Similarly, whether or not it is located within a virtual cylinder D3 having a predetermined radius centered on a straight extension line connecting the display position P7 determined to be the next start point and the display position P8 of the next point of the start point P7, The next display position P8 of the next point is sequentially determined from the display position P9, and the display position P8 immediately before the display position P9 determined for the first time not to be located in the virtual cylinder D3 is connected to the display position P7 of the start point. Display trajectory information including the coefficient of the equation representing the straight line T3 is calculated.
Similarly, whether or not it is located within a virtual cylinder D4 having a predetermined radius centered on a straight extension line connecting the display position P8 determined next as the start point and the display position P9 of the next point of the start point P8, The next display position P9 is determined in order from the next display position P10, and the display position P9 immediately before the display position P10 determined for the first time not to be located in the virtual cylinder D4 is connected to the start display position P8. Display trajectory information composed of coefficients of an expression representing the straight line T4 is calculated.

Similarly, whether or not it is located within a virtual cylinder D5 having a predetermined radius centered on a straight extension line connecting the display position P9 determined next as the start point and the display position P10 of the next point of the start point P9, The next display position P10 of the next point is sequentially determined from the display position P11, and the display position P11 immediately before the display position P12 determined for the first time not to be located within the virtual cylinder D5 is connected to the display position P9 of the start point. Display trajectory information including the coefficient of the equation representing the straight line T5 is calculated.
Similarly, whether or not it is located within a virtual cylinder D6 having a predetermined radius centered on a straight extension line connecting the display position P11 determined as the next start point and the display position P12 of the next point of the start point P11. The next display position P12 of the next point is sequentially determined from the display position P13, and the display position P14 immediately before the display position P15 that is determined for the first time not to be located in the virtual cylinder D6 is connected to the display position P11 of the start point. Display trajectory information including the coefficient of the equation representing the straight line T6 is calculated.
Similarly, whether or not it is located within a virtual cylinder D7 having a predetermined radius centered on a straight extension line connecting the display position P14 determined next as the start point and the display position P15 of the next point of the start point P14, Since the display position P16 is determined in order from the display position P16 next to the display position P15 of the next point and there is no display position up to P17 in the virtual cylinder D7, a straight line T7 connecting the display position P17 and the display position P14 of the start point is obtained. Display trajectory information composed of coefficients of the expression to be expressed is calculated.

  When the calculation of the display trajectory information is calculated at the time of playback of the 3D moving image or immediately before the playback, the motion expression information generation unit 17 temporarily stores the calculated display trajectory information (step S606). The reproduction processing unit 16 is notified of the completion of calculation of the display trajectory information (step S607). When the display trajectory information is calculated when the 3D metadata is created, the motion expression information generation unit 17 transmits the calculated display trajectory information to the 3D metadata generation / editing unit 19, and the 3D metadata generation / editing unit The unit 19 registers the display trajectory information in the 3D metadata DB 14 in association with the 3D metadata ID. Then, the motion expression information generation unit 17 sets a set of the display position of each sample point of the 3D metadata to be synthesized and displayed and the elapsed time from the reproduction start time at the display position when the reproduction processing unit 16 reproduces the 3D moving image. Based on the display trajectory information of the 3D metadata, a display position corresponding to the elapsed time from the playback start time of the 3D moving image is calculated (step S608), and an instruction to display the 3D metadata at the display position is issued. The processing to be performed is performed on the reproduction processing unit 16 (step S609). As a result, the reproduction processing unit 16 reproduces the 3D moving image and performs a process of combining and displaying the 3D metadata on the 3D moving image (step S610).

FIG. 14 is a diagram showing a processing flow of 3D metadata composition display.
FIG. 15 is a diagram showing an outline of processing of 3D metadata composition display.
Next, 3D metadata composition display processing on a 3D moving image using display trajectory information will be described with reference to FIGS. 14 and 15.
When reproducing the 3D moving image, the reproduction processing unit 16 associates the 3D metadata recorded in the 3D metadata DB 14 with the ID of the 3D moving image and the ID of the 3D metadata. Read (step S701). The reproduction processing unit 16 notifies the motion expression information generation unit 17 of the ID of the 3D metadata to be synthesized and displayed and inquires about the display position of the 3D metadata (step S702). In the case of the embodiment in which the display trajectory information is registered in a pre-calculated sample point information table (FIG. 3), the reproduction processing unit 16 uses the sample point information table (based on the 3D metadata ID ( The display trajectory information is acquired from FIG. 3) and transmitted to the motion expression information generation unit 17, and the motion expression information generation unit 17 temporarily stores the ID of the 3D metadata and the display trajectory information in association with each other.

  When the motion expression information generation unit 17 receives the notification of the ID of the 3D metadata, the display position of each sample point stored in association with the ID, the elapsed time from the reproduction start time at the display position, and the display locus Information is read from the storage unit in the processing unit (step S703). Then, the current elapsed time from the reproduction start time is obtained from the reproduction processing unit 16 (step S704), and 2 indicating the elapsed time closest to the current elapsed time based on the elapsed time from the reproduction start time of each sample point. The display position of one sample point is specified (step S705). Then, the ratio w of the current elapsed time in the difference between the elapsed times of the two specified sample points is obtained (step S706). Then, the position of the ratio w in the trajectory indicated by the display trajectory information passing through the two specified sample points is calculated as the display position of the 3D metadata at the current elapsed time (step S707), and the display position is reproduced. The unit 16 is notified (step S708). The reproduction processing unit 16 synthesizes and displays 3D metadata at the display position on the 3D moving image received from the motion expression information generation unit 17 (step S709). By repeating this process every predetermined time, a moving image in which 3D metadata moves on the 3D moving image can be displayed on the 3D display 15.

FIG. 16 is a first diagram showing a processing flow of editing 3D metadata during moving image reproduction.
FIG. 17 is a first diagram illustrating an overview of 3D metadata editing processing during moving image playback.
Next, processing in which the user edits 3D metadata in a situation where 3D moving images and 3D metadata are being reproduced will be described. Here, editing of 3D metadata means correcting a display trajectory in which 3D metadata moves on a 3D moving image.
First, the reproduction processing unit 16 reproduces a 3D moving image (step S801), and 3D metadata is synthesized and displayed on the moving image (step S802). In such a situation, the user operates the spatial mouse 11 to move the mouse pointer displayed on the 3D display of the spatial mouse 11, matches the display position of the 3D metadata, and further presses the selection button. To do. Then, the spatial mouse control unit 12 receives a 3D metadata selection event from the spatial mouse 11 (step S803), and sends a selection start notification triggered by the reception of the selection event to the sample point generation unit 18 (step S804). When the sample point generation unit 18 receives the selection start notification, the sample point generation unit 18 compares the current display position of the mouse pointer with the display position of the 3D metadata to determine whether the mouse pointer has selected the 3D metadata ( Step S805). In this determination, a process for determining that the 3D metadata is selected when the display position of the mouse pointer matches the display position of the 3D metadata, and the display position of the mouse pointer is within a certain distance from the display position of the 3D metadata. If it is within a certain distance, one of the processes for determining that the 3D metadata is selected is performed. If the processing of determining that the 3D metadata is selected if the display position of the mouse pointer is within a certain distance from the display position of the 3D metadata, the spatial mouse 11 aligns the mouse pointer with the 3D metadata. Even if it is difficult, the user can easily select.

  If the sample point generator 18 determines that the 3D metadata is selected by the mouse pointer, the sample point generator 18 notifies the 3D metadata generator / editor 19 of the ID of the 3D metadata. Then, the 3D metadata generation / editing unit 19 waits until it receives a movement request for the selected 3D metadata. When the user wants to correct the display trajectory of the 3D metadata selected using the spatial mouse 11 on the 3D moving image, the user moves the mouse pointer by operating the spatial mouse while selecting the 3D metadata. When the spatial mouse control unit 12 detects the movement amount of the 3D metadata by the operation of the spatial mouse 11 (step S806), the spatial mouse control unit 12 notifies the reproduction processing unit 16 and the sample point generation unit 18 of the movement amount. When the reproduction processing unit 16 receives the notification of the movement amount, the reproduction processing unit 16 moves and displays the sample point displayed on the 3D moving image based on the movement amount, and is selected along with the movement display of the sample point. The 3D metadata is also moved and displayed (step S807). Further, the sample point generation unit 18 generates a movement request storing the movement amount of the 3D metadata displayed by the reproduction processing unit 16 on the 3D moving image and the ID of the 3D metadata, and the 3D metadata generation editing unit 19. Send to. Then, the 3D metadata generation / editing unit 19 temporarily stores the movement amount of the received 3D metadata and transfers the movement request to the motion expression information generation unit 17.

  When the movement expression information generation unit 17 receives the movement request, the movement amount indicated by the movement request is detected, and the display trajectory information is recalculated to change the entire display trajectory of the 3D metadata by the detected movement amount. Is performed (step S808). Here, the moving amount is indicated by each moving amount of (x, y, z) in the virtual space of the 3D moving image. Therefore, the motion expression information generation unit 17 identifies display trajectory information from the ID of the 3D metadata indicated by the movement request, and converts the mathematical expression representing the trajectory indicated by the display trajectory information by the amount indicated by the movement amount. Then, each coefficient of the mathematical expression representing the display trajectory after the movement is calculated, and the display trajectory information indicating the calculated coefficient is stored in association with the 3D metadata ID as new display trajectory information (step S809). The motion expression information generation unit 17 calculates the display position of the 3D metadata in the elapsed time from the reproduction start time based on the new display trajectory information after recalculation (step S810), and notifies the reproduction processing unit 16 of the display position. As a result, the reproduction processing unit 16 synthesizes and displays the 3D metadata of the new display locus on the 3D moving image (step S811). In addition, the motion expression information generation unit 17 transmits new display trajectory information after recalculation to the 3D metadata generation / editing unit 19. When the 3D metadata generation / editing unit 19 receives a notification of editing completion from the spatial mouse control unit 12 via the sample point generation unit 18 (for example, the 3D metadata generation / editing unit 19 selected by the user operating the spatial mouse 11). This event is notified to the spatial mouse control unit 12 by clicking on the 3D moving image other than the data), and the new display trajectory information after the recalculation is associated with the ID of the selected 3D metadata. It records in the sample point information table in metadata DB14 (step S812).

FIG. 18 is a second diagram illustrating a processing flow of editing 3D metadata during moving image reproduction.
FIG. 19 is a second diagram showing an overview of 3D metadata editing processing during moving image playback.
In the processing in steps S801 to 8012 described above, an example in which the entire display trajectory of the 3D metadata is moved and new display trajectory information is recalculated according to the movement amount of the selected 3D metadata has been described. However, an instruction to change the display position of a sample point having the selected 3D metadata may be received, and new display trajectory information in which only the display position of the sample point is changed may be recalculated.

  That is, in this case, Steps S801 to S805 are performed as described above. When the user wants to correct the display trajectory of the 3D metadata selected by the user using the spatial mouse 11 on the 3D moving image, the user operates the spatial mouse while selecting a sample point in the display trajectory of the 3D metadata. Move the mouse pointer. When the spatial mouse control unit 12 detects the movement amount due to the operation of the spatial mouse 11 (step S906), it notifies the reproduction processing unit 16 and the sample point generation unit 18 of the movement amount. Upon receiving the movement amount notification, the reproduction processing unit 16 moves and displays the sample points displayed on the 3D moving image based on the movement amount (step S907). In addition, the sample point generation unit 18 outputs a movement request storing the movement amount of the sample point on the 3D moving image displayed by the reproduction processing unit 16 and the ID of the 3D metadata corresponding to the sample point to the 3D metadata. The data is transmitted to the generation / editing unit 19. Then, the 3D metadata generation / editing unit 19 temporarily stores the movement amount of the received 3D metadata, and transfers a movement request indicating the movement amount to the motion expression information generation unit 17.

  When the movement expression information generation unit 17 receives the movement request, the movement expression indicated by the movement request is detected, and the movement trajectory at the moved sample point in the display trajectory of the 3D metadata is detected by the detected movement amount. The recalculation process of the display trajectory information to be changed is performed (step S908). Here, the moving amount is indicated by each moving amount of (x, y, z) in the virtual space of the 3D moving image. Therefore, the motion expression information generation unit 17 specifies display trajectory information from the ID of the 3D metadata indicated by the movement request, and among the sample points indicated by the display trajectory information, the display position of the moved sample point is determined. Then, each coefficient of the mathematical expression representing the display trajectory is calculated, and the display trajectory information indicating the calculated coefficient is stored in association with the 3D metadata ID as new display trajectory information (step S909). The motion expression information generation unit 17 calculates the display position of the 3D metadata at the current elapsed time from the reproduction start time based on the new display trajectory information after recalculation (step S910). By receiving the notification, the reproduction processing unit 16 composites and displays the 3D metadata of the new display locus on the 3D moving image (step S911). In addition, the motion expression information generation unit 17 transmits new display trajectory information after recalculation to the 3D metadata generation / editing unit 19. When the 3D metadata generation / editing unit 19 receives a notification of editing completion from the spatial mouse control unit 12 via the sample point generation unit 18, the 3D metadata generation / editing unit 19 (for example, the 3D metadata selected by the user operating the spatial mouse 11). This event is notified to the spatial mouse control unit 12 by clicking on the 3D moving image other than the data), and the new display trajectory information after the recalculation is associated with the ID of the selected 3D metadata. It records in the sample point information table in metadata DB14 (step S912).

As described above, the embodiment of the present invention has been described. According to the above-described processing, 3D metadata to be synthesized and displayed on a 3D moving image can be easily created using a space mouse (up, down, left, and right of the space mouse in space). This can be done only by moving back and forth.
In addition, according to the present invention, at the time of creating 3D metadata, not all the positions on the 3D moving image designated by the spatial mouse are held as 3D metadata display trajectory information, and only a plurality of discrete sample points are stored. Is stored as trajectory information, so that data to be stored can be reduced.
In addition, according to the present invention, editing of the display trajectory of 3D metadata can be performed by an easy operation using a space mouse (only movement of the space mouse up and down, left and right and back and forth in space).
Further, according to the present invention, when a user selects a display track when editing a display track of 3D metadata, the 3D metadata is selected within a certain range without directly specifying the 3D metadata. Therefore, it is possible to easily select 3D metadata for editing.
In addition, since a fixed template, a moving speed and a moving direction of each 3D metadata display trajectory can be stored as motion expression information, search of 3D metadata displayed in the 3D moving image is performed. When the function is provided, it is possible to perform processing for searching for 3D metadata displayed on the 3D moving image by designating the template, the moving speed, and the moving direction as the data to be searched. Become.
Further, according to the present invention, when the user edits the 3D metadata displayed at the time of playback of the 3D moving image, the display trajectory after the editing is immediately calculated, and the composite display of the 3D metadata is performed based on the display trajectory. Therefore, 3D metadata can be edited in real time.
Further, according to the present invention, since the display position of the 3D metadata is automatically specified by moving the spatial mouse when the user creates the 3D metadata, the display locus of the 3D metadata is easily input. be able to.

  Note that the above-described three-dimensional moving image reproduction apparatus has a computer system therein. Each process described above is stored in a computer-readable recording medium in the form of a program, and the above process is performed by the computer reading and executing the program. Here, the computer-readable recording medium means a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, or the like. Alternatively, the computer program may be distributed to the computer via a communication line, and the computer that has received the distribution may execute the program.

  The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

It is a block diagram which shows the structure of a three-dimensional moving image reproducing device. It is a figure which shows 3D metadata information table. It is a figure which shows a sample point information table. It is a figure which shows the processing flow at the time of preparation of 3D metadata. It is a figure which shows the processing flow of a composite display of 3D metadata. It is a 1st figure which shows the processing flow of a display trace information calculation process. It is a 1st figure which shows the outline | summary of a display trace information calculation process. It is a 2nd figure which shows the processing flow of a display trace information calculation process. It is a 2nd figure which shows the outline | summary of a display trace information calculation process. It is a 3rd figure which shows the processing flow of a display trace information calculation process. It is a 3rd figure which shows the outline | summary of the calculation process of display locus information. It is a 4th figure which shows the processing flow of a display locus | trajectory information calculation process. It is a 4th figure which shows the outline | summary of a display trace information calculation process. It is a figure which shows the processing flow of 3D metadata synthetic | combination display. It is a figure which shows the process outline | summary of 3D metadata synthetic | combination display. It is a 1st figure which shows the processing flow of edit of 3D metadata. It is a 1st figure which shows the outline | summary of the edit process of 3D metadata. It is a 2nd figure which shows the processing flow of edit of 3D metadata. It is a 2nd figure which shows the outline | summary of the edit process of 3D metadata.

Explanation of symbols

11 ... Spatial mouse 12 ... Spatial mouse control unit 13 ... 3D animation DB
14 ... 3D metadata DB
DESCRIPTION OF SYMBOLS 15 ... 3D display 16 ... Reproduction | regeneration processing part 17 ... Motion expression information generation part 18 ... Sample point generation part 19 ... 3D metadata production | generation edit part

Claims (17)

A three-dimensional moving image reproducing device for reproducing and outputting a three-dimensional moving image stored in advance to a three-dimensional moving image display means,
A designated position detecting means for detecting a position of the spatial input device existing in the real space and detecting a designated position by the spatial input device in the three-dimensional moving image according to a change in the position;
Among the designated positions by the spatial input device, the display position of the additional information in the three-dimensional moving image is either the position specifying information received from the spatial input device or the designated position by the spatial input device at a predetermined time interval. A plurality of additional information display position detecting means for detecting the
A display time for detecting an elapsed time from the start time for displaying the additional information in the three-dimensional moving image based on a start time of the three-dimensional moving image and a time at which a display position of the additional information is detected. Detection means;
Display attribute information including the display position of the additional information and the elapsed time from the start time of the three-dimensional moving image for displaying the additional information at the display position is displayed for each display position of the additional information. A plurality of display attribute information recording means for recording on the means;
A three-dimensional moving image reproduction apparatus comprising: Display trajectory calculation means for calculating display trajectory information for specifying a display trajectory connecting the plurality of display positions from the plurality of display positions of the additional information, and
The display attribute information recording unit records a plurality of the display attribute information further including the calculated display trajectory information in a display attribute information storage unit for each display position of the additional information. 3D video playback device. The three-dimensional moving image reproducing apparatus according to claim 2, wherein the display trajectory information is each coefficient of an expression representing each straight line connecting the plurality of display positions of the additional information. The three-dimensional moving image reproduction apparatus according to claim 2, wherein the display trajectory information is each coefficient of an expression representing a spline curve connecting the plurality of display positions of the additional information. A plurality of continuous display positions among the display positions of the additional information in the detected three-dimensional moving image are straight lines connecting the display position of the start point and the display position of the next point. One of the display positions that are determined in order from the display position next to the display position of the next point in order to determine whether they are located within a virtual cylinder with a predetermined radius centered on a line, and are determined not to be located in the virtual cylinder for the first time Display trajectory calculation means for calculating display trajectory information consisting of coefficients of an expression representing a straight line connecting the previous display position and the display position of the start point, and
The display trajectory calculation means repeatedly calculates the display trajectory information starting from the display position immediately before the display position determined not to be positioned for the first time in the virtual cylinder,
The display attribute information recording unit records a plurality of the display attribute information further including the calculated display trajectory information in a display attribute information storage unit for each display position of the additional information. 3D video playback device. A trajectory template specifying means for comparing a display trajectory connecting the plurality of display positions of the additional information, which is represented by the display trajectory information, with a predetermined trajectory template and specifying a matching trajectory template; Prepared,
The display attribute information recording unit records a plurality of the display attribute information further including the trajectory template in a display attribute information storage unit for each display position of the additional information. The three-dimensional moving image reproducing device according to claim 5. Based on the two consecutive display positions of the plurality of detected display positions of the additional information in the three-dimensional moving image and the display times of the display positions, the additional information at the two consecutive display positions is displayed. Movement information calculation means for calculating a movement speed and a movement direction;
The display attribute information recording unit records a plurality of the display attribute information further including the calculated moving speed and moving direction in a display attribute information storage unit for each display position of the additional information. The three-dimensional moving image reproduction device according to claim 1 or claim 2 or claim 5. 3D moving image reproduction means for reproducing and outputting the 3D moving image to 3D moving image display means;
Additional information display means for reading the display attribute information of the additional information from the display attribute information storage means, and displaying the additional information in the three-dimensional moving image based on the display attribute information;
Additional information selection presence / absence determining means for determining whether the additional information is selected based on a designated position by the spatial input device;
Designated position movement detecting means for detecting movement of the designated position by the spatial input device;
When the movement of the designated position is detected while the additional information is selected, the display trajectory information after the movement of the additional information displayed in the three-dimensional moving image is displayed on the additional information. Trajectory information, and trajectory recalculation means for recalculating based on the movement amount of the specified position,
The said additional information display means displays the said additional information in the said three-dimensional moving image based on the display locus | trajectory information after the said movement. The Claim 3 characterized by the above-mentioned. Dimensional video playback device. A processing method in a three-dimensional moving image reproducing apparatus for reproducing and outputting a three-dimensional moving image stored in advance to a three-dimensional moving image display means,
The designated position detecting means of the three-dimensional moving image reproducing device detects the position of the spatial input device existing in real space, and determines the designated position by the spatial input device in the three-dimensional moving image according to the change of the position. Detect
Position specifying information received from the spatial input device by the additional information display position detecting means of the three-dimensional moving image reproduction device, of the positions specified by the spatial input device, the display position of the additional information in the three-dimensional moving image. Or a plurality of detections based on any of the designated positions by the spatial input device at a predetermined time interval,
Based on the start time of the three-dimensional moving image and the time when the display position of the additional information is detected by the display time detecting means of the three-dimensional moving image reproduction device, the additional information is displayed in the three-dimensional moving image. Detect the elapsed time from the start time to display,
The display attribute information recording means of the three-dimensional moving image reproduction apparatus includes the display position of the additional information and the elapsed time from the start time of the three-dimensional moving image that displays the additional information at the display position. A plurality of pieces of information are recorded in the display attribute information storage unit for each display position of the additional information. The display trajectory calculating means of the three-dimensional moving image reproduction device calculates display trajectory information for specifying a display trajectory connecting the plurality of display positions from the plurality of display positions of the additional information,
The display attribute information recording unit of the three-dimensional moving image reproduction device records a plurality of the display attribute information further including the calculated display trajectory information in a display attribute information storage unit for each display position of the additional information. The processing method according to claim 9, wherein the processing method is characterized. The display trajectory calculating means of the three-dimensional moving image reproduction apparatus has a plurality of continuous display positions among the display positions of the additional information in the plurality of detected three-dimensional moving images as starting points of the plurality of continuous display positions. It is determined in order from the display position next to the display position of the next point whether it is located in a virtual cylinder having a predetermined radius centered on a straight line extending from the display position to the display position of the next point. Calculating display trajectory information composed of coefficients of a formula representing a straight line connecting the display position immediately before the display position determined not to be located in the cylinder for the first time and the display position of the start point;
The display trajectory calculating means of the three-dimensional moving image reproduction device repeatedly calculates the display trajectory information starting from the display position immediately before the display position determined not to be positioned for the first time in the virtual cylinder,
The display attribute information recording unit of the three-dimensional moving image reproduction device records a plurality of the display attribute information further including the calculated display trajectory information in a display attribute information storage unit for each display position of the additional information. The processing method according to claim 9, wherein the processing method is characterized. The trajectory template specifying means of the three-dimensional moving image reproducing device compares a display trajectory connecting the plurality of display positions of the additional information represented by the display trajectory information with a predetermined trajectory template, Identify matching trajectory templates,
The display attribute information recording unit of the three-dimensional moving image reproduction apparatus records a plurality of the display attribute information further including the trajectory template in a display attribute information storage unit for each display position of the additional information. The processing method according to claim 9. The movement information calculation means of the three-dimensional moving image reproduction apparatus is based on two consecutive display positions among the display positions of the additional information in the plurality of detected three-dimensional moving images and the display times of the display positions. , Calculating the moving speed and moving direction of the additional information at each of the two consecutive display positions,
The display attribute information recording unit of the three-dimensional moving image reproduction apparatus records a plurality of the display attribute information further including the calculated moving speed and moving direction in the display attribute information storage unit for each display position of the additional information. The processing method according to any one of claims 9 to 11, wherein: Three-dimensional moving image reproducing means of the three-dimensional moving image reproducing device reproduces and outputs the three-dimensional moving image to a three-dimensional moving image display means;
The additional information display unit of the three-dimensional moving image reproduction apparatus reads the display attribute information of the additional information from the display attribute information storage unit, and the additional information is included in the three-dimensional moving image based on the display attribute information. Display
The additional information selection presence / absence determining means of the three-dimensional moving image reproduction device determines whether the additional information is selected based on a designated position by the spatial input device,
The designated position movement detecting means of the three-dimensional moving image reproducing device detects the movement of the designated position by the spatial input device,
When the trajectory recalculation unit of the three-dimensional moving image reproduction apparatus detects the movement of the designated position while the additional information is selected, the additional information displayed in the three-dimensional moving image is displayed. Recalculate the display trajectory information after the movement based on the display trajectory information related to the additional information and the amount of movement of the designated position,
11. The additional information display unit of the three-dimensional moving image reproduction device displays the additional information in the three-dimensional moving image based on the display trajectory information after the movement. 14. The processing method according to any one of items 13. A computer of a three-dimensional moving image reproducing apparatus for reproducing and outputting a three-dimensional moving image stored in advance to a three-dimensional moving image display means,
A designated position detecting means for detecting a position of the spatial input device existing in the real space and detecting a designated position by the spatial input device in the three-dimensional moving image according to a change in the position;
Among the designated positions by the spatial input device, the display position of the additional information in the three-dimensional moving image is either the position specifying information received from the spatial input device or the designated position by the spatial input device at a predetermined time interval. A plurality of additional information display position detecting means for detecting the
A display time for detecting an elapsed time from the start time for displaying the additional information in the three-dimensional moving image based on a start time of the three-dimensional moving image and a time at which a display position of the additional information is detected. Detection means,
Display attribute information including the display position of the additional information and the elapsed time from the start time of the three-dimensional moving image for displaying the additional information at the display position is displayed for each display position of the additional information. A plurality of display attribute information recording means for recording on the means;
Program to function as. Said computer further
Display trajectory calculation means for calculating display trajectory information for specifying a display trajectory connecting the plurality of display positions from the plurality of display positions of the additional information;
Function as
The display attribute information recording unit records a plurality of the display attribute information further including the calculated display trajectory information in a display attribute information storage unit for each display position of the additional information. Program. Said computer further
3D moving image reproduction means for reproducing and outputting the 3D moving image to 3D moving image display means;
Additional information display means for reading the display attribute information of the additional information from the display attribute information storage means and displaying the additional information in the three-dimensional moving image based on the display attribute information;
Additional information selection presence / absence determining means for determining whether the additional information is selected based on a designated position by the spatial input device;
Designated position movement detecting means for detecting movement of the designated position by the spatial input device;
When the movement of the designated position is detected while the additional information is selected, the display trajectory information after the movement of the additional information displayed in the three-dimensional moving image is displayed on the additional information. Trajectory recalculation means for recalculating based on trajectory information and the amount of movement of the specified position,
Function as
The program according to claim 16, wherein the additional information display unit displays the additional information in the three-dimensional moving image based on the display trajectory information after the movement.

Images