JP2002335442A - Scene description program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a scene description program that satisfies functions of producing a video object and can be applicable to production for broadcast purpose video images. SOLUTION: The scene description program functionally realizes a computer by means of each program of the scene description program that is means such as; a display means 5a that displays a video object of an optional shape being a component of each scene; a frame unit description means 5c that describes a display start time/end time of the video object of an optional shape in the unit of frames; and an object edit means 5g that edits the video object of the optional shape in the unit of frames described by the frame unit description means 5c.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a scene description program that defines (describes) how a part (video object) such as a person or a background behaves on a scene (scene).

[0002]

2. Description of the Related Art In recent years, people, objects, backgrounds, and the like are regarded as "parts" (hereinafter referred to as video objects) constituting one screen, and the video objects are extracted from the screen (video object extraction) and extracted. An object-based coding technique has been proposed in which each video object is encoded, the encoded video object is stored and transmitted, decoded, decoded, and then video synthesized. In this object encoding technique, when a video object is synthesized with an extracted video object, a program language called a “scene description language” (hereinafter, referred to as a scene description program) that describes information for returning the video object to its original location. ).

[0003] This scene description program is famously known as BIFS (Binary Format for
r Scenes), SMIL (Synchroniz)
ed Multimedia Integration
Language), BML (Broadcast)
Markup Language), EDL (Edit
Decision Lists).

[0004] Briefly describing each of them, BIFS is a VRML (Virtual Reality Mode).
MPEG-4 based on Ling Language)
Is a scene description language. It has a rich description function of 3D-CG and 2D-CG, and can handle video of any shape. SMIL is an interactive presentation description language. It is mainly used for describing Web sites, and can handle only rectangular video. BML is a multimedia coding system for digital broadcasting, and depends on the hardware of the receiver. Therefore, the function is restricted depending on the hardware, but BML is currently the only practical scene description language. . EDL is a list of edit points of a VTR and is very primitive and cannot be called a scene description language strictly, but is the only general-purpose scene description method currently used in video production. .

[0005]

However, the above-described scene description program has the following problems. (1) BIFS was able to manage the display time (start / end time) of a video object only at millisecond intervals. (2) SMIL and BML could not handle video objects of arbitrary shape, and could only manage the display time of video objects at millisecond intervals. (3) EDL was of arbitrary shape. The video objects cannot be handled, and the spatial arrangement of the video objects cannot be changed.

[0006] Further, in the existing scene description program, since a video object is described as a completed material, it is not possible to perform advanced processing of the video object, and it is only possible to display a given video scene as it is. Did not. That is, the existing scene description programs have disadvantages, and lack a function of creating another video object by combining video objects, and for example, it has been difficult to apply the present invention to video production for broadcasting. .

[0007] An object of the present invention is to provide a scene description program which solves the above-mentioned problems of the prior art and has a function of producing a video object, and which can be applied to, for example, video production for broadcasting. It is in.

[0008]

According to a first aspect of the present invention, there is provided a scene description program, comprising: display means for displaying an image object having an arbitrary shape which constitutes each scene; This is for causing a computer to function as a frame unit description unit described in units, and an object editing unit for editing the video object of the arbitrary shape in frame units described by the frame unit description unit.

According to this configuration, since the video object displayed by the display means is described in frame units by the frame unit description means, it is edited in frame units by the object editing means. In general, a start point of a specific portion of a video object is referred to as an in point, and an end point is referred to as an out point. With these configurations, the in point and the out point of a video object having an arbitrary shape are described and edited in frame units. be able to. Note that the video object may be configured to include audio objects such as audio integrally or separately.

[0010] The scene description program according to claim 2 is
2. The scene description program according to claim 1, wherein the frame unit description means includes a parameter that changes according to elapsed time, and includes a parameter description means that changes the parameter.

According to this configuration, the description in the frame unit description means can be freely changed according to the elapsed time by changing the parameters by the parameter description means.

According to a third aspect of the present invention, there is provided a scene description program,
A scene description program according to claim 1 or 2, further comprising video effect description means for describing at least one of a scene change and an image deformation of a video object.

According to this configuration, the video effect is given to each video object having an arbitrary shape by the video effect description means.

According to a fourth aspect of the present invention, there is provided a scene description program,
4. The scene description program according to claim 3, wherein the video effect description unit includes a parameter that changes with elapsed time, and includes a parameter description unit that changes the parameter.

According to this configuration, the description in the video effect description means can be freely changed according to the elapsed time by changing the parameters by the parameter description means.

According to a fifth aspect of the present invention, there is provided a scene description program,
The scene description program according to any one of claims 1 to 4, further comprising a space arrangement description unit that describes a spatial arrangement of the image object when the image object is displayed on a display unit. It is characterized by that.

According to this configuration, the position at which the video object is displayed on the display means can be freely changed by the space arrangement description means.

A scene description program according to claim 6 is
6. The scene description program according to claim 5, wherein the space arrangement description means includes a parameter that changes according to elapsed time, and includes a parameter description means that changes the parameter.

According to this configuration, the description in the space arrangement description means can be freely changed according to the elapsed time by changing the parameters by the parameter description means.

[0020]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS One embodiment of the present invention will be described below in detail with reference to the drawings. (Schematic Description of Scene Description Program) FIG. 1 is an explanatory diagram for explaining the position of a scene description program in object-based encoding. In the object-based encoding, one scene (scene) drawn on the left side of FIG. 1 is extracted (separated) into a plurality of video objects constituting the scene by an “object extraction” process. It is encoded for each video object, stored, transmitted, decoded for each video object after transmission, and video-composited by a “video composition” process.

"Object extraction" process and "image composition"
A "scene description program" defines information that links a process. That is, for example, the person included in the scene is
It is extracted as a "person" video object, and at the time of this extraction, the position information and the like displayed in the original scene are defined (described) by the scene description program. Then, in the "video composition" process, the video object of "person" is composed so that the original scene is reproduced based on the defined position information and the like.

(Video Object Editing Apparatus) FIG. 2 is a block diagram for explaining the functions of the video object editing apparatus activated by the scene description program. As shown in FIG. 2, the video object editing device 1 includes a display output unit 3,
It comprises a main control unit 5, a storage unit 7, an input unit 9, a storage medium handling unit 11, and an external connection unit 13.

The video object editing apparatus 1 adds a scene description file to a video object extracted from a certain scene (scene) or a video object shot alone by chroma key shooting or the like, and changes the contents of the scene description file. By describing, a scene is synthesized from video objects (usually a plurality).

In this embodiment, it is assumed that audio data (audio object) such as audio is integrally attached to a video object. However, the audio objects can also be handled separately by the scene description program as “parts” constituting the scene.

The display output unit 3 is a display device such as a general CRT or liquid crystal, and includes a sound output unit (not shown). A GUI is displayed on the display output unit 3 by a basic program (OS) developed in the main control unit 5, and a scene description program is activated on the OS. Edit screen (description screen) is displayed.

The main control section 5 is composed of a CPU, a memory and the like, and controls the video object editing apparatus 1. The main control unit 5 includes a basic program (OS)
Are deployed, and a scene description program is running on this OS. That is, the main control unit 5 executes each program of the scene description program, that is, the object display unit 5a, the object separation unit 5b, the frame unit description unit 5c, the video effect description unit 5d, the parameter description unit 5e, the space arrangement description unit 5f, The object editing means 5g is functionally realized.

The object display means 5a is a program for displaying a video object on the display output unit 3. A video object is displayed by the object display means 5a in accordance with the specification of the display output unit 3,
The description contents of each of the description means 5c to 5f are reflected and displayed on the display output unit 3.

The object separating means 5b is included in the video object editing apparatus 1 when the video object editing apparatus 1 exchanges a scene (scene) composed of a certain video object via the storage medium handling unit 11 or the external connection unit 13. It is a program that separates objects individually. In the object separating unit 5b, a portion where the display of the video object is defined is extracted from the scene description file defining the transferred scene,
Each video object used in the scene is extracted.

The frame unit description means 5c is a program for defining (controlling) a video object in frame units by adding a scene description file to the video object. The frame unit description means 5c defines that the video object exists in a specific scene, and a display start time and a display end time at which the video object is displayed in the specific scene are defined in frame units. You. That is, the frame unit description means 5
c describes the in point (display start time) and the out point (display end time) in the video object in frame units.

The scene description file is a file for describing (defining) the behavior of a video object, and one file is always generated for one video object. The frame unit is, for example, the minimum unit in time notation of a video obtained by dividing one second into about 30 frames (in the case of a general television). Here, to add,
In a conventional scene description program (excluding EDL), a video object can be managed only in milliseconds.

The frame unit description means 5c is the minimum essential block (program unit) of the scene description program, and as shown in FIG. 1, a person video object, a monster video object, and a background video object. It is possible to attach a scene description file to a video object having an arbitrary shape.

The video effect description means 5d describes a dissolve or wipe, which is a video expression used for a scene change connecting frames, describes an image deformation effect of a video object, and describes a 3D-image. Various effects (effects) such as a “page turning effect” using CG are described.

The parameter description means 5e freely changes the parameters described in the frame unit description means 5c and the video effect description means 5d, which change according to the elapsed time. There are two types of parameter change methods, one is a parameter change method that changes the parameter only once, and the other is a change method that changes the parameter continuously.

The changing method of changing the parameter only once is, for example, at a specific time (for example, 10: 00: 0
5:00, 5 seconds after just at 10:00)
There are methods such as doubling the size of the video object or halving the size.

In a changing method for continuously changing parameters, first, a file (hereinafter, referred to as a metadata file) that describes a time code and a value in the time code is prepared separately from a scene description file. .
Then, a value at a specific time code is extracted and set as a parameter. In other words, the parameters described by the frame unit description means 5c and the video effect description means 5d in the scene description file are defined as functions having the time code values as arguments, and the parameters themselves are associated with the time code values. Let it change. For example, a file describing display coordinates in each frame is prepared, and the display position of the video object can be temporally changed.

The space arrangement description means 5f calculates the size of the image frame of the entire scene (the relationship between the display surface of the display output unit 3 and the scene (scene) constituted by the video object),
It describes (defines) the display position of the video object on the display surface of the display output unit 3, before and after the overlap of the video objects, and the like.

The object editing unit 5g refers to a scene description file described by the frame unit description unit 5c, the video effect description unit 5d, the parameter description unit 5e, and the spatial arrangement description unit 5f for each video object, and This is to be composed and edited.

The storage unit 7 is composed of an HDD, a large-capacity memory, etc., is directly connected to the main control unit 5, stores a basic program, a scene description program, and the like, and also stores a single video object or scene description. Scenes synthesized and edited by the program are stored.

The input unit 9 is composed of a general keyboard, a pointing device such as a mouse, and the like. The operator who operates the video object editing apparatus 1 can use various functions and commands (for example, , Parameters) and the like.

The storage medium handling unit 11 is a general FD, C
It is composed of a D-ROM drive, MO drive, DVD drive and the like, and reads and writes these storage media. The handled storage medium stores, for example, a video object photographed by chromakey alone or a specific scene (scene) formed by a video object photographed by an existing broadcasting device.

The external connection unit 13 includes a port for connecting the video object editing apparatus 1 to an external device, a network or the like. In this embodiment, the external connection unit 13 is not connected to the network. However, when the external connection unit 13 is connected to the network, the image (video Image consisting of objects)
Can be imported as is and edited. Alternatively, an encoded video object can be exchanged, decoded by the main control unit 5 (in this case, the main control unit 5 needs to be provided with a decoding unit), and edited.

(Operation of Video Object Editing Apparatus) First, the video object editing apparatus 1
1 or a scene (scene) from a recording medium or another device connected to the network via the external connection unit 13 (S1). Then, based on the control of the object display means 5a of the main control unit 5, the scene is displayed on the display surface of the display output unit 3 (S2), and the video object editing apparatus 1 checks whether the scene is composed of the video object. It is determined whether it is (S3). If the scene is not composed of a video object, a display indicating "cannot be edited" is displayed on the display output unit 3 (S4). Then, it instructs a process in an object extracting device (not shown) capable of extracting a video object from a normal image.

In this embodiment, the description has been given assuming that the video object editing apparatus 1 can handle only scenes (scenes) composed of video objects.
An object extraction function (corresponding to an object extraction device) for extracting a video object may be provided.
Further, it is not always necessary to display the scene on the display output unit 3 after giving / receiving the scene, and each video object may be displayed after the process of S5 described later is completed.

If it is determined in S3 that the transmitted / received scene is composed of video objects, it is separated into individual video objects by the object separating means 5b (S5). Then, the operator of the video object editing apparatus 1 defines each video object by giving a name (object name) (S6). Next, a scene description file corresponding to the named video object is generated by the frame unit description means 5c (S7).

Further, the fact that the video object exists in the specific scene is described (defined) in the scene description file by the frame unit description means 5c, and the time at which the video object exists in the scene is set. Then, the frame point (in point) of the video object to be displayed in the scene and the frame point (out point) of the video object to be displayed are described.

Next, the video effect of the video object is described in the scene description file by the video effect description means 5d (S8). The video effects include, for example, a dissolve in which frames are switched from one frame to another and a transition effect such as a wipe in which the next frame of the displayed frame gradually increases, and 3D-CG.
Advanced video effects fused with are described.

The spatial arrangement description means 5f describes the spatial display coordinates (display position) of the video object in the scene description file (S9). That is, the spatial arrangement description means 5f describes the X, Y, and Z coordinates of the video object. For example, when the operator faces the display surface of the display output unit 3 and the upper left end of the display surface is used as the base point, the larger the value of the X coordinate of the video object is, the more the value of the Y coordinate is increased to the right. The video object is displayed as lower as possible, and as the value of the Z coordinate is larger, as it is closer.

Further, the parameters described by the frame unit description means 5c or the space arrangement description means 5f are freely changed by the parameter description means 5e (S10). For example, for two moving video objects (for example, “fish”) that exist in a specific scene (for example, a background video object is underwater), the Y coordinate of the Y coordinate described by the The value is described so as to be changed (the Y coordinate is reversed) each time these video objects overlap. Then 2
A scene (scene) can be described in which each time one video object (“fish”) moves underwater, the video object (“fish”) seen in front is switched.

In this embodiment, the function of generating a scene description file is provided in the frame description means 5c of the scene description program. (Description file generating means), the order of the steps S7 to S10 can be freely changed.

Here, the outline of the description contents of the scene description file will be described with reference to FIG. One scene is shown in FIG.
"Scene (head =“ 00: 00: 1 ”) shown in the upper part of FIG.
0:00 ")" (hereinafter referred to as a scene number). This scene is divided into a spatial arrangement description section, a temporal arrangement description section, a video effect description section, and a parameter change description section, and the behavior of each video object is described.

The spatial arrangement description section is composed of spatial arrangement description means 5
The display position of each video object is described by X coordinates, Y coordinates, and Z coordinates. The temporal arrangement description unit is described by the frame unit description unit 5c, and includes the start time of the existence of each video object in the scene, the display start time (in point), and the display end time (ou) of the video object.
t point) is described.

The video effect description section is described by the video effect description means 5d. The parameter change description part is described by the parameter description means 5d, and in this embodiment, the X coordinate of the display position of the video object named fish1 is stored in the file fish_mot.
Is described to be changed to date (1).

The file fish_mot is an external file to which a scene description file can be referred.
Shows the contents of the external file fish1_mot. The left column in FIG. 5 is the time code, and the middle column is dat.
e (1), and the right column in the figure represents date (2). After programming with the scene description program,
At the time of scene reproduction, for example, the time in the scene is 00: 0.
At 0:00:03, date (1) is 15,
date (2) is -46. As a result, the display coordinates of the video object fish1 are X coordinate 15, Y
The coordinates become -46.

Returning to the flowchart of FIG. 3, the description of the video object editing apparatus 1 will be continued. When the description (definition) of the scene description file of one video object is completed, the object editing device 1 describes or presents another video object to the operator (S11), and asks the operator for a determination. If another video object is to be described, the process returns to S6, where the video object is defined, or a previously defined video object is selected and described. If no other video object is described, a plurality of video objects are synthesized by the object editing means 5g so as to form a specific scene (S
12). A specific scene is configured based on a scene number.

The video object synthesized by the image editing unit 5g is displayed on the display output unit 3 (S13), and the operator checks the synthesized video object and determines whether to end the editing (S13). S14),
If the editing is to be repeated (redo), the process returns to S6, and if it is determined that the editing is to be ended, the edited (composited video) scene is stored in the storage unit 7 and the process is terminated.

(Example of Sample of Video Object and Scene Description Program (Example of Description)) FIGS. 6 and 7 show samples of individual video objects. Also, FIG.
8 shows an example of a scene composed of the video objects shown in FIG. FIG. 6 illustrates fish1 to fish5, which are image objects of “fish”, having various shapes, plant, which is an image object of “plant”, and sand, which is an image object of “gravel”. In addition,
It is assumed that fish2 and fish4 have the same shape but different colors. FIG. 7 also shows a rock image object of “rock” and a bubblesand image object of “gravel with air bubbles”.

In FIG. 8 in which these video objects are video-composited, bubblesand, which is a video object of “gravel with blowing bubbles”, is placed at the front, and then a video object of “rock” is placed. A ro
ck is arranged, followed by a plant that is a video object of "plant", and a background (space) is configured, and a scene in which fish1 to fish5 are swimming in this space is illustrated. .

FIG. 9 shows an outline of a program described by the scene description program. <Scen
e start = "00: 00: 00: 00"> indicates the time code at the head of the scene. <Layout
<width height> is a description that defines the size of the entire scene. That is, it is a description corresponding to the spatial arrangement description section. <Region id = "r_sand">
Is a definition of the display position of the sand, left is the X coordinate, top is the Y coordinate, and z-index indicates the context.

<Timeline> is a description corresponding to a temporal arrangement description section. <Effect> is a description corresponding to the video effect description part. <Animation> is a description corresponding to a parameter change description part. In the program shown in FIG. 9, a general-purpose XML (eXtens
Although the description has been made based on the example described in I. Markup Language, the present invention is not limited to the XML.

This embodiment has the following effects. With respect to a so-called scene description program in object-based coding, a video object having an arbitrary shape can be handled as compared with a conventional scene description program, and can be managed and edited in frame units by the frame unit description means 5c.

Since an arbitrary video object displayed by the display output unit 3 is described in frame units by the frame unit description unit 5c, it is edited in frame units by the object editing unit 5g. For this reason, a scene (scene) can be produced by editing a video object having an arbitrary shape on a frame basis, and can be applied to, for example, production of a broadcast video.

Further, since each scene description file has an independent timeline (by the temporal arrangement description unit), each scene is reproduced (displayed) from the middle of each scene (video object at an arbitrary time). You can also.

The video effect description means 5d applies a video effect to each video object having an arbitrary shape, so that it is possible to produce a scene incorporating a high-level video effect. The position at which the video object is displayed on the display output unit 3 can be freely changed by the space arrangement description means 5f. Also, the description in the frame unit description means 5c or the video effect description means 5d can be freely changed according to the elapsed time by changing the parameters by the parameter description means 5e, so that various effects can be applied to a specific scene. Can be.

The present invention has been described based on one embodiment, but the present invention is not limited to this. For example, a program developed in the main control unit 5 of the video object editing apparatus 1 can be stored in a storage medium and handled, or distributed via a network or the like.

[0065]

According to the first aspect of the present invention, since the video object displayed by the display means is described in frame units by the frame unit description means, the video object is edited in frame units by the object editing means. Is done. In point of video object of arbitrary shape
The out point can be described and edited in frame units.

According to the second aspect of the present invention, the description in the frame unit description means can be freely changed according to the elapsed time by changing the parameters by the parameter description means, so that various effects can be produced for a specific scene. Can be applied.

According to the third aspect of the present invention, a video effect is given to each arbitrarily shaped video object by the video effect description means, so that various scenes composed of arbitrarily shaped video objects can be produced. Can be.

According to the fourth aspect of the present invention, the description in the video effect description means is provided by the parameter description means.
By changing the parameter, it can be changed freely according to the elapsed time, so that various effects can be applied to a specific scene.

According to the fifth aspect of the present invention, the position at which the video object is displayed on the display means can be freely changed by the spatial arrangement description means, and is used, for example, as a broadcast scene description program. can do.

According to the sixth aspect of the present invention, the description in the space arrangement description means is performed by the parameter description means.
By changing the parameter, it can be changed freely according to the elapsed time, so that various effects can be applied to a specific scene.

[Brief description of the drawings]

FIG. 1 is an explanatory diagram for explaining the outline (positioning in object-based encoding) of a scene description program according to the present invention.

FIG. 2 is a block diagram of a video object editing device.

FIG. 3 is a flowchart illustrating an operation of the video object editing apparatus.

FIG. 4 is a diagram illustrating the description contents of a scene description file.

FIG. 5 is a diagram illustrating a description example of “fish” of a video object;

FIG. 6 is a diagram illustrating an example of a video object.

FIG. 7 is a diagram illustrating an example of a video object.

8 is a diagram showing an example of a scene constituted by the video objects shown in FIGS. 6 and 7. FIG.

FIG. 9 is a diagram illustrating an example of a scene description program.

[Explanation of symbols]

 REFERENCE SIGNS LIST 1 video object editing device 3 display output unit 5 main control unit 5c frame unit description unit 5d video effect description unit 5e parameter description unit 5f space layout description unit 5g object editing unit 7 storage unit 9 input unit 11 storage medium handling unit 13 external connection Department

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) H04N 5/92 H04N 7/13 Z // H04N 7/24 G11B 27/02 B (72) Inventor Osamu Mizuno 1-10-11 Kinuta, Setagaya-ku, Tokyo Japan Broadcasting Corporation Research Institute (72) Inventor Yoshiaki Kakuro 1-1-10 Kinuta, Setagaya-ku, Tokyo Japan Broadcasting Corporation F-Term (Reference) ) 5B050 AA09 BA11 EA01 EA19 EA24 5C023 AA06 AA13 AA14 AA17 AA34 AA38 BA02 BA12 CA03 CA08 DA04 DA08 EA03 5C053 FA14 FA23 FA27 GB06 GB09 JA22 KA04 KA05 KA24 LA06 5C059 KK36 MA29 MB10 ACA13 SS06 CD07 CM06

Claims (6)

[Claims] 1. Display means for displaying video objects of arbitrary shape constituting each scene, frame unit description means for describing display start time and end time of the video object of arbitrary shape in frame units, frame unit description means A scene description program for causing a computer to function as object editing means for editing the video object of an arbitrary shape in frame units described by 2. The scene description program according to claim 1, wherein the frame unit description unit includes a parameter that changes according to an elapsed time, and includes a parameter description unit that changes the parameter. 3. The scene description program according to claim 1, further comprising a video effect description unit that describes at least one of a scene change and an image deformation of a video object. 4. The scene description program according to claim 3, wherein the video effect description unit includes a parameter that changes according to an elapsed time, and includes a parameter description unit that changes the parameter. 5. The image processing apparatus according to claim 1, further comprising a spatial arrangement description unit for describing a spatial arrangement of the video object when the video object is displayed on a display unit. The scene description program according to claim 1. 6. The scene description program according to claim 5, wherein the space arrangement description means includes a parameter that changes according to an elapsed time, and includes a parameter description means that changes the parameter.