KR100810649B1 - System and method for moving picture file and multimedia file synthesis - Google Patents

Abstract

A system and a method for combining a video file and a multimedia file are provided to express boundaries naturally and smoothly by rendering images of a multimedia file to draw them newly. A video data decoder(20) extracts a still image of each frame in a video file. A rendering engine(30) renders a still image of each frame of a multimedia file to be combined with the video file to generate a rendered still image of each frame of the multimedia file. An image combining engine(40) synchronizes the still image of each frame of the video file extracted by the video data decoder(20) and the rendered still image of each frame of the multimedia file and combines them to product a combined still image of each frame. An encoder(90) encodes the combined still image of each frame produced by the image combining engine(40) into a combined video file.

Description

System and method for moving picture file and multimedia file synthesis

1 is a block diagram of a system for synthesizing a video file and a multimedia file according to the present invention;

2 is a flowchart of Embodiment 1 of a method for synthesizing a video file and a multimedia file according to the present invention;

3 is a flowchart of Embodiment 2 of a method of synthesizing a video file and a multimedia file according to the present invention;

4 is a flowchart of Embodiment 3 of a method of synthesizing a video file and a multimedia file according to the present invention;

5 is a conceptual diagram illustrating a state in which frames of a video file and a multimedia file are synchronized according to the present invention.

FIG. 6A is a capture diagram illustrating an unclear boundary line when a video file and a multimedia file are synthesized according to the present invention. FIG.

Figure 6b is a capture diagram showing a clear boundary when the video file and the multimedia file according to the present invention synthesized.

7A is a capture diagram illustrating a portion to be synthesized between a video file and a multimedia file according to the present invention.

 FIG. 7B is a capture diagram illustrating the disappearance of a portion designated as a transparent color of a multimedia image when the video file and the multimedia file are mutually synthesized according to the present invention. FIG.

8A is a frame-by-frame capture diagram of a multimedia file according to the present invention, FIG. 8B is a frame-by-frame capture diagram of a video file according to the present invention, and FIG. 8C is a cross-synthesized state of a video file and a multimedia file according to the present invention. Frame-by-frame capture representation.

9 is a capture diagram showing a user operation for the mutual synthesis of a video file and a multimedia file according to the present invention.

※ Explanation of code about main part of drawing ※

100: A system for composing video files and multimedia files

10: Video player 20: Video data decoder

30: rendering engine 40: video synthesis engine

50: buffer memory 60: audio data decoder

70: sound extractor 80: speech synthesis engine

90: encoder

The present invention relates to a synthesis technique of a video file and a multimedia file.

In general, for still images that do not have time elements such as documents, pictures, and photographs that do not express movements, when editing with other still images, editing is performed using an editing program such as Photoshop.

However, there is a problem in that a work time is increased as the number of still pictures increases in synthesizing the moving picture file consisting of continuous still pictures as described above and a multimedia file such as Flash.

Therefore, even if a system for automatically synchronizing and synthesizing frames of a video file and a multimedia file is proposed, there is a problem in that a problem of overlapping between an image of a video file and a multimedia file occurs during synthesis.

In addition, since the processing of the boundary of the synthesized image is unnatural, there is a problem that the quality of the synthesized video file is inferior.

In addition, since the synthesis method is a method of uploading a video file and a multimedia file together to an encoder and encoding the frame by synchronizing the frame, a user may not insert a specific part of another video file while playing one video file.

The present invention has been proposed to solve the above problems,

Firstly, an object of the present invention is to provide a system and method for synthesizing a video file and a multimedia file, which smoothly and naturally process the boundary of the synthesized image when the video file and the multimedia file are synthesized.

Secondly, an object of the present invention is to provide a system and method for synthesizing a video file and a multimedia file to solve the problem of image overlap between the video file when the video file and the multimedia file are synthesized.

Thirdly, an object of the present invention is to provide a system and method for synthesizing a video file and a multimedia file, when the user selects another multimedia file to be synthesized while playing a specific video file.

Fourthly, an object of the present invention is to provide a system for synthesizing a video file and a multimedia file and a method thereof to increase the speed of synthesis when synthesizing a video file and a multimedia file.

A system for synthesizing a video file and a multimedia file according to the present invention for achieving the above object includes a video data decoder for extracting a still image for each frame from the video file; A rendering engine for generating a still image for each frame of the multimedia file by rendering a still image for each frame of the multimedia file to be synthesized with the video file; An image synthesis engine for synthesizing a frame-by-frame still image of the video file extracted by the video data decoder and a frame-by-frame still image of the multimedia file rendered by the rendering engine, and synthesizing each frame to produce a frame-by-frame composite still image; And an encoder for encoding the frame-by-frame composite still image produced by the image synthesis engine into a composite video file.

The system for synthesizing a video file and a multimedia file according to the present invention includes a video data decoder for extracting a still image for each frame from the video file; An image synthesizing engine for synthesizing a still image for each frame of a moving image file extracted by the video data decoder and a still image for each frame of a multimedia file to be synthesized frame by frame and synthesizing each frame to generate a composite still image for each frame; A rendering engine configured to render an image of the multimedia file in the synthesized frame-by-frame composite still image to generate a multimedia file on the frame-by-frame composite still image; And an encoder for encoding a frame-by-frame composite still image from which the image is generated by the rendering engine into a composite video file.

The system for synthesizing a video file and a multimedia file according to the present invention includes a video data decoder for extracting a still image for each frame from the video file; A buffer memory in which a still picture for each frame of the video file is constructed; A rendering engine for directly rendering an image of the multimedia file on the frame-by-frame still image of the video file constructed in the buffer memory to generate the multimedia file on the frame-by-frame composite still image; And an encoder for encoding a frame-by-frame composite still image in which the image is generated by the rendering engine into a composite video file.

On the other hand, a method for synthesizing a video file and a multimedia file according to the present invention comprises the steps of: a) extracting a still image for each frame from the video file using a video data decoder; b) generating a still image for each frame of the multimedia file by using a rendering engine to render a still image for each frame of the multimedia file to be synthesized with the moving image file; c) Synthesis-by-frame composite still image by synchronizing frame-by-frame still image of the video file extracted by the video data decoder and frame-by-frame still image of the multimedia file rendered by the rendering engine by frame and synthesizing each other Producing a; And d) encoding a frame-by-frame synthesis still image produced by the image synthesis engine into a synthetic video file using an encoder.

The method for synthesizing a video file and a multimedia file according to the present invention includes: a) extracting a still image for each frame from a video file using a video data decoder; b) synthesizing the frame-by-frame still image of the video file extracted from the video data decoder and the frame-by-frame still image of the multimedia file to be synthesized by frame and synthesizing each other to produce a composite still image by frame step; c) a rendering engine configured to render an image of the multimedia file in the synthesized frame-by-frame composite still image and generate the multimedia file on the frame-by-frame composite still image; And d) encoding a frame-by-frame composite still image generated by the rendering engine into a composite video file using an encoder.

The method for synthesizing a video file and a multimedia file according to the present invention includes: a) extracting a still image for each frame from a video file using a video data decoder; b) constructing a still picture for each frame of the video file in a buffer memory; c) directly generating an image of the multimedia file on the frame-by-frame still image of the video file constructed in the buffer memory using a rendering engine to generate the multimedia file on the frame-by-frame composite still image; And d) encoding a frame-by-frame composite still image generated by the rendering engine into a composite video file using an encoder.

Hereinafter, with reference to the drawings will be described the specific configuration and operation of the present invention.

As shown in FIG. 1, the system 100 for synthesizing a video file and a multimedia file according to the present invention basically includes a video player 10, a video data decoder 20, a rendering engine 30, and an image synthesis engine ( 40, an audio data decoder 60, a sound extractor 70, a speech synthesis engine 80, an encoder 90 and a buffer memory 50 are provided.

Moving picture (moving picture, moving picture) continuously used in the present invention refers to formats such as MPEG, MOV, RealVideo, Avi, etc., multimedia files are flash, DMF (Digital Mobile Flash), VIS (Vector Image Solution), ANIMATION Refers to a format such as GIF.

The terms defined and used in the present specification and claims, such as video files and multimedia files, should not be construed in a dictionary meaning, and the inventors should properly interpret the concept of terms in order to best explain their invention. Based on the principle that it can be defined, it should be interpreted as meaning and concept corresponding to the technical idea of the present invention.

Hereinafter, the first embodiment of the present invention will be described in detail by component.

First, the video player 10 is largely provided with components such as storage means (not shown), display means (not shown), and key input means (not shown).

The storage means stores a video file (2), a codec for decompressing and playing the video file (2), and a multimedia file (3) to be synthesized in the video file (2). In addition, the storage means is responsible for the role of the storage space in which the synthesized video encoded by the encoder described later is stored in real time.

In addition, the display means serves to express a video to be played back to the user, and the key input means generates a user's operation signal, that is, a portion that the user wants to synthesize, i.e., a start frame is designated and the video being played. It plays the role of selecting and uploading the multimedia file to be synthesized from the user.

The video data decoder 20 extracts a still image for each frame from a video file. Extracting a still image for each frame from a video file is a well-known technique, and thus a detailed description thereof will be omitted.

The rendering engine 30 renders a still image for each frame of the multimedia file to be synthesized with the video file and generates an image for each frame of the multimedia file.

In this case, rendering refers to a process of creating an image by injecting reality by considering external information such as a light source, a location, and a color.

That is, the multimedia file records position coordinate values of pixels, which are the smallest unit for representing an image, and color values of pixels to be input to the position coordinates, and is recorded in the multimedia file by the rendering operation of the rendering engine as described above. The image to be represented in the multimedia file is actually drawn according to the position coordinates and colors of the pixels.

The image synthesis engine 40 synchronizes the frame-by-frame still image of the video file extracted by the video data decoder and the frame-by-frame still image of the multimedia file rendered by the rendering engine, by frame, as shown in FIG. 5. It is responsible for producing composite still images for each frame by synthesizing each other.

Here, the simple compositing of the still image of the moving image file and the still image of the multimedia file may cause the overlapping problem of images. Accordingly, the image synthesis engine 40 transparently processes pixels having a predetermined color value as a transparent color in the frame-by-frame still image of the multimedia file when compositing with the frame-by-frame still image of the corresponding video file (see FIGS. 7A and 7B). )

In addition, the image synthesis engine 40 ignores the corresponding pixel in the video file data corresponding to the pixel having the color value designated as opacity in the frame-by-frame still image of the multimedia file when the video file data is combined with the frame-by-frame still image. The pixel of the still picture for each frame is displayed first.

For example, in a multimedia file, when the still image is yellow, the border of the image is black, and the background color is gray, the gray is pre-specified as a transparent color so that it disappears when compositing with the video file. Is opaque to disregard the pixels of the still image of the video file corresponding to the yellow and black, and to display the yellow and black of the multimedia file first.

FIG. 8A is a frame-by-frame still image of a multimedia file, FIG. 8B is a frame-by-frame still image of a video file being played back, and FIG. 8C is a frame-by-frame composite still image synthesized and displayed by the image synthesis engine 40. will be.

The encoder 90 is provided to encode the frame-by-frame synthesized still image synthesized by the image synthesis engine 40 as described above.

The encoder 90 compresses the composite still image for each frame and the synthesized speech data synthesized by the speech synthesis engine 80 to be described later using a lossy or lossless compression method using a specific codec to produce a composite video file. Do it.

So far, image synthesis has been described, and hereinafter, voice synthesis will be described.

An audio data decoder 60 for extracting sound data from the video file, a sound extractor 70 for extracting sound data from the multimedia file, and synthesized sound by synthesizing sound data of the video file and sound data of the multimedia file The speech synthesis engine 80 generates data and delivers the data to the encoder 90.

The synthesized voice data synthesized by the voice synthesis engine 80 is transferred to the encoder, and the encoder is configured to reflect the synthesized voice data in the synthesized video file to produce a synthesized video.

A second embodiment of a system for synthesizing a video file and a multimedia file of the present invention will be described below.

A system for synthesizing a video file and a multimedia file according to Embodiment 1 of the present invention includes a still image for each frame extracted from the video data decoder 20, and a still image for each frame of the multimedia file that has been already rendered and rendered by the rendering engine. Images are combined with each other.

However, since the boundary portion of the multimedia file is a mixture of colors designated as transparent and opaque colors, the synthesis engine does not accurately reflect only the image to be displayed as opaque in the still image of the multimedia file, and thus may be unnatural as shown in FIG. 6A.

Therefore, in the second embodiment of the present invention, in order to overcome the unnaturalness of the boundary portion, if the video data decoder 20 extracts the still picture for each frame from the video file, the still picture for each frame of the multimedia file is not pre-rendered. The frame-by-frame still image of the extracted video file and the frame-by-frame still image of the multimedia file to be synthesized are synchronized by frame and synthesized with each other.

Subsequently, the image of the multimedia file is generated on the frame-by-frame composite still image by rendering the image of the frame-by-frame still image of the multimedia file on the frame-by-frame composite still image synthesized using the rendering engine 30.

As a result, the unnaturalness of the boundary occurs in the synthesis between the still image of the video file and the still image of the multimedia file in which the image is already drawn by rendering, so that the image of the multimedia file can be newly drawn by rendering the synthesized frame-by-frame composite still image. As shown in 6b, the boundary line can be clearly drawn.

Next, Example 3 of the present invention will be described.

In the third embodiment of the present invention, the frame-by-frame still image is extracted from the video file extracted by the video data decoder 20 in the buffer memory 50, and the frame-by-frame of the video file constructed in the buffer memory 50 using the rendering engine. A multimedia file is generated on a frame-by-frame composite still image by directly rendering an image of a frame-by-frame still image on a still image.

Accordingly, the third embodiment of the present invention directly renders the image of the multimedia file on the frame-by-frame still image of the video file constructed in the buffer memory 50 without the image synthesizing engine 40, thereby saving time required for synthesis. Can be.

Hereinafter, a method of synthesizing a video file and a multimedia file of the present invention having the above configuration will be described.

Example 1

As shown in FIG. 2 and FIG. 9, a user first plays a video file using the video player 10 and uses a function such as select to select a portion to start compositing from the video file being played. If specified, the video being played is paused.

At this time, the portion that the user temporarily pauses is designated as the start frame (S100). Subsequently, in the paused video, the user selects and inserts a multimedia file to be synthesized after the start frame (s102).

Accordingly, the video data decoder 20 extracts a still image for each frame of the video file after the start frame from the video file (S104).

Subsequently, a still image for each frame of the multimedia file to be synthesized with the moving image file is rendered using the rendering engine 30 to generate an image for each frame of the multimedia file (S106).

Subsequently, the frame-by-frame still image of the video file extracted by the video data decoder 20 and the frame-by-frame still image of the multimedia file rendered by the rendering engine 30 are synchronized by frame using the image synthesis engine 40. By synthesizing with each other, a composite still image for each frame is produced (s108).

Meanwhile, sound data is extracted from the video file using the audio data decoder 60 (s110), sound data is extracted from the multimedia file using the sound extractor 70 (s112), and the speech synthesis engine 80 is used. By synthesizing the sound data of the video file and the sound data of the multimedia file with each other to generate the synthesized sound data (s114) and delivers to the encoder (90).

As a result, the encoder 90 generates a composite video file by performing encoding using the synthesized sound data and the composite still image for each frame (S116).

Subsequently, the video player 10 expresses the composite video file to the user in real time after the user specified start frame (S118).

Example 2

As shown in FIG. 3, if a user first plays a video file using the video player 10 and designates a portion of the playing video file to start compositing using a function such as select, The video being played will be paused.

At this time, the portion that the user temporarily pauses is designated as the start frame (S200). Subsequently, in the paused video, the user selects and uploads a multimedia file to be synthesized after the start frame (s202).

Accordingly, the video data decoder 20 extracts a still image for each frame of the video file after the start frame from the video file (s204).

Subsequently, frame-by-frame synthesis is performed by synchronizing frame-by-frame still images of video files extracted from the video data decoder 20 and frame-by-frame still images of a multimedia file to be synthesized by frames and synthesizing each other using the image synthesis engine 40. A still image is produced (s206).

Subsequently, by rendering each image of the frame-by-frame still image of the multimedia file in the synthesized frame-by-frame composite still image, the multimedia file is generated on the frame-by-frame composite still image (S208).

Meanwhile, sound data is extracted from the video file using the audio data decoder 60 (s210), sound data is extracted from the multimedia file using the sound extractor 70 (s212), and the speech synthesis engine 80 is used. By synthesizing the sound data of the video file and the sound data of the multimedia file with each other to generate the synthesized sound data (s214) and transmits to the encoder 90.

As a result, the encoder 90 generates a composite video file by performing encoding using the synthesized sound data and the composite still image for each frame (S216).

Subsequently, the video player 10 expresses the composite video file to the user in real time after the start frame designated by the user (S218).

Example 3

As shown in FIG. 4, if a user first plays a video file using the video player 10 and designates a portion of the playing video file to start compositing using a function such as select, The video being played will be paused.

At this time, the portion that the user temporarily pauses is designated as the start frame (S300). Subsequently, in the paused video, the user selects and inserts a multimedia file to be synthesized after the start frame (insert frame) (S302).

Accordingly, the video data decoder 20 extracts a still image for each frame of the moving image file after the start frame from the moving image file (S304).

First, the still image for each frame is extracted from the video file using the video data decoder 20 (S304).

Subsequently, a still picture for each frame of the video file is constructed in a buffer memory (s306).

Subsequently, the rendering engine 30 directly renders the image of the multimedia file on the frame-by-frame still image of the video file constructed in the buffer memory 50 to generate the multimedia file on the frame-by-frame composite still image (S308).

Meanwhile, sound data is extracted from the video file using the audio data decoder 60 (s310), sound data is extracted from the multimedia file using the sound extractor 70 (s312), and the voice synthesis engine 80 is used. By synthesizing the sound data of the video file and the sound data of the multimedia file with each other to generate a synthesized sound data (s314) and delivers to the encoder (90).

As a result, the encoder 90 generates a composite video file by performing encoding using the synthesized sound data and the composite still image for each frame (S316).

Subsequently, the video player 10 expresses the composite video file to the user in real time after the start frame designated by the user (S318).

As mentioned above, although the present invention has been described by means of a limited configuration and drawings, the technical idea of the present invention is not limited to the above, and by those skilled in the art to which the present invention pertains, Various modifications and variations may be made without departing from the scope of the appended claims.

According to the present invention by the above configuration,

First, the present invention synchronizes a frame-by-frame still image of a video file extracted by a video data decoder and a frame-by-frame still image of a multimedia file to be synthesized frame by frame, and synthesizes each other. This will render the image.

Therefore, by rendering an image of a multimedia file of the composite still image of each frame synthesized and newly drawing, a borderline can be expressed naturally and smoothly.

Second, according to the present invention, a pixel having a predetermined color value as a transparent color in a frame-by-frame still image of the multimedia file is transparently processed when compositing with a frame-by-frame still image of a corresponding video file, and opaque in the frame-by-frame still image of the multimedia file. Pixels having a designated color value are displayed first when compositing the corresponding still image for each frame of corresponding video file data.

Therefore, it is possible to effectively combine video and multimedia files by eliminating image overlap.

Third, in the present invention, a start frame is designated by a user to start synthesizing a video file using a video player, and a multimedia file to be synthesized after the start frame is selected.

Therefore, the composite video file encoded by the encoder can be displayed to the user in real time after the start frame.

Fourth, the present invention directly renders an image of a multimedia file on a frame-by-frame still image of a video file constructed in a buffer memory, thereby generating a multimedia file on a frame-by-frame composite still image.

Therefore, since a separate synthesis process can be omitted by the buffer memory, the synthesis can be performed at a high speed.

Claims (13)

A video data decoder for extracting a still picture for each frame from a video file; A rendering engine for generating a still image for each frame of the multimedia file by rendering a still image for each frame of the multimedia file to be synthesized with the video file; An image synthesis engine for synthesizing the frame-by-frame still image of the video file extracted by the video data decoder and the frame-by-frame still image of the multimedia file rendered by the rendering engine, and synthesizing each frame to produce a composite still image for each frame; And System for synthesizing a video file and a multimedia file, characterized in that it comprises an encoder for encoding the frame-by-frame composite still image produced by the image synthesis engine into a composite video file. A video data decoder for extracting a still picture for each frame from a video file; An image synthesizing engine for synthesizing a still image for each frame of a moving image file extracted by the video data decoder and a still image for each frame of a multimedia file to be synthesized frame by frame and synthesizing each frame to generate a composite still image for each frame; A rendering engine for rendering an image of the multimedia file in the synthesized frame-by-frame composite still image; And And an encoder for encoding a frame-by-frame composite still image in which the image is generated by the rendering engine into a composite video file. A video data decoder for extracting a still picture for each frame from a video file; A buffer memory in which a still picture for each frame of the video file is constructed; A rendering engine configured to directly render an image of a frame-by-frame still image of the multimedia file on a frame-by-frame still image of the video file constructed in the buffer memory to generate a frame-by-frame composite still image of the multimedia image; And And an encoder for encoding a frame-by-frame composite still image in which the image is generated by the rendering engine into a composite video file. The method according to any one of claims 1 to 3, A start frame to be synthesized from the video file by the user, a multimedia file to be synthesized after the start frame is selected, and a video player for displaying the encoded video file encoded by the encoder to the user in real time after the start frame; A system for compositing video files and multimedia files. The method of claim 1, The image synthesis engine Pixels having a color value predetermined as a transparent color in the frame-by-frame still image of the multimedia file are transparently processed when compositing with the frame-by-frame still image of the corresponding video file And a pixel having a color value designated as opacity in the frame-by-frame still image of the multimedia file is displayed first when composing the corresponding video file data with the frame-by-frame still image. The method of claim 4, wherein An audio data decoder for extracting sound data from the video file; A sound extractor for extracting sound data from the multimedia file; And Extracting and synthesizing the sound data of the video file and the sound data of the multimedia file, and generating a synthesized voice data and transmitting the synthesized voice data to the encoder so that the encoder can reflect the synthesized voice data in the synthesized video file. A system for synthesizing a video file and a multimedia file. The method of claim 1, The multimedia file is a system for synthesizing a video file and a multimedia file, characterized in that any one of the format of Flash, Digital Mobile Flash (DMF), Vector Image Solution (VIS), ANIMATION GIF. a) extracting a still picture for each frame from a moving picture file using a video data decoder; b) generating a still image for each frame of the multimedia file by using a rendering engine to render a still image for each frame of the multimedia file to be synthesized with the moving image file; c) Synthesis-by-frame composite still image by synchronizing frame-by-frame still image of the video file extracted by the video data decoder and frame-by-frame still image of the multimedia file rendered by the rendering engine by frame and synthesizing each other Producing a; And and d) encoding a frame-by-frame composite still image produced by the image synthesis engine into a composite video file using an encoder. a) extracting a still picture for each frame from a moving picture file using a video data decoder; b) synthesizing the frame-by-frame still image of the video file extracted by the video data decoder and the frame-by-frame still image of the multimedia file to be synthesized by frame and synthesizing each other to produce a composite still image by frame step; c) rendering the multimedia file in the synthesized frame-by-frame synthesized still image to generate the multimedia file on the frame-by-frame composite still image; And and d) encoding a frame-by-frame composite still image generated by the rendering engine into a composite video file using an encoder. a) extracting a still picture for each frame from a moving picture file using a video data decoder; b) constructing a still picture for each frame of the video file in a buffer memory; c) generating a frame-by-frame composite still image by directly rendering an image of the frame-by-frame still image of the multimedia file on the frame-by-frame still image of the video file constructed in the buffer memory using a rendering engine; And and d) encoding a frame-by-frame composite still image generated by the rendering engine into a composite video file using an encoder. The method according to any one of claims 8 to 10, Prior to step a), a ') using the video player, designating a start frame to start synthesizing the video file from a user; And a '') further comprising the step of receiving a multimedia file to be synthesized after the start frame by the user using the video player, After step d), d ') a method of synthesizing a video file and a multimedia file, further comprising displaying the synthesized video file encoded by the encoder to the user in real time after the start frame using a video player. The method according to any one of claims 8 to 10, e) extracting sound data from the video file using an audio data decoder; f) extracting sound data from the multimedia file using a sound extractor; And g) Synthesizing the sound data of the video file and the sound data of the multimedia file by using a voice synthesis engine to generate synthesized sound data and transmitting the synthesized sound data to the encoder in step d) to reflect the synthesized sound data on the synthesized video file. Method for synthesizing a video file and a multimedia file, characterized in that it further comprises the step of enabling. The method of claim 8, Transparently processing pixels having a predetermined color value as a transparent color in the frame-by-frame still image of the multimedia file when compositing with the frame-by-frame still image of a corresponding video file; And Pixels having color values designated as opacity in the frame-by-frame still image of the multimedia file are first displayed when synthesizing the corresponding video file data with the frame-by-frame still image. How to.

Images