KR20010035099A - Streaming Hypervideo System using Automatic Scene Change Detection and Controlling Method - Google Patents

Abstract

PURPOSE: A streaming hypervideo system using scene change automatic detection and a method for controlling the same are provided to perform scenes by a shot without an additional hardware. And in respect to a passed shot, the device enable to change to a pertinent shot with selecting a main frame. CONSTITUTION: A motion picture is reproduced in a streaming hypervideo working tool(S1). A scene change is automatically detected by using directshow technique(S2). An arbitrary frame is selected in the analyzed motion picture representative frames and a frame where an object to insert anchor information to whole or a part of the scene applicable to the selected frame, is selected(S3). A specific object is automatically tracked related to the selected frame(S10). The anchor information are inserted into the applicable frame of the tracked object and the applicable position of the applicable frame(S14). The worked anchor information are stored in files. The worked streaming hypervideo is manufactured as web documents(S14).

Description

Streaming Hypervideo System using Automatic Scene Change Detection and Controlling Method

The present invention relates to a streaming hypervideo system and a control method thereof. In particular, after a scene change automatic detection process is performed, a hyper hyper video system is authored and streamed by inserting anchor information into an area of a tracked object after tracking a specific object in a shut unit. By automatically generating web documents including hypervideo and allowing the player to properly display anchor information in a predefined form, it is possible to efficiently manage and edit digital video data on a shut-off basis as well as to automatically generate web documents. The present invention relates to a streaming hypervideo system that provides excellent performance in design and development and provides a user-friendly and convenient screen and a control method thereof.

Hyper Video System was first researched at MIT in 1996, and started from the prototype called HyperCafe. It has been developed in MIT, Korea, and overseas for the system configuration and screen interface. HyperCafe, International Business Machines Corporation (IBM), Veon, Inc. It has been studied in academia and companies such as Telecommunication Co., Ltd., Samsung Electronics Co., Ltd. and Korea First Tech.

MIT

HyperCafe connects the video clips in multi-threads based on the storyspace so that the user can watch the video clips of the story as shown in FIG. 1A or the user's choice as shown in FIG. 1B. You can watch video clips of a specific story. In addition, as can be seen in Figure 1 (b) by sharing the video clips on the different storyline with each other to create an efficient story composition. This system allows the user to enter a virtual cafe existing on the Internet and passively watch the other person talk or selectively view the field of interest. The system can be used to conceptualize a storyline, to capture relevant scenes using two or more cameras to acquire analog video data, to convert the acquired analog video data into digital video clips, and to edit them in a predetermined order. It consists of a script production step to construct. HyperVideo created through each of these steps connects to other video clips using video data and time and space script files created in the authoring process.

However, the HyperCafe system has a problem in that it is inconvenient for general users to use by using a script generator in the form of an editable text rather than displaying the hypervideo authoring tool in a graphical user interface environment. The visual authoring tool of FIG. 2 is not yet implemented. In addition, since most of existing video data such as movies and dramas are provided as a single storyline, it is not suitable for reprocessing existing videos and providing them to users.

2. International Business Machines Corporation (IBM)

Interactive video created through the HyperVideo authoring tool is delivered to each individual over the Internet. At this time, each user sees this using a specific player, and it is impossible to know which object the anchor information is inserted in the video displayed on the screen. Therefore, the player corresponding to the client must deliver this information to each user in a proper way. IBM informed the user about the existence of the anchor area by changing the visual properties of the anchor area within the frame. Also, for the users who do not want to change the video itself, the screen is displayed in the outside area of the video rather than changing the property of the screen, and when the mouse is over the anchor area, the shape of the mouse is modified. As a method of displaying on the outside of the screen, a method of displaying the anchor region B on the frame by placing the anchor indicator A as shown in FIG. 3 has been filed in Korea as a patent. This patent relates to a method for indicating the position of a video hot anchor and does not include the creation of hypervideo.

Following the development of a user interface with minimal screen interference, the company released a hypervideo authoring tool called HotVideo, which became a hypervideo system. The system adds anchor information to an object in a video by designating a start frame and an end frame in which a user wants an object as a key frame and linear interpolation of the object between the two frames. In addition, a frame with a hot anchor with a script information file set apart by the player is interpreted and displayed on the screen. 4 shows an example of authoring a hypervideo using a hypervideo authoring tool. The HotVideo system automatically tracks the objects between the two frames by using linear interpolation by specifying the start and end frames. It also uses standard video formats such as AVI and MPEG to enhance scalability. HotVideo Although the start and end frames are user defined, linear interpolation is used.

Most of the objects that want to insert anchor information in the video exist below the unit of shut and even if not, there exists the unit smaller than the unit of shut. However, HotVideo uses the user-defined key frame to distinguish shut, which is the smallest unit of anchor information insertion in a video. Therefore, for authoring using HotVideo, it is necessary to manually set the starting key frame and the ending key frame each time. In addition, in order to re-author the screen that has passed in time, it is inconvenient to review the video from the beginning.

Veon, Inc.

Beon Corporation was established in 1998 to build a single system called VeonSuite, which consists of hypervideo authoring tools, players and media servers suitable for the Internet environment, and provides various services such as e-commerce, advertising, and education system. The system was developed for the purpose of doing so.

Veon Suite is composed of the HyperVideo authoring tools, VeonStudio, VeonPlayer, and Media Server. Veon Studio is used together with the client program, Veon Player, to select specific objects in the video to be used as hot spot areas. You can connect various resources such as video, HTML document, sound, etc. to designated hot spots, and create web documents including hypervideos.

Veon Studio enables frame-by-frame authoring of objects in the frame in various shapes such as rectangles, circles, and polygons. It also adds automatic tracking for moving objects to automatically define hot spot areas for desired objects. Have In addition, the method of expressing a desired object when watching a hyper video also provides various types of interfaces such as showing only an outline, giving a translucent effect, or overlaying a specific image. In addition, like the interface taken by IBM, for the user who only wants to watch the video, the screen interference is minimized by changing the shape of the mouse when the mouse is located on the object.

Beon player, a client program used by the general user, plays a hypervideo created by the authoring tool, and is displayed in a web browser, so it can be shown to the user in various forms according to the design designed by the web developer. . In addition, Benon Suit stores various interaction information of users as log file and uses it for service. 5 shows the configuration of a Beon suit composed of a player, a media server, and an authoring tool.

However, Beon Studio, like HotVideo, also needs to set the start and end key frames each time. In addition, in order to re-author the screen that has passed in time, it is inconvenient to review the video from the beginning.

4. Gauri Information Communication Co., Ltd.

In 1998, it applied for a patent for a video hypermedia system in Korea. This patent uses SunVideo board based on SparcStation hardware to encode / decode video based on JPEG video so that the similarity between frames can be compared and authored and played in shut unit. Referring to FIG. 6, after reading a video (S1), a shut is searched to find a start frame of a shut. (S2) If a shut is found, a check is performed to determine the end of the video (S3). Save to a file (S10) and ends. Otherwise, in the middle of the video, an anchor is set in the first frame of the found shut (S4). The anchor is automatically set up using the interpolation method until the last frame of shut (S5). The user of the authoring tool checks whether the anchor information of the last frame is properly set (S6), and if it is properly set, proceeds to reading the video (S2) and repeats until the end of the video. If the anchor information of the last frame is not properly set, the anchor information of the last frame is corrected (S7) and interpolation is performed on the intermediate frames again. As a result, it is determined whether the anchor information of the intermediate frames is properly set (S8), and if it is properly set, the process proceeds to the step of reading a video (S2) and repeats until the end of the video. If the anchor information of the intermediate frames is not properly set, after splitting the corresponding shut into several shuts (S9), the anchor information is set to the first frame (S4) to perform anchor information insertion.

This series of processes is configured to perform the authoring process in the shut unit, which is the minimum unit of the anchor information insertion process in the video. However, it is inconvenient to review the video from the beginning in order to re-author the screen that has passed in time. In other words, it is inconvenient to edit an excessively shut shut because the writing is performed in a shut-down step that progresses in time. In addition, the conversion to JPEG files requires the use of special hardware or even takes a lot of encoding / decoding time. This problem has the same problem for client computers used by users other than the authoring tool.

5. Samsung Electronics Co., Ltd.

Samsung Electronics Co., Ltd. also has a patent published in 1998, the contents of which relates to a video signal recording and playback method including hyper-anchor information. In this method, hyperanchor information of a specific frame has information at both ends of a diagonal line in a rectangular area, and a method of recording and reproducing a video signal using the same is disclosed. However, this method also mentions the method of setting up the area and displaying the set area in the authoring tool and the player, and is far from the shut down process.

6. Korea Firstec

Korea Firstec has developed a hypervideo system under the name of DIMPEC SYSTEM (Direct Motion Picture Electronic Commerce System), which is a video direct electronic commerce system, and applies it directly to e-commerce service. This system is a service that interworks with a shopping mall by obtaining information of a specific object in a desired frame after temporarily storing the frame by selecting a mouse when there is a desired product while watching a drama. DIMPEC utilizes the anchor information without any change in the video or the mouse by minimizing interference on the video. The DIMPEC player is included in the web browser for the internet environment, and the authoring tool is not open to the public, and is currently operating in conjunction with the drama mall (http://www.dramamall.co.kr). 7 illustrates a process of linking a video frame to a shopping mall. In FIG. 7A, when a desired product is found while watching a video, a screen is clicked to temporarily store it, and the screen is stored in a temporary space and displayed on the right side of the player. If you select the one displayed on the right side of the player, the stored screen is created as another window of the browser, and when a desired object is selected in the new window, it moves to the shopping mall and purchases the product as shown in (c).

However, the DIMPEC system is created for service purposes, and the authoring tool is not open to the public, and the player can also indirectly interfere with the video by creating additional windows, and the user cannot know the authored information desired. The desired product may not be provided and accordingly has a disadvantage of storing unnecessary screens.

Related research on hypervideo systems has been conducted in many places, including Andersen Consulting LLP, Progressive Networks, and Microsoft, in addition to the above-mentioned system. have.

references

[Sawhney 96] N. Sawhney, D. Balcom, and I. Smith, ″ HyperCafe: Narrative and Aesthetic Properties of Hypervideo, ″ Proceedings of Hypertext '96: Seventh ACM Conf. on Hypertext, Washington DC, pp. 1-10, 1996.

[VEON] Veon, inc., ″ Streaming Hypervideo and Dynamic Hypervideo, ″ International Application Published under the Patent Cooperation Treaty (PCT) [WO 99/10822], 1999

[IBM1] International Business Machines Corporation, ″ Method for Marking the Location of a Video Hot Anchor, ″ Domestic Patent Publication [Special 1998-041946], 1998.

[IBM2] International Business Machines Corporation, ″ Method and Apparatus for Integrating Hyperlinks into Video, ″ Domestic Patent Publication [Spec. 2000-0016192], 2000.

[Gauri98] Gauri Information & Communication Co., Ltd., ″ Video Hyper Media System, ″ Domestic Patent Publication [98-10873], 1998

[Samsung 98] Samsung Electronics Co., Ltd., ″ Video Signal Recording and Playback Method Including Hyper Anchor Information, ″ Domestic Patent Publication [Special 1998-0023624], 1998

In general, an object to set an anchor in a video exists in a shut unit. Even if there are objects that want to anchor at the same time in two consecutive shuts, their positions cannot be retrieved automatically. Therefore, the minimum unit for anchoring an object in a video is shut down. However, in the conventional hypervideo system, as described above, in order to author streaming hypervideo, a method of manually setting two frames, a start frame at which anchor information starts and an end frame at which anchor information ends, is mostly used. . Meanwhile, in the case of Gauri Information Communication, a shut is detected based on the similarity between frames, and the start and end frames of the detected shut are extracted and used to insert the anchor information. However, even in this case, as shown in the flowchart of FIG. 6, it is difficult to insert anchor information for a shut that has already passed in time by inserting anchor information in a sequential order of shuts. In addition, in the process of inserting the anchor information after the detection of the shut, if the object to insert the anchor information for the intermediate frames in the shut is not properly set by interpolation, the method includes forcibly dividing the shut (S9). An object that tries to insert anchor information in a shut may not necessarily exist from the start frame to the end frame of a shut, but the transition of a traditional shut down of a screen switching by a special effect such as a camera transition or a fade in, fade out, or dissolve This is a violation of the classification. Moreover, because it is configured on special hardware that uses SpcarcStation's SunVideo board, it is less general and requires expensive system.

Accordingly, the present invention has been invented to solve the above problems, to perform a shut-down operation in a short time without the need for special hardware, and to move to the corresponding shut by selecting the representative frame of the shut even for the shut already passed. It provides easy authoring process without additional professional training process, so it is easy to provide streaming hypervideo system and automatic control method using scene change automatic detection that can maximize the convenience of streaming hypervideo authoring. There is this.

It is still another object of the present invention to perform a streaming hypervideo authoring process more conveniently and quickly, and to produce a result as a web document, for example, an HTML document. Through the edited web document, the end user can watch without special hardware and automatically provide the anchor information on the user interaction generated during the viewing process without any interference on the screen to maximize the spreading and utilization of streaming hypervideo. It is an object of the present invention to provide a streaming hypervideo system using detection and a control method thereof.

In order to achieve the above object, the present invention provides a video analysis step of performing a video transition automatic detection function in a short time by using the DirectShow technology of Microsoft while playing a video in a streaming hypervideo authoring tool, Selecting any frame among the analyzed video representative frames and selecting a frame in which an object to which anchor information is to be inserted for all or part of the scene corresponding to the selected frame exists; and automatically selecting a specific frame for the selected frame. Tracking, inserting anchor information into the frame of the tracked object and the corresponding position of the frame, saving the authored anchor information to a file, and organizing pages of frequently used web documents. Have a layout of document pages such as 2 split and 2 split Streaming hypervideo system and its control using automatic scene change detection, which can be selected immediately or has a function of splitting and merging web documents arbitrarily horizontally or vertically and naming each frame within a web page. Provide a method.

1 is a control method of a conventional hypervideo system (HyperCafe)

2 is a configuration diagram of a system of a conventional HyperCafe.

3 is a method for displaying a link area within a video in a conventional hypervideo system.

4 is a user interface of a tool for authoring hypervideo of a conventional hypervideo system (HotVideo).

5 is a configuration of a conventional hyper video system (VeonSuite)

6 is a flowchart of a conventional hypervideo system.

7 is a process of linking a conventional hyper video system (DIMPEC) and a shopping mall

8 is a block diagram of a streaming hypervideo system

9 is a control method of a streaming hypervideo authoring tool

10 is an image analysis process using a direct show

11 is a configuration of the video analysis function

12 is a configuration of a streaming hypervideo player

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

As shown in FIG. 8, the system of the present invention stores a web server system unit 2 for authoring streaming hypervideo, an authored streaming hypervideo and anchor information file, and a web document for viewing streaming hypervideo. It consists of a web server system unit 15 and a user interface unit 1 for playing streaming hypervideo obtained from a web server via a network and delivering anchor information.

The web server system unit 2 for authoring the streaming hypervideo may input a user input control signal, for example, an anchor information control signal for selecting and controlling a specific video, selecting a specific shut down, or specifying link information on the screen. The video panel analysis section 10 and the key panel section which perform a scene change over the key panel section 4, the specific video selected through the key panel section 4, and output the result to the output device section 3 ( 4) The object anchoring unit 11 for inserting anchor information at the position of the object while tracking a specific object in the shut selected by 4) and the content producing unit 12 and the author for generating the created anchor information as a web document. A hypervideo database (15) containing stored streaming hypervideo, anchor information files and web documents, and information stored in the hypervideo database (15) according to user requirements. It consists of a communication device section 8 for transmitting the beam. The object anchoring unit 11 also stores the frame information and the position information of the dynamic object in the temporary buffer 2 (14), and then hypervideos the contents of the web document and the temporary buffer 2 (14) generated by the web document management unit 12. It is stored in the database (15).

At this time, the contents stored in the temporary buffer 2 (14) is the additional information to be displayed on the screen according to the selection of the object, for example, the web page related to the object containing the anchor information and the frame containing the object containing the anchor information The start frame and the end frame, the position of the object including the anchor information existing in the frame, and the display method of the object including the anchor information are stored. Here, the display method of the object including the anchor information may be displayed as a basic shape such as a rectangle, a triangle, a polygon, or a bitmap image, and the displayed shape is also shown as an outline highlighting effect. You can give it a reversed or semi-transparent effect to match the screen, or make it visible only when the mouse is over it or always.

Meanwhile, the user interface unit 1 controls playback, pause, and stop of the streaming hypervideo obtained from the web server and the communication device unit 8 which obtains the streaming hypervideo, the anchor information file, and the web document. The display unit 5 determines whether anchor information is displayed by using the video control unit 5 and the key panel unit 4 such as the movement of a mouse and the anchor information file which is obtained in advance and stored in the temporary buffer 7 and outputs the determination result. And an anchor information management unit 6 which sends a signal to the output device unit 3 in the form defined in 3) and delivers the information to the user when the user selects the anchor information.

In streaming hypervideo authoring tools, compressed video data is generally used due to the low bandwidth of the Internet. However, in order to process various video data, it is necessary to configure not only compressed video data but also scene change detection for uncompressed video data. Therefore, the present invention uses an analysis method that can support as many file formats as possible using the direct show function under the Microsoft Windows operating system, rather than using the structural characteristics of a specific format. The Windows operating system uses the direct show function to play general video clips and play the same structure regardless of file formats such as AVI and MPEG.

10 illustrates an image analysis process of performing a scene change detection using a direct show, through a transform filter 911 performing a decoding operation regardless of a file format, entering an input of a renderer filter 912 and displaying the image on an actual screen. Becomes In the present invention, in order to support a scene change detection function, whether a video is compressed or uncompressed, a scene change analysis is processed by intercepting an image output from the output of a transform filter which is already decoded by a Windows operating system. As a result, various types of video data can be quickly authored, and even a newly released video format can detect scene change if a video codec exists and can be arbitrarily accessed. That is, the image resulting from the transform filter 911 proceeds to the image analysis module 920 at the same time as the renderer filter 912 receives. The obtained image is stored in the temporary buffer 921 and obtains an image of consecutive frames and is stored in another temporary buffer 922. The two images of the consecutive frames stored are determined by the scene change detection module 923 to determine whether a scene change has occurred, and when a scene change occurs, a representative frame of the scene is displayed on the screen (924).

In order to detect a scene change by comparing two consecutive images obtained by using a direct show technique, a scene change detection technique for an uncompressed video may be applied. The scene change detection process may be a pixel scene change detection method or a frame change scene detection method. In the present invention, as an example of the scene change detection process, a color histogram is used during the scene change detection process on a frame basis. Since the streaming hypervideo authoring tool uses arbitrary video data whose genre is not determined, the frame-based detection method, which is the most commonly used method, is used as an example.

In the scene detection method on a frame-by-frame basis, feature values of the entire frame such as a brightness histogram, a color histogram, and a histogram of a difference image may be used for scene change detection. In the case of the streaming hypervideo authoring tool, since random video data is used as a resource, a color histogram based on a color model is used to calculate an average of the difference in the histogram according to each component to detect a scene transition. We also used a double comparison method, Zhang's method, to detect progressive scene transitions (HJ Zhang, et. Al., ″ Automatic Partitioning of Full-Motion Video, ″ Multimedia Systems, Vol. 1, No. 1, pp. 10-28, 1993.). The double comparison method uses two thresholds of different sizes to detect gradual transitions caused by film editing effects. Use a high threshold for cuts with large differences in each component of the two frames, and a low threshold for progressive transitions where the differences in components are low and maintain similar magnitudes of variation over time. The frame was extracted.

In the present invention, by performing the scene change detection function together with the above-described direct show technology, it is possible to pre-index the moving picture and perform the shut-by-shot editing. This not only provides an effective authoring function for large-capacity videos, but can also help to add basic editing functions such as cutting / pasting videos. In addition, since most objects in the video exist in shut units, there is an advantage of providing a lower limit for object tracking in object tracking. 11 is a diagram illustrating an analysis process of a video. The video analysis process stores the indexing information of the video in the temporary buffer 1014 through the automatic scene change detection step 1011, and corrects the indexing information 1012 so that the wrongly detected scene change can be corrected manually. Include. After the scene change detection manual correction step, the author proceeds to authoring the streaming hypervideo 1013 on a shut basis, and updates the data of the web server 1015 with the streaming hypervideo anchor information file obtained as a final result.

The streaming hypervideo player obtains the anchor information file from the Internet, analyzes it, and displays it on the screen in a predetermined manner when the frame is played. The authoring tool can display a polygon outline, a polygon with a translucent effect, or a bitmap image on the screen for displaying the screen of the player, and can be displayed only when the mouse is positioned on the object. 12 is configured to minimize the user's interference by displaying on the screen in two steps. This process is to examine and determine the user's motion before going to the video renderer 1115, which is displayed on the screen. First, the anchor information data in the web server is stored in the temporary buffer 1116 in advance, and it is determined using the anchor information in the frame currently being played using the same. In addition, if the location of the mouse is located in the anchor information area (1113), the user can select the display 1114 in a specific form determined on the video screen. This configuration method is intended to provide an effective interface for both users who only want to watch a video and users who want to acquire information in the video.

On the other hand, the content production unit provides a function of providing a web document of a basic form to reconstruct an existing video into a streaming hypervideo in a streaming hypervideo authoring tool and publish it on the web. This function provides web administrators with the basic framework for autonomous web page composition. The browser screen can be divided into the desired form. The most basic page type is divided into two frames. It is divided into a frame that can be included and a frame that can be moved to anchor information of streaming hypervideo. In addition, it is possible to easily configure web pages by designating frequently used page layouts and to freely divide or merge the frames within the pages, so that web designers can easily decorate web documents including streaming hypervideo.

As described above, the present invention enables the authoring operation in the shut unit, which is the minimum unit of the anchor information insertion or the unit of the shut down unit, by performing the automatic scene change detection in the streaming hypervideo authoring tool, as well as in the object tracking of the object anchoring unit. This has the effect of providing a lower limit so that object tracking no longer proceeds at the end of the scene. In addition, for the past shuts, the representative frame of the shut can be selected and moved directly to the shut for editing, and the authored result can be directly generated as a web document having a layout desired by the author.

Another feature of the present invention is to display the anchor information in two steps by minimizing the interference of the screen in order to satisfy various conveniences of users who watch streaming hypervideo, and users who only want to watch the video and users who are interested in the anchor information in the video. It has the effect of effectively delivering streaming hypervideo to everyone.

By doing so, in addition to the existing hypervideo system, hypervideo authoring can be accessed and edited more quickly and intuitively in the shut unit, which is the minimum unit of anchor information insertion. In addition, the streaming hyper-video system can be used as a core technology for interactive multimedia solutions such as Internet TV or interactive TV, and can be efficiently applied to various interactive multimedia services such as e-commerce, VOD service, and online education.

Claims (5)

Before analyzing the authoring streaming hyper-video in the streaming hyper-video system, a video analysis unit including automatic transition detection, object anchoring unit for tracking the object in the shut detected in the step and insert the anchor information in the frame and location, A streaming hypervideo system having a streaming hypervideo authoring tool comprising a content production unit for generating an authored streaming hypervideo as a web document. The streaming hypervideo system of claim 1, wherein the video analyzing unit is configured to automatically perform scene change detection on various videos through a screen direct access function. The apparatus of claim 1, wherein the video analyzer includes a scene change detection function, displays a representative frame of the detected scene on a screen, and when selecting the representative frame of the scene, starting from the start frame or any intermediate frame of the scene. Streaming hypervideo systems that can do the job. The method of claim 1, wherein the object anchoring unit performs object tracking on a shut-by-shot basis to automatically track an area defined by a user for an object existing in a corresponding scene, inserting anchor information at a time and a spatial location, and inserting the inserted area. Streaming hypervideo systems that specify settings such as circles, squares, polygons, geometric shapes such as circles, rectangles, and polygons, or display translucently with a specific bitmap image or screen, and output the contents set together with hypervideo playback. If there is anchor information in the video that the user is watching, the screen shape is minimized to customers who want to watch the video only by changing the shape of the mouse or displaying on the screen, and the customer who is interested in the anchor information in the video Streaming hyper-video system, when the mouse pointer moves to the area with anchor information in the video, in the specific form as defined in the above section.