KR20020015123A - Multimedia library system for individual - Google Patents

Abstract

PURPOSE: A personal multimedia library system and a library storing and browsing method are provided to easily compose a multimedia library corresponding to a personal environment and to enable a user to search data by indexing data and storing the indexed data in a home storage. CONSTITUTION: A personal multimedia library system comprises a receiver(42) receiving the multimedia data delivered from a terminal, a home storage(41) storing the received data, an indexer(43) indexing the multimedia data, and a browser(45) searching and browsing the indexed multimedia data. The terminal is one of a TV, a digital movie camera, a digital still camera, and a cellular phone having the function of a temporal storage. With respect to a personal library storing and browsing method using a music program broadcasted via a TV, a music program is recognized. A section of a music is obtained from the program. Additive information of the music section is obtained. Respective music broadcasting sections are indexed and stored. A user selects a specific section among the indexed music broadcasting sections. The selected section is displayed. Therefore, the user searches and browses the desired data at a desired timing.

Description

Personal multimedia library system and method for storing and browsing personal multimedia library using such system {Multimedia library system for individual}

The present invention builds a personal library by receiving and indexing and storing various types of digital data from a mobile device such as a movie camera or a fixed device such as a TV, and a personal multimedia library system that can search and display it again. It is about.

In the related art, many methods for indexing moving picture media data to search or display only a portion desired by a user have been introduced.

For example, by segmenting arbitrary video data in units of shots and grouping the video data into content centers again, a user may enable scene-oriented video search and display.

In addition, the news video may be indexed for each subject to display only an event section of a portion desired by the user.

Most user interfaces of the existing technologies provide a way for a user to easily find a desired section by arranging key frames representing each section.

By applying such a conventional video indexing technology to data that can be transmitted from a mobile device such as TV broadcast data or a movie camera, it is possible to manage and use a personal multimedia data library.

The present invention relates to such a personal multimedia library system, and proposes an environment capable of receiving video data from various devices centering on the storage where the library is to be constructed, and a device environment for automatically or semi-automatically indexing data. .

In addition, we propose a data management method for effectively configuring a personal library, a device to help indexing, and a specific embodiment applying such an environment.

1 is a block diagram showing a schematic configuration of a personal multimedia library system of the present invention.

2 is a flowchart illustrating a process of storing and browsing a personal music album using a music broadcasting program of a TV according to an embodiment of the present invention.

3 is a flowchart illustrating a process of storing an educational broadcast program in a personal library according to another embodiment of the present invention for storing and browsing a personal multimedia library.

FIG. 4 is a flowchart illustrating another example of a method for storing and browsing a personal multimedia library according to the present invention, wherein a process of producing a celebrity album capable of producing an album based on the characters of celebrities appearing on a TV is provided.

5 is a view illustrating another embodiment of a method for storing and browsing a personal multimedia library according to the present invention, wherein indexing information for personalizing a movie acquired by using a movie camera as a terminal, such as a home video, may be automatically acquired. Flowchart showing the process.

The present invention relates to a system for building, maintaining and using a personal multimedia library system, and to a specific application applied on such a system.

The present invention provides a terminal device capable of acquiring and transmitting multimedia data to a designated address;

Receiving means for receiving the multimedia data transmitted from the terminal device and storing the same in mass storage means;

Mass storage means for storing the received data,

An indexer for indexing multimedia data,

Personal multimedia library system consisting of a browser for searching and browsing the indexed multimedia data.

The transmission means is characterized in that it comprises both wireless communication or wired communication.

The terminal device capable of acquiring and transmitting the multimedia data may be one of a TV, a digital movie camera, a digital still camera, and a mobile phone in which a camera function is combined.

The terminal apparatus capable of acquiring and transmitting the multimedia data may further include temporary storage means for temporarily storing the acquired data.

The terminal device further including the temporary storage stores data acquired in the temporary storage means and transmits the data to a designated address when a user request or when the amount of stored data exceeds a threshold associated with the capacity of the temporary storage. do.

The terminal device may further include means for acquiring data indexing information or information to assist in indexing simultaneously with data acquisition.

The indexing information that can be obtained at the same time as data acquisition is

When shooting with a digital movie camera, shot information that can be automatically saved using the On / Off signal, time information at the time of data acquisition provided by a mobile device such as a digital camera, and shooting by a mobile device such as a digital camera It is characterized in that it is one of the information that can be stored as data after inputting necessary information as a voice signal and converting it into text through speech recognition again.

An indexer for indexing the multimedia data,

Means for acquiring shot information of a moving picture, means for grouping shots using semantic or physical information, and means for assigning semantic information to each shot or group of shot groups.

The browser for retrieving and browsing the indexed multimedia data includes search means capable of searching specific data, or parts of data, using semantic and physical information indexed and stored by the indexer, and the semantic information. And means for browsing in association with the stored data.

In addition, when the terminal device is a TV, in the personal library system using a music program that is aired on the TV,

Recognizing a music program;

A unit section acquiring step of acquiring a section in which one piece of music is broadcast in one recognized program;

An additional information acquiring step of acquiring additional information added to one music broadcasting section;

A storage step of indexing and storing each music broadcasting section;

A selection step of selecting a desired section among the indexed and stored music broadcasting sections;

A personal music album storage and retrieval method comprising a display step in which a selected specific music broadcasting section is displayed.

In the acquiring the additional information, the additional information added to one music broadcasting section may include one of a singer information, song meaning information such as a song title, and one or a combination of key frames representing the corresponding section. do.

In the additional information acquiring step, the key frame representing the corresponding section is characterized by using a scene in which the singer's face is caught close up.

In the selecting step of selecting a section desired by the user, the search is performed by using additional information, or by arranging keyframes for each section and allowing the user to select the section.

The unit section acquisition in the unit section acquisition step may include broadcast transmission data such as EPG or caption, or a combination of shot segmentation, audio, face region extraction, and recognition technology.

In addition, when the terminal device is a TV, in the personal multimedia library system using the educational broadcast data broadcast on the TV

Recognizing educational programs;

Segmenting sections for each unit in a recognized program; and

An acquisition step of a representative frame representing a segment segmented into one unit,

A subject information acquisition step of adding subject information of the education covered in the segment segmented into one unit,

Storing the indexed data,

A selection step of selecting a desired section among the indexed and stored sections;

A personal education broadcast storage and retrieval method comprising a display step of displaying a selected specific education broadcast section.

In the representative frame acquiring step, the representative frame is characterized by selecting a frame in which the subject is displayed as text.

The subject information acquiring step may be obtained by using broadcast transmission information such as caption and EPG or by applying character region extraction and character recognition technology.

In addition, when the terminal device is a TV, in a personal multimedia library system using a broadcast program broadcasted on TV,

Specifying which screens the user will configure as a library,

Saving the specified screen as an image in temporary storage,

Adding semantic information to a specified image,

Transmitting the acquired scene image and semantic information to a mass storage means;

And storing the transmitted scene image and semantic information in accordance with the indexing structure.

In addition, when the terminal device is a TV, in a personal library system using a broadcast program that is aired on a TV,

Recognizing the scene where a person appears;

Acquiring a scene in which the character appears to satisfy a certain condition;

Identifying the characters in the acquired scene,

Acquiring person information such as the id of the person;

A celebrity automatic album storage method comprising the step of constructing a celebrity album by using the acquired scene image and the character information.

In the step of acquiring a scene in which the character appears to satisfy a specific condition, the specific condition may be set to appear close to the screen at a ratio equal to or greater than an arbitrary threshold value.

In the personal multimedia library system using the digital camera,

Video data acquisition step using a digital camera,

Obtaining shot index information using an on / off signal,

A time information acquisition step using the shooting date information,

Indexing the acquired data using the same;

Personal library storage and browsing method comprising the step of searching or browsing the indexed data using the indexing information.

The method may further include storing and browsing a personal multimedia library including recognizing a user voice and inputting a main keyword.

The present invention having such a feature will be described with reference to the embodiments shown in the accompanying drawings.

1 is a diagram showing an entire system configuration block for building a personal library.

First, a device capable of acquiring a moving image, such as the TV 10 or the movie camera 20, stores the moving image data acquired through the broadcasting station 50 or the recording in a temporary storage means included in each device. .

In this case, indexing information or information that may help indexing may be stored together, for example, as follows.

In the case of the TV 10, EPG or caption information is transmitted, and when receiving it, this information is used as main information at the time of indexing.

In the case of the movie camera 20, the on / off signal can be used directly as a shot index when shooting.

In addition, the time information provided by the movie camera 20 is also used as the main information, and by including the voice recognition means, the user can add a simple keyword or annotation by voice.

The data acquired with this information is automatically transmitted to the mass storage means (home storage) 41 when there is no margin by the user's command or by the data stored in the temporary storage means (2).

In this way, the terminal apparatus 10, 20, 30 capable of acquiring data includes a transmitter 1 and a transmitter controller 3 capable of controlling and transmitting data of the storage means 2. And Temporal Storage (2).

In this case, the data is transmitted by wire in the case of a terminal such as the TV 10, and in the case of a mobile terminal such as the movie camera 20, it may be wirelessly transmitted using the Internet.

The data transmitted as described above is stored in the mass storage means 41 of the system device 40 installed at home. In this case, the transmitted data is indexed and stored using the already obtained index information and additionally extracted index information.

For this purpose, not only the mass storage means 41 but also a receiver 42 for receiving the transmitted data and an indexer 43 for indexing the transmitted data are required in the home.

The indexed and stored data is used when a user wants to see specific data.

In order to select specific data at a time desired by the user, the system device 40 includes a searcher 44 and a browser 45 to provide a user with a search or browsing user interface. In order to select data, the data may be searched using semantic information such as a person's name, or may provide a method of viewing and selecting a desired section by arranging representative frames for each section.

As described above, when the acquired data is indexed, acquisition of index information may be partially performed by the terminal apparatuses 10, 20, and 30, or may be performed by a separate indexer 43 after being transmitted.

Meanwhile, FIG. 2 is a flowchart illustrating an execution process of a method of storing and browsing a personal music album using a music broadcasting program of the TV 10 as an embodiment of the present invention.

First, a process of distinguishing whether a program received from a TV is a music program or not is performed.

In this case, if the user selects a specific music program, only the corresponding program may be distinguished among the music programs.

In this process, the program can be divided into EPG.

When it is confirmed that the received broadcast is the corresponding program, the next indexing process is performed.

First, we need to segment the program into music units.

This process is, for example, in the case of a song program, which indexes data in units of a section where a song is broadcasted. If the corresponding information can be obtained from EPG or caption information, indexing can be performed more easily. If this information is difficult to obtain, this process is carried out using a more complex method.

For example, indexing is possible by first segmenting shots and then grouping shots into consecutive audio information units using audio information.

The indexed data is stored in the mass storage means 41. When the user's query is given, the indexed data is searched using the indexing information to display the result.

The query of the user may be made based on semantic information, for example, a search for a section in which a singer sang a song, or the section may be selected by user selection in a list of representative frames of each section.

Meanwhile, FIG. 3 is a flowchart illustrating a process of storing an educational broadcast program in a personal library as another embodiment of the present invention.

This embodiment shows an educational broadcast library system configuration process for enhancing the effectiveness of education by indexing educational broadcast programs by unit and subject and storing them for every date and then re-watching only the necessary sections again.

First, when a broadcast program is received, it is determined whether the program is an educational broadcast program, and if it is corrected, segmentation is performed for each unit, and then a representative frame is extracted for each section.

In this case, the segmentation for each unit may be applied to the character area extraction and the character recognition technology due to the characteristics of educational broadcasting, and the representative frame is set as the scene in which the subject character is displayed to easily find out which unit the user is.

The content of the recognized character can be used as keyword or text information of each unit, and the data is stored in the mass storage means 41 together with this information.

In order to search for the desired section, the user can use the section name or the topic (for example, a function type in the case of mathematics) as the query contents, or list the representative frames in which the topic of each section is well-known and select the section. Select to display it.

4 is a flowchart illustrating a production process of a celebrity album capable of producing an album centered on the characters of celebrities who appear on the TV 10 as another embodiment of the present invention.

First, it is checked whether the received program of the TV 10 is a target program.

If it is correct, only the section in which the person appeared is extracted and then a character identification is performed to confirm whether the person who appeared in the section is the desired performer.

A part of extracting a section in which a person appears may use a face region extraction algorithm and the like, and a method of identifying a performer may be performed by a face recognition technique and an empirical method.

Through this process, if the corresponding section is a section in which the desired performer appears, a process of storing a scene frame to be composed as an album as an image is performed.

At this time, the selection of the scene frame to be composed as an album may be arbitrarily extracted in the corresponding section, but may catch only the frame in which the performer's face is close up based on the size of the face area.

Character image stored in this process can be stored with the identification information (identification) extracted previously can be browsed and searched by the user.

As such, in FIG. 4, a system for automatically constructing a celebrity album is presented.

In addition to these methods, the same function can be performed by providing an interface that allows the user to manually save a desired frame as an image for use in an album, in which case the user can add additional semantic information that can be used for searching. It includes.

As another embodiment of the present invention, FIG. 5 is a flow diagram illustrating a process of automatically acquiring indexing information for personalizing a moving picture acquired by using the movie camera 20 as a terminal device, such as a home video. It is a chart.

The image acquisition device including such a function is not limited to the movie camera. First, an example of a movie camera introduced in the drawing will include an image acquisition means 21 and an image data buffer 22 to store the same as a conventional digital movie camera.

In addition, an indexing buffer 24 is further included to further add indexing information, and the on / off control device 25 and the indexing buffer 24 are connected to the bus to store the on / off signal as shot information as the indexing information. The On / Off control device 25 transmits this information to the indexing buffer 24 as shot information whenever there is an On / Off signal.

The other information stored in the indexing buffer 24 is photographing date information, and in addition, the voice acquiring means 22 and the voice recognition means 23 may be used to include text information such as a main word.

When more than a certain amount of data is stored in the data buffer 26 and the indexing buffer 24, the data is automatically transmitted to the address given by the transmission means 29, and the transmitted data is emptied in the indexing buffer 24.

A control unit (CU) 27 is included to control such an operation, and an address storage space 28 in which an address to be transmitted is stored is included.

The indexing buffer 24, the data buffer 26, and the address storage space 28 described above may use the same memory or different memories.

As described above, the present invention indexes and stores various data in a mass storage from a mobile image acquisition device such as a TV broadcast or a movie camera, so that the user can search and look for desired data when desired. It will provide a system to easily organize the library.

In particular, it is possible to produce and organize data sensitive to a particular generation's preferences, such as a music album or a celebrity album, and to provide a desired service for a portion that requires selective repeat viewing, such as an educational broadcast, according to a user's needs.

In this age of multimedia contents, the field of multimedia content is very important, and it is significant in that it provides a way to create and manage personal content using a general home appliance system. It's no longer just recording, it's about creating your own content, and using the concept of transferring from local storage to mass storage to manage the hardware limitations of memory limitations.

Claims (21)

Receiving means for receiving the multimedia data transmitted from the terminal device and storing the same in mass storage means; Mass storage means for storing the received data, An indexer for indexing multimedia data A personal multimedia library system comprising a browser for searching and browsing indexed multimedia data. The personal multimedia library system of claim 1, wherein the terminal device capable of acquiring and transmitting the multimedia data is one of a TV, a digital movie camera, a digital still camera, and a mobile phone in which a camera function is combined. The personal multimedia library system according to claim 1 or 2, wherein the terminal apparatus capable of acquiring and transmitting the multimedia data further comprises temporary storage means for temporarily storing the acquired data. 4. The terminal of claim 3, wherein the terminal device further including the temporary storage means stores a data obtained in the temporary storage means, and there is a user's request or a designated address when the amount of stored data exceeds a threshold associated with the capacity of the temporary storage means. Personal multimedia library system, characterized in that for transmitting data. 3. The personal multimedia library system according to claim 1 or 2, wherein the terminal apparatus further comprises means for acquiring data indexing information or information to assist in indexing simultaneously with data acquisition. The method of claim 5, wherein the indexing information that can be obtained at the same time as the data acquisition, When shooting with a digital movie camera, shot information that can be automatically saved using the On / Off signal, time information at the time of data acquisition provided by a mobile device such as a digital camera, and shooting by a mobile device such as a digital camera Personal multimedia library system, characterized in that one or a combination of information that can be stored, such as data after inputting the necessary information to the voice signal at the time, and converted back to text through speech recognition. The method of claim 1, wherein the indexer for indexing the multimedia data, Means for acquiring shot information of a moving picture, means for grouping shots using semantic or physical information, and means for assigning semantic information to each shot or group of shot groups. Library system. The method of claim 1, wherein the browser for searching and browsing the indexed multimedia data may search for specific data, or a portion of data, using semantic and physical information indexed and stored by the indexer. And means for browsing said semantic information in association with stored data. In a personal multimedia library system using a music program broadcast on TV, Recognizing a music program; A unit section acquiring step of acquiring a section in which one piece of music is broadcast in one recognized program; An additional information acquiring step of acquiring additional information added to one music broadcasting section; A storage step of indexing and storing each music broadcasting section; A selection step of selecting a desired section among the indexed and stored music broadcasting sections; And storing and retrieving a personal music album, wherein the selected music broadcasting section is displayed. 10. The method of claim 9, wherein in the acquiring the additional information, the additional information added to one music broadcasting section is one or more of singer information, song meaning information such as a song title, and key frames representing the corresponding section. Personal multimedia library storage and browsing method characterized in that the combination. The personal multimedia library storage and browsing method according to claim 10, wherein in the additional information acquisition step, a key frame representing a corresponding section uses a scene in which a singer's face is close up. The personal multimedia library storage according to claim 9, wherein in the selection step of selecting a desired section by the user, the search is performed by using additional information or the user selects a keyframe after listing each section. And browsing method. 10. The personal multimedia apparatus according to claim 9, wherein the unit section acquisition is performed by using broadcast transmission data such as EPG or caption, or a combination of shot segmentation, audio, face region extraction, and recognition technology. How to save and browse your library. In the personal library system using the educational broadcast data broadcast on TV, Recognizing educational programs; Segmenting sections for each unit in a recognized program; and An acquisition step of a representative frame representing a segment segmented into one unit, A subject information acquisition step of adding subject information of the education covered in the segment segmented into one unit, Storing the indexed data, A selection step of selecting a desired section among the indexed and stored sections; A personal multimedia library storage and browsing method comprising a personal education broadcast storage and retrieval process comprising a display step of displaying a selected specific education broadcast section. The personal multimedia library storage and browsing method according to claim 14, wherein in the representative frame obtaining step, the representative frame selects a frame in which a subject is displayed as text. The personal multimedia library storage and browsing method of claim 14, wherein the obtaining of the subject information comprises acquiring broadcast transmission information such as caption and EPG, or applying character region extraction and character recognition technology. In a personal multimedia library system using a broadcast program broadcast on TV, Specifying which screens the user will configure as a library, Saving the specified screen as an image in temporary storage, Adding semantic information to a specified image, Transmitting the acquired scene image and semantic information to a mass storage means; And storing the transmitted scene image and semantic information according to an indexing structure. In a personal multimedia library system using a broadcast program broadcast on TV, Recognizing the scene where a person appears; Acquiring a scene in which the character appears to satisfy a certain condition; Identifying the characters in the acquired scenes, Acquiring person information such as the id of the person; A personal multimedia library storage and browsing method, comprising the step of constructing a celebrity album using the acquired scene image and character information. 19. The personal multimedia library storage and browsing according to claim 18, wherein in the step of acquiring a scene in which the character appears to satisfy a specific condition, the specific condition is set to appear close to the screen at a ratio of a threshold or more. Way. In the personal multimedia library system using a digital camera, Video data acquisition step using a digital camera, Obtaining shot index information using an on / off signal, A time information acquisition step using the shooting date information, Acquisition data indexing step, A method of storing and browsing a personal multimedia library comprising searching for or browsing indexed data using indexing information. 21. The method of claim 20, further comprising the step of recognizing a user's voice and inputting key keywords.