KR100713501B1 - Method of moving picture indexing in mobile phone - Google Patents

Abstract

The present invention relates to a method for indexing a specific keyframe for searching and browsing a video on a mobile communication terminal. More particularly, the present invention relates to a method for receiving indexing by displaying a corresponding keyframe having a scene change in a thumbnail form among video frames. It is about. To this end, according to the present invention, a scene transition frame detecting step of detecting a key frame in which the scene change has occurred among the key frames of the digital video and recognizes the scene change frame, and selects a predetermined number of representative frames to be used for indexing from the scene change frame. A representative frame selection step, a thumbnail frame generation step of generating a thumbnail frame having a reduced capacity of the selected representative frame, and an indexing step of registering indexing of the representative frame.

 Video, MPEG, Index, Indexing, Frames, Thumbnails, Browsing, Mobile Terminals

Description

How to index digital video on mobile devices {METHOD OF MOVING PICTURE INDEXING IN MOBILE PHONE}             

1 is a block diagram of a mobile communication terminal for video indexing according to the present invention.

2 shows each database table in the index storage.

3 is a flowchart illustrating a process of indexing a video in a mobile communication terminal according to the present invention.

4 is a flowchart illustrating a process of detecting a scene change frame.

5 is a bar graph of the difference between the luminance block average brightness values for each frame.

FIG. 6 is a diagram illustrating a representative frame displayed in a thumbnail form on a mobile communication terminal window.

The present invention relates to a method for indexing a specific keyframe for searching and browsing a video on a mobile communication terminal. More particularly, the present invention relates to a method for receiving indexing by displaying a corresponding keyframe having a scene change in a thumbnail form among video frames. It is about.

In recent years, mobile communication terminals have made great strides and can record and play large-capacity video clips over one hour in the terminal. In order to search and browse such a video, a user has to play a rewind / rewind of the video repeatedly.

In detail, in order to search and browse a video in a mobile communication terminal, a search and a browsing are performed by extracting a key frame in a scene unit. In other words, the most basic search and browsing method is to use the properties of the start time and the end time of the video. The keyframe of a specific time zone selected by the user is displayed one-dimensionally, and the search and browsing is performed in a form asking whether the user wants to play or not. lost.

However, the conventional one-dimensional interface has a disadvantage in that it is difficult to express a lot of content in a limited display space because a large number of keyframes must be displayed at once to convey the entire content. In particular, since similar scenes are repeated in a genre of content such as a movie or a drama, there is a problem in that the user cannot easily find a desired scene by providing a lot of unnecessary information to the user.

As a result, video content retrieval on the conventional mobile communication terminal is only a level that the mobile communication terminal derives only one-way still image search results for the user. Also, in browsing, the keyframe extracted from the video data was used for the browsing, but the keyframe was not efficiently represented during the browsing process. This search and browsing technique is not an efficient search method because it is a search and browsing method using an independent tree formed by different mechanisms rather than using a single tree structure.

The present invention has been made to solve the above problems, the purpose of performing a search and browsing by detecting a keyframe that has been changed based on a specific threshold in order to search and browse the video displayed on the mobile communication terminal do. In addition, by selecting a frame to be displayed in the form of thumbnails on the terminal among the scene-transformed keyframes, the indexing of text / voice is registered so that later searching and browsing can be easily performed by using indexing.

In order to achieve the above object, the present invention provides a scene transition frame detecting step of detecting a scene transition frame among key frames of a digital video and recognizing it as a scene transition frame, and a predetermined number of representatives to be used for indexing among the scene transition frames. A representative frame selection step of selecting a frame, a thumbnail frame generation step of generating a thumbnail frame with a reduced capacity of the selected representative frame, and an indexing step of registering the indexing of the representative frame.

In addition, a method of detecting a keyframe in which a scene change has occurred in the scene change frame detecting step includes: a first step of reading an Nth keyframe and an N + 1th keyframe to determine an average brightness value of each luminance block; If the difference between the luminance block average brightness values of the Nth keyframe and the N + 1th frame obtained above exceeds a predetermined threshold, it is recognized that a scene change has occurred and the corresponding N + 1 key key is registered as the scene change frame. In the second step, if the N + 1st keyframe is not the last keyframe of the video, the N + 2th keyframe, which is the next keyframe, is read and the first and second steps are repeated.

In the above description, the key frame refers to a frame in which all images of the frame are compressed and stored in video compression. In general, video compression compresses an entire image of a corresponding frame by a key frame, and the images between the key frames are delta frames in which only the key frame and the changed image information are compressed. Therefore, if the video is stream-compressed only with keyframes, it has many advantages in terms of accurate frame-by-frame editing, fast search and cut editing, and interface editing, but has a low compression ratio.

An embodiment of the present invention will be described taking an MPEG compressed video as an example. Therefore, the present invention will be described using an I-frame functioning as a key frame in MPEG as an example, but the present invention can be implemented by using key frames in the compression algorithm even in a video having a general compression algorithm other than the MPEG compression algorithm. Will be self-explanatory.

Hereinafter, the detailed description of the preferred embodiments of the present invention will be described with reference to the accompanying drawings. It should be noted that the same components in the figures represent the same numerals wherever possible.

In the following description, specific details are set forth in order to provide a more thorough understanding of the present invention. It will be apparent to those skilled in the art that the present invention may be readily implemented by variations of these detailed descriptions.

1 is a block diagram illustrating a necessary block in a mobile communication terminal that provides an indexing function by detecting a representative frame in a video in the mobile communication terminal according to an embodiment of the present invention.

The I-frame storage unit 102 is a place where keyframes of a compressed video are stored, and an I-frame which functions as a keyframe in the MPEG compression algorithm is stored. Therefore, a spatially compressed I-frame in 8x8 block units is sequentially stored through a discrete cosine transform (DCT), and an intra-frame coded code in which an entire frame video is compressed among MPEG bit stream frames is stored. Only I-frames are stored in this buffer. If an MPEG video uses both an intra-frame I-frame and an inter-frame B-frame and a P-frame in which only the amount of information change from the previous frame is stored, that is, I- In case of an MPEG frame having an IBP GOP structure using all three frames, a B-frame and a P-frame, only an I-frame is extracted and stored in the I-frame storage unit 102.

The scene change detector 104 sequentially retrieves a plurality of I-frames stored in the I-frame storage to detect an I-frame having a scene change. There may be various methods for detecting whether there is a scene change, such as comparing brightness differences between I-frames. According to an embodiment of the present invention, the average brightness value of the luminance block of the previous I-frame and the luminance of the current I-frame may be used. The difference between the block average brightness values is compared, and when the difference exceeds a predetermined threshold value, a method of determining that a scene change has occurred is used.

If you express it as a formula,

는 n번째 프레임에서 휘도 블록의 평균 밝기 값을 의미한다. 프레임간 차

이 특정 임계치

를 넘게 되는 경우(

), n번째 프레임에서 장면 전환이 일어났다고 판단하고, n번째 프레임을 장면전환이 일어난 장면전환 프레임으로 설정한다. 상기의 함수는

n번째 프레임의 휘도 블록의 평균 밝기 값을 구하는 함수로서, 기존의 다양한 알고리즘을 통하여 휘도 블록의 평균 밝기 값을 구할 수 있음은 자명할 것이다. Here, M and N represent the total number of 8x8 blocks in the vertical and horizontal directions, respectively. For example, in the case of 352 x 288 MPEG, M = 44 and N = 36.

Denotes an average brightness value of the luminance block in the nth frame. Interframe difference

This particular threshold

Is greater than (

), it is determined that the scene change has occurred in the nth frame, and the nth frame is set as the scene change frame in which the scene change has occurred. The function above

As a function of obtaining the average brightness value of the luminance block of the nth frame, it will be apparent that the average brightness value of the luminance block can be obtained through various algorithms.

The representative frame selector 108 selects a representative frame from the scene change frames detected by the scene change detector 104. The representative frame is one selected from among the scene change frames detected by the scene change detection unit as a frame to be displayed on the window of the mobile communication terminal.

That is, if there are many scene change frames detected, it is inefficient to display all of them in thumbnail form in the mobile communication terminal window. Therefore, several representative scene change frames are selected and displayed in thumbnail form in the mobile communication terminal.

The number and method of the scene change frame (representative frame) to be displayed above may be arbitrarily selected, but are evenly selected as the representative frame among the detected scene change frames. There may be various embodiments for this, but in the present invention, first, the number of the representative frame is set from the user, and based on this, the number of the representative frame as the number of the set is selected from the scene change frame do. Detailed description of the embodiment will be described in more detail with the flowchart of FIG. 3 to be described later.

The thumbnail frame generator 110 generates a reduced image of the representative frame to be displayed in the window of the mobile communication terminal. If the representative frame is displayed in the form of a thumbnail in the window of the mobile communication terminal, there is a problem that the loading time is long due to the video capacity of the representative frame, so that the thumbnail frame is generated by reducing the capacity of the representative frame to the size of the thumbnail to display the thumbnail. to be. Accordingly, the thumbnail frame generation unit generates a thumbnail frame in which the image capacity of the representative frame is reduced and stores the thumbnail frame in the thumbnail frame database 118.

The indexing unit 106 is a place for inserting a simple text memo or voice memo into the extracted representative frame, and the user may index the text memo for each thumbnail type image displayed in the mobile communication terminal window. Therefore, in the future, the user can search for a specific representative frame in the form of a query using a touch pad or voice recognition, or play the desired representative frame from the indexed representative frame.

The index storage unit 112 includes an I-frame unique number database 116 storing the original I-frame unique number of the selected representative frame, a thumbnail frame database 118 storing thumbnail frames reduced in capacity of the representative frame, The indexing database 114 stores the indexing of the representative frame. The indexing database 114, the I-frame unique number database 116, and the thumbnail frame database 118 are respectively shown in the table table of FIG. As they are associated, they form a database.

For example, in the case of the first selected representative frame 202, the corresponding I-frame unique number (e.g., I567), the thumbnail frame unique number (e.g., S1) of the representative frame, the corresponding indexed text (or voice) The registration is stored.

3 is a flowchart illustrating a process of extracting and indexing a representative frame from I-frames of a video according to the present invention.

First, the mobile communication terminal detects 302 an I-frame in which a scene change is performed in a predetermined video. The scene change frame detection 302 determines that a scene change has occurred when the difference between the luminance block average values between the previous and subsequent frames exceeds a threshold.

The process of detecting the scene change frame 302 is described in more detail in the flowchart of FIG. 4.

First, a step 402 of reading the first I-frame second I-frame in the I-frame storage. The difference between the luminance block average brightness values between the first I-frame read and the second I-frame is calculated (404).

Calculation of the luminance block average brightness value is as described in the description of the scene change detection unit of FIG.

By the equation.

는 n번째 프레임에서 휘도 블록의 평균 밝기 값을 의미하는데, 상기 계산 결과 프레임간 차

이 특정 임계치

를 넘게 되는 경우(

), n번째 프레임에서 장면 전환이 일어났다고 판단하고, n번째 프레임을 장면전환이 일어난 장면전환 프레임으로 등록(408)한다.

Denotes the average brightness value of the luminance block in the nth frame, and the difference between the frames as a result of the calculation

This particular threshold

Is greater than (

), it is determined that the scene change has occurred in the nth frame, and the nth frame is registered as the scene change frame in which the scene change has occurred (408).

를 넘는 4번째 I-프레임(502), 9번째 I-프레임(504)이 장면전환 프레임으로 검출될 것이다.That is, as shown in FIG. 5, when the difference between the luminance block average brightness values exceeds a specific threshold, the corresponding frame is detected as a scene change frame.

A fourth I-frame 502 and a ninth I-frame 504 will be detected as the transition frame.

After determining whether the luminance block average brightness value exceeds the threshold 406, it is determined whether the read I-frame is the last I-frame of the video (408). If the frame read from the I-frame storage unit is the last frame of the video, the transition frame detection step ends. Otherwise, the next (next) I-frame is read (412) and the luminance block average brightness value of the corresponding frame is read. Repeat steps 404, 406 and 408 above if the threshold of.

On the other hand, after the detection of the scene change frame as described above, a representative frame is selected (304).

The representative frame selection selects a frame to be indexed from the detected scene change frames, and first selects the number of scene change frames to be indexed. To this end, a user is selected to determine how many scene transition frames are representative frames.

For example, 0.5%, 1%, 2%, and 3% of the detected scene change frames are used as the representative frame number, but the UI (User Interface) is provided to be selected by the user of the mobile communication terminal. If the movie's transition frame is 1000 frames, 5 for the 0.5% representative frame user setting via the UI, 10 for the 1% representative frame setting, and 20 for the 2% representative frame setting. If there is a representative frame setting of 3%, 30 is set as the number of representative frames.

When the number of representative frames is determined, as many scene transition frames as extracted representative frame numbers are extracted from the scene transition frames. The extraction is to be evenly extracted from the scene change frame.

For example, it is assumed that there are a total of P scene change frames and among them, Q representative frames (P * 1/100) representing 1% are detected.

;

;

; .....,;

;

;

; ......, ;

;

}의 집합형태로 장면전환이 일어난 P개의 장면전환 프레임이 검출되었고, 여기서 Q개의 대표 프레임을 추출하는 경우를 가정한다. In other words, {

;

;

; .....,;

;

;

; ......,;

;

} P scene transition frames in which a scene change has occurred in the form of a set are detected, and it is assumed here that Q representative frames are extracted.

,

,

의 장면전환 프레임을 선택한 후,

,

,

,

형태로 첫 번째 장면전환 프레임(

)의 이후 프레임(

), 마지막 장면전환 프레임(

)의 이전 프레임(

), 그리고 가운데 장면전환 프레임(

)의 이전 프레임(

)과 이후 프레임(

)을 차례대로 선택하여 총 Q개의 대표 프레임 군{

;

;

;

;

;

;

} (여기서 Q = 7)을 형성한다. In the above scene change frame set

,

,

After selecting the transition frame of,

,

,

,

In the first transition frame (

Frame after

), The last transition frame (

) 'S previous frame (

), And the center transition frame (

) 'S previous frame (

) And subsequent frames (

) In turn to select a total of Q representative frames {

;

;

;

;

;

;

} (Where Q = 7).

After the representative frame selection 304 is performed as described above, a process 306 of generating a thumbnail frame of the extracted representative frame is performed. In order to display the representative frames in a thumbnail form for searching and indexing in a mobile communication terminal window, a thumbnail frame having a reduced capacity in a thumbnail form is generated. When displaying the extracted representative frames in the form of thumbnails as they are, since a problem such as loading time delay occurs, a separate thumbnail frame having a reduced capacity is generated for the display in the form of thumbnails. Accordingly, the thumbnail frame is formed by capacity reduction of the representative frame, and various reduction methods such as size reduction or quantization rate reduction may be used.

After the thumbnail frame generation 306 as described above, the generated thumbnail frame is displayed to have an indexing process 308 for the corresponding frame. Indexing may be performed by text or voice in a corresponding frame. For this purpose, a mobile communication terminal provides a user interface (UI) for indexing. As shown in FIG. 6, the mobile communication terminal displays the thumbnail frame in the form of a thumbnail, and the user inputs his or her indexing 604 by clicking the thumbnail frame 602.

By the indexing as described above, the terminal user can easily play and watch from the indexed frame by text input or voice input without having to repeatedly repeat / rewind to see a specific portion of a predetermined video.

In the above description of the present invention, a specific embodiment such as a mobile communication terminal has been described, but various modifications can be made without departing from the scope of the present invention. Therefore, the scope of the present invention is not to be determined by the embodiments described above, but will be apparent in the claims as well as equivalent scope.

 That is, as described above, the present invention has an effect of efficiently searching and browsing movies on mobile communication terminals by performing keyframe-based text / voice indexing. In addition, by displaying the frame in the form of a thumbnail, the user can easily search and browse.

Claims (6)

A scene transition frame detection step of detecting a key frame having a scene change among the key frames of the digital video and recognizing it as a scene change frame; A representative frame selection step of selecting a predetermined number of representative frames to be used for indexing from the scene change frame; A thumbnail frame generation step of generating a thumbnail frame having a reduced capacity of the selected representative frame; An indexing step of registering the indexing of the representative frame; The thumbnail frame is a method of indexing a digital video on a mobile communication terminal, characterized in that the capacity is reduced to the thumbnail frame size. The method of claim 1, wherein the detecting of the keyframe in which the scene change occurred in the scene change frame detection step comprises: A first step of reading the Nth keyframe and the N + 1th keyframe to find the average brightness value of each luminance block; If the difference between the obtained average brightness value of the Nth keyframe and the N + 1th keyframe luminance block exceeds a predetermined threshold, it recognizes that there is a scene change and registers the N + 1th keyframe as the scene change frame. The second step, A third step of reading the N + 2 th keyframe, which is the next keyframe, and repeating the first and second steps if the N + 1 th keyframe is not the last keyframe of the video; Indexing a digital video on a mobile communication terminal comprising a. The mobile communication terminal of claim 1, wherein selecting the representative frame in the representative frame selection step selects the representative frame by a predetermined ratio of the number of the scene change frames detected in the scene change frame detection step. How to index digital video on the web. 4. The method of claim 3, wherein the predetermined ratio is provided by a user with a setting menu user interface (UI) to be set by a user of the mobile communication terminal. 5. The method of claim 1, wherein the indexing in the indexing step is indexing by text or voice. The method of claim 1, wherein after the indexing step, the registered indexed content is displayed together with a thumbnail frame.

Images