JPH10232884A - Method and device for processing video software - Google Patents

Abstract

PROBLEM TO BE SOLVED: To exactly grasp an entire video software by detecting a position to change the state of video software constitutive data and generating the summary of video software by extracting several representative images from the video software based on that information. SOLUTION: A superimposed character discrimination and storage part 4 discriminates and segments superimposed characters (caption). The information (timing information) of position to let the superimposed characters appear is sent to a cut discrimination and storage part 1. The superimposed characters are preserved on a hard disk 6 (or RAM) together with the information of this position. The cut discrimination and storage part 1 receives the information of position to let the superimposed characters appear and extracts a still picture from the prescribed position of correspondent cut (scene). The hard disk 6 stores the superimposed characters, still pictures and cut images clipped and designated by a viewer during the reproduction of representative images. According to a program stored in a ROM 3 and the hard disk 6, a CPU 2 executes processing for generating the summary.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a video software processing method and a video software processing device, and more particularly, to extracting a representative image from the contents of a two-hour drama recorded on a DVD (digital video disc), for example. For example, the present invention relates to a video software processing method and a video software processing device for displaying the whole video software in a short time, for example, 5 minutes, at high speed and accurately by displaying still images in sequence for several seconds per screen.

[0002] First, the usage of terms will be refused.
First, in this specification, video software means DVD stored video, digital TV (television) broadcast, ground wave T
V broadcast, video files used on the Internet and personal computers, computer graphics, movies, etc.
DVD, hard disk, magnetic tape, film, etc., regardless of their data format or medium format, a certain number of continuous images (frames),
It refers to everything that appeals to the viewer (user) about the scene, the atmosphere of the place, and the like (a certain degree excludes 4-frame cartoons, etc.).

In this specification, in some places including each claim, for example, “position information” and “position information”
So, I use several words that differ only in the presence or absence of the "no" character between them. This is the only meaning used in order to improve the readability and readability in the context, and the meaning is the same. It may have been obvious, but I will refuse it just in case. The same applies to “scene” and “cut”, “viewer” and “user”, “image” and “still image”, “screen”, and the like. Further, the terms may be abbreviated as appropriate. For example, video software is referred to as "video software", an operation including processing according to the present invention is referred to as "essence play", and an abstract generated by the processing is referred to as "essence", "video essence" or "VE". I have something to say.

[0004]

2. Description of the Related Art Modern society is moving toward the multimedia age. Information industries that support these eras include the following. Information primary industry = writer, newspaper reporter, photographer, etc. (information production) Information secondary industry = publishing / newspaper company, record company, broadcasting station, etc. (information processing) Information tertiary industry = bookstore, newspaper delivery, rental video, etc. Information distribution) Automation is progressing in information processing and distribution. For this reason, the number of people involved in information production is increasing.

The information industry is supported by the electronics industry, and various hardware and software that contribute to the advancement of the information industry are being provided. As a result of this,
The following situation is created.・ Information producers will increase, and large amounts of cobblestone content will be produced.・ Flood of video information due to development of new media such as multi-channel TV and DVD.・ Individual information intake time is limited and cannot be handled.

[0006] There is also a transformation of television (from fun to search). In other words, in the days when there were few channels, the whole family enjoyed common images. In the multi-channel era, personalization to search for the video you want to watch is progressing, and program selection, zapping (frequent switching of channels with a remote control), and use of multi-screens are becoming popular. I have.

[0007] By the way, an image is viewed along the flow of time, and it is necessary to take a certain amount of time to watch it. However, in the movement of the world as described above, the current state of time saving for video is as follows: Time shift by video: When you want to watch, watch the recording at your leisure.・ Time saving with video ・ ・ ・ ・ ・ ・ ・ Fast forward and digest play to see the main points.・ Two screen video ・ ・ ・ ・ ・ ・ ・ Enjoy sports and music at the same time. Play one and receive the other. (Video: Originally means "video", but here refers to VCR, recording, or using these).

[0008] As mentioned above, images are time consuming.
But human time is limited. If you can easily grasp the point of the video anywhere in a short time,
You can select only the ones you really want to see (works, programs, etc.) from many images and watch them slowly at your own pace. Such desires existed even before being called the multimedia age. Conventionally, the above-mentioned fast forward and the like have been used for that purpose. In recent years, a video deck equipped with a function called the digest play, that is, a fast-forward screen of about 1.5 to 2 times,
There is also a commercially available one equipped with a function of reproducing a sound at a normal reproduction speed together with the function of allowing the user to know the content in about half the normal time.

[0009]

However, the fast forward,
Conventional methods such as digest play are all complicated and extremely unsatisfactory for viewers (users).
That is, in these conventional methods, for example, according to the digest play, the sound can be reproduced at a normal speed, but the entire screen is fast-forwarded. For this reason, the image is blurred and difficult to see, and the viewer himself has to judge whether or not the point represents the point of the video software. In some cases, the ears too), fast forward, play, review, pause, and so on. In this method, if you want to find only specific scenes (cuts, scenes), rabbits are horns, and eyes and nerves are quite tired to try to grasp the whole picture of video software.

[0010] As described above, even though it is said that a large amount of video information is being supplied or that the era of advanced technology is beginning to occur, the ability of human beings and the freedom of human beings can be improved. This does not mean that time will increase significantly. Because of this, ・ I am not busy watching video software slowly.・ I want to quickly extract what I want to see from a large amount of video software.・ I want to see many video software freely in a short time at my own pace. There is also a new demand for this.

It is an object of the present invention to provide a method which responds to such human time-saving needs and which enables a new method different from the conventional method to accurately grasp the entire video software.

[0012]

In order to achieve the above object, according to the present invention, a position where a state of at least one of image, sound, subtitle and other video software configuration data changes is detected, and the detected position is detected. Based on this information, some representative images are extracted from the video software, and a summary of the video software is generated (claim 1). Also, for at least one of image, sound, subtitle, and other video software configuration data, a position at which the state changes is detected, and the information of the detected position is converted into a summary of the video software including several representative images. Is added to the video software as position information for generating (Claim 2).

[0013] In addition, for at least one of image software, image data, subtitle data, and other video software configuration data, information on a position added to the video software as a position at which the state changes is read out, and based on the position information. Then, some representative images are extracted from the video software, and a summary of the video software is generated (claim 3). In addition, for at least one of image, sound, subtitle, and other video software configuration data, a position where the state changes is detected, and some representative images related to the detected position are extracted. (Claim 4).

[0014] Further, for at least one of image, sound, subtitle and other video software configuration data, a position at which the state changes is detected, extracted as a representative image relating to the position, and added to the video software. The digests are read out and sequentially reproduced (claim 5). In the video software processing method according to any one of claims 1, 3 and 5, when the generated summary or the added summary is reproduced, a command is issued from a viewer. In this case, the video software is reproduced in a normal state from the vicinity of the position where the reproduced representative image is extracted (claim 6). Further, in the video software processing method according to any one of claims 1, 3, 5, and 6,
When the video software is reproduced in the normal state, if there is a command from the viewer, a summary of the video software is generated from the vicinity of the position (claim 7). .

[0015] Further, for at least one of image, sound, subtitle and other video software configuration data, a position detecting means for detecting a position at which the state changes, and the video software based on the detected position information. And summarizing means for extracting some representative images from the video software and generating a summarization of the video software. Also,
For at least one of image, sound, subtitle, and other video software configuration data, a position detection unit that detects a position at which the state changes, and information on the detected position is used to generate a summary of the video software. A position information adding means for adding the position information to the video software is provided (claim 9).

Position information reading means for reading out information on a position added to the video software as a position at which a state of at least one of image software, subtitles, and other video software configuration data is changed;
A summary generating means for generating a summary of the video software based on the position information (claim 10). In addition, for at least one of the image software, the sound, the caption, and other video software configuration data, a position detection unit that detects a position where the state changes, and, based on the detected position information, Image adding means for extracting the representative image of the image software and adding the representative image as a summary to the video software.

[0017] Further, for at least one of image, sound, subtitle, and other video software configuration data, a position at which the state changes is detected, and extracted from the video software based on information of the detected position. There is provided a reproducing means for sequentially reproducing the representative images added to the video software as a summary (claim 12). In the video software processing device according to any one of claims 8, 10, and 12, when the generated summary or the added summary is reproduced, a command is issued from a viewer. In this case, there is provided a reproducing means for reproducing the video software in a normal state from the vicinity of the position where the reproduced representative image is extracted (claim 13). Further, in the video software processing device according to any one of claims 8, 10, 12, and 13, when the viewer receives a command from the viewer when playing the video software in a normal state, Summarization means is provided for executing the summarization from the vicinity of the reproduced position (claim 14).

Further, in the video software processing method according to any one of claims 1 to 7, a distance in a multidimensional space is calculated for a histogram of each cut representative image representing each cut. Then, by detecting the positions at which the state of the configuration data changes, a group is formed by putting together the objects having the short distances, and each position representative of each group is extracted (claim 15).

In the video software processing apparatus according to any one of claims 8 to 14, a distance calculation for calculating a distance in a multidimensional space for a histogram of each image representing each cut. The position detecting means is realized by means, a grouping means for grouping together the objects having the short distances, and a group representative detecting means for detecting a position representative of each group.
6).

(Function) In the present invention, attention is paid to the position where the video software configuration data changes, for example, images, sounds, subtitle codes, etc. It is represented by a set (summary) of each image representing the neighborhood, and specifically, this is represented by 1
The still image is displayed in sequence for several seconds on the screen (by frame advance display),
For example, the entire image of the drama or the like can be accurately grasped in five minutes. In the present invention, this summary is created using various features such as sounds, characters, and images, and using image processing technology by a computer. When this system is used, for example, the video software VS of “Chibi Maruko-chan” having a length of 30 minutes in FIG. 2 is summarized into a summary composed of several images VD and subtitles CP shown on the right (note that FIG. The picture is quoted from the comic "Chibi Maruko-chan.")

[0021]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The details of the present invention will be described below.
To facilitate understanding, an essence sprayer 10 as an example of the embodiment shown in FIG. 1 will be described first, and then various developments of the present invention will be described. That is, the essence player 10 is a digital video player 20.
And is used in connection with a cut determination storage unit 1, a subtitle determination storage unit 4, a screen synthesis unit 5, a hard disk 6, a CPU 2,
It includes a ROM 3, an operation unit 7, a display 8, and the like.

The function of each section is as follows. Caption discrimination storage unit 4: discriminates and cuts out captions (captions). Information on the position where the caption appears (timing information)
To the cut discrimination storage unit 1. The caption is stored on the hard disk 6 (or RAM) together with the information on the position. Cut discrimination storage unit 1: receives information on the position where a caption appears, and from a predetermined position of a corresponding cut (scene),
A still image is taken out (still image = one screen (one frame). This corresponds to a representative image described in the claims). Still images are addressed and stored on a hard disk (or RAM)
Is stored in

The predetermined position refers to, for example, a position (center of a scene) such as the PC in FIG. It is presumed that a screen (image) representative of the scene often appears a little after the scene is switched, and here, the center of the scene is listed at a predetermined position, but the appearance position of the characteristic image depends on the genre. Is different, the predetermined position is arbitrarily determined.
The viewer or the video software maker may try the abstract extraction once, and determine the predetermined position based on the result.

Hard disk 6: Stores cut images designated by the viewer for cutting during reproduction of subtitles, still images, and representative images. CPU2, ROM3: The CPU2 executes processing according to the present invention in accordance with programs stored in the ROM3 and the hard disk 6.

The operation unit 7 has the following various buttons. Switching between EP / NP essence play and normal play FWD During essence play, go to the next screen BK 戻 り Return to the previous screen CUT 〃 Cut out and save to hard disk 6 Display 8: Display of video software or representative screen (summary).

The still picture and the subtitles are provided with positional information of the original video and information indicating the mutual relationship between them, and a link is established (the information is associated with each other). The position information is represented by, for example, the storage address of the DVD, the number of the image (frame) counted from the beginning, the normal reproduction time from the beginning to that, and the like. Based on this, for example, the switching operation between the essence play and the normal play by the EP / NP button, and the cutting and saving of one screen or a continuous screen of a desired length near the still image extraction position are executed.

The switching operation between the essence play and the normal play is performed, for example, when a cut desired by the user is found from the reproduction of a still image + subtitles (essence play). (Normal playback) or during normal play, in accordance with the user's command, moving from that position to still image + subtitle playback processing. In addition, the essence user can adjust the screen feed speed according to the speed at which the subtitles are read, and view the subtitles.

When the CUT button is pressed during the essence play, one screen corresponding to the CUT button or a scene (cut) of a desired length is stored on the hard disk 6 as video data. A file name is given to this image data.
If the text can be extracted from the subtitles, it can be used as the file name. This makes it easier to search later. Also, by specifying the files in order, the saved scenes may be freely connected and edited.

There are cases where subtitles are attached to video software by character codes and cases where they are incorporated into image data as video itself. In the former case, the character code may be read according to the storage format of the medium. The appearance position (timing data) can also be easily grasped in relation to the storage position. Since the story can be understood only by the subtitles, it is interesting to tailor the extracted subtitle data into one still image and summarize the subtitles only.

When a caption is incorporated in an image as a video itself, the presence or absence of the caption is detected by using the kanji OCR method, which has recently been improved in accuracy. It is enough to be able to recognize that there are some characters on the screen.For example, focus on the lower part or both ends of the image, which is generally the subtitle display position, and capture the image with a coarse resolution. The presence or absence of subtitles may be determined based on whether or not any character can be read from.

Hereinafter, the relationship between video, cut, and subtitle will be exemplified for each type of video. [1] Conventional subtitled movie In a conventional subtitled movie, subtitles are printed on a film. Therefore, the processing method is as described above. If C = cut (scene), S = caption, G = still image or short moving image, the respective relationships are as follows, for example. Video = C1 + C2 + C3 +... Cx The cut is several seconds to several tens of seconds, and there are cases in which subtitles are attached and no subtitles. Subtitles may be once per cut or several times. Cut subtitles (superimpose) C1 S11 + S12 C20 C3 S31 + S32 + S33 Cx Sx1 + Sx2 + Sx3 + Sx4

In this case, the extraction of the essence image is performed, for example, as follows. 1. Determine and extract subtitles. 2. List the subtitles in the display order. S11 + S12, S31 + S32 + S33,... Sx1 + Sx2 + S
x3 + Sx4 3. One still image at the position where subtitles were first displayed in each cut
Cut out pieces. C1 G1 C2-C3 G3 Cx Gx 4. Each still image and subtitle have an address, and are stored on the hard disk 6.

The relationship between these and the essence play is as follows. 1. When watching a normal playback video on a DVD or digital video, pressing the essence play button (EP / NP button) displays a still image and subtitles. 2. For example, if you press the essence play button while playing the cut of C3, G3 + S31 will be displayed, and by pressing the next page button (FWD button), G3 + S32, G3 + S33 ... Gx + Sx1, Gx + Sx2,.
・ And the essence screen are displayed in order. Press the return button (BK) to return to the previous screen. 3. Normal play button (EP / NP) during essence play
Button), you can watch normal video from the beginning of the cut. 4. When the cut button (CUT) is pressed during the essence play, the image of the cut can be stored in the hard disk 6.

[2] Video + Caption (Caption in Image Format) (1) Structure of video and caption. In some cases, the video and subtitles are configured as separate screens, and are combined and displayed during playback. The relationship in this case is as follows. Video = C1 + C2 + C3 + ... Cx subtitle = S1 + S2 + S3 + ... Sy cut subtitle (superimposed) C1 S1 + S2 C2 0 C3 S3 + S4 + S5 · · · · · · Cx Sy

(2) Extraction method of essence image 1. Subtitles are arranged in display order. S1, S2, S3, S4,... Sy 2. Cut out one still image with subtitles displayed first in each cut. C1 G1 C2-C3 G3------------------------------------------------------------------------------------

(3) Essence Play 1. When watching an image on a DVD or digital video, pressing the essence play button displays a still image and subtitles. 2. For example, if you press the essence play button while playing a cut of C3, G3 + S3 is displayed. By pressing the next page button, you can see the essence screen as G3 + S4, G3 + S5. Press the return button to return to the previous screen. 3. If you press the normal play button during essence play, you can watch normal video from the beginning of the cut. 4. If you press the crop button during essence play, the video of the cut can be saved in the storage device.

[3] Video + Subtitle (Text Data) When the subtitle is not image data but text data, the text data is temporarily expanded into an image to form a subtitle, and displayed together with the video during reproduction. Also in this case, since the cut at which the subtitle is displayed is specified, there is no need to determine the subtitle from the video. Essence play processing and operation can be performed in the same manner as in [2].

The embodiment has been described above.
The invention can be implemented in a further wide variety of forms. Hereinafter, these will be described in detail. Item numbers will be added here again. First, VE is extracted from the original video. Therefore, the problem is to understand the characteristics of the original video and what kind of summary video can be created from it. First, this point will be described.

1. Elements of Original Video (Multimedia Data) The original video for extracting the VE is a multimedia video used for DVD, television, video, personal computer, and the like, and is roughly classified into the following two types. 1) Stream video Video that has existed in the past, such as movies and videos. It changes according to a certain flow. 2) Link video (hyper jump video) A video that is used in games and the like in recent years, where the link destination changes according to the user's selection and the story development changes. VE targets both of these two types, but at present, "stream video" is more, so for now, this will be the target.

These video softwares include the following elements. 1) Character code (TC) A character code is indispensable for a hearing-impaired person as an explanation of a screen. It will be included in much of the multimedia data to be produced. 2) Image characters (TI) Those that appear as characters but are not in character code.
This kind of character can be used if it is possible to know whether it is not a natural image but a character. This processing can be performed by the existing character recognition method as described above.

3) Still image (S) An image displayed as a still image on the screen for a certain period of time. T
V or a still image file used for a personal computer or the like. 4) Moving image (D) An image itself such as a movie or a video, which represents the motion of a subject in a number of continuous frames. In the case of a TV, there are 30 images per second. If the movement of the screen is fast, the resolution of one still image may decrease. In that case, a moving image video of about 1 second is cut out instead of a still image. This is also included in the “representative image” described in the claims.

5) Sound (A) The sound includes voice and other sounds.・ Speech is a human voice and has a clear meaning (in a broad sense, “speech” includes all “sounds”, but here, it is used in a narrow sense as described above. No strict distinction is made)). The “other sounds” include, for example, music, sound effects, and surrounding sounds. Audio and other sounds are often separated and stored in multimedia data in advance.

2. Configuration of VE All the multimedia elements described above can be used as VE components. However, in order to see this in a short time, it is effective to use still images and characters and use short moving images as needed. 1) Still image... The user views this as turning a page. 2) Characters: If there is character code data, this is displayed. If there is an image character, paste it on the screen. It is good to make it stand still on a screen, or to display it, for example, so that it may flow right and left. 3) Moving picture image: This is started when the user turns the page. 4) Sound: Start when the user turns the page. If the page feed is fast, you do not need to output.

3. User's operation when viewing VE 1) The user turns the VE so that the pages of the book are turned. 2) One page is a still image or a moving image. 3) The page has a sound, and when the user opens the page, the sound is output as described above (it can be repeated). 4) It is better to prepare an automatic turning mode. For example,
In the case of a still image, it is preferable to automatically advance to the next page at intervals set by the viewer, for example, every few seconds. It is preferable that the moving image starts moving as soon as the page is opened, and proceeds to the next page when the moving image ends. Since it is difficult to hear the sound when the reproduction speed is increased, it is better to output the sound mainly with the sound effect. 5) Link to the original video (hyper jump) If you want to see the original video while watching the VE, press the button. This alone links to the original video, and you can return to VE again by pressing it again.

4. Parameters That Can Be Set by User When using VE, parameters that can be set by the user include, for example, the following. 1) The size of the summarization rate 2) Automatic turning / manual turning 3) Summarization method (motivation of representative image extraction) A default summarization method may be defined. In addition, it is better to allow the viewer to select. Using a plurality of motives, for example, a representative image may be extracted for both a position where a human voice is present and a position where a cheer is raised.

[Example of motivation]-Switching of scenes-Periodic extraction at regular intervals-Extraction of climax of sounds such as applause and cheers When a VE button is attached to a video or the like, the user presses this button to immediately display the VE image. Can be taken out.
Clicking the VE button on a personal computer allows you to flip the VE video.

5. VE Extraction Method VE extraction uses a comprehensive relationship of information as multimedia. Information such as the state, sound, and characters of the moving image is used to find the timing of extracting the still image or moving image. Although this is the same as that described above, if it is described again in the sense of tidying, the following method can be considered as a method of extracting the VE.

1) Cutting out the top screen The screen at the start of the original video is simply cut out (extracted) as a title (representative screen). 2) Periodic clipping Periodic clipping of the screen at regular intervals (every 10 seconds depending on the summarization rate). 3) Cut out a cut Take out one scene and cut out one still image from it. 4) Subtitle cutout If there is a subtitle in a part of the screen, this is cut out as a still image.

5) Extraction of a scene with sound A scene without sound is skipped, and only a scene with sound is extracted. 6) Character pattern screen A scene in which character information is clearly displayed on the screen (such as a pattern display) is cut out as one representative screen. 7) Climax Cut out a short-time video image during climax when the sound becomes loud such as applause and loud cheers.

6. Assumed Targets and Extraction of VEs The following is an example of specific VE motives (element change positions) for various stream videos. Video genre VE extraction motive News Scene with pattern (flip) Drama Scene with subtitles Scene with audio Documentary Audio scene English conversation with subtitles Sports Applause, cheering scenes and their surroundings (audio climax) Animation Subtitles One scene A long stationary scene A scene with sound TV shopping A scene with subtitles (You can see information such as price) A song program A scene where music starts (Judgment from sound) Educational program A scene with patterns Variety show A scene where cheers rise orchestra Music start scene (periodic extraction) Weather forecast Stationary scene (pattern (flip) = a board on which characters and pictures are written, used by speakers on TV)

7. Summarization rate Summarization rate = time to see VE with automatic flipping at standard speed / time to see original video. Using VE, 1/1 of the original video display time
The contents can be viewed in the time of 0 to 1/30 (summarization rate).

The time required to view the VE depends on the operation of the user. Users can look at the page slowly, send the video quickly,
You may also look at automatic page turning. VE extracted from original video of 2 hours (120 minutes) in case of automatic turning
Takes about 4 to 12 minutes.

8. Implementation of VE The implementation of the VE technology includes, for example, the following various types. 1) Addition to Video Software When VE is standardized, VE is generally added to any video as standard. The video software production side adds VE data at the production stage.

2) Incorporation into Hardware VE can also be incorporated into various hardware (devices). It may be manufactured and sold as a dedicated LSI chip. 3) Use VE as software The VE technology is realized as computer software, and is executed by being incorporated into a personal computer or the like. The user can optionally use VE for various images. It can also be used when retrieving video from the Internet.

The present invention is further applicable to existing video software other than multimedia. Hereinafter, these will be described. First, the summary video finally taken out is configured as follows. (1) Still image (2) Short video image (3) Character (equivalent to subtitle of movie) (4) Sound

(1) Original Video Input Device A normal video capture device, such as a video deck or a laser disk deck, is used. (2) Summarization processing method 1) Simple method-If the original image is composed of N images and M images are to be extracted from this to generate a summary, a still image should be created for every N / M images Take it out. This is the simplest method.・ For sound, remove silence parts and take out parts where speech is words. The words, music, noise (roar), etc. can be relatively distinguished by frequency analysis. Only the audio near the video of the still image to be extracted is extracted. -If character data is given, the character displayed near the still image is extracted and superimposed on the still image.

2) Difference method (finding of scene switching timing) In this method, a video that changes greatly with time is found (such as scene switching), and the video is taken out by paying attention to this. When the scene is changed, the time until the next scene change is checked, and a screen at an intermediate time is taken out as a still image (the position of PC in FIG. 3). Alternatively, a moving image corresponding to a certain time width of an intermediate portion where the scene switching occurs is extracted as a summary image. -Handling of characters In the case of multimedia data, characters near the extracted still image / moving image are extracted from "character data" and superimposed on the still image. Otherwise, the subtitle is determined and displayed.・ Handling of audio The audio near the extracted still image / moving image is extracted from the “audio data”, and the audio is output only once or repeatedly while the still image is displayed.

The following method can be used as an image processing method. 1) Pixel aggregation processing An image is a collection of points such as 300 × 260 as a two-dimensional spread. Each dot is colored. In a television picture, such a picture is 3 per second.
0 sheets are required. Since the pixel configuration of this image is usually too large for subsequent processing, pixel aggregation is performed. For example, if the images are aggregated at 4x4 points, the original video is 1/1
6 data volumes. If it is taken out as 8x8, it can be reduced to 1/64.

This has the effect of discarding small changes even for images that change slowly when zooming up / down. After performing such preprocessing, the respective values, a (t, x, y), are extracted. Here, t: time x, y: coordinates of the aggregated video a: a color value of the point (x, y), a = R +
Generally, G + B is used (R, G, and B are values of three primary color information).

2) Aggregation processing in the time direction Video data that changes with time is redundant and is wasteful to handle as it is. Therefore, any one of the following processes is performed on the data a (t, x, y) of each point on which the preprocessing has been performed. (1) Periodic sampling (2) Temporal difference comparison processing of video Data of each of the above points constituting data of one image, a
A temporal difference is obtained for (t, x, y). d (t, x, y) = a (t, x, y) −a (t−Δt,
x, y) where Δt: appropriate time width

This value is totaled for the entire image (x, y). In terms of the formula,

(Equation 1) Becomes This indicates the amount of change (difference) between two images that are temporally adjacent by Δt.

Here, N Da (t) are taken out in descending order of the value. N Da (t) thus taken out
3 is shown in FIG. CS1 and CS2 where the value is large, that is, where the difference between the images is large, indicate that there is a large change in the image, that is, there is a high possibility that the scene has been switched there. So, this CS
Considering between 1 and CS2 as one scene, one representative image is extracted from this. Although different depending on the genre, a still image that clearly represents a scene is generally located near the center of the scene. Therefore, in the example of FIG.
C is a position where the representative screen is taken out.

For each frame constituting the video software, the degree of change between images (switching of scenes) is detected by focusing on, for example, image data of one horizontal scanning line near the center of each frame. You may do it. Specifically, for example, the image data for one horizontal scanning line is
It is divided into N sections, and an average value is obtained for each section. Then, for each section, a difference between the previous image and the average value of the section is calculated. The difference is summed for each frame, and the position where the value is large, that is, CS in FIG.
1 or a position corresponding to CS2 is obtained, and this is assumed to be a scene switching position similar to the above.

The summary video has the following features. Users can turn the page at any time they like. In the case of a still image, it is simple. In the case of a moving image, a part is displayed as a moving image in a short time within one page, and can be repeatedly viewed by an instruction.

Hereinafter, modified examples will be described. First, at the time of summary playback, four representative screens constituting the summary screen may be displayed on the display at a time (the four representative screens are synthesized,
One screen is displayed.) The representative image originally starts from one video software. Therefore, each is related to each other (has a story), and even if a plurality of these are displayed on the screen at once, the viewer can easily perceive the contents. Therefore, if, for example, N representative images are displayed on the screen at a time, the summary can be simply reproduced one by one in N times of the time of still display. Also, in view of the fact that the video software has a story, it may be easier to perceive the contents in a format in which a plurality of representative images are displayed at a time than in a format in which the images are reproduced one by one.

The present invention may be realized by a dedicated device, or may be stored in a hard disk or the like as one of the application programs, and may be called up as necessary and executed on a computer. They can also be stored on floppy disks and distributed. It is also possible to manufacture a dedicated chip and incorporate it into a personal computer, a DVD player, a game player, or a VTR. When the operation is performed by a personal computer or the like, the “EP / NP” button or the like in the embodiment may be assigned to a screen or a keyboard.

A plurality of summaries may be added to one piece of video software so that the viewer can selectively view. For example, it is also possible to add, to the video software, two summaries that are motivated by the appearance position of the caption and that are motivated by the position where cheers are raised. This processing can be applied to the embodiment in which the position information is added to the video software. Also, both the position information and the summary image may be added to the video software. In this case, a normal video software playback device (D) having no function of generating the summary from the position information is provided.
VD players, VCRs, etc.) can also perform summary playback.

The video software to which the representative screen (summary) or the position information is added can be transmitted on an independent medium such as a DVD or a video tape, or transmitted via a radio wave, a network, an individual communication line, or another medium. I can do it. At this time, various data (captions, sounds, etc.) near the representative screen extraction position may also be transmitted together, and may be reproduced by the viewer's selection. Based on the motivation selected by viewers and administrators,
It is also possible to extract representative images from video software and accumulate them on an appropriate storage medium such as a hard disk or a video tape. In this way, in the video library etc.
The search for the target video software can be performed accurately and easily. It can also be used to manage video software owned by individuals.

Another embodiment 30 is shown in FIGS. This embodiment 30 is one embodiment of the video software processing method according to claim 15 and the video software processing device according to claim 16, and mainly includes C
It is realized on PU2 and ROM3.

For example, in a drama, a performance is performed at a certain place (background), the cut is changed, and a subsequent performance is performed at one or several other places, and then returns to the original place. In many cases, you can continue acting. The summary is
The content of the video software is more easily perceived when the number of representative images constituting the video software is appropriate, and when they represent different scenes from various aspects.

The third embodiment shown in FIGS. 4 to 10
0 can respond to such a demand, and creates a histogram for each of the cut representative images extracted from each of the cuts, and from the viewpoint of the distance of these histograms in a multidimensional space, each histogram, that is, each histogram, The similarity of the cut representative images is obtained, and a process of approximating the close ones, that is, the close ones are grouped into the same group is repeated, and these many cut representative images are aggregated into an appropriate number, for example, several tens of groups. , A process of extracting a representative image from each group.

As a result, the contents of each representative image become multifaceted representing different scenes of the video software, and the number is reduced to an appropriate number. In addition, the similarity judgment performed for each screen performed here is based on the characteristic that the screen of the video software generally has, that is, when the cut changes, the distribution (waveform, envelope) of the histogram becomes visible there. The advantage is that it can be implemented uniformly and mechanically regardless of the genre of the video software.

The embodiment 30 shown in FIGS. 4 to 10 will be described below. The heading numbers are added again in the description of this embodiment. 1) Cut out the cut. Video software consists of many cuts. One cut is a few seconds to a few tens of seconds long. At the boundary where the cut changes, the data of each frame changes significantly.
As a method of detecting this, one of the discrimination methods in the above-described embodiment is used. Here, a histogram (video histogram) (H) of each frame (image) effective for this is used. Use the method that you use.

FIG. 5 shows an example of the histogram. This histogram is created by the histogram creating unit 31. The horizontal axis plots the information of the color (R, G, B) of each pixel of each image, or the brightness and the hue (Y, U, V), and the vertical axis displays the pixel such as the color in one image. Is plotted (the number of pixels). In the example of FIG. 5, the horizontal axis represents the luminance Y in 256 steps, and the vertical axis represents the frequency of occurrence of pixels having the luminance Y (the number of pixels). In order to simplify the processing, it is convenient to divide the horizontal axis into 10 to 20 steps and collectively express them as shown in FIG.

The histogram can simply represent the characteristics of the image, and can be effectively used in various ways. Here, first, as described above, it is used for discrimination of a cut. That is, the difference calculation unit 32 obtains the difference Ha of each histogram H (t), and detects a portion where the change is large. The position where the difference is large represents the cut switching position.

H (t) = H (t, y1, y2, y3,..., YN) where H (t): histogram of the frame at time t yn: color or color of the n-th division Frequency of brightness information (n = 1, 2,..., N, where N = the number of sections on the horizontal axis)

[0077] ΔHa = H (t + 1) a _{-H (t) = Σ y n} (t + 1) -y n (t) ( the sum of the absolute values of the differences of the components). In the figure,
For example, as shown in FIG. If a portion having a change equal to or more than a certain value is taken out from this portion, the change of cut can be detected.

2) Extraction of a representative cut histogram. Since a continuous video, for example, a continuous image of a one-hour drama is cut into a number of cuts, a representative histogram (Hs: representative histogram) of each cut is created by the representative histogram extracting unit 33 next. For this, for example, a histogram is created by extracting the frame at the first time, the frame at the time near the center, and the frame at the end time in the cut. A time average for the entire histogram of one cut may be created and used as a representative.

This is expressed by the following equation: Hs = Hs (ys1, ys2, ys3,..., YsN) where Hs: a representative histogram of the cut s (s = 1,
2,..., S) ysn: the color of the n-th division or the frequency of the color or brightness information (n =
1, 2,..., N). (Average over the representative screen of the cut and the time of the cut)

3) Calculation of the degree of approximation of the cut representative histogram. Next, the distance between representative cut histograms Hs is calculated by the representative histogram distance calculating unit 34 (s,
t = 1, 2, 3,..., S). The distance Dst is the distance between the representative histogram of a certain cut s and the representative histogram of another cut t. This value (Dst) is the absolute value of the difference between the corresponding sections ysn of each histogram. It is calculated as the sum of values or square values. In terms of the formula,

(Equation 2) It is. When the number of cut representative histograms is s, the number of distances Dst between the representative histograms thus obtained is the same as the number of games in the league match, and s ×
(S-1) ÷ 2.

4) Grouping. Next, small ones having a small inter-cut distance Dst are put together to form an appropriate number (M) of groups. This processing is performed by the grouping processing unit 36. Various mathematical methods can be used for this processing, but the simplest method is as follows. First, the cut representative histograms s and t at the minimum distance among Dst are obtained. And this s
And t into one group. Next, the remaining distance Dst
And finds the combination of s and t having the minimum distance from and group them.

Initially, s histograms corresponding to the number of all cuts exist independently, and the group has the same number of s histograms.
Exists. Here, if the grouping operation is performed once, the group is reduced by one. Therefore, if this grouping operation is repeated SM times, the number of groups becomes M. As a result, s representative histograms of each cut can be grouped into an appropriate number (M) of groups as shown in FIG.

5) Extraction of representative images of each group. When the group division is completed, a representative image is extracted from each group. This process is executed by the representative image extracting unit 37. In this case, one group representative image may be randomly selected from each group.
The center of gravity Hg (symbol x in FIG. 8) of the group is obtained, and the image closest to the center of gravity Hg is set as a representative image.

To determine the center of gravity Hg, first, for each histogram belonging to the group, each y
Average the values of sn. In terms of the formula,

(Equation 3)

(Equation 4) It is expressed as

Then, the distance between each histogram j and the group center of gravity Hg is obtained. The histogram with the smallest distance, that is, the image of the histogram closest to the group centroid is extracted as a representative image of the group. These representative images may be aggregated as they are and summarized. FIG. 9 shows the flow up to this point.

In this embodiment, a grouping process is performed. Therefore, when the processing result is reflected in the extraction of the representative image, a summary expressing the contents of the original video software can be generated more accurately. Specifically, each image included in the summary is assigned a priority, and a representative image is selected as described below according to the priority.

1) Reflecting the number of cuts (representative histograms) included in the group. If the number of cuts that make up a group is large, that is, if similar images or cuts appear many times in one piece of video software, the group is considered to have significant significance in that video software. . Therefore, it is considered effective to take out a representative image of a number proportional to the frequency, instead of taking out only one representative image. In this case, if the extracted ones are similar, the sense of having a plurality of representative images feels less.
Therefore, in this case, it is preferable to select as many items as possible as far as possible and take out a plurality of items.

2) Reflection of total time of cut. Even when the total of the durations of the cuts, which are the bases of the histograms constituting the group, is long, it is considered that the group also has significant significance in the video software. Therefore, in order to increase these priorities, it is also effective to take out some representative images in proportion to the total number of cut times for each group. 3) Ignoring infrequent cuts. If the number of cuts constituting the group is small, the group may be ignored and the representative image may not be used for the summary.

The display of the representative image has two ideas. First, they may be distributed on a large screen and displayed in large numbers at the same time. In other words, it is an expression format like "bird and beast caricature". In this way, the summary can be grasped spatially rather than temporally. Then, when one of the representative images arranged spatially is clicked, it is good to make the representative image start moving as a moving image.
The original video software may be played back from the cut.

At this time, it is easy to understand that weighting is applied to the display size of each representative image from the viewpoints such as 1), 2) and 3). That is, as shown in FIG. 10, a histogram having a large number of histograms belonging to the group or a graph having a large total duration of the cut has a large display area and is arranged at the center. In this way, the viewer can feel the touch of the video software at a glance. The modification of the first embodiment also applies to the embodiment described here.

[0091]

As described above, according to the present invention,
By paying attention to the video software configuration data desired by the viewer, a video summary can be automatically extracted. The viewer can freely see it at his own pace in a short time, and can easily and accurately grasp the entire image of the video software. This makes it possible to easily access a large amount of video.

Further, the summary video has a considerably smaller data amount than the original video, and may be suitable for communication and portable terminals. In the future, the data volume itself will not be a problem due to the high-speed optical fiber communication network, large capacity memory, and high-speed CPU, but the small amount of information itself means that more types of information can be transmitted by the same communication means. There is an advantage that can be. There is also an advantage that it can be easily implemented even in the transition period to a high-speed communication network.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating an example of an embodiment.

FIG. 2 is an explanatory diagram showing an example of original video software and an extracted abstract (representative screen).

FIG. 3 is a graph showing an example of a difference between image data.

FIG. 4 is a block diagram showing another embodiment.

FIG. 5 is a graph showing an example of a histogram of one image.

FIG. 6 is a graph showing an example of a simplified histogram.

FIG. 7 is a graph showing an example of a histogram difference.

FIG. 8 is a conceptual diagram showing an example of grouping each cut (histogram) in an n-dimensional space.

FIG. 9 is a conceptual diagram showing a summary generation process in another embodiment.

FIG. 10 is a plan view showing a summary display example of a spatial arrangement.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Cut discrimination storage part 2 ... CPU 3 ... ROM 4 ... Subtitle discrimination storage part 5 ... Screen synthesis part 6 ... Hard disk 7 ... Operation unit 8 ... Display 10 ... Essen sprayer 20 ... DVD player 30 ... Other embodiment examples 31 ... Histogram creation unit 32 ... Difference calculation unit 33 ... Representative histogram extraction unit 34 ... Representative histogram distance calculation unit 36 ... Grouping processing unit 37 ... Representative image extraction unit

Claims (16)

[Claims] At least one of image, sound, subtitle, and other video software configuration data detects a position at which the state changes, and, based on the information on the detected position, divides some of the video software from the video software. A video software processing method, comprising: extracting a representative image and generating a summary of the video software. 2. At least one of image, sound, subtitle, and other video software configuration data is detected at a position where the state changes, and the information of the detected position is converted to the video comprising several representative images. A video software processing method, wherein the video software processing method is added to the video software as position information for generating a software summary. 3. At least one of image, sound, subtitle, and other video software configuration data is read as information on a position added to the video software as a position at which the state changes, and based on the position information. And
Extracting some representative images from the video software,
A video software processing method comprising generating a summary of the video software. 4. For at least one of image, sound, subtitle, and other video software configuration data, a position at which the state changes is detected, and some representative images related to the detected position are extracted as a summary. A video software processing method, which is added to the video software. 5. At least one of image, sound, subtitle, and other video software configuration data is detected at a position where the state changes, extracted as a representative image relating to the position, and added to the video software. A video software processing method, wherein a digest is read and sequentially reproduced. 6. When reproducing the generated summary or the added summary, if there is a command from a viewer, the video software is normally operated from the vicinity of the position where the reproduced representative image is extracted. 6. The video software processing method according to claim 1, wherein the video software is reproduced. 7. The video software according to claim 1, wherein when the video software is reproduced in a normal state, when a viewer gives an instruction, a summary of the video software is generated from the vicinity of the position. The video software processing method according to claim 5. 8. A position detecting means for detecting a position where at least one of image, sound, subtitle and other video software configuration data changes in state, and said video software based on the detected position information. And a summary generating means for extracting some representative images from the image and generating a summary of the video software. 9. A position detecting means for detecting a position at which a state of at least one of image, sound, subtitle, and other video software configuration data changes, and information on the detected position is summarized by the video software. And a position information adding unit for adding position information to the image software as position information for generating the image software. 10. Position information reading means for reading information of a position added to the video software as a position at which a state of at least one of image, sound, subtitle, and other video software configuration data is changed; A video software processing device, comprising: a summary generating unit configured to generate a video software summary based on the position information. 11. A position detecting means for detecting a position at which at least one of image, sound, subtitle and other video software configuration data changes in state, and said video software based on information of the detected position. An image adding means for extracting some representative images from the image software and adding the representative images as a summary to the video software. 12. At least one of image, sound, subtitle, and other video software configuration data detects a position at which the state changes, and is extracted from the video software based on information of the detected position. A video software processing device comprising a reproducing unit for sequentially reproducing a representative image added to the video software as a summary. 13. At the time of reproducing the generated summary or the added summary, if there is a command from a viewer, the video software is placed in a normal state from a position near the position where the reproduced representative image is extracted. 10. A reproducing means for reproducing in the step (c).
3. The video software processing device according to any one of 2. 14. A reproducing apparatus according to claim 8, further comprising a summarizing means for executing the summarization from the vicinity of the position where the video software is reproduced when a command is received from a viewer during reproduction of the video software in a normal state. The video software processing device according to any one of claims 10, 12, and 13. 15. A distance in a multidimensional space is calculated for a histogram of each of the cut representative images representing each cut, and a group in which the distances are short is formed to form a group. By extracting the position,
The video software processing method according to any one of claims 1 to 7, wherein a position at which a state of the configuration data changes is detected. 16. A distance calculating means for calculating a distance in a multidimensional space with respect to a histogram of each image representing each cut, a grouping means for grouping together those having a short distance, 15. The video software processing apparatus according to claim 8, wherein the position detection unit is realized by a group representative detection unit that detects a position representative of a group.