JP2012142645A - Audio/video reproducing apparatus, audio/video recording and reproducing apparatus, audio/video editing apparatus, audio/video reproducing method, audio/video recording and reproducing method, and audio/video editing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an audio/video recording and reproducing apparatus capable of selecting a proper image as a thumb nail.SOLUTION: An audio/video reproducing method includes extracting a scene boundary from image information, generating entry information for accessing the extracted scene boundary, selecting a thumb nail image representing scenes divided by the extracted scene boundary, displaying the selected thumb nail image, allowing an arbitrary thumb nail to be selected from the displayed thumb nails, and reproducing a scene corresponding to the selected thumb nail, and selects a picture a prescribed period after the scene boundary as the thumb nail image.

Description

  The present invention relates to a video / audio reproduction device that divides a video to be recorded into scene units and generates a thumbnail image corresponding to the scene so that a scene desired to be viewed can be visually selected by thumbnail selection at the time of reproduction. It is about.

  In the conventional video recording / playback device, when playing from the playback start position of the video, the playback start time is selected by the title name or chapter number, so the user can select what content before playback. Couldn't know if the video will be played.

  In order to solve such a problem, by generating thumbnail data using image data before video encoding processing at the time of recording and displaying the generated thumbnail data on a graphics screen for selecting a playback title There is a method of informing the user of the playback content (Patent Document 1). Furthermore, in order to make the image of thumbnail data indicating the content of a title or a specific section (chapter or scene) in the title more appropriate image content, a method of determining the position of the video that is the source of the thumbnail data has been devised. (Patent Documents 2 and 3). On the other hand, there is also a method for determining a thumbnail image by using a scene change occurrence interval in a scene in order to express the content of a specific scene as a continuous still image (Patent Document 4).

JP 2006-148731 A (page 14, FIG. 9) JP 2006-229821 A (page 4, FIG. 1) JP 2003-274361 A (page 4, FIG. 3) Japanese Patent Laid-Open No. 2001-298711 (page 3, FIG. 25)

  However, in the conventional video recording / playback apparatus, the start and end positions of a scene are separated by chapters or entries, and images after a fixed time from the start of playback or after a fixed time from the entry position are used as thumbnails. Therefore, there is a difference between the content of the selected thumbnail image and the content of the video to be played back, and the user sometimes feels uncomfortable. In addition, when a picture of a scene with a large change in motion is selected as a thumbnail, there is a problem that it is not possible to know what the thumbnail image is displayed.

  The present invention has been made to solve the above-described problem, and is a video / audio recording / reproducing apparatus capable of dividing video / audio content to be recorded into scenes and selecting an appropriate image as a thumbnail for each scene. The purpose is to provide.

A video reproduction apparatus according to the present invention includes a scene boundary extraction unit that extracts a scene boundary from video information;
Entry generating means for generating entry information for accessing the scene boundary extracted by the scene boundary extracting means;
Thumbnail generation means for selecting a thumbnail image representative of the scene divided by the scene boundary extraction means;
Thumbnail display means for displaying a thumbnail image selected by the thumbnail generation means;
A thumbnail selection means for selecting an arbitrary thumbnail from the displayed thumbnails;
Means for playing a scene corresponding to the selected thumbnail,
The thumbnail generation means selects a picture after a predetermined period from the scene boundary as a thumbnail image.

  According to the video / audio reproduction device of the present invention, since a picture after a predetermined period from a scene boundary is selected as a thumbnail image, it is possible to divide the video / audio content into scenes and to select an appropriate image as a thumbnail for each scene. it can.

1 is a block diagram showing a configuration of a video / audio recording / reproducing apparatus according to Embodiment 1. FIG. It is an example of the display screen in scene selection operation. It is a figure which shows the relationship between a scene and a thumbnail image. It is a figure which shows the relationship between a scene change and a thumbnail image. It is a figure which shows the relationship between a scene change and a thumbnail image. It is a figure which shows the relationship between a scene change and a thumbnail image. It is a flowchart which shows entry information production | generation operation | movement. It is a flowchart which shows a thumbnail production | generation operation | movement. It is a flowchart which shows a thumbnail production | generation operation | movement. It is a flowchart which shows a thumbnail display operation | movement. 6 is a block diagram showing a configuration of a video / audio recording / reproducing apparatus according to Embodiment 2. FIG. It is a figure which shows an example of the scene boundary detection method. 6 is a block diagram showing a configuration of a video / audio recording / reproducing apparatus according to Embodiment 3. FIG.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing a configuration of a video / audio recording / reproducing apparatus 1 according to Embodiment 1 of the present invention. The video / audio recording / reproducing apparatus 1 includes a digital tuner 3, an analog tuner 4, an external input terminal 5, a monitor output terminal 31, and a network terminal 30 as interfaces with external devices. The digital tuner 3 and the analog tuner 4 are connected to the antenna 91 and can receive digital and analog broadcasts. The external input terminal 5 mainly corresponds to composite input called analog input, S video input, LR audio input, and the like. A monitor 93 is connected to the monitor output terminal 31 so that video and audio signals generated by the video / audio recording / reproducing apparatus 1 can be displayed and output. The video / audio recording / reproducing apparatus 1 includes a remote control receiver 6 for remotely operating the apparatus, and a remote control code received by infrared rays or the like is transmitted to the system control unit 2.

  The system control unit 2 controls the video / audio recording / reproducing apparatus 1 according to a control signal of a remote controller based on a user's request. When a recording request is made by the user, the system control unit 2 activates the recording application 9 and instructs execution of recording. The recording application 9 controls the recording control 10 and manages the start and end of recording and the recording operation during recording. When a playback request is made by the user, the system control unit 2 activates the playback application 11 and instructs playback execution. The playback application 11 controls the playback control 14 to start playback of the specified content.

  With reference to FIG. 1, the recording function of the digital broadcast in the audio / video recording / reproducing apparatus 1 is demonstrated. The digital broadcast received by the digital tuner 3 is sent to the input stream control 8 as a full TS (Transport Stream). The input stream control 8 performs a process of temporarily storing the data and writing the AV stream recording unit 17 in a steady manner in order to stably record the AV stream that is the video / audio data in the AV stream recording unit 17. The input stream control 8 also supports a partial TS in which packets that do not need to be recorded from the full TS are omitted. In addition to the TS PID and section information detection processing, the GOP (Group Of Picture) start position in the recorded video stream is also detected. The AV stream recording unit 17 is configured by an HDD (Hard Disc Drive) or the like.

  When recording an AV stream on the HDD, the address information and the time information of the AV stream recorded on the HDD are associated with each other in order to improve scene searchability during playback and to achieve random seek at an arbitrary time during special playback. Address map is required. The recording control 10 generates an address map based on the GOP start position information detected by the input stream control 8 during recording, and records it in the AV management information recording unit 18. The AV management information recording unit 18 records the above-described address map, as well as the recorded program title, recording start time, recording end time, broadcast service name, channel number, video codec information, audio codec information, and program details information. And the like are recorded, and read out to the reproduction control 14 as necessary.

  Next, analog broadcasting and a video signal recording function input from outside will be described. From the analog broadcast received by the antenna 91, only the signal of the channel designated by the user by the analog tuner 4 is extracted. The signal extracted by the analog tuner 4 is converted into a video signal, converted into a digital signal by an AD / DA converter (not shown), and then input to the AV encoder 7. The AV encoder 7 encodes the video signal into MPEG-2 video, encodes the audio signal by AAC, multiplexes the encoded video and audio signal with the MPEG-2 Transport Stream, and generates an AV stream. The AV stream is sent to the input stream control 8 and recorded in the AV stream recording unit 17. Similarly, the audio / video signal input from the external input 5 is also AD / DA converted, converted to an AV stream by the AV encoder 7, and recorded in the AV stream recording unit 17.

  With reference to FIG. 1, the reproduction function in the video / audio recording / reproducing apparatus 1 will be described. When the user presses a “recorded title list” button provided on a remote controller (not shown), all or part of the recorded program titles are displayed on the screen. The user moves the cursor to a desired title by using the up / down / left / right cursor keys provided on the remote controller, and selects a title to be reproduced by pressing the “OK” button. The selected title is transmitted from the remote controller to the remote controller receiver 6 as an infrared signal, converted into a software code, and then transmitted to the system control unit 2 to update the graphics screen. If the playback standby state or the recording state is not set, these codes are sent to the playback application 11, the corresponding playback operation function is called, and the playback state is changed. Hereinafter, a specific reproduction operation will be described.

  When the user determines the reproduction title, the reproduction application 11 instructs the reproduction control 14 to reproduce the designated title. The reproduction control 14 reads out reproduction control information related to the title from the AV management information recording unit 18 and inputs it to a RAM (not shown). The playback control 14 uses the address map of the playback control information to read address information corresponding to the playback start time of the stream to be played next, reads the AV stream at that address from the AV stream recording unit 17, and outputs the output stream. Read to control 16. The AV decoder 15 decodes the AV stream sent from the output stream control 16 in the order of transfer, and outputs the decoded AV stream from the monitor output terminal 31 to the monitor 93. The output stream control 16 reads the AV stream recorded in the AV stream recording unit 17 so as to decode the video / audio data so that the video / audio is not interrupted by causing the AV decoder 15 to underflow and overflow. Furthermore, the output stream control 16 uses a DMA (Direct Memory Access) transfer function, which is an H / W assist, for the AV stream temporarily stored in accordance with the buffer state of the AV decoder 15. Transfer to.

  Next, the operation of the scene detection unit 50 in the video / audio recording / reproducing apparatus 1 will be described. The scene detection unit 50 performs scene detection by analyzing video and audio of an AV stream temporarily recorded in the input stream control 8 during a recording operation. When the head of the GOP is detected in the input stream control 8 during recording, the analysis decoder 21 extracts a key frame (I picture) from the head of the GOP, and decodes the I picture. The decoded luminance signal (Y) and color difference signal (U, V) data are input to the frame buffer 22. The frame buffer 22 has a size capable of holding at least two images of decoded YUV data. The scene boundary extraction 23 obtains the difference between the two images input to the frame buffer 22, and determines that a scene change has occurred if the difference between the images is equal to or greater than a preset threshold, and determines the scene boundary.

  As a technique for obtaining a difference between two images, a means for obtaining a histogram of each image to be compared and accumulating an absolute difference value of each element of the histogram can be considered. Alternatively, a motion vector or the like that is an encoding parameter in the AV stream may be detected. Further, it may be realized by capturing changes in the composition of characters using face data detected by face detection, face recognition, or the like.

  The position determined as the scene boundary by the scene boundary extraction 23 is the scene reproduction start position. The entry generation 24 creates reproduction entry information including address information and time information of the scene reproduction start position. Such a scene is generally called a chapter. Further, the entry position or the information thereof may be called a chapter mark or simply a chapter. The entry information generated in this way is recorded in the metadata recording unit 19.

  When the scene boundary is detected by the scene boundary extraction 23, the thumbnail generation 25 extracts the corresponding image information from the frame buffer 22, converts it into a bitmap format, and generates a thumbnail image. The generated thumbnail image is recorded in the metadata recording unit 19 as thumbnail information (details of the thumbnail selection method will be described later). The thumbnail format may be a format such as TIFF or JPEG in addition to the bitmap. In addition to the method of recording thumbnail image data, time information and address information of an AV stream in which a thumbnail image exists may be recorded in the metadata recording unit 19 as thumbnail information. In this case, it is possible to reduce the system load without requiring generation time of actual data for thumbnails, and to reduce the area of the metadata recording unit 19 and reduce the time required for data transfer and the system load. .

  Next, a method for playing back an arbitrary scene using the entry information and thumbnail information in the video / audio recording / playback apparatus 1 will be described. FIG. 2 shows a state in which the playback video of the AV stream and the graphics image are superimposed on the monitor 93. For the thumbnail image, the thumbnail information stored in the metadata recording unit 19 is read out by the metadata control unit 12, converted into an image in the thumbnail display 13, and then superimposed on the reproduced video by the graphics superimposition 20. Is displayed.

  When the user selects a desired title, the playback application 11 instructs the playback control 14 to play the selected title. The reproduction control 14 reads the AV stream from the AV stream recording unit 17 to the output stream control 16. The AV stream read by the output stream control 16 is decoded by the AV decoder 15 and displayed on the monitor 93.

  When the user presses a dedicated button on the remote control provided for scene selection, the playback application 11 informs the metadata control unit 12 of the title to be played back and elapsed time information of the playback video. The metadata control unit reads five thumbnails before and after the elapsed time from the metadata control unit 12. The thumbnail selection frame 28 generates a thumbnail selection frame and outputs it to the thumbnail display 13. The thumbnail display 13 expands the metadata of the thumbnail as an image and outputs it to the graphics overlay together with the selection frame.

  As a result of the above processing, as shown in FIG. 2, a total of five thumbnails including before and after the thumbnail (Scene 17) corresponding to the current scene are displayed. In the initial state in which thumbnails are displayed, the remote control selection frame is displayed in accordance with the currently reproduced scene. Here, since the scene 17 is being reproduced, a remote control selection frame is displayed for the thumbnail of Scene17. The user can select a desired scene by moving the remote control selection frame left and right using the left and right keys of the remote control. The thumbnail selection 28 generates a selection frame that moves to the left and right in conjunction with this remote control operation. Furthermore, if the left and right cursor keys are pressed while the remote control selection frame has reached the right or left edge of the screen, the entire displayed thumbnail moves in the opposite direction to the pressed cursor key, so that the current display is displayed. A thumbnail of the scene that has not been displayed is displayed. When the user moves the cursor to a desired thumbnail and presses the “OK” key, the playback application 11 causes the playback control 14 to pause the playback state of the content being played back.

  Furthermore, the playback application 11 instructs the playback control 14 to restart playback from the time specified by the entry information of the scene, based on the entry information indicating the start position of the scene selected by the user. The playback control 14 uses the address map information recorded in the AV management information recording unit 18 to convert the entry time information into AV stream address information and sends it to the output stream control 16. Based on the address information from the playback control 14, the output stream control 16 reads the AV stream data of the selected scene from the AV stream recording unit 17 and outputs it to the coder 15 in AV. The AV decoder 15 decodes the AV stream data from the output stream control 16. As a result, the scene corresponding to the thumbnail selected by the user can be reproduced.

  FIG. 3 is an explanatory diagram showing the relationship between the thumbnail and the entry that is the start position of the scene. Here, a scene change, which is a video change point, is used as a scene boundary. Note that the scene boundary is not limited to a scene change, and may be detected based on, for example, a section where no sound is generated, a codec or format change point, a frequency, the number of channels, and a quantization number change position. Alternatively, the scene may be divided at predetermined time intervals such as 5 minutes, 10 minutes, 15 minutes, or time intervals specified by the user. A scene may be configured for a specific event given from the outside. In each scene divided in this way, a thumbnail image that displays the contents of the scene is generated.

  As shown in FIG. 3, the present embodiment is characterized in that thumbnails are the pictures after the head of the scene, not the pictures at the head of the scene. For images where the video is changing continuously, it is appropriate to adopt the position as an entry that is the boundary of the scene, but it is appropriate to adopt that image as a thumbnail image that is a representative image of the scene. Not. In general, when one image is extracted from a continuously changing video, it is often difficult to know what is displayed only by the image. For this reason, it is inappropriate to use a part of the moving video as a thumbnail image used to select a scene to be played back by the user, and a still image or an image close to a still image should be adopted. Is considered appropriate. Such a problem can be solved by using a picture after a predetermined time instead of a scene start position as a thumbnail image.

  As a method of using a picture that is delayed in time from a scene boundary as a thumbnail, there is a method of using a thumbnail image at a position after a certain time, for example, 3 seconds after detecting a scene change (FIG. 4). In addition, after the scene change is detected, the scene change amount may be selected to be equal to or less than a threshold value, and a certain time elapses (for example, 3 seconds) when the threshold value is less than the threshold value may be selected (FIG. 5). In addition, when the video data is data having motion vector information, a place where the amount of motion vector falls below a certain threshold value may be selected. In this way, by using a picture after a predetermined time from the scene boundary as a thumbnail, it is possible to provide the user with an image that matches the contents of each scene as an image for scene selection.

  FIG. 4 is an explanatory diagram illustrating an example of a thumbnail image selection method. In FIG. 4, the horizontal axis represents the picture number, and the vertical axis represents the image difference value between two adjacent pictures. Here, in order to simplify the description, one picture per second is set as an analysis target. As the image difference value, for example, a histogram of the luminance signal (Y) and the color difference signal (U, V) of two pictures is generated, and the cumulative sum of the difference absolute values of each element is divided by the total number of elements. Also good. In this case, the image difference value takes a value between 0 and 1 at the minimum. When the image difference value exceeds a preset scene change determination threshold, it is determined that a scene change has occurred between pictures.

  In FIG. 4, since the scene change determination threshold is exceeded at picture number P52, it is determined that a scene change has occurred between pictures P51 and P52, and picture P52 is set as a scene boundary. Therefore, the entry information is recorded in the metadata recording unit 19 with the picture P52 as a scene start position, that is, an entry. As described above, the image of the picture P52 is not adopted as a thumbnail corresponding to the scene, and the picture P55 after a predetermined time has elapsed from the picture 52 (here, after 3 seconds have elapsed) is adopted as a thumbnail image. The picture P55 is detected by the input stream control 8, decoded by the analysis decoder 21, and then input to the frame buffer 22. The thumbnail generation 25 generates a thumbnail based on the data of the picture P55 input to the frame buffer 22 and records it in the metadata recording unit 19.

  In this way, a scene change, that is, an image that has passed a predetermined time from the entry position is used as a thumbnail. It is possible to prevent an image from being selected as a thumbnail.

  FIG. 5 is an explanatory diagram illustrating another example of a thumbnail image selection method. In the method shown in FIG. 5, a thumbnail image is selected using a thumbnail determination threshold in addition to the scene change determination threshold. The determination threshold for thumbnail is a threshold for selecting a thumbnail image, and a thumbnail image is selected when the image difference value becomes equal to or less than the threshold after a scene change occurs.

FIG. 6 is an explanatory diagram illustrating another example of a thumbnail image selection method. In the method shown in FIG.
A picture P58 is selected as a thumbnail image after a predetermined time has elapsed (here, 3 seconds have elapsed) after the image difference value falls below the thumbnail determination threshold. As a result, it is possible to select an image during a period in which images having the same contents are continuously displayed (in this example, from picture P55 to picture P58) as a thumbnail image. Therefore, an image whose contents can be easily recognized can be displayed as a thumbnail, and the scene video and the thumbnail image can be more easily associated with each other.

  In the example shown in FIG. 6, a picture after a predetermined time has elapsed since the scene change value of the video falls below the thumbnail determination threshold is used as a thumbnail image. Picture P57) may be employed.

Hereinafter, the operation of the video / audio recording / reproducing apparatus 100 according to the present embodiment will be described using a flowchart.
FIG. 7 is a diagram showing a flow of entry information generation. When recording is started, AV streams are sequentially input to the input stream control 8 (S101). The input stream control 8 detects a GOP header from the AV stream, analyzes and extracts only an I picture called a key frame (S102). When the key frame is extracted, the picture is decoded (S103), and the image difference value is calculated (S104). If it is determined that no scene change will occur (No in S105), the processing of steps S102 to S104 is performed for the next picture.

  On the other hand, when a scene change is detected (Yes in S105), the location where the scene change has occurred is determined to be a scene boundary, and entry information for that position is generated (S106). The generated entry information is recorded in the metadata recording unit 19 (S107). The above operation is performed until the recording is completed and the input stream control 8 detects the end of the stream input. When the input stream control 8 detects a flag indicating the end of the stream from the AV stream (Yes in S108), the recording process ends (S109).

FIG. 8 is a flowchart illustrating an example of operations up to thumbnail image generation. The flowchart of FIG. 8 shows an operation in the case where the picture shown in FIG. 4 is a thumbnail image after a predetermined time has elapsed (after 3 seconds) from the occurrence of the scene change.
As described above with reference to FIG. 7, the input stream control 8 extracts an I picture that is a key frame from the AV stream (S102), and performs a decoding process (S103). After the decoding process, it is determined whether a thumbnail generation timer is set (S104). This thumbnail generation timer is set in the subsequent step S114 after the scene change is detected, that is, after the scene boundary is detected, and thereafter is subtracted every time the key frame is decoded (S105). The thumbnail generation timer is for counting a predetermined time (3 seconds) until a thumbnail image is selected after a scene change is detected. When the predetermined time elapses and the thumbnail generation timer times out (S106), the thumbnail generation timer is used. In the generation 25, a thumbnail image is generated (S107). When the thumbnail image is generated, the thumbnail information is recorded in the metadata recording unit 19 (S108). When the predetermined time has not elapsed until the thumbnail image is generated (No in S106), or when the generation of the thumbnail image is completed (S109), the image difference value of the key frame decoded in step S104 is calculated. (S110), a scene change determination process is performed (S111). When a scene change is detected, entry information indicating a scene boundary is generated (S112) and recorded in the metadata recording unit 19 (S113). When the entry information is recorded, a thumbnail generation timer is set, and the key frame extraction process (S102) is resumed for the subsequent recording stream.
The above processing is performed until the input of the recording stream is completed (S115).

  Here, the timer is subtracted every time a picture is processed, but a timeout may be counted using a timer function existing in the apparatus. Further, the system time superimposed on the AV stream to be recorded may be used.

FIG. 9 is a flowchart illustrating an example of operations up to thumbnail image generation. The flowchart of FIG. 9 shows the operation in the case where the picture shown in FIG. 6 is a thumbnail image after a predetermined time has elapsed (after 3 seconds have elapsed) after the image change value has fallen below the thumbnail determination threshold after the occurrence of the scene change. ing.
In FIG. 9, the processing steps shown in steps S102 to S104 are the same as those described in FIG. After calculating the image difference value in step S104, it is determined whether or not an entry that is a scene start position has been determined (S105). Whether or not the entry has been determined is determined based on whether or not the scene start state is set in the subsequent step S116. That is, a scene change is detected in step S113, entry information generation (S114) and entry information recording (S115) are performed, and then a scene start state is set (S116). If the entry has been determined, it is determined whether the image difference value is equal to or smaller than the thumbnail determination threshold value (S106). If it is determined that it is less than or equal to the determination threshold for thumbnail, the thumbnail generation count is subtracted (S107). When the thumbnail generation count times out after a lapse of a predetermined period after the image difference value falls below the thumbnail determination threshold (S108), thumbnail information is generated (S109) and recorded in the metadata recording unit 19 (S110). When the thumbnail image generation process ends, the scene start state is canceled (S111). Thus, the next scene boundary detection process is performed. On the other hand, if it is determined that the image difference value is equal to or smaller than the thumbnail determination threshold (No in S106), the count value of the thumbnail generation counter is reset (S112), and thumbnail generation is performed until the image difference value is again equal to or smaller than the thumbnail determination threshold. No count is subtracted.
The above processing is performed until the input of the recording stream is completed (S117).

FIG. 10 is a flowchart showing a reproduction operation by thumbnail selection.
When the user inputs a desired process to the video recording / reproducing apparatus 100 using a remote controller or the like, a command encoded by the remote control reception 6 is input to the system control unit 2 (S102). The system control unit 2 branches the process according to the command. When a playback start instruction is input (S103), a playback start instruction is issued to the playback application 11 (S104). As a result, reading of the AV stream from the AV stream recording unit 17 to the output stream control 16 is started, and decoding and image output are started in the AV decoder 15.
Next, when the user presses the “scene select” button provided on the remote controller, a thumbnail display command is sent and a thumbnail display instruction is issued (S105). Thereby, a thumbnail image is read from the metadata recording unit 19 (S106), and the read thumbnail image is displayed (S107).
Next, when the user presses the left and right keys on the remote controller, a selection frame movement instruction is issued (S108), and a process of drawing by moving the selection frame displayed by the thumbnail selection 28 is performed (step S109). When the user finds the desired scene and presses the “decision” key, the system controller 2 is instructed to decide (S110). Next, the playback application 11 is instructed to extract entry information, and the playback control 14 reads the playback control information from the AV management information recording unit 18 and performs a seek operation at the address position of the designated scene ( S111). If a reproduction end instruction is given, the process ends (S113). Otherwise, a new command is input (S102).
With the above operation, the user can play back from the entry position of the corresponding scene by designating the thumbnail on the screen.

  According to the video / audio recording / reproducing apparatus 100 according to the present embodiment described above, an image after a predetermined time can be employed as a thumbnail image from an entry position corresponding to a scene change. As a result, it is possible to eliminate a problem that occurs when an image immediately after entry with a high possibility of continuing the scene change is adopted as a thumbnail image, and to select a more appropriate image as the thumbnail. Specifically, it is possible to avoid a video with intense movement or a video that is displayed instantaneously, and an image close to a still image with little movement can be used as a thumbnail. In addition, since the thumbnail image is generated by selecting an image in a period in which a highly correlated image such as a still image continues in time, it is possible to provide a thumbnail image that matches the contents of the scene and the user can The selected thumbnail image can be confirmed in the video of the scene, and the user can play the scene by selecting the thumbnail image with peace of mind.

  Moreover, since the condition for selecting the thumbnail is only the passage of time, the load on the system for calculating the thumbnail generation determination condition can be reduced. In this embodiment, since the AV stream to be reproduced is directly analyzed to extract entry information and select a thumbnail image, the entry information and time information of the thumbnail image are associated with the AV stream. For this reason, it is possible to perform reproduction by associating the elapsed reproduction time of the AV stream with the time indicated by the entry information and the thumbnail image.

  In this embodiment, as a method for generating an entry that is a scene boundary and a method for generating a thumbnail, an AV stream recorded in an AV stream recording unit such as an HDD is targeted. However, the method is limited to an encoded stream. For example, image data before encoding may be used. Also, a decoder implemented by H / W or the like may be used as the analysis decoder 21. Further, a dedicated H / W encoder may be used for the process of generating thumbnails. Further, since the apparatus can be configured without modifying the tuners 3 and 4, the AV encoder 7, the input stream control 8, and the AV stream recording unit 17 related to recording, it can be easily installed in a conventional video / audio recording / reproducing apparatus. can do.

  In addition, in the thumbnail generation 25, if the thumbnail image to be generated is not suitable as a thumbnail image, another thumbnail image may be generated instead of being used as a thumbnail image. Examples of images that are not suitable as thumbnail images include monochrome images such as black and white, images that do not exist as images, images that are overexposed by the camera flash, etc. An image that does not change whether it is reflected or an image with a low contrast can be considered. Thus, not only selecting an appropriate image as a thumbnail but also performing a thumbnail generation process based on inappropriate data can prevent a problem that the system hangs.

  In the present embodiment, the thumbnail image is selected and generated after the scene boundary is extracted.For example, an appropriate image is detected as a thumbnail, and then the scene boundary that is temporally ahead is detected. Extraction may be performed. In this case, scene boundary detection (for example, scene change detection) need not always be performed, and a reduction in processing load can be expected. Furthermore, since a scene is divided only when an appropriate image is detected as a thumbnail image, an image inappropriate as a thumbnail image is not selected as a thumbnail.

  When scene boundary detection is performed with a fixed value, that is, when a scene is divided at intervals of 5 minutes, the load for scene boundary detection can be reduced. Furthermore, when generating thumbnails in parallel with the recording operation, the length of the scene is known in advance, so that the process of selecting the best thumbnail image in the scene can be easily realized, resulting in a more appropriate result. It becomes possible to select a thumbnail image.

  The video / audio recording / reproducing apparatus 100 according to the present embodiment has a reproducing function, but may be a recording apparatus having only a recording function. Such a recording apparatus can be realized by providing an interface function for outputting the recorded AV stream, reproduction control information, and metadata to the outside. Further, a playback apparatus having a playback function according to the present embodiment may be configured using data output from such an interface or a medium on which data is recorded.

Embodiment 2. FIG.
FIG. 11 is a block diagram showing a configuration of a video / audio recording / reproducing apparatus 101 according to Embodiment 2 of the present invention. The video / audio recording / reproducing apparatus 101 according to this embodiment is characterized in that an audio boundary extraction 26 is provided instead of the scene boundary detection 23 in the video / audio recording / reproducing apparatus 100 shown in FIG.

  In the present embodiment, the analysis decoder 21 also performs decoding on encoded audio data in the AV stream. The audio boundary extraction 26 determines, for example, a section in which silence is occurring in the audio data input through the frame buffer 22 as a scene boundary. The entry 25 creates entry information based on the determination result of the scene boundary and records it in the metadata recording unit 19 as in the first embodiment.

  The speech analysis method is not limited to silence detection, and speech frequency analysis may be performed, and speech information characteristic classification may be performed based on the frequency distribution. In the voice boundary extraction 26, frequency analysis is performed in units of audio frames on a voice signal in a certain section, and a distribution of each frequency is obtained. Next, it is determined which of the frequency distribution patterns classified in advance into “conversation”, “music”, “sports”, “animal voice”, “noise”, etc. Furthermore, a section window composed of a plurality of audio frames is provided, and the frequency distribution pattern with the highest frequency is determined as the corresponding section window pattern. Then, the head of the section window in which the pattern has changed is set as the scene boundary.

  FIG. 12 is an explanatory diagram showing a method for detecting a scene boundary by audio frequency analysis. As shown in FIG. 12A, the audio signal is divided into audio frames. Frequency analysis of audio data is performed in units of divided audio frames, and a frequency distribution pattern closest to each frequency distribution is obtained. In the example shown in FIG. 12A, a section window composed of three audio frames is provided, and the most frequent frequency distribution pattern is obtained for each section window, as shown in FIG. A place where the frequency distribution pattern associated with the section window changes is determined to be a scene boundary which is a transition of the scene, and the scene is configured as shown in FIG.

Next, a method for selecting thumbnail images based on audio information will be described. The video information and audio information delimiters do not necessarily match, but since video changes often occur at the audio delimiters, the scene boundary (the shape pattern of the frequency distribution changes as shown in FIG. 12C). The image of the section window boundary) should not be a thumbnail image. Therefore, a portion where the same frequency distribution pattern continues in the section window is defined as a thumbnail. For example, in the section windows N + 1 to N + 3 where “Sports” continues, AFm + 7, which is the central audio frame in the central section window N + 2, is selected as a thumbnail image.
Subsequent thumbnail generation processing is the same as in the first embodiment.

  According to the present embodiment, scene boundary detection and thumbnail image selection are performed based on audio information, and analysis is performed as compared with the case where these processes are performed based on video information as in the first embodiment. The processing can be reduced in weight and the processing speed can be increased. In particular, when a silent section is detected from an audio signal, the analysis process becomes very light and the process can be completed more quickly. In addition, since audio information is used, accuracy is further improved by dividing a scene into a program characterized by audio, such as a music program or a sports program.

  Further, by combining video and audio to detect a scene boundary and select a thumbnail image, more appropriate processing can be performed. Further, the present invention is not limited to video information and audio information, and various data such as caption information, EPG data superimposed on broadcast waves, copyright management information, and the like may be used. By using these data, accurate scene division becomes possible.

Embodiment 3 FIG.
FIG. 13 is a block diagram showing a configuration of the video / audio recording / reproducing apparatus 102 according to the third embodiment of the present invention. The video / audio recording / reproducing apparatus 102 according to the present embodiment reads the AV stream recorded in the AV stream recording unit 17 to detect a scene boundary and select a thumbnail image. In the video / audio recording apparatus 102 shown in FIG. 13, the configuration other than the dubbing stream control 27 is the same as that of the video / audio recording / reproducing apparatus 100 shown in FIG.

  The dubbing stream control 27 reads the AV stream from the AV stream recording unit 17 and inputs the data of the AV stream to be analyzed to the analysis decoder 21. The dubbing stream control 27 is normally used as a dubbing buffer. In this embodiment, the dubbing stream control 27 is used to convert the AV stream read from the AV stream recording unit 17. Input to the decoder for analysis.

  When the recording is finished, the dubbing stream control 27 sequentially reads the AV stream recorded in the AV stream recording unit 17 and inputs it to the coder 21 for analysis. When the GOP start code is detected, the analysis decoder 21 transfers a packet including the first I picture to the analysis decoder 21. The analysis decoder 21 decodes the I picture. Subsequent operations are the same as those in the first embodiment.

  In this way, by adopting a configuration in which the detection of the scene boundary and the selection of the thumbnail image are performed after the recording is completed, the processing load is not low but the processing load is low as in the first embodiment. Therefore, stable processing can be performed. In addition, when analyzing an AV stream during recording, it is necessary to detect a scene boundary and generate a thumbnail image at a speed that is the same as or higher than the speed of the recording operation. It does not require a special analysis process. In addition, since the processing is performed after recording, it is possible to analyze only for a specific period (for example, the backbone excluding the CM section), thereby reducing the total analysis load and contributing to the reduction of power consumption. Further, since the process can be performed on the title once recorded, it can be used when the content of the title is changed by an editing operation or the like.

100, 101, 102 Recorded audio recording / playback device, 2 system control unit, 3 digital tuner, 4 analog tuner, 5 external input terminal, 6 remote control receiving unit, 7 AV encoder, 8 input stream control, 9 recording application, 10 recording control 11 playback application, 12 metadata control unit, 13 thumbnail display, 14 playback control, 15 AV decoder, 16 output stream control, 17 AV stream recording unit, 18 AV management information recording unit, 19 metadata recording unit, 20 graphic superposition , 21 Analysis decoder, 22 frame buffer, 23 scene boundary extraction, 24 entry generation, 25 thumbnail generation, 26 audio boundary extraction, 27 dubbing stream control, 28 thumbnail selection, 30 network terminal, 31 monitor output terminal , 91 antenna, 92 network, 93 monitors, 94 remote control, 95 screen

Claims (33)

Scene boundary extraction means for extracting a scene boundary from video information;
Entry generating means for generating entry information for accessing the scene boundary extracted by the scene boundary extracting means;
Thumbnail generation means for selecting a picture representing the scene divided by the scene boundary extraction means to generate a thumbnail image;
Thumbnail display means for displaying thumbnail images generated by the thumbnail generation means;
A thumbnail selection means for selecting an arbitrary thumbnail from the displayed thumbnails;
Means for playing a scene corresponding to the selected thumbnail,
The video / audio reproduction device, wherein the thumbnail generation means selects a picture after a predetermined period from a scene boundary as a thumbnail image. 2. The video / audio reproduction apparatus according to claim 1, wherein the thumbnail generation means analyzes the video information and selects a picture with little video change as a thumbnail image. 2. The video / audio reproduction device according to claim 1, wherein the thumbnail generation means detects a difference value between pictures constituting a scene and selects a picture having the difference value equal to or less than a predetermined threshold value as a thumbnail image. 2. The video / audio reproduction device according to claim 1, wherein the thumbnail generation means includes metadata storage means for generating address information of a picture corresponding to the generated thumbnail image as thumbnail information and storing the thumbnail information. . 2. The video / audio reproduction device according to claim 1, wherein the scene boundary extraction means detects the scene boundary based on the detection result of the scene change. 2. The video / audio reproduction device according to claim 1, wherein the scene boundary extraction unit detects a scene boundary based on a change in frequency distribution of the audio data. 2. The video / audio reproduction device according to claim 1, wherein the scene boundary extraction means detects a scene boundary based on data superimposed on the video information. 2. The video / audio reproduction apparatus according to claim 1, wherein the scene boundary extraction unit detects a scene boundary based on address information of a picture corresponding to the generated thumbnail image. Scene boundary extraction means for extracting a scene boundary from video information;
Entry generating means for generating entry information for accessing the scene boundary extracted by the scene boundary extracting means;
Thumbnail generation means for selecting a picture representing the scene divided by the scene boundary extraction means to generate a thumbnail image;
Metadata recording means for recording the entry information and thumbnail information that is address information of a picture corresponding to the generated thumbnail image;
The video / audio editing apparatus, wherein the thumbnail generation unit selects a picture after a predetermined period from a scene boundary as a thumbnail image. 10. The video / audio editing apparatus according to claim 9, wherein the thumbnail generation means detects a difference value between pictures constituting a scene, and selects a picture having the difference value equal to or less than a predetermined threshold value as a thumbnail image. Extract scene boundaries from video information,
Generate entry information to access the extracted scene boundary,
Select a picture that represents the scene divided by the extracted scene boundary to generate a thumbnail image,
Display the generated thumbnail image,
A process of playing a scene corresponding to a thumbnail arbitrarily selected from the displayed thumbnail images;
A video / audio reproduction method, wherein a picture after a predetermined period from a scene boundary is selected as a thumbnail image. 12. The video / audio reproduction method according to claim 11, wherein the video information is analyzed, and a picture with little video change is selected as a thumbnail image. 12. The video / audio reproduction method according to claim 11, wherein a difference value between pictures constituting a scene is detected, and a picture having the difference value equal to or less than a predetermined threshold is selected as a thumbnail image. 12. The video / audio reproduction method according to claim 11, wherein address information of a picture corresponding to the generated thumbnail image is stored as thumbnail information. 12. The video / audio reproduction method according to claim 11, wherein a scene boundary is detected based on a detection result of a scene change. 12. The video / audio reproduction method according to claim 11, wherein a scene boundary is detected based on a change in frequency distribution of the audio data. 12. The video / audio reproduction method according to claim 11, wherein a scene boundary is detected based on data superimposed on the video information. Extract scene boundaries from video information,
Generate entry information to access the extracted scene boundary,
Select a picture that represents the scene divided by the extracted scene boundary to generate a thumbnail image,
A process of recording the entry information and thumbnail information that is address information of a picture corresponding to the generated thumbnail image;
A video / audio editing method, wherein a picture after a predetermined period from a scene boundary is selected as a thumbnail image. 19. The video / audio editing method according to claim 18, wherein a difference value between pictures constituting a scene is detected, and a picture having the difference value equal to or less than a predetermined threshold is selected as a thumbnail image. Recording means for recording video information;
Scene boundary extracting means for extracting a scene boundary from the video information;
Entry generating means for generating entry information for accessing the scene boundary extracted by the scene boundary extracting means;
Thumbnail generation means for selecting a picture representing the scene divided by the scene boundary extraction means to generate a thumbnail image;
Thumbnail display means for displaying thumbnail images generated by the thumbnail generation means;
A thumbnail selection means for selecting an arbitrary thumbnail from the displayed thumbnails;
Means for playing a scene corresponding to the selected thumbnail,
The video / audio recording / reproducing apparatus, wherein the thumbnail generation means selects a picture after a predetermined period from a scene boundary as a thumbnail image. 21. The video / audio recording / reproducing apparatus according to claim 20, wherein the thumbnail generation means detects a difference value between pictures constituting a scene and selects a picture having the difference value equal to or smaller than a predetermined threshold as a thumbnail image. . 21. The video / audio recording / reproducing according to claim 20, wherein the thumbnail generation means includes metadata storage means for generating address information of a picture corresponding to the generated thumbnail image as thumbnail information and storing the thumbnail information. apparatus. 21. The video / audio recording / reproducing apparatus according to claim 20, wherein the scene boundary extracting means detects a scene boundary based on a scene change detection result. 21. The video / audio recording / reproducing apparatus according to claim 20, wherein the scene boundary extraction unit detects a scene boundary based on a change in frequency distribution of the audio data. 21. The video / audio recording / reproducing apparatus according to claim 20, wherein the scene boundary extraction unit detects a scene boundary based on data superimposed on the video information. 21. The video / audio recording / reproducing apparatus according to claim 20, wherein when an inappropriate thumbnail image is generated, the thumbnail generation unit generates a thumbnail image different from the thumbnail image. Record video information,
Extracting scene boundaries from the video information;
Generate entry information to access the extracted scene boundary,
Select a picture that represents the scene divided by the extracted scene boundary to generate a thumbnail image,
Display the generated thumbnail image,
A process of playing a scene corresponding to a thumbnail arbitrarily selected from the displayed thumbnail images;
A video / audio recording / reproducing method, wherein a picture after a predetermined period from a scene boundary is selected as a thumbnail image. 28. The video / audio recording / reproducing method according to claim 27, wherein a difference value between pictures constituting a scene is detected, and a picture having the difference value equal to or less than a predetermined threshold is selected as a thumbnail image. 28. The video / audio recording / reproducing method according to claim 27, wherein address information of a picture corresponding to the generated thumbnail image is stored as thumbnail information. 28. The video / audio recording / reproducing method according to claim 27, wherein a scene boundary is detected based on a detection result of the scene change. 28. The video / audio recording / reproducing method according to claim 27, wherein a scene boundary is detected based on a change in frequency distribution of the audio data. 28. The video / audio recording / reproducing method according to claim 27, wherein a scene boundary is detected based on data superimposed on the video information. 28. The video / audio recording / reproducing method according to claim 27, wherein when an inappropriate thumbnail image is generated, a thumbnail image different from the thumbnail image is generated.

Images