JP5540376B2 - Frame-split image generation apparatus and program - Google Patents

Description

  The present invention relates to a frame-split image generating apparatus and a program, and more particularly, to a frame-split image generating apparatus and a program for generating a frame-split image representing video content and presentation impression.

  Conventionally, there has been much research on video summarization. Many of them are techniques for detecting a point where a cut (a point at which the camera is switched), a high sound volume, or a specific object (for example, the main character's face) in an image appears as an important point (key frame). That is, the conventional method is intended to summarize the video based on the content shown in the video.

  For example, there is known a comic generation apparatus that automatically detects a comic image by detecting a person area from a video signal and an audio signal of a television program and the like, recognizing a dialogue, adding a speech balloon and an effect line ( Patent Document 1).

  By the way, an image showing a smiling person can be interpreted in various ways depending on how the image is shown. For example, when taking a picture of a woman inviting to eat, a woman looks adorable at a high angle (taken from the top of a person), while a woman at a low angle (taken from the bottom of a person) Looks like a high pressure. Thus, in order to interpret the meaning of the content shown in the video, information on how it is shown is indispensable.

  And how the audience receives the image of the subject is called the grammar of the movie, and it has been compiled in the first half of the 20th century and has reached today. Therefore, from the physical structure such as camera work (lens work, angle, shot size, lighting, subject and camera movement) and sound (sound effects, speech sounds, music) It is considered that an impression can be predicted (hereinafter referred to as a production impression).

JP 2003-85572 A

  In the conventional video summarization method, even if a person's face and object can be seen, the information indicating how the person is reflected is not stored. There is a problem that not enough information is stored to interpret what it means.

  The present invention has been made to solve the above-described problems, and provides a frame-divided image generation apparatus and a program that can generate a frame-divided image representing the contents represented by the video signal and the presentation impression. Objective.

In order to achieve the above object, the frame-split image generating apparatus according to the present invention is configured to specify the specific image impression for each of a plurality of types of specific presentation impressions based on an input video signal or an audio signal corresponding to the video signal. For each of the plurality of types of specific effect impressions, a plurality of image portions corresponding to the video portions detected by the effect detection unit from the video frame of the video signal. Frame selection means for selecting a video frame, and for each of the plurality of types of specific presentation impressions , the frame selection is performed for each frame divided into a plurality of frames corresponding to the specific presentation impression. generating means for generating each respectively assigned panel layout images the image frame selected as the specific effect impression by means only contains the The shape of the boundaries of each frame of Ma split has a shape corresponding to the type of the specific effect impression.

The program according to the present invention allows a computer to detect a video portion having a specific presentation impression for each of a plurality of types of specific presentation impressions based on an input video signal or an audio signal corresponding to the video signal. Effect detection means, frame selection means for selecting a plurality of video frames corresponding to the video portion detected by the effect detection means, from the video frames of the video signal , for each of the plurality of types of specific effect impressions , and Each of the plurality of types of specific effect impressions is selected as the specific effect impression by the frame selection unit for each frame that corresponds to the specific effect impression and is divided into a plurality of frames. program der to function as generating means for generating each respectively assigned panel layout images the video frame Te, the shape of the boundary of each frame of the panel layout, and a shape corresponding to the type of the specific effect impression.

  According to the present invention, the effect detection means detects a video portion having a specific effect impression based on the input video signal or an audio signal corresponding to the video signal. The frame selection means selects a plurality of video frames corresponding to the video portion detected by the effect detection means from the video frames of the video signal.

  Then, the generation unit generates a frame-divided image corresponding to a specific effect impression and assigning the video frame selected by the frame selection unit to each frame divided into a plurality of frames.

  In this way, the content represented by the video signal is generated by generating a frame-divided image in which a plurality of video frames corresponding to a video portion having a specific effect impression are assigned to each frame allocated to the frame corresponding to the specific effect impression. At the same time, it is possible to generate a frame-divided image representing what kind of psychological impression it gives to the audience.

  The effect detection means according to the present invention detects a video portion having a specific effect impression for each of a plurality of types of specific effect impressions based on the video signal or the audio signal, and the frame selection means detects the plurality of types of specification effects. For each of the production impressions, a plurality of video frames corresponding to the video part in which the specific production impression is detected is selected from the video frames of the video signal, and the generation unit A frame-divided image in which a video frame selected as a specific effect impression is assigned to each frame that corresponds to a specific effect impression can be generated. As a result, for a plurality of types of effect impressions, it is possible to generate a frame-divided image representing the contents represented by the video signal and the effect impressions.

  The shape of the boundary of each frame of the above-mentioned frame division can be made to be a shape according to a specific type of effect impression.

Specific effect impression of the includes a tense scenes, a panel layout that corresponds to the tense scene, can be obtained by dividing into a plurality of frames by a boundary including a diagonal line. This can be a panel layout showing that it is tense scene.

Further, the effect detection means detects a cut that is a boundary of continuous shooting based on a difference in color information between successive video frames of the video signal, and tightens a predetermined range in which the number of cut detections is equal to or greater than a threshold. it can be detected as an image portion to be a scene.

  The above-mentioned specific performance impression includes a shocking scene, and the frame division corresponding to the shocking scene is divided into a plurality of frames by a boundary including diagonal lines, and is larger in size than other frames. It can have a frame. As a result, it is possible to divide the frame into scenes that represent a shocking scene.

  Further, the effect detection means can detect a video portion that becomes a shocking scene based on an audio signal corresponding to the video signal.

  The specific production impression includes a conversation scene or a slow scene, and a frame layout corresponding to the conversation scene or the slow scene is regularly arranged, and a plurality of frames each corresponding to a size. It can be divided into two. As a result, the frame can be divided into frames representing a conversation scene or a slow scene.

  Further, the effect detection means can detect a video portion that becomes a conversation scene or a slow scene based on an audio signal corresponding to the video signal.

  As described above, according to the frame-split image generating apparatus and program of the present invention, a plurality of video frames corresponding to a video portion having a specific effect impression are assigned to each frame that corresponds to the specific effect impression. By generating the assigned frame divided image, it is possible to generate a frame divided image representing what kind of psychological impression it gives to the audience together with the contents represented by the video signal.

It is the schematic which shows the structure of the video equipment which concerns on the 1st Embodiment of this invention. It is a figure which shows a RGB histogram. It is an image figure which shows the example of the frame division corresponding to the scene with a feeling of tension. It is an image figure which shows the example of the frame division corresponding to a shocking scene. It is an image figure which shows the example of the frame division corresponding to a conversation scene. (A) It is an image figure which shows the example of a frame, (B) It is an image figure which shows a mode that the shape of the frame was superimposed on the frame. It is a flowchart which shows the content of the frame division | segmentation image generation process routine in the video equipment which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the content of the tension | tensile_strength scene frame image generation process routine in the video equipment which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the content of the cut frequency calculation process routine in the video equipment which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the content of the impact scene frame image generation process routine in the video equipment which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the content of the conversation scene frame image generation process routine in the video equipment which concerns on the 1st Embodiment of this invention. It is an image figure which shows the example of the frame division corresponding to a normal scene.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. An example in which the present invention is applied to a video device that records and reproduces a video signal will be described.

  As shown in FIG. 1, the video equipment 10 according to the first embodiment includes a CPU, a RAM, and a ROM that stores a program for executing a frame-divided image generation processing routine described later. Is configured as follows.

  The video equipment 10 includes a movie storage unit 12 that stores movie data composed of video and audio signals input via a recording medium such as a DVD, and movie data that acquires movie data to be processed from the movie storage unit 12. An acquisition unit 14, an effect detection unit 16 that detects a video portion having a specific effect impression, and a frame allocation storage unit 18 that stores frame allocation data prepared in advance for each type of the specific effect impression, are detected. A frame division acquisition unit 20 that acquires frame division data corresponding to a specific production impression from the frame division storage unit 18 and a plurality of key frames are selected from the video signal of the movie data according to the detected specific production impression. The key frame selection unit 22 and the selected key frame are allocated to each frame of the acquired frame division data to generate a frame division image, which is used as a summary of movie data. The panel layout image generation unit 24 to be stored in the movie storage section 12, the movie data and the panel layout image stored in the movie storage unit 12, and a display control unit 26 to be displayed on the display device 40 such as a display. A frame is a transmission unit in video and is a single image. A scene is a portion where a person who has watched a video or a manga receives a semantic collection or production. A frame is a unit obtained by dividing a page by a line segment, and a page is one side of a piece of paper in a cartoon of a paper medium. The production is an impression received from a video or comic by a person who views the video or comic.

  Next, the principle of detecting a plurality of types of specific effect impressions will be described. First, the case where a scene with a sense of tension (or a scene with movement) is detected as a specific performance impression will be described.

  In movies, there are various techniques for producing a sense of urgency. For example, it is a method of producing a continuous high tone as a sound effect, moving a camera or subject largely, or frequently switching shots within a certain time. In the present embodiment, attention is paid to the last method. This technique aims to change the viewer's field of view and disturb the psychological state. In other words, it can be said that the strained part in the video is a part where cuts frequently appear in a short time in the video. Therefore, it is considered that the tightened part can be detected by detecting the part where the cut frequency protrudes in the video. Here, a cut is a shot boundary, and a shot is a portion of an image that has been continuously captured by a single camera.

  In the present embodiment, the effect detection unit 16 sequentially obtains the cut frequency in the analysis window while shifting the analysis window for T seconds by 1 second from the beginning of the video signal of the input movie data. The cut is obtained by performing threshold processing on the difference in color distribution between frames in the video. When the cut frequency in the analysis window is higher than a predetermined threshold, the position of the analysis window is detected as a location that becomes a tense scene. Note that the threshold regarding the cut frequency may be obtained in advance by experiments or the like. For example, the threshold is set to 7 (times / 10 seconds).

  Note that, in addition to normal switching, there is an effect such as a dissolve (an effect of switching smoothly while mixing video frames of adjacent shots) as a method of connecting between shots. In the present embodiment, only normal switching is targeted as one that contributes most to a sense of urgency. In many cases, different stage apparatuses are photographed before and after the shot boundary, and abrupt changes occur in the color distribution. Therefore, the effect detection unit 16 calculates an RGB histogram as shown in FIG. 2 as the color distribution for each of the current frame and the next frame, and determines the difference between the histograms for each color channel as a color distribution difference ( Bhattacharya) Find the distance. Here, the batcha rear distance is defined as the amount of self-information for the simultaneous probability of two independent events, and is invariant to affine transformation (translation, rotation, expansion / contraction). I do not receive it. If the average distance between the RGB channels is equal to or greater than a preset threshold, it is determined that a cut has occurred between the frames.

  It should be noted that the threshold value for the batcha rear distance may be determined in advance through experiments or the like. For example, when the cut frequency calculation range (analysis window) is T = 10 seconds and the cut frequency is calculated while shifting T / 2 = 5 seconds, the threshold for the batch rear distance is set to 0.1 to 0.4. Is changed in increments of 0.02 to obtain a value 0.36 in which the slope of the ROC (Receiver Operating Characteristic) curve is maximized, and this is used as a threshold value.

Further, the threshold value for the cut frequency may be set in advance, for example, seven times by an experiment or the like. In this case, the analysis window position exceeding this is detected as a part that becomes a tight scene that is rarely seen in the statistics, and the cut (A) as a true value detected visually, the cut (B) detected by the program As for the correct answer cut (C) among the cuts detected by the program, the recall and precision were calculated by the following equations (1) and (2). As a result, the recall was 53.8% and the precision was 81.4%.
Reproducibility = | C | / | A | (1)
Compliance rate = | C | / | B | (2)

  Next, a case where a shocking scene is detected as a specific effect impression will be described.

  The effect detection unit 16 sequentially corresponds to the video signal in the analysis window while shifting the analysis window for T seconds (for example, 1 second) by T / 2 seconds from the beginning of the input video data of the movie data. Calculate the short-term average power of the audio signal. The location of the shocking scene is obtained by performing threshold processing on the short-time average power in the video. If the short-term average power in the analysis window is greater than a predetermined threshold, the position of the analysis window is detected as a shocking scene. Note that the threshold for the short-term average power may be obtained in advance by experiments or the like.

  Next, a case where a conversation scene (slow scene) is detected as a specific effect impression will be described.

  The effect detection unit 16 sequentially performs voice detection processing in the analysis window while shifting the analysis window for T seconds (for example, 62.5 ms) from the beginning of the input video signal of the movie data by T / 2 seconds. And determine whether voice is detected. When voice is detected, the analysis window is marked as a voice section. The effect detection unit 16 calculates the ratio of the voice section of the past M seconds (for example, 10 seconds), and when the ratio of the voice section is equal to or greater than a predetermined threshold (for example, 0.4), the position of the analysis window is calculated. Detect as a conversation scene.

  Next, the principle of this embodiment will be described.

  The production technique similar to the image is also developed in the comics. For example, Non-Patent Document 1 (Takahiro Akita, “From“ Koma ”to“ Film ””, Manga and Manga Movie, NTT Publishing, 2005) and Non-Patent Document 2 (Osamu Takeuchi, “Introduction to Manga Expression Studies”, Chikuma Shobo, In the comics using books as a medium, when dividing a scene into pages and dividing the page into frames, the number of pages, the number of frames, the size of the frames, By changing the shape, it controls how the audience receives the picture drawn on the frame (in the present embodiment, these are collectively referred to as frame division). It is thought that the object can be expressed in the same direction in both movies and comics.

  In the present embodiment, a location where a specific effect is made is detected from the movie, and a key frame in the vicinity thereof is arranged in a comic frame having a similar effect impression so that it is reflected in that location. We propose a method for summarizing the subject and the impression of the performance at the same time.

  Next, the frame division data prepared in advance for each type of specific effect impression will be described.

  FIG. 3 shows an example of frame division corresponding to a scene with tension. In frame division corresponding to a scene with a sense of tension, when attention is paid to the shape of the boundary between frames, diagonal lines are frequently used. The slanted line has a property that makes the audience feel the movement in the grammar of the movie, and is considered to be a common rule between movies and comics in order to produce a dynamic aspect. It is done. Also, since the line of sight is moved from frame to frame without gazing at the specific frame, the difference in frame size is not set so that the core frame stands out. In the frame allocation storage unit 18, one type of frame allocation data representing frame allocation as shown in FIG. 3 is stored in advance corresponding to a scene with a sense of tension.

  FIG. 4 shows an example of frame division corresponding to a shocking scene. There is a frame that is larger than the other frames as it is identified as a core frame, and an impact is given by the large frame. The large frame is surrounded by borders that are not substantially diagonal, while it has several small frames that are surrounded by boundaries that are diagonal lines.

  For example, in the frame division corresponding to the shocking scene, the incident that occurred here is drawn on the large frame on the right, and the surprised faces of the three people surrounding the incident are displayed on each of the three vertical columns on the left. Draw. As a result, it is possible to produce a shocking scene in which the incident occurred is very shocking, and the person's time has been stopped as if the time has stopped. In the frame allocation storage unit 18, one type of frame allocation data representing frame allocation as shown in FIG. 4 is stored in advance corresponding to a shocking scene.

  FIG. 5 shows an example of frame division corresponding to a conversation scene (slow scene). A plurality of frames of medium size and approximately the same size are regularly arranged. The shift of the boundary between the frames guides the line of sight in the order of the frame images. In the frame allocation storage unit 18, one type of frame allocation data representing frame allocation as shown in FIG. 5 is stored in advance corresponding to the conversation scene.

  The key frame selection unit 22 selects, from the analysis window in which a scene with a sense of tension is detected, the top frame for six shots equal to the number of frames divided as a frame corresponding to the scene with a sense of tension.

  The key frame selection unit 22 includes the first frame of the shot including the position of the analysis window in which the shocking scene is detected and the first frame for the subsequent three shots (when combined, the first frame for four shots equal to the number of frames divided) Frame) is selected as the key frame corresponding to the shocking scene.

  The key frame selection unit 22 selects, from the first shot in the range where the conversation scene is detected, the first frame for eight shots equal to the number of frames divided as a frame corresponding to the conversation scene.

  The frame division image generation unit 24 allocates the selected key frames to each frame division corresponding to the specific effect impression, generates a frame division image, and the movie storage unit 12 as a summary of movie data. Remember me.

  In assignment to a frame, first, as shown in FIG. 6B, the shape of the frame is enlarged / reduced at a ratio inscribed in the frame of the video, and superimposed on the frame so that the centers of gravity coincide. In this state, in the frame as shown in FIG. 6A, the area in the frame that overlaps the frame is copied and allocated to the frame as it is. Note that the order of the frames of the frame division data is determined in advance, and the key frame is assigned to each frame by combining the order of the frame numbers of the selected key frames and the order of the frames. For example, the frame order of the frame division data is determined such that the closer the frame position is to the upper right, the faster the order is, and the lower the right position is, the slower the order is.

  The display control unit 26 receives the corresponding movie data and frame allocation from the movie storage unit 12 when browsing of movie data is instructed by the operation unit (not shown) or the remote operation unit (not shown) of the video equipment 10. The image data is read out, and frame display image data or movie data as a summary of movie data is displayed on the display device 40.

  Next, the operation of the video equipment 10 according to the present embodiment will be described. First, movie data is input to the video equipment 10 via a recording medium, and the input movie data is stored in the movie storage unit 12.

  Then, in the video equipment 10, a frame-divided image generation processing routine shown in FIG. 7 is executed.

  First, in step 100, movie data that is a generation target of a frame-split image is acquired from the movie storage unit 12. Then, in step 102, a frame-cut image corresponding to a scene with a sense of tension is generated and stored in the movie storage unit 12, and in step 104, a frame-cut image corresponding to a shocking scene is generated, Store in the storage unit 12.

  In the next step 106, a frame-cut image corresponding to the conversation scene is generated and stored in the movie storage unit 12, and the frame-cut image generation processing routine is terminated.

  The step 102 is realized by a tight scene frame image generation processing routine shown in FIG.

  In step 110, time t is set to 0, and in step 112, the cut frequency is calculated within the analysis window that is in the range of T seconds from time t.

  In step 114, it is determined whether or not the cut frequency calculated in step 112 is equal to or greater than a threshold value. If the cut frequency is less than the threshold value, the process proceeds to step 116. If so, it is determined that a tense scene has been detected, and the process proceeds to step 120.

  In step 116, it is determined whether or not the analysis window has reached the video end position. If the analysis window has reached the video end position, it is determined that no tension scene has been detected, and a frame-split image corresponding to the tension scene is determined. Without generating, the tension scene frame image generation processing routine is terminated.

  If it is determined in step 116 that the analysis window has not reached the video end position, in step 118 the time t is advanced by 1 second to shift the analysis window, and the process returns to step 112.

  In step 120, the frame number of the first frame of each shot in the analysis window in which the cut frequency is determined to be greater than or equal to the threshold value in step 114 is extracted. In the next step 122, frame division data corresponding to a scene with a sense of tension is acquired from the frame division storage unit 18.

  Then, in step 124, the frame number of the top frame extracted in step 120 is selected by the number of frames equal to the frame number of the frame division data acquired in step 122 in order from the top, and from the video signal, Select multiple keyframes.

  In step 126, each of the key frames selected in step 124 is sequentially assigned to each frame of the frame division data to generate a frame division image. As a summary of the movie data acquired in step 100, movie storage is performed. The data is stored in the unit 12 and the process returns to step 116.

  The step 112 is realized by a cut frequency calculation processing routine shown in FIG.

  First, in step 130, the first frame in the analysis window is set as a target frame, and the cut frequency is set to an initial value 0.

  In step 132, the RGB histogram of the target frame is calculated. In step 134, the RGB histogram of the next frame of the target frame is calculated.

  In step 136, the virtual distance between the RGB histograms calculated in steps 132 and 134 (the average value of the virtual distance between the RGB channels) is calculated. Next, in step 138, it is determined whether or not the virtual distance calculated in step 136 is greater than or equal to a threshold value. If the calculated virtual distance is less than the threshold value, the interval between the target frame and the next frame is determined. It is determined that there is no cut, and the process proceeds to step 142. On the other hand, if the calculated virtual distance is greater than or equal to the threshold, it is determined that there is a cut between the target frame and the next frame, and in step 140, the cut frequency is incremented and counted, and the process proceeds to step 142.

  In step 142, it is determined whether or not the above processing has been completed for all the frames in the analysis window. If the next frame has reached the final frame, it is determined that the above processing has been completed for all the frames. Then, the cut frequency calculation processing routine is terminated using the current cut frequency as a calculation result. On the other hand, if the next frame has not reached the final frame, in step 144, the next frame is set as the target frame, and the process returns to step 132.

  Step 104 is realized by the impact scene frame image generation processing routine shown in FIG.

  In step 150, time t is set to 0, and in step 152, short-term average power is calculated within an analysis window that is in the range of T seconds from time t. For example, the short-time average power is calculated by calculating the root mean square of the speech waveform sample values for T seconds.

  In step 154, it is determined whether or not the short-time average power calculated in step 152 is equal to or greater than a threshold value. If the short-time average power is less than the threshold value, the process proceeds to step 156. If the short time average power is greater than or equal to the threshold, it is determined that a shocking scene has been detected, and the process proceeds to step 160.

  In step 156, it is determined whether or not the analysis window has reached the video end position. If the analysis window has reached the video end position, it is determined that no shocking scene has been detected, and the shocking scene is handled. The impact scene frame image generation processing routine is terminated without generating the frame-cut image.

  If it is determined in step 156 that the analysis window has not reached the video end position, in step 158, the time t is advanced by T / 2 seconds and the analysis window is shifted by T / 2. Return to 152.

  In step 160, the frame number of the first frame of the shot including the position of the analysis window in which the short-term average power is determined to be equal to or greater than the threshold value in step 154 is extracted. In the next step 162, frame division data corresponding to a shocking scene is acquired from the frame division storage unit 18.

  In step 164, a key frame having the frame number of the first frame extracted in step 160 is selected from the video signal, and each first frame of a plurality of subsequent shots is selected as a key frame from the video signal. The key frames corresponding to the number of frames equal to the number of frames of the frame division data acquired in step 122 are selected.

  In step 166, each of the key frames selected in step 164 is sequentially assigned to each frame of the frame division data to generate a frame division image, and a movie storage unit as a summary of the movie data acquired in step 100 above 12 and return to step 156.

  Step 106 is realized by the conversation scene frame image generation processing routine shown in FIG.

  In step 170, time t is set to 0, and in step 172, sound detection processing is performed on the sound signal of the movie data acquired in step 100 in the analysis window that is in the range of T seconds from time t. . It should be noted that since a conventionally known method may be used in the voice detection process, description regarding the voice detection process is omitted.

  In step 174, it is determined whether or not sound is detected by the sound detection processing in step 172. If no sound is detected, the process proceeds to step 176. On the other hand, the sound is detected. If yes, go to Step 180.

  In step 176, it is determined whether or not the analysis window has reached the video end position. If the analysis window has reached the video end position, it is determined that no conversation scene has been detected, and a frame-split image corresponding to the conversation scene is obtained. Without generating, the conversation scene frame image generation processing routine is terminated.

  If it is determined in step 176 that the analysis window has not reached the video end position, in step 178 the time t is advanced by T / 2 seconds to shift the analysis window by T / 2, Return to 172.

  In step 180, the section of the analysis window in which it is determined that the voice is detected in step 174 is marked as a voice section. In the next step 182, the ratio of the section marked as the voice section in the range of the past M seconds including the section of the current analysis window is calculated.

  In step 184, it is determined whether or not the ratio calculated in step 182 is equal to or greater than the threshold value. If the ratio is less than the threshold value, the process proceeds to step 178. If there is, it is determined that a conversation scene has been detected, and the process proceeds to step 184.

  In step 186, frame division data corresponding to the conversation scene is acquired from the frame division storage unit 18.

  In step 188, the head frame is selected as a key frame from the head shot of the video signal in the range of the past M seconds determined in step 184 that the proportion of the audio section is equal to or greater than the threshold, and a plurality of subsequent frames are selected. The first frame of each shot is selected from the video signal as a key frame, and key frames corresponding to the number of frames equal to the number of frames of the frame division data acquired in step 186 are selected.

  In step 190, each of the key frames selected in step 188 is sequentially assigned to each frame of the frame division data to generate a frame division image, and a movie storage unit as a summary of the movie data acquired in step 100 12 and return to step 176.

  When the user operates the operation unit (not shown) or the remote operation unit (not shown) of the video device 10 to instruct to browse movie data, the movie storage unit 12 is displayed by the display control unit 26 of the video device 10. From the instructed movie data and the frame-divided image corresponding to each type of the specific presentation impression corresponding to the movie data, and according to each type of the specific presentation impression as a summary of the movie data The frame-divided image is displayed on the display device 40, and the movie data is displayed on the display device 40.

  As described above, according to the video equipment according to the first embodiment, a plurality of key frames corresponding to the video portion detected as the specific effect impression are prepared in advance to express the specific effect impression. By creating a frame-by-frame image assigned to each frame of the comic frame, the frame-by-frame image showing what the video signal represents and what psychological impression it gives to the audience, Can be generated as Further, for a plurality of types of presentation impressions, a frame-divided image representing what kind of psychological impression it gives to the audience together with the contents represented by the video signal can be generated.

  Conventionally, video summarization has been performed on the basis of what appears in the video. However, the interpretation of what the subject in the video means in the entire video is possible only when there is information indicating how the subject appears in the video. Therefore, in the present embodiment, a part having a specific effect impression is detected in the video, and a key frame in the vicinity of the part is assigned to the comic frame division according to the effect impression, thereby A method of summarizing both impressions was proposed.

  Next, a second embodiment will be described. Since the configuration of the video equipment according to the second embodiment is the same as that of the first embodiment, the same reference numerals are given and description thereof is omitted.

  In the second embodiment, the shape and size of each frame divided based on the physical characteristics of the video is determined, and the area cut out from the frame based on the object recognition result for the frame This is different from the first embodiment.

  In the second embodiment, the frame division acquisition unit 20 acquires frame division data corresponding to the detected specific production impression from the frame division storage unit 18, and the acquired frame division data and the specific production impression are obtained. Based on the detected physical characteristics of the video portion, the shape and size of each frame of the frame division data is determined, and frame division data corresponding to the detected specific effect is generated.

  The frame division image generation unit 24 detects the position of an object (for example, a human face) in a video frame by object recognition or the like in the allocation to the frame, and uses the grammar of the video and the position of the detected object. Based on this, an area to be cut out from the frame to the frame is determined, and the determined area in the frame is copied to the frame as it is and assigned.

  Note that other configurations and operations of the video equipment according to the second embodiment are the same as those of the first embodiment, and thus description thereof is omitted.

  In the first embodiment and the second embodiment described above, the case where the frame-cut image corresponding to the scene with a sense of tension, the shocking scene, and the conversation scene is generated is described as an example. However, the present invention is not limited to this. In addition, a frame division image corresponding to an ordinary scene may be generated. For example, a part that is not a tense scene, a shocking scene, or a conversation scene is detected as a video part corresponding to a normal scene, a key frame is selected, and a frame allocation corresponding to the normal scene is selected. May be assigned to each frame to generate a frame-divided image corresponding to an ordinary scene. For example, frame division data corresponding to an ordinary scene as shown in FIG. 12 may be prepared in advance. The shape and size of each frame division corresponding to an ordinary scene may be determined so as not to have the characteristics seen in the frame division of other effect impressions.

  Further, although the case where the frame-divided image is generated from the video data such as movie data has been described as an example, the frame-divided image may be generated from the video data obtained by video shooting or the like. It is possible to extract high-impact summaries by extracting characteristic images from video obtained by video shooting, and to create impressive albums. Further, the present invention may be applied to generating a frame-cut image from a still image such as a digital photo image. For example, the user may select a specific effect impression type, and the input digital photo image may be arranged in each frame frame corresponding to the selected specific effect impression to generate a frame-divided image. Good. A photo album can be generated using frame division, and an album having a production effect can be automatically generated.

  Further, although the case where movie data is input via a recording medium has been described as an example, the present invention is not limited to this, and broadcast video data may be stored as input.

  Further, as an example of calculating the batch distance between RGB histograms as the difference in color distribution between video frames, the present invention is not limited to this. For example, the difference between other image feature amounts between frames May be calculated.

  In the present specification, the embodiment has been described in which the program is installed in advance. However, the program may be provided by being stored in a storage medium such as a CDROM.

DESCRIPTION OF SYMBOLS 10 Video equipment 12 Movie memory | storage part 14 Movie data acquisition part 16 Effect detection part 18 Frame allocation memory | storage part 20 Frame allocation acquisition part 22 Key frame selection part 24 Frame allocation image generation part 26 Display control part 40 Display apparatus

Claims (6)

Production detection means for detecting a video portion having the specific production impression for each of a plurality of types of specific production impressions based on an input video signal or an audio signal corresponding to the video signal;
Frame selection means for selecting a plurality of video frames corresponding to the video portion detected by the effect detection means from the video frames of the video signal for each of the plurality of types of specific presentation impressions ;
Each of the plurality of types of specific effect impressions is selected as the specific effect impression by the frame selection unit for each frame that corresponds to the specific effect impression and is divided into a plurality of frames. a generating means each for generating a respective assigned frame split images the video frame,
A panel layout image generating apparatus comprising,
A frame-divided image generating apparatus in which a shape of a boundary between the frame-divided frames is a shape corresponding to the specific effect impression type. The specific performance impression includes a tense scene in which cuts frequently appear in a short time in the video ,
The frame division corresponding to the tense scene is divided into a plurality of frames by a boundary including an oblique line ,
The effect detection means detects a cut that is a boundary of continuous shooting based on a difference in color information between successive video frames of the video signal, and sets the predetermined range in which the number of detections of the cut is equal to or greater than a threshold value, The frame-split image generating apparatus according to claim 1 , wherein the frame-divided image generating apparatus detects a video portion that is a scene . The specific performance impression includes a shocking scene,
The frame division corresponding to the shocking scene is divided into a plurality of frames by a boundary including an oblique line, and has a frame that is larger than the other frames ,
The effect detection means calculates the short-time average power of the audio signal corresponding to the video signal in the analysis window based on the audio signal corresponding to the video signal, and the calculated short-time average power is greater than a threshold value. 3. The frame-split image generating apparatus according to claim 1, wherein a video signal of the analysis window is detected as a video portion that becomes the shocking scene. The specific performance impression includes a conversation scene or a slow scene,
The panel layout corresponding to the conversation scenes or slow scene, are regularly arranged, and any one of claims 1 to claim 3 in which each size is assumed to have divided into a plurality of frames which correspond The frame-split image generating apparatus according to claim 1. 5. The frame-divided image generation device according to claim 4 , wherein the effect detection means detects a video portion that becomes the conversation scene or a slow scene based on an audio signal corresponding to the video signal. Computer
An effect detection means for detecting a video portion having the specific effect impression for each of a plurality of types of specific effect impressions based on an input video signal or an audio signal corresponding to the video signal,
Frame selection means for selecting a plurality of video frames corresponding to the video portion detected by the effect detection means from the video frames of the video signal for each of the plurality of types of specific presentation impressions ; and
Each of the plurality of types of specific effect impressions is selected as the specific effect impression by the frame selection unit for each frame that corresponds to the specific effect impression and is divided into a plurality of frames. wherein a program for functioning as a generator, each for generating a respective assigned frame split image video frame,
The program which made the shape of the boundary of each frame of the said frame division the shape according to the kind of said specific production impression.

Images