JP2014006912A - Information processing apparatus, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing apparatus and an information processing method for automatically extracting an image and displaying the image so that the details can be easily grasped.SOLUTION: An information processing apparatus according to the present invention includes: an image analysis unit that analyzes image data to generate image meta data that is meta data related to a feature amount that characterizes the image; a frame information determination unit that extracts a plurality of images from the image data on the basis of the image meta data, and determines frame information related to frames for arranging the extracted plurality of images; and image display data generation unit that generates image display data including at least the frame information and data of the extracted images.

Description

  The present invention relates to an information processing apparatus, an information processing method, and a program.

  With the development of information processing technology, digital imaging devices such as digital still cameras and digital video cameras are becoming popular. With regard to these digital imaging devices, technical development has been actively conducted on both a technique for improving convenience during imaging and a technique used when displaying captured data to a user.

  As an example of a technique for improving convenience during imaging, for example, in Patent Document 1 below, a face area of a person as a subject is detected, and the subject is focused on the change in the size of the face area and the moving speed. A technique for verifying the possibility of blurring is disclosed.

  Moreover, as an example of a technique used when displaying captured data to a user, there is a technique disclosed in Patent Document 2 below, for example. This technology displays a thumbnail image related to an event experienced by a user on a display screen, and displays a speech generated according to the event as a balloon for a highly important thumbnail image representing the event. .

JP 2008-22300 A JP 2005-141297 A

  By the way, in order to grasp the contents of moving image data generated by an imaging apparatus capable of moving image shooting such as a digital video camera, the time that can be shot has increased along with the development of technology, which requires enormous time. . At this time, even if a technique for displaying a plurality of thumbnail images and displaying a speech balloon for a highly important thumbnail image, such as the technique described in Patent Document 2, It may be difficult to grasp the relationship with other images. Therefore, there has been a demand for a technique for automatically extracting an effective image for grasping the contents of a certain moving image and displaying the images so that the contents can be easily grasped.

  Accordingly, the present invention has been made in view of the above problems, and an object of the present invention is to process information that can automatically extract an image and display it for easy understanding of the contents. To provide an apparatus, an information processing method, and a program.

  In order to solve the above-described problem, according to an aspect of the present invention, an image analysis unit that analyzes image data and generates image metadata that is metadata relating to a feature amount that characterizes the image is based on the image metadata. Extracting a plurality of images from the image data and determining frame information regarding a frame for arranging the extracted images; the frame information; and the extracted image data An information processing apparatus is provided that includes an image display data generation unit that generates image display data including at least.

  In order to solve the above-described problem, according to another aspect of the present invention, image data is analyzed, image metadata that is metadata relating to a feature amount characterizing the image is generated, and the image metadata Extracting a plurality of images from the image data based on the image data, determining frame information on a frame for arranging the extracted plurality of images, the frame information, and the extracted image data; Generating an image display data including at least the information processing method.

  In order to solve the above-described problem, according to still another aspect of the present invention, an image analysis function for analyzing image data and generating image metadata that is metadata relating to a feature amount characterizing the image is performed on a computer. A frame information determination function for extracting a plurality of images from the image data based on the image metadata and determining frame information regarding a frame for arranging the extracted plurality of images; There is provided a program for realizing an image display data generation function for generating image display data including at least the extracted image data.

  According to the present invention, image data is analyzed to generate image metadata that is metadata related to a feature amount that characterizes the image, and a plurality of images are extracted from the image data based on the image metadata. At the same time, frame information relating to a frame for arranging the plurality of extracted images is determined, and image display data including at least the frame information and the extracted image data is generated.

  As described above, according to the present invention, a plurality of images are extracted based on the image metadata relating to the feature amount that characterizes the image, arranged, and displayed, so that an image effective for grasping the contents is automatically obtained. And can be displayed so that the contents can be easily grasped.

It is explanatory drawing for demonstrating the comic display of the moving image which concerns on the 1st Embodiment of this invention. It is a block diagram for demonstrating the structure of the information processing apparatus concerning the embodiment. It is explanatory drawing for demonstrating comic display data. It is a block diagram for demonstrating the structure of the moving image analysis part which concerns on the embodiment. It is explanatory drawing for demonstrating moving image metadata. It is explanatory drawing for demonstrating moving image metadata. It is a block diagram for demonstrating the structure of the comic display conversion part which concerns on the same embodiment. It is explanatory drawing for demonstrating the calculation method of the digest score which concerns on the same embodiment. It is explanatory drawing for demonstrating the extraction method of the representative frame image which concerns on the same embodiment. It is explanatory drawing for demonstrating the animation frame image which concerns on the embodiment. It is explanatory drawing for demonstrating the camera work metadata utilization part which concerns on the embodiment. It is explanatory drawing for demonstrating the image size determination part which concerns on the embodiment. It is explanatory drawing for demonstrating the image size determination part which concerns on the embodiment. It is explanatory drawing for demonstrating the frame image arrangement | positioning part which concerns on the embodiment. It is explanatory drawing for demonstrating the frame image arrangement | positioning part which concerns on the embodiment. It is explanatory drawing for demonstrating the frame image arrangement | positioning part which concerns on the embodiment. It is explanatory drawing for demonstrating the frame image arrangement | positioning part which concerns on the embodiment. It is explanatory drawing for demonstrating the image arrangement | positioning which considered camerawork. It is explanatory drawing for demonstrating the image arrangement | positioning which considered camerawork. It is explanatory drawing for demonstrating the effect image arrangement | positioning part which concerns on the embodiment. It is explanatory drawing for demonstrating the effect image arrangement | positioning part which concerns on the embodiment. It is explanatory drawing for demonstrating the effect image arrangement | positioning part which concerns on the embodiment. It is a flowchart for demonstrating the whole flow of the information processing method which concerns on the embodiment. 4 is a flowchart for explaining a method of selecting a representative frame image according to the embodiment. It is a flowchart for demonstrating the selection method of the animation frame image according to camera work. 4 is a flowchart for explaining a method for dynamically arranging representative frame images according to the embodiment. 4 is a flowchart for explaining a method for dynamically arranging representative frame images according to the embodiment. It is a flowchart for demonstrating the arrangement | positioning method of the representative frame image in consideration of camera work. It is a flowchart for demonstrating the arrangement method of the effect image in consideration of camera work. It is a flowchart for demonstrating the determination method of the reproduction speed of a thumbnail animation. It is a block diagram for demonstrating the 1st modification of the moving image analysis part which concerns on the embodiment. It is explanatory drawing for demonstrating the arrangement method of the effect image which concerns on the modification. It is explanatory drawing for demonstrating the arrangement method of the effect image which concerns on the modification. It is explanatory drawing for demonstrating the arrangement method of the effect image which concerns on the modification. It is explanatory drawing for demonstrating the arrangement method of the effect image which concerns on the modification. It is explanatory drawing for demonstrating the arrangement method of the effect image which concerns on the modification. It is explanatory drawing for demonstrating the arrangement method of the effect image which concerns on the modification. It is a block diagram for demonstrating the structure of the information processing apparatus which concerns on the 2nd Embodiment of this invention. It is explanatory drawing for demonstrating the structure of the still image analysis part which concerns on the same embodiment. It is explanatory drawing for demonstrating the structure of the comic display conversion part which concerns on the same embodiment. It is a flowchart for demonstrating the whole flow of the information processing method which concerns on the embodiment. It is a flowchart for demonstrating the determination method of the image size of a still image. It is a block diagram for demonstrating the structure of the information processing method which concerns on the 3rd Embodiment of this invention. It is a block diagram for demonstrating the hardware constitutions of the information processing apparatus which concerns on each embodiment of this invention.

  Exemplary embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

The description will be made in the following order.
(1) First Embodiment (1-1) Outline of Comic Display (1-2) Configuration of Information Processing Device (1-3) Information Processing Method (1-4) First Modification (2) Second Embodiment (2-1) Configuration of Information Processing Device (2-2) Information Processing Method (3) Third Embodiment (3-1) Configuration of Information Processing Device (4) Hardware configuration of information processing apparatus according to each embodiment (5) Summary

(First embodiment)
<About comic display overview>
First, before describing the information processing apparatus and the information processing method according to the first embodiment of the present invention in detail, an outline of comic display of a moving image according to the present embodiment will be described with reference to FIG. FIG. 1 is an explanatory diagram for explaining comic display of a moving image according to the present embodiment.

  The information processing apparatus according to the present embodiment automatically extracts an image of an important scene from a plurality of images included in moving image data when single or a plurality of arbitrary moving image data is designated. For example, as illustrated in FIG. 1, the information processing apparatus rearranges the extracted image in a manga manner and displays it on the display screen.

  Consider a display area 301 as shown in FIG. This display area 301 corresponds to a page in a normal manga. The display area 301 is divided into a plurality of frames 303, and an image (for example, a thumbnail image) 305 included in the moving image data is arranged in each frame.

  In addition, on the thumbnail image 305, various balloons 307, screen effects such as a concentrated line 309, an effect line 311, and the like are automatically arranged based on moving image metadata described below. The balloon 307, the concentrated line 309, the effect line 311 and the like are freely arranged inside and outside the frame 303 like a manga on a normal paper medium.

  In addition, there may be a case where the user of the information processing apparatus wants to understand what kind of scene a certain thumbnail image 305 is. At this time, the user operates an input device provided in the information processing apparatus such as a mouse and a keyboard to move the position selection object 313 such as a mouse pointer to a desired thumbnail image 305, and is in a state called a mouse over. To. Then, the information processing apparatus executes an animation using the thumbnail image 305 as a start image in the frame for the thumbnail image 305 on which the position selection object 313 is superimposed.

  Although only one display area 301 is shown in FIG. 1, the size of the display area 301 can be changed depending on the size of the display screen or the like, and a plurality of display areas 301 can be included in one display screen. Can also be arranged.

  Since the information processing apparatus arranges the images constituting the moving image in the order of the time information associated with each image, the user can easily grasp the contents of the moving image data in the same manner as reading a normal manga. It is possible. Further, since a screen effect such as a balloon (hereinafter also referred to as an effect) is automatically arranged according to the moving image metadata associated with the thumbnail image, the contents can be confirmed while enjoying the moving image data.

<Configuration of information processing device>
Hereinafter, the configuration of an information processing apparatus capable of realizing comic display of a moving image as described above will be described in detail with reference to FIGS.

[Overall configuration of information processing device]
First, the overall configuration of the information processing apparatus according to the present embodiment will be described in detail with reference to FIG. FIG. 2 is a block diagram for explaining the configuration of the information processing apparatus according to the present embodiment.

  As shown in FIG. 2, for example, the information processing apparatus 10 according to the present embodiment includes a moving image data acquisition unit 101, a moving image analysis unit 103, an audio extraction unit 105, a comic display conversion unit 107, a frame image data generation unit 109, a comic A display data generation unit 111 is provided. The information processing apparatus 10 further includes a comic display execution unit 113, a display control unit 115, and a storage unit 117. In addition to these processing units, the information processing apparatus 10 includes a communication control unit (not shown) that controls communication performed with an arbitrary device via an arbitrary communication network, for example. You may do it.

  The moving image data acquisition unit 101 includes, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a communication device, and the like. The video data acquisition unit 101 stores video data corresponding to a video specified by the user in response to a user operation performed on the information processing device 10. Obtained from recording media. Here, the moving image data means moving image data composed of a plurality of images and sounds associated with time information. In addition, the moving image data acquisition unit 101 acquires moving image data from another device connected to the information processing device 10, or acquires moving image data from various servers connected via a communication network such as the Internet. It is possible. The moving image data acquisition unit 101 transmits the acquired moving image data to the moving image analysis unit 103 and the audio extraction unit 105, respectively.

  The moving image analysis unit 103 includes, for example, a CPU, a ROM, a RAM, and the like. The moving image analysis unit 103 analyzes the transmitted moving image data and extracts a feature amount that characterizes the moving image. In addition, the moving image analysis unit 103 generates moving image metadata, which is metadata related to the feature amount, using the extracted feature amount. Feature amount extraction is performed for both image data and audio data constituting the moving image data, and the generated moving image metadata includes feature amount metadata relating to image data and feature amount metadata relating to audio data. It is. The moving image analysis unit 103 transmits the generated moving image metadata to the comic display conversion unit 107 described later. In addition to the moving image metadata generated by analyzing the moving image, the moving image analysis unit 103 converts the metadata that can be acquired on an imaging device such as a camcorder into moving image metadata used by the comic display conversion unit 107 described later. It is also possible to use as. Here, examples of metadata that can be acquired on the imaging device include shooting during recording, operation history of favorite buttons, zoom-in buttons, zoom-out buttons, and the like. In addition, the moving image analysis unit 103 transmits the image data included in the moving image data transmitted from the moving image data acquisition unit 101 to the frame image data generation unit 109 described later.

  The moving image analysis unit 103 will be described in detail later.

  The voice extraction unit 105 is constituted by, for example, a CPU, a ROM, a RAM, and the like. The audio extraction unit 105 extracts audio data from the moving image data transmitted from the moving image data acquisition unit 101. When extracting the audio data, the audio extraction unit 105 can refer to a database regarding the format of the moving image data stored in the ROM, the storage unit 117, or the like. Note that the voice extraction unit 105 may perform a conversion process such as compressing the extracted voice data as necessary. The audio extraction unit 105 transmits the extracted audio data of the moving image to the comic display data generation unit 111 described later.

  The comic display conversion unit 107 includes, for example, a CPU, a ROM, a RAM, and the like. The comic display conversion unit 107 extracts a plurality of images from the moving image data based on the moving image metadata, and arranges and converts the plurality of images extracted by dividing a predetermined display area into frames, Frame information including information related to an image arranged on the frame is generated. The comic display conversion unit 107 transmits the generated frame information to a frame image data generation unit 109 and a comic display data generation unit 111 described later.

  The comic display conversion unit 107 will be described in detail later.

  The frame image data generation unit 109 includes, for example, a CPU, a ROM, a RAM, and the like. The frame image data generation unit 109 generates frame image data based on the image data transmitted from the moving image analysis unit 103 and the frame information transmitted from the comic display conversion unit 107. More specifically, the frame image data generation unit 109 extracts a frame image described in the frame information from a plurality of frame images constituting the moving image. Further, the frame image data generation unit 109 performs an enlargement / reduction process on the extracted frame image based on the information described in the frame information to obtain a thumbnail image that matches the size of the frame. Further, the frame image data generation unit 109 generates an image for animation reproduction based on the frame information. The frame image data generation unit 109 transmits the generated frame image data (that is, thumbnail image data and image data for animation reproduction) to the comic display data generation unit 111.

  The comic display data generation unit 111 includes, for example, a CPU, a ROM, a RAM, and the like. The comic display data generation unit 111 uses the various information transmitted from the voice extraction unit 105, the comic display conversion unit 107, and the frame image data generation unit 109, and the effect data acquired from the storage unit 117 and the like, and uses the comic display data. Is generated. For example, as shown in FIG. 3, the comic display data is configured to include at least frame information described in, for example, an XML format, frame image data, audio data, and effect data. The information processing apparatus 10 and other apparatuses can generate comic display screens in various applications by reading and displaying the frame information described in the XML format.

  For example, comic display can be realized by a stand-alone network application using Visual C ++ (registered trademark), a mobile application such as Java / Brew, a browser application using Flash, Javascript, or the like. For example, a Flash file (SWF file) is created using Action Script, and the frame information that is the arrangement information XML of the comic image is read to view the comic display screen using a general Web browser. It is possible.

  Further, by creating comic display data with Flash, more interactive content expression is possible. For example, it is possible to implement such as turning a page by a button or turning a page by scrolling with a mouse wheel. Also, by displaying animations in time-series order, even if there are many frames, it is possible to read them smoothly without getting lost. Further, when a thumbnail image is clicked and audio is reproduced, the currently reproduced frame can be emphasized more clearly by increasing the transparency of all the frames other than the clicked frame. Thus, by outputting the frame information as an XML format file, it can be referred to from various applications including Flash.

  The comic display data generation unit 111 transmits the generated comic display data to the comic display execution unit 113. Further, the comic display data generation unit 111 may provide the generated comic display data to other information processing apparatuses via various recording media such as CD-ROM and DVD-ROM, and communication networks such as the Internet. Is possible.

  The comic display execution unit 113 includes, for example, a CPU, a ROM, a RAM, and the like. The comic display execution unit 113 executes the comic display as shown in FIG. 1, for example, by executing an application for executing the comic display data transmitted from the comic display data generation unit 111. The application for executing the comic display data may be a dedicated application for executing comic display or a general application such as a Web browser. The comic display execution unit 113 executes the comic display data and causes the display unit (not shown) to display the comic display via the display control unit 115 described later.

  The display control unit 115 includes, for example, a CPU, a ROM, a RAM, and the like. The display control unit 115 acquires data corresponding to the contents to be displayed on the display screen recorded in the storage unit 117 and displays the data on the display screen. When a signal indicating movement of the position selection object such as a mouse pointer is transmitted from the input unit provided in the information processing apparatus 10, the movement of the position selection object is displayed on the display screen in accordance with the transmitted signal. Display. In addition, when requested to change the display content from the comic display execution unit 113, the display control unit 115 changes the display content displayed on the display screen based on the request from the comic display execution unit 113. .

  The storage unit 117 stores various types of moving image data such as moving image data shot by a video camera, a mobile phone, and the like, and moving image content such as TV content. The storage unit 117 stores object data displayed on the display screen. The object data referred to here includes, for example, arbitrary parts constituting a graphical user interface (GUI) such as icons, buttons, and thumbnails.

  In addition, the storage unit 117 stores various parameters, intermediate progress of processing, and various databases that need to be saved when the information processing apparatus 10 according to the present embodiment performs some processing, as appropriate. It may be recorded. The storage unit 117 includes a moving image data acquisition unit 101, a moving image analysis unit 103, an audio extraction unit 105, a comic display conversion unit 107, a frame image data generation unit 109, a comic display data generation unit 111, a comic display execution unit 113, and display control. The unit 115 and the like can freely read and write.

[Configuration of Movie Analysis Unit 103]
Next, the configuration of the moving image analysis unit 103 according to the present embodiment will be described in detail with reference to FIGS. 4 to 6. FIG. 4 is a block diagram for explaining the configuration of the moving image analysis unit 103 according to this embodiment, and FIGS. 5 and 6 are explanatory views for explaining moving image metadata.

  For example, as illustrated in FIG. 4, the moving image analysis unit 103 includes a moving image data division unit 131, a face detection unit 133, a scene transition detection unit 135, a camera work detection unit 137, an audio analysis unit 139, and a moving image meta unit. A data generation unit 141.

  The moving image data dividing unit 131 is constituted by, for example, a CPU, a ROM, a RAM, and the like. The moving image data dividing unit 131 divides the moving image data transmitted from the moving image data acquiring unit 101 into image data and audio data. The moving image data division unit 131 transmits the obtained moving image data to the frame image generation unit 109, the face detection unit 133, the scene transition detection unit 135, and the camera work detection unit 137. In addition, the moving image data division unit 131 transmits the obtained audio data to the audio analysis unit 139.

  The face detection unit 133 is configured by, for example, a CPU, a ROM, a RAM, and the like. The face detection unit 133 detects the face of a person existing in the frame image for all the frame images constituting the moving image. Further, when a human face is present in the frame image, the face detection unit 133 determines the number of faces present in the image, whether the existing face is a smile, the position of the face, and the face. Further detection is performed for various information related to the face, such as the direction and whether the face is zoomed up. When performing the detection as described above, the face detection unit 133 can refer to, for example, a detection database stored in advance in the storage unit 117 or the like, or can execute a detection program or the like. is there.

  Among the detection results as described above, when the result can be expressed by binary values of “presence / absence” such as presence / absence of a face, smile / no smile, and zoom-up, the face detection unit 133 The value “1” is output when “present”, and the value “0” is output when “not present”. In addition, the face detection unit 133 may output these detection results themselves, such as the direction in which the face is facing or the number of faces present in the image, and in accordance with a predetermined conversion method. The detection result converted into a certain value may be output as the detection result.

  The face detection unit 133 transmits these detection results to the moving image metadata generation unit 141 described later.

  The scene transition detection unit 135 is constituted by, for example, a CPU, a ROM, a RAM, and the like. The scene transition detection unit 135 detects whether or not a scene transition has occurred for all the frame images constituting the moving image, and outputs the detection result as a value called a scene change score. The scene change score is a numerical value indicating how much the currently focused frame image differs from the previous frame image. In other words, the scene change score can be said to be the degree of similarity between the currently focused frame image and the previous frame image. This scene change score can be defined by a color histogram, for example. In performing the detection as described above, the scene transition detection unit 135 refers to, for example, a scene transition detection database stored in advance in the storage unit 117 or the like, or executes a detection program or the like. Is possible.

  The scene transition detection unit 135 transmits the obtained scene change score to the moving image metadata generation unit 141 described later.

  The camera work detection unit 137 includes, for example, a CPU, a ROM, a RAM, and the like. The camera work detection unit 137 detects camera work indicating how an imaging device such as a video camera is moved when capturing a moving image for all frame images constituting the moving image. More specifically, the camera work detection unit 137 detects, for example, the direction and distance in which the imaging apparatus has moved by comparing the currently focused frame image with the previous frame image. At this time, the camera work detection unit 137 may perform a camera work detection process based on the detection result of the scene transition detection unit 135. The camera work detection unit 137 can refer to a camera work detection database or the like stored in advance in the storage unit 117 or execute a detection program or the like, for example.

  Here, the distance traveled by the imaging device can be expressed as, for example, the number of pixels of an image that moves between frames. Examples of camera work detected in this way include fix, pan, tilt, roll, zoom in / zoom out, and the like. Here, the term “fix” means that an image is taken with the imaging device fixed. Further, pan (panoramic viewing) means that the image pickup apparatus is moved in the horizontal direction and the image is picked up, and tilt means that the image pickup apparatus is shaken in the vertical direction and picked up. In addition, the roll means that an image is picked up by rotating the imaging device around a certain axis.

  The camera work detection unit 137 transmits the camera work detection result as described above to the moving image metadata generation unit 141 described later.

  The voice analysis unit 139 includes, for example, a CPU, a ROM, a RAM, and the like. The audio analyzing unit 139 analyzes the audio data transmitted from the moving image data dividing unit 131 and calculates a feature amount related to the audio data attached to the moving image. More specifically, the sound analysis unit 139 analyzes whether or not the following sound exists in the sound data associated with all the frame images constituting the moving image. In other words, the voice analysis unit 139 praises the applause as to whether the voice data is conversation, laughter, cheers like “Oo”, plosives such as “Phan”, etc. Audio data classification processing, such as whether the sound represents music or music. Such voice data classification processing can be executed, for example, by referring to a voice analysis database or the like stored in advance in the storage unit 117 or by executing a voice analysis program or the like.

  The voice analysis unit 139 may also analyze the volume of the voice data, for example, convert the volume into a numerical value and output it. In addition, the voice analysis unit 139 determines whether the voiced person is a man, a woman, or a child if the voice data is generated by a person such as a conversation or a laughter. Can also be analyzed. These analyzes can be performed, for example, by analyzing the amplitude of the corresponding waveform, the frequency of the audio data, etc. with reference to a spectrum representing the audio.

  The audio analysis unit 139 transmits the analysis result as described above to the moving image metadata generation unit 141.

  The moving image metadata generation unit 141 includes, for example, a CPU, a ROM, a RAM, and the like. The moving image metadata generation unit 141 collects the detection results and analysis results transmitted from the face detection unit 133, the scene transition detection unit 135, the camera work detection unit 137, the audio analysis unit 139, etc., and characterizes the moving image Generate video metadata for.

  FIG. 5 illustrates an example of detection results and analysis results included in the moving image metadata generated by the moving image metadata generation unit 141. As shown in FIG. 5, the moving image metadata includes metadata relating to feature amounts extracted from image data and metadata relating to feature amounts extracted from audio data. FIG. 6 illustrates the feature quantities related to “face presence / absence”, “some face”, “conversation presence / absence”, and “laughing voice presence / absence” among a plurality of feature quantities included in the moving image metadata. . In FIG. 6, the value of each feature amount is illustrated in a graph. In each graph shown in FIG. 6, the horizontal axis represents the time associated with the moving image data or the number assigned for convenience to the frame image constituting the moving image, and the vertical axis represents the feature amount. Although FIG. 6 illustrates the case where each feature amount is 1 or 0, the present invention can be similarly applied even when each feature amount is a continuous value other than 0 or 1. It is.

  As shown in FIG. 6, in the moving image metadata, the values of the feature amounts are individually described in time series for all the images constituting the moving image and the sound associated with the images. In addition, the moving image metadata does not need to be in the form of a graph as shown in FIG. 6, and each number is given in the form of a table with headings (columns) given as numbers for convenience to the frame images constituting the moving image. The feature amount may be described.

  The moving image metadata generation unit 141 transmits the moving image metadata generated as described above to the comic display conversion unit 107.

  The analysis method performed by each processing unit as described above is merely an example, and is not limited to the above example. In addition to the above-described processing unit, the moving image analysis unit 103 may further include various analysis units according to moving image metadata to be generated.

[Composition of comic display conversion unit 107]
Next, the configuration of the comic display conversion unit 107 according to the present embodiment will be described in detail with reference to FIGS. First, the overall configuration of the comic display conversion unit 107 will be described in detail with reference to FIG. FIG. 7 is a block diagram for explaining the configuration of the comic display conversion unit 107 according to the present embodiment.

  For example, as shown in FIG. 7, the comic display conversion unit 107 includes a digest score calculation unit 151, a thumbnail number determination unit 153, a camera work metadata use unit 155, an image size determination unit 157, and a frame image arrangement unit. 159 and an effect image arrangement unit 161. The comic display conversion unit 107 further includes a frame information generation unit 163 that collects various types of information generated by the above-described processing unit and generates frame information.

  The digest score calculation unit 151 includes, for example, a CPU, a ROM, a RAM, and the like. The digest score calculation unit 151 is a digest score (hereinafter abbreviated as DS), which is a value obtained by weighting and adding a plurality of feature amounts to each of a plurality of images constituting a movie based on the transmitted movie metadata. Yes.)

  For example, as shown in FIG. 8, a weighting coefficient is set in advance for the feature amount in the moving image metadata used for calculating the digest score. In FIG. 8, a weighting coefficient 0.7 is set for the feature value Face relating to the presence or absence of a face, and a weighting factor 1.2 is set for the feature value FaceCounts relating to the number of faces. A weighting factor of 1.5 is set for the feature value FaceZoom related to the presence or absence of the face zoom, and a weighting factor of 1.0 is set to the feature value FaceSmile related to the presence or absence of a smile.

  Further, not only the moving image metadata related to the image data as described above, but also the moving image metadata related to the audio data is set with the same weighting coefficient. For example, in FIG. 8, the weighting coefficient 1.0 is set for the feature amount Speech regarding the presence / absence of conversation, and the weighting factor 2.2 is set for the feature amount Laughter regarding the presence / absence of laughter. In addition, a weighting coefficient 3.6 is set for the feature amount Cheering regarding the presence or absence of cheers, and a weighting factor 1.8 is set for the feature amount Clapping regarding the presence or absence of a plosive sound. In addition, a weighting coefficient 2.7 is set for the feature quantity Applase regarding the presence / absence of praise, and a weighting coefficient 1.2 is set for the feature quantity Musical regarding the presence / absence of music.

  Using these weighting coefficients and each feature quantity included in the moving image metadata, the digest score DS is obtained by multiplying the feature quantity by the weighting coefficient related to the feature quantity, and all the feature quantities used for calculating the DS. It is calculated as the sum of.

  Here, the weighting coefficient shown in FIG. 8 is a coefficient indicating how much each feature amount contributes to the climax in the video scene. In the example of FIG. 8, the weighting coefficients for the feature amount regarding the presence / absence of the zoom of the face, the feature amount regarding the presence / absence of the laughing voice, the feature amount regarding the presence / absence of the cheer, the feature amount regarding the presence / absence of the plosive, Is set to a large value. This is based on an empirical rule that video scenes with these sounds and video scenes in which a person is zoomed in are often “interesting” or “exciting”. Further, a parameter obtained by statistical learning may be used as a weighting coefficient, or a user may input and set each weighting coefficient by himself / herself. Further, these weighting coefficients may be updated by feedback from the user.

  In this way, by weighting various video metadata, it is possible to extract scenes with various characteristics as exciting scenes, and automatically detect the exciting scenes while reflecting the characteristics of the animation Is possible.

  The digest score calculation unit 151 transmits the digest score DS calculated in this way to the thumbnail number determination unit 153 described later.

  The thumbnail number determination unit 153 includes, for example, a CPU, a ROM, a RAM, and the like. The number-of-thumbnail determination unit 153 extracts thumbnail images to be arranged in a frame from a plurality of images constituting the moving image data based on the transmitted digest score, and determines the number of thumbnail images. Hereinafter, a method for extracting thumbnail images will be described in detail with reference to FIG.

  As described above, the digest score is calculated by the digest score calculation unit 151 for all the frame images constituting the moving image. Therefore, the thumbnail number determination unit 153 calculates the distribution of the digest score and the median (Median) of the digest score in the moving image data of interest for all frame images constituting the moving image. Thereafter, as shown in FIG. 9, for example, the thumbnail number determination unit 153 extracts a frame image having a digest score equal to or greater than the calculated median value as a thumbnail image candidate. By extracting thumbnail image candidates in this manner, it is possible to extract a rising scene while cutting a frame image that has a low digest score and is not likely to be rising.

  In the following, the thumbnail number determining unit 153 will be described in detail step by step which frame is used as a thumbnail image among frame images having a digest score greater than or equal to the median value. At this time, if all frame images having a digest score equal to or higher than the median value are simply adopted, a large number of frames with similar images are arranged, resulting in a problem that the arrangement of the frames becomes monotonous. Also, simply adopting the digest scores in the descending order only arranges large frames of the same size, and does not provide dynamic frame allocation. Therefore, in order to avoid such a situation as much as possible, a thumbnail image is selected as follows.

  First, the thumbnail number determination unit 153 determines whether or not the scene change score included in the moving image metadata is equal to or greater than a predetermined threshold. When the scene change score is large, it is possible to detect a scene change as if the scene has changed to another scene. Here, when the scene change score is larger than the threshold value, the thumbnail number determination unit 153 considers that the image is blurred due to a large movement of the camera, and removes it without adopting it as a thumbnail image. In particular, in private video content, when the camera moves suddenly, the scene change score increases rapidly. Since the image at this time is often blurred and does not appear well, it is not used for the thumbnail image.

  Next, the thumbnail number determination unit 153 calculates the distribution and median of the digest scores. As described above, the median value is used to determine whether to adopt a frame image as a thumbnail image.

  Then, the thumbnail number determination unit 153 sets the threshold value T = the median value if the number of frame images of the thumbnail image candidates is equal to or larger than the predetermined setting value m, and sets the threshold value T = the minimum digest score if the number is less than m. Here, the set value m can be set to about 100, for example. This is a process for preventing the number of display frames from becoming extremely small in the case of a video whose playback time is too short.

  Next, the thumbnail number determination unit 153 does not employ a candidate whose digest score is less than T or the same value is continuous. This is a process for preventing a situation in which a frame with a high digest score is adopted as a climax scene and frames with the same climax are continuously adopted in the same scene. If frames with the same excitement level continue, similar frames of the same size will be lined up continuously, giving a monotonous impression, so such frames are not adopted.

  Thereafter, the thumbnail number determination unit 153 calculates the total number of adopted frame images, and performs a thinning process when the total number is equal to or greater than a predetermined threshold. The threshold value at this time is determined according to the number of pages to be displayed. Of the adopted frames, only the part that fits in the page is adopted for the thumbnail image. For example, the thumbnail number determination unit 153 may sort the frame images in the order of the digest score and thin them out at regular intervals. Thus, it is possible to perform the thinning process while maintaining the digest score distribution.

  Finally, the thumbnail number determination unit 153 performs a thinning process when frames having a small digest score continue. When comics are displayed on a moving image, if similar frames appear in succession, the frame division becomes tidy and a very monotonous impression is received. Since frames with a small size also have a small digest score, it is possible to divide frames more like a comic by preferentially thinning out such portions. The frame number remaining in this manner is used by the thumbnail number determination unit 153 as a frame (that is, a thumbnail image) in comic display. Hereinafter, this thumbnail image is also referred to as a representative frame (or representative frame image).

  In addition, when the thumbnail number determination unit 153 selects a frame image to be used as a thumbnail image as described above, an image used for animation of the thumbnail image in the frame (hereinafter also referred to as an animation frame image). Select. More specifically, as shown in FIG. 10, for example, the thumbnail number determination unit 153 uses, as animation frame images, frame images for a predetermined time that are temporally continuous to the extracted thumbnail images (representative frame images). Extract. Further, when extracting the animation frame image, attention is not paid to the size of the digest score. The “predetermined time” used for the extraction can be set to an arbitrary value. For example, the “predetermined time” is set to a value such that the animation time is about 1 second together with the representative frame. For example, when the frame rate of the moving image data is 30 FPS (Frames Per Second), the number of frame images for one second is about 30 frames.

  The number-of-thumbnail determination unit 153 uses the information regarding the thumbnail image (representative frame) and the number of thumbnail images extracted as described above and the information regarding the animation frame image as the number-of-images information. Here, the information on the number of thumbnail images and the number of thumbnail images includes, for example, information such as how many frame images from the top of the moving image frame are selected as thumbnail images, and how many thumbnail images are in total. Have been described. The thumbnail number determination unit 153 transmits this number information to an image size determination unit 157 and a frame information generation unit 163 described later.

  Further, in the above description, the case where the thumbnail number determination unit 153 selects the animation frame image so as to be a preset time has been described. However, it is also possible to dynamically change the number of animation frame images to be selected by using the feature amount relating to camera work included in the moving image metadata. This process is executed, for example, in the camera work metadata utilization unit 155 described below.

  The camera work metadata usage unit 155 is constituted by, for example, a CPU, a ROM, a RAM, and the like. The camera work metadata utilization unit 155 determines the number of images extracted as an animation frame image by the thumbnail number determination unit 153 based on moving image metadata related to camera work.

  First, prior to the description of the camerawork metadata usage unit 155, a brief description will be given of what phenomenon occurs when an animation frame image is selected without considering camerawork.

  For example, as shown in FIG. 1, an animation (thumbnail animation) that is started by overlaying a position selection object such as a mouse pointer on an image is useful for grasping the content of the entire video, and is used as an index or summary of the video content. Is possible. However, if you simply play the animation without taking camerawork into consideration, if the camera moves greatly and the screen changes significantly, the screen (that is, the frame) becomes small and difficult to see, and the animation itself looks unnatural. There is a possibility that. In addition, when the image changes abruptly (for example, when a person on the screen suddenly changes), there is a possibility that the animation story itself as a summary of the moving image content fails. Also, simply playing the animation can make the animation appear monotonous. For this reason, it is necessary to devise a method in which a portion where the camera is moving greatly is cut from the animation and playback is switched when the scene is switched. Therefore, in the present embodiment, by providing a camera work metadata utilization unit 155 as described below, it is possible to select an animation frame image considering camera work.

  The number information is transmitted from the thumbnail number determination unit 153 to the camera work metadata utilization unit 155. The camera work metadata utilization unit 155 calculates an integrated value of the camera work based on the feature amount related to the camera work described in the moving image metadata related to the animation frame image described in the number information. Here, there are two types of integral values of camera work, one obtained by integrating the absolute value of camera work and one obtained by integrating the camera work value itself (without considering the absolute value). However, in the present embodiment, any integral value can be used. Note that the integral value of the absolute value of camera work means the sum of distances moved by the camera. When using an integral value that does not take into account the absolute value, when the camera returns to the original position / orientation, the camera work integral value becomes 0, and the camera work amount is evaluated only with the relative value from the initial position. Can not. However, it has a feature that it is strong against the effects of camera shake and noise. In addition, when using the integral value of the absolute value of camera work, it may be affected by camera shake or noise, but the integral value will always increase no matter what direction the camera moves. The feature is that the integrated value can be easily evaluated from the total moving distance.

  The camera work metadata utilization unit 155 determines the number of animation frame images using the calculated integral value of camera work and a preset threshold value TH. Hereinafter, a method for determining the number of animation frame images by the camera work metadata using unit 155 will be described in detail with reference to FIG. FIG. 11 illustrates a method of determining the number of animation frame images using the integral value of the absolute value of camera work. When using an integrated value of camera work that does not consider the absolute value, the integrated value may be a negative value. Therefore, the threshold value TH is set so that the absolute value is equal to both positive and negative values (that is, the threshold value + TH). And -TH) can be determined.

  For example, as shown in FIG. 11, the camera work metadata utilization unit 155 detects a frame i in which the integral value of the absolute value of the camera work is equal to or greater than a predetermined threshold value TH, and sets a section 0 to i to a predetermined set value n. Divide into n equal parts. Thereafter, the camera work metadata utilization unit 155 selects an animation frame image using the value of i / n. For example, in FIG. 11, if the origin 0 is the position of the representative frame, (i / n), 2 × (i / n),..., N × (i / n) = i is closest to the value. A frame image having a frame number is selected as an animation frame image.

  Here, the set value n is a value corresponding to the number of animation frame images determined according to the integrated value of camera work. For example, n = 2 to 12, and n is set according to the value of the integrated value. = 2, 4, 8, 12, etc. More specifically, when the integrated value of camera work is large, the value of n is set to a small value, and when the integrated value of camera work is small, the value of n is set to a large value. The set value n can be determined by an arbitrary method using the integrated value of camera work. For example, n can be determined based on the following criteria. In the following notation, FPS represents the frame rate of moving image data.

(1) When integral value of camera work> (0.1 × FPS × 2): n = 2
(2) When (0.1 × FPS × 2) ≧ integral value of camera work> (0.1 × FPS × 1): n = 4
(3) When (0.1 × FPS × 1) ≧ camera work integral value> (0.1 × FPS × 0.5): n = 8
(4) (0.1 × FPS × 0.5) ≧ camera work integral value: n = 12

  In this way, by determining the set value n corresponding to the number of animation frame images, the animation frame image can be dynamically determined according to the size of the camera work. That is, as shown in FIG. 11A, when the camera work is relatively small, the integrated value of the camera work does not readily reach the threshold value TH. Therefore, the number of frames tends to be close to 30 when the integrated value of the camera work reaches the threshold value TH. Further, when the integral value of camera work is relatively small, the set value n is set to be large, so that many images are selected from relatively wide sections 0 to i. Since an image with a small camera work does not change much, the animation reproduction can be performed more smoothly by increasing the number of animation frame images.

  When the camera work is relatively large as shown in FIG. 11B, the integrated value of the camera work can easily reach the threshold value TH. Therefore, when the integrated value of camera work reaches the threshold value TH, the value i tends to be small. When the integral value of camera work is relatively large, the set value n is set to be small, so that there are few images selected from the relatively narrow sections 0 to i. Since an image with a large camera work changes greatly, it is possible to select adjacent frame images (that is, a high similarity) by reducing the number n of animations.

  Also, the camera work metadata utilization unit 155 sets the playback speed of the animation composed of the thumbnail image and the animation frame image according to the magnitude of the integrated value of the camera work. More specifically, when the number of selected animation frame images is large, the camera work metadata utilization unit 155 sets the reproduction speed to a large value and reproduces the animation quickly. Further, when the number of selected animation frame images is small, the camera work metadata utilization unit 155 sets the reproduction speed to a small value and reproduces the animation slowly. Specifically, the animation playback speed is set as follows, for example.

(1) When the number of animations (n) is 2 or less: Playback speed = 1 (FPS)
(2) When the number of animations (n) is more than 2 and 4 or less: Playback speed = 2 (FPS)
(3) When the number of animations (n) is greater than 4 and less than or equal to 8: Playback speed = 8 (FPS)
(4) When the number of animations (n) exceeds eight: playback speed = 24 (FPS)

  The camerawork metadata utilization unit 155 can employ a similar frame image from among a plurality of animation frame image candidates by performing the processing as described above, and can prevent natural scene switching. Can produce simple animations.

  Next, returning to FIG. 7, the image size determination unit 157 will be described. The image size determination unit 157 is constituted by, for example, a CPU, a ROM, a RAM, and the like. The image size determination unit 157 determines the size of the thumbnail image arranged in the predetermined display area based on the digest score.

  More specifically, the image size determination unit 157 sorts all the images constituting the moving image in the order of the digest score as shown in FIG. 12, for example. In addition, the image size determination unit 157 determines that the upper number% (for example, upper 3%) of the digest score is an outlier, and does not use it for determining the image size. Thereafter, the image size determination unit 157 divides the digest score that is greater than or equal to the median value and less than the outlier value into m according to the digest score, and generates m stages of image sizes. Here, the value of the set value m can be set to an arbitrary value according to the size of the display area to be displayed on the screen.

  FIG. 12 illustrates the case where the image size is set in two stages. Here, the image size B in FIG. 12 is smaller than the image size A. For the thumbnail image described in the number information transmitted from the thumbnail number determination unit 153, the image size determination unit 157 has the digest score of the image of the area of the image size A shown in FIG. Determine which of the areas it belongs to. The image size determination unit 157 determines the image size for all the thumbnail images described in the number information in accordance with the determination result. By determining the image size in this way, an image having a relatively large digest score has a large image size, and an image having a relatively small digest score has a small image size. Thereby, in the information processing apparatus 10 according to the present embodiment, the image size can be automatically determined according to the degree of excitement.

  In addition, when there is moving image metadata indicating that a face exists in the image for the selected thumbnail image, the image size determination unit 157 determines the face that is the area in which the face is displayed. Set the image size so that the area is enlarged. When there are a plurality of face areas in the image, the image size determination unit 157 enlarges the size of the image data so as to include all the face areas.

  For example, as illustrated in FIG. 13, when the frame image a having only one face area is selected as the thumbnail image, the image size determination unit 157 displays the comic area so that the face area is enlarged. Determine the image size at. When the frame image b having two face areas is selected as the thumbnail image, the image size determination unit 157 enlarges the image so as to include the two face areas.

  The image size determination unit 157 uses the information related to the thumbnail image size (and thus the size of the animation frame image) set in this way as frame image information, and transmits the frame image information to the frame image arrangement unit 159 and the frame information generation unit 163 described later. To do.

  Next, returning to FIG. 7, the frame image arrangement unit 159 will be described. The frame image arrangement unit 159 includes, for example, a CPU, a ROM, a RAM, and the like. The frame image arrangement unit 159 arranges the thumbnail images whose image sizes have been determined on the frames in a predetermined order. Hereinafter, the frame arrangement process performed by the frame image arrangement unit 159 will be described in detail with reference to FIGS. 14 to 17.

  As described above, the frame image arrangement unit 159 dynamically arranges thumbnail images whose image sizes have been determined on the display area. At this time, the frame image arrangement unit 159 implements frame division that reflects the content of the video by arranging thumbnail images in chronological order of the video. In a general comic, a plurality of lines basically exist in one page, and each line includes a plurality of frames. Comic readers read the frames contained in one line to the left and right, read to the very end, go to the next line, and understand the contents of the comic. In this method, as in an actual comic, thumbnail images are arranged on the left and right for each line, and the lines are arranged from top to bottom.

  When realizing the same page turning direction as a right-bound book, the frame image arrangement unit 159 arranges thumbnail images from the right side to the left side when arranging thumbnail images in a certain row. I will do it. Conversely, when realizing the same page turning direction as a left-bound book, the frame image arrangement unit 159 displays thumbnail images from the left side to the right side when arranging thumbnail images in a certain row. Place it.

  FIG. 14 is an explanatory view showing an arrangement example of thumbnail images. 14, assuming the same page turning direction as a right-bound book, 16 thumbnail images (indicated as Pic. 1 to Pic. 16 in the drawing) are arranged in the display area 301. Yes. In FIG. 14, Pic. 1-Pic. The numbering of 16 is assumed to be along the time series of moving images.

  At this time, the frame image arrangement unit 159 sets an area having the same height as the minimum image size of the thumbnail image as the arrangement unit frame 501. This arrangement unit frame 501 is a line for arranging thumbnail images. The frame image arrangement unit 159 can realize the image arrangement as shown in FIG. 14 by arranging the thumbnail images while enlarging the arrangement unit frame 501. Further, if the thumbnail images are simply arranged, a gap may be generated between the images. Therefore, in such a case, the thumbnail image is enlarged and displayed, and a comic-like frame division is realized.

  Hereinafter, a specific description will be given with reference to FIGS. First, the frame image arrangement unit 159 sets an arrangement unit frame 501 (hereinafter also referred to as a unit frame 501) based on the minimum size (minimum height) of an image to be arranged. The frame image arrangement unit 159 arranges thumbnail images using this unit frame 501.

  As shown in FIG. 15A, the frame image arrangement unit 159 confirms whether the thumbnail image is smaller than the unit frame 501, and the image Pic. 1 is placed. Next, the frame image arrangement unit 159 displays the next thumbnail image Pic. 2 is arranged, the height of the unit frame 501 and the thumbnail image Pic. Compare with the height of 2 to see if the image can be placed. In the example shown in FIG. 15A, the thumbnail image Pic. 1 and Pic. 2 have the same height, the frame image arrangement unit 159 displays the thumbnail image Pic. 2, Pic. 1 next to the right.

  Next, as shown in FIG. 15B, the thumbnail image Pic. Having a height larger than the unit frame 501 is displayed. The case where 3 is arrange | positioned is demonstrated. The frame image arrangement unit 159 displays the height of the unit frame 501 and the thumbnail image Pic. Compare the height of 3. In this case, since the height of the thumbnail image is higher than the height of the unit frame 501, the frame image placement unit 159 increases the height of the unit frame 501 as shown in FIG. Rearrange thumbnail images from the beginning of the line.

  When the height of the thumbnail image is smaller than the unit frame 501, the frame image arrangement unit 159 checks whether the thumbnail image can be arranged. If a thumbnail image can be arranged, the image is arranged at that position, and the arrangement position is moved to a position below the arranged image to check whether the next thumbnail image can be arranged. As a result, as shown in FIG. 1 to Pic. 2 is arranged, Pic. 1 and Pic. 2 to the right of Pic. 3 will be arranged.

  As shown in FIG. 16A, the thumbnail image Pic. In the case where 6 is arranged, when the frame 501 protrudes downward from the unit frame 501, the frame image arrangement unit 159 changes the size of the unit frame 501, and arranges the thumbnail images from the beginning of the changed line. At this time, Pic. 5 next to the image Pic. 6 has a large image size. If it is arranged below 5, it will protrude from the unit frame 501. Therefore, the frame image arrangement unit 159 displays the thumbnail image Pic. 6 to Pic. 5 next to the right. At this time, as shown in FIG. 16B, when a gap is generated between the previous image and the current image, the frame image arrangement unit 159 displays the previous thumbnail image. Fill in the gaps by zooming in. That is, in the case as shown in FIG. 16B, as shown in FIG. By enlarging 5, the gap is filled.

  Also, as shown in FIG. 17A, when an image is arranged and when it protrudes rightward beyond the right end of the display area 301, the frame image arrangement unit 159 fills all the current lines. As a thing, move on to the next line. In the next row, the frame image placement unit 159 changes the size of the unit frame 501 by the method as described above, and as shown in FIG. 4 is arranged. Also in this case, as in the case of FIG. 16, when a gap is generated between the previous image and the current image, the gap is filled by enlarging the previous image. In the case shown in FIG. 17, the frame image placement unit 159 displays the Pic. 3 is enlarged so that no gap is generated.

  The frame image arrangement unit 159 arranges the thumbnail images according to the procedure as described above, and when all the thumbnail images are arranged, the arrangement process ends.

  When the arrangement of the thumbnail images is completed, the frame image arrangement unit 159 displays frame arrangement information, which is information indicating which thumbnail image is arranged in what size and where, the effect image arrangement unit 161 described later and the frame information. The data is transmitted to the generation unit 163. In addition, when the thumbnail image enlargement process is performed to fill the gap generated in the image arrangement, the frame image arrangement unit 159 notifies the image size determination unit 157 and the frame information generation unit 163 to that effect. The frame image information may be requested to be corrected.

  The thumbnail image size determination by the image size determination unit 157 and the thumbnail image arrangement by the frame image arrangement unit 159 may also be performed in consideration of camera work. For example, as shown in FIG. 1-Pic. Consider a case where there are three thumbnail images 3 and the camerawork direction is horizontal for each image (that is, the camerawork is pan). In this case, the image size determination unit 157 may determine the size and shape of each thumbnail image according to the camera work direction and the camera work speed. Further, the frame image arrangement unit 159 may arrange each thumbnail image along the direction of the camera work.

  In the example shown in FIG. 18, since the camera work is pan, the frame image arrangement unit 159 arranges the thumbnail images in the horizontal direction as shown in the lower part of FIG. The image size determination unit 157 processes each thumbnail image so as to be vertically long, and determines the width of the thumbnail image according to the camera work speed. That is, when the camera work speed is slow, the thumbnail image width is set wide, and when the camera work speed is fast, the thumbnail image width is set narrow.

  Also, as shown in FIG. 19, when the camera work is tilted, the frame image arrangement unit 159 may arrange the thumbnail images in the vertical direction. Further, the image size determination unit 157 may process each thumbnail image so as to be horizontally long and determine the height of the thumbnail image according to the camera work speed.

  As described above, by changing the image size, the shape of the image, and the arrangement direction of the image in accordance with the direction and speed of the camera work, it becomes possible to display a more comical thumbnail image.

  Returning to FIG. 7 again, the effect image arrangement unit 161 will be described. The effect image arrangement unit 161 includes, for example, a CPU, a ROM, a RAM, and the like. The effect image arrangement unit 161 arranges at least one of a balloon and a screen effect based on the moving image metadata with respect to the frame on which the thumbnail image is arranged. For example, the effect image arrangement unit 161 arranges at least one of a balloon and a screen effect using moving image metadata regarding audio data included in the moving image data.

  FIG. 20 shows the correspondence between audio metadata and speech balloons. As shown in FIG. 20, the effect image arrangement unit 161 arranges a balloon for a frame on which a thumbnail image having corresponding audio metadata is arranged. Thereby, in the information processing apparatus 10 according to the present embodiment, a more comic-like expression is possible. Note that the balloon may be realized by vector data in the SWF format, assuming that it is displayed on the Web. Thereby, it is possible to achieve both animation processing such as capacity saving and enlargement / reduction.

  If the thumbnail image includes moving image metadata indicating that a face is present, the effect image arrangement unit 161 arranges the balloon so that the balloon does not cover the face area. Thereby, it is possible to create a comic display image that is more like a comic. By doing so, it is possible to express a predetermined speech balloon as if it represents the person's dialogue. Specifically, the effect image placement unit 161 first compares the aspect ratios of the frames, and if the frames are vertically long, the balloons are vertically arranged, and if the frames are horizontally long, the effect images are arranged horizontally. At this time, the effect image arrangement unit 161 compares the center coordinates of the face area with the center coordinates of the frame, and arranges the balloon in the direction opposite to the position where the face exists. For example, when the frame is vertically long and the center coordinates of the face area are below the center coordinates of the frame, the effect image placement unit 161 places the balloon above the frame.

  Furthermore, the effect image arrangement unit 161 may change the size of the balloon depending on the duration of the audio metadata. For example, if the duration is set to 1 second or less, 3 seconds or less, more than 3 levels, and the longer the sound continues, the more important the sound metadata is, the higher the importance of the effect image. The arrangement unit 161 can express a more important scene with emphasis.

  In addition, the effect image arrangement unit 161 can change the color of the balloon according to the type of the audio metadata. For example, the “Speech” metadata is added to the conversational voice scene, but the “Speech” metadata is further divided into three further categories “male”, “female”, and “children”. Can be added. These are metadata provided by extracting the conversational voices of men, women, and children, respectively. The effect image arrangement unit 161 can change the color of the balloon by setting in advance colors such as blue, red, and yellow for each metadata. This makes it possible to express speaker differences in comics by color.

  Further, the effect image arrangement unit 161 can arrange screen effects such as concentrated lines and effect lines based on the camerawork feature amount included in the moving image metadata. Here, the effect line is a plurality of parallel lines arranged along the horizontal direction, and the concentrated line is a plurality of lines arranged radially from the center with respect to a certain region. The effect image arrangement unit 161 may arrange concentrated lines or effect lines according to the type and orientation of the camera work, and may increase or decrease the density and number of the concentrated lines or effect lines according to the speed of the camera work.

  For example, as illustrated in FIG. 21, when the thumbnail image includes moving image metadata indicating that a face area exists and moving image metadata indicating that camera work called zoom-in has been performed, A plurality of concentrated lines 309 are arranged around the region. In this way, the thumbnail image with the face up as shown in FIG. 21 is arranged after the thumbnail image in which the concentrated lines are arranged, so that the comic display screen becomes more comic. In addition, the number of concentrated lines 309 is increased when the zoom-in speed is high, and is decreased when the zoom-in speed is low.

  In addition, in the thumbnail image in which the moving image metadata indicating that the camera work is pan is present, the effect image arranging unit 161 arranges the effect line along the camera work direction, for example, as illustrated in FIG. In the case of the effect line, as in the case of the concentrated line, many are arranged when the camera work speed is fast, and few are arranged when the camera work speed is slow.

  In this way, by arranging images related to screen effects such as concentration lines and effect lines according to camera work, the comic display screen becomes more comical, and the user of the information processing apparatus 10 visually recognizes the content of the moving image. It becomes possible to grasp while having fun.

  Returning to FIG. 7 again, the frame information generation unit 163 will be described. The frame information generation unit 163 includes, for example, a CPU, a ROM, a RAM, and the like. The frame information generation unit 163 generates frame information regarding each frame on which thumbnail images are arranged. The frame information is information including various types of information generated by each processing unit included in the comic display conversion unit 107, such as the number information, the frame image information, the frame layout information, and the effect layout information. The information processing apparatus 10 is necessary to generate a comic display screen such as where and in what size the thumbnail image should be arranged and what kind of effect should be arranged by referring to the frame information. Can be obtained.

  Heretofore, an example of the function of the information processing apparatus 10 according to the present embodiment has been shown. Each component described above may be configured using a general-purpose member or circuit, or may be configured by hardware specialized for the function of each component. In addition, the CPU or the like may perform all functions of each component. Therefore, it is possible to appropriately change the configuration to be used according to the technical level at the time of carrying out the present embodiment.

  It should be noted that a computer program for realizing each function of the information processing apparatus according to the present embodiment as described above can be produced and installed in a personal computer or the like. In addition, a computer-readable recording medium storing such a computer program can be provided. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like. Further, the above computer program may be distributed via a network, for example, without using a recording medium.

<About information processing method>
Next, an information processing method according to the present embodiment (more specifically, a method for displaying comics for moving image data) will be described in detail with reference to FIGS. 23 to 29 are flowcharts for explaining the information processing method according to the present embodiment.

[Overall flow of information processing method]
First, the overall flow of the information processing method according to the present embodiment will be described in detail with reference to FIG.

  The moving image data acquisition unit 101 of the information processing apparatus 10 according to the present embodiment first acquires moving image data corresponding to a moving image specified by the user in response to a user operation (step S101). The moving image data acquisition unit 101 may acquire moving image data from the storage unit 117 or the like in the own device or may acquire from various recording media attached to the own device. The moving image data acquisition unit 101 may acquire moving image data from another device connected wirelessly or by wire, or may acquire it from another device connected via a communication network such as the Internet. Good. The moving image data acquisition unit 101 transmits the acquired moving image data to the moving image analysis unit 103 and the audio extraction unit 105.

  The audio extraction unit 105 extracts audio data from the moving image data transmitted from the moving image data acquisition unit 101 (step S103), and transmits the acquired audio data to the comic display data generation unit 111.

  On the other hand, the moving image analysis unit 103 analyzes the moving image data transmitted from the moving image data acquisition unit 101, and generates moving image metadata that is metadata relating to the feature amount characterizing the moving image corresponding to the transmitted moving image data (step). S105). The moving image analysis unit 103 transmits the generated moving image metadata to the comic display conversion unit 107.

  Next, the digest score calculation unit 151 of the comic display conversion unit 107 calculates a digest score for all images (frame images) constituting the moving image based on the transmitted moving image metadata. The digest score calculation unit 151 transmits the calculated digest score to the thumbnail number determination unit 153.

  Subsequently, the thumbnail number determination unit 153 selects a representative frame image to be used as a thumbnail image using the transmitted digest score and moving image metadata (step S107). Further, when the selection of the representative frame image is completed, the thumbnail number determination unit 153 selects an animation frame image used for the thumbnail animation. The number-of-thumbnail determination unit 153 generates number information including information for specifying the selected representative frame image, information about the number of representative frame images, information about an animation frame image related to the representative frame image, and the like. The thumbnail number determination unit 153 transmits the generated number information to the image size determination unit 157 and the frame information generation unit 163.

  It should be noted that the analysis result by the camera work metadata utilization unit 155 can be reflected when determining the playback speed of the animation frame image or thumbnail animation.

  Next, the image size determination unit 157 determines an image size for displaying the selected representative frame image in comics based on the transmitted number information, moving image metadata, digest score, and the like (step S109). The image size determination unit 157 generates frame image information including information related to the image size of the representative frame image, and transmits the frame image information to the frame image arrangement unit 159 and the frame information generation unit 163.

  Subsequently, the frame image arrangement unit 159 automatically arranges the representative frame image in the display area using the transmitted frame image information, moving image metadata, and the like, and determines the arrangement of the representative frame image (step). S111). When the arrangement of the representative frame image is determined, the frame image arrangement unit 159 generates frame arrangement information including information such as which representative frame image is arranged at which position, and an effect image arrangement unit 161 and a frame information generation unit 163. And transmit.

  Next, the effect image arrangement unit 161 arranges an effect image such as a balloon based on the transmitted frame arrangement information, moving image metadata, and the like (step S113). When the effect image placement unit 161 finishes placing the effect image, the effect image placement unit 161 generates effect placement information including information about which effect is placed where, and transmits the effect placement information to the frame information generation unit 163.

  It should be noted that when determining the image size, the frame image arrangement, and the effect image arrangement, it is possible to use the feature amount relating to the camera work included in the moving image metadata.

  Subsequently, the frame information generation unit 163 generates frame information based on the number information, frame image information, frame arrangement information, effect arrangement information, and the like transmitted from each processing unit (step S115). The frame information generation unit 163 transmits the generated frame information to the frame image data generation unit 109 and the comic display data generation unit 111.

  Next, the frame image data generation unit 109 performs a process of adjusting the size and shape of the frame image used for comic display based on the image data and the frame information, and generates the frame image data used for comic display. Generate (step S117). The frame image data generation unit 109 transmits the generated frame image data to the comic display data generation unit 111.

  The comic display data generation unit 111 generates audio data used for comic display based on the audio data transmitted from the audio extraction unit 105 and the frame information transmitted from the comic display conversion unit 107. Further, the comic display data generation unit 111 generates comic display data, which is a set of data necessary for realizing comic display, using the frame information, audio data, frame image data, and effect data. (Step S119).

  By executing the generated comic display data, the information processing apparatus 10 can display a comic display that summarizes the content of the moving image on the display screen.

[How to select representative frame images]
Next, the flow of the representative frame image selection method will be described in detail with reference to FIG. FIG. 24 is a flowchart for explaining a representative frame image selection method according to this embodiment.

  First, the thumbnail number determination unit 153 determines whether the scene change score (SCS) included in the moving image metadata is equal to or greater than a predetermined threshold (step S201). If the scene change score is greater than the threshold, the thumbnail number determination unit 153 considers that the image is blurred due to a large movement of the camera, and does not select it as a representative frame image (step S203).

  Next, the thumbnail number determination unit 153 calculates the distribution and median of the digest score (DS) (step S205). As described above, the median value is used to determine whether to select a frame image as a representative frame image.

  Then, the thumbnail number determination unit 153 determines whether or not the number of valid frames of the representative frame image candidate is equal to or greater than a predetermined setting value N (step S207). If the number of valid frames is N or more, the threshold value T = median value, and if it is less than N, the threshold value T = the minimum digest score value.

  Next, the thumbnail number determination unit 153 does not select a candidate whose digest score is less than T or the same value continues as a representative frame image. Further, candidates whose digest score is equal to or greater than T and whose same value is not continuous are selected from the representative frame image candidates.

  Subsequently, the thumbnail number determination unit 153 determines whether or not the above-described processing has been performed for all frames (step S211). If all the frames have not been processed, the process returns to step S207 to perform the process.

  If the processing has been completed for all frames, the thumbnail number determination unit 153 calculates the total number of adopted frame images and determines whether this total is equal to or greater than a predetermined threshold (step S213). If the number of adopted frame images is equal to or greater than a predetermined threshold, the thumbnail number determination unit 153 performs a thinning process at regular intervals so as to leave a digest score distribution (step S215).

  Next, the thumbnail number determination unit 153 performs a thinning process when frames having a small digest score continue (step S217). The thumbnail number determination unit 153 selects the frame image remaining in this way as a frame (that is, a representative frame image) in comic display (step S219).

[How to select an animation frame image according to camera work]
Next, a method for selecting an animation frame image corresponding to camera work will be described in detail with reference to FIG. FIG. 25 is a flowchart for explaining a method of selecting an animation frame image according to camera work.

  First, the camera work metadata utilization unit 155 determines whether or not the currently focused frame image exceeds the final frame image (step S301). If the focused frame image exceeds the final frame image, the camera work metadata utilization unit 155 ends the process. When the final frame image is not exceeded, the camera work metadata utilization unit 155 performs the following processing.

  Next, the camera work metadata utilization unit 155 determines whether or not the currently focused frame image is a representative frame image to be displayed (step S303). If the currently focused frame is not the representative frame to be displayed, the camera work metadata utilization unit 155 performs processing on the next representative frame (step S305). When the currently focused frame is a representative frame to be displayed, the camera work metadata using unit 155 refers to the feature amount related to the camera work included in the moving image metadata. More specifically, the camera work metadata utilization unit 155 calculates the absolute value of the camera work regarding pan, tilt, roll, and zoom (step S307).

  Subsequently, the camera work metadata utilization unit 155 performs an integration process in which absolute values of camera work are sequentially added to the animation frame image selected in relation to the representative frame image. The camerawork metadata utilization unit 155 determines whether or not the integration value has become equal to or greater than a predetermined threshold every time addition of one animation frame image is completed (step S309). If the integral value is equal to or greater than the predetermined threshold value, the camera work metadata utilization unit 155 performs the process of step S313 described later. If the integral value is less than the predetermined threshold, the camera work metadata utilization unit 155 determines whether or not the feature amount related to the camera work has been added for a predetermined time (for example, one second) or more. Judgment is made (step S311). If more than a predetermined number of frames are added, the camera work metadata using unit 155 stores the number of frames currently focused on (step S313). In addition, when the frame of a predetermined time or more is not added, the camera work metadata utilization unit 155 pays attention to the next animation frame image (step S317), and returns to step S307 to perform processing.

  Next, the camera work metadata utilization unit 155 determines the number of animation frame images according to the integrated value of the camera work (step S315). Subsequently, the camera work metadata using unit 155 determines an animation frame image to be used for the thumbnail animation based on the determined number of animation frame images and the stored frame number.

  By performing the processing as described above, the information processing method according to the present embodiment can perform natural thumbnail animation according to camera work.

[Dynamic layout method of representative frame images]
Next, a method for dynamically arranging representative frame images according to the present embodiment will be described in detail with reference to FIGS. 26A and 26B. FIG. 26A and FIG. 26B are flowcharts for explaining a method for dynamically arranging representative frame images according to the present embodiment.

  First, the frame image placement unit 159 creates a placement unit frame used when placing the representative frame image (step S401). At this time, the frame image arrangement unit 159 makes the arrangement unit frame the same height as the minimum image size determined by the image size determination unit 157.

  Next, the frame image arrangement unit 159 determines whether or not the representative frame image to be arranged is smaller than the arrangement unit frame (step S403). When the height of the representative frame image to be arranged is larger than the height of the arrangement unit frame, the frame image arrangement unit 159 performs processing in steps S405 to S409 described later. Further, when the height of the representative frame image to be arranged is equal to or less than the height of the arrangement unit frame, the frame image arrangement unit 159 performs processing after step S411 described later.

  When the height of the representative frame image to be arranged is larger than the height of the arrangement unit frame, the frame image arrangement unit 159 expands the height of the arrangement unit frame to the height of the representative frame image to be arranged (step S405). Next, the frame image arrangement unit 159 returns to the beginning of the line (that is, the beginning of the arrangement unit frame in which the height has been enlarged) (step S407), and starts image rearrangement from the beginning of the line (step S409). ).

  Further, when the height of the representative frame image to be arranged is equal to or less than the arrangement unit frame, the frame image arrangement unit 159 determines whether or not the representative frame image to be arranged can be arranged (step S411).

  When the representative frame image to be arranged can be arranged, the frame image arrangement unit 159 arranges the representative frame image at the corresponding place (step S413). Subsequently, the frame image placement unit 159 moves the frame on which the image is to be placed to the next frame (step S415). Here, when the height of the representative frame image arranged in step S413 is less than the height of the arrangement unit frame, the frame image arrangement unit 159 displays the frame located below the image arranged in step S413. Is the next frame in step S415. Further, when the height of the representative frame image arranged in step S413 is equal to the height of the arrangement unit frame, the frame image arrangement unit 159 selects a frame located on the right side or the left side of the image arranged in step S413. Assume that the next frame in S415.

  Next, the frame image arrangement unit 159 determines whether or not all the representative frame images have been arranged (step S417). When all the representative frame images have been arranged, the frame image arrangement unit 159 ends the frame arrangement process. When all the representative frame images have not been arranged, the frame image arrangement unit 159 performs the processing from step S403 on for the next representative frame image.

  In step S411, when the representative frame image to be arranged protrudes in the vertical direction (height direction) of the arrangement unit frame, the frame image arrangement unit 159 advances the image immediately before the image to be currently arranged. The image arranged on the frame is enlarged (step S419). Next, the frame image arrangement unit 159 changes the focused frame to a horizontally adjacent frame (step S421). Thereafter, the frame image arrangement unit 159 returns to the beginning of the line (step S423), and starts rearranging the images from the beginning of the line (step S425).

  In step S411, when the representative frame image to be arranged protrudes in the horizontal direction (width direction) of the arrangement unit frame, the frame image arrangement unit 159 displays the frame immediately preceding the image to be arranged. The image arranged in (2) is enlarged (step S427). Next, the frame image arrangement unit 159 moves to the next line (step S429), and resets the size of the arrangement unit frame to the original size (step S431). Thereafter, the frame image arrangement unit 159 returns to step S401 to resume the arrangement process of the representative frame image.

  By performing the representative frame image arrangement process according to the procedure described above, the information processing apparatus 10 according to the present embodiment can dynamically arrange the representative frame images in time series.

[Regarding how to arrange representative frame images in consideration of camera work]
Next, with reference to FIG. 27, a representative frame image arrangement method considering camera work will be described in detail. FIG. 27 is a flowchart for explaining a method for arranging representative frame images in consideration of camera work.

  Hereinafter, a case will be described in which the arrangement process of the representative frame image in consideration of the camera work is performed after the arrangement of the representative frame image described with reference to FIGS. 26A and 26B is finished, for example. However, in FIG. 26A and FIG. 26B, it is also possible to arrange representative frame images while taking camerawork into consideration.

  First, the frame image arrangement unit 159 determines whether there is moving image metadata indicating that panning or tilting camera work has been performed (step S501). If the camera work is not pan or tilt, the frame image placement unit 159 ends the representative frame image rearrangement process in consideration of the camera work. If the camerawork is pan or tilt, the frame image placement unit 159 determines whether the corresponding image is an extracted face area (step S503). When the face area is not included, the frame image placement unit 159 equally divides the frame section where the camera work exists (step S505).

  Next, the frame image arrangement unit 159 stores all the detected face areas and the number of frames (step S507).

  Subsequently, the frame image placement unit 159 determines whether the camera work is pan for the corresponding representative frame image based on the moving image metadata (step S509). When the camera work is pan, the frame image arrangement unit 159 sets the representative frame image (frame image) to be vertically long, and arranges the corresponding image horizontally along the camera work direction (step S511). When the camera work is tilted, the frame image placement unit 159 sets the representative frame image (frame image) to be horizontally long, and places the corresponding image vertically along the camera work direction (step S513). ).

  Next, the image size determination unit 157 determines the size of the representative frame image (frame image) according to the designation of the portrait or landscape orientation transmitted from the frame image placement unit 159 and the camera work speed obtained from the moving image metadata. Is determined (step S515).

  Subsequently, the frame image arrangement unit 159 corrects the arrangement of other frames as necessary (step S617).

  By performing the representative frame image arrangement process in consideration of the camera work as described above, it is possible to make the comic display richer in comic expression.

[How to place effect images in consideration of camera work]
Next, with reference to FIG. 28, a method for arranging an effect image in consideration of camera work will be described in detail. FIG. 28 is a flowchart for explaining an effect image arrangement method in consideration of camera work.

  First, the effect image arrangement unit 161 determines whether there is moving image metadata indicating that pan or tilt camera work has been performed (step S601). If the camera work is panning or tilting, the effect image placement unit 161 performs processing for generating an effect line effect image (step S603). If the camera work is not pan or tilt but zoom, for example, the effect image placement unit 161 performs a concentrated line effect image generation process (step S605).

  Below, the case where an effect line effect image is produced | generated first is demonstrated. The effect image arrangement unit 161 detects the frame image in which the camera work is started and the frame image in which the camera work is completed (step S607). Next, the effect image arrangement unit 161 determines whether or not a face area exists in the image based on the moving image metadata (step S609).

  If a face area exists, the effect image placement unit 161 generates an effect line effect image from the edge of the image to the periphery of the face area (step S611). If the face area does not exist, the effect image placement unit 161 generates an effect line effect image at the periphery of the image (step S613). The effect image placement unit 161 determines the density, length, number, and the like of the effect line effect image according to the camera work speed.

  When the generation of the effect line effect image is completed, the effect image placement unit 161 stores the size of the effect line effect image and the frame image on which the effect line effect image is placed (step S615).

  Next, a case where a concentrated line effect image is generated will be described. The effect image arrangement unit 161 detects the frame image where the camera work is started and the frame image where the camera work is finished (step S617). Next, the effect image arrangement unit 161 determines whether or not a face area exists in the image based on the moving image metadata (step S619).

  If a face area exists, the effect image placement unit 161 generates a concentrated line effect image around the face area (step S621). If the face area does not exist, the effect image placement unit 161 generates a concentrated line effect image at the periphery of the image (step S623). The effect image placement unit 161 determines the density, length, number, and the like of the concentrated line effect image according to the camera work speed.

  When the generation of the concentrated line effect image is completed, the effect image placement unit 161 stores the size of the concentrated line effect image and the frame image for which the concentrated line effect image has been placed (step S615).

  By performing the effect image arrangement processing in consideration of the camera work as described above, it is possible to make the comic display richer in comic expression.

[How to determine the animation playback speed]
Next, a method for determining the playback speed of the thumbnail animation will be described in detail with reference to FIG. FIG. 29 is a flowchart for explaining a method of determining the playback speed of the thumbnail animation.

  In the above description, the case where the camera work metadata using unit 155 determines the animation playback speed in advance has been described. However, as will be described below, the comic display execution unit 113 sets the playback speed during animation playback. Can be determined.

  First, the comic display execution unit 113 refers to comic display data and determines whether or not the frame (representative frame image) includes an effect image (step S701).

  If the effect image is a frame, the comic display execution unit 113 reproduces the effect line effect based on the comic display data (step S703), and the process proceeds to step S705. If the effect image does not exist, the comic display execution unit 113 executes the process of step S705 described below.

  Next, the comic display execution unit 113 refers to the comic display data and acquires the number of images used for the thumbnail animation (step S405). The comic display execution unit 113 determines the reproduction speed of the animation using the number of animation frame images and a reference as shown below, for example (step S707).

(1) When the number of animations (n) is 2 or less: Playback speed = 1 (FPS)
(2) When the number of animations (n) is more than 2 and 4 or less: Playback speed = 2 (FPS)
(3) When the number of animations (n) is greater than 4 and less than or equal to 8: Playback speed = 8 (FPS)
(4) When the number of animations (n) exceeds eight: playback speed = 24 (FPS)

  Subsequently, the comic display execution unit 113 reproduces the thumbnail animation based on the determined reproduction speed (step S709).

  By performing such processing, it is possible to determine the playback speed in consideration of the camera work when the thumbnail animation is played back.

<About the first modification>
Then, the 1st modification of the information processing apparatus 10 which concerns on this embodiment is demonstrated in detail, referring FIGS. 30-32.

  In addition to the function of the video analysis unit 103 according to the first embodiment, the video analysis unit 103 of the information processing apparatus 10 according to the present modification detects a person from the video data or detects a subject focused by the user. Or detecting an object designated by the user. Further, the comic display conversion unit 107 of the information processing apparatus 10 according to the present modification further has a function of arranging an effect image avoiding occlusion according to the detection result by the moving image analysis unit 103.

[About the structure of the video analysis unit]
First, the moving image analysis unit 103 according to this modification will be described in detail with reference to FIG. FIG. 30 is a block diagram for explaining the configuration of the moving image analysis unit 103 according to this modification.

  The moving image analysis unit 103 according to the present modification includes a moving image data division unit 131, a face detection unit 133, a scene transition detection unit 135, a camera work detection unit 137, an audio analysis unit 139, and a moving image metadata generation unit 141. It has the following processing units. That is, the moving image analysis unit 103 according to this modification further includes a person detection unit 171, a subject detection unit 173, and an object detection unit 175.

  The moving image data division unit 131, the face detection unit 133, the scene transition detection unit 135, the camera work detection unit 137, and the audio analysis unit 139 according to this modification are the same as the respective processing units according to the first embodiment of the present invention. It has a structure and produces the same effect. Therefore, detailed description is omitted below.

  In addition, the moving image metadata generation unit 141 according to this modification is output from the face detection unit 133, the scene transition detection unit 135, the camera work detection unit 137, the person detection unit 171, the subject detection unit 173, and the object detection unit 175. Based on the result and the like, moving image metadata is generated. The other functions of the moving image metadata generation unit 141 according to the modified example are the same as those of the moving image metadata generation unit 141 according to the first embodiment of the present invention, and the modified example has substantially the same effect. Since it is obtained, detailed description is omitted below.

  The person detection unit 171 includes, for example, a CPU, a ROM, a RAM, and the like. The person detection unit 171 detects a portion corresponding to a person from transmitted image data using a classifier (Classifier) generated in advance using a statistical learning process such as the AdaBoost method.

  This classifier is generated, for example, by integrating a clothes classifier and a contour classifier. A clothing classifier is a relatively strong classifier composed of a plurality of weak classifiers generated by statistical learning. By using the characteristics of a person's clothes, a region of a human image is included in a transmitted image. Used to identify whether or not it exists. Further, the contour classifier is a relatively strong classifier made up of a plurality of weak classifiers generated by statistical learning, and the region of a person's image is included in a transmitted image using a person's contour. Used to identify whether or not it exists.

  When the image data is transmitted from the moving image data dividing unit 131, the person detection unit 171 extracts clothes feature points and contour feature points from the transmitted image data. The person detection unit 171 calculates a clothing feature value and a contour feature value based on each extracted feature point. The person detection unit 171 performs calculation by substituting the calculated clothing feature value and contour feature value into the classifier, and identifies whether or not the person is recognized (detected) in the image based on the obtained calculation result. Output the result.

  The person detection unit 171 according to this modification focuses on two types of feature amounts, that is, a person's clothes and a contour. Accordingly, the person detection unit 171 can detect a person from an image if it can sufficiently extract at least one feature amount from the transmitted image.

  The person detection unit 171 transmits a detection result indicating whether or not a person is present in the transmitted image to the moving image metadata generation unit 141. In addition, when a plurality of persons are detected in the image, the person detection unit 171 may transmit the number of detected persons (number of persons) to the moving image metadata generation unit 141.

  Note that, when a person is present in the image, the person detection unit 171 may transmit position information indicating the position of the pixel corresponding to the person to the moving image metadata generation unit 141. By outputting such position information, other processing units according to this modification can easily know at which position in the image the person is present. In addition, the person detection unit 171 may transmit the calculation result (score output from the discriminator) of the image subjected to the person detection process to the moving image metadata generation unit 141.

  The subject detection unit 173 includes, for example, a CPU, a ROM, a RAM, and the like. The subject detection unit 173 uses, as a subject, an object on the image that is estimated to be noticed by the user when the user glances at the image, that is, an object that is estimated to be turned by the user. To detect.

  The subject detection unit 173 performs subject detection by paying attention to, for example, the degree of blur (degree of blur) of the transmitted image, the degree of appropriateness of exposure, the degree of appropriateness of white balance, and the like. The subject detection process by the subject detection unit 173 will be described in more detail as follows.

  When the image data is transmitted from the moving image data dividing unit 131, the subject detection unit 173 generates a luminance information map indicating information on luminance for each region of the image based on the transmitted image. In addition, the subject detection unit 173 generates a color information map indicating information regarding colors constituting the image for each region of the image based on the transmitted image. In addition, the subject detection unit 173 generates an edge information map indicating information about edges for each region of the image based on the transmitted image. In addition, the subject detection unit 173 generates a face information map indicating information regarding the face of a person as a subject for each region of the image based on the transmitted image. In addition, the subject detection unit 173 generates a motion information map indicating information regarding motion for each region of the image based on the transmitted image.

  The information included in these information maps is information indicating feature amounts of features that are included more in the region including the subject, and the information map is arranged in correspondence with each region of the image to which the information is transmitted. It is. That is, the information map is information indicating the feature amount in each area of the transmitted image. In addition, each above-mentioned information map is an example to the last, Information maps other than the above-mentioned thing may be produced | generated, and only one part may be produced | generated among the above-mentioned each information map.

  The subject detection unit 173 linearly combines the generated information maps to generate a subject map. That is, the information (feature amount) of each area of each information map described above is weighted and added for each area at the same position, and a subject map is generated.

  Subsequently, the subject detection unit 173 uses the transmitted image and the generated subject map as an object to be processed in an area including the subject in the transmitted image as an index (defocus blur index). ) Is generated. In addition, the subject detection unit 173 generates an index (exposure index) indicating the degree of appropriateness of exposure in the entire transmitted image based on the transmitted image. In addition, the subject detection unit 173 generates an index (white balance index) indicating the degree of appropriateness of white balance in the entire transmission image based on the transmitted image.

  The subject detection unit 173 linearly combines these generated indices using preset weights to obtain a subject score for the transmitted image. The subject detection unit 173 detects, for example, an area where the calculated subject score is equal to or greater than a predetermined threshold as a subject.

  The subject detection unit 173 transmits a detection result indicating whether or not a subject is present in the transmitted image to the moving image metadata generation unit 141. Further, the subject detection unit 173 may transmit the number of detected subjects to the moving image metadata generation unit 141 when a plurality of subjects are detected in the image.

  Note that the subject detection unit 173 may transmit position information indicating the position of the pixel corresponding to the subject to the moving image metadata generation unit 141 when the subject exists in the image. By outputting such position information, other processing units according to the present modification can easily know at which position in the image the subject is present. In addition, the subject detection unit 173 may transmit any or all of the calculated subject score, defocus index, exposure index, and white balance index to the moving image metadata generation unit 141.

  The object detection unit 175 includes, for example, a CPU, a ROM, a RAM, and the like. The object detection unit 175 detects an object registered in advance in the database from the transmitted images. Therefore, it is possible for the user of the information processing apparatus 10 according to the present modification to detect an object reflecting the user's preference from the image by registering data characterizing various objects in the database in advance. It becomes.

  The object detection unit 175 performs edge detection processing on an image including a general object or an object specified by the user of the information processing apparatus 10 to generate an edge image. The object detection unit 175 extracts a local feature amount at an edge point on the generated edge image and a point that supports the edge point, and associates the extracted point with the geometric positional relationship and registers it in the database. deep.

  When the image data is transmitted from the moving image data dividing unit 131, the object detection unit 175 performs edge detection processing on the transmitted image and generates an edge image. The object detection unit 175 calculates a local feature amount at an edge point on the edge image using the generated edge image. The object detection unit 175 searches for the feature quantity registered in the database and the geometric positional relationship of the edge points, and determines whether data similar to the generated local feature quantity or the like is registered. If pre-registered data exists, the object detection unit 175 outputs the detection result to the moving image metadata generation unit 141 assuming that the corresponding object exists in the image. Further, when a plurality of objects are detected in the image, the object detection unit 175 may transmit the types and number of detected objects to the moving image metadata generation unit 141.

  The moving image analysis unit 103 according to the present modification includes such a processing unit, so that in addition to the functions of the moving image analysis unit 103 according to the first embodiment, the moving image analysis unit 103 detects a person from the moving image or the subject focused on by the user Or an object designated by the user can be detected. Thereby, the moving image analysis unit 103 according to the present modification can analyze the content of the moving image from more viewpoints. As a result, the information processing apparatus 10 according to the present modification can generate a more natural and powerful comic display screen in which the content of the moving image is reflected.

  Note that the person detection method, the subject detection method, and the object detection method in the above description are merely examples, and details of each detection process executed by the moving image analysis unit 103 according to the present modification are limited to the above examples. Do not mean. The moving image analysis unit 103 according to the present modification can detect a person, a subject, and an object using processing other than the method described above.

  Each of the face detection unit 133, the person detection unit 171, the subject detection unit 173, and the object detection unit 175 includes a person (object) detected first in the image, a person (object) with the highest detection accuracy, a predetermined It is also possible to preferentially detect a predetermined detection target such as a person (face) facing the direction of the person or a smiling person.

  In the above description, the detection result transmitted to the moving image metadata generation unit 141 is merely an example. In addition to the detection results described above, when there are detection results that can be used as feature quantities that characterize moving images, these detection results can be used as moving image metadata.

[About effect image placement processing]
Next, an effect image arrangement process performed by the comic display conversion unit 107 according to the present modification will be described in detail with reference to FIGS. 31A to 31E and FIG. FIG. 31A to FIG. 31E and FIG. 32 are explanatory diagrams for explaining the effect image arrangement method according to the present modification.

  The effect image placement unit 161 of the comic display conversion unit 107 may place the effect image based on the policy described below, according to the analysis result of the moving image by the moving image analysis unit 103.

  For example, when a face area with high detection accuracy exists in the image, the effect image placement unit 161 emphasizes the face area with respect to the entire frame including the face area as shown in FIG. 31A. A concentrated line effect may be placed. Here, as an example of a face area with high detection accuracy, an area in which a numerical value (score) calculated during face detection processing is equal to or greater than a predetermined threshold value can be given. As a result, as shown in FIG. 31A, the entire frame becomes an effect image arrangement area, which is an area in which the effect image is arranged, and the effect image arrangement unit 161 arranges the concentrated line effect in this effect image arrangement area. It will be followed.

  In addition, the moving image analysis unit 103 notifies the comic display conversion unit 107 of a result that a specific area (a face area, a person area, a subject area, an object area) or the like is not detected and only a sound is detected. Cases can also arise. At this time, as shown in FIG. 31B, the effect image arrangement unit 161 may arrange a predetermined effect image indicating that sound is present in the entire frame area.

  In addition, there may be a case where the moving image analysis unit 103 notifies the comic display conversion unit 107 of a result that an area with low detection accuracy is detected and a sound is also detected. At this time, the effect image arrangement unit 161 sets the effect image arrangement area so as to avoid the detected area (the area where the detection accuracy is not high), and arranges the balloon effect corresponding to the sound in the effect image arrangement area. May be. At this time, it is preferable that the effect image placement unit 161 sets the direction of the effect image to be placed in consideration of the relative positional relationship between the detected region with low accuracy and the effect image placement region. Here, the size of the effect image arrangement area to be set is appropriately determined according to the aspect ratio of the frame.

  For example, as shown in FIG. 31C, when a face area with low detection accuracy is detected on the right side of the frame and sound is also detected, the effect image placement unit 161 avoids the detected face area. Set the effect image placement area on the left side of. In addition, since the detected sound is often related to the detected face area, the effect image placement unit 161 sets the direction of the balloon effect to be placed from the right side to the left side of the frame where the face area is located. .

  If an area with low detection accuracy is detected, placing an effect image to emphasize the detected area will highlight the wrong detection result when the detection fails. It becomes. Therefore, as shown in FIG. 31C, by arranging the effect image while avoiding the detection area, even if an erroneous detection result is output, a natural effect is given to the processing target image. The possibility of being able to be increased.

  In addition, there may be a case where the moving image analysis unit 103 notifies the comic display conversion unit 107 of a result that a plurality of areas are detected and a sound is also detected. At this time, the effect image arrangement unit 161 sets a larger area that includes a plurality of detection areas. The effect image arrangement unit 161 may set an effect image arrangement area so as to avoid the set area, and may arrange a balloon effect corresponding to sound in the effect image arrangement area. At this time, it is preferable that the effect image placement unit 161 sets the direction of the effect image to be placed in consideration of the relative positional relationship between the set larger region and the effect image placement region.

  Further, not only when a person or a face is detected, but also when an object detected as a subject or a general object is detected as shown in FIG. Can arrange an effect image as described above.

  When the effect image placement unit 161 places the effect image while considering the direction of the effect image, for example, as shown in FIG. 32, the position of the center of gravity of the entire frame and the position of the center of gravity of the detected area The relative positional relationship is calculated to determine the direction of the effect image. As a result, the effect image placement unit 161 can effectively place the effect image by further increasing the relationship between the effect image and the detected area while avoiding occlusion.

(Second Embodiment)
Next, an information processing apparatus and an information processing method according to the second embodiment of the present invention will be described in detail with reference to FIGS. The information processing apparatus and the information processing method according to the present embodiment automatically extract effective images for grasping the contents of moving images and still images using the analysis results of moving images and still images, and the extracted images Is displayed so that it is easy to grasp the contents. Thereby, when recording regarding a certain event is performed using both a moving image and a still image, the content of a certain event can be easily grasped without distinguishing between a moving image and a still image.

<Configuration of information processing device>
First, the configuration of the information processing apparatus according to the present embodiment will be described in detail with reference to FIGS. 33 to 35. FIG. 33 is a block diagram for explaining the configuration of the information processing apparatus according to the present embodiment. FIG. 34 is a block diagram for explaining the configuration of the still image analysis unit 1003 according to this embodiment, and FIG. 35 is a block diagram for explaining the configuration of the comic display conversion unit 1005 according to this embodiment. is there.

[Overall configuration of information processing device]
As shown in FIG. 33, for example, the information processing apparatus 10 according to the present embodiment includes a moving image data acquisition unit 101, a moving image analysis unit 103, an audio extraction unit 105, a frame image data generation unit 109, a comic display data generation unit 111, and A comic display execution unit 113 is provided. The information processing apparatus 10 further includes a display control unit 115, a storage unit 117, a still image data acquisition unit 1001, a still image analysis unit 1003, and a comic display conversion unit 1005. In addition to these processing units, the information processing apparatus 10 includes a communication control unit (not shown) that controls communication performed with an arbitrary device via an arbitrary communication network, for example. You may do it.

  Here, the moving image data acquisition unit 101, the moving image analysis unit 103, the sound extraction unit 105, the frame image data generation unit 109, the comic display data generation unit 111, the comic display execution unit 113, the display control unit 115, and the storage unit 117 Since it has the same configuration as each processing unit according to the first embodiment and has substantially the same effect, detailed description will be omitted below.

  The still image data acquisition unit 1001 includes, for example, a CPU, a ROM, a RAM, a communication device, and the like. The still image data acquisition unit 1001 has inserted still image data corresponding to a still image specified by the user into the storage unit 117 or the information processing device 10 in response to a user operation performed on the information processing device 10. Obtained from various recording media. Here, it is assumed that time information regarding the time when the still image data is generated is associated with the still image data. The still image data acquisition unit 1001 acquires still image data from other devices connected to the information processing device 10, or acquires still image data from various servers connected via a communication network such as the Internet. It is possible to do. The still image data acquisition unit 1001 transmits the acquired still image data to the still image analysis unit 1003 and the frame image data generation unit 109, respectively.

  The still image analysis unit 1003 includes, for example, a CPU, a ROM, a RAM, and the like. The still image analysis unit 1003 analyzes the transmitted still image data and extracts a feature amount that characterizes the still image. Further, the still image analysis unit 1003 generates still image metadata, which is metadata related to the feature amount, using the extracted feature amount. The still image analysis unit 1003 transmits the generated still image metadata to the comic display conversion unit 1005 described later. The still image analysis unit 1003 uses, in addition to still image metadata generated by analyzing a still image, metadata that can be acquired on an imaging device such as a digital camera in a comic display conversion unit 1005 described later. It can also be used as still image metadata. Here, as metadata that can be acquired on the imaging apparatus, for example, an operation history of a favorite button, a zoom-in button, a zoom-out button, or the like can be cited.

  The still image analysis unit 1003 will be described in detail later.

  The comic display conversion unit 1005 includes, for example, a CPU, a ROM, a RAM, and the like. The comic display conversion unit 1005 extracts a plurality of images from the moving image data and the still image data based on the moving image metadata and the still image metadata. In addition, the comic display conversion unit 1005 performs a manga-like layout conversion of a plurality of images extracted by dividing a predetermined display area into frames, and generates frame information including information regarding images arranged in the respective frames. The comic display conversion unit 1005 transmits the generated frame information to the frame image data generation unit 109 and the comic display data generation unit 111.

  The comic display conversion unit 1005 will be described in detail later.

[Configuration of Still Image Analysis Unit 1003]
Next, the configuration of the still image analysis unit 1003 according to the present embodiment will be described in detail with reference to FIG.

  As illustrated in FIG. 34, for example, the still image analysis unit 1003 according to the present embodiment includes a face detection unit 1011, a person detection unit 1013, a subject detection unit 1015, an object detection unit 1017, a time stamp determination unit 1019, and a still image meta. A data generation unit 1021 is mainly provided.

  Here, the face detection unit 1011, the person detection unit 1013, the subject detection unit 1015, and the object detection unit 1017 are the same as the respective processing units according to the first embodiment of the present invention and the first modification of the first embodiment. In order to achieve the same effect, detailed description will be omitted below.

  The time stamp determination unit 1019 includes, for example, a CPU, a ROM, a RAM, and the like. The time stamp determination unit 1019 determines when the still image data is generated with reference to time information associated with the still image data transmitted from the still image data acquisition unit 1001. The time stamp determination unit 1019 transmits the time stamp of the still image data to a still image metadata generation unit 1021 described later.

  By transmitting the time stamp as one of the still image metadata to the still image metadata generation unit 1021, the other processing unit of the information processing apparatus 10 identifies when the corresponding still image is generated. It becomes possible. By specifying such a time stamp, the frame image arrangement unit 159 of the comic display conversion unit 1005 can easily determine at which position of the plurality of images selected from the moving image the extracted still image should be inserted. It becomes possible to judge.

  The still image metadata generation unit 1021 includes, for example, a CPU, a ROM, a RAM, and the like. The still image metadata generation unit 1021 is a feature quantity that characterizes a still image based on detection results transmitted from the face detection unit 1011, the person detection unit 1013, the subject detection unit 1015, the object detection unit 1017, and the time stamp determination unit 1019. Still image metadata is generated. That is, the still image metadata includes metadata related to the feature amount extracted from the image data.

  The still image metadata generation unit 1021 transmits the still image metadata generated as described above to the comic display conversion unit 1005.

The configuration of the still image analysis unit 1003 according to the present embodiment has been described above.
Note that FIG. 34 illustrates a case where the still image analysis unit 1003 includes a face detection unit 1011, a person detection unit 1013, a subject detection unit 1015, and an object detection unit 1017, but the still image analysis unit 1003 is described above. Some of the processing units may be provided.

  In the information processing apparatus 10 according to the present embodiment, the case where each of the moving image analysis unit 103 and the still image analysis unit 1003 includes a processing unit (each detection unit) having the same function has been described. However, each detection unit shared by the moving image analysis unit 103 and the still image analysis unit 1003 may be provided in the information processing apparatus 10.

[Configuration of Comic Display Conversion Unit 1005]
Next, the configuration of the comic display conversion unit 1005 according to the present embodiment will be described in detail with reference to FIG.

  The comic display conversion unit 1005 includes a digest score calculation unit 151, a thumbnail number determination unit 153, a camera work metadata use unit 155, a frame image arrangement unit 159, an effect image arrangement unit 161, a frame information generation unit 163, and an image size determination unit 1031. Is mainly provided.

  Here, the digest score calculation unit 151, the thumbnail number determination unit 153, the camera work metadata use unit 155, the frame image arrangement unit 159, the effect image arrangement unit 161, and the frame information generation unit 163 are the first embodiment of the present invention. And since it has the structure similar to each process part which concerns on the 1st modification of 1st Embodiment, and there exists substantially the same effect, below, detailed description is abbreviate | omitted.

  The image size determination unit 1031 is constituted by, for example, a CPU, a ROM, a RAM, and the like. Similar to the image size determining unit 157 according to the first embodiment, the image size determining unit 1031 determines the size of the thumbnail image of the moving image arranged in the predetermined display area based on the digest score.

  When there is moving image metadata indicating that a face exists in the image for the selected thumbnail image, the image size determination unit 1031 selects a face region that is a region where the face is displayed. Set the image size so that it is enlarged. When there are a plurality of face areas in the image, the image size determination unit 1031 enlarges the size of the image data so as to include all the face areas.

  The image size determination unit 1031 determines the size of the still image thumbnail image arranged in the predetermined display area based on the still image metadata transmitted from the still image analysis unit 1003. The image size determination unit 1031 can also select a still image to be used for comic display for a plurality of still images input to the information processing apparatus 10 based on the still image metadata and the still image size. . For example, the image size determination unit 1031 may preferentially select a still image having metadata indicating that a face / person / subject has been detected. The image size determination unit 1031 may preferentially select a large image size based on the image size of the still image determined by the image size determination method described below. The method for determining the image size of a still image based on still image metadata will be described in detail later.

  The image size determination unit 1031 uses the information regarding the size of the moving image and the still image thumbnail image (and thus the size of the animation frame image) set as described above as the frame image information, and the frame image arrangement unit 159 and the frame information generation unit. 163 to be transmitted.

  Heretofore, an example of the function of the information processing apparatus 10 according to the present embodiment has been shown. Each component described above may be configured using a general-purpose member or circuit, or may be configured by hardware specialized for the function of each component. In addition, the CPU or the like may perform all functions of each component. Therefore, it is possible to appropriately change the configuration to be used according to the technical level at the time of carrying out the present embodiment.

  It should be noted that a computer program for realizing each function of the information processing apparatus according to the present embodiment as described above can be produced and installed in a personal computer or the like. In addition, a computer-readable recording medium storing such a computer program can be provided. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like. Further, the above computer program may be distributed via a network, for example, without using a recording medium.

<About information processing method>
Subsequently, the information processing method according to the present embodiment will be described in detail with reference to FIGS. 36 and 37. FIG. 36 is a flowchart for explaining the overall flow of the information processing method according to the present embodiment. FIG. 37 is a flowchart for explaining a method of determining the image size of a still image.

[Overall flow of information processing method]
First, the overall flow of the information processing method according to the present embodiment will be described in detail with reference to FIG.

  First, the moving image data acquisition unit 101 of the information processing apparatus 10 according to the present embodiment acquires moving image data corresponding to a moving image specified by the user in accordance with a user operation (step S1001). The moving image data acquisition unit 101 may acquire moving image data from the storage unit 117 or the like in the own device or may acquire from various recording media attached to the own device. The moving image data acquisition unit 101 may acquire moving image data from another device connected wirelessly or by wire, or may acquire it from another device connected via a communication network such as the Internet. Good. The moving image data acquisition unit 101 transmits the acquired moving image data to the moving image analysis unit 103 and the audio extraction unit 105.

  Further, the still image data acquisition unit 1001 of the information processing apparatus 10 according to the present embodiment acquires still image data corresponding to the still image specified by the user in response to a user operation (step S1001). The still image data acquisition unit 1001 may acquire the still image data from the storage unit 117 or the like in the own device, or may acquire from various recording media attached to the own device. The still image data acquisition unit 1001 may acquire still image data from another device connected wirelessly or by wire, or may acquire it from another device connected via a communication network such as the Internet. May be. The still image data acquisition unit 1001 transmits the acquired still image data to the still image analysis unit 1003 and the frame image data generation unit 109.

  The moving image analysis unit 103 analyzes the moving image data transmitted from the moving image data acquisition unit 101, and generates moving image metadata that is metadata relating to a feature amount characterizing the moving image corresponding to the transmitted moving image data (step S1003). . The moving image analysis unit 103 transmits the generated moving image metadata to the comic display conversion unit 1005.

  Further, the audio extraction unit 105 extracts audio data from the moving image data transmitted from the moving image data acquisition unit 101 (step S1005), and transmits the acquired audio data to the comic display data generation unit 111.

  On the other hand, the still image analysis unit 1003 analyzes the still image data transmitted from the still image data acquisition unit 1001 and performs still image meta data that is metadata relating to a feature amount characterizing the still image corresponding to the transmitted still image data. Data is generated (step S1007). The still image analysis unit 1003 transmits the generated still image metadata to the comic display conversion unit 1005.

  Next, the digest score calculation unit 151 of the comic display conversion unit 1005 calculates a digest score for all images (frame images) constituting the moving image based on the transmitted moving image metadata. The digest score calculation unit 151 transmits the calculated digest score to the thumbnail number determination unit 153.

  Subsequently, the thumbnail number determination unit 153 selects a representative frame image to be used as a thumbnail image using the transmitted digest score and moving image metadata (step S1009). Further, when the selection of the representative frame image is completed, the thumbnail number determination unit 153 selects an animation frame image used for the thumbnail animation. The number-of-thumbnail determination unit 153 generates number information including information for specifying the selected representative frame image, information about the number of representative frame images, information about an animation frame image related to the representative frame image, and the like. The thumbnail number determination unit 153 transmits the generated number information to the image size determination unit 1031 and the frame information generation unit 163.

  It should be noted that the analysis result by the camera work metadata utilization unit 155 can be reflected when determining the playback speed of the animation frame image or thumbnail animation.

  Next, the image size determination unit 1031 determines an image size for displaying the selected representative frame image in comics based on the transmitted number information, moving image metadata, digest score, and the like (step S1011). In addition, the image size determination unit 1031 determines the image size of the still image based on the still image metadata transmitted from the still image analysis unit 1003 (step S1013). The image size determination unit 1031 generates frame image information including information related to the image size of the representative frame image, and transmits the frame image information to the frame image arrangement unit 159 and the frame information generation unit 163.

  The image size determination unit 1031 can also select a still image to be used for comic display for a plurality of still images input to the information processing apparatus 10 based on the still image metadata and the still image size. . For example, the image size determination unit 1031 may preferentially select a still image having metadata indicating that a face / person / subject has been detected. The image size determination unit 1031 may preferentially select a large image size based on the image size of the still image determined by the image size determination method described below.

  Subsequently, the frame image arrangement unit 159 automatically arranges the representative frame image in the display area using the transmitted frame image information, moving image metadata, still image metadata, and the like, and arranges the representative frame image. Is determined (step S1015). When the arrangement of the representative frame image is determined, the frame image arrangement unit 159 generates frame arrangement information including information such as which representative frame image is arranged at which position, and an effect image arrangement unit 161 and a frame information generation unit 163. And transmit.

  Next, the effect image arrangement unit 161 arranges an effect image such as a balloon based on the transmitted frame arrangement information, moving image metadata, still image metadata, and the like (step S1017). At this time, the effect image arrangement unit 161 arranges the effect images based on the effect image arrangement method as described in the first embodiment of the present invention and the first modification of the first embodiment. . When the effect image placement unit 161 finishes placing the effect image, the effect image placement unit 161 generates effect placement information including information about which effect is placed where, and transmits the effect placement information to the frame information generation unit 163.

  It should be noted that when determining the image size, the frame image arrangement, and the effect image arrangement, it is possible to use the feature amount relating to the camera work included in the moving image metadata.

  Subsequently, the frame information generation unit 163 generates frame information based on the number information, frame image information, frame arrangement information, effect arrangement information, and the like transmitted from each processing unit (step S1019). The frame information generation unit 163 transmits the generated frame information to the frame image data generation unit 109 and the comic display data generation unit 111.

  Next, the frame image data generation unit 109 performs a process of adjusting the size and shape of the frame image used for comic display based on the image data and the frame information, and generates the frame image data used for comic display. Generate (step S1021). The frame image data generation unit 109 transmits the generated frame image data to the comic display data generation unit 111.

  The comic display data generation unit 111 generates audio data used for comic display based on the audio data transmitted from the audio extraction unit 105 and the frame information transmitted from the comic display conversion unit 1005. Further, the comic display data generation unit 111 generates comic display data, which is a set of data necessary for realizing comic display, using the frame information, audio data, frame image data, and effect data. (Step S1023).

  By executing the generated comic display data, the information processing apparatus 10 can display a comic display that summarizes the content of the moving image on the display screen.

  It should be noted that the frame image generated from the still image can be animated by moving one still image when browsing after the comic is generated. In addition, for a frame image generated from a still image, it is possible to select an image to be used for animation from a plurality of still images by still image clustering, and perform animation using the selected image.

[How to determine the image size of still images]
Next, a method for determining the image size of a still image performed by the image size determination unit 1031 will be described in detail with reference to FIG.

  When determining the image size of the still image, the image size determining unit 1031 first sets a parameter representing the image size of the still image to an initial value (step S1101).

  Subsequently, the image size determination unit 1031 refers to the still image metadata transmitted from the still image analysis unit 1003, and determines whether or not two or more face regions are detected in the still image being processed (step S1). S1103). If two or more face areas are not detected, the image size determination unit 1031 executes step S1111 described later. If two or more face areas are detected, the image size determination unit 1031 increases a parameter representing the image size (step S1105). Next, the image size determination unit 1031 determines whether or not the size of the detected face area is 80 pixels or more (step S1107). If the size of the face area is 80 pixels or more, the image size determination unit 1031 increases a parameter representing the image size (step S1109). If the size of the face area is not 80 pixels or more, the image size determination unit 1031 executes step S1111 described later.

  Next, the image size determination unit 1031 refers to the still image metadata and determines whether or not the defocus index of the still image being processed is 50 or more (step S1111). When the defocus index is 50 or more, there is a possibility that the image is blurred. Therefore, the image size determination unit 1031 decreases the parameter representing the image size (step S1113), and thereafter, step S1115 described later is performed. Run. On the other hand, if the defocus index is not 50 or more, the image size determination unit 1031 executes step S1117 described later.

  Subsequently, the image size determination unit 1031 calculates a subject region score defined by (area of a region detected as a subject / distance from the image center) (step S1115), and the calculated subject region score is 0.75. It is determined whether or not this is the case (step S1117). When the subject area score is 0.75 or more, the image size determination unit 1031 increases a parameter indicating the size of the image (step S1119), and executes step S1121 described later. If the subject area score is not greater than or equal to 0.75, the image size determination unit 1031 executes step S1121 described later.

  Next, the image size determination unit 1031 refers to the final value of the parameter representing the image size, and determines the image size based on the parameter value (step S1121).

  The image size determination unit 1031 can determine the image size used for comic display by performing processing based on the still image metadata in such a procedure.

  Note that the threshold values shown in the above description are merely examples, and may be set to arbitrary values according to various factors such as the shooting conditions and size of the still image processed by the image size determination unit 1031. Is possible.

(Third embodiment)
Next, an information processing apparatus according to the third embodiment of the present invention will be described in detail with reference to FIG. The information processing apparatus according to the present embodiment automatically extracts effective images for grasping the contents of a series of still images using the analysis result of the still images, and the contents of the extracted images can be easily grasped. To display.

<Configuration of information processing device>
FIG. 38 is a block diagram for explaining the configuration of the information processing apparatus according to the present embodiment.
The information processing apparatus 10 according to the present embodiment includes a frame image data generation unit 109, a comic display data generation unit 111, a comic display execution unit 113, a display control unit 115, and a storage unit 117, for example, as illustrated in FIG. . The information processing apparatus 10 according to the present embodiment further includes a still image data acquisition unit 1001, a still image analysis unit 1003, and a comic display conversion unit 1101.

  The frame image data generation unit 109, the comic display data generation unit 111, the comic display execution unit 113, the display control unit 115, and the storage unit 117 are each processing according to the first embodiment, the first modified example, and the second embodiment. It has the same configuration as the unit and has the same effect. Therefore, detailed description is omitted below.

  In addition, the still image data acquisition unit 1001 and the still image analysis unit 1003 have the same configuration as the still image data acquisition unit 1001 and the still image analysis unit 1003 according to the second embodiment, and have the same effects. Hereinafter, detailed description is omitted.

  The comic display conversion unit 1101 includes, for example, a CPU, a ROM, a RAM, and the like. The comic display conversion unit 1101 divides a predetermined display area into frames based on the still image metadata transmitted from the still image analysis unit 1003, arranges and converts a plurality of still images in a manga manner, and arranges them in each frame. The frame information including the information regarding the processed image is generated. The comic display conversion unit 1101 transmits the generated frame information to the frame image data generation unit 109 and the comic display data generation unit 111.

  Since the detailed configuration of the comic display conversion unit 1101 is similar to the detailed configuration of the comic display conversion unit 1005 according to the second embodiment of the present invention, detailed description thereof will be omitted below.

  The still image data acquisition unit 1001 of the information processing apparatus 10 according to the present embodiment acquires still image data corresponding to a still image specified by the user and transmits the still image data to the still image analysis unit 1003 in response to a user operation. The still image analysis unit 1003 analyzes the still image data transmitted from the still image data acquisition unit 1001 and generates still image metadata, which is metadata related to the feature amount characterizing the still image corresponding to the transmitted still image data. Generate. The still image analysis unit 1003 transmits the generated still image metadata to the comic display conversion unit 1101.

  The image size determination unit 1031 of the comic display conversion unit 1101 determines the image size of the still image based on the still image metadata transmitted from the still image analysis unit 1003. The image size determination unit 1031 generates frame image information including information regarding the image size of the representative frame image, and transmits the frame image information to the frame image arrangement unit 159 and the frame information generation unit 163 of the comic display conversion unit 1101.

  The frame image arrangement unit 159 of the comic display conversion unit 1101 automatically arranges the representative frame image in the display area by using the transmitted frame image information and still image metadata, and arranges the arrangement of the representative frame image. decide. When determining the arrangement of the representative frame image, the frame image arrangement unit 159 generates frame arrangement information including information such as which representative frame image is arranged at which position, and the effect image arrangement unit 161 of the comic display conversion unit 1101 and It is transmitted to the frame information generation unit 163.

  Next, the effect image arrangement unit 161 of the comic display conversion unit 1101 arranges an effect image such as a balloon based on the transmitted frame arrangement information, still image metadata, and the like. At this time, the effect image arrangement unit 161 arranges the effect images based on the effect image arrangement method as described in the first embodiment of the present invention and the first modification of the first embodiment. . When the effect image placement unit 161 finishes placing the effect image, the effect image placement unit 161 generates effect placement information including information about which effect is placed where, and transmits the effect placement information to the frame information generation unit 163.

  Subsequently, the frame information generation unit 163 of the comic display conversion unit 1101 generates frame information based on frame image information, frame arrangement information, effect arrangement information, and the like transmitted from each processing unit. The frame information generation unit 163 transmits the generated frame information to the frame image data generation unit 109 and the comic display data generation unit 111.

  Next, the frame image data generation unit 109 performs a process of adjusting the size and shape of the frame image used for comic display based on the image data and the frame information, and generates the frame image data used for comic display. Generate. The frame image data generation unit 109 transmits the generated frame image data to the comic display data generation unit 111.

  The comic display data generation unit 111 uses the frame information, the frame image data, and the effect data to generate comic display data that is a set of data necessary for realizing comic display.

  By executing the generated comic display data, the information processing apparatus 10 can display a comic display that summarizes the content of the moving image on the display screen.

  Heretofore, an example of the function of the information processing apparatus 10 according to the present embodiment has been shown. Each component described above may be configured using a general-purpose member or circuit, or may be configured by hardware specialized for the function of each component. In addition, the CPU or the like may perform all functions of each component. Therefore, it is possible to appropriately change the configuration to be used according to the technical level at the time of carrying out the present embodiment.

  It should be noted that a computer program for realizing each function of the information processing apparatus according to the present embodiment as described above can be produced and installed in a personal computer or the like. In addition, a computer-readable recording medium storing such a computer program can be provided. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like. Further, the above computer program may be distributed via a network, for example, without using a recording medium.

(About hardware configuration)
Next, the hardware configuration of the information processing apparatus 10 according to each embodiment of the present invention will be described in detail with reference to FIG. FIG. 39 is a block diagram for explaining a hardware configuration of the information processing apparatus 10 according to each embodiment of the present invention.

  The information processing apparatus 10 mainly includes a CPU 901, a ROM 903, and a RAM 905. The information processing apparatus 10 further includes a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, and a connection port 923. And a communication device 925.

  The CPU 901 functions as an arithmetic processing unit and a control unit, and controls all or a part of the operation in the information processing apparatus 10 according to various programs recorded in the ROM 903, the RAM 905, the storage device 919, or the removable recording medium 927. The ROM 903 stores programs used by the CPU 901, calculation parameters, and the like. The RAM 905 primarily stores programs used in the execution of the CPU 901, parameters that change as appropriate during the execution, and the like. These are connected to each other by a host bus 907 constituted by an internal bus such as a CPU bus.

  The host bus 907 is connected to an external bus 911 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 909.

  The input device 915 is an operation unit operated by the user, such as a mouse, a keyboard, a touch panel, a button, a switch, and a lever. Further, the input device 915 may be, for example, remote control means (so-called remote controller) using infrared rays or other radio waves, or an external connection device such as a mobile phone or a PDA corresponding to the operation of the information processing device 10. 929 may be used. Furthermore, the input device 915 includes an input control circuit that generates an input signal based on information input by a user using the above-described operation means and outputs the input signal to the CPU 901, for example. The user of the information processing apparatus 10 can input various data and instruct a processing operation to the information processing apparatus 10 by operating the input device 915.

  The output device 917 is configured by a device capable of visually or audibly notifying acquired information to the user. Examples of such devices include CRT display devices, liquid crystal display devices, plasma display devices, EL display devices and display devices such as lamps, audio output devices such as speakers and headphones, printer devices, mobile phones, and facsimiles. For example, the output device 917 outputs results obtained by various processes performed by the information processing apparatus 10. Specifically, the display device displays results obtained by various processes performed by the information processing device 10 as text or images. On the other hand, the audio output device converts an audio signal composed of reproduced audio data, acoustic data, and the like into an analog signal and outputs the analog signal.

  The storage device 919 is a data storage device configured as an example of a storage unit of the information processing device 10. The storage device 919 includes, for example, a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 919 stores programs executed by the CPU 901 and various data, and acoustic signal data and image signal data acquired from the outside.

  The drive 921 is a recording medium reader / writer, and is built in or externally attached to the information processing apparatus 10. The drive 921 reads information recorded on a removable recording medium 927 such as a mounted magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs the information to the RAM 905. In addition, the drive 921 can write a record on a removable recording medium 927 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory. The removable recording medium 927 is, for example, a DVD medium, an HD-DVD medium, a Blu-ray (registered trademark) medium, or the like. Further, the removable recording medium 927 may be a compact flash (registered trademark) (CompactFlash: CF), a memory stick, an SD memory card (Secure Digital memory card), or the like. Further, the removable recording medium 927 may be, for example, an IC card (Integrated Circuit card) on which a non-contact IC chip is mounted, an electronic device, or the like.

  The connection port 923 is a port for directly connecting a device to the information processing apparatus 10. An example of the connection port 923 is a USB (Universal Serial Bus) port, i. There are IEEE 1394 ports such as Link, SCSI (Small Computer System Interface) ports, and the like. As another example of the connection port 923, there are an RS-232C port, an optical audio terminal, a high-definition multimedia interface (HDMI) port, and the like. By connecting the external connection device 929 to the connection port 923, the information processing apparatus 10 directly acquires acoustic signal data and image signal data from the external connection device 929, or acquires acoustic signal data and image signals from the external connection device 929. Or provide data.

  The communication device 925 is a communication interface including a communication device for connecting to the communication network 931, for example. The communication device 925 is, for example, a communication card for wired or wireless LAN (Local Area Network), Bluetooth, or WUSB (Wireless USB). The communication device 925 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various communication. The communication device 925 can transmit and receive signals and the like according to a predetermined protocol such as TCP / IP, for example, with the Internet or other communication devices. The communication network 931 connected to the communication device 925 is configured by a wired or wireless network, and may be, for example, the Internet, a home LAN, infrared communication, radio wave communication, satellite communication, or the like. .

  Heretofore, an example of the hardware configuration capable of realizing the function of the information processing apparatus 10 according to each embodiment of the present invention has been shown. Each component described above may be configured using a general-purpose member, or may be configured by hardware specialized for the function of each component. Therefore, it is possible to change the hardware configuration to be used as appropriate according to the technical level at the time of carrying out this embodiment.

<Summary>
As described above, by using this method, it is possible to display private videos of various events (for example, drinking parties, trips, etc.) like comics. Private videos that are only a few minutes long can be expressed in a few pages, depending on the size of the frames, and can be viewed more easily. Further, by generating comic display data as content such as Flash, it is possible to turn the page on a web browser or the like and browse while reproducing the sound of the designated frame.

  In this method, the moving image metadata is extracted by analyzing the moving image data by the moving image analysis unit. However, when the moving image is TV content, information such as subtitle data and EPG data can be used. It is. Thereby, for example, it is also possible to display a person's lines in a balloon.

  Thus, according to this method, the contents of the moving image can be summarized so that the contents of the moving image can be understood at a glance. As a result, it is possible to save time for reproducing all the moving images, and to roughly view the whole without missing an important scene. In addition, the climax scene can be effectively expressed by reflecting the scene size. In addition, since the list of moving images is improved, it is easy to search for a specific moving image from a large number of moving images.

  In addition, according to the present technique, various comic expressions such as frame division and speech balloons that make a user want to see a moving image are possible. As a result, the extraction of the exciting scene becomes a dynamic frame division as it is, which is useful as a fun part of the comic. In addition, the face extraction function can be used for a comic without hiding a frame showing a good face in a balloon. In addition, the speech data classification function can generate speech bubbles corresponding to speech lines. In addition, it is possible to more easily understand a moving image scene by using various balloons. In addition, it is possible to prevent a phenomenon in which a moving image is taken, but it becomes troublesome as it is taken and cannot be seen, and interest in the moving image can be aroused. In addition, by expressing the moving image as a comic image, it is possible to relieve the extended feeling of private video.

  Further, comic display data in this method can be described in various scripts and languages such as Flash, JavaScript, Java / BREW. For example, when Flash or the like is used, it is possible to create interactive content such as turning a page or playing a sound. In addition, since browsing with a general Web browser is possible, it is possible to eliminate the complexity of installing a dedicated application. In addition, since a list of moving image contents can be displayed, the expression and convenience of a moving image sharing site and the like can be improved. Thereby, comic display data can be uploaded on a network and sharing of a moving image with others can be promoted. In addition, this method can improve the compatibility with pasting on homepages and blogs because of its high listability and interesting expression.

  In addition to the detection results of face areas, various recognition and detection technologies such as person detection, subject detection, and object detection can be used to extract various images useful for understanding the content. It is possible to easily grasp the contents.

  Also, by changing the placement position of the effect image according to the detection accuracy of the detected area, the position of the detected area, etc., it becomes possible to highlight and display important objects in the image, It is possible to prevent the effect image from hiding an important object.

  In addition, it is possible to convert a variety of media content such as video content as well as still image content together into a comic display so that the user can easily enjoy digests of multiple media. it can.

  The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present invention.

The following configurations also belong to the technical scope of the present invention.
(1)
An image analysis unit that analyzes the image data and generates image metadata that is metadata relating to a feature amount that characterizes the image;
A frame information determination unit that extracts a plurality of images from the image data based on the image metadata and determines frame information regarding a frame for arranging the extracted plurality of images;
An image display data generation unit that generates image display data including at least the frame information and the extracted image data;
An information processing apparatus comprising:
(2)
Time information is associated with the image data,
The information processing apparatus according to (1), wherein the frame information determination unit extracts the image data in time order based on the time information.
(3)
The information processing apparatus according to (1) or (2), wherein the frame information determination unit automatically adjusts the frame according to the size of the image, and performs layout conversion of the extracted image.
(4)
The frame information determination unit determines the size of the plurality of images arranged in each frame based on the generated image metadata, and the frame information determination unit is any one of (1) to (3) The information processing apparatus described.
(5)
The frame information determination unit
A digest score calculation unit that calculates a digest score that is a value obtained by weighting and adding the feature amount to the image data based on the image metadata;
A thumbnail number determining unit that extracts thumbnail images to be arranged in the frame from the image data based on the digest score, and determines the number of thumbnail images;
An image size determining unit that determines the size of a thumbnail image arranged in a predetermined display area based on the digest score;
A frame image arrangement unit that arranges the thumbnail images of which the image size has been determined in the frame in a predetermined order;
A frame information generation unit that generates the frame information regarding the frame in which the thumbnail image is arranged;
The information processing apparatus according to any one of (1) to (4), further including:
(6)
The information processing apparatus according to (5), wherein the thumbnail number determination unit further extracts a plurality of images for a predetermined time continuous to the extracted thumbnail images as animation images.
(7)
The image metadata includes image metadata related to camera work when capturing the image,
The frame information determination unit further includes a camera work metadata utilization unit that determines the number of images extracted as the animation image by the thumbnail number determination unit based on image metadata related to the camera work. The information processing apparatus according to 5) or (6).
(8)
The camera work metadata using unit is configured to select the animation image from a plurality of images existing between the extracted thumbnail image and an image corresponding to a point in time when the integrated value of the camera work becomes a predetermined threshold value. The information processing apparatus according to (7), wherein the image is selected.
(9)
The information processing apparatus according to (8), wherein the camera work metadata utilization unit sets a reproduction speed of an animation composed of the thumbnail image and the animation image according to a magnitude of an integral value of the camera work. .
(10)
The frame information determination unit further includes an effect image arrangement unit that arranges at least one of a balloon and a screen effect based on the image metadata with respect to the frame on which the thumbnail image is arranged. The information processing apparatus according to any one of 9).
(11)
The information processing apparatus according to (10), wherein the effect image arrangement unit arranges at least one of the balloon and the screen effect using the image metadata regarding audio data included in the image data.
(12)
The information processing unit according to (10) or (11), wherein the effect image arrangement unit arranges the screen effect on the thumbnail image using image metadata related to camerawork when the image is captured. apparatus.
(13)
The image size determination unit enlarges a face area, which is an area where a face is displayed, into the thumbnail image with respect to an image including the image metadata indicating that a face is present in the image (5). The information processing apparatus according to any one of (12) to (12).
(14)
The image metadata includes at least one of image metadata indicating that a person is present in the image, image metadata relating to a subject existing in the image, and image metadata relating to a pre-registered object existing in the image. In addition,
The effect image arrangement unit includes image metadata indicating that a person is present in the image, image metadata relating to a subject existing in the image, or image metadata relating to a pre-registered object existing in the image. The information processing apparatus according to any one of (10) to (13), wherein the screen effect is arranged based on the information.
(15)
The effect image arrangement unit is configured to detect an area in which the predetermined object is detected based on a relative positional relationship between a centroid of an area in which the predetermined object is detected in the thumbnail image and a centroid of the predetermined display area. The information processing apparatus according to any one of (10) to (14), wherein the screen effect is arranged in a region other than the above.
(16)
The image size determination unit, when there are a plurality of the face areas in the image, performs an enlargement process so as to include all the face areas and sets the thumbnail image as any one of (13) to (15) The information processing apparatus as described in any one.
(17)
Analyzing image data and generating image metadata, which is metadata relating to a feature amount characterizing the image;
Extracting a plurality of images from the image data based on the image metadata and determining frame information relating to a frame for arranging the extracted plurality of images;
Generating image display data including at least the frame information and the extracted image data;
Including an information processing method.
(18)
On the computer,
An image analysis function that analyzes image data and generates image metadata that is metadata related to a feature amount that characterizes the image;
A frame information determination function for extracting a plurality of images from the image data based on the image metadata and determining frame information regarding a frame for arranging the extracted plurality of images;
An image display data generation function for generating image display data including at least the frame information and the extracted image data;
A program to realize

DESCRIPTION OF SYMBOLS 10 Information processing apparatus 101 Movie data acquisition part 103 Movie analysis part 105 Audio | voice extraction part 107,1005,1101 Comic display conversion part 109 Frame image data generation part 111 Comic display data generation part 113 Comic display execution part 115 Display control part 117 Storage part 131 video data dividing unit 133, 1011 face detection unit 135 scene transition detection unit 137 camera work detection unit 139 audio analysis unit 141 video metadata generation unit 151 digest score calculation unit 153 thumbnail number determination unit 155 camera work metadata use unit 157 1031 Image size determination unit 159 Frame image arrangement unit 161 Effect image arrangement unit 163 Frame information generation unit 1001 Still image data acquisition unit 1003 Still image analysis unit 171, 1013 Human detection unit 173, 10 15 Object detection unit 175, 1017 Object detection unit 1019 Time stamp determination unit 1021 Still image metadata generation unit

Claims (18)

An image analysis unit that analyzes the image data and generates image metadata that is metadata relating to a feature amount that characterizes the image;
A frame information determination unit that extracts a plurality of images from the image data based on the image metadata and determines frame information regarding a frame for arranging the extracted plurality of images;
An image display data generation unit that generates image display data including at least the frame information and the extracted image data;
An information processing apparatus comprising: Time information is associated with the image data,
The information processing apparatus according to claim 1, wherein the frame information determination unit extracts the image data in time order based on the time information.   The information processing apparatus according to claim 1, wherein the frame information determination unit automatically adjusts the frame according to a size of the image, and performs layout conversion of the extracted image.   The information processing apparatus according to claim 2, wherein the frame information determination unit determines a size of the plurality of images arranged in each of the frames based on the generated image metadata. The frame information determination unit
A digest score calculation unit that calculates a digest score that is a value obtained by weighting and adding the feature amount to the image data based on the image metadata;
A thumbnail number determining unit that extracts thumbnail images to be arranged in the frame from the image data based on the digest score, and determines the number of thumbnail images;
An image size determining unit that determines the size of a thumbnail image arranged in a predetermined display area based on the digest score;
A frame image arrangement unit that arranges the thumbnail images of which the image size has been determined in the frame in a predetermined order;
A frame information generation unit that generates the frame information regarding the frame in which the thumbnail image is arranged;
The information processing apparatus according to claim 4, further comprising:   The information processing apparatus according to claim 5, wherein the thumbnail number determination unit further extracts a plurality of images for a predetermined time continuous with the extracted thumbnail images as animation images. The image metadata includes image metadata related to camera work when capturing the image,
The frame information determination unit further includes a camera work metadata utilization unit that determines the number of images extracted as the animation image by the thumbnail number determination unit based on image metadata related to the camera work. Item 6. The information processing device according to Item 5.   The camera work metadata using unit is configured to select the animation image from a plurality of images existing between the extracted thumbnail image and an image corresponding to a point in time when the integrated value of the camera work becomes a predetermined threshold value. The information processing apparatus according to claim 7, wherein the image is selected.   The information processing apparatus according to claim 8, wherein the camera work metadata utilization unit sets a playback speed of an animation composed of the thumbnail image and the animation image according to a magnitude of an integral value of the camera work. .   The said frame information determination part is further provided with the effect image arrangement | positioning part which arrange | positions at least any one of a balloon and a screen effect based on the said image metadata with respect to the flame | frame in which the said thumbnail image is arrange | positioned. Information processing device.   The information processing apparatus according to claim 10, wherein the effect image arrangement unit arranges at least one of the balloon and the screen effect using the image metadata regarding audio data included in the image data.   The information processing apparatus according to claim 10, wherein the effect image arrangement unit arranges the screen effect on the thumbnail image using image metadata related to camera work when the image is captured.   The image size determination unit enlarges a face area, which is an area where a face is displayed, into the thumbnail image with respect to an image in which the image metadata indicating that a face exists in the image exists. 5. The information processing apparatus according to 5. The image metadata includes at least one of image metadata indicating that a person is present in the image, image metadata relating to a subject existing in the image, and image metadata relating to a pre-registered object existing in the image. In addition,
The effect image arrangement unit includes image metadata indicating that a person is present in the image, image metadata relating to a subject existing in the image, or image metadata relating to a pre-registered object existing in the image. The information processing apparatus according to claim 10, wherein the screen effect is arranged based on the information.   The effect image arrangement unit is configured to detect an area in which the predetermined object is detected based on a relative positional relationship between a centroid of an area in which the predetermined object is detected in the thumbnail image and a centroid of the predetermined display area. The information processing apparatus according to claim 10, wherein the screen effect is arranged in an area other than the area.   The information processing apparatus according to claim 13, wherein when there are a plurality of the face areas in the image, the image size determination unit performs an enlargement process so as to include all the face areas to form the thumbnail image. . Analyzing image data and generating image metadata, which is metadata relating to a feature amount characterizing the image;
Extracting a plurality of images from the image data based on the image metadata and determining frame information relating to a frame for arranging the extracted plurality of images;
Generating image display data including at least the frame information and the extracted image data;
Including an information processing method. On the computer,
An image analysis function that analyzes image data and generates image metadata that is metadata related to a feature amount that characterizes the image;
A frame information determination function for extracting a plurality of images from the image data based on the image metadata and determining frame information regarding a frame for arranging the extracted plurality of images;
An image display data generation function for generating image display data including at least the frame information and the extracted image data;
A program to realize

Images