JP5598159B2 - Image processing apparatus, imaging system, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an imaging system, an image processing method, and a program.

2. Description of the Related Art Conventionally, a method for classifying an image by comparing the characteristics of an image such as a photograph taken with a digital camera or the like with the characteristics of a learning image classified in advance by category is known. For example, Patent Document 1 discloses a method for classifying images based on image feature amounts.
Patent Document 1 JP 2009-009402 A

  Various subjects are captured in the moving image data captured by the digital video camera or the digital camera, and the characteristics of the captured subject change with progress. Therefore, it has been difficult to automatically select a characteristic scene from moving image data.

  In order to solve the above-described problem, in the first aspect of the present invention, an image processing apparatus that selects a feature scene from input moving image data, the moving image data is analyzed, and the moving image data is A category specifying unit for specifying which category of a plurality of predetermined categories for identifying image contents, and moving image data among time-series data of a plurality of feature amounts extracted from moving image data There is provided an image processing apparatus comprising: a selection unit that selects a feature scene included in moving image data based on time-series data of feature amounts associated with a category to which.

  It should be noted that the above summary of the invention does not enumerate all the necessary features of the present invention. In addition, a sub-combination of these feature groups can also be an invention.

2 shows a configuration example of an image processing apparatus 100 according to the present embodiment. 2 shows an operation flow of the image processing apparatus 100 according to the present embodiment. An example of the moving image data 210 which concerns on this embodiment is shown. An example of the time series data of the category feature amount corresponding to the moving image data 210 according to the present embodiment is shown. An example of time-series data of scene feature amounts corresponding to the moving image data 210 according to the present embodiment is shown. The modification of the image processing apparatus 100 which concerns on this embodiment is shown. 2 shows an exemplary hardware configuration of a computer 1900 according to the present embodiment.

  Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. In addition, not all the combinations of features described in the embodiments are essential for the solving means of the invention.

  FIG. 1 shows a configuration example of an image processing apparatus 100 according to the present embodiment. The image processing apparatus 100 selects a feature scene from the input moving image data. The image processing apparatus 100 may select a feature scene from moving image data captured by a digital video camera, a digital camera, or the like, and record it in association with the moving image data, and displays the feature scene of the moving image data as a thumbnail. May be. Here, the feature scene in the present embodiment refers to an impressive scene in which the content of the moving image data can be grasped among the scenes captured in the moving image data. The feature scene may be a still image or short moving image data.

  For example, a feature scene is a scene in which a child is running in moving image data capturing an athletic meet, a scene in which a goal is scored in moving image data capturing a soccer game, or sightseeing in moving image data capturing a family trip. It may be a scene determined to be impressive by the user, such as a family gathering scene at a spot. The image processing apparatus 100 includes a category specifying unit 110, a selection unit 120, an extraction unit 130, and a processing unit 140.

  The category identifying unit 110 analyzes the moving image data and identifies which category the moving image data belongs to among a plurality of predetermined categories for identifying image contents. The category specifying unit 110 may specify which category of a plurality of categories for identifying in which event the moving image data is a moving image captured.

  Moreover, the category specifying unit 110 identifies which category the moving image data to be analyzed belongs to. For example, the category specifying unit 110 includes events (events) taken by users such as “cherry blossom viewing”, “seaside”, “festival”, “wedding”, etc., as moving image data belonging to a predetermined category. To identify. The category specifying unit 110 includes a plurality (n) of discriminators 115 and a specifier 117.

  Each of the plurality of discriminators 115 determines whether the moving image data belongs to a category associated with each discriminator 115 based on one or more category feature amounts extracted from the moving image data. The discriminator 115 may extract a feature amount corresponding to a corresponding category among a plurality of predetermined categories. For example, the category specifying unit 110 has the same number of discriminators 115 as the number of predetermined categories, and each discriminator 115 has a one-to-one correspondence with each category and features according to the respective categories. Extract each amount. The discriminator 115 may extract one or more feature amounts corresponding to one category. Here, the feature amount corresponding to the category is set as the category feature amount.

  The discriminator 115 may decompose the input moving image data into a plurality of still image data of frames forming a moving image, and extract category feature amounts of the plurality of still image data, respectively. Here, the discriminator 115 may extract a category feature amount of one or more still image data from among a plurality of still image data and use it as a category feature amount of the input moving image data.

  The discriminator 115 may use information corresponding to the number of edges included in one or more still image data as the category feature amount. Specifically, the discriminator 115 divides one still image data area into a plurality of areas each having a predetermined number of pixels, and then determines edges included in some or all of the plurality of divided areas. You may accumulate numbers. Note that an edge is a portion where luminance or chromaticity changes by a predetermined amount or more between adjacent pixels.

  The discriminator 115 may calculate the category feature amount by digitizing color information of one or more still image data. For example, the discriminator 115 converts a color space of one still image data into an HSV color space, and then generates a histogram based on the occurrence frequency of each color constituting the color space. The discriminator 115 may use numerical information on the occurrence frequency of each color as a category feature amount.

  The discriminator 115 may calculate the category feature amount by digitizing luminance information of one or more still image data. For example, the discriminator 115 uses the average and variance of the luminance of one still image data as the category feature amount. For example, the discriminator 115 may convert the color space of one still image data into the HSV color space and then use the energy and variance in the converted color space as the category feature amount.

  The discriminator 115 may calculate a category feature amount based on meta information included in one or more still image data. Specifically, when the input moving image data is moving image data generated in conformity with MPEG-7, information about the edge and the color information indicated in the video signal feature descriptor of the moving image data are displayed. Information on occurrence frequency, information on luminance, and the like may be used as category feature amounts.

  The discriminator 115 may extract the category feature amount for each of a plurality of still image data forming the input moving image data, and extract the time series data of the category feature amount. Alternatively, the discriminator 115 may extract a category feature amount of still image data that is a part of moving image data.

  The discriminator 115 identifies the category of each still image data based on the category feature amount extracted for each still image data. The discriminator 115 may specify a category using a category feature quantity vector having a plurality of category feature quantities as components. For example, the discriminator 115 uses a category feature amount such as a value based on the chromaticity of the image and a value based on the luminance of the image as a vector component.

  Each discriminator 115 compares the category feature amount with a predetermined reference amount, and determines whether the moving image data belongs to the category corresponding to the discriminator 115 based on the difference between the category feature amount and the reference amount. You can do it. Here, the predetermined reference amount may be set for each category associated with each discriminator 115. In addition, when the classifier 115 classifies a category using the category feature vector, the classifier 115 compares the category with a predetermined reference vector to determine whether the moving image data belongs to the corresponding category. You can do it. In this case, the reference vector may be a vector having a reference amount for each category feature amount as a component.

  Each discriminator 115 may calculate a distance between the category feature vector and the reference vector, and may determine whether or not each belongs to a category corresponding to the discriminator 115 based on the calculated distance. For example, the discriminator 115 discriminates that the input still image data belongs to the category associated with the discriminator 115 when the category feature vector is within a predetermined distance from the reference vector. .

  Further, the discriminator 115, when the category feature amount vector of some still image data among a plurality of still image data forming the input moving image data is within a predetermined distance range from the reference vector May determine that the partial still image data belongs to the category associated with the classifier 115. When each discriminator 115 determines that it belongs to the corresponding category, the discriminator 115 may send information of the determination result indicating that it belongs to the category to the specifying unit 117. Here, each discriminator 115 may send information about the distance from the reference vector to the identifying unit 117 in addition to the determination result that the classifying unit 115 belongs to the category.

  The identifying unit 117 identifies the category to which the input moving image data belongs based on the determination result information received from the plurality of identifying units 115. For example, when the identifying unit 117 receives a determination result indicating that it belongs to a corresponding category from one discriminator 115 for a plurality of still image data forming the input moving image data, the input moving image data is It identifies that it belongs to the category associated with the discriminator 115.

  Further, when the identification device 117 receives a determination result indicating that a part of a plurality of still image data belongs to a corresponding category from one discriminator 115, the identifying device 117 converts a part of the input moving image data to the discriminator 115. You may specify the category associated with. In this case, when the identifying unit 117 receives a determination result indicating that other still image data belongs to the corresponding category from the other identifying unit 115, the identifying unit 117 determines the region of the input moving image data. It may be divided into two or more and specified as belonging to a category associated with each discriminator 115 that has transmitted discrimination results for two or more regions.

  Further, when the identification device 117 receives the determination result indicating that the same still image data belongs to the corresponding category from two or more identification devices 115, the identification device 117 responds to the distance information from the reference vector received from each identification device 115. Thus, the category of still image data may be specified. For example, the identifying unit 117 may identify the category associated with the classifier 115 that has transmitted the result having the shortest distance from the reference vector among the plurality of discrimination results as the category of the still image data. The specifying unit 117 may specify a category for each still image data forming the input moving image data, and specify a category of the moving image data. The identifying unit 117 sends information on the identified category to the selection unit 120.

  The selection unit 120 is included in the moving image data based on the time-series data of feature amounts associated with the category to which the moving image data belongs among the time-series data of the plurality of feature amounts extracted from the input moving image data. Select the feature scene to be displayed. The selection unit 120 may select, as a feature scene, a portion of moving image data in which time-series data of feature amounts satisfies a predetermined condition.

  When selecting a feature scene included in moving image data belonging to one category, the selection unit 120 has time-series data of feature amounts different from the plurality of category feature amounts used to specify the category of moving image data. A feature scene may be selected based on Accordingly, the selection unit 120 can select a feature scene of the moving image data based on the feature corresponding to the category to which the moving image data belongs. Here, a feature amount corresponding to the feature scene is set as a scene feature amount. The selection unit 120 includes a feature detector 125 and a feature selector 127.

  The feature detector 125 is provided for each category type, and extracts one or more scene feature amounts corresponding to each category. The feature detector 125 may extract a scene feature amount corresponding to a predetermined feature scene.

  For example, when the category specifying unit 110 specifies moving image data as a “cherry-blossom viewing” category, one feature detector 125 corresponding to the “cherry-blossom viewing” category uses a feature scene such as “a scene in which many flowers are captured”. The feature quantity corresponding to is extracted. In this case, the feature detector 125 may use “flower color”, “flower shape”, and the like as feature amounts. In the example in the figure, feature detectors 125 are provided in the selection unit 120 according to the number of n discriminators 115, and respectively output scene feature amounts corresponding to n categories.

  Similar to the discriminator 115, the feature detector 125 may extract scene feature amounts of a plurality of still image data forming a moving image from the input moving image data. Here, the feature detector 125 may extract a scene feature amount of still image data in a part of a moving image from among a plurality of still image data, and output the scene feature amount in time series.

  The feature detector 125 may use information corresponding to the number of edges included in one or more still image data as the scene feature amount. The feature detector 125 may calculate the scene feature amount by digitizing color information of one or more still image data. The feature detector 125 may use the occurrence frequency of each color as a scene feature amount.

  Further, the feature detector 125 may calculate the scene feature amount by digitizing luminance information of one or more still image data. As an example, the feature detector 125 may convert the color space of one still image data into an HSV color space, and then use the energy and dispersion in the converted color space as scene feature amounts. The feature detector 125 may calculate a scene feature amount based on meta information included in one or more still image data.

  Further, the feature detector 125 may extract audio data that is output at the same time that one or more still image data is displayed, and calculate a scene feature amount. For example, the feature detector 125 may use the volume of audio data output together with the display of one still image data as the scene feature amount. Instead, the feature detector 125 may filter the audio data and use the volume of the high-pitched sound portion or the low-pitched sound portion as the scene feature amount.

  Further, the feature detector 125 may extract audio data included in the moving image data, convert the extracted audio data into text data, and calculate a scene feature amount. For example, the feature detector 125 converts the content spoken in the moving image data into text, and calculates a feature amount based on a matching amount with a predetermined keyword.

  The feature detector 125 detects a feature scene based on the scene feature amount extracted for each still image data. The feature detector 125 may detect a feature scene using a scene feature quantity vector having a plurality of scene feature quantities as components. For example, the feature detector 125 uses a value based on a change in a specific color included in a specific area of the image, a value based on a change in luminance of the image, or the like as a vector component.

  The feature detector 125 detects, as a feature scene, a portion of moving image data in which time-series data of feature amounts associated with a category to which the moving image data belongs satisfies a predetermined condition. For example, the feature detector 125 may detect, as a feature scene, a still image corresponding to data having the largest feature amount among the extracted time series data of scene feature amounts. Instead of this, the feature detector 125 may detect a plurality of still images within a predetermined number of frames from the largest feature amount in the time series data as feature scenes.

  Instead, the feature detector 125 may detect one or more still images having a value larger than a predetermined reference value in the time series data as a feature scene. Here, when there is no still image having a value larger than the reference value, the feature detector 125 may detect the feature scene by lowering the reference value. The feature detector 125 may decrease the reference value until a feature scene is detected.

  Instead, the feature detector 125 may detect a still image corresponding to data having the smallest feature amount among the time-series data of the selected scene feature amounts as a feature scene. Instead, the feature detector 125 may detect a plurality of still images within a predetermined number of frames from the smallest feature amount in the time series data as feature scenes. Instead, the feature detector 125 may detect one or more still images that are smaller than a predetermined reference value in the time series data as a feature scene. The feature detector 125 sends the detected feature scene information to the feature selector 127.

  The feature selector 127 detects the feature scene information detected by the feature detector 125 that detects the feature scene corresponding to the category identified by the identifier 117 from the plurality of feature scene information received from the plurality of feature detectors 125. select. Here, when the specifying unit 117 specifies a plurality of categories, the feature selector 127 may select information on the feature scene detected by the feature detector 125 corresponding to each specified category. The feature selector 127 sends information on the selected feature scene to the extraction unit 130.

  The extraction unit 130 receives feature scene information and extracts feature scenes included in the moving image data. Thereby, the extraction unit 130 can extract a feature scene corresponding to the specified category from the input moving image data. The extracting unit 130 may extract the feature scene by designating the position of the feature scene included in the moving image data in a time stamp or still image arrangement order. The extraction unit 130 may send information on the position of the feature scene included in the moving image data to the processing unit 140.

  Based on the information on the position of the feature scene received from the extraction unit 130, the processing unit 140 is information indicating that it is a feature scene with respect to still image data corresponding to the position of the feature scene included in the input moving image data. The moving image data is processed so as to be embedded. The processing unit 140 outputs moving image data having feature scene information.

  FIG. 2 shows an operation flow of the image processing apparatus 100 according to the present embodiment. The image processing apparatus 100 inputs the input moving image data to the category specifying unit 110 and the selection unit 120 to calculate a category feature amount and a scene feature amount. The image processing apparatus 100 may execute the process of calculating the category feature amount and the scene feature amount in parallel.

  The plurality of classifiers 115 of the category specifying unit 110 respectively input moving image data, and extract a plurality of category feature amounts corresponding to the plurality of classifiers 115 from a plurality of still image data forming a moving image (S500). ). Each of the plurality of classifiers 115 obtains one or more category feature amounts corresponding to one predetermined category such as “cherry blossom viewing”, “seaside”, “festival”, “wedding” from a plurality of still image data. Extract.

  Next, the category specifying unit 110 specifies the category of the input moving image data (S510). The plurality of classifiers 115 determine whether or not the plurality of still image data belong to the category corresponding to each classifier 115 based on the plurality of category feature amounts extracted. Here, each discriminator 115 may discriminate by comparing a plurality of category feature amounts of still image data with a category reference amount.

  For example, the discriminator 115 extracts a category feature amount from a plurality of moving image data to be specified in advance in a corresponding category, and totals the range of the reference amount and / or the statistical distribution of the category feature amount. May be. The category of the input moving image data may be discriminated based on being included in the range of the distribution of the category feature amount thus aggregated. The discriminator 115 that has determined that the input still image data belongs to the corresponding category among the plurality of discriminators 115 sends the discrimination result to the identifying unit 117.

  The identifying unit 117 identifies the category of the input moving image data based on the determination result received from the identifying unit 115. The identifier 117 sends information on the identified category to the feature selector.

  On the other hand, the selection unit 120 selects a feature scene included in the moving image data. The plurality of feature detectors 125 of the selection unit 120 respectively input moving image data, and extract a plurality of scene feature amounts corresponding to the plurality of feature detectors 125 from a plurality of still image data forming a moving image. (S520). Each of the plurality of feature detectors 125 corresponds to one predetermined category specified by the category specifying unit 110 such as “cherry blossom viewing”, “seaside”, “festival”, “wedding”, etc. from a plurality of still image data. One or more scene feature amounts are extracted.

  Next, each of the plurality of feature detectors 125 detects a feature scene from time-series data of a plurality of scene feature amounts in order to select a feature scene of the input moving image data. The plurality of feature detectors 125 send the detected feature scenes to the feature selector 127, respectively.

  The feature selector 127 detects the feature scene information detected by the feature detector 125 that detects the feature scene corresponding to the category identified by the identifier 117 from the plurality of feature scene information received from the plurality of feature detectors 125. Select (S530). For example, when the identifying unit 117 identifies the input moving image as the “seaside” category, the feature selector 127 detects the “seaside” feature scene by extracting the “seaside” scene feature amount from the input moving image. The feature scene information received from the feature detector 125 is selected. The feature selector 127 sends information on the selected feature scene to the extraction unit 130.

  The extraction unit 130 receives the feature scene information and extracts the feature scene included in the input moving image data (S540). The extraction unit 130 sends the information on the position of the feature scene included in the moving image data to the processing unit 140, and the processing unit 140 embeds the information on the feature scene in the input moving image data. With the operation flow of the image processing apparatus 100 according to the present embodiment, the image processing apparatus 100 can extract a feature scene of moving image data.

  That is, for example, in the case of moving image data capturing sports, the image processing apparatus 100 extracts feature scenes using feature amounts such as spectator excitement and player movement, and in the case of moving image data capturing a pet, A feature scene can be extracted from a scene group using a feature amount corresponding to a category, such as extracting a feature scene using a smile of the owner and a pet gesture as a feature amount. Therefore, the image processing apparatus 100 can automatically select a characteristic scene from the moving image data.

  Here, the extraction unit 130 may be a still image generation unit that generates a still image representing the selected feature scene. For example, when the selected feature scene is a plurality of still images, the extraction unit 130 extracts one still image from the plurality of still images. In this case, the extraction unit 130 may extract a still image corresponding to the largest feature amount or the smallest feature amount among the plurality of still images. When there are a plurality of pieces of moving image data to be input, the extraction unit 130 may generate a still image that represents a feature scene corresponding to the plurality of pieces of moving image data. Accordingly, the extraction unit 130 can display thumbnails as still images representing a plurality of moving image data.

  In the above-described embodiments, the example in which the selection unit 120 selects one feature scene with respect to the input moving image data has been described. Instead, the selection unit 120 selects a plurality of feature scenes. Also good. For example, the feature selector 127 selects still image data indicating an extreme value as a feature scene in addition to the maximum value of the time series data of the scene feature amount.

  When selecting a plurality of feature scenes, the selection unit 120 may rank each of the plurality of feature scenes based on the scene feature amount. As a result, the image processing apparatus 100 can extract a plurality of feature scenes from the input moving image data while ranking them. The extraction unit 130 may display thumbnails according to the ranking. As a result, a plurality of characteristic scenes of moving image data can be automatically selected and displayed in ranking.

  FIG. 3 shows an example of the moving image data 210 according to the present embodiment. The moving image data 210 may be formed from a plurality of still image data 215 arranged in time series. In the example in the drawing, the moving image data 210 is formed from 24 still image data 215.

  FIG. 4 shows an example of time series data of category feature values corresponding to the moving image data 210 according to the present embodiment. The example in the figure shows category feature amount time-series data 302, time-series data 304, and time-series data 306 corresponding to three moving image data 210. If the video of the same category is not captured in the middle of the moving image data and the video of the same category is captured, if the still image data 215 forming the moving image data 210 is used, The amount is a value within a substantially constant range as in the example in the figure.

  The discriminator 115 that identifies the “seaside” category may identify whether or not the time-series data belongs to the “seaside” category based on a predetermined distance from the “seaside” reference vector. For example, in the case of “seaside” moving image data, category feature amounts are extracted by the color of the sea, the color of the beach, the color of the rocks forming the coast, and the like. Here, a category feature amount may be extracted from a plurality of videos to be categorized as “seaside” and a statistical distribution range of the reference amount and the category feature amount may be aggregated in advance. From the distribution of the category feature amount extracted in this way, time-series data included in a predetermined reference value range may be specified as the “seaside” category.

  Here, in advance, the manufacturer of the image processing apparatus 100 selects a feature amount as a category feature amount from a lot of “seaside” moving image data by machine learning or manual optimization, and sets a category reference vector. May be. Further, the user may be optimized by learning in accordance with the change of the feature scene selected by the image processing apparatus 100 and its category.

  In the case where the example in the figure is the extraction result of the discriminator 115 that extracts with such category feature amount, the discriminator 115 indicates that the time series data 302 and 304 included in the distance range close to the reference vector ”Category. Here, since the distance of each time series data is the number of still image data, the discriminator 115 may specify using the average value of the distance. In the example in the figure, the example in which the same number of data as the still image data 215 forming the moving image data 210 is extracted as the time series data of the category feature amount has been described. A feature amount may be extracted. As a result, the category specifying unit 110 can execute the process of specifying the category at high speed.

  For example, in the case of a category such as “cherry-blossom viewing”, the category feature amount of “cherry-blossom viewing” may be extracted by paying attention to the color generated in the moving image data, similarly to the “seaside” category. However, the discriminator 115 that identifies the category “Seaside” extracts the color and the like generated in the “Seaside” video as a feature amount, so that moving image data such as “Hanami” is, for example, the time-series data 306. Thus, the “seaside” is distributed at an unspecified distance. Thereby, the discriminator 115 can specify a predetermined category from the time series data of the category feature amount.

  FIG. 5 shows an example of time-series data of scene feature amounts corresponding to the moving image data 210 according to the present embodiment. The time series data 402 and the time series data 404 indicate time series data of scene feature amounts corresponding to the time series data 302 and the time series data 304 in FIG. Since the scene feature amount extracts a characteristic scene from the moving image data, the scene feature amount does not have a substantially constant value in time series unlike the category feature amount.

  Here, the scene feature amount time-series data is an extraction result of the feature detector 125 that extracts the scene feature amount associated with the category specified by the category specifying unit 110. For example, the feature detector 125 may detect an area where the sea color is generated, an area where the sand beach color is generated, an area where the rock color is generated, and the sound of the waves. Extract scene features such as volume, child face recognition, and child voice volume. Here, in advance, the manufacturer of the image processing apparatus 100 may select a scene feature amount from a lot of “seaside” moving image data by machine learning or manual optimization.

  For example, when an image of a seaside landscape is captured, the area where the color of the sea is generated, the area where the color of the sandy beach is generated, and the area where the color of the rock is generated may not change, and the feature detector 125 extracts the feature to be extracted. Even if the numerical value as a quantity is large, time series data that does not change much is detected. Therefore, for example, the feature detector 125 detects the feature scene by changing the time-series data as in the time-series data 402 in the figure by adding the feature quantity of the volume of the wave sound as the scene feature quantity. be able to.

  Here, even if the sound of the waves is loud, if the sea is not imaged, the feature amount due to the area where the color of the sea is generated decreases, so such a scene is not extracted. Therefore, the feature detector 125 can extract still image data A of a dynamic scene in which the sea is imaged and a large wave is hit against the seaside as a feature scene. Still image data A and a predetermined number of frames before and after the data may be used as the feature scene. As a result, a scene in which a large wave is hit against the coast can be extracted as a feature scene.

  Further, even when moving image data specified in the same “seaside” category is used, when a child's face is mainly imaged, the scene feature amount focusing on the color of the sea is calculated to be low. Even in this case, the feature detector 125 is picked up as shown in the time-series data 404 in the figure by extracting the scene feature amount including the child face recognition, the voice volume of the child, and the like. It is possible to detect the feature scene by changing the feature amount according to the feature of the child. As a result, for example, still image data B, which is a scene where a child playing at the beach laughs out loud, can be extracted as a feature scene.

  FIG. 6 shows a modification of the image processing apparatus 100 according to the present embodiment. In the image processing apparatus 100 of the present modification, the same reference numerals are given to the substantially same operations as those of the image processing apparatus 100 according to the present embodiment shown in FIG. After specifying the category of the input moving image data, the image processing apparatus 100 extracts a scene feature amount corresponding to the specified category and selects a feature scene.

  The specifying unit 117 of the category specifying unit 110 specifies the category of the input moving image data. The identifying unit 117 sends the identified category information and the input moving image data to the feature selecting unit 127 of the selecting unit 120. The feature selector 127 selects the feature detector 125 that extracts a scene feature amount corresponding to the received category, and inputs the received moving image data. The selected feature detector 125 extracts scene feature amount time-series data from the input moving image data, and detects a feature scene. The feature detector 125 sends the detected feature scene information and the input moving image data to the extraction unit 130.

  The extraction unit 130 extracts feature scenes from the received moving image data and feature scene information. The extraction unit 130 may extract feature scenes from the received moving image data and display them as thumbnails. The extraction unit 130 sends the input moving image data and feature scene information to the processing unit 140. The processing unit 140 processes and embeds the received feature scene information into the received moving image data. The processing unit 140 outputs the processed moving image data.

  The feature scene of the input moving image data can be selected by the modification of the image processing apparatus 100 according to the present embodiment. In addition, in this modified example, the scene feature amount is extracted after specifying the category of the input moving image data. Therefore, the feature detector 125 corresponding to the specified category can be appropriately selected. A feature scene can be selected without operating an irrelevant feature detector 125.

  The image processing apparatus 100 according to the present embodiment has been described by way of example in which moving image data is input and a feature scene is selected. Furthermore, the image processing apparatus 100 may be mounted on an imaging system. The imaging system corresponds to an imaging unit that generates moving image data obtained by imaging a subject, an image processing device 100 that selects a feature scene from moving image data captured by the imaging unit, and moving image data captured by the imaging unit. In addition, a recording unit that records information representing the feature scene selected by the image processing apparatus on a recording medium may be provided. Thus, the imaging system can select a feature scene of the captured moving image data while imaging the subject.

  FIG. 7 shows an example of the hardware configuration of a computer 1900 according to this embodiment. A computer 1900 according to this embodiment is connected to a CPU peripheral unit having a CPU 2000, a RAM 2020, a graphic controller 2075, and a display device 2080 that are connected to each other by a host controller 2082, and to the host controller 2082 by an input / output controller 2084. An input / output unit having a communication interface 2030, a hard disk drive 2040, and a DVD drive 2060; a legacy input / output unit having a ROM 2010, a flexible disk drive 2050, and an input / output chip 2070 connected to the input / output controller 2084; Is provided.

  The host controller 2082 connects the RAM 2020 to the CPU 2000 and the graphic controller 2075 that access the RAM 2020 at a high transfer rate. The CPU 2000 operates based on programs stored in the ROM 2010 and the RAM 2020 and controls each unit. The graphic controller 2075 acquires image data generated by the CPU 2000 or the like on a frame buffer provided in the RAM 2020 and displays it on the display device 2080. Instead of this, the graphic controller 2075 may include a frame buffer for storing image data generated by the CPU 2000 or the like.

  The input / output controller 2084 connects the host controller 2082 to the communication interface 2030, the hard disk drive 2040, and the DVD drive 2060, which are relatively high-speed input / output devices. The communication interface 2030 communicates with other devices via a network. The hard disk drive 2040 stores programs and data used by the CPU 2000 in the computer 1900. The DVD drive 2060 reads a program or data from the DVD-ROM 2095 and provides it to the hard disk drive 2040 via the RAM 2020.

  The input / output controller 2084 is connected to the ROM 2010, the flexible disk drive 2050, and the relatively low-speed input / output device of the input / output chip 2070. The ROM 2010 stores a boot program that the computer 1900 executes at startup and / or a program that depends on the hardware of the computer 1900. The flexible disk drive 2050 reads a program or data from the flexible disk 2090 and provides it to the hard disk drive 2040 via the RAM 2020. The input / output chip 2070 connects the flexible disk drive 2050 to the input / output controller 2084 and inputs / outputs various input / output devices via, for example, a parallel port, a serial port, a keyboard port, a mouse port, and the like. Connect to controller 2084.

  A program provided to the hard disk drive 2040 via the RAM 2020 is stored in a recording medium such as the flexible disk 2090, the DVD-ROM 2095, or an IC card and provided by the user. The program is read from the recording medium, installed in the hard disk drive 2040 in the computer 1900 via the RAM 2020, and executed by the CPU 2000.

  A program installed on the computer 1900 and causing the computer 1900 to function as the image processing apparatus 100 includes a category specifying module, a selection module, an extraction module, and a processing module. These programs or modules work on the CPU 2000 or the like to cause the computer 1900 to function as the category specifying unit 110, the selection unit 120, the extraction unit 130, and the processing unit 140, respectively.

  The information processing described in these programs is read by the computer 1900, whereby the category specifying unit 110, the selecting unit 120, and the extracting unit, which are specific means in which the software and the various hardware resources described above cooperate. 130 and function as the processing unit 140. And the specific image processing apparatus 100 according to the intended purpose is constructed | assembled by implement | achieving the calculation or processing of the information according to the intended purpose of the computer 1900 in this embodiment by these specific means.

  As an example, when communication is performed between the computer 1900 and an external device or the like, the CPU 2000 executes a communication program loaded on the RAM 2020 and executes a communication interface based on the processing content described in the communication program. A communication process is instructed to 2030. Under the control of the CPU 2000, the communication interface 2030 reads transmission data stored in a transmission buffer area provided on a storage device such as the RAM 2020, the hard disk drive 2040, the flexible disk 2090, or the DVD-ROM 2095, and sends it to the network. The reception data transmitted or received from the network is written into a reception buffer area or the like provided on the storage device. As described above, the communication interface 2030 may transfer transmission / reception data to / from the storage device by a DMA (direct memory access) method. Instead, the CPU 2000 transfers the storage device or the communication interface 2030 as a transfer source. The transmission / reception data may be transferred by reading the data from the data and writing the data to the communication interface 2030 or the storage device of the transfer destination.

  In addition, the CPU 2000 includes all or necessary portions of files or databases stored in an external storage device such as the hard disk drive 2040, DVD drive 2060 (DVD-ROM 2095), and flexible disk drive 2050 (flexible disk 2090). Are read into the RAM 2020 by DMA transfer or the like, and various processes are performed on the data on the RAM 2020. Then, CPU 2000 writes the processed data back to the external storage device by DMA transfer or the like. In such processing, since the RAM 2020 can be regarded as temporarily holding the contents of the external storage device, in the present embodiment, the RAM 2020 and the external storage device are collectively referred to as a memory, a storage unit, or a storage device. Various types of information such as various programs, data, tables, and databases in the present embodiment are stored on such a storage device and are subjected to information processing. Note that the CPU 2000 can also store a part of the RAM 2020 in the cache memory and perform reading and writing on the cache memory. Even in such a form, the cache memory bears a part of the function of the RAM 2020. Therefore, in the present embodiment, the cache memory is also included in the RAM 2020, the memory, and / or the storage device unless otherwise indicated. To do.

  In addition, the CPU 2000 performs various operations, such as various operations, information processing, condition determination, information search / replacement, etc., described in the present embodiment, specified for the data read from the RAM 2020 by the instruction sequence of the program. Is written back to the RAM 2020. For example, when performing the condition determination, the CPU 2000 determines whether the various variables shown in the present embodiment satisfy the conditions such as large, small, above, below, equal, etc., compared to other variables or constants. When the condition is satisfied (or not satisfied), the program branches to a different instruction sequence or calls a subroutine.

  Further, the CPU 2000 can search for information stored in a file or database in the storage device. For example, in the case where a plurality of entries in which the attribute value of the second attribute is associated with the attribute value of the first attribute are stored in the storage device, the CPU 2000 displays the plurality of entries stored in the storage device. The entry that matches the condition in which the attribute value of the first attribute is specified is retrieved, and the attribute value of the second attribute that is stored in the entry is read, thereby associating with the first attribute that satisfies the predetermined condition The attribute value of the specified second attribute can be obtained.

  The program or module shown above may be stored in an external recording medium. As the recording medium, in addition to the flexible disk 2090 and the DVD-ROM 2095, an optical recording medium such as DVD or CD, a magneto-optical recording medium such as MO, a tape medium, a semiconductor memory such as an IC card, and the like can be used. Further, a storage device such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet may be used as a recording medium, and the program may be provided to the computer 1900 via the network.

  As mentioned above, although this invention was demonstrated using embodiment, the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

The order of execution of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before” or “prior to”. It should be noted that the output can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first”, “next”, etc. for convenience, it means that it is essential to carry out in this order. It is not a thing. According to the present application, the following configurations are also disclosed.
(Item 1)
An image processing apparatus for selecting a feature scene from input moving image data,
A category specifying unit that analyzes the moving image data and specifies which category of a plurality of predetermined categories for identifying the image content of the moving image data;
A feature scene included in the moving image data is selected based on time-series data of feature amounts associated with a category to which the moving image data belongs among time-series data of feature amounts extracted from the moving image data. A selection section to
An image processing apparatus comprising:
(Item 2)
The image processing apparatus selects a feature scene from the captured moving image data,
The category specifying unit according to Item 1, wherein the category specifying unit specifies which category of a plurality of categories for identifying in which event the moving image data is a moving image captured. Image processing device.
(Item 3)
The image processing apparatus according to item 1 or 2, wherein the category specifying unit specifies to which category the moving image data belongs based on one or more feature amounts extracted from the moving image data.
(Item 4)
When the selection unit selects a feature scene included in moving image data belonging to one category, a time series of feature amounts different from the plurality of feature amounts used for specifying the category of the moving image data 4. The image processing device according to item 3, wherein a feature scene is selected based on data.
(Item 5)
The selection unit is any one of items 1 to 4 for selecting, as a feature scene, a portion of moving image data in which time-series data of feature amounts associated with a category to which the moving image data belongs satisfy a predetermined condition. The image processing apparatus according to one item.
(Item 6)
6. The image processing device according to item 5, wherein when the plurality of feature scenes are selected, the selection unit ranks each of the plurality of feature scenes based on a feature amount.
(Item 7)
The image processing device according to any one of items 1 to 6, further comprising a still image generation unit that generates a still image representing the selected feature scene.
(Item 8)
An imaging system for capturing moving image data,
An imaging unit that generates moving image data obtained by imaging a subject;
The image processing apparatus according to claim 1, wherein a feature scene is selected from moving image data captured by the imaging unit.
A recording unit that records information representing the feature scene selected by the image processing apparatus on a recording medium in association with moving image data captured by the imaging unit;
An imaging system comprising:
(Item 9)
An image processing method for selecting a feature scene from input moving image data,
A category specifying step of analyzing the moving image data and specifying which of the plurality of predetermined categories for identifying the image content the moving image data belongs to;
A feature scene included in the moving image data is selected based on time-series data of feature amounts associated with a category to which the moving image data belongs among time-series data of feature amounts extracted from the moving image data. And a selection stage to
An image processing method comprising:
(Item 10)
A program for causing a computer to function as an image processing device that selects a feature scene from input moving image data,
The program causes the computer to
A category specifying unit that analyzes the moving image data and specifies which category of a plurality of predetermined categories for identifying the image content of the moving image data;
A feature scene included in the moving image data is selected based on time-series data of feature amounts associated with a category to which the moving image data belongs among time-series data of feature amounts extracted from the moving image data. A selection section to
Program to make it work.

100 image processing apparatus, 110 category specifying unit, 115 classifier, 117 specifying unit, 120 selecting unit, 125 feature detector, 127 feature selecting unit, 130 extracting unit, 140 processing unit, 210 moving image data, 215 still image data, 302, 304, 306, 402, 404 Time series data, 1900 computer, 2000 CPU, 2010 ROM, 2020 RAM, 2030 communication interface, 2040 hard disk drive, 2050 flexible disk drive, 2060 DVD drive, 2070 I / O chip, 2075 graphics Controller, 2080 display device, 2082 Host controller, 2084 I / O controller, 2090 flexible disk, 2095 DVD-ROM

Claims (9)

An image processing apparatus for selecting a feature scene from input moving image data,
By analyzing the moving picture data, the moving image data, and a category specifying unit for specifying whether belongs to the category of the multiple categories defined Me pre,
Based on the feature quantity the moving image data associated with the category to which belongs among the plurality of feature amounts extracted from the moving image data, a selection unit for selecting a feature scenes included in the moving image data,
Equipped with a,
An image processing apparatus in which a feature amount associated with a category to which the moving image data belongs is a color .   The selection unit sets the occurrence frequency of the color of the moving image data as a feature amount associated with a category to which the moving image data belongs.
  The image processing apparatus according to claim 1.   The selection unit sets the color area in the moving image data as a feature amount associated with a category to which the moving image data belongs.
  The image processing apparatus according to claim 1. The image processing apparatus selects a feature scene from the captured moving image data,
The category specifying unit, the moving image data, from the claims 1 to identify whether they fall in a category of a plurality of categories to identify whether the moving image captured in any event 3 The image processing apparatus according to any one of the above. The image according to any one of claims 1 to 4, wherein the category specifying unit specifies a category to which the moving image data belongs based on one or more feature amounts extracted from the moving image data. Processing equipment. The image processing apparatus according to any one of claims 1, further comprising a still image generation unit for generating a still image representative of said selected feature scenes further 5. An imaging system for capturing moving image data,
An imaging unit that generates moving image data obtained by imaging a subject;
The image processing apparatus according to any one of claims 1 to 6 , wherein a feature scene is selected from moving image data captured by the imaging unit.
A recording unit that records information representing the feature scene selected by the image processing apparatus on a recording medium in association with moving image data captured by the imaging unit;
An imaging system comprising: An image processing method for selecting a feature scene from input moving image data,
By analyzing the moving picture data, the moving image data, and category specific step of identifying whether belongs to the category of the multiple categories defined Me pre,
Based on the feature quantity the moving image data associated with the category to which belongs among the plurality of feature amounts extracted from the moving image data, a selecting step of selecting the feature scenes included in the moving image data,
Equipped with a,
An image processing method in which a feature amount associated with a category to which the moving image data belongs is a color . A program for causing a computer to function as an image processing device that selects a feature scene from input moving image data,
The program causes the computer to
By analyzing the moving picture data, the moving image data, and a category specifying unit for specifying whether belongs to the category of the multiple categories defined Me pre,
Based on the feature quantity the moving image data associated with the category to which belongs among the plurality of feature amounts extracted from the moving image data, a selection unit for selecting a feature scenes included in the moving image data,
With
Feature quantity the moving image data associated with the category belongs, the program to function in an image processing apparatus to be color.

Images