JP2014229092A - Image processing device, image processing method and program therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately extract a scene of importance from image data.SOLUTION: An image processing device comprises: a motion extraction unit that extracts motion information on an image to be inputted on the basis of the image; and a characteristic amount calculation unit that calculates the amount of a characteristic indicative of an important scene in which there is a motion in a subject within the image on the basis of the extracted motion information. Herein, the subject is an image of an image area having a relatively large motion among the image areas out of a plurality of image areas within the image, and information indicative of a direction of the motion of the subject is included in th motion information. The characteristic amount calculation unit calculates the amount of the characteristic on the basis of the direction of the motion indicated by the motion information.

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program thereof.

  In order to perform highlight reproduction or digest reproduction of video clips (moving image data) or video data, a technique for extracting a specific scene (for example, an important scene) from these image data is known (for example, Patent Documents). 1). In this patent document, a camera is provided with a camera motion sensor, a global motion is calculated to form a plurality of video segments, each segment is labeled according to a series of camera motion classes, and an important scene is determined from the labeled segments. A technique for extracting candidates is disclosed. This global motion is obtained by calculating camera work obtained from a camera motion sensor and video.

Special table 2009-539273 gazette

However, in the important scene candidate extraction method disclosed in Patent Document 1, it is necessary to provide a camera motion sensor in the camera.
Therefore, the present invention has been made in view of the above circumstances, and provides an image processing apparatus, an image processing method, and a program for accurately extracting an important scene from image data without using a special sensor. The purpose is to provide.

  One aspect of the present invention provides a motion extraction unit that extracts motion information of an image based on an input image, and an important scene in which a subject in the image has motion based on the extracted motion information. An image processing apparatus comprising: a feature amount calculation unit that calculates a feature amount to be indicated.

  Further, according to one embodiment of the present invention, a motion extraction procedure for extracting motion information of an image based on an input image, and an object in the image has motion based on the extracted motion information. It is an image processing method characterized by having a feature amount calculation procedure for calculating a feature amount indicating a scene.

  One embodiment of the present invention is a motion extraction procedure for extracting motion information of an image based on an image input to a computer included in the image processing apparatus, and the image based on the extracted motion information. This is a program for executing a feature amount calculation procedure for calculating a feature amount indicating an important scene in which a subject moves.

  According to the present invention, an important scene can be accurately extracted from image data.

1 is a schematic diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present invention. It is a schematic diagram which shows an example of the frame structure of the moving image of this embodiment, and the direction of the motion of the to-be-photographed object between frames. It is a schematic diagram which shows an example of the motion calculation result by the motion extraction part of this embodiment. It is a flowchart which shows an example of operation | movement of the image processing apparatus of this embodiment.

[Embodiment]
Hereinafter, an embodiment of an image processing apparatus 10 according to the present invention will be described with reference to the drawings. FIG. 1 is a schematic diagram showing a configuration of an image processing apparatus 10 according to an embodiment of the present invention.

  As shown in FIG. 1, the image processing apparatus 10 according to the present embodiment is realized as a function of a personal computer, for example, and outputs an important scene extracted from input images (processing target images). That is, the image processing apparatus 10 extracts important scenes included in the image data to be processed. Here, the important scene is an image of a scene that is viewed by a viewer or a photographer (in the following description, simply referred to as a viewer) of the image. Since the viewer pays attention to a temporal change in the image, that is, a portion having movement, the image processing apparatus 10 extracts a portion having movement in the image as a subject of the image, and this subject is characteristic. Scenes that move smoothly are extracted as important scenes. That is, the important scene is an image of a scene where the subject is moving among the images to be processed.

  The image processing apparatus 10 is not limited to being realized as a function of a personal computer. For example, the image processing apparatus 10 may be built in a mobile phone or a camera. In addition, the image here is not necessarily limited to a moving image, but in the following description, an example in which an input image (image to be processed) is a moving image will be described.

Hereinafter, the configuration of the image processing apparatus 10 will be described.
The image processing apparatus 10 includes a feature amount calculation unit 100, a discriminator generation unit 200, an important scene extraction unit 300, and a storage unit 400.
The storage unit 400 stores an identifier (identification information) that identifies an important scene. The discriminator is information for identifying an important scene based on a feature amount indicating an important scene (also referred to as an important scene feature amount h * (etch asterisk) in the following description). This important scene feature amount h * is information indicating the direction of movement of the subject between certain frames in the moving image to be processed. The direction of movement of the subject between frames of the moving image will be described with reference to FIG.

  FIG. 2 is a schematic diagram illustrating an example of a frame structure of a moving image and a direction of movement of a subject between frames according to the present embodiment. Hereinafter, an XY orthogonal coordinate system is set with the top left vertex of each frame as the origin, and the description will be made with reference to this XY orthogonal coordinate system. In this XY orthogonal coordinate system, the horizontal direction of each frame is taken as the X axis, and the vertical direction of each frame is taken as the Y axis. As shown in FIG. 2A, the moving image of the present embodiment includes a plurality of frames (for example, frames 1 to n) arranged in time series. This frame is composed of a plurality of pixels (for example, 1920 × 1080 pixels).

  Here, the motion of the image between the frames can be obtained by comparing the images of a plurality of frames (for example, two). As shown in FIG. 2B, a case where a soccer ball image is included in frames 1 and 4 will be described as a specific example. In this frame 1, an image of a soccer ball is displayed at a position Y1. In addition, an image of a soccer ball is displayed on the frame 4 at a position Y2. When the images of these two frames (frame 1 and frame 4) are compared, the position of the soccer ball image has moved from position Y1 to position Y2, that is, in the -Y direction. In this way, the motion of the soccer ball image can be obtained by comparing the images of the two frames. Here, the movement of the image is also referred to as an optical flow OF. That is, in the case of the example of FIG. 2B, the optical flow OF1 of the soccer ball can be obtained by comparing the images of the two frames. This optical flow OF has two types of values (the frame width direction (X-axis direction) component and the height direction (Y-axis direction) component) representing the motion of the image in the frame for each pixel constituting the frame. doing.

  Returning to FIG. 1, the description of the image processing apparatus 10 will be continued. The image processing apparatus 10 extracts an important scene through two processes, a training process and an identification process. Among these, the training process means that an important scene feature amount h * of an input image (for example, a moving image) is calculated, and a discriminator (identification information) based on the calculated important scene feature amount h * is stored in the storage unit 400. It is a process of memorizing. The image input in this training process is a training image. The image input in this identification process is an image to be processed. Here, the training process will be described first.

  The image processing apparatus 10 receives a plurality of training images for each moving image category in the training process. The image processing apparatus 10 calculates the important scene feature amount h * for each of the input training images. Here, the moving image category is information for classifying the moving image content based on the characteristics of the direction of the optical flow OF. More specifically, the moving image category is information that classifies the content of the moving image based on the degree of variation in the direction of the optical flow OF. Here, the degree of variation in the direction of the optical flow OF is, for example, when the direction of the optical flow OF is classified into 8 directions, the optical flow OF of a certain moving image has a class with each direction in the 8 directions as a class. The degree of frequency variation. As a specific example, the moving image category includes “a full view image of a stadium during a soccer game”, “a close-up image of a soccer individual technique”, and the like. The feature of the optical flow OF in the video category “entire view image of the stadium during a soccer game” is, for example, movement of a soccer ball moving toward one of the left and right goals. In this case, the player image moves less than the soccer ball image. Viewers who watch this video pay attention to the image of the soccer ball. That is, the viewer who has viewed this moving image recognizes an image area having a relatively large movement as an object, and pays attention to the image of the object. In other words, the subject here is an image of an image region that has a relatively large movement among the image regions among a plurality of image regions in the image. In the following description, the subject is also referred to as a foreground. In this case, the subject of interest (foreground) is a soccer ball. That is, in this example, the movement in which the subject of interest moves in the left-right direction (X-axis direction) of the screen is a feature of the optical flow OF of the moving image category “entire view image of the stadium during a soccer game”. . In other words, for example, the feature of the optical flow OF of the video category “entire view image of the stadium during a soccer game” is that the main direction of the optical flow OF is the left-right direction (X-axis direction) of the screen. is there.

  In addition, the feature of the optical flow OF in the moving image category “close-up image of a soccer individual technique” is, for example, a movement in which a soccer ball is lifted and moved up and down. In this case, the image of the person lifting is less moving than the image of the soccer ball. Viewers who watch this video pay attention to the image of the soccer ball. Therefore, in this case, the subject to be noticed, that is, the foreground is a soccer ball. That is, the movement of the noted subject (foreground) in the vertical direction (Y-axis direction) of the screen is a feature of the optical flow OF of the moving image category “close-up image of soccer individual technique”.

[Configuration of Image Processing Apparatus 10 (Training Process)]
Next, the configuration of the image processing apparatus 10 in the training process will be described. In this training process, a discriminator for training images is generated by the feature amount calculation unit 100 and the discriminator generation unit 200. The feature amount calculation unit 100 includes an optical flow extraction unit 110, a foreground optical flow extraction unit 120, and an important scene feature amount calculation unit 130. The feature amount calculation unit 100 extracts motion information of the image based on the input image, and generates a classifier of the image based on the extracted motion information.

  Specifically, the optical flow extraction unit 110 extracts the optical flow OF of the input training image. When the training image data is input, the optical flow extraction unit 110 samples the training image at a predetermined time interval. This predetermined time interval is, for example, every three frames as shown in FIG. In this case, when the training image data is input, the optical flow extraction unit 110 samples the training image every three frames.

  Next, the optical flow extraction unit 110 calculates the optical flow OF of the training image by the following equation (1) based on the two frames before and after the sampling.

  Here, (x, y) in Equation (1) represents pixel coordinates in the frame for which the optical flow OF has been calculated. Further, vx (x, y) and vy (x, y) in the formula (1) indicate optical flow components in the width direction (X-axis direction) and the height direction (Y-axis direction), respectively. The subscripts x and y of vx and vy are written in subscripts in the formula (1) and the following formulas. That is, the optical flow extraction unit 110 extracts the motion information of the image for each predetermined pixel that configures the image. Here, the predetermined pixel may be each pixel in the frame, or may be a pixel at a certain pixel interval L (for example, L = 5 pixels). Hereinafter, a case where the optical flow extraction unit 110 calculates the optical flow OF for each pixel in the frame will be described.

  The foreground optical flow extraction unit 120 extracts the foreground optical flow FOF from the training image optical flow OF extracted by the optical flow extraction unit 110. Specifically, the foreground optical flow extraction unit 120 calculates the average value v ￣ (buoy bar) and the standard deviation σ (sigma) of the optical flow OF for each frame by the equations (2) and (3). To do. Here, N is the number of optical flows OF calculated for each frame.

  Next, the foreground optical flow extraction unit 120 extracts the foreground optical flow FOF based on the calculated average value v ￣ (buoy bar) and standard deviation σ (sigma) of the optical flow OF. Specifically, the foreground optical flow extraction unit 120 subtracts the average value v ￣ (buoy bar) from the magnitude v (x, y) of the optical flow OF, that is, the residual (v (x, y y) -Foreground optical flow FOF is calculated based on -v￣ (buoy bar)). For example, if the foreground optical flow extraction unit 120 is larger than the standard deviation σ, the foreground optical flow OF (v (x, y)) is converted into the foreground optical flow FOF (v * (x, y)) (buoy asterisk X Y) (see equation (4)). Note that the subscript * (asterisk) of v is written in superscript in both the formula (4) and the following formulas.

  Next, the foreground optical flow extraction unit 120 quantizes the foreground optical flow FOF (v * (x, y)) in a plurality of directions, and calculates the frequency in each direction. Specifically, the foreground optical flow extraction unit 120 quantizes the direction of the non-zero foreground optical flow FOF (v * (x, y)) into eight directions, and generates a direction histogram (frequency for each direction) h. Calculate (see formulas (5) and (6)). Here, the direction θ (x, y) is a direction indicated by radians.

  That is, in the feature quantity calculation unit 100, the optical flow extraction unit 110 extracts motion information (optical flow OF) of the image based on the input image, and the foreground optical flow extraction unit 120 extracts the motion that has been extracted. Based on the information, motion information (foreground optical flow FOF) indicating the motion of the subject in the image is extracted.

  In addition, the direction histogram h indicates an image region having a relatively large movement within a certain frame. That is, the foreground optical flow extraction unit 120 can extract an image region having a relatively large motion within a certain frame as a foreground (subject) by calculating the direction histogram h.

  Next, the important scene feature amount calculation unit 130 cyclically rearranges the remaining components from the direction histogram h calculated by Equation (5) and Equation (6) as the first component having the highest frequency. The h * (etch asterisk) is calculated as an important scene feature amount. That is, the important scene feature quantity calculation unit 130 calculates a feature quantity indicating an important scene based on the extracted motion information indicating the motion of the subject. Here, the important scene feature amount calculation unit 130 calculates the feature amount indicating the important scene based on the direction of the motion having the highest appearance frequency among the motion directions indicated by the motion information extracted for each predetermined pixel. To do. For example, if the k = third component h3 of the direction histogram h is the largest in a certain frame, the important scene feature amount h * of this frame is expressed by the following equation (7).

  The important scene feature amount calculation unit 130 calculates the important scene feature amount h * in all the sampled frames. Next, the important scene feature quantity calculation unit 130 associates the calculated important scene feature quantity h * with the image category data indicating the category of the image and causes the storage unit 400 to store them. That is, the important scene feature quantity calculation unit 130 calculates the important scene feature quantity h * based on the extracted motion information. That is, the important scene feature quantity calculation unit 130 calculates the important scene feature quantity h * based on the information indicating the category (kind) of the input image and the extracted optical flow OF (motion information).

  As described above, the important scene feature quantity calculation unit 130 calculates the important scene feature quantity h * based on the direction of movement of the subject indicated by the foreground optical flow FOF. That is, the important scene feature quantity calculation unit 130 calculates the important scene feature quantity h * based on the direction of motion indicated by the motion information.

  Further, the important scene feature quantity calculation unit 130 calculates the important scene feature quantity h * based on the direction histogram h. The direction histogram h is information indicating the appearance frequency of the direction of motion indicated by the foreground optical flow FOF (motion information) extracted for each predetermined pixel. That is, the important scene feature quantity calculation unit 130 calculates the important scene feature quantity h * based on the appearance frequency of the motion direction indicated by the motion information extracted for each predetermined pixel.

  The discriminator generation unit 200 generates a discriminator (identification information) that identifies an important scene of an image. Specifically, the discriminator generation unit 200 discriminates based on the important scene feature quantity h * calculated by the important scene feature quantity calculation unit 130, input important scene data, and input image category data. Create a container. The important scene data is information indicating an important scene among the scenes of the training image input to the feature amount calculation unit 100. That is, the important scene data is information indicating a correct scene that the classifier should identify as an important scene among the scenes of the training image. The image category data is data indicating a category of training image data input to the feature amount calculation unit 100.

  The classifier generation unit 200 generates a classifier by, for example, a known machine learning method (for example, a classification method using bag-of-words and SVM).

  Further, the discriminator generation unit 200 can generate a discriminator for each category of the image. Here, the important scene feature amount h * may be different for each category of the image. Therefore, by preparing a discriminator for identifying an important scene for each category of an image, it is possible to prepare a discriminator with high discrimination accuracy adapted to each category. That is, for each image category, by preparing a discriminator that is generated based on the important scene feature quantity h * corresponding to this category, the image processing apparatus 10 can accurately identify the important scene.

[Configuration of Image Processing Device 10 (Identification Process)]
Next, the configuration of the image processing apparatus 10 in the identification process will be described. In this identification process, the important scene extraction unit 300 extracts an important scene from the processing target image based on the important scene feature amount h * of the training image stored in the storage unit 400 and the processing target image. To do. A specific configuration of the important scene extraction unit 300 will be described below.

  The important scene extraction unit 300 includes a feature amount calculation unit 320 corresponding to the above-described feature amount calculation unit 100, and an important scene determination unit 330. When the image data to be processed is input, the feature amount calculation unit 320 receives the optical flow FOF (v * (x, y)) of the foreground of the image to be processed in the same manner as the feature amount calculation unit 100 described above. Extract.

  In addition, the feature amount calculation unit 320, based on the extracted foreground optical flow FOF (v * (x, y)) of the processing target image, the important scene feature amount h * (etch asterisk) of the processing target image. Is calculated.

  The important scene determination unit 330 includes image category data indicating the category of the image to be processed, a discriminator stored for each image category in the storage unit 400, and the processing target image calculated by the feature amount calculation unit 320. Based on the important scene feature amount h *, an important scene is determined from the image to be processed. Specifically, the important scene determination unit 330 applies the important scene feature amount h * of the processing target image calculated by the feature amount calculation unit 320 to the discriminator stored in the storage unit 400, thereby performing input. It is determined whether the scene of the processed image to be processed is an important scene.

  As described above, the important scene determination unit 330 determines an important scene based on the input image category data. Specifically, the important scene determination unit 330 reads out the classifiers associated with the image category indicated by the input image category data from among the classifiers stored in the storage unit 400. Thereby, the important scene determination unit 330 can use the classifiers classified for each category of the image as the application target of the important scene feature amount h * of the image to be processed. That is, since the important scene determination unit 330 can select a discriminator suitable for the category of the image to be processed, the important scene can be determined with high accuracy.

The important scene determination unit 330 outputs an image indicating the important scene determined as described above as important scene image data.
Up to this point, the basic configuration of the important scene extraction performed by the image processing apparatus 10 has been described. Hereinafter, a more specific configuration of the important scene extraction performed by the image processing apparatus 10 will be described.

[More Specific Configuration of Image Processing Apparatus 10]
In the above description, the feature amount calculation unit 100 calculates the optical flow OF of the training image and the optical flow FOF of the foreground without changing the number of pixels of the input training image data, but is not limited thereto. Specifically, the feature amount calculation unit 100 calculates the optical flow OF of the training image and the optical flow FOF of the foreground by reducing the number of pixels of the training image data. For example, when the input training image is 1920 × 1080 pixels, the optical flow extraction unit 110 of the feature amount calculation unit 100 changes the size of the training image to 320 × 240 pixels, and the optical flow OF and The foreground optical flow FOF is calculated. At this time, the optical flow extraction unit 110 determines that the aspect ratio (for example, 16: 9) of the input training image is different from the aspect ratio (4: 3) of the image after the image size is changed. When the image size is changed, the image is trimmed. Thereby, the feature quantity calculation unit 100 can reduce the amount of calculation for calculating the optical flow OF. Thereby, the feature-value calculation part 100 can reduce the influence by this noise component, when the noise component is mixed in the input training image.

  In the above description, the feature amount calculation unit 100 calculates the optical flow OFF of the training image and the optical flow FOF of the foreground for all the pixels in the frame of the input training image. Here, in the frame of the moving image, the region that the viewer pays attention to is near the center of the frame, and the four corners of the frame may be hardly noticed. Therefore, the feature amount calculation unit 100 does not have to calculate the optical flow OF of the training image and the optical flow FOF of the foreground for the peripheral pixels in the frame of the training image. That is, the feature quantity calculation unit 100 only needs to calculate the optical flow OF of the training image and the optical flow FOF of the foreground for the pixel at the center portion in the frame of the training image. As a more specific example, the feature amount calculation unit 100 calculates the pixels in the ellipse EO with the radius threshold value α, that is, the pixels represented by the following equation (8) when W and H are the horizontal width and height of the frame. The optical flow OF is calculated by (x, y). In this equation (8), 0 <α ≦ 1.

FIG. 3 shows a specific example in which the feature amount calculation unit 100 calculates the optical flow OF of the training image and the optical flow FOF of the foreground using the pixels in the ellipse EO represented by the equation (8) as the calculation target.
FIG. 3 is a schematic diagram illustrating an example of a motion calculation result by the feature amount calculation unit 100 of the present embodiment. For example, the optical flow extraction unit 110 of the feature amount calculation unit 100 calculates the optical flow OF of the training image using the pixels within the ellipse EO shown in FIG. In addition, the foreground optical flow extraction unit 120 of the feature amount calculation unit 100, for example, as shown in FIG. Calculate as With this configuration, the feature amount calculation unit 100 can reduce the amount of calculation for calculating information (optical flow OF, foreground optical flow FOF) indicating image motion.

  In the above description, the feature amount calculation unit 100 calculates the optical flow OFF of the training image and the optical flow FOF of the foreground for all the pixels in the frame of the input training image. The feature amount calculation unit 100 may calculate an optical flow OF and a foreground optical flow FOF for pixels thinned out at a predetermined interval among the pixels in the frame. As an example, the feature amount calculation unit 100 calculates an optical flow OF and a foreground optical flow FOF for each pixel interval L (for example, L = 5 pixels) (see Expression (9) and Expression (10)).

  Also with this configuration, the feature amount calculation unit 100 can reduce the amount of calculation for calculating the optical flow OF.

  Further, the optical flow extraction unit 110 calculates noise and noise before calculating the average value v￣ (buoy bar) and the standard deviation σ (sigma) of the optical flow OF by the above-described equations (2) and (3). The optical flow OF (v (x, y)) is calculated by using the threshold v0 (buoy zero) of the optical flow to be regarded and the threshold σ0 (sigma zero) of the standard deviation of the foreground optical flow. (See equation (11).)

  As an example, the optical flow extraction unit 110 calculates the optical flow OF (v (x, y)) with the threshold v0 = 2 and the threshold σ0 = 2 pixels.

  With this configuration, the optical flow extraction unit 110 can remove noise when calculating the optical flow OF, and can reduce erroneous detection of the foreground optical flow FOF.

  The foreground optical flow extraction unit 120 has been described as calculating the foreground optical flow FOF (v * (x, y)) (buoy asterisk x wai) based on the above-described equation (4). Not limited to this. The foreground optical flow extraction unit 120 may be configured to calculate the foreground optical flow FOF using a threshold β (β ≧ 0), as shown in Expression (12). This threshold value β is, for example, threshold value β = 1.

  With this configuration, the foreground optical flow extraction unit 120 can adjust the calculated foreground optical flow FOF in accordance with, for example, the category of the moving image and the length (duration) of the moving image scene. .

  Further, the important scene determination unit 330 may output an image by combining two or more important scenes into one important scene. In this case, if the time interval between two adjacent important scenes is equal to or less than a certain time T (for example, time T = 2 seconds), the important scene determination unit 330 will perform the two previous important scenes and the subsequent important scenes. Of the scenes, the time from the start time of the previous important scene to the end time of the subsequent important scene may be output together as one important scene. By configuring in this way, the important scene determination unit 330 can output the important scene so that the important scene is not shredded.

[Operation of Image Processing Apparatus 10]
Next, the operation of the image processing apparatus 10 according to the present embodiment will be described with reference to FIG.
FIG. 4 is a flowchart showing an example of the operation of the image processing apparatus 10 of the present embodiment. First, the operation in the training process will be described, and then the operation in the identification process will be described.

  In the training process, the optical flow extraction unit 110 extracts the optical flow OF of the input training image (step S10).

  Next, the foreground optical flow extraction unit 120 extracts the foreground optical flow FOF from the optical flow OF of the training image extracted by the optical flow extraction unit 110 (step S20).

Next, the foreground optical flow extraction unit 120 quantizes the foreground optical flow FOF extracted in step S20 in a plurality of directions (step S30).
Next, the foreground optical flow extraction unit 120 calculates the frequency in each direction for the foreground optical flow FOF quantized in each direction in step S30 (step S40). Thereby, the direction histogram h is calculated.

  Next, the important scene feature quantity calculation unit 130 calculates the important scene feature quantity h * (etch asterisk) based on the direction histogram h calculated in step S40 (step S50).

  Next, the feature amount calculation unit 100 determines whether or not the important scene feature amount h * has been calculated for all training images (step S60). Here, the training images include images of various categories. Since the feature amount calculation unit 100 calculates the important scene feature amount h * for each category of the training image, the feature amount calculation unit 100 calculates the important scene feature amount h * for each of the plurality of training images. If the feature amount calculation unit 100 determines that the important scene feature amount h * has been calculated for all training images (step S60: YES), the feature amount calculation unit 100 advances the process to step S70. When it is determined that the important scene feature value h * has not been calculated for all training images (step S60: NO), the feature value calculating unit 100 calculates the important scene feature value h * for the next training image. The process returns to step S10.

  Next, the discriminator generation unit 200 generates a discriminator based on the important scene feature amount h * calculated for each image category in step S50, the input image category data, and the important scene data. The discriminator generation unit 200 associates the input image category data with the generated discriminator, stores them in the storage unit 400, and ends the training process (step S70).

  Next, in the identification process, the feature amount calculation unit 320 calculates the important scene feature amount h * in the same manner as in steps S10 to S50 described above (steps S100 to S140).

  Next, the important scene determination unit 330 calculates the important scene feature amount h * of the processing target image calculated in steps S100 to S140, the classifier stored in the storage unit 400 in step S70, and the input image. Based on the category, an important scene is extracted from the processing target image, and the process ends (step S150).

  As described above, the image processing apparatus 10 according to the present embodiment includes the feature amount calculation unit 100 and the classifier generation unit 200. Accordingly, the image processing apparatus 10 extracts a subject included in the input image (for example, a moving image), and calculates an important scene feature amount h * based on the movement of the subject. Therefore, the image processing apparatus 10 can calculate the important scene feature amount h * without requiring additional information such as audio information when the input image is taken and sensor information of the camera. That is, according to the image processing apparatus 10, an important scene can be accurately extracted from image data without using a special sensor.

  In addition, when a moving image is shot with a camera following a moving subject, the subject is almost fixed and the background moves greatly. Also in this case, according to the image processing apparatus 10 of the present embodiment, the foreground optical flow FOF can be calculated without erroneously recognizing a largely moving background as a subject (foreground).

  In the above-described embodiment, the example in which the image processing apparatus 10 extracts the important scene based on the moving image category has been described. However, the present invention is not limited to this. For example, since the image processing apparatus 10 extracts an important scene based on a general machine learning method, the image processing apparatus 10 can extract an important scene without calculating a feature amount for each category of a moving image. Thereby, the image processing apparatus 10 can simplify the structure.

  In the above-described embodiment, the example in which the image processing apparatus 10 associates the input moving image category with the calculated feature amount has been described, but the present invention is not limited thereto. For example, the image processing apparatus 10 may determine a moving image category in the self apparatus based on a general machine learning technique, and associate the determined moving image category with the calculated feature amount. Thereby, since the image processing apparatus 10 can automate the category determination of the moving image, the operation can be facilitated.

  Further, some functions of the image processing apparatus 10 described above may be realized by a computer. In this case, the image processing program for realizing the function may be recorded on a computer-readable recording medium, and the image processing program recorded on the recording medium may be read by the computer system and executed. Good. Here, the “computer system” includes an OS (Operating System) and peripheral device hardware. The “computer-readable recording medium” refers to a portable recording medium such as a flexible disk, a magneto-optical disk, an optical disk, and a memory card, and a storage device such as a magnetic hard disk built in the computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, it may include a device that holds a program for a certain period of time, such as a volatile memory inside a computer system serving as a server device or a client. Further, the above program may be for realizing a part of the functions described above, or may be realized by a combination with the program already recorded in the computer system. .

As mentioned above, although embodiment of this invention was explained in full detail with reference to drawings, the specific structure is not restricted to that embodiment, The design of the range which does not deviate from the summary of this invention, etc. are included.

  DESCRIPTION OF SYMBOLS 10 ... Image processing apparatus, 100 ... Feature-value calculation part, 110 ... Optical flow extraction part, 120 ... Foreground optical flow extraction part, 130 ... Important scene feature-value calculation part, 200 ... Discriminator production | generation part, 300 ... Important scene extraction , 320... Feature amount calculation unit, 330... Important scene determination unit, 400.

Claims (9)

A motion extraction unit that extracts motion information of the image based on the input image;
Based on the extracted motion information, a feature amount calculating unit that calculates a feature amount indicating an important scene in which the subject in the image has a motion;
An image processing apparatus comprising: The movement information is information indicating movement of the subject in the image,
The motion extraction unit
Based on the input image, extract motion information indicating the motion of the subject in the image,
The feature amount calculation unit includes:
The image processing apparatus according to claim 1, wherein a feature amount indicating the important scene is calculated based on movement information indicating the extracted movement of the subject. The image processing apparatus according to claim 1, wherein the subject is an image of an image area that has a relatively large movement between the image areas among a plurality of image areas in the image. . The movement information includes information indicating the direction of movement of the subject,
The feature amount calculation unit includes:
The image processing apparatus according to any one of claims 1 to 3, wherein the feature amount is calculated based on a direction of the motion indicated by the motion information. The motion extraction unit
Based on the input image, the motion information of the image is extracted for each predetermined pixel constituting the image,
The feature amount calculation unit includes:
The image processing apparatus according to claim 4, wherein a feature amount indicating the important scene is calculated based on an appearance frequency of the motion direction indicated by the motion information extracted for each predetermined pixel. The feature amount calculation unit includes:
The feature amount indicating the important scene is calculated based on the direction of the motion having the highest appearance frequency among the directions of the motion indicated by the motion information extracted for each predetermined pixel. 5. The image processing apparatus according to 5. The feature amount calculation unit is further input with information indicating the type of the image,
The feature amount calculation unit includes:
The feature amount indicating the important scene is calculated based on the input information indicating the type of the image and the extracted motion information. An image processing apparatus according to 1. A motion extraction procedure for extracting motion information of the image based on the input image;
Based on the extracted motion information, a feature amount calculation procedure for calculating a feature amount indicating an important scene in which a subject in the image has a motion;
An image processing method comprising: In the computer provided in the image processing apparatus,
A motion extraction procedure for extracting motion information of the image based on the input image;
Based on the extracted motion information, a feature amount calculation procedure for calculating a feature amount indicating an important scene in which a subject in the image has a motion;
A program for running

Images