WO2016031573A1 - Image-processing device, image-processing method, program, and recording medium - Google Patents

Image-processing device, image-processing method, program, and recording medium Download PDF

Info

Publication number
WO2016031573A1
WO2016031573A1 PCT/JP2015/072818 JP2015072818W WO2016031573A1 WO 2016031573 A1 WO2016031573 A1 WO 2016031573A1 JP 2015072818 W JP2015072818 W JP 2015072818W WO 2016031573 A1 WO2016031573 A1 WO 2016031573A1
Authority
WO
WIPO (PCT)
Prior art keywords
person
still image
still
motion
noted
Prior art date
Application number
PCT/JP2015/072818
Other languages
French (fr)
Japanese (ja)
Inventor
学斌 胡
Original Assignee
富士フイルム株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 富士フイルム株式会社 filed Critical 富士フイルム株式会社
Publication of WO2016031573A1 publication Critical patent/WO2016031573A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/76—Television signal recording
    • H04N5/91—Television signal processing therefor

Definitions

  • the present invention relates to an image processing apparatus, an image processing method, a program and a recording medium for extracting and outputting still image data from moving image data.
  • a scene of a best shot (a person captured in a moving image) that can not be captured (still difficult to capture) with a still image, such as a moment when a child blows out a candle on a birthday May be included in a scene that preferably represents the motion of
  • moving images include those with little movement of persons, those with low importance, those with poor composition, those with poor image quality, and the like.
  • Patent Document 1 relates to a person action search device that enables a person to quickly reproduce moving image data from a recorded position at which a person is recorded.
  • a person action search device that enables a person to quickly reproduce moving image data from a recorded position at which a person is recorded.
  • a representative image of the recognized person is extracted, and after the virtual center of gravity of the human image appears in the photographed image in the representative image, it disappears out of the photographed image
  • a bird's-eye view image is created by synthesizing tracking lines which are movement trajectories up to.
  • Patent Document 2 relates to a technique for extracting a representative frame that clearly represents a video included in the moving image data from the moving image data.
  • a representative frame image a frame image with the largest evaluation value output by the face state determination unit.
  • An object of the present invention is an image processing apparatus capable of automatically extracting and outputting still image data of a still image corresponding to a scene of a best shot from moving image data by solving the problems of the prior art, and an image processing method , Providing a program and a recording medium.
  • the present invention provides a still image data extraction unit that extracts still image data of a plurality of frames from moving image data;
  • An attention person detection unit that detects an attention person who is a person to be processed from each of a plurality of still images corresponding to still image data of a plurality of frames;
  • a movement trajectory detection unit for tracking the movement of the person of interest in the moving image corresponding to the moving image data to detect the movement locus of the person of interest based on the detection result of the person of interest in the plurality of still images;
  • Motion analysis unit The present invention provides an image processing apparatus including: a still image data output unit that outputs still image data of a still image having an evaluation value with respect to the motion of a person of interest out of a plurality of frames of still image data
  • a person registration unit for registering a person to be processed among persons photographed in a moving image as a registered person
  • the person-of-interest detection unit detects a person who matches the registered person or a person whose similarity is equal to or more than a threshold value as a person-of-interest from among each of the plurality of still images.
  • the person-of-interest detection unit extracts the face of the person from each of the plurality of still images, and performs the main person determination on the face image of the extracted person's face, thereby extracting the face.
  • the target person detection unit further detects the face area of the target person in the still image
  • the motion trajectory detection unit is based on the face area of the target person and is an arbitrary face corresponding to the face region of the target person in the still image of the current frame and the face region of the target person in the still image of the frame next to the current frame. Based on the position of the detection area in the still image of the next frame, the similarity with the face area of the target person in the still image of the current frame is compared with the detection area in the position and the position detection area is compared. It is preferable to track the movement of the target person in the moving image by detecting to which detection region of the still image of the next frame the face region of the target person in the still image has moved.
  • the motion trajectory detection unit preferably tracks the movement of the person of interest for each of the regions obtained by dividing the region of the upper body of the person of interest into a predetermined number.
  • the motion locus detection unit generates an integral image of the still image of the next frame, and uses the generated integral image to all included in the detection area at any position in the still image of the next frame.
  • the calculation of the sum of the luminance values of the image is sequentially repeated for detection regions at a plurality of positions.
  • the motion locus detection unit track the movement of the noted person by using an average displacement method.
  • the motion analysis unit defines in advance a motion trajectory for the motion of the target person, and detects a portion similar to the motion trajectory defined in advance from the motion trajectory of the target person detected by the motion trajectory detection unit. By doing this, it is preferable to analyze the movement of the noted person and calculate an evaluation value for the movement of the noted person according to the type of the movement of the noted person.
  • the motion analysis unit analyzes the motion of the target person based on the motion history image of the target person as the motion trajectory of the target person, and calculates an evaluation value for the motion of the target person.
  • the target person detection unit further detects the position of the target person in the still image, the size of the target person in the still image, and the region of the target person in the still image
  • the motion trajectory detection unit further detects the length of the motion trajectory of the person of interest and the movement pattern of the person of interest, Furthermore, the importance of each of the plurality of still images is determined based on at least one of the length of the movement locus of the target person, the position of the target person in the still image, and the size of the target person in the still image.
  • An importance determination unit that calculates an evaluation value of the importance based on the determined importance for each of the plurality of still images; Based on at least one of the position of the target person in the still image, the size of the target person in the still image, and the movement pattern of the target person, the quality of each composition of the plurality of still images is analyzed.
  • a composition analysis unit that calculates an evaluation value of the composition based on the quality of the analyzed composition for each of the still images; Image quality determination that determines the image quality of each of a plurality of still images based on the area of the person of interest in the still image, and calculates an evaluation value of the image quality based on the determined image quality for each of the plurality of still images Equipped with
  • the still image data output unit is configured to evaluate at least one of the evaluation value for the motion of the person of interest, the evaluation value of importance, the evaluation value of the composition, and the evaluation value of the image quality among the plurality of still images. It is preferable to output still image data of one or more still images whose overall evaluation value is greater than or equal to a threshold value.
  • the composition analysis unit defines in advance the movement pattern of the person of interest, and the person of interest is moved with the movement pattern defined in advance from among the movement loci of the person of interest detected by the movement locus detection unit. A portion is detected, and the composition of the still image corresponding to the portion where the noted person is moving is analyzed as good according to a predefined movement pattern, and the evaluation value of the composition of the still image analyzed as good is analyzed as good. It is preferable that the calculation is performed so as to be higher than the evaluation value of the composition of the still image which is not performed.
  • the target person detection unit further detects the orientation of the target person's face in the still image, Furthermore, based on the orientation of the face of the target person detected by the target person detection unit, the top and bottom of the still image corresponding to the still image data output from the still image data output unit is captured when the moving image is captured It is preferable to have a top-bottom correction unit that corrects the top and bottom of the still image corresponding to the still image data output from the still image data output unit so as to be the same as the top and bottom of the device.
  • the still image data extraction unit extracts still image data of a plurality of frames from moving image data; Detecting a noted person who is a person to be processed from each of a plurality of still images corresponding to still image data of a plurality of frames;
  • the motion locus detection unit detects the movement locus of the target person by tracking the movement of the target person in the moving image corresponding to the moving image data based on the detection result of the target person in the plurality of still images;
  • the motion analysis unit analyzes the motion of the watched person in the moving image based on the motion trajectory of the watched person, and for each of the plurality of still images, the motion of the watched person with respect to the motion of the watched person based on the analyzed motion of the watched person.
  • the still image data output unit provides an image processing method including the steps of outputting, among still image data of a plurality of frames, still image data of a still image whose evaluation value for the motion of the person of interest is equal to or greater than a threshold.
  • the motion trajectory detection unit detects the length of the motion trajectory of the person of interest and the movement pattern of the person of interest;
  • the importance determination unit determines the importance of each of the plurality of still images based on at least one of the length of the motion locus of the target person, the position of the target person in the still image, and the size of the target person in the still image.
  • the composition analysis unit analyzes the quality of each of the plurality of still images based on at least one of the position of the target person in the still image, the size of the target person in the still image, and the movement pattern of the target person. Calculating an evaluation value of the composition based on the quality of the analyzed composition for each of a plurality of still images;
  • the image quality determination unit determines the image quality of each of the plurality of still images based on the area of the person of interest in the still image, and the image quality evaluation value based on the determined image quality for each of the plurality of still images.
  • the still image data output unit calculates the evaluation value for the motion of the person of interest, the evaluation value of importance, the evaluation value of composition, and the evaluation value of image quality Outputting still image data of one or more still images whose overall evaluation value is greater than or equal to a threshold value.
  • the moving image was taken with the still image corresponding to the still image data output from the still image data output unit based on the face orientation of the noted person detected by the noted person detection unit by the elevation correction unit And correcting the top and bottom of the still image corresponding to the still image data output from the still image data output unit so as to be the same as the top and bottom of the photographing apparatus at the time.
  • the present invention also provides a program for causing a computer to execute the steps of the image processing method described above.
  • the present invention also provides a computer readable recording medium having recorded thereon a program for causing a computer to execute the steps of the image processing method described above.
  • the scene of the best shot is automatically detected from the moving image, and the scene of the best shot from still image data of a plurality of frames extracted from the moving image data Still image data corresponding to the still image can be output.
  • FIG. 1 It is a block diagram of one embodiment showing composition of an image processing device of the present invention.
  • the left side of (A) to (C) is a conceptual diagram of an example showing the movement locus of the noted person, and the right side is a conceptual diagram of an example showing the movement history image of the noted person.
  • (A) is a conceptual diagram of an example showing a still image rotated 90 ° to the left
  • (B) shows a still image corrected to the top and bottom by rotating the still image shown in (A) 90 ° to the right
  • FIG. 1 is a block diagram of an embodiment showing the configuration of the image processing apparatus of the present invention.
  • the image processing apparatus 10 shown in the figure automatically detects a scene of a best shot from a moving image, and outputs still image data of a still image corresponding to the scene of the best shot.
  • Still image data extraction unit 14 attention person detection unit 16, motion trajectory detection unit 18, motion analysis unit 20, importance determination unit 22, composition analysis unit 24, image quality determination unit 26, still image data
  • An output unit 28 and a top-bottom correction unit 30 are provided.
  • the target person registration unit 12 registers a target person to be processed as a registered person among persons photographed in a moving image corresponding to moving image data.
  • the notable person registration unit 12 can register, for example, a person designated by the user among persons photographed in a moving image as a registered person.
  • the focused person registration unit 12 can register an image of a registered person (a face image or the like for specifying the focused person).
  • the still image data extraction unit 14 extracts still image data of a plurality of frames from the moving image data.
  • the still image data extraction unit 14 can extract, for example, still image data of all frames (each frame) of moving image data.
  • still image data of one frame may be extracted, for example, every two frames, for each fixed number of frames.
  • still image data of a frame of an arbitrary section of a moving image corresponding to moving image data may be extracted.
  • the focused person detection unit 16 selects a person to be processed from among each of a plurality of still images corresponding to still image data of a plurality of frames extracted from moving image data by the still image data extraction unit 14. The person of interest is detected.
  • the noted person detection unit 16 detects, for example, the presence or absence of a person in each of a plurality of still images, and detects an image of the detected person and, for example, an image of a registered person registered in the noted person registration unit 12. By comparing (compare face images and the like), it is possible to specify a person who matches or is similar to the registered person (a person whose similarity is equal to or higher than a threshold) among the detected persons as the noted person.
  • the target person detection unit 16 extracts the face of the person from each of the plurality of still images, and the face person is extracted by performing the central person determination on the face image of the extracted face of the person. From among the persons, the person determined to be the center person by the center person determination can be identified as the person of interest.
  • the same person determination process is performed on a plurality of face images, and the plurality of face images are classified into an image group including face images of the same person. Subsequently, one or more persons of the persons classified into the image group are determined as the main character, and one or more persons having high relevancy with the main character among the persons other than the main character are determined as the important persons. Further, based on the face image of each registered person registered in the focused person registration unit 12, the person corresponding to each image group can be specified.
  • a person with the largest number of face images detected can be determined as the main character, or a person other than the main character with a large number of still images photographed with the main character can be determined as the important person.
  • the distance between the face image of the main character and the face image of a person other than the main character photographed in the same still image may be calculated, and a person whose distance between the face images is close may be determined as an important person.
  • the difference between the shooting date and time information of the still image in which the main character was shot and the shooting date and time information of the still image in which the person other than the main character was shot, and the shooting position information of the still image in which the main character was shot and the person other than the main character The important person may be determined based on one or both of the difference from the still image shooting position information.
  • the focused person detection unit 16 detects the position of the focused person, the size of the focused person, the area of the focused person, the upper body area of the focused person, the position of the face of the focused person, and the size of the face of the focused person in the still image.
  • the face area of the person of interest, the direction of the face of the person of interest, etc. can be detected.
  • the detection method of the attention person in a still image the detection method of the face of the attention person, etc. are known, detailed explanation is omitted here, but the specific detection method is not limited at all. Also, the method of detecting the person of interest is not limited at all.
  • the motion locus detection unit 18 tracks the movement of the target person in the moving image corresponding to the moving image data based on the detection result of the target person in the plurality of still images by the target person detection unit 16. It detects the movement locus of a person. Further, the motion locus detection unit 18 can detect the length of the motion locus of the person of interest, the movement pattern of the person of interest, and the like by detecting the motion locus of the person of interest.
  • the movement locus of the person of interest is a region of interest (ROI: Region of Interest), for example, as shown on the left side of FIGS. It is possible to use the ones represented in.
  • ROI Region of Interest
  • a motion history image MHI: Motion History Image
  • the action history image is a view showing the action history of the person of interest, for example, by changing the color at regular intervals.
  • the motion locus detection unit 18 is, for example, based on the face area of the noted person, an arbitrary face corresponding to the face area of the noted person in the still image of the current frame and the face area of the noted person in the still image of the next frame. Based on the position of the detection area in the still image of the next frame, the similarity with the face area of the target person in the still image of the current frame is compared with the detection area in the position and the position detection area is compared. It is possible to track the movement of the target person in the moving image by detecting to which detection region of the still image of the next frame the face region of the target person in the still image has moved.
  • the upper body region of the person of interest is divided into a fixed number, for example, four regions, and the movement of the person of interest is similarly tracked for each of the five regions in total. , Can improve the tracking success rate.
  • the integral image of the still image of the next frame (that is, each frame) is generated, and the total amount of luminance values is calculated using the generated integral image, thereby reducing the amount of calculation and processing. Can be speeded up.
  • the integral image for example, assuming that the coordinates of the pixels of the still image increase from left to right and from top to bottom of the still image, the pixel at each coordinate is from the pixel at the upper left to each coordinate It is an image having an integral value of luminance values up to the pixel.
  • the use of the integral image is not limited for the purpose of reduction of calculation amount and speeding up of processing, for example, mean shift (Mean Shift) method etc. Various methods can be used. Since the mean displacement method is also known, its detailed description is omitted.
  • the motion analysis unit 20 analyzes the motion of the target person in the moving image, based on the motion track of the target person detected by the motion track detection unit 18, for example, the motion track of the region of interest such as a face region, For each of the plurality of still images, the evaluation value for the motion of the target person is calculated based on the analyzed motion of the target person.
  • the motion analysis unit 20 defines, for example, a motion trajectory for the motion of the person of interest, for example, a motion trajectory of when the person of interest is running, and the motion trajectory of the person of interest detected by the motion trajectory detection unit 18. From the inside, the motion of the person of interest is analyzed by detecting a portion similar to the previously defined movement trajectory. Then, when the motion of the target person is a running motion, it is possible to calculate the evaluation value for the motion of the target person according to the type of the motion of the target person, such as what evaluation value.
  • the motion analysis unit 20 analyzes the motion of the target person based on the motion history image as shown on the right side of FIGS. 2A to 2C as the motion locus of the target person, and the motion of the target person is obtained. An evaluation value can be calculated.
  • the motion analysis unit 20 analyzes the motion of the noted person based on the motion history image, so that the noted person is running from the right to the left in the figure as shown on the right of FIG. 2 (A). As shown on the right side of the figure (B), the noted person is moving only the right hand while standing still, and as shown on the right side of the figure (C), the noted person is falling on the ground It can be recognized that something is picked up. In addition, it is possible to calculate an evaluation value for the movement of the person of interest based on whether or not the person of interest is moving, at which position, in which direction, or the like.
  • the importance degree determination unit 22 determines a plurality of still images based on at least one of the length of the motion locus of the target person, the position of the target person in the still image, and the size of the target person in the still image.
  • the degree of importance of each of the plurality of still images is determined, and the evaluation value of the degree of importance is calculated based on the determined degree of importance for each of the plurality of still images.
  • the importance degree determination unit 22 determines that, in the moving image, the importance degree of the still image corresponding to the scene in which the movement locus of the target person is long is high. In addition, it is determined that the still image in which the person of interest is photographed in the central part and the still image in which the person of interest is largely photographed (the size of the person of interest is equal to or larger than the threshold) have high importance. Then, the evaluation value of the importance is calculated to be higher as the importance is higher.
  • the composition analysis unit 24 composes each of the plurality of still images based on at least one of the position of the target person in the still image, the size of the target person in the still image, and the movement pattern of the target person.
  • the quality of the image is analyzed, and the evaluation value of the composition is calculated based on the quality of the analyzed composition for each of the plurality of still images.
  • the composition analysis unit 24 may, for example, have a still image in which the person of interest is photographed at the center, or a composition of still images in which the person of interest is largely photographed (the size of the person of interest is equal to or larger than the threshold). Is analyzed to be better than the composition of a still image not captured in the central part or a still image in which the person of interest is not captured appreciably. Then, the evaluation value of the composition of the still image analyzed as good can be calculated to be higher than the evaluation value of the composition of the still image not analyzed as good.
  • the composition analysis unit 24 defines in advance a movement pattern of the target person, for example, a movement pattern in which the target person moves from the left end to the right end of the moving image. From the trajectory, a portion in which the person of interest is moving is detected by a predefined movement pattern. Then, it analyzes that the composition of the still image corresponding to the portion where the noted person is moving with the predefined movement pattern is good, and the evaluation value of the composition of the still image analyzed as good is not analyzed as good. It can be calculated to be higher than the evaluation value of the composition of the still image.
  • the image quality determination unit 26 determines the image quality of each of the plurality of still images based on the region of interest person in the still image, for example, the region of interest such as the face region.
  • the evaluation value of the image quality is calculated based on the determined image quality.
  • the still image extracted from the moving image may or may not have high image quality depending on the compression method of the moving image data.
  • blurring or blurring may occur in the still image due to defocusing or camera shake, or the luminance, color tone, contrast or the like may not be appropriate.
  • the image quality judgment unit 26 judges that the image quality of the still image is good. Then, for still images judged to have good image quality, the evaluation value of image quality can be calculated to be higher as the image quality is better.
  • the still image data output unit 28 selects still image data of a still image corresponding to the scene of the best shot among still image data of a plurality of frames extracted from the moving image data by the still image data extraction unit 14
  • the evaluation value for the motion of the person of interest, or the evaluation value for the motion of the person of interest, and the evaluation value of importance, the evaluation value of composition, and the evaluation value of at least one of the evaluation values of image quality Still image data of a still image whose value is equal to or greater than a threshold is output.
  • the top / bottom correction unit 30 determines the top / bottom position of the still image corresponding to the still image data output from the still image data output unit 28
  • the top and bottom of the still image corresponding to the still image data output from the still image data output unit 28 is corrected so as to be the same as the top and bottom of the photographing apparatus when the moving image is captured.
  • FIG. 3A is a conceptual view of an example showing a still image rotated 90 degrees to the left.
  • a still image can be obtained by rotating the imaging device by 90 degrees to the right when capturing a moving image.
  • the top-bottom correction unit 30 rotates the still image shown in FIG. 6A by 90 ° to the right so that the top and bottom of the still image are the same as the top and bottom of the imaging device when the moving image is captured.
  • FIG. 6B the top and bottom of the still image can be corrected.
  • the person-of-interest detection unit 16 detects and detects each of the two or more persons of interest from a plurality of still images. It is possible to sequentially identify who is the watched person who has been Further, in this case, the motion locus detection unit 18, the motion analysis unit 20, the importance degree determination unit 22, the composition analysis unit 24, the image quality determination unit 26, the still image data output unit 28, and the top-bottom correction unit 30 Processing is sequentially performed on each of the noted persons.
  • step S1 a person designated by the user is registered as an attention person by the attention person registration unit 12 (step S2).
  • the still image data extraction unit 14 extracts, for example, still image data of all the frames from the moving image data (step S2). That is, as shown in FIG. 5, still images of all frames are extracted from the moving image.
  • a notable person registered in the notable person registration unit 12 is detected by the notable person detection unit 16 out of each of the still images of all the frames extracted by the still image data extraction unit 14 (step S3). ).
  • the person of interest is identified in each of the still images of all the frames, and the position of the person of interest in each of the still images of all the frames, as shown by the frame in FIG. The size, the area of the noted person, etc. are detected.
  • the motion locus detection unit 18 tracks the movement of the target person in the moving image, for example, the movement of the region of interest shown by the frame in FIG.
  • the motion locus of the person of interest is detected (step S4).
  • a motion locus of a person of interest representing in a line the locus of movement of a region of interest such as a face region
  • An operation history image as shown on the right of A) to (C) can be obtained.
  • the motion analysis unit 20 analyzes the motion of the target person in the moving image based on the motion track of the target person detected by the motion track detection unit 18. Then, for each of the still images of all the frames, an evaluation value for the motion of the target person is calculated based on the analyzed motion of the target person (step S5-1).
  • the importance degree determination unit 22 determines the importance degree of each of all still images based on the length of the motion locus of the person of interest, the position of the person of interest in the still image, and the size of the person of interest. Then, the evaluation value of the importance is calculated based on the determined importance for each of the still images of all the frames (step S5-2).
  • composition analysis unit 24 analyzes the quality of each composition of all still images based on the position of the target person in the still image, the size of the target person, and the movement pattern of the target person. Then, for each of the still images of all the frames, the evaluation value of the composition is calculated based on the quality of the analyzed composition (step S5-3).
  • the image quality determination unit 26 determines the image quality of each of the still images of all the frames based on the area of the person of interest in the still images. Then, for each of all still images, the image quality evaluation value is calculated according to the determined image quality, and in the case of the present embodiment, the degree of blur (step S5-4). For example, determination of blurring of the region of interest shown by a frame in FIG. 6 is performed, and the evaluation value of the image quality is calculated to be lower as the degree of blurring is larger.
  • the order of calculating the evaluation value of the motion of the person of interest, the evaluation value of importance, the evaluation value of the composition, and the evaluation value of the image quality is not limited at all, and can be calculated in any order. Also, these evaluation values can be calculated in parallel, that is, simultaneously.
  • Still image data of at least one still image is output (step S6).
  • FIG. 7 is a graph of an example showing the comprehensive evaluation value of each of the still images of all the frames extracted from the moving image.
  • the vertical axis of the figure represents the comprehensive evaluation value of each still image, and the horizontal axis represents time (frame).
  • the person of interest is detected by the person of interest detection unit 16 and the motion locus of the person of interest is detected by the motion locus detection unit 18;
  • still image data of a still image whose overall evaluation value is equal to or greater than a threshold is output.
  • the top and bottom of the still image are the same as the top and bottom of the imaging device when the moving image is captured.
  • the top and bottom of the still image is corrected so as to be (step S7).
  • the image processing apparatus 10 for example, based on the comprehensive evaluation value including the evaluation value for the motion of the target person in the moving image, the evaluation value of the importance of the still image, the evaluation value of the composition and the evaluation value of the image quality. Automatically detects the scene of the best shot from the moving image, and extracts the still image data of the still image corresponding to the scene of the best shot from the still image data of all the frames extracted from the moving image data be able to.
  • each component of the device may be configured by dedicated hardware, or each component may be configured by a programmed computer.
  • the method of the present invention can be implemented, for example, by a program for causing a computer to execute each of the steps. It is also possible to provide a computer readable recording medium in which the program is recorded.
  • the present invention is basically as described above.
  • the present invention has been described above in detail, but the present invention is not limited to the above embodiment, and it goes without saying that various improvements and changes may be made without departing from the spirit of the present invention.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Analysis (AREA)
  • Television Signal Processing For Recording (AREA)
  • Studio Devices (AREA)

Abstract

　In this image-processing device, a person-of-interest-detecting unit detects a person of interest from each of a plurality of still images corresponding to still image data on a plurality of frames extracted from moving image data. A motion-trajectory-detecting unit detects motion trajectory by tracking the movement of the person of interest in the moving images on the basis of the detection results for the person of interest. An action-analyzing unit analyzes the actions of the person of interest in the moving images on the basis of the motion trajectory and calculates evaluation values for the actions of the person of interest on the basis of the actions of the person interest for each of the plurality of still images. In addition, a still-image-data-outputting unit outputs still image data on still images that have an evaluation value for the actions of the person of interest at or above a threshold value, the still-image-data-outputting unit outputting the still image data from the still image data on the plurality of frames.

Description

画像処理装置、画像処理方法、プログラムおよび記録媒体IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, PROGRAM, AND RECORDING MEDIUM
　本発明は、動画像データから静止画像データを抽出して出力する画像処理装置、画像処理方法、プログラムおよび記録媒体に関するものである。 The present invention relates to an image processing apparatus, an image processing method, a program and a recording medium for extracting and outputting still image data from moving image data.
　近年、一般家庭においてもたくさんの動画像が撮影されている。撮影された動画像の中には、例えば、子供が誕生日にろうそくを吹き消す瞬間等のように、静止画像では捉えきれない（撮影が難しい）ベストショットのシーン（動画像に撮影された人物の動作を好適に表すシーン）が含まれている可能性がある。その一方で、動画像の中には、人物の動きが少ないもの、重要度が低いもの、構図が悪いもの、画質が悪いもの等が含まれている場合がある。 In recent years, many moving pictures have been taken also in general homes. Among the captured moving images, for example, a scene of a best shot (a person captured in a moving image) that can not be captured (still difficult to capture) with a still image, such as a moment when a child blows out a candle on a birthday May be included in a scene that preferably represents the motion of On the other hand, there are cases where moving images include those with little movement of persons, those with low importance, those with poor composition, those with poor image quality, and the like.
　従って、動画像の中からベストショットのシーンを見つけ出し、静止画像として抽出するのには非常に手間がかかるという問題がある。 Therefore, there is a problem that it takes much time to find the best shot scene from the moving image and extract it as a still image.
　ここで、本発明に関連性のある先行技術文献として、特許文献１，２がある。 Here, there are patent documents 1 and 2 as prior art documents relevant to the present invention.
　特許文献１は、人が録画されている録画位置から素早く動画データを再生できるようにした人物行動検索装置に関するものである。同文献には、撮影画像上に人が認識されると、認識された人の代表画像を抽出し、代表画像に、人物画像の仮想重心点が撮像画像内に現れてから撮像画像外に消えるまでの移動軌跡である追跡線を合成した栞画像を作成することが記載されている。 Patent Document 1 relates to a person action search device that enables a person to quickly reproduce moving image data from a recorded position at which a person is recorded. In the document, when a person is recognized on a photographed image, a representative image of the recognized person is extracted, and after the virtual center of gravity of the human image appears in the photographed image in the representative image, it disappears out of the photographed image It has been described that a bird's-eye view image is created by synthesizing tracking lines which are movement trajectories up to.
　特許文献２は、動画像データから当該動画像データに含まれる映像を端的に表す代表フレームを抽出する技術に関するものである。同文献には、動画像データの所定の時間区間から当該区間の内容（映像）を良く表す代表フレームを１枚あるいは複数枚抽出することが記載されている。また、同文献には、代表フレーム画像として、顔状態判定部が出力する評価値が最大となったフレーム画像を抽出することが記載されている。 Patent Document 2 relates to a technique for extracting a representative frame that clearly represents a video included in the moving image data from the moving image data. In the document, it is described that one or a plurality of representative frames well representing the contents (video) of the section are extracted from a predetermined time section of moving image data. Further, it is described in the same document to extract, as a representative frame image, a frame image with the largest evaluation value output by the face state determination unit.
特開２００９－７５８０２号公報JP, 2009-75802, A 特開２０１０－１０９５９２号公報JP, 2010-109592, A
　本発明の目的は、従来技術の問題点を解消し、動画像データからベストショットのシーンに対応する静止画像の静止画像データを自動で抽出して出力することができる画像処理装置、画像処理方法、プログラムおよび記録媒体を提供することにある。 An object of the present invention is an image processing apparatus capable of automatically extracting and outputting still image data of a still image corresponding to a scene of a best shot from moving image data by solving the problems of the prior art, and an image processing method , Providing a program and a recording medium.
　上記目的を達成するために、本発明は、動画像データから、複数のフレームの静止画像データを抽出する静止画像データ抽出部と、
　複数のフレームの静止画像データに対応する複数枚の静止画像の各々の中から、処理対象とする人物である注目人物を検出する注目人物検出部と、
　複数枚の静止画像における注目人物の検出結果に基づいて、動画像データに対応する動画像における注目人物の動きを追跡して注目人物の運動軌跡を検出する運動軌跡検出部と、
　注目人物の運動軌跡に基づいて、動画像における注目人物の動作を分析し、複数枚の静止画像の各々について、分析された注目人物の動作に基づいて、注目人物の動作に対する評価値を算出する動作分析部と、
　複数のフレームの静止画像データの中から、注目人物の動作に対する評価値が閾値以上である静止画像の静止画像データを出力する静止画像データ出力部とを備える画像処理装置を提供するものである。 In order to achieve the above object, the present invention provides a still image data extraction unit that extracts still image data of a plurality of frames from moving image data;
An attention person detection unit that detects an attention person who is a person to be processed from each of a plurality of still images corresponding to still image data of a plurality of frames;
A movement trajectory detection unit for tracking the movement of the person of interest in the moving image corresponding to the moving image data to detect the movement locus of the person of interest based on the detection result of the person of interest in the plurality of still images;
Based on the motion locus of the target person, the motion of the target person in the moving image is analyzed, and for each of a plurality of still images, the evaluation value for the motion of the target person is calculated based on the analyzed motion of the target person. Motion analysis unit,
The present invention provides an image processing apparatus including: a still image data output unit that outputs still image data of a still image having an evaluation value with respect to the motion of a person of interest out of a plurality of frames of still image data.
　さらに、動画像に撮影された人物のうち、処理対象とする人物を登録人物として登録する人物登録部を備え、
　注目人物検出部は、複数枚の静止画像の各々の中から、登録人物と一致した人物ないし類似度が閾値以上の人物を注目人物として検出するものであることが好ましい。 And a person registration unit for registering a person to be processed among persons photographed in a moving image as a registered person,
Preferably, the person-of-interest detection unit detects a person who matches the registered person or a person whose similarity is equal to or more than a threshold value as a person-of-interest from among each of the plurality of still images.
　また、注目人物検出部は、複数枚の静止画像の各々の中から人物の顔を抽出し、抽出された人物の顔の顔画像に対し中心人物判定を行うことにより、顔が抽出された人物の中から、中心人物判定により中心人物であると判定された人物を注目人物として検出するものであることが好ましい。 The person-of-interest detection unit extracts the face of the person from each of the plurality of still images, and performs the main person determination on the face image of the extracted person's face, thereby extracting the face. Among the above, it is preferable to detect a person who is determined to be the center person by the center person determination as the person of interest.
　また注目人物検出部は、さらに、静止画像における注目人物の顔領域を検出するものであり、
　運動軌跡検出部は、注目人物の顔領域に基づいて、現在のフレームの静止画像における注目人物の顔領域と、現在のフレームの次のフレームの静止画像における注目人物の顔領域に相当する任意の位置の検出領域とを比較し、現在のフレームの静止画像における注目人物の顔領域との類似度が閾値以上である、次のフレームの静止画像における検出領域の位置に基づいて、現在のフレームの静止画像における注目人物の顔領域が、次のフレームの静止画像のどの位置の検出領域に移動しているかを検出することにより、動画像における注目人物の動きを追跡するものであることが好ましい。 Further, the target person detection unit further detects the face area of the target person in the still image,
The motion trajectory detection unit is based on the face area of the target person and is an arbitrary face corresponding to the face region of the target person in the still image of the current frame and the face region of the target person in the still image of the frame next to the current frame. Based on the position of the detection area in the still image of the next frame, the similarity with the face area of the target person in the still image of the current frame is compared with the detection area in the position and the position detection area is compared. It is preferable to track the movement of the target person in the moving image by detecting to which detection region of the still image of the next frame the face region of the target person in the still image has moved.
　また、運動軌跡検出部は、注目人物の顔領域に加えて、注目人物の上半身の領域を一定数に分割した領域のそれぞれについて、注目人物の動きを追跡するものであることが好ましい。 In addition to the face area of the person of interest, the motion trajectory detection unit preferably tracks the movement of the person of interest for each of the regions obtained by dividing the region of the upper body of the person of interest into a predetermined number.
　また、運動軌跡検出部は、次のフレームの静止画像の積分画像を生成し、生成された積分画像を利用して、次のフレームの静止画像において、任意の位置の検出領域内に含まれる全ての画像の輝度値の総和を算出することを、複数の位置の検出領域について順次繰り返すものであることが好ましい。 In addition, the motion locus detection unit generates an integral image of the still image of the next frame, and uses the generated integral image to all included in the detection area at any position in the still image of the next frame. Preferably, the calculation of the sum of the luminance values of the image is sequentially repeated for detection regions at a plurality of positions.
　また、運動軌跡検出部は、平均変位法を利用して、注目人物の動きを追跡するものであることが好ましい。 Further, it is preferable that the motion locus detection unit track the movement of the noted person by using an average displacement method.
　また、動作分析部は、注目人物の動作に対する運動軌跡をあらかじめ定義しておき、運動軌跡検出部により検出された注目人物の運動軌跡の中から、あらかじめ定義された運動軌跡と類似した部分を検出することにより、注目人物の動作を分析し、注目人物の動作の種類に応じて、注目人物の動作に対する評価値を算出するものであることが好ましい。 In addition, the motion analysis unit defines in advance a motion trajectory for the motion of the target person, and detects a portion similar to the motion trajectory defined in advance from the motion trajectory of the target person detected by the motion trajectory detection unit. By doing this, it is preferable to analyze the movement of the noted person and calculate an evaluation value for the movement of the noted person according to the type of the movement of the noted person.
　また、動作分析部は、注目人物の運動軌跡として、注目人物の動作履歴画像に基づいて、注目人物の動作を分析し、注目人物の動作に対する評価値を算出するものであることが好ましい。 Preferably, the motion analysis unit analyzes the motion of the target person based on the motion history image of the target person as the motion trajectory of the target person, and calculates an evaluation value for the motion of the target person.
　また、注目人物検出部は、さらに、静止画像における注目人物の位置、静止画像における注目人物の大きさ、および、静止画像における注目人物の領域を検出するものであり、
　運動軌跡検出部は、さらに、注目人物の運動軌跡の長さ、および、注目人物の移動パターンを検出するものであり、
　さらに、注目人物の運動軌跡の長さ、静止画像における注目人物の位置、静止画像における注目人物の大きさのうちの少なくとも１つに基づいて、複数枚の静止画像の各々の重要度を判定し、複数枚の静止画像の各々について、判定された重要度に基づいて、重要度の評価値を算出する重要度判定部と、
　静止画像における注目人物の位置、静止画像における注目人物の大きさ、注目人物の移動パターンのうちの少なくとも１つに基づいて、複数枚の静止画像の各々の構図の良否を分析し、複数枚の静止画像の各々について、分析された構図の良否に基づいて、構図の評価値を算出する構図分析部と、
　静止画像における注目人物の領域に基づいて、複数枚の静止画像の各々の画質を判定し、複数枚の静止画像の各々について、判定された画質に基づいて、画質の評価値を算出する画質判定部とを備え、
　静止画像データ出力部は、複数枚の静止画像の中から、注目人物の動作に対する評価値と、重要度の評価値、構図の評価値、および、画質の評価値のうちの少なくとも１つの評価値との総合評価値が閾値以上である１枚以上の静止画像の静止画像データを出力するものであることが好ましい。 Further, the target person detection unit further detects the position of the target person in the still image, the size of the target person in the still image, and the region of the target person in the still image,
The motion trajectory detection unit further detects the length of the motion trajectory of the person of interest and the movement pattern of the person of interest,
Furthermore, the importance of each of the plurality of still images is determined based on at least one of the length of the movement locus of the target person, the position of the target person in the still image, and the size of the target person in the still image. An importance determination unit that calculates an evaluation value of the importance based on the determined importance for each of the plurality of still images;
Based on at least one of the position of the target person in the still image, the size of the target person in the still image, and the movement pattern of the target person, the quality of each composition of the plurality of still images is analyzed. A composition analysis unit that calculates an evaluation value of the composition based on the quality of the analyzed composition for each of the still images;
Image quality determination that determines the image quality of each of a plurality of still images based on the area of the person of interest in the still image, and calculates an evaluation value of the image quality based on the determined image quality for each of the plurality of still images Equipped with
The still image data output unit is configured to evaluate at least one of the evaluation value for the motion of the person of interest, the evaluation value of importance, the evaluation value of the composition, and the evaluation value of the image quality among the plurality of still images. It is preferable to output still image data of one or more still images whose overall evaluation value is greater than or equal to a threshold value.
　また、構図分析部は、注目人物の移動パターンをあらかじめ定義しておき、運動軌跡検出部により検出された注目人物の運動軌跡の中から、あらかじめ定義された移動パターンで注目人物が移動している部分を検出し、あらかじめ定義された移動パターンで注目人物が移動している部分に対応する静止画像の構図はよいと分析し、よいと分析された静止画像の構図の評価値が、よいと分析されていない静止画像の構図の評価値よりも高くなるように算出するものであることが好ましい。 In addition, the composition analysis unit defines in advance the movement pattern of the person of interest, and the person of interest is moved with the movement pattern defined in advance from among the movement loci of the person of interest detected by the movement locus detection unit. A portion is detected, and the composition of the still image corresponding to the portion where the noted person is moving is analyzed as good according to a predefined movement pattern, and the evaluation value of the composition of the still image analyzed as good is analyzed as good. It is preferable that the calculation is performed so as to be higher than the evaluation value of the composition of the still image which is not performed.
　また、注目人物検出部は、さらに、静止画像における注目人物の顔の向きを検出するものであり、
　さらに、注目人物検出部により検出された注目人物の顔の向きに基づいて、静止画像データ出力部から出力された静止画像データに対応する静止画像の天地が、動画像が撮影された時の撮影装置の天地と同じになるように、静止画像データ出力部から出力された静止画像データに対応する静止画像の天地を補正する天地補正部を備えることが好ましい。 Further, the target person detection unit further detects the orientation of the target person's face in the still image,
Furthermore, based on the orientation of the face of the target person detected by the target person detection unit, the top and bottom of the still image corresponding to the still image data output from the still image data output unit is captured when the moving image is captured It is preferable to have a top-bottom correction unit that corrects the top and bottom of the still image corresponding to the still image data output from the still image data output unit so as to be the same as the top and bottom of the device.
　また、本発明は、静止画像データ抽出部が、動画像データから、複数のフレームの静止画像データを抽出するステップと、
　注目人物検出部が、複数のフレームの静止画像データに対応する複数枚の静止画像の各々の中から、処理対象とする人物である注目人物を検出するステップと、
　運動軌跡検出部が、複数枚の静止画像における注目人物の検出結果に基づいて、動画像データに対応する動画像における注目人物の動きを追跡して注目人物の運動軌跡を検出するステップと、
　動作分析部が、注目人物の運動軌跡に基づいて、動画像における注目人物の動作を分析し、複数枚の静止画像の各々について、分析された注目人物の動作に基づいて、注目人物の動作に対する評価値を算出するステップと、
　静止画像データ出力部が、複数のフレームの静止画像データの中から、注目人物の動作に対する評価値が閾値以上である静止画像の静止画像データを出力するステップとを含む画像処理方法を提供する。 Further, according to the present invention, the still image data extraction unit extracts still image data of a plurality of frames from moving image data;
Detecting a noted person who is a person to be processed from each of a plurality of still images corresponding to still image data of a plurality of frames;
The motion locus detection unit detects the movement locus of the target person by tracking the movement of the target person in the moving image corresponding to the moving image data based on the detection result of the target person in the plurality of still images;
The motion analysis unit analyzes the motion of the watched person in the moving image based on the motion trajectory of the watched person, and for each of the plurality of still images, the motion of the watched person with respect to the motion of the watched person based on the analyzed motion of the watched person. Calculating an evaluation value;
The still image data output unit provides an image processing method including the steps of outputting, among still image data of a plurality of frames, still image data of a still image whose evaluation value for the motion of the person of interest is equal to or greater than a threshold.
　さらに、注目人物検出部が、静止画像における注目人物の位置、静止画像における注目人物の大きさ、および、静止画像における注目人物の領域を検出するステップと、
　運動軌跡検出部が、注目人物の運動軌跡の長さ、および、注目人物の移動パターンを検出するステップと、
　重要度判定部が、注目人物の運動軌跡の長さ、静止画像における注目人物の位置、静止画像における注目人物の大きさのうちの少なくとも１つに基づいて、複数枚の静止画像の各々の重要度を判定し、複数枚の静止画像の各々について、判定された重要度に基づいて、重要度の評価値を算出するステップと、
　構図分析部が、静止画像における注目人物の位置、静止画像における注目人物の大きさ、注目人物の移動パターンのうちの少なくとも１つに基づいて、複数枚の静止画像の各々の構図の良否を分析し、複数枚の静止画像の各々について、分析された構図の良否に基づいて、構図の評価値を算出するステップと、
　画質判定部が、静止画像における注目人物の領域に基づいて、複数枚の静止画像の各々の画質を判定し、複数枚の静止画像の各々について、判定された画質に基づいて、画質の評価値を算出するステップと、
　静止画像データ出力部が、複数枚の静止画像の中から、注目人物の動作に対する評価値と、重要度の評価値、構図の評価値、および、画質の評価値のうちの少なくとも１つの評価値との総合評価値が閾値以上である１枚以上の静止画像の静止画像データを出力するステップとを含むことが好ましい。 And detecting the position of the target person in the still image, the size of the target person in the still image, and the region of the target person in the still image;
The motion trajectory detection unit detects the length of the motion trajectory of the person of interest and the movement pattern of the person of interest;
The importance determination unit determines the importance of each of the plurality of still images based on at least one of the length of the motion locus of the target person, the position of the target person in the still image, and the size of the target person in the still image. Determining the degree of importance, and calculating an evaluation value of the degree of importance based on the degree of importance determined for each of the plurality of still images;
The composition analysis unit analyzes the quality of each of the plurality of still images based on at least one of the position of the target person in the still image, the size of the target person in the still image, and the movement pattern of the target person. Calculating an evaluation value of the composition based on the quality of the analyzed composition for each of a plurality of still images;
The image quality determination unit determines the image quality of each of the plurality of still images based on the area of the person of interest in the still image, and the image quality evaluation value based on the determined image quality for each of the plurality of still images. Calculating the
The still image data output unit, among a plurality of still images, at least one of the evaluation value for the motion of the person of interest, the evaluation value of importance, the evaluation value of composition, and the evaluation value of image quality Outputting still image data of one or more still images whose overall evaluation value is greater than or equal to a threshold value.
　さらに、注目人物検出部が、静止画像における注目人物の顔の向きを検出するステップと、
　天地補正部が、注目人物検出部により検出された注目人物の顔の向きに基づいて、静止画像データ出力部から出力された静止画像データに対応する静止画像の天地が、動画像が撮影された時の撮影装置の天地と同じになるように、静止画像データ出力部から出力された静止画像データに対応する静止画像の天地を補正するステップとを含むことが好ましい。 And detecting the direction of the face of the person of interest in the still image.
The moving image was taken with the still image corresponding to the still image data output from the still image data output unit based on the face orientation of the noted person detected by the noted person detection unit by the elevation correction unit And correcting the top and bottom of the still image corresponding to the still image data output from the still image data output unit so as to be the same as the top and bottom of the photographing apparatus at the time.
　また、本発明は、上記に記載の画像処理方法の各々のステップをコンピュータに実行させるためのプログラムを提供する。 The present invention also provides a program for causing a computer to execute the steps of the image processing method described above.
　また、本発明は、上記に記載の画像処理方法の各々のステップをコンピュータに実行させるためのプログラムが記録されたコンピュータ読み取り可能な記録媒体を提供する。 The present invention also provides a computer readable recording medium having recorded thereon a program for causing a computer to execute the steps of the image processing method described above.
　本発明によれば、動画像における注目人物の動作に対する評価値、もしくは、注目人物の動作に対する評価値と、静止画像の重要度の評価値、構図の評価値および画質の評価値のうちの少なくとも１つの評価値との総合評価値に基づいて、動画像の中からベストショットのシーンを自動で検出し、動画像データから抽出された複数のフレームの静止画像データの中から、ベストショットのシーンに対応する静止画像の静止画像データを出力することができる。 According to the present invention, at least one of the evaluation value for the motion of the target person in the moving image or the evaluation value for the motion of the target person, the evaluation value of the importance of the still image, the evaluation value of the composition and the evaluation value of the image quality Based on the overall evaluation value with one evaluation value, the scene of the best shot is automatically detected from the moving image, and the scene of the best shot from still image data of a plurality of frames extracted from the moving image data Still image data corresponding to the still image can be output.
本発明の画像処理装置の構成を表す一実施形態のブロック図である。It is a block diagram of one embodiment showing composition of an image processing device of the present invention. （Ａ）～（Ｃ）の左側は、注目人物の運動軌跡を表す一例の概念図、右側は、注目人物の動作履歴画像を表す一例の概念図である。The left side of (A) to (C) is a conceptual diagram of an example showing the movement locus of the noted person, and the right side is a conceptual diagram of an example showing the movement history image of the noted person. （Ａ）は、左に９０°回転された静止画像を表す一例の概念図、（Ｂ）は、（Ａ）に示す静止画像を右に９０°回転させて天地が補正された静止画像を表す一例の概念図である。(A) is a conceptual diagram of an example showing a still image rotated 90 ° to the left, (B) shows a still image corrected to the top and bottom by rotating the still image shown in (A) 90 ° to the right It is a conceptual diagram of an example. 図１に示す画像処理装置の動作を表す一例のフローチャートである。It is a flowchart of an example showing operation | movement of the image processing apparatus shown in FIG. 動画像から全てのフレームの静止画像が抽出された様子を表す一例の概念図である。It is a conceptual diagram of an example showing signs that the still picture of all the frames was extracted from a moving picture. 図５に示す全てのフレームの静止画像の各々から検出された人物の領域が枠で囲まれた様子を表す一例の概念図である。It is a conceptual diagram of an example showing a mode that the area | region of the person detected from each of the still image of all the frames shown in FIG. 5 was surrounded by the frame. 動画像から抽出された全てのフレームの静止画像の各々の総合評価値を表す一例のグラフである。It is a graph of an example showing the comprehensive evaluation value of each of a still picture of all the frames extracted from a moving picture. 図５に示す全てのフレームの静止画像の中から、総合評価が閾値以上である静止画像に星印が付与された様子を表す一例の概念図である。It is a conceptual diagram of an example showing a mode that the star mark was provided to the still image whose comprehensive evaluation is more than a threshold value among the still images of all the frames shown in FIG.
　以下に、添付の図面に示す好適実施形態に基づいて、本発明の画像処理装置、画像処理方法、プログラムおよび記録媒体を詳細に説明する。 Hereinafter, an image processing apparatus, an image processing method, a program, and a recording medium of the present invention will be described in detail based on preferred embodiments shown in the attached drawings.
　図１は、本発明の画像処理装置の構成を表す一実施形態のブロック図である。同図に示す画像処理装置１０は、動画像からベストショットのシーンを自動で検出し、ベストショットのシーンに対応する静止画像の静止画像データを出力するものであり、注目人物登録部１２と、静止画像データ抽出部１４と、注目人物検出部１６と、運動軌跡検出部１８と、動作分析部２０と、重要度判定部２２と、構図分析部２４と、画質判定部２６と、静止画像データ出力部２８と、天地補正部３０とを備えている。 FIG. 1 is a block diagram of an embodiment showing the configuration of the image processing apparatus of the present invention. The image processing apparatus 10 shown in the figure automatically detects a scene of a best shot from a moving image, and outputs still image data of a still image corresponding to the scene of the best shot. Still image data extraction unit 14, attention person detection unit 16, motion trajectory detection unit 18, motion analysis unit 20, importance determination unit 22, composition analysis unit 24, image quality determination unit 26, still image data An output unit 28 and a top-bottom correction unit 30 are provided.
　注目人物登録部１２は、動画像データに対応する動画像に撮影された人物のうち、処理対象とする注目人物を登録人物として登録するものである。 The target person registration unit 12 registers a target person to be processed as a registered person among persons photographed in a moving image corresponding to moving image data.
　注目人物登録部１２は、例えば、動画像に撮影された人物のうち、ユーザにより指定された人物を登録人物として登録することができる。また、注目人物登録部１２は、登録人物の画像（注目人物を特定するための顔画像等）を登録しておくことができる。 The notable person registration unit 12 can register, for example, a person designated by the user among persons photographed in a moving image as a registered person. In addition, the focused person registration unit 12 can register an image of a registered person (a face image or the like for specifying the focused person).
　続いて、静止画像データ抽出部１４は、動画像データから、複数のフレームの静止画像データを抽出するものである。 Subsequently, the still image data extraction unit 14 extracts still image data of a plurality of frames from the moving image data.
　静止画像データ抽出部１４は、例えば、動画像データの全てのフレーム（各々のフレーム）の静止画像データを抽出することができる。しかし、本発明はこれに限定されず、一定数のフレーム毎に、例えば、２フレーム毎に１つのフレームの静止画像データを抽出してもよい。また、動画像データに対応する動画像の任意の区間のフレームの静止画像データのみを抽出してもよい。 The still image data extraction unit 14 can extract, for example, still image data of all frames (each frame) of moving image data. However, the present invention is not limited to this, and still image data of one frame may be extracted, for example, every two frames, for each fixed number of frames. Alternatively, only still image data of a frame of an arbitrary section of a moving image corresponding to moving image data may be extracted.
　続いて、注目人物検出部１６は、静止画像データ抽出部１４により動画像データから抽出された複数のフレームの静止画像データに対応する複数枚の静止画像の各々の中から、処理対象とする人物である注目人物を検出するものである。 Subsequently, the focused person detection unit 16 selects a person to be processed from among each of a plurality of still images corresponding to still image data of a plurality of frames extracted from moving image data by the still image data extraction unit 14. The person of interest is detected.
　注目人物検出部１６は、例えば、複数枚の静止画像の各々において、人物の有無を検出し、検出された人物の画像と、例えば、注目人物登録部１２に登録された登録人物の画像とを比較（顔画像等を比較）することにより、検出された人物の中から、登録人物に一致ないし類似した人物（類似度が閾値以上の人物）を、注目人物として特定することができる。 The noted person detection unit 16 detects, for example, the presence or absence of a person in each of a plurality of still images, and detects an image of the detected person and, for example, an image of a registered person registered in the noted person registration unit 12. By comparing (compare face images and the like), it is possible to specify a person who matches or is similar to the registered person (a person whose similarity is equal to or higher than a threshold) among the detected persons as the noted person.
　あるいは、注目人物検出部１６は、複数枚の静止画像の各々の中から人物の顔を抽出し、抽出された人物の顔の顔画像に対し中心人物判定を行うことにより、顔が抽出された人物の中から、中心人物判定により中心人物であると判定された人物を、注目人物として特定することができる。 Alternatively, the target person detection unit 16 extracts the face of the person from each of the plurality of still images, and the face person is extracted by performing the central person determination on the face image of the extracted face of the person. From among the persons, the person determined to be the center person by the center person determination can be identified as the person of interest.
　中心人物判定では、例えば、複数枚の顔画像に対して同一人物判定処理が行われ、複数枚の顔画像が、同一人物の顔画像からなる画像群に分類される。続いて、画像群に分類された人物のうちの１以上の人物が主人公として決定され、主人公以外の人物のうち主人公と関連性の高い１以上の人物が重要人物として判定される。
　また、注目人物登録部１２に登録された各々の登録人物の顔画像に基づいて、各々の画像群に対応する人物を特定することができる。 In the center person determination, for example, the same person determination process is performed on a plurality of face images, and the plurality of face images are classified into an image group including face images of the same person. Subsequently, one or more persons of the persons classified into the image group are determined as the main character, and one or more persons having high relevancy with the main character among the persons other than the main character are determined as the important persons.
Further, based on the face image of each registered person registered in the focused person registration unit 12, the person corresponding to each image group can be specified.
　例えば、顔画像の検出数が最も多い人物を主人公として決定したり、主人公以外の人物のうち、主人公と共に撮影された静止画像の数が多い人物を重要人物と判定したりすることができる。
　また、同一の静止画像に撮影された主人公の顔画像と主人公以外の人物の顔画像の間の距離を算出し、顔画像の間の距離が近い人物を重要人物と判定してもよい。
　主人公が撮影された静止画像の撮影日時情報と主人公以外の人物が撮影された静止画像の撮影日時情報との差分、および主人公が撮影された静止画像の撮影位置情報と主人公以外の人物が撮影された静止画像の撮影位置情報との差分の一方もしくは両方に基づいて重要人物を判定してもよい。 For example, a person with the largest number of face images detected can be determined as the main character, or a person other than the main character with a large number of still images photographed with the main character can be determined as the important person.
In addition, the distance between the face image of the main character and the face image of a person other than the main character photographed in the same still image may be calculated, and a person whose distance between the face images is close may be determined as an important person.
The difference between the shooting date and time information of the still image in which the main character was shot and the shooting date and time information of the still image in which the person other than the main character was shot, and the shooting position information of the still image in which the main character was shot and the person other than the main character The important person may be determined based on one or both of the difference from the still image shooting position information.
　また、注目人物検出部１６は、静止画像における、注目人物の位置、注目人物の大きさ、注目人物の領域、注目人物の上半身の領域、注目人物の顔の位置、注目人物の顔の大きさ、注目人物の顔領域、注目人物の顔の向き等を検出することができる。 Also, the focused person detection unit 16 detects the position of the focused person, the size of the focused person, the area of the focused person, the upper body area of the focused person, the position of the face of the focused person, and the size of the face of the focused person in the still image. The face area of the person of interest, the direction of the face of the person of interest, etc. can be detected.
　なお、静止画像における注目人物の検出方法、注目人物の顔の検出方法等は公知であるから、ここでは詳細な説明は省略するが、その具体的な検出方法は何ら限定されない。また、注目人物の検出方法も何ら限定されない。 In addition, since the detection method of the attention person in a still image, the detection method of the face of the attention person, etc. are known, detailed explanation is omitted here, but the specific detection method is not limited at all. Also, the method of detecting the person of interest is not limited at all.
　続いて、運動軌跡検出部１８は、注目人物検出部１６による、複数枚の静止画像における注目人物の検出結果に基づいて、動画像データに対応する動画像における注目人物の動きを追跡して注目人物の運動軌跡を検出するものである。また、運動軌跡検出部１８は、注目人物の運動軌跡を検出することにより、注目人物の運動軌跡の長さや、注目人物の移動パターン等を検出することができる。 Subsequently, the motion locus detection unit 18 tracks the movement of the target person in the moving image corresponding to the moving image data based on the detection result of the target person in the plurality of still images by the target person detection unit 16. It detects the movement locus of a person. Further, the motion locus detection unit 18 can detect the length of the motion locus of the person of interest, the movement pattern of the person of interest, and the like by detecting the motion locus of the person of interest.
　ここで、注目人物の運動軌跡は、関心領域（ＲＯＩ：Region of Interest）、例えば、図２（Ａ）～（Ｃ）の左側に示すように、注目人物の顔領域が移動した軌跡をライン状に表したものを使用することができる。また、注目人物の運動軌跡として、同図（Ａ）～（Ｃ）の右側に示すように、動作履歴画像（ＭＨＩ：Motion History Image）を使用してもよい。動作履歴画像は、注目人物の動作の履歴を、例えば、一定時間毎に色を変えて表したものである。動作履歴画像を利用することにより、動作履歴画像における、注目人物の位置、注目人物の大きさ、注目人物の移動箇所、注目人物の移動方向等を知ることができる。 Here, the movement locus of the person of interest is a region of interest (ROI: Region of Interest), for example, as shown on the left side of FIGS. It is possible to use the ones represented in. Also, as shown in the right side of the figures (A) to (C), a motion history image (MHI: Motion History Image) may be used as the motion trajectory of the person of interest. The action history image is a view showing the action history of the person of interest, for example, by changing the color at regular intervals. By using the operation history image, it is possible to know the position of the target person, the size of the target person, the moving position of the target person, the moving direction of the target person, and the like in the operation history image.
　運動軌跡検出部１８は、例えば、注目人物の顔領域に基づいて、現在のフレームの静止画像における注目人物の顔領域と、その次のフレームの静止画像における注目人物の顔領域に相当する任意の位置の検出領域とを比較し、現在のフレームの静止画像における注目人物の顔領域との類似度が閾値以上である、次のフレームの静止画像における検出領域の位置に基づいて、現在のフレームの静止画像における注目人物の顔領域が、その次のフレームの静止画像のどの位置の検出領域に移動しているかを検出することにより、動画像における注目人物の動きを追跡することができる。 The motion locus detection unit 18 is, for example, based on the face area of the noted person, an arbitrary face corresponding to the face area of the noted person in the still image of the current frame and the face area of the noted person in the still image of the next frame. Based on the position of the detection area in the still image of the next frame, the similarity with the face area of the target person in the still image of the current frame is compared with the detection area in the position and the position detection area is compared. It is possible to track the movement of the target person in the moving image by detecting to which detection region of the still image of the next frame the face region of the target person in the still image has moved.
　ここで、注目人物の顔領域を検出するだけでは、静止画像における注目人物の位置、注目人物の大きさ等が時間の経過に伴って変化することにより、注目人物の動きを追跡することが困難となる場合がある。この場合、注目人物の顔領域に加えて、注目人物の上半身の領域を一定数、例えば、４つの領域に分割し、合計５つの領域のそれぞれについて、同様に注目人物の動きを追跡することにより、追跡の成功率を向上させることができる。 Here, it is difficult to track the movement of the target person by detecting the face area of the target person by changing the position of the target person in the still image, the size of the target person, etc. with the passage of time. It may be In this case, in addition to the face region of the person of interest, the upper body region of the person of interest is divided into a fixed number, for example, four regions, and the movement of the person of interest is similarly tracked for each of the five regions in total. , Can improve the tracking success rate.
　また、現在のフレームの静止画像における注目人物の顔領域と、その次のフレームの静止画像における検出領域との類似度を求める場合、次のフレームの静止画像において、現在のフレームの静止画像における注目人物の顔領域に対応する位置の検出領域を検出するために、任意の位置の検出領域内に含まれる全ての画素の輝度値の総和を算出することを、複数の位置の検出領域について順次繰り返す必要がある。そのため、各々のフレーム毎に輝度値の総和を計算するための計算量は膨大となる。 In addition, when the similarity between the face area of the person of interest in the still image of the current frame and the detection area in the still image of the next frame is determined, the attention in the still image of the current frame in the still image of the next frame In order to detect a detection area at a position corresponding to the face area of a person, calculating the sum of luminance values of all pixels included in the detection area at an arbitrary position is sequentially repeated for detection areas at a plurality of positions. There is a need. Therefore, the amount of calculation for calculating the sum of luminance values for each frame is enormous.
　この場合、次のフレーム（つまり、各々のフレーム）の静止画像の積分画像を生成し、生成された積分画像を利用して輝度値の総和を算出することにより、その計算量を削減し、処理を高速化することができる。積分画像とは、例えば、静止画像の画素の座標が、静止画像の左から右へ、かつ、上から下へ向かうに従って大きくなるとすると、各々の座標の画素が、左上の画素から各々の座標の画素までの輝度値の積分値を持つ画像である。 In this case, the integral image of the still image of the next frame (that is, each frame) is generated, and the total amount of luminance values is calculated using the generated integral image, thereby reducing the amount of calculation and processing. Can be speeded up. In the integral image, for example, assuming that the coordinates of the pixels of the still image increase from left to right and from top to bottom of the still image, the pixel at each coordinate is from the pixel at the upper left to each coordinate It is an image having an integral value of luminance values up to the pixel.
　なお、積分画像を利用して、注目人物の顔領域に相当する領域内に含まれる全ての画素の輝度値の総和を算出する方法は公知であるから、ここでは、その詳細な説明は省略する。また、注目人物の動きを追跡する場合に、計算量の削減や、処理の高速化の目的のために、積分画像を利用することは限定されず、例えば、平均変位（Mean Shift）法等の各種の方法を利用することができる。平均変位法についても公知であるから、その詳細な説明は省略する。 In addition, since a method of calculating the sum of the luminance values of all the pixels included in the area corresponding to the face area of the person of interest using the integral image is known, the detailed description thereof is omitted here. . In addition, when tracking the movement of the person of interest, the use of the integral image is not limited for the purpose of reduction of calculation amount and speeding up of processing, for example, mean shift (Mean Shift) method etc. Various methods can be used. Since the mean displacement method is also known, its detailed description is omitted.
　続いて、動作分析部２０は、運動軌跡検出部１８により検出された注目人物の運動軌跡、例えば、顔領域等の関心領域の運動軌跡に基づいて、動画像における注目人物の動作を分析し、複数枚の静止画像の各々について、分析された注目人物の動作に基づいて、注目人物の動作に対する評価値を算出するものである。 Subsequently, the motion analysis unit 20 analyzes the motion of the target person in the moving image, based on the motion track of the target person detected by the motion track detection unit 18, for example, the motion track of the region of interest such as a face region, For each of the plurality of still images, the evaluation value for the motion of the target person is calculated based on the analyzed motion of the target person.
　動作分析部２０は、例えば、注目人物の動作に対する運動軌跡、例えば、注目人物が走っている時の運動軌跡をあらかじめ定義しておき、運動軌跡検出部１８により検出された注目人物の運動軌跡の中から、あらかじめ定義された運動軌跡と類似した部分を検出することにより、注目人物の動作を分析する。そして、注目人物の動作が、走っている動作である場合には、評価値がいくつというように、注目人物の動作の種類に応じて、注目人物の動作に対する評価値を算出することができる。 The motion analysis unit 20 defines, for example, a motion trajectory for the motion of the person of interest, for example, a motion trajectory of when the person of interest is running, and the motion trajectory of the person of interest detected by the motion trajectory detection unit 18. From the inside, the motion of the person of interest is analyzed by detecting a portion similar to the previously defined movement trajectory. Then, when the motion of the target person is a running motion, it is possible to calculate the evaluation value for the motion of the target person according to the type of the motion of the target person, such as what evaluation value.
　また、動作分析部２０は、注目人物の運動軌跡として、図２（Ａ）～（Ｃ）の右側に示すような動作履歴画像に基づいて、注目人物の動作を分析し、注目人物の動作に対する評価値を算出することができる。 Further, the motion analysis unit 20 analyzes the motion of the target person based on the motion history image as shown on the right side of FIGS. 2A to 2C as the motion locus of the target person, and the motion of the target person is obtained. An evaluation value can be calculated.
　動作分析部２０は、動作履歴画像に基づいて注目人物の動作を分析することにより、図２（Ａ）の右側に示すように、注目人物が、同図中、右側から左側へ走っていること、同図（Ｂ）の右側に示すように、注目人物が、静止した状態で右手だけを動かしていること、同図（Ｃ）の右側に示すように、注目人物が、地面に落ちているものを拾っていること等を認識することができる。また、注目人物が、動いているのか否か、どの位置で、どの方向へ動いているか等に基づいて、注目人物の動作に対する評価値を算出することができる。 The motion analysis unit 20 analyzes the motion of the noted person based on the motion history image, so that the noted person is running from the right to the left in the figure as shown on the right of FIG. 2 (A). As shown on the right side of the figure (B), the noted person is moving only the right hand while standing still, and as shown on the right side of the figure (C), the noted person is falling on the ground It can be recognized that something is picked up. In addition, it is possible to calculate an evaluation value for the movement of the person of interest based on whether or not the person of interest is moving, at which position, in which direction, or the like.
　続いて、重要度判定部２２は、注目人物の運動軌跡の長さ、静止画像における注目人物の位置、静止画像における注目人物の大きさのうちの少なくとも１つに基づいて、複数枚の静止画像の各々の重要度を判定し、複数枚の静止画像の各々について、判定された重要度に基づいて、重要度の評価値を算出するものである。 Subsequently, the importance degree determination unit 22 determines a plurality of still images based on at least one of the length of the motion locus of the target person, the position of the target person in the still image, and the size of the target person in the still image. The degree of importance of each of the plurality of still images is determined, and the evaluation value of the degree of importance is calculated based on the determined degree of importance for each of the plurality of still images.
　例えば、注目人物の運動軌跡が長い場合（長さが閾値以上である場合）、その注目人物に対する撮影者の関心度は高いと推測できる。そのため、重要度判定部２２は、動画像の中で、注目人物の運動軌跡が長いシーンに対応する静止画像の重要度が高いと判定する。また、注目人物が中央部に撮影されている静止画像や、注目人物が大きく撮影されている（注目人物の大きさが閾値以上である）静止画像の重要度は高いと判定する。そして、重要度が高くなるほど、重要度の評価値が高くなるように算出する。 For example, when the motion locus of the person of interest is long (when the length is equal to or greater than the threshold), it can be estimated that the photographer's interest in the person of interest is high. Therefore, the importance degree determination unit 22 determines that, in the moving image, the importance degree of the still image corresponding to the scene in which the movement locus of the target person is long is high. In addition, it is determined that the still image in which the person of interest is photographed in the central part and the still image in which the person of interest is largely photographed (the size of the person of interest is equal to or larger than the threshold) have high importance. Then, the evaluation value of the importance is calculated to be higher as the importance is higher.
　続いて、構図分析部２４は、静止画像における注目人物の位置、静止画像における注目人物の大きさ、注目人物の移動パターンのうちの少なくとも１つに基づいて、複数枚の静止画像の各々の構図の良否を分析し、複数枚の静止画像の各々について、分析された構図の良否に基づいて、構図の評価値を算出するものである。 Subsequently, the composition analysis unit 24 composes each of the plurality of still images based on at least one of the position of the target person in the still image, the size of the target person in the still image, and the movement pattern of the target person. The quality of the image is analyzed, and the evaluation value of the composition is calculated based on the quality of the analyzed composition for each of the plurality of still images.
　構図分析部２４は、例えば、注目人物が中央部に撮影されている静止画像や、注目人物が大きく撮影されている（注目人物の大きさが閾値以上である）静止画像の構図は、注目人物が中央部に撮影されていない静止画像や、注目人物が大きく撮影されていない静止画像の構図よりもよいと分析する。そして、よいと分析された静止画像の構図の評価値が、よいと分析されていない静止画像の構図の評価値よりも高くなるように算出することができる。 The composition analysis unit 24 may, for example, have a still image in which the person of interest is photographed at the center, or a composition of still images in which the person of interest is largely photographed (the size of the person of interest is equal to or larger than the threshold). Is analyzed to be better than the composition of a still image not captured in the central part or a still image in which the person of interest is not captured appreciably. Then, the evaluation value of the composition of the still image analyzed as good can be calculated to be higher than the evaluation value of the composition of the still image not analyzed as good.
　また、構図分析部２４は、注目人物の移動パターン、例えば、動画像の左端から右端まで注目人物が移動する移動パターンをあらかじめ定義しておき、運動軌跡検出部１８により検出された注目人物の運動軌跡の中から、あらかじめ定義された移動パターンで注目人物が移動している部分を検出する。そして、あらかじめ定義された移動パターンで注目人物が移動している部分に対応する静止画像の構図はよいと分析し、よいと分析された静止画像の構図の評価値が、よいと分析されていない静止画像の構図の評価値よりも高くなるように算出することができる。 In addition, the composition analysis unit 24 defines in advance a movement pattern of the target person, for example, a movement pattern in which the target person moves from the left end to the right end of the moving image. From the trajectory, a portion in which the person of interest is moving is detected by a predefined movement pattern. Then, it analyzes that the composition of the still image corresponding to the portion where the noted person is moving with the predefined movement pattern is good, and the evaluation value of the composition of the still image analyzed as good is not analyzed as good. It can be calculated to be higher than the evaluation value of the composition of the still image.
　続いて、画質判定部２６は、静止画像における注目人物の領域、例えば、顔領域等の関心領域に基づいて、複数枚の静止画像の各々の画質を判定し、複数枚の静止画像の各々について、判定された画質に基づいて、画質の評価値を算出するものである。 Subsequently, the image quality determination unit 26 determines the image quality of each of the plurality of still images based on the region of interest person in the still image, for example, the region of interest such as the face region. The evaluation value of the image quality is calculated based on the determined image quality.
　動画像から抽出された静止画像は、動画像データの圧縮方式によって、画質がよい場合も悪い場合もある。また、ピンボケ、手ぶれ等によって、静止画像にボケやブレが発生している場合や、輝度、色合い、コントラスト等が適切ではない場合もある。しかし、背景等の画質が悪い場合でも、関心領域である注目人物の顔領域や体の領域の画質がよい場合、画質判定部２６は、その静止画像の画質はよいと判定する。そして、画質がよいと判定された静止画像について、画質がよいほど、画質の評価値が高くなるように算出することができる。 The still image extracted from the moving image may or may not have high image quality depending on the compression method of the moving image data. In addition, blurring or blurring may occur in the still image due to defocusing or camera shake, or the luminance, color tone, contrast or the like may not be appropriate. However, even if the image quality of the background or the like is poor, if the image quality of the face area or body area of the target person who is the region of interest is good, the image quality judgment unit 26 judges that the image quality of the still image is good. Then, for still images judged to have good image quality, the evaluation value of image quality can be calculated to be higher as the image quality is better.
　続いて、静止画像データ出力部２８は、静止画像データ抽出部１４により動画像データから抽出された複数のフレームの静止画像データの中から、ベストショットのシーンに対応する静止画像の静止画像データとして、注目人物の動作に対する評価値が、もしくは、注目人物の動作に対する評価値と、重要度の評価値、構図の評価値、および、画質の評価値のうちの少なくとも１つの評価値との総合評価値が閾値以上である静止画像の静止画像データを出力するものである。 Subsequently, the still image data output unit 28 selects still image data of a still image corresponding to the scene of the best shot among still image data of a plurality of frames extracted from the moving image data by the still image data extraction unit 14 The evaluation value for the motion of the person of interest, or the evaluation value for the motion of the person of interest, and the evaluation value of importance, the evaluation value of composition, and the evaluation value of at least one of the evaluation values of image quality Still image data of a still image whose value is equal to or greater than a threshold is output.
　最後に、天地補正部３０は、注目人物検出部１６により検出された注目人物の顔の向きに基づいて、静止画像データ出力部２８から出力された静止画像データに対応する静止画像の天地が、動画像が撮影された時の撮影装置の天地と同じになるように、静止画像データ出力部２８から出力された静止画像データに対応する静止画像の天地を補正するものである。 Finally, based on the orientation of the face of the target person detected by the target person detection unit 16, the top / bottom correction unit 30 determines the top / bottom position of the still image corresponding to the still image data output from the still image data output unit 28 The top and bottom of the still image corresponding to the still image data output from the still image data output unit 28 is corrected so as to be the same as the top and bottom of the photographing apparatus when the moving image is captured.
　図３（Ａ）は、左に９０°回転された静止画像を表す一例の概念図である。このような静止画像は、動画像を撮影する時に、撮影装置を右に９０°回転させて撮影することにより得られる。天地補正部３０は、静止画像の天地が、動画像が撮影された時の撮影装置の天地と同じになるように、同図（Ａ）に示す静止画像を右に９０°回転させることにより、同図（Ｂ）に示すように、静止画像の天地を補正することができる。 FIG. 3A is a conceptual view of an example showing a still image rotated 90 degrees to the left. Such a still image can be obtained by rotating the imaging device by 90 degrees to the right when capturing a moving image. The top-bottom correction unit 30 rotates the still image shown in FIG. 6A by 90 ° to the right so that the top and bottom of the still image are the same as the top and bottom of the imaging device when the moving image is captured. As shown in FIG. 6B, the top and bottom of the still image can be corrected.
　なお、注目人物検出部１６は、注目人物登録部１２に二人以上の人物が登録されている場合、複数枚の静止画像の中から、二人以上の注目人物のそれぞれを検出して、検出された注目人物が誰なのかを順次特定することができる。また、この場合、運動軌跡検出部１８、動作分析部２０、重要度判定部２２、構図分析部２４、画質判定部２６、静止画像データ出力部２８、天地補正部３０は、その二人以上の注目人物のそれぞれについて順次処理を行う。 When two or more persons are registered in the person-of-interest registration unit 12, the person-of-interest detection unit 16 detects and detects each of the two or more persons of interest from a plurality of still images. It is possible to sequentially identify who is the watched person who has been Further, in this case, the motion locus detection unit 18, the motion analysis unit 20, the importance degree determination unit 22, the composition analysis unit 24, the image quality determination unit 26, the still image data output unit 28, and the top-bottom correction unit 30 Processing is sequentially performed on each of the noted persons.
　次に、図４に示すフローチャートを参照して、図１に示す画像処理装置１０の動作を説明する。 Next, the operation of the image processing apparatus 10 shown in FIG. 1 will be described with reference to the flowchart shown in FIG.
　図４のフローチャートに示すように、まず、注目人物登録部１２により、動画像に撮影された人物のうち、例えば、ユーザにより指定された人物が注目人物として登録される（ステップＳ１）。 As shown in the flowchart of FIG. 4, first, among the persons photographed in the moving image, for example, a person designated by the user is registered as an attention person by the attention person registration unit 12 (step S1).
　続いて、静止画像データ抽出部１４により、例えば、動画像データから、全てのフレームの静止画像データが抽出される（ステップＳ２）。つまり、図５に示すように、動画像から、全てのフレームの静止画像が抽出される。 Subsequently, the still image data extraction unit 14 extracts, for example, still image data of all the frames from the moving image data (step S2). That is, as shown in FIG. 5, still images of all frames are extracted from the moving image.
　なお、動画像データから静止画像データが抽出された後、注目人物の登録を行ってもよい。 Note that after still image data is extracted from moving image data, registration of a target person may be performed.
　続いて、注目人物検出部１６により、静止画像データ抽出部１４により抽出された全てのフレームの静止画像の各々の中から、注目人物登録部１２に登録された注目人物が検出される（ステップＳ３）。これにより、全てのフレームの静止画像の各々において、注目人物が特定されるとともに、図６に枠で囲んで示すように、全てのフレームの静止画像の各々における、注目人物の位置、注目人物の大きさ、注目人物の領域等が検出される。 Subsequently, a notable person registered in the notable person registration unit 12 is detected by the notable person detection unit 16 out of each of the still images of all the frames extracted by the still image data extraction unit 14 (step S3). ). As a result, the person of interest is identified in each of the still images of all the frames, and the position of the person of interest in each of the still images of all the frames, as shown by the frame in FIG. The size, the area of the noted person, etc. are detected.
　続いて、運動軌跡検出部１８により、全てのフレームの静止画像における注目人物の検出結果に基づいて、動画像における注目人物の動き、例えば、図６に枠で囲んで示す関心領域の動きが追跡されて注目人物の運動軌跡が検出される（ステップＳ４）。これにより、例えば、図２（Ａ）～（Ｃ）の左側に示すように、注目人物の運動軌跡として、顔領域等の関心領域が移動した軌跡をライン状に表したものや、同図（Ａ）～（Ｃ）の右側に示すような動作履歴画像を得ることができる。 Subsequently, based on the detection results of the target person in the still images of all the frames, the motion locus detection unit 18 tracks the movement of the target person in the moving image, for example, the movement of the region of interest shown by the frame in FIG. The motion locus of the person of interest is detected (step S4). Thus, for example, as shown in the left side of FIGS. 2 (A) to 2 (C), a motion locus of a person of interest representing in a line the locus of movement of a region of interest such as a face region An operation history image as shown on the right of A) to (C) can be obtained.
　続いて、動作分析部２０により、運動軌跡検出部１８により検出された注目人物の運動軌跡に基づいて、動画像における注目人物の動作が分析される。そして、全てのフレームの静止画像の各々について、分析された注目人物の動作に基づいて、注目人物の動作に対する評価値が算出される（ステップＳ５－１）。 Subsequently, the motion analysis unit 20 analyzes the motion of the target person in the moving image based on the motion track of the target person detected by the motion track detection unit 18. Then, for each of the still images of all the frames, an evaluation value for the motion of the target person is calculated based on the analyzed motion of the target person (step S5-1).
　また、重要度判定部２２により、注目人物の運動軌跡の長さ、静止画像における注目人物の位置、注目人物の大きさに基づいて、全ての静止画像の各々の重要度が判定される。そして、全てのフレームの静止画像の各々について、判定された重要度に基づいて、重要度の評価値が算出される（ステップＳ５－２）。 Also, the importance degree determination unit 22 determines the importance degree of each of all still images based on the length of the motion locus of the person of interest, the position of the person of interest in the still image, and the size of the person of interest. Then, the evaluation value of the importance is calculated based on the determined importance for each of the still images of all the frames (step S5-2).
　また、構図分析部２４により、静止画像における注目人物の位置、注目人物の大きさ、注目人物の移動パターンに基づいて、全ての静止画像の各々の構図の良否が分析される。そして、全てのフレームの静止画像の各々について、分析された構図の良否に基づいて、構図の評価値が算出される（ステップＳ５－３）。 Further, the composition analysis unit 24 analyzes the quality of each composition of all still images based on the position of the target person in the still image, the size of the target person, and the movement pattern of the target person. Then, for each of the still images of all the frames, the evaluation value of the composition is calculated based on the quality of the analyzed composition (step S5-3).
　また、画質判定部２６により、静止画像における注目人物の領域に基づいて、全てのフレームの静止画像の各々の画質が判定される。そして、全ての静止画像の各々について、判定された画質、本実施形態の場合、ボケブレの程度に応じて、画質の評価値が算出される（ステップＳ５－４）。
　例えば、図６に枠で囲んで示す関心領域のボケブレの判定が行われて、ボケブレの程度が大きいほど、画質の評価値が低くなるように算出される。 Also, the image quality determination unit 26 determines the image quality of each of the still images of all the frames based on the area of the person of interest in the still images. Then, for each of all still images, the image quality evaluation value is calculated according to the determined image quality, and in the case of the present embodiment, the degree of blur (step S5-4).
For example, determination of blurring of the region of interest shown by a frame in FIG. 6 is performed, and the evaluation value of the image quality is calculated to be lower as the degree of blurring is larger.
　なお、注目人物の動作に対する評価値、重要度の評価値、構図の評価値、画質の評価値を算出する順序は何ら限定されず、任意の順序で算出することができる。また、これらの評価値を並列に、つまり、同時に算出するようにすることもできる。 The order of calculating the evaluation value of the motion of the person of interest, the evaluation value of importance, the evaluation value of the composition, and the evaluation value of the image quality is not limited at all, and can be calculated in any order. Also, these evaluation values can be calculated in parallel, that is, simultaneously.
　続いて、静止画像データ出力部２８により、静止画像データ抽出部１４により動画像データから抽出された全てのフレームの静止画像データの中から、図７に示すように、ベストショットのシーンに対応する静止画像の静止画像データとして、注目人物の動作に対する評価値、重要度の評価値、構図の評価値、画質の評価値の総合評価値（各評価値の加算値等）が閾値以上である１枚以上の静止画像の静止画像データが出力される（ステップＳ６）。 Subsequently, among still image data of all the frames extracted from the moving image data by the still image data extraction unit 14 by the still image data output unit 28, as shown in FIG. As still image data of a still image, the evaluation value for the motion of the person of interest, the evaluation value of importance, the evaluation value of composition, the comprehensive evaluation value of evaluation value of image quality (addition value of each evaluation value etc.) is above threshold 1 Still image data of at least one still image is output (step S6).
　ここで、図７は、動画像から抽出された全てのフレームの静止画像の各々の総合評価値を表す一例のグラフである。同図の縦軸は、各々の静止画像の総合評価値を、横軸は時間（フレーム）を表す。同図に示すように、全ての静止画像のうちの、注目人物検出部１６により注目人物が検出され、かつ、運動軌跡検出部１８により注目人物の運動軌跡が検出された静止画像の中から、図８に星印を付与して示されているように、総合評価値が閾値以上である静止画像の静止画像データが出力される。 Here, FIG. 7 is a graph of an example showing the comprehensive evaluation value of each of the still images of all the frames extracted from the moving image. The vertical axis of the figure represents the comprehensive evaluation value of each still image, and the horizontal axis represents time (frame). As shown in the figure, out of all the still images, the person of interest is detected by the person of interest detection unit 16 and the motion locus of the person of interest is detected by the motion locus detection unit 18; As shown by adding an asterisk in FIG. 8, still image data of a still image whose overall evaluation value is equal to or greater than a threshold is output.
　そして最後に、天地補正部３０により、注目人物検出部１６により検出された注目人物の顔の向きに基づいて、静止画像の天地が、動画像が撮影された時の撮影装置の天地と同じになるように、静止画像の天地が補正される（ステップＳ７）。 Finally, based on the orientation of the face of the target person detected by the target person detection unit 16 by the top-bottom correction unit 30, the top and bottom of the still image are the same as the top and bottom of the imaging device when the moving image is captured. The top and bottom of the still image is corrected so as to be (step S7).
　上記のように、画像処理装置１０では、例えば、動画像における注目人物の動作に対する評価値、静止画像の重要度の評価値、構図の評価値および画質の評価値を含む総合評価値に基づいて、動画像の中からベストショットのシーンを自動で検出し、動画像データから抽出された全てのフレームの静止画像データの中から、ベストショットのシーンに対応する静止画像の静止画像データを抽出することができる。 As described above, in the image processing apparatus 10, for example, based on the comprehensive evaluation value including the evaluation value for the motion of the target person in the moving image, the evaluation value of the importance of the still image, the evaluation value of the composition and the evaluation value of the image quality. Automatically detects the scene of the best shot from the moving image, and extracts the still image data of the still image corresponding to the scene of the best shot from the still image data of all the frames extracted from the moving image data be able to.
　本発明の装置は、装置が備える各々の構成要素を専用のハードウェアで構成してもよいし、各々の構成要素をプログラムされたコンピュータで構成してもよい。
　本発明の方法は、例えば、その各々のステップをコンピュータに実行させるためのプログラムにより実施することができる。また、このプログラムが記録されたコンピュータ読み取り可能な記録媒体を提供することもできる。 In the device of the present invention, each component of the device may be configured by dedicated hardware, or each component may be configured by a programmed computer.
The method of the present invention can be implemented, for example, by a program for causing a computer to execute each of the steps. It is also possible to provide a computer readable recording medium in which the program is recorded.
　本発明は、基本的に以上のようなものである。
　以上、本発明について詳細に説明したが、本発明は上記実施形態に限定されず、本発明の主旨を逸脱しない範囲において、種々の改良や変更をしてもよいのはもちろんである。 The present invention is basically as described above.
The present invention has been described above in detail, but the present invention is not limited to the above embodiment, and it goes without saying that various improvements and changes may be made without departing from the spirit of the present invention.

Claims (17)

  1. 　動画像データから、複数のフレームの静止画像データを抽出する静止画像データ抽出部と、
    　前記複数のフレームの静止画像データに対応する複数枚の静止画像の各々の中から、処理対象とする人物である注目人物を検出する注目人物検出部と、
    　前記複数枚の静止画像における前記注目人物の検出結果に基づいて、前記動画像データに対応する動画像における前記注目人物の動きを追跡して前記注目人物の運動軌跡を検出する運動軌跡検出部と、
    　前記注目人物の運動軌跡に基づいて、前記動画像における注目人物の動作を分析し、前記複数枚の静止画像の各々について、前記分析された注目人物の動作に基づいて、前記注目人物の動作に対する評価値を算出する動作分析部と、
    　前記複数のフレームの静止画像データの中から、前記注目人物の動作に対する評価値が閾値以上である静止画像の静止画像データを出力する静止画像データ出力部とを備える画像処理装置。     A still image data extraction unit for extracting still image data of a plurality of frames from moving image data;
    A noted person detection unit that detects a noted person who is a person to be processed out of each of a plurality of still images corresponding to still image data of the plurality of frames;
    A motion locus detection unit for tracking the movement of the target person in the moving image corresponding to the moving image data based on the detection result of the target person in the plurality of still images and detecting the movement locus of the target person; ,
    The motion of the target person in the moving image is analyzed based on the motion locus of the target person, and for each of the plurality of still images, the motion of the target person is analyzed based on the motion of the target person analyzed. A motion analysis unit that calculates an evaluation value;
    An image processing apparatus, comprising: a still image data output unit that outputs still image data of a still image having an evaluation value with respect to the motion of the person of interest among a plurality of still image data of the plurality of frames.
  2. 　さらに、前記動画像に撮影された人物のうち、前記処理対象とする人物を登録人物として登録する人物登録部を備え、
    　前記注目人物検出部は、前記複数枚の静止画像の各々の中から、前記登録人物と一致した人物ないし類似度が閾値以上の人物を前記注目人物として検出するものである請求項１に記載の画像処理装置。     And a person registration unit for registering, as a registered person, the person to be processed among the persons photographed in the moving image.
    The noted person detection unit detects a person who matches the registered person or a person whose similarity is equal to or more than a threshold value as the noted person from each of the plurality of still images. Image processing device.
  3. 　前記注目人物検出部は、前記複数枚の静止画像の各々の中から人物の顔を抽出し、前記抽出された人物の顔の顔画像に対し中心人物判定を行うことにより、前記顔が抽出された人物の中から、前記中心人物判定により中心人物であると判定された人物を前記注目人物として検出するものである請求項１に記載の画像処理装置。 The noted person detection unit extracts the face of a person from each of the plurality of still images, and performs the central person determination on the face image of the face of the extracted person to extract the face. The image processing apparatus according to claim 1, wherein a person who is determined as the central person by the central person determination is detected as the noted person among the persons.
  4. 　前記注目人物検出部は、さらに、前記静止画像における注目人物の顔領域を検出するものであり、
    　前記運動軌跡検出部は、前記注目人物の顔領域に基づいて、現在のフレームの静止画像における注目人物の顔領域と、前記現在のフレームの次のフレームの静止画像における注目人物の顔領域に相当する任意の位置の検出領域とを比較し、前記現在のフレームの静止画像における注目人物の顔領域との類似度が閾値以上である、前記次のフレームの静止画像における検出領域の位置に基づいて、前記現在のフレームの静止画像における注目人物の顔領域が、前記次のフレームの静止画像のどの位置の検出領域に移動しているかを検出することにより、前記動画像における注目人物の動きを追跡するものである請求項１～３のいずれか１項に記載の画像処理装置。     The noted person detection unit further detects a face area of the noted person in the still image,
    The motion locus detection unit corresponds to the face region of the person of interest in the still image of the current frame and the face region of the person of interest in the still image of the next frame of the current frame based on the face region of the person of interest Based on the position of the detection area in the still image of the next frame, the similarity of the still image of the current frame with the face area of the target person being equal to or greater than a threshold value. Tracking the movement of the target person in the moving image by detecting to which detection region of the still image of the next frame the face region of the target person in the still image of the current frame is moved The image processing apparatus according to any one of claims 1 to 3, wherein
  5. 　前記運動軌跡検出部は、前記注目人物の顔領域に加えて、前記注目人物の上半身の領域を一定数に分割した領域のそれぞれについて、前記注目人物の動きを追跡するものである請求項４に記載の画像処理装置。 The motion locus detection unit is configured to track the movement of the noted person for each of the regions obtained by dividing the region of the upper body of the noted person into a predetermined number in addition to the face region of the noted person. Image processing apparatus as described.
  6. 　前記運動軌跡検出部は、前記次のフレームの静止画像の積分画像を生成し、前記生成された積分画像を利用して、前記次のフレームの静止画像において、任意の位置の前記検出領域内に含まれる全ての画像の輝度値の総和を算出することを、複数の位置の前記検出領域について順次繰り返すものである請求項４または５に記載の画像処理装置。 The motion trajectory detection unit generates an integral image of a still image of the next frame, and using the generated integral image, the still image of the next frame is detected within the detection area at an arbitrary position. 6. The image processing apparatus according to claim 4, wherein calculating the sum of luminance values of all the included images is sequentially repeated for the detection areas at a plurality of positions.
  7. 　前記運動軌跡検出部は、平均変位法を利用して、前記注目人物の動きを追跡するものである請求項４または５に記載の画像処理装置。 The image processing apparatus according to claim 4, wherein the motion locus detection unit tracks the movement of the noted person using an average displacement method.
  8. 　前記動作分析部は、前記注目人物の動作に対する運動軌跡をあらかじめ定義しておき、前記運動軌跡検出部により検出された注目人物の運動軌跡の中から、前記あらかじめ定義された運動軌跡と類似した部分を検出することにより、前記注目人物の動作を分析し、前記注目人物の動作の種類に応じて、前記注目人物の動作に対する評価値を算出するものである請求項１～７のいずれか１項に記載の画像処理装置。 The motion analysis unit defines in advance a motion trajectory for the motion of the noted person, and a portion similar to the previously-defined motion trajectory among motion trajectories of the noted person detected by the motion trajectory detection unit. 8. The method according to any one of claims 1 to 7, wherein the motion of the noted person is analyzed by detecting the movement of the noted person, and the evaluation value for the motion of the noted person is calculated according to the type of the motion of the noted person. The image processing apparatus according to claim 1.
  9. 　前記動作分析部は、前記注目人物の運動軌跡として、前記注目人物の動作履歴画像に基づいて、前記注目人物の動作を分析し、前記注目人物の動作に対する評価値を算出するものである請求項１～７のいずれか１項に記載の画像処理装置。 The motion analysis unit analyzes the motion of the noted person based on the motion history image of the noted person as the motion locus of the noted person, and calculates an evaluation value for the motion of the noted person. The image processing apparatus according to any one of 1 to 7.
  10. 　前記注目人物検出部は、さらに、前記静止画像における注目人物の位置、前記静止画像における注目人物の大きさ、および、前記静止画像における注目人物の領域を検出するものであり、
    　前記運動軌跡検出部は、さらに、前記注目人物の運動軌跡の長さ、および、前記注目人物の移動パターンを検出するものであり、
    　さらに、前記注目人物の運動軌跡の長さ、前記静止画像における注目人物の位置、前記静止画像における注目人物の大きさのうちの少なくとも１つに基づいて、前記複数枚の静止画像の各々の重要度を判定し、前記複数枚の静止画像の各々について、前記判定された重要度に基づいて、前記重要度の評価値を算出する重要度判定部と、
    　前記静止画像における注目人物の位置、前記静止画像における注目人物の大きさ、前記注目人物の移動パターンのうちの少なくとも１つに基づいて、前記複数枚の静止画像の各々の構図の良否を分析し、前記複数枚の静止画像の各々について、前記分析された構図の良否に基づいて、前記構図の評価値を算出する構図分析部と、
    　前記静止画像における注目人物の領域に基づいて、前記複数枚の静止画像の各々の画質を判定し、前記複数枚の静止画像の各々について、前記判定された画質に基づいて、前記画質の評価値を算出する画質判定部とを備え、
    　前記静止画像データ出力部は、前記複数枚の静止画像の中から、前記注目人物の動作に対する評価値と、前記重要度の評価値、前記構図の評価値、および、前記画質の評価値のうちの少なくとも１つの評価値との総合評価値が閾値以上である１枚以上の静止画像の静止画像データを出力するものである請求項１～９のいずれか１項に記載の画像処理装置。     The noted person detection unit further detects the position of the noted person in the still image, the size of the noted person in the still image, and the area of the noted person in the still image;
    The motion trajectory detection unit further detects a length of a motion trajectory of the noted person and a movement pattern of the noted person,
    Furthermore, each of the plurality of still images is important based on at least one of the length of the movement locus of the noted person, the position of the noted person in the still image, and the size of the noted person in the still image. A degree of importance determining unit that determines the degree of importance and calculates an evaluation value of the degree of importance based on the determined degree of importance for each of the plurality of still images;
    The quality of each of the plurality of still images is analyzed based on at least one of the position of the target person in the still image, the size of the target person in the still image, and the movement pattern of the target person. A composition analysis unit configured to calculate an evaluation value of the composition based on the quality of the analyzed composition for each of the plurality of still images;
    The image quality of each of the plurality of still images is determined based on the area of the person of interest in the still image, and the evaluation value of the image quality is determined based on the determined image quality for each of the plurality of still images. And an image quality determination unit that calculates
    Among the plurality of still images, the still image data output unit includes an evaluation value for the motion of the person of interest, an evaluation value of the importance, an evaluation value of the composition, and an evaluation value of the image quality. The image processing apparatus according to any one of claims 1 to 9, wherein still image data of one or more still images having a total evaluation value with at least one evaluation value of or greater than a threshold is output.
  11. 　前記構図分析部は、前記注目人物の移動パターンをあらかじめ定義しておき、前記運動軌跡検出部により検出された注目人物の運動軌跡の中から、前記あらかじめ定義された移動パターンで注目人物が移動している部分を検出し、前記あらかじめ定義された移動パターンで注目人物が移動している部分に対応する静止画像の構図はよいと分析し、前記よいと分析された静止画像の構図の評価値が、前記よいと分析されていない静止画像の構図の評価値よりも高くなるように算出するものである請求項１０に記載の画像処理装置。 The composition analysis unit defines in advance the movement pattern of the noted person, and the noted person moves with the previously-defined movement pattern from among the movement locus of the noted person detected by the movement locus detection unit. Detected, and the composition of the still image corresponding to the part in which the noted person is moving according to the predefined movement pattern is analyzed as good, and the evaluation value of the composition of the still image analyzed as good is 11. The image processing apparatus according to claim 10, wherein the image processing apparatus is calculated to be higher than the evaluation value of the composition of the still image not analyzed as being good.
  12. 　前記注目人物検出部は、さらに、前記静止画像における注目人物の顔の向きを検出するものであり、
    　さらに、前記注目人物検出部により検出された注目人物の顔の向きに基づいて、前記静止画像データ出力部から出力された静止画像データに対応する静止画像の天地が、前記動画像が撮影された時の撮影装置の天地と同じになるように、前記静止画像データ出力部から出力された静止画像データに対応する静止画像の天地を補正する天地補正部を備える請求項１～１１のいずれか１項に記載の画像処理装置。     The noted person detection unit further detects the direction of the face of the noted person in the still image,
    Furthermore, based on the orientation of the face of the target person detected by the target person detection unit, the moving image is taken of the top and bottom of a still image corresponding to the still image data output from the still image data output unit. 12. The image processing apparatus according to claim 1, further comprising: a top-bottom correction unit configured to correct the top-bottom of the still image corresponding to the still image data output from the still-image data output unit so as to be the same as the top An image processing apparatus according to claim 1.
  13. 　静止画像データ抽出部が、動画像データから、複数のフレームの静止画像データを抽出するステップと、
    　注目人物検出部が、前記複数のフレームの静止画像データに対応する複数枚の静止画像の各々の中から、処理対象とする人物である注目人物を検出するステップと、
    　運動軌跡検出部が、前記複数枚の静止画像における前記注目人物の検出結果に基づいて、前記動画像データに対応する動画像における前記注目人物の動きを追跡して前記注目人物の運動軌跡を検出するステップと、
    　動作分析部が、前記注目人物の運動軌跡に基づいて、前記動画像における注目人物の動作を分析し、前記複数枚の静止画像の各々について、前記分析された注目人物の動作に基づいて、前記注目人物の動作に対する評価値を算出するステップと、
    　静止画像データ出力部が、前記複数のフレームの静止画像データの中から、前記注目人物の動作に対する評価値が閾値以上である静止画像の静止画像データを出力するステップとを含む画像処理方法。     A still image data extraction unit extracting still image data of a plurality of frames from moving image data;
    Detecting a noted person who is a person to be processed from each of a plurality of still images corresponding to the still image data of the plurality of frames;
    The movement locus detection unit detects the movement locus of the notable person by tracking the movement of the notable person in the moving image corresponding to the moving image data based on the detection result of the notable person in the plurality of still images Step to
    The motion analysis unit analyzes the motion of the watched person in the moving image based on the motion locus of the watched person, and, for each of the plurality of still images, the motion of the watched person based on the analyzed motion of the watched person. Calculating an evaluation value for the movement of the noted person;
    And D. outputting the still image data of the still image whose evaluation value for the motion of the person of interest is equal to or greater than the threshold value out of the still image data of the plurality of frames.
  14. 　さらに、前記注目人物検出部が、前記静止画像における注目人物の位置、前記静止画像における注目人物の大きさ、および、前記静止画像における注目人物の領域を検出するステップと、
    　前記運動軌跡検出部が、前記注目人物の運動軌跡の長さ、および、前記注目人物の移動パターンを検出するステップと、
    　重要度判定部が、前記注目人物の運動軌跡の長さ、前記静止画像における注目人物の位置、前記静止画像における注目人物の大きさのうちの少なくとも１つに基づいて、前記複数枚の静止画像の各々の重要度を判定し、前記複数枚の静止画像の各々について、前記判定された重要度に基づいて、前記重要度の評価値を算出するステップと、
    　構図分析部が、前記静止画像における注目人物の位置、前記静止画像における注目人物の大きさ、前記注目人物の移動パターンのうちの少なくとも１つに基づいて、前記複数枚の静止画像の各々の構図の良否を分析し、前記複数枚の静止画像の各々について、前記分析された構図の良否に基づいて、前記構図の評価値を算出するステップと、
    　画質判定部が、前記静止画像における注目人物の領域に基づいて、前記複数枚の静止画像の各々の画質を判定し、前記複数枚の静止画像の各々について、前記判定された画質に基づいて、前記画質の評価値を算出するステップと、
    　前記静止画像データ出力部が、前記複数枚の静止画像の中から、前記注目人物の動作に対する評価値と、前記重要度の評価値、前記構図の評価値、および、前記画質の評価値のうちの少なくとも１つの評価値との総合評価値が閾値以上である１枚以上の静止画像の静止画像データを出力するステップとを含む請求項１３に記載の画像処理方法。     Further, the target person detection unit detects the position of the target person in the still image, the size of the target person in the still image, and the region of the target person in the still image;
    The motion trajectory detection unit detects the length of the motion trajectory of the noted person and the movement pattern of the noted person;
    The importance level determination unit determines the plurality of still images based on at least one of the length of the motion locus of the noted person, the position of the noted person in the still image, and the size of the noted person in the still image. Determining an importance of each of the plurality of still images, and calculating an evaluation value of the importance based on the determined importance for each of the plurality of still images.
    A composition analysis unit composes each of the plurality of still images based on at least one of the position of the target person in the still image, the size of the target person in the still image, and the movement pattern of the target person. Analyzing the quality of each of the plurality of still images based on the quality of the analyzed composition, and calculating an evaluation value of the composition;
    The image quality determination unit determines the image quality of each of the plurality of still images based on the area of the person of interest in the still image, and the image quality determination unit determines the quality of each of the plurality of still images based on the determined image quality. Calculating an evaluation value of the image quality;
    Among the plurality of still images, the still image data output unit includes an evaluation value for the motion of the person of interest, an evaluation value of the importance, an evaluation value of the composition, and an evaluation value of the image quality. And D. outputting still image data of one or more still images having an overall evaluation value with at least one evaluation value of or greater than a threshold value.
  15. 　さらに、前記注目人物検出部が、前記静止画像における注目人物の顔の向きを検出するステップと、
    　天地補正部が、前記注目人物検出部により検出された注目人物の顔の向きに基づいて、前記静止画像データ出力部から出力された静止画像データに対応する静止画像の天地が、前記動画像が撮影された時の撮影装置の天地と同じになるように、前記静止画像データ出力部から出力された静止画像データに対応する静止画像の天地を補正するステップとを含む請求項１３または１４に記載の画像処理方法。     Furthermore, the noted person detection unit detects the direction of the face of the noted person in the still image;
    The moving image is the moving image of the still image corresponding to the still image data output from the still image data output unit based on the orientation of the face of the noted person detected by the noted person detection unit. And correcting the top and bottom of the still image corresponding to the still image data output from the still image data output unit so as to be the same as the top and bottom of the photographing apparatus at the time of photographing. Image processing method.
  16. 　請求項１３～１５のいずれか１項に記載の画像処理方法の各々のステップをコンピュータに実行させるためのプログラム。 A program for causing a computer to execute each step of the image processing method according to any one of claims 13 to 15.
  17. 　請求項１３～１５のいずれか１項に記載の画像処理方法の各々のステップをコンピュータに実行させるためのプログラムが記録されたコンピュータ読み取り可能な記録媒体。 A computer readable recording medium having recorded thereon a program for causing a computer to execute the steps of the image processing method according to any one of claims 13 to 15.
PCT/JP2015/072818 2014-08-29 2015-08-12 Image-processing device, image-processing method, program, and recording medium WO2016031573A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2014-176402 2014-08-29
JP2014176402A JP2016052013A (en) 2014-08-29 2014-08-29 Image processing device, image processing method, program and recording medium

Publications (1)

Publication Number Publication Date
WO2016031573A1 true WO2016031573A1 (en) 2016-03-03

Family

ID=55399470

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2015/072818 WO2016031573A1 (en) 2014-08-29 2015-08-12 Image-processing device, image-processing method, program, and recording medium

Country Status (2)

Country Link
JP (1) JP2016052013A (en)
WO (1) WO2016031573A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020010974A1 (en) * 2018-07-12 2020-01-16 腾讯科技（深圳）有限公司 Image processing method and device, computer readable medium and electronic device

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106023894B (en) * 2016-08-09 2019-01-22 深圳市华星光电技术有限公司 A kind of driving method and drive system for reducing AMOLED and showing ghost
JP6778398B2 (en) * 2017-07-20 2020-11-04 京セラドキュメントソリューションズ株式会社 Image forming apparatus, image forming method and image forming program
JP6737247B2 (en) * 2017-07-20 2020-08-05 京セラドキュメントソリューションズ株式会社 Image processing apparatus, image processing method, and image processing program
EP4047923A4 (en) 2020-12-22 2022-11-23 Samsung Electronics Co., Ltd. Electronic device comprising camera and method of same
KR20220090054A (en) * 2020-12-22 2022-06-29 삼성전자주식회사 Electronic device with camera and method thereof

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009200713A (en) * 2008-02-20 2009-09-03 Sony Corp Image processing device, image processing method, and program
WO2010116715A1 (en) * 2009-04-07 2010-10-14 パナソニック株式会社 Image pickup device, image pickup method, program and integrated circuit
JP2011175599A (en) * 2010-02-25 2011-09-08 Canon Inc Image processor, and processing method and program thereof
JP2011239021A (en) * 2010-05-06 2011-11-24 Nikon Corp Moving image creation device, imaging device and moving image creation program

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009200713A (en) * 2008-02-20 2009-09-03 Sony Corp Image processing device, image processing method, and program
WO2010116715A1 (en) * 2009-04-07 2010-10-14 パナソニック株式会社 Image pickup device, image pickup method, program and integrated circuit
JP2011175599A (en) * 2010-02-25 2011-09-08 Canon Inc Image processor, and processing method and program thereof
JP2011239021A (en) * 2010-05-06 2011-11-24 Nikon Corp Moving image creation device, imaging device and moving image creation program

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020010974A1 (en) * 2018-07-12 2020-01-16 腾讯科技（深圳）有限公司 Image processing method and device, computer readable medium and electronic device
US11282182B2 (en) 2018-07-12 2022-03-22 Tencent Technology (Shenzhen) Company Limited Image processing method and apparatus, computer-readable medium, and electronic device

Also Published As

Publication number Publication date
JP2016052013A (en) 2016-04-11

Similar Documents

Publication Publication Date Title
WO2016031573A1 (en) Image-processing device, image-processing method, program, and recording medium
US9734612B2 (en) Region detection device, region detection method, image processing apparatus, image processing method, program, and recording medium
US10417773B2 (en) Method and apparatus for detecting object in moving image and storage medium storing program thereof
KR101071352B1 (en) Apparatus and method for tracking object based on PTZ camera using coordinate map
JP6482195B2 (en) Image recognition apparatus, image recognition method, and program
JP5484184B2 (en) Image processing apparatus, image processing method, and program
JP6389801B2 (en) Image processing apparatus, image processing method, program, and recording medium
JP4373840B2 (en) Moving object tracking method, moving object tracking program and recording medium thereof, and moving object tracking apparatus
JP6579950B2 (en) Image analysis apparatus, program, and method for detecting person appearing in captured image of camera
JP6362085B2 (en) Image recognition system, image recognition method and program
JP4764172B2 (en) Method for detecting moving object candidate by image processing, moving object detecting method for detecting moving object from moving object candidate, moving object detecting apparatus, and moving object detecting program
JP6389803B2 (en) Image processing apparatus, image processing method, program, and recording medium
US10574904B2 (en) Imaging method and electronic device thereof
JP2012212373A (en) Image processing device, image processing method and program
US9947106B2 (en) Method and electronic device for object tracking in a light-field capture
KR20170053807A (en) A method of detecting objects in the image with moving background
JP4913801B2 (en) Shielding object image identification apparatus and method
JP2008288684A (en) Person detection device and program
JP2008287594A (en) Specific movement determination device, reference data generation device, specific movement determination program and reference data generation program
González et al. Single object long-term tracker for smart control of a ptz camera
JP6798609B2 (en) Video analysis device, video analysis method and program
JP6548788B2 (en) IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, PROGRAM, AND RECORDING MEDIUM
KR101468347B1 (en) Method and arrangement for identifying virtual visual information in images
JP5539565B2 (en) Imaging apparatus and subject tracking method
CN113033350B (en) Pedestrian re-identification method based on overlook image, storage medium and electronic equipment

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15836773

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 15836773

Country of ref document: EP

Kind code of ref document: A1