JP2019022166A - Image processing apparatus, image processing method, and image processing program - Google Patents

Abstract

To provide a technology to support setting a composition in extracting still image data from dynamic image data.SOLUTION: An image processing apparatus comprises: a still image data creation unit that creates a plurality of pieces of still image data from dynamic image data; and a print layout setting unit that analyzes face images displaying the faces displayed in the still image data, specifies subject directions that are each a direction of at least one of the face and the line of sight of the face on the basis of the analysis, and sets a composition in which a wide space is arranged toward the specified subject directions rather than the opposite side of the subject directions.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing technique for extracting still image data from moving image data.

  In recent years, with improvement in performance and image quality of video cameras and smartphones, generation of still image data that can be extracted from moving image data has been performed. In view of such a situation, for example, Patent Document 1 proposes an image processing method including a layout step of enlarging / reducing and positioning so that a face image is arranged in a preset area of the entire image area. ing. Accordingly, Patent Document 1 states that a person image can be automatically arranged in an appropriate layout.

JP 2000-270198 A

  However, since the aspect ratio of the moving image is generally different from the aspect ratio of the still image, the composition intended when the moving image is captured may not be maintained when editing the aspect ratio of the still image. Furthermore, in video shooting, both the photographer and the subject may move, and it is generally difficult to shoot with the composition in mind.

  The present invention has been made in view of such a situation, and an object of the present invention is to provide a technique for supporting composition setting when extracting still image data from moving image data.

  The image processing apparatus of the present invention analyzes a still image data generation unit that generates a plurality of still image data from moving image data, a face image that displays a face displayed in the still image data, and based on the analysis And a still image data extraction unit that sets a composition in which a larger space is arranged on the side of the direction than the opposite side of the direction according to at least one direction of the face and the line of sight of the face.

  An image forming apparatus according to the present invention includes the image processing apparatus and an image forming unit that forms an image on a print medium.

  According to the image processing method of the present invention, a still image data generation step of generating a plurality of still image data from moving image data, a face image displaying a face displayed in the still image data are analyzed, and based on the analysis And a still image data extraction step of setting a composition in which a larger space is arranged on the side of the direction than the opposite side of the direction according to at least one direction of the face and the line of sight of the face.

  The image processing program of the present invention analyzes a still image data generation unit that generates a plurality of still image data from moving image data, and a face image that displays a face displayed in the still image data, and performs the analysis Based on at least one direction of the face and the line of sight of the face, the image processing apparatus functions as a still image data extraction unit that sets a composition in which a larger space is arranged on the side of the direction than the opposite side of the direction Let

  According to the present invention, a technique for supporting composition setting when extracting still image data from moving image data is provided.

2 is a block diagram showing a functional configuration of the image forming apparatus 100 according to the first embodiment of the present invention. It is a flowchart which shows the content of the still image acquisition process which concerns on 1st Embodiment. It is a data flow diagram which shows the content of the frame image data generation process which concerns on 1st Embodiment. It is explanatory drawing which shows the outline | summary of the frame image extraction process which concerns on 1st Embodiment. It is a flowchart which shows the content of the frame image extraction process which concerns on 1st Embodiment. It is a flowchart which shows the content of the still image acquisition process which concerns on 2nd Embodiment. It is a flowchart which shows the content of the frame image extraction process which concerns on 2nd Embodiment. It is explanatory drawing which shows the outline | summary of the frame image extraction process which concerns on 2nd Embodiment. It is a flowchart which shows the content of the still image acquisition process which concerns on 3rd Embodiment. It is a flowchart which shows the content of the person registration process which concerns on 3rd Embodiment. It is a flowchart which shows the content of the feature-value calculation process which concerns on 3rd Embodiment. It is a flowchart which shows the content of the still image acquisition process which concerns on 4th Embodiment. It is a flowchart which shows the content of the print output process which concerns on 4th Embodiment. It is a flowchart which shows the content of the printing target image selection process which concerns on 4th Embodiment. It is explanatory drawing which shows the operation display screen in the printing target image selection process which concerns on 4th Embodiment. It is a flowchart which shows the content of the print layout setting process which concerns on 4th Embodiment. It is explanatory drawing which shows the operation display screen in the print layout setting process which concerns on 4th Embodiment. It is explanatory drawing which shows an example of the print layout setting process which concerns on 4th Embodiment. It is explanatory drawing which shows the other example of the printing layout setting process which concerns on 4th Embodiment.

Hereinafter, modes for carrying out the present invention (hereinafter referred to as “embodiments”) will be described in the following order with reference to the drawings.
A. First embodiment:
B. Second embodiment:
C. Third embodiment:
D. Fourth embodiment:
E. Variation:

A. First embodiment:
FIG. 1 is a block diagram showing a functional configuration of an image forming apparatus 100 according to the first embodiment of the present invention. The image forming apparatus 100 includes a control unit 110, an image forming unit 120, an operation display unit 130, a storage unit 140, and a communication interface unit 150 (also referred to as a communication I / F unit). The image forming apparatus 100 is connected to the smartphone 200 via short-range wireless communication via the communication interface unit 150. Thereby, the image forming apparatus 100 can receive the moving image data generated by imaging with the smartphone 200.

  In this embodiment, near field communication uses CLASS 2 of BLUETOOTH (registered trademark). CLASS 2 of BLUETOOTH (registered trademark) is a communication with an output of 2.5 mW, and is a short-range wireless communication that enables communication between the image forming apparatus 100 and the smartphone 200 within about 10 m.

  The control unit 110 includes main storage means such as RAM and ROM, and control means such as MPU (Micro Processing Unit) and CPU (Central Processing Unit). The control unit 110 also has controller functions related to various I / O, USB (Universal Serial Bus), bus, and other interfaces such as hardware. The control unit 110 controls the entire image forming apparatus 100.

  The control unit 110 includes a frame image generation unit 111, a feature amount calculation unit 112, a frame image extraction unit 113, a frame memory 114, and a cycle setting unit 115. The image forming unit 120 forms an image on a print medium.

  The operation display unit 130 includes a display 131 and an operation processing unit 132. The display 131 functions as a touch panel and displays various menus as an input screen. The operation processing unit 132 includes a start button and accepts a user operation input.

  The storage unit 140 is a storage device including a hard disk drive or a flash memory that is a non-temporary recording medium, and stores a control program and data for processing executed by the control unit 110, respectively. The storage unit 140 further stores a person registration data in which a still image storage area 141 for storing frame image data and face image data for detecting a human face (also referred to as registered face image data) are registered. Region 142. The person registration data storage area 142 also stores data representing a threshold for extracting frame image data in accordance with general face detection.

  FIG. 2 is a flowchart showing the content of the still image acquisition process according to the first embodiment. In step S <b> 10, the user executes feature selection processing using the operation display unit 130. In the feature selection process, the user can select the type of feature quantity to be detected in the still image to be extracted as a condition for extracting still image data from moving image data.

  The types of feature amounts include a feature amount for face detection and a feature amount for detecting a specific person. The feature amount for face detection can quantitatively represent that a human face is simply displayed large. The feature amount for detecting a specific person can identify the specific person from other persons and quantitatively represent that the face of the specific person is displayed large. A feature amount registration method for detecting a specific person will be described in detail in a third embodiment.

  At this time, the cycle setting unit 115 causes the operation display unit 130 to display an operation display screen (not shown) that receives a cycle setting input that is a sampling cycle setting that is a temporal sampling interval of the frame image. Thereby, the user can set a sampling period.

  In step S20, the frame image generation unit 111 of the control unit 110 functions as a still image data generation unit, and executes frame image data generation processing. In the frame image data generation process, the frame image generation unit 111 generates frame image data from moving image data MD having a frame rate of 30 fps, for example.

  FIG. 3 is a data flow diagram showing the contents of the frame image data generation processing according to the first embodiment. In FIG. 3, a data flow diagram is shown on the upper side, and a GOP (Group of Pictures) is shown on the lower side. The data flow diagram shows the flow of frame image data from the moving image data MD. The frame image data is configured as YUV image data. The frame image data generation process is a process for extracting a plurality of frame image data from the moving image data MD, and is executed by the frame image generation unit 111.

  Examples of the frame image data generation process include MPEG-4 (ISO / IEC 14496) and H.264. H.264 is included. In the frame image data generation process, the frame image generation unit 111 generates frame image data from an I frame (Intra-coded Frame), a P frame (Predicted Frame), and a B frame (Bi-directional Predicted Frame).

  An I frame is a frame that is encoded without using inter-frame prediction. The I frame is also called an intra frame or a key frame. The I frame constitutes a GOP together with a P frame (Predicted Frame) and a B frame (Bi-directional Predicted Frame). The P frame enables generation of frame image data by inter-frame processing with an I frame or a P frame. The B frame enables generation of frame image data by inter-frame processing before and after the I frame, P frame, and other B frames.

  The moving image data is generated from a plurality of frame image data arranged in time series. A plurality of frame image data is often approximated between frames before and after time series. Inter-frame prediction is a technique for predicting a current frame image from a previous frame image in time series using such characteristics of moving image data.

  Specifically, this is a technique for estimating the movement of each pixel block and DCT-transforming / quantizing the difference of the pixel block between the moved frames to increase the compression rate in GOP units. P frames can be reproduced from I frames using motion vectors. The motion vector is a movement vector of each pixel block.

  The frame image generation unit 111 generates frame image data as YUV image data including luminance data and color difference data by performing inverse discrete cosine transform (also referred to as inverse DCT transform) on the I frame. The inverse DCT transform is executed for each 8 × 8 pixel or 16 × 16 pixel block, for example. The frame image generation unit 111 stores the reproduced frame image data in the frame memory 114 of the control unit 110.

  The frame image generation unit 111 generates difference data by performing inverse discrete cosine transform on the P frame and the B frame. The frame image generation unit 111 performs inter-frame processing using the difference data and the motion vector, and generates frame image data. The motion vector is data generated when the moving image data MD is encoded. This processing is performed using MPEG-4 or H.264. H.264 is a normal decoding process.

  The frame image generation unit 111 executes frame image data generation processing based on the P frame and the B frame, and stores all the frame image data in the frame memory 114. When the frame image data is stored in the frame memory 114, the frame image generation unit 111 stores a number representing the time-series order of the frame image data in association with each frame image data.

  FIG. 4 is an explanatory diagram showing an outline of the frame image extraction processing according to the first embodiment. FIG. 4 shows a first frame image group FG1, a second frame image group FG2, and a third frame image group FG3. The first frame image group FG1 shows 80 frame image data F1 to F80 as processing targets. The second frame image group FG2 shows frame image data F1 to F80 during sampling processing. The third frame image group FG3 shows frame image data F1 to F80 after completion of the frame image extraction process.

  In step S30, the frame image generation unit 111 executes a sampling process. In the present embodiment, the feature amount calculation unit 112 determines a sampling target frame at a sampling period of 8 frames (0.27 seconds in the case of 30 fps) based on a user's selection. In this example, the sampling target frames are eight frame image data F9, F18, F27, F36, F45, F54, F63, and F72 represented by diamond symbols in the first frame image group FG1. The image data of the sampling target frame is also called sampling target still image data.

  In step S40, the feature amount calculation unit 112 of the control unit 110 performs a feature amount calculation process. In the feature amount calculation process, the feature amount calculation unit 112 calculates a feature amount based on the selected feature amount type. The feature quantity calculation unit 112 can perform detection and identification of a person using, for example, the well-known OpenCV (Open Source Computer Vision Library). The feature amount of the sampling target still image data is also referred to as a first feature amount.

  Specifically, the feature amount calculation unit 112 is configured to specify a specific person based on registered face image data registered in the person registration data storage area 142 and, for example, a HOG (Histograms of Oriented Gradients) feature or a Haar-Like feature. The feature quantity that quantitatively represents the feature of is calculated. In the calculation of the feature amount, machine learning such as SVM (Support Vector Machine) can be used.

  In step S50, the frame image extraction unit 113 executes a threshold setting process. In the threshold value setting process, the frame image extraction unit 113 reads data representing a threshold value for extracting frame image data in accordance with face detection from the person registration data storage area 142, and determines the threshold value using the data. . The threshold value is automatically calculated by machine learning and stored in the person registration data storage area 142.

  FIG. 5 is a flowchart showing the contents of the frame image extraction process (step S60) according to the first embodiment (see FIG. 3). In step S61, the frame image extraction unit 113 determines whether each sampling target frame is an extraction target. The frame image extraction unit 113 determines whether or not it is an extraction target based on whether or not the feature amount of each sampling target frame is greater than or equal to a threshold value. The threshold value is a value read from the person registration data storage area 142.

  If the sampling target frame is the extraction target, the frame image extraction unit 113 advances the process to step S62. In this example, the sampling target frame whose feature amount is equal to or greater than the threshold value is three frame image data F18, F45, and F54 in the first frame image group FG1. The sampling target frame (sampling target still image data) is also called first extraction target still image data when it is an extraction target.

  In step S62, the feature amount calculation unit 112 extracts, as extraction candidate frame image data, frame image data (adjacent eight in this example) within a preset range from a sampling target frame whose feature amount is equal to or greater than a threshold value. (Also referred to as still image data)) (see second frame image group FG2). The sampling target frames F18, F45, and F54 whose feature amount is equal to or greater than the threshold are also referred to as extraction target sampling target frames.

  Specifically, for example, for the sampling target frame F18, the feature amount calculation unit 112 selects the eight subsequent frame image data F19 to F26. Further, the feature amount calculation unit 112 selects the eight subsequent frame image data F46 to F53 for the sampling target frame F45, and the eight subsequent frame image data F55 to F5 for the sampling target frame F54. Select F62.

  In step S63, the feature amount calculation unit 112 determines whether the sampling target frame immediately before the extraction target sampling target frame is the extraction target sampling target frame. If the immediately preceding sampling target frame is the extraction target sampling target frame, the feature amount calculation unit 112 proceeds to step S65. If the immediately preceding sampling target frame is not the extraction target sampling target frame, the feature amount calculating unit 112 performs the processing. Proceed to S64.

  Specifically, for the sampling target frame F18, the feature amount calculating unit 112 advances the process to step S64 because the immediately preceding sampling target frame F9 is not the extraction target sampling target frame. This also applies to the sampling target frame F45. On the other hand, for the sampling target frame F54, the feature amount calculation unit 112 advances the processing to step S65 because the immediately preceding sampling target frame F45 is the extraction target sampling target frame.

  In step S64, the feature amount calculation unit 112, as extraction candidate frame image data, frame image data (also referred to as adjacent still image data) within a preset range from the extraction target sampling target frame (eight in this example). .) Specifically, for the sampling target frame F18, eight frame image data F10 to F17 are selected as extraction candidate frame image data, and for the sampling target frame F45, eight sampling target frames F37 to F44 are extraction candidates. Selected as frame image data.

  On the other hand, for the extraction target sampling target frame F54, the previous frame image data F46 to F53 are not selected. This is because the frame image data F46 to F53 have already been selected as the eight frame image data on the rear side of the extraction target sampling target frame F45.

  Note that the preset range includes the sampling target still image data (for example, sampling target frame F45) immediately before the first extraction target still image data (for example, sampling target frame F54) and the first extraction target still image data. It can be set with the sampling target still image data immediately after (for example, sampling target frame F63).

  As described above, the feature amount calculation unit 112 selects the frame image data F10 to F17, F19 to F26, F37 to F44, F46 to F53, and F55 to F62 as extraction candidate frame image data. That is, the feature amount calculation unit 112 sets a sampling target frame at a predetermined frame interval and calculates a feature amount, so that a part of the frame images that are likely to be extracted from a large number of frame image data. It is possible to calculate feature amounts (also referred to as second feature amounts) by limiting to data.

  This method was created based on the following findings by the present inventors. That is, a characteristic part included in each frame image of a moving image rarely appears only in a local frame (a frame image corresponding to an extremely short time). For example, when calculating a feature amount using a human face as a feature, the face is not detected from only a few frame images in the moving image, and is captured in a certain number of consecutive frames (for example, several tens of frames). Conceivable.

  The inventor of the present application considers the nature of the moving image as described above, detects a series of scenes in which the subject whose characteristic amount is to be calculated is photographed in the sampling target frame, and frame image data ( We created a method to calculate the feature value only for frames near the sampling target frame. As a result, the inventor of the present application has realized a method for solving the trade-off problem between the calculation accuracy of the feature amount and the processing burden in the process of extracting still image data from moving image data.

  In step S65, the feature amount calculation unit 112 executes feature amount calculation processing. In this example, the feature amount calculation process is the same as the feature amount calculation process in step S40. That is, in this example, the second feature amount is the same as the first feature amount, but may be a feature amount different from each other.

  In step S66, the frame image extraction unit 113 functions as a still image data extraction unit and executes an extraction target selection process. In the extraction target selection process, the frame image extraction unit 113 performs the same process as the process of step S61, determines whether or not the feature amount of each extraction candidate frame image data is equal to or greater than the threshold value, and the feature amount is the threshold value. The extraction candidate frame image data as described above is selected.

  In this example, the frame image extraction unit 113 includes the frame image data F12 to F17, F19 to F21, F40 to F44, F46 to F47, F50 to F53, and F55, as shown in the third frame image group FG3. ... F61 are selected as second extraction target still image data. The second extraction target still image data constitutes extraction target frame image data together with the extraction target sampling target frame (also referred to as first extraction target still image data).

  In step S67, the frame image extraction unit 113 executes an extraction target flag process. In the extraction target flag process, the frame image extraction unit 113 sets a flag indicating that it is an extraction target in the extraction target frame image data.

  In step S68, the frame image extraction unit 113 executes grouping processing. In the grouping process, the frame image extraction unit 113 groups a plurality of extraction target frame image data for which a flag is set around the extraction target sampling target frame (see FIG. 3).

  Specifically, the frame image extraction unit 113 generates a group of the first frame image data group FE1 around the extraction target sampling target frame F18 (see the third frame image group FG3). The first frame image data group FE1 includes ten extraction target frame image data F12 to F21 that are continuous in time series.

  Further, the frame image extraction unit 113 generates groups of the second frame image data group FE2 and the third frame image data group FE3, respectively, with the extraction target sampling target frames F45 and F54 as the center. The second frame image data group FE2 includes eight extraction target frame image data F40 to F47 that are continuous in time series. The third frame image data group FE3 includes twelve extraction target frame image data F50 to F61 that are continuous in time series.

  In this example, since the frame image data F48 and F49 between the two frame image data F47 and F50 are not selected as the extraction target frame image data, the second frame image data group FE2 and the third frame image data group Grouped with FE3. However, if the frame image data F48 and F49 are selected as the extraction target frame image data, the third frame image data group FE3 is grouped in a state of being merged with the second frame image data group FE2. It will be.

  The control unit 110 repeatedly executes the processing from step S20 to step S60 until the final frame image data (step S70).

  In step S80, the control unit 110 executes frame image data output processing. In the frame image data output process, the control unit 110 stores the first frame image data group FE1, the second frame image data group FE2, and the third frame image data group FE3 in the still image storage area 141 of the storage unit 140.

  Thus, the first frame image data group FE1, the second frame image data group FE2, and the third frame image data group FE3 are managed as three continuous image data files and can be handled for each group. In the still image storage area 141, the extraction target frame image data of the first frame image data group FE1, the second frame image data group FE2, and the third frame image data group FE3 are each DCT converted and stored as JPEG still image data. (See FIG. 3).

  As described above, according to the image forming apparatus 100 according to the first embodiment, a series of scenes in which a subject whose feature amount is to be calculated is detected by a sampling target frame, and frame image data constituting the scene is detected. The feature quantity can be calculated only for. Thereby, it is possible to solve the trade-off problem between the calculation accuracy of the feature amount and the processing burden in the process of extracting still image data from moving image data.

  In the first embodiment, frame image data is generated for all frames, and sampling is performed at a predetermined cycle from the frame image data. However, the present invention is not limited to such a method. Specifically, for example, when the sampling target frame is detected, only the sampling target frame may be generated from the moving image data.

  Specifically, when the sampling target frame is an I frame, frame image data can be generated using only the I frame. This is because the I frame is a frame that retains information of all the frames and is encoded without using inter-frame prediction. When the sampling target frame is a P frame, frame image data can be generated using I frames and P frames related in inter-frame prediction. When the sampling target frame is a B frame, frame image data can be generated using the I frame, the P frame, and the B frame related in inter-frame prediction.

  Furthermore, for example, only I frames may be sampled. In this way, since it is not necessary to perform inter-frame prediction, the processing load can be significantly reduced. Alternatively, only the I frame and the P frame may be sampled. This is because the B frame is not used for the inter-frame prediction of the P frame, so that the frame image data can be generated from only the I frame and the P frame.

  In the sampling of the I frame, or the I frame and the P frame, the sampling period can be realized by, for example, sampling a frame that coincides with the sampling period or a frame having the maximum interval within the sampling period. If neither the I frame nor the P frame exists within the sampling period, the frame image data may be generated using the B frame based on the determination.

B. Second embodiment:
FIG. 6 is a flowchart showing the content of the still image acquisition process according to the second embodiment. FIG. 7 is a flowchart showing the contents of the frame image extraction process according to the second embodiment. The still image acquisition process according to the second embodiment is different from the still image acquisition process according to the first embodiment in that the frame image extraction process (step S60) is changed to the frame image extraction process (step S600). The frame image extraction process (step S600) has a feature in that a frame image in the vicinity of the maximum value of the feature amount in time series is extracted.

  The feature amount is set so as to quantitatively represent the still image desired by the user. Therefore, the maximum value of the feature amount is intended to indicate that it is a frame image at the moment when an image as intended is taken when the photographer sets up the subject. In this example, the image as intended means an image in which a face of a specific person can be photographed from the front or a face of a person with a high smile degree is photographed. The smile degree can be calculated by a statistical identification method using, for example, a well-known 3D model fitting method, and using the shape of the eyes and mouth, and images around the eyes and mouth.

  However, the image having the maximum feature amount does not necessarily match the image desired by the user. Specifically, for example, an image showing a child's facial expression desired by the parent as viewed from the parent user does not necessarily have a maximum smile level. In the second embodiment, by extracting a frame image in the vicinity of the maximum value of the feature amount, the frame image is narrowed down based on the smile level or the like, and the user can select a favorite frame image from the frame image.

  In step S610, the frame image extraction unit 113 executes a local maximum search process. In the maximum value search process, the frame image extraction unit 113 searches for the maximum value of the feature amount in time series. The maximum value means a local maximum value (that is, in the vicinity of a certain point) of the feature amount.

  FIG. 8 is an explanatory diagram showing an outline of the frame image extraction processing according to the second embodiment. FIG. 8 shows a feature amount curve FC representing the variation of the feature amount in time series. In this example, the feature amount curve FC has a first maximum value LM1, a second maximum value LM2, and a third maximum value LM3. In FIG. 8, the feature values are normalized so that the minimum value is 0 and the maximum value is 1. The first maximum value LM1 has a feature value of 0.65, the second maximum value LM2 has a feature value of 0.8, and the third maximum value LM3 has a feature value of 0.5.

  In this example, the feature amount curve FC is configured as a spline curve generated using the feature amount of the sampling target frame. The spline curve includes a curve that passes through the feature amount of the sampling target frame and a spline curve that does not necessarily pass through the feature amount of the sampling target frame (for example, a B-spline curve). The feature amount curve FC is not limited to a spline curve, and may be an approximate curve that passes through the feature amount of the sampling target frame or its vicinity.

  In step S620, the frame image extraction unit 113 executes a threshold value calculation process. In the threshold calculation process, the frame image extraction unit 113 calculates the threshold as 80%, which is a predetermined ratio set in advance with respect to the maximum value in this example. Specifically, for example, the threshold value Th1 for the first maximum value LM1 (feature amount 0.65) is 0.52, and the threshold value Th2 for the second maximum value LM2 (feature amount 0.8) is 0.64. Thus, the threshold value Th3 for the third maximum value LM3 (feature amount 0.5) is 0.4.

  In step S630, the frame image extraction unit 113 executes an adjacent minimum value search process. In the adjacent minimum value search process, the frame image extraction unit 113 searches for a minimum value exceeding the threshold value in the vicinity of each maximum value. Specifically, the frame image extraction unit 113 can find a local minimum value AL exceeding the threshold Th3 (0.4) in the vicinity of the third local maximum value LM.

  In step S640, the frame image extraction unit 113 proceeds to step S655 when the adjacent minimum value exists, and proceeds to step S650 when the adjacent minimum value does not exist. In this example, the frame image extraction unit 113 advances the process to step S650 for the first maximum value LM1 and the second maximum value LM2. On the other hand, the process proceeds to step S655 for the start time of the third maximum value LM, and the process proceeds to step S650 for the end time of the third maximum value LM.

  In step S650, the frame image extraction unit 113 acquires the intersection time with the threshold value. The intersection time with the threshold value can be obtained by using a spline curve, or can be obtained as the time of frame image data having a feature quantity that is larger than the threshold value and closest to the threshold value. In this example, the frame image extraction unit 113 acquires the start time t11 and the end time t12 for the first maximum value LM1, and acquires the start time t21 and the end time t22 for the second maximum value LM2. For the third maximum value LM3, the end time t32 is acquired.

  In step S655, the frame image extraction unit 113 acquires the time of the adjacent minimum value. The time of the adjacent minimum value can be obtained using a spline curve, or can be obtained as the time of frame image data in the vicinity of the adjacent minimum value. In this example, the frame image extraction unit 113 acquires the start time t31 that is the time of the adjacent minimum value AL for the third maximum value LM3.

  In step S660, the frame image extraction unit 113 functions as a still image data extraction unit and executes an extraction target selection process. In the extraction target selection process, the frame image extraction unit 113 selects frame image data within a predetermined time range between the start time and the end time in the vicinity of each local maximum value as extraction target frame image data. The predetermined time range is, in other words, the time between each maximum value and the start time (or the time of the adjacent minimum value), and the time between each maximum value and the end time (or the time of the adjacent minimum value). Set using.

  In this example, the frame image extraction unit 113 selects frame image data between the start time t11 and the end time t12 for the first maximum value LM1, and the start time t21 and the end time for the second maximum value LM2. The frame image data between the time t22 and the third maximum value LM3 is selected between the start time t31 and the end time t32.

  In step S670, the frame image extraction unit 113 executes an extraction target flag process. In the extraction target flag process, the frame image extraction unit 113 sets a flag indicating that it is an extraction target in the extraction target frame image data.

  In step S680, the frame image extraction unit 113 performs a grouping process. In the grouping process, the frame image extraction unit 113 groups a plurality of extraction target frame image data set with a flag for each local maximum value (see FIG. 3). Thus, the frame image extraction unit 113 uses the frame image data groups FE1a, FE2a, and FE3a as continuous image data files for the first maximum value LM1, the second maximum value LM2, and the third maximum value LM3, respectively. Can be generated.

  As described above, the frame image extraction processing according to the second embodiment searches for the maximum value of the feature value, and uses the intersection time (or the time of the adjacent minimum value) with the threshold value of 80% of the maximum value. Frame image data in the vicinity of the local maximum value is extracted. As a result, when the peak shape (or inclination) of the maximum value is steep, the image forming apparatus 100 enables efficient extraction by reducing the number of frames to be extracted, while the peak shape (or inclination of the maximum value). ) Is moderate, it is possible to increase the number of frames to be extracted, assuming that an intended image can be stably acquired over a long period of time.

  In this embodiment, the intersection time of the maximum value with the threshold value of 80% is used. However, the numerical value is not limited to 80%, and a time setting method for determining the vicinity of the maximum value of the feature amount. Is not limited to the method using the threshold and the crossing time. The frame image extraction process according to the second embodiment reduces the number of frames to be extracted when the peak shape of the maximum value is gentle, while extracting the frame when the peak shape of the maximum value is gentle. What is necessary is just to be comprised so that a number may be increased.

  In the present embodiment, the feature amount curve FC is configured as an approximate curve generated using the feature amount of the sampling target frame, but it is not always necessary to use the feature amount of the sampling target frame. The frame image data may be used.

C. Third embodiment:
FIG. 9 is a flowchart showing the contents of still image acquisition processing according to the third embodiment. In the still image acquisition process according to the third embodiment, a person registration process (step S100) is added, and a feature quantity calculation process (step S40) is changed to a feature quantity calculation process (step S400). This is different from the still image acquisition processing according to the first embodiment and the second embodiment.

  FIG. 10 is a flowchart showing the contents of the person registration process according to the third embodiment. In step S110, the user executes moving image data reading processing for person registration processing. In the moving image data reading process, the user uses the operation display unit 130 to select moving image data for person registration processing and instruct to read moving image data. The moving image data can be read by the image forming apparatus 100 using, for example, wireless communication (not shown) or a portable storage medium (not shown).

  In step S120, the feature amount calculation unit 112 of the control unit 110 executes a person detection process. In the person detection process, the feature amount calculation unit 112 generates frame image data from the moving image data, and extracts a person detection area that is an image area having a feature that seems to be a person from the still image represented by the frame image data file. Extract. The feature amount calculation unit 112 can extract a person detection region using machine learning such as SVM (Support Vector Machine) based on, for example, HOG (Histograms of Oriented Gradients) feature amounts.

  In step S130, the feature amount calculation unit 112 performs a person classification process. In the person classification process, the feature amount calculation unit 112 classifies a person in the person detection area as, for example, someone in the family registered in advance. In this embodiment, it is assumed that father A, mother B, son C, and daughter D are registered in advance by the user using the operation display unit 130 as a family.

  The feature amount calculation unit 112 selects frame image data in which a human face is displayed in a size larger than a preset image area, and automatically classifies the frame image data into a plurality of groups and displays them on the operation display unit 130. To do. The user selects and inputs whether each of the plurality of groups corresponds to the father A, the mother B, the son C, the daughter D, or another person via the operation display unit 130.

  Further, the user can perform a correction operation in response to a misrecognition that a still image of father A is included in the group of son C, for example. Thereby, the feature quantity computing unit 112 can improve the accuracy of machine learning. The feature amount calculation unit 112 generates a database using father A, mother B, son C, and daughter D as records. In the database, HOG feature amounts of face images of father A, mother B, son C, and daughter D are registered.

  In step S140, the feature amount calculation unit 112 executes a clothing selection process. In the clothing selection process, the feature amount calculation unit 112 extracts HOG feature amounts for the clothing worn by the father A, mother B, son C, and daughter D from the frame image data. Thereby, the feature amount calculation unit 112 can specify a person using the HOG feature amount of the clothing image in addition to the HOG feature amount of the facial images of the father A, mother B, son C, and daughter D. . This is because each person often wears the same clothes, and different persons tend to wear different clothes.

  In step S150, the feature amount calculation unit 112 executes database registration processing. In the database registration process, the feature amount calculation unit 112 stores a database for the father A, the mother B, the son C, and the daughter D in the person registration data storage area 142 of the storage unit 140. In the database, father A, mother B, son C, and daughter D are recorded, and in addition to the HOG feature value of each face image, the HOG feature value of the dress image, machine learning data of the face image, and machine learning data of the dress image , Height and other user-enterable attribute data.

  The user can also register the face image and clothes image data of each person using still image data captured by the digital camera. The feature amount calculation unit 112 can generate the HOG feature amount of the face image and the HOG feature amount of the clothing image using such image data, and register them in the database. In the present embodiment, it is assumed that the HOG feature amount is generated based on YUV image data with a small calculation load in image recognition.

  FIG. 11 is a flowchart showing the contents of the feature amount calculation processing according to the third embodiment. In the feature amount calculation process (step S400), the feature amount calculation unit 112 detects a candidate region that is a region having a predetermined feature by executing a candidate region high-speed detection process with a relatively small processing load. A feature amount for specifying a person in each candidate area is calculated (the processing load is relatively large).

  In step S410, the feature amount calculation unit 112 executes candidate area high-speed detection processing. In the candidate area high-speed detection process, the feature amount calculation unit 112 detects a candidate area from the frame image data. The candidate area high-speed detection processing employs a detection processing method configured so that processing speed is prioritized over person recognition processing (described later).

  Specifically, the feature amount calculation unit 112 performs a widely used face detection process, for example, a multi-stage filter process using a Haar-like detector. However, the multi-stage filter processing is configured to give priority to calculation speed by reducing the number of stages. Therefore, the candidate area high-speed detection process is allowed to detect an area that the user does not want as a candidate area.

  In step S420, the feature amount calculation unit 112 executes candidate area cutout processing. In the candidate area cut-out process, the feature amount calculation unit 112 acquires candidate area information, which is information indicating coordinates and size indicating the position of each candidate area detected in the frame image data that is bitmap image data. Bitmap image data of each candidate area is generated using the area information.

  In step S430, the feature amount calculation unit 112 compares the total number of pixels (area) of the bitmap image data of the plurality of candidate regions with the total number of pixels (total area) of the frame image data. The inventor of the present application has found a problem that a plurality of candidate regions overlap each other, and the sum of the number of pixels of the plurality of candidate regions exceeds the number of pixels of the frame image data, and the calculation amount may increase.

  When the sum of the number of pixels in the plurality of candidate areas is less than the number of pixels in the frame image data, the feature amount calculating unit 112 proceeds to step S440, and the sum of the number of pixels in the plurality of candidate areas is the number of pixels in the frame image data. If so, the process proceeds to step S450. Thereby, the feature amount calculation unit 112 can solve the problem of increase in the amount of calculation described above.

  In step S440, the feature amount calculation unit 112 calculates each feature amount based on the bitmap image data of each candidate area, and outputs the maximum value as the feature amount. The maximum value is set because there should be only one image of the same person for each bitmap image data. In step S450, the feature amount calculation unit 112 calculates and outputs a feature amount based on the bitmap image data as the entire frame image data.

  Specifically, the feature amount calculation unit 112 performs a person recognition process. The person recognition process is a process that quantitatively represents the possibility (accuracy) of a specific person as a feature amount. In the person recognition process, the feature selected in step S10 for each of the plurality of frame image data stored in the frame memory 114 (in this example, the daughter D is selected in the feature selection process (step S10)). .) Is included. The feature amount calculation unit 112 attempts to detect the daughter D from frame image data as YUV image data including luminance data and color difference data.

  The feature quantity calculation unit 112 can perform detection and identification of a person using, for example, the well-known OpenCV (Open Source Computer Vision Library). The feature amount calculation unit 112 detects a person from the frame image data, and determines whether or not the detected face of the person is the face of the daughter D. This determination is made using the HOG feature amount of the daughter D's face image.

  When the accuracy of the determination as to whether or not the face is the face of the daughter D is low, the feature amount calculation unit 112 determines whether or not it is the daughter D using the HOG feature amount of the clothes image of the daughter D. The HOG feature amount of the clothing image can be used supplementarily when the daughter D is photographed from the lateral direction and the size of the face image is small. The feature amount calculation unit 112 can also quantify the smile level of the daughter D and use it for calculating the feature amount. As described above, the feature amount calculation unit 112 can execute a complicated process with a large calculation amount for each candidate region.

  As described above, in the frame image extraction processing according to the third embodiment, candidate region high-speed detection processing is executed to detect a candidate region that is a region having a predetermined characteristic, and a large calculation load is detected in each detected candidate region. Perform feature calculation. As a result, the third embodiment can reduce the calculation target of the feature quantity, so that the processing load is relatively large and the calculation method with high accuracy can be achieved without overloading the calculation quantity of the feature quantity as a whole. Can be adopted.

  In the present embodiment, the candidate area high-speed detection process (step S410) and the person recognition process (steps S440 and S450) are different types of processes, but only the same type of process is different only in accuracy. It may be.

  In addition, the present embodiment can be mounted in combination with at least one of the first embodiment and the second embodiment, or can be mounted without combining with either the first embodiment or the second embodiment. is there. Furthermore, the frame image extraction process according to the first to third embodiments may be executed not on the image forming apparatus 100 but on the smartphone 200 functioning as the image processing apparatus.

D. Fourth embodiment:
FIG. 12 is a flowchart showing the contents of still image acquisition processing according to the fourth embodiment. The still image acquisition process according to the fourth embodiment is a still image according to the first to third embodiments in that at least the frame image data output process (step S80) is changed to a print output process (step S800). It differs from the acquisition process.

  FIG. 13 is a flowchart showing the contents of print output processing according to the fourth embodiment. In this example, the image forming apparatus 100 is installed in a convenience store or the like, and can be printed out according to money input or other deposit processing.

  In step S <b> 810, the user inserts money from a money insertion slit (not shown) of the image forming apparatus 100. In this example, it is assumed that the user has inserted 500 yen coins into the image forming apparatus 100. In the image forming apparatus 100, usable print sizes include an L plate (127 mm × 89 mm) and a 2L plate (127 mm × 178 mm). In this example, it is assumed that the L size (127 mm × 89 mm) is set as an initial setting as the print size, and the unit price is 30 yen.

  In step S820, the control unit 110 functions as a printable sheet number calculation unit, and calculates the printable sheet number. Until the print size is set, the control unit 110 prints 16 sheets (= 500 yen (amount deposited by the deposit process) ÷ 30 yen (unit price)) based on the initial setting (L version). Calculate as

  In step S830, control unit 110 executes a group number calculation process. A group is a plurality of continuous frame image data (constituting a continuous image data file) grouped by grouping processing (steps S68 and S680). The group includes, for example, a first frame image data group FE1, a second frame image data group FE2, and a third frame image data group FE3 (first embodiment) and three frame image data groups FE1a, FE2a, and FE3a (second embodiment). Form).

  In step S840, control unit 110 determines whether the number of printable sheets is equal to or greater than the number of groups. If the printable number is less than the number of groups, the control unit 110 advances the process to step S850. If the printable number is greater than or equal to the number of groups, the control unit 110 advances the process to step S860. In step S850, control unit 110 executes a first print target image selection process. The second print target image selection process will be described later.

  FIG. 14 is a flowchart showing the contents of the print target image selection process according to the fourth embodiment. In step S851, the feature amount calculation unit 112 calculates an average feature amount for each group. The average feature amount can be calculated as, for example, an average value of feature amounts of the top 10 frame image data of each group.

  FIG. 15 is an explanatory diagram showing an operation display screen in the print target image selection process according to the fourth embodiment. FIG. 15A displays a representative still image for each group as a scene selection screen. In this example, it is assumed that a plurality of frame image data groups included in each group constitute each scene. On the operation display screen of FIG. 15, it is possible to perform enlargement / reduction, movement, etc. of the displayed image by operating (pinch-in, pinch-out, drag, etc.) on the operation display unit 130 functioning as a touch panel.

  In FIG. 15A, the display 131 of the operation display unit 130 has three representative still images F55, F43, and F17, a scroll icon SC1 for scrolling in the high evaluation direction, and scrolling in the low evaluation direction. And a display area RC for displaying the remaining number of sheets (printable sheet number) and the number of remaining scenes. The display 131 functions as a touch panel, and the screen can be scrolled by swiping.

  In step S852, the display 131 displays representative still images in descending order of the average feature amount. The display 131 displays based on display data generated by the control unit 110 that functions as a display control unit. The display data is data for displaying a representative still image in association with a characteristic area. The display data is configured to display the characteristic regions of each of the plurality of representative still images so that the relationship with each of the plurality of representative still images can be understood. The order in which the representative still images are displayed is not limited to the average feature amount, and may be an order determined using the feature amount.

  In this example, the display 131 displays the three representative still images F55, F43, and F17 in descending order of the average feature amount from the left side. The representative still image F55 is a still image represented by frame image data F55 that represents the third frame image data group FE3. The representative still image F43 is a still image represented by frame image data F43 representing the second frame image data group FE2. The representative still image F17 is a still image represented by frame image data F17 representing the first frame image data group FE1 (see FIG. 4 of the first embodiment).

  In this example, the display 131 displays characteristic images Q55, Q43, and Q17 that are images of characteristic areas above the representative still images F55, F43, and F17. Thereby, the user can confirm the standard (or reason) by which the group including the representative still images F55, F43, and F17 is automatically selected. In this example, it is assumed that each group of representative still images F55, F43, and F17 is extracted based on a feature amount for extracting a still image including the daughter D having a high smile degree as a subject.

  Specifically, the feature image Q55 is an image of a candidate region used for calculating the feature amount when the representative still image F55 is extracted in the frame image extraction process according to the third embodiment. The feature image Q43 is an image of a candidate area used for calculating the feature amount when the representative still image F43 is extracted. The feature image Q17 is an image of a candidate area used for calculating the feature amount when the representative still image F17 is extracted.

  In step S853, the user selects a group as scene selection processing. In this example, it is assumed that the user selects a scene by touching the representative still image F43 (or feature image Q43) displayed on the display 131, and touches and confirms the scene selection icon M1.

  FIG. 15B displays two still images F42 and F44 before and after the representative still image F43 as an image selection screen. In FIG. 15B, the display 131 has three still images F42, F43, and F44, feature images Q42, Q43, and Q44, a display area RC that displays the remaining number and the number of remaining scenes, and time series. A scroll icon SC3 for scrolling to the front side and a scroll icon SC4 for scrolling to the rear side are displayed.

  As a result, the following trade-off problem can be solved. That is, after extracting still image data from moving image data, in order to select a desired still image from a large number of still images, a request to display a list with a reduced size of each still image There is. On the other hand, when the size of each still image is reduced, a characteristic image area (for example, a face image) of each still image becomes smaller, which is a trade-off problem that makes it difficult to select a still image.

  In step S854, the user selects frame image data as an image selection process. In this example, it is assumed that the user selects a still image by touching the still image F42 (or the feature image Q42) displayed on the display 131, and touches and confirms the image selection icon M2.

  In step S855, the user executes print layout setting processing. In the print layout setting process, the user extracts a part of the still image F42 and sets the print layout of the L size or 2L size print size.

  FIG. 16 is a flowchart showing the contents of the print layout setting process according to the fourth embodiment. FIG. 17 is an explanatory diagram showing an operation display screen in the print layout setting process according to the fourth embodiment. FIG. 17A shows an enlarged human face displayed in the still image F42. Also on the operation display screen of FIG. 17, it is possible to perform enlargement / reduction, movement, etc. of the displayed image by operating (pinch-in, pinch-out, drag, etc.) on the operation display unit 130 functioning as a touch panel.

  In this example, the display 131 includes a feature image Q42 in which the still image F42 displays the face of the daughter D, an extracted image F42c in which the face of the son C is displayed, and two other persons who are not registered in the person registration data storage area 142. It shows that it includes two extracted images F42a and F42b that display the face. The three extracted images F42a, F42b, and F42c are extracted by the face detection process by the feature amount calculation unit 112 according to the change of the still image F42.

  In step S8551, the user functions as a print layout setting unit and executes subject person selection processing. In the subject person selection process, the user refers to the face of the son C and the face of the daughter D, and changes the scroll icon SC5 for changing to the front still image in time series and the rear still image. The still image can be changed within the group by touching the scroll icon SC6. In this example, it is assumed that the user selects the face of the daughter D and the face of the son C, and touches and confirms the person selection icon M3. The three extracted images F42a, F42b, and F42c are extracted and changed according to the change of the still image.

  FIG. 17B shows an enlarged still image F42L obtained by performing the print layout setting process on the still image F42. FIG. 17B further shows the extracted images F42a and F42b so that it can be seen that the faces of two other people are excluded from the print target area that is the print target area set in the print layout. it's shown. The still image F42L employs a print layout in which the face of the daughter D, the face of the son C, and the body are selected while excluding the faces of two other people. The still image F42L is automatically subjected to print layout setting processing by the following method. The user can touch the layout adjustment M4 to adjust (change) the print layout, and touch the vertical / horizontal change M5 to change the vertical and horizontal directions of the L version.

  In step S8552, the control unit 110 executes subject number counting processing. In the subject number counting process, the control unit 110 counts the number of selected faces as the number of subjects. In step S8553, control unit 110 determines whether or not the number of subjects is one. If the number of subjects is not one, control unit 110 advances the process to step S8554, and if the number of subjects is one, control unit 110 advances the process to step S8555.

  FIG. 18 is an explanatory diagram illustrating an example of a print layout setting process according to the fourth embodiment. FIG. 18 shows a still image F101 that displays the faces of two persons H1 and H2 in order to easily explain the print layout setting process.

  In step S8554, the control unit 110 sets a rectangle that includes all the subjects. Specifically, the control unit 110 automatically sets a rectangle as a bounding box BB1 that includes the faces of all the two persons H1 and H2 and their surroundings.

  FIG. 19 is an explanatory diagram illustrating another example of the print layout setting process according to the fourth embodiment. FIG. 19 shows still images F102 and F103 displaying the face of one person H3 in order to easily explain the print layout setting process. In the still image F102, the person H3 faces both the face and the line of sight in the shooting direction which is the direction of the photographer. In the still image F103, the person H3 has the face and the line of sight facing leftward.

  When the number of subjects is one, the control unit 110 executes print layout setting processing based on the line-of-sight direction of the subject. In step S8555, the control unit 110 performs a gaze direction estimation process. In the line-of-sight direction estimation process, the control unit 110 can perform estimation using, for example, the method proposed in JP-A-8-322796.

  JP-A-8-322796 is based on the analysis of a face image, that is, the image of the eye part is cut out from the image of the face part, the feature quantity representing the eye position information is extracted, and the feature quantity is compared with the standard pattern. We have proposed a method for estimating the gaze direction. The feature amount includes density distributions in the horizontal direction and the vertical direction corresponding to the black eyes of the eye portion of the grayscale image.

  In step S8556, the control unit 110 sets a print layout by arranging a space in the line-of-sight direction. In photography, a composition with a sense of stability is generally adopted so that a space is arranged in the direction of the line of sight of the subject or the direction of the face. In this way, the control unit 110 can set a print layout according to the subject direction that is at least one of the line-of-sight direction and the face direction, and can automatically adopt a stable composition.

  In the still image F102, the control unit 110 automatically sets a rectangle as a bounding box BB2 that includes the person H3 and its periphery from the still image F102. Since the face and line of sight of the person H3 are directed in the direction of the photographer, the control unit 110 adopts a composition that is spread evenly from the bounding box BB2 in both side directions.

  On the other hand, in the still image F103, the control unit 110 automatically sets a rectangle as a bounding box BB3 including the person H3 and its surroundings from the still image F103. Since the face and line of sight of the person H3 are directed to the left side, the control unit 110 employs a composition in which a relatively large space is arranged on the left side from the bounding box BB3.

  It should be noted that the larger the angle of the subject direction relative to the shooting direction, the larger the ratio of the space arranged on the subject direction side to the space arranged on the opposite side of the subject direction. In addition, when there are a plurality of persons and the subject directions of a plurality of faces are in the same direction, a wide space may be arranged on the same direction side.

  In step S8557, control unit 110 executes a margin setting process. In the margin setting process, the control unit 110 automatically arranges general margins assumed when a person is photographed with a camera and sets the frame frames Fr1, Fr2, and Fr3. This setting is performed assuming an aspect ratio of the L version (127 mm × 89 mm) as an initial setting.

  Since the aspect ratio of a moving image is generally different from the aspect ratio of a still image, it may not be possible to maintain the composition intended when shooting the moving image when editing the aspect ratio of the still image. Furthermore, in video shooting, both the photographer and the subject may move, and it is generally difficult to shoot with the composition in mind.

  In order to deal with such a problem, in the present embodiment, the control unit 110 realizes an appropriate composition that is assumed when a person is photographed with a still camera, and automatically arranges a general margin to form a frame frame. Fr1, Fr2, and Fr3 can be set to the L version. In this way, the intended composition is maintained when shooting moving images, and further, support is provided such as semi-automatic setting of a stable and desirable composition without having to be aware of the composition when shooting moving images, thereby reducing the burden on the user. Can be reduced.

  In step S856 (see FIG. 14), print setting processing is executed. In the print setting process, the user sets the number of prints and the contents of the image process, and touches setting completion M6 (see FIG. 17B) to complete the print setting process.

  In step S857, the control unit 110 executes a printable sheet number update process. In the printable sheet number update process, the control unit 110 calculates the remaining printable sheet number, assuming that the print process is executed for the still image that has been set. In this example, since printing of one still image F42L is set, the number of printable sheets (remaining number) is updated from 16 to 15 in the display area RC, and the number of remaining scenes is updated from 18 to 17. Is done.

  The control unit 110 executes the processing from step S851 to step S857 to the final group as the first print target image selection processing (step S858). In this example, the number of remaining scenes is the number of unprocessed groups displayed after a group including still image data selected as a print processing target.

  On the other hand, when the number of printable sheets is equal to or greater than the number of groups, control unit 110 advances the process to step S860 (see FIG. 13). In step S860, control unit 110 executes a second print target image selection process. The second print target image selection process is different from the first print target image selection process in which the representative still images are displayed in descending order of the average feature amount in that the representative still images are displayed for each group in time series order.

  When the number of printable sheets is equal to or greater than the number of groups, at least one print output process can be performed for each group. Therefore, the control unit 110 can display the representative still images for each group in time series order. On the other hand, when the number of printable sheets is less than the number of groups, the user desires to select a group for performing print output processing, and therefore displays representative still images in descending order of the average feature amount.

  Further, the control unit 110 can display the number of printable sheets (remaining number of sheets) and the number of remaining scenes in the display area RC of the operation display unit 130. Thereby, the user can smoothly select a group (scene). For example, when the number of printable sheets is larger than the number of remaining scenes (number of remaining groups), the user can additionally select an image from each group.

  When still image data is extracted from moving image data, it is generally difficult to predict the number of still images extracted. For example, when printing using an image forming device installed at a convenience store or the like on a trip, the number of extractions cannot be predicted, so the burden of selecting a still image to be printed while considering the fee to be paid Is big. This embodiment can reduce the burden for selecting desired still image data from a plurality of still image data extracted from moving image data.

  In step S870, the control unit 110 forms an image on the print medium using the image forming unit 120 in response to pressing of the operation processing unit (start switch) 132. The print target is still image data for which the print setting process has been completed in step S856 (see FIG. 14).

  Note that the control unit 110 may immediately form an image on the print medium upon completion of the print setting process. However, according to the above-described embodiment, there is an advantage that still image data can be selected again by permitting cancellation of the print setting process.

  As described above, the still image acquisition processing according to the fourth embodiment selects a scene (group) from the still image data extracted from the moving image data while confirming the state of the face of the subject, and requests the desired group from the group. The still image to be selected can be selected smoothly.

  In the still image acquisition process according to the fourth embodiment, the layout setting process can be automatically performed with an appropriate composition in consideration of the state of the subject from the selected still image. As described above, the still image acquisition process according to the fourth embodiment can realize an appropriate still image printout while reducing the burden on the user in the printing process using moving image data.

E. Variation:
The present invention can be implemented not only in the above embodiments but also in the following modifications.

  In the above embodiment, the present invention is applied to the image forming apparatus. However, the present invention can also be applied to an apparatus that functions as an image processing apparatus such as a smartphone or a personal computer.

DESCRIPTION OF SYMBOLS 100 Image forming apparatus 110 Control part 111 Frame image generation part 112 Feature-value calculating part 113 Frame image extraction part 114 Frame memory 115 Period setting part 120 Image formation part 130 Operation display part 140 Storage part 150 Communication interface part 200 Smartphone

Claims (8)

A still image data generating unit that generates a plurality of still image data from moving image data;
Analyzing a face image that displays the face displayed in the still image data, and identifying a subject direction that is at least one of the face and the line of sight of the face based on the analysis, and the identified subject direction A print layout setting unit for setting a composition for arranging a wider space on the subject direction side than the opposite side;
An image processing apparatus comprising: The image processing apparatus according to claim 1,
The print layout setting unit is configured such that the larger the angle of the subject direction with respect to the photographing direction that is the direction in which the face is photographed, the larger the space arranged on the subject direction side with respect to the space arranged on the opposite side of the subject direction. An image processing apparatus that arranges the spaces so that the ratio of the two becomes larger. The image processing apparatus according to claim 1, wherein:
The print layout setting unit determines whether or not the number of faces displayed in the still image data is single, and if the number of faces is single, sets the composition, and An image processing apparatus for setting a composition surrounding the plurality of faces when the number of faces is plural. The image processing apparatus according to claim 3,
In the case where the number of the faces is plural and the subject direction of the plural faces is in the same direction, the print layout setting unit is configured to arrange a wide space on the same direction side. The image processing device to be set. The image processing apparatus according to any one of claims 1 to 4,
When a plurality of the faces are displayed in the still image data, display data for displaying the still image data is generated so that at least one of the plurality of faces can be selected,
The still image data extraction unit is an image processing device that sets the composition based on the selected face. An image forming apparatus,
The image processing apparatus according to any one of claims 1 to 5,
When a plurality of the faces are displayed in the still image data, the still image data is displayed based on display data for displaying the still image data so that at least one of the plurality of faces can be selected. An operation display for displaying
An image forming unit that forms an image on a print medium;
An image forming apparatus comprising: A still image data generation step of generating a plurality of still image data from moving image data;
Analyzing a face image that displays the face displayed in the still image data, and identifying a subject direction that is at least one of the face and the line of sight of the face based on the analysis, and the identified subject direction A print layout setting step for setting a composition for arranging a wider space on the subject direction side than the opposite side;
An image processing method comprising: A still image data generation unit that generates a plurality of still image data from moving image data; and a face image that displays a face displayed in the still image data; and based on the analysis, the face and the face The image processing apparatus functions as a print layout setting unit that identifies a subject direction that is at least one direction of the line of sight and sets a composition in which a wider space is arranged on the subject direction side than the opposite side of the identified subject direction An image processing program to be executed.