JP6778398B2 - Image forming apparatus, image forming method and image forming program - Google Patents

Description

The present invention relates to an image forming technique for extracting still image data from moving image data.

In recent years, with the improvement of performance and image quality of video cameras and smartphones, it has come to be performed to generate still image data that can be extracted from moving image data. Based on this situation, a technique has been proposed that makes it possible to create an album or the like by extracting a desired still image from moving image data. For example, Patent Document 1 proposes a technique in which a change in the shooting status of moving image data is grasped from a change characteristic of a feature amount included in a frame image of moving image data, and a frame image showing this change is selected and displayed. ..

JP-A-2007-82240

However, sufficient studies have not been made on how to reduce the burden on the user when printing a still image extracted from moving image data. When extracting still image data from moving image data, it is difficult to predict the number of still images extracted. For example, when performing printing processing using an image forming apparatus installed in a convenience store or the like while traveling, it is assumed that the burden of selecting a still image to be printed is heavy while considering the fee to be paid. ..

The present invention has been made in view of such a situation, and an object of the present invention is to provide a technique for reducing the burden of selecting a still image to be printed while considering a fee to be paid.

The present invention provides an image forming apparatus that executes a printing process in response to a deposit process. The image forming apparatus calculates a still image data generation unit that generates a plurality of still image data from moving image data, a feature amount of the still image data, and extracts the still image data based on the feature amount. Using a still image data extraction unit that generates at least one group that is continuous in time series from the plurality of extracted still image data, the amount of money deposited in the deposit process, and the unit price of the print process. A printable number calculation unit that calculates the number of printable sheets, and an operation display unit that displays the groups in an order determined by using the feature amount when the number of printable sheets is less than the number of the groups. And.

The present invention provides an image forming method that executes a printing process in response to a deposit process. The image forming method includes a still image data generation step of generating a plurality of still image data from moving image data, a feature amount of the still image data is calculated, and the still image data is extracted based on the feature amount. Using the still image data extraction step of generating at least one group that is continuous in time series from the plurality of extracted still image data, the amount of money deposited in the deposit process, and the unit price of the print process. A printable number calculation step of calculating the printable number of sheets, and an operation display step of displaying the group in an order determined by using the feature amount when the printable number of sheets is less than the number of the group. And.

The present invention provides an image forming program for controlling an image forming apparatus that executes a printing process in response to a deposit process. The image forming program calculates a feature amount of the still image data, a still image data generation unit that generates a plurality of still image data from the moving image data, and extracts the still image data based on the feature amount. A still image data extraction unit that generates at least one group that is continuous in time series from a plurality of extracted still image data, and prints using the amount deposited in the deposit process and the unit price of the print process. An image forming unit for calculating the number of printable sheets, and an operation display unit for displaying the groups in an order determined by using the feature amount when the number of printable sheets is less than the number of the groups. Make the device work.

According to the present invention, there is provided a technique for reducing the burden of selecting a still image to be printed while considering a fee to be paid.

It is a block diagram which shows the functional structure of the image forming apparatus 100 which concerns on 1st Embodiment of this invention. It is a flowchart which shows the content of the still image acquisition processing which concerns on 1st Embodiment. It is a data flow diagram which shows the content of the frame image data generation processing which concerns on 1st Embodiment. It is explanatory drawing which shows the outline of the frame image extraction processing which concerns on 1st Embodiment. It is a flowchart which shows the content of the frame image extraction processing which concerns on 1st Embodiment. It is a flowchart which shows the content of the still image acquisition processing which concerns on 2nd Embodiment. It is a flowchart which shows the content of the frame image extraction processing which concerns on 2nd Embodiment. It is explanatory drawing which shows the outline of the frame image extraction processing which concerns on 2nd Embodiment. It is a flowchart which shows the content of the still image acquisition processing which concerns on 3rd Embodiment. It is a flowchart which shows the content of the person registration process which concerns on 3rd Embodiment. It is a flowchart which shows the content of the feature amount calculation process which concerns on 3rd Embodiment. It is a flowchart which shows the content of the still image acquisition processing which concerns on 4th Embodiment. It is a flowchart which shows the content of the print output processing which concerns on 4th Embodiment. It is a flowchart which shows the content of the print target image selection process which concerns on 4th Embodiment. It is explanatory drawing which shows the operation display screen in the print target image selection process which concerns on 4th Embodiment. It is a flowchart which shows the content of the print layout setting process which concerns on 4th Embodiment. It is explanatory drawing which shows the operation display screen in the print layout setting process which concerns on 4th Embodiment. It is explanatory drawing which shows an example of the print layout setting process which concerns on 4th Embodiment. It is explanatory drawing which shows another example of the print layout setting process which concerns on 4th Embodiment.

Hereinafter, embodiments for carrying out the present invention (hereinafter, referred to as “embodiments”) will be described in the following order with reference to the drawings.
A. First Embodiment:
B. Second embodiment:
C. Third Embodiment:
D. Fourth Embodiment:
E. Modification example:

A. First Embodiment:
FIG. 1 is a block diagram showing a functional configuration of the image forming apparatus 100 according to the first embodiment of the present invention. The image forming apparatus 100 includes a control unit 110, an image forming unit 120, an operation display unit 130, a storage unit 140, and a communication interface unit 150 (also referred to as a communication I / F unit). The image forming apparatus 100 is connected to the smartphone 200 by short-range wireless communication via the communication interface unit 150. As a result, the image forming apparatus 100 can receive the moving image data generated by imaging with the smartphone 200.

For short-range wireless communication, in this embodiment, CLASS2 of BLUETOOTH (registered trademark) is used. BLUETOOTH (registered trademark) CLASS2 is a communication with an output of 2.5 mW, and is a short-range wireless communication capable of communicating within a distance of about 10 m between the image forming apparatus 100 and the smartphone 200.

The control unit 110 includes main storage means such as RAM and ROM, and control means such as MPU (Micro Processing Unit) and CPU (Central Processing Unit). In addition, the control unit 110 has controller functions related to interfaces such as various I / O, USB (Universal Serial Bus), bus, and other hardware. The control unit 110 controls the entire image forming apparatus 100.

The control unit 110 includes a frame image generation unit 111, a feature amount calculation unit 112, a frame image extraction unit 113, a frame memory 114, and a cycle setting unit 115. The image forming unit 120 forms an image on the print medium.

The operation display unit 130 includes a display 131 and an operation processing unit 132. The display 131 functions as a touch panel and displays various menus as input screens. The operation processing unit 132 includes a start button and accepts a user's operation input.

The storage unit 140 is a storage device including a hard disk drive, a flash memory, or the like, which are non-temporary recording media, and stores control programs and data for processing executed by the control unit 110, respectively. The storage unit 140 further stores a person registration data 141 in which a still image storage area 141 for storing frame image data and face image data (also referred to as registered face image data) for detecting a person's face are registered. It has a region 142. The person registration data storage area 142 also stores data representing a threshold value for extracting frame image data in response to general face detection.

FIG. 2 is a flowchart showing the contents of the still image acquisition process according to the first embodiment. In step S10, the user executes the feature selection process using the operation display unit 130. In the feature selection process, the user can select the type of feature amount to be detected in the still image to be extracted as a condition for extracting the still image data from the moving image data.

There are various types of feature amounts, such as a feature amount for face detection and a feature amount for detecting a specific person. The feature amount for face detection can simply quantitatively represent that a human face is displayed in a large size. The feature amount for detecting a specific person can identify a specific person from other persons and quantitatively represent that the face of the specific person is displayed in a large size. The method of registering the feature amount for detecting a specific person will be described in detail in the third embodiment.

At this time, the cycle setting unit 115 causes the operation display unit 130 to display an operation display screen (not shown) that accepts a cycle setting input that is a setting of a sampling cycle that is a temporal sampling interval of the frame image. This allows the user to set the sampling period.

In step S20, the frame image generation unit 111 of the control unit 110 functions as a still image data generation unit and executes the frame image data generation process. In the frame image data generation process, the frame image generation unit 111 generates frame image data from the moving image data MD having a frame rate of, for example, 30 fps.

FIG. 3 is a data flow diagram showing the contents of the frame image data generation process according to the first embodiment. In FIG. 3, a data flow diagram is shown on the upper side, and GOP (Group of Pictures) is shown on the lower side. The data flow diagram shows the flow of frame image data from the moving image data MD. The frame image data is configured as YUV image data. The frame image data generation process is a process of extracting a plurality of frame image data from the moving image data MD, and is executed by the frame image generation unit 111.

For frame image data generation processing, for example, MPEG-4 (ISO / IEC 14496) or H.M. The processing specified in 264 is included. In the frame image data generation process, the frame image generation unit 111 generates frame image data from an I frame (Intra-coded Frame), a P frame (Predicted Frame), and a B frame (Bi-directional Predicted Frame).

An I-frame is a frame that is encoded without using inter-frame prediction. The I frame is also called an intra frame or a key frame. The I frame constitutes a GOP together with a P frame (Predicted Frame) and a B frame (Bi-directed Predicted Frame). The P frame enables the generation of frame image data by inter-frame processing with the I frame and the P frame. The B frame enables the generation of frame image data by inter-frame processing before and after the I frame, the P frame, and other B frames.

The moving image data is generated from a plurality of frame image data arranged in chronological order. The plurality of frame image data are often approximated between the frames before and after the time series. Inter-frame prediction is a technique for predicting the current frame image from the previous frame image in chronological order by utilizing the properties of such moving image data.

Specifically, it is a technique that estimates the movement of each pixel block and DCT transforms and quantizes the difference between the pixel blocks after the movement to increase the compression rate in GOP units. The P-frame can be reproduced from the I-frame using motion vectors. The motion vector is a motion vector of each pixel block.

The frame image generation unit 111 generates frame image data as YUV image data including brightness data and color difference data by performing an inverse discrete cosine transform (also called an inverse DCT transform) on an I frame. The inverse DCT transform is performed, for example, for each 8 × 8 pixel or 16 × 16 pixel block. The frame image generation unit 111 stores the reproduced frame image data in the frame memory 114 of the control unit 110.

The frame image generation unit 111 generates difference data by performing an inverse discrete cosine transform on the P frame and the B frame. The frame image generation unit 111 generates frame image data by executing interframe processing using the difference data and the motion vector. The motion vector is data generated at the time of encoding the motion image data MD. This process is performed by MPEG-4 or H.M. This is a normal decoding process specified in 264.

The frame image generation unit 111 executes a frame image data generation process based on the P frame or the B frame, and stores all the frame image data in the frame memory 114. When the frame image data is stored in the frame memory 114, the frame image generation unit 111 stores a number representing the time-series order of the frame image data in association with each frame image data.

FIG. 4 is an explanatory diagram showing an outline of the frame image extraction process according to the first embodiment. FIG. 4 shows a first frame image group FG1, a second frame image group FG2, and a third frame image group FG3. The first frame image group FG1 shows 80 frame image data F1 to F80 as processing targets. The second frame image group FG2 shows the frame image data F1 to F80 during the sampling process. The third frame image group FG3 shows the frame image data F1 to F80 after the frame image extraction process is completed.

In step S30, the frame image generation unit 111 executes the sampling process. In the present embodiment, the feature amount calculation unit 112 determines the sampling target frame at a sampling period of 8 frame intervals (0.27 seconds in the case of 30 fps) based on the user's selection. In this example, the sampling target frames are eight frame image data F9, F18, F27, F36, F45, F54, F63, and F72 represented by diamond symbols in the first frame image group FG1. The image data of the sampling target frame is also called sampling target still image data.

In step S40, the feature amount calculation unit 112 of the control unit 110 executes the feature amount calculation process. In the feature amount calculation process, the feature amount calculation unit 112 calculates the feature amount based on the selected feature amount type. The feature amount calculation unit 112 can execute the detection and identification of a person by using, for example, a well-known OpenCV (Open Source Computer Vision Library). The feature amount of the still image data to be sampled is also called the first feature amount.

Specifically, the feature amount calculation unit 112 is a specific person based on the registered face image data registered in the person registration data storage area 142 and, for example, a HOG (Histograms of Oriented Grades) feature or a Har-Like feature. Calculate the features that quantitatively represent the features of. Machine learning such as SVM (Support Vector Machine) can be used in the calculation of the feature amount.

In step S50, the frame image extraction unit 113 executes the threshold value setting process. In the threshold value setting process, the frame image extraction unit 113 reads data representing a threshold value for extracting frame image data in response to face detection from the person registration data storage area 142, and determines the threshold value using the data. .. The threshold value is automatically calculated by machine learning and stored in the person registration data storage area 142.

FIG. 5 is a flowchart showing the contents of the frame image extraction process (step S60) according to the first embodiment (see FIG. 3). In step S61, the frame image extraction unit 113 determines whether or not each sampling target frame is an extraction target. The frame image extraction unit 113 determines whether or not it is an extraction target based on whether or not the feature amount of each sampling target frame is equal to or greater than a threshold value. The threshold value is a value read from the person registration data storage area 142.

When the sampling target frame is the extraction target, the frame image extraction unit 113 proceeds to the process in step S62. In this example, the sampling target frames whose feature amounts are equal to or larger than the threshold value are three frame image data F18, F45, and F54 in the first frame image group FG1. The sampling target frame (sampling target still image data) is also referred to as a first extraction target still image data when it is an extraction target.

In step S62, the feature amount calculation unit 112 receives frame image data (adjacent to each other) within a preset range (in this example, the rear eight) from the sampling target frame whose feature amount is equal to or greater than the threshold value as extraction candidate frame image data. (Also called still image data) is selected (see the second frame image group FG2). The sampling target frames F18, F45, and F54 whose feature amounts are equal to or larger than the threshold value are also referred to as extraction target sampling target frames.

Specifically, the feature amount calculation unit 112 selects, for example, eight frame image data F19 to F26 on the rear side of the sampling target frame F18. Further, the feature amount calculation unit 112 selects eight frame image data F46 to F53 on the rear side for the sampling target frame F45, and eight frame image data F55 on the rear side for the sampling target frame F54. Select F62.

In step S63, the feature amount calculation unit 112 determines whether or not the sampling target frame immediately before the extraction target sampling target frame is the extraction target sampling target frame. The feature amount calculation unit 112 advances the processing to step S65 when the immediately preceding sampling target frame is the extraction target sampling target frame, and steps the processing when the immediately preceding sampling target frame is not the extraction target sampling target frame. Proceed to S64.

Specifically, the feature amount calculation unit 112 proceeds to step S64 for the sampling target frame F18 because the sampling target frame F9 immediately before the sampling target frame F18 is not the extraction target sampling target frame. This point is the same for the sampling target frame F45. On the other hand, the feature amount calculation unit 112 proceeds to step S65 because the sampling target frame F45 immediately before the sampling target frame F54 is the extraction target sampling target frame.

In step S64, the feature amount calculation unit 112 uses the extraction candidate frame image data as frame image data (also referred to as adjacent still image data) within a preset range (in this example, the front eight) from the extraction target sampling target frame. .) Is selected. Specifically, for the sampling target frame F18, eight frame image data F10 to F17 are selected as extraction candidate frame image data, and for the sampling target frame F45, eight sampling target frames F37 to F44 are extraction candidates. Selected as frame image data.

On the other hand, for the extraction target sampling target frame F54, the frame image data F46 to F53 on the front side thereof are not selected. This is because the frame image data F46 to F53 have already been selected as the eight frame image data on the rear side of the extraction target sampling target frame F45.

The preset range includes the sampling target still image data (for example, sampling target frame F45) immediately before the first extraction target still image data (for example, sampling target frame F54) and the first extraction target still image data. It can be set with the sampling target still image data (for example, sampling target frame F63) immediately after.

In this way, the feature amount calculation unit 112 selects the frame image data F10 to F17, F19 to F26, F37 to F44, F46 to F53, and F55 to F62 as extraction candidate frame image data. That is, the feature amount calculation unit 112 sets the sampling target frame at a predetermined frame interval and calculates the feature amount, so that a part of the frame image that is likely to be the extraction target among a large number of frame image data The feature quantity (also called the second feature quantity) can be calculated only for the data.

This method was created based on the following findings by the inventor of the present application. That is, the characteristic portion included in each frame image of the moving image rarely appears only in the local frame (frame image corresponding to an extremely short time). For example, when calculating the feature amount with a person's face as a feature, the face is not detected only from the few frame images in the moving image, and it is said that the face is shot in a certain continuous frame (for example, several tens of frames). Conceivable.

In consideration of such a property of a moving image, the inventor of the present application detects a series of scenes in which a subject whose feature amount is to be calculated is photographed in a sampling target frame, and frame image data (frame image data) constituting the scene. We created a method to calculate the features only for the frames in the vicinity of the sampling target frame). As a result, the inventor of the present application has realized a method of solving the problem of trade-off between the calculation accuracy of the feature amount and the processing load in the process of extracting the still image data from the moving image data.

In step S65, the feature amount calculation unit 112 executes the feature amount calculation process. In this example, the content of the feature amount calculation process is the same as the feature amount calculation process in step S40. That is, in this example, the second feature amount is the same as the first feature amount, but can be different from each other.

In step S66, the frame image extraction unit 113 functions as a still image data extraction unit and executes the extraction target selection process. In the extraction target selection process, the frame image extraction unit 113 executes the same process as the process in step S61, determines whether or not the feature amount of each extraction candidate frame image data is equal to or greater than the threshold value, and the feature amount is the threshold value. The above extraction candidate frame image data is selected.

In this example, the frame image extraction unit 113 includes frame image data F12 to F17, F19 to F21, F40 to F44, F46 to F47, F50 to F53, and F55, as shown in the third frame image group FG3. ~ F61 is selected as the second still image data to be extracted. The second extraction target still image data constitutes the extraction target frame image data together with the extraction target sampling target frame (also referred to as the first extraction target still image data).

In step S67, the frame image extraction unit 113 executes the extraction target flag processing. In the extraction target flag processing, the frame image extraction unit 113 sets a flag indicating the extraction target in the extraction target frame image data.

In step S68, the frame image extraction unit 113 executes the grouping process. In the grouping process, the frame image extraction unit 113 groups a plurality of extraction target frame image data for which flags are set, centering on the extraction target sampling target frame (see FIG. 3).

Specifically, the frame image extraction unit 113 generates a group of the first frame image data group FE1 centering on the extraction target sampling target frame F18 (see the third frame image group FG3). The first frame image data group FE1 includes ten extraction target frame image data F12 to F21 that are continuous in time series.

Further, the frame image extraction unit 113 generates each group of the second frame image data group FE2 and the third frame image data group FE3, respectively, centering on the extraction target sampling target frames F45 and F54. The second frame image data group FE2 includes eight extraction target frame image data F40 to F47 that are continuous in time series. The third frame image data group FE3 includes twelve consecutive frame image data F50 to F61 to be extracted in chronological order.

In this example, since the frame image data F48 and F49 between the two frame image data F47 and F50 are not selected as the frame image data to be extracted, the second frame image data group FE2 and the third frame image data group It is grouped with FE3. However, if the frame image data F48 and F49 are selected as the frame image data to be extracted, the third frame image data group FE3 is grouped in a state of being merged with the second frame image data group FE2. It will be.

The control unit 110 repeats the processes of steps S20 to S60 until the final frame image data (step S70).

In step S80, the control unit 110 executes the frame image data output process. In the frame image data output process, the control unit 110 stores the first frame image data group FE1, the second frame image data group FE2, and the third frame image data group FE3 in the still image storage area 141 of the storage unit 140.

As a result, the first frame image data group FE1, the second frame image data group FE2, and the third frame image data group FE3 are managed as three continuous image data files and can be handled for each group. In the still image storage area 141, the extraction target frame image data of the first frame image data group FE1, the second frame image data group FE2, and the third frame image data group FE3 are DCT-converted and stored as PEG still image data. (See FIG. 3).

As described above, according to the image forming apparatus 100 according to the first embodiment, a series of scenes in which a subject whose feature amount is to be calculated is photographed is detected in a sampling target frame, and frame image data constituting the scene is detected. The feature quantity can be calculated only for. This makes it possible to solve the problem of trade-off between the calculation accuracy of the feature amount and the processing load in the process of extracting the still image data from the moving image data.

In the first embodiment, frame image data is generated for all frames and sampled from the frame image data at a predetermined cycle, but the method is not limited to this method. Specifically, for example, when detecting a sampling target frame, only the sampling target frame may be generated from the moving image data.

Specifically, when the sampling target frame is an I frame, the frame image data can be generated only by the I frame. This is because the I frame is a frame that holds the information of all the frames and is encoded without using the inter-frame prediction. When the sampling target frame is a P frame, the frame image data can be generated by using the related I frame and P frame in the inter-frame prediction. When the sampling target frame is a B frame, the frame image data can be generated by using the related I frame, P frame, and B frame in the inter-frame prediction.

Further, for example, only the I frame may be sampled. In this way, it is not necessary to execute the inter-frame prediction, so that the processing load can be significantly reduced. Alternatively, only the I frame and the P frame may be sampled. This is because the B frame is not used for the inter-frame prediction of the P frame, so that the frame image data can be generated only from the I frame and the P frame.

In sampling I-frames, or I-frames and P-frames, a sampling period can be realized by sampling frames that match the sampling period or frames with the maximum interval within the sampling period, for example. If neither the I frame nor the P frame exists in the sampling cycle, the B frame may be used to generate the frame image data based on the determination.

B. Second embodiment:
FIG. 6 is a flowchart showing the contents of the still image acquisition process according to the second embodiment. FIG. 7 is a flowchart showing the contents of the frame image extraction process according to the second embodiment. The still image acquisition process according to the second embodiment is different from the still image acquisition process according to the first embodiment in that the frame image extraction process (step S60) is changed to the frame image extraction process (step S600). The frame image extraction process (step S600) is characterized in that a frame image in the vicinity of the maximum value of the feature amount in the time series is extracted.

The feature amount is set so as to quantitatively indicate that the still image is a still image desired by the user. Therefore, the maximum value of the feature amount is intended to indicate that the image as intended when the photographer sets up the subject is a frame image at the moment when the image is taken. In this example, the intended image means an image in which the face of a specific person can be photographed large from the front, or the face of a person with a high degree of smile is photographed. The degree of smile can be calculated by, for example, using a well-known 3D model fitting method, and a statistical identification method using an image of the shape of the eyes and mouth and images around the eyes and mouth.

However, the image having the maximum feature amount does not always match the image desired by the user. Specifically, for example, from the perspective of a parent user, an image showing a child's facial expression desired by the parent does not always have a maximum smile level. In the second embodiment, by extracting a frame image in the vicinity of the maximum value of the feature amount, the frame image is narrowed down by the degree of smile or the like, and the user can select a favorite frame image from the frame images.

In step S610, the frame image extraction unit 113 executes the maximum value search process. In the maximum value search process, the frame image extraction unit 113 searches for the maximum value of the feature amount in the time series. The maximum value means the local maximum value (that is, in the vicinity of a certain point) of the feature quantity.

FIG. 8 is an explanatory diagram showing an outline of the frame image extraction process according to the second embodiment. FIG. 8 shows a feature amount curve FC showing a change in the feature amount in a time series. In this example, the feature amount curve FC has a first maximum value LM1, a second maximum value LM2, and a third maximum value LM3. In FIG. 8, the feature amount is normalized so that the minimum value is 0 and the maximum value is 1. The first maximum value LM1 has a feature amount of 0.65, the second maximum value LM2 has a feature amount of 0.8, and the third maximum value LM3 has a feature amount of 0.5.

In this example, the feature curve FC is configured as a spline curve generated using the features of the sampling target frame. The spline curve includes a curve that passes through the feature amount of the sampling target frame and a spline curve (for example, a B-spline curve) that does not necessarily pass through the feature amount of the sampling target frame. The feature amount curve FC is not limited to the spline curve, and may be an approximate curve that passes through the feature amount of the sampling target frame or its vicinity.

In step S620, the frame image extraction unit 113 executes the threshold value calculation process. In the threshold calculation process, the frame image extraction unit 113 calculates the threshold value as 80%, which is a preset predetermined ratio to the maximum value in this example. Specifically, for example, the threshold Th1 for the first maximum value LM1 (feature amount 0.65) is 0.52, and the threshold Th2 for the second maximum value LM2 (feature amount 0.8) is 0.64. Therefore, the threshold Th3 with respect to the third maximum value LM3 (feature amount 0.5) is 0.4.

In step S630, the frame image extraction unit 113 executes the adjacent minimum value search process. In the adjacent minimum value search process, the frame image extraction unit 113 searches for a minimum value that exceeds the threshold value in the vicinity of each maximum value. Specifically, the frame image extraction unit 113 can find a minimum value AL exceeding the threshold value Th3 (0.4) in the vicinity of the third maximum value LM.

In step S640, the frame image extraction unit 113 advances the process to step S655 when the adjacent minimum value exists, and proceeds to the process to step S650 when the adjacent minimum value does not exist. In this example, the frame image extraction unit 113 proceeds to step S650 for the first maximum value LM1 and the second maximum value LM2. On the other hand, the process proceeds to step S655 for the start time of the third maximum value LM, and the process proceeds to step S650 for the end time of the third maximum value LM.

In step S650, the frame image extraction unit 113 acquires the intersection time with the threshold value. The intersection time with the threshold value can be obtained by using a spline curve, or can be obtained as the time of frame image data having a feature amount larger than the threshold value and closest to the threshold value. In this example, the frame image extraction unit 113 acquires the start time t11 and the end time t12 for the first maximum value LM1, and acquires the start time t21 and the end time t22 for the second maximum value LM2. , The final time t32 is acquired for the third maximum value LM3.

In step S655, the frame image extraction unit 113 acquires the time of the adjacent minimum value. The time of the adjacent minimum value can be obtained by using a spline curve, or can be obtained as the time of the frame image data in the vicinity of the adjacent minimum value. In this example, the frame image extraction unit 113 acquires the start time t31, which is the time of the adjacent minimum value AL for the third maximum value LM3.

In step S660, the frame image extraction unit 113 functions as a still image data extraction unit and executes the extraction target selection process. In the extraction target selection process, the frame image extraction unit 113 selects frame image data within a predetermined time range between the start time and the end time in the vicinity of each maximum value as extraction target frame image data. The predetermined time range is, in other words, the time between each maximum value and the start time (or the time of the adjacent minimum value), and the time between each maximum value and the end time (or the time of the adjacent minimum value). Is set using.

In this example, the frame image extraction unit 113 selects frame image data between the start time t11 and the end time t12 for the first maximum value LM1, and the start time t21 and the end time for the second maximum value LM2. The frame image data between the time t22 and the time t22 is selected, and the frame image data between the start time t31 and the end time t32 is selected for the third maximum value LM3.

In step S670, the frame image extraction unit 113 executes the extraction target flag processing. In the extraction target flag processing, the frame image extraction unit 113 sets a flag indicating the extraction target in the extraction target frame image data.

In step S680, the frame image extraction unit 113 executes the grouping process. In the grouping process, the frame image extraction unit 113 groups a plurality of extraction target frame image data for which flags are set for each maximum value (see FIG. 3). As a result, the frame image extraction unit 113 uses the frame image data groups FE1a, FE2a, and FE3a as continuous image data files for the first maximum value LM1, the second maximum value LM2, and the third maximum value LM3, respectively. Can be generated.

As described above, the frame image extraction process according to the second embodiment searches for the maximum value of the feature amount and uses the intersection time (or the time of the adjacent minimum value) with the threshold value of 80% of the maximum value. The frame image data in the vicinity of the maximum value of the quantity is extracted. As a result, the image forming apparatus 100 enables efficient extraction by reducing the number of frames to be extracted when the peak shape (or inclination) of the maximum value is steep, while the peak shape (or inclination) of the maximum value is enabled. ) Is gentle, it is possible to increase the number of frames to be extracted, assuming that the intended image can be stably acquired for a long period of time.

In the present embodiment, the intersection time with the threshold value of 80% of the maximum value is used, but the numerical value is not limited to 80%, and a method of setting the time for determining the vicinity of the maximum value of the feature amount. Is not limited to the method using the threshold value and the crossing time. In the frame image extraction process according to the second embodiment, the number of frames to be extracted is reduced when the peak shape of the maximum value is gentle, and the frame to be extracted is performed when the peak shape of the maximum value is gentle. It suffices if it is configured to increase the number.

Further, in the present embodiment, the feature amount curve FC is configured as an approximate curve generated by using the feature amount of the sampling target frame, but it is not always necessary to use the feature amount of the sampling target frame, and all of them. The frame image data of may be used.

C. Third Embodiment:
FIG. 9 is a flowchart showing the contents of the still image acquisition process according to the third embodiment. In the still image acquisition process according to the third embodiment, the person registration process (step S100) is added, and the feature amount calculation process (step S40) is changed to the feature amount calculation process (step S400). This is different from the still image acquisition process according to the first embodiment and the second embodiment.

FIG. 10 is a flowchart showing the contents of the person registration process according to the third embodiment. In step S110, the user executes a moving image data reading process for the person registration process. In the moving image data reading process, the user selects the moving image data for the person registration process using the operation display unit 130 and instructs the reading of the moving image data. The moving image data can be read by the image forming apparatus 100 by using, for example, wireless communication (not shown) or a portable storage medium (not shown).

In step S120, the feature amount calculation unit 112 of the control unit 110 executes the person detection process. In the person detection process, the feature amount calculation unit 112 generates frame image data from the moving image data, and generates a person detection area which is an image area having characteristics considered to be a person from the still image represented by the frame image data file. Extract. The feature amount calculation unit 112 can extract a person detection region by using machine learning such as SVM (Support Vector Machine) based on, for example, a HOG (Histograms of Oriented Gradients) feature amount.

In step S130, the feature amount calculation unit 112 executes the person classification process. In the person classification process, the feature amount calculation unit 112 classifies a person in the person detection area into, for example, a person in a family registered in advance. In the present embodiment, it is assumed that the father A, the mother B, the son C, and the daughter D are pre-registered as a family by the user using the operation display unit 130.

The feature amount calculation unit 112 selects frame image data in which a person's face is displayed in a size larger than a preset image area, automatically classifies the frame image data into a plurality of groups, and displays the frame image data on the operation display unit 130. To do. The user selects and inputs whether each of the plurality of groups corresponds to the father A, the mother B, the son C and the daughter D, or another person via the operation display unit 130.

The user can further perform a correction operation in response to a false recognition that, for example, the group of son C contains a still image of father A. As a result, the feature amount calculation unit 112 can improve the accuracy of machine learning. The feature amount calculation unit 112 generates a database with the father A, the mother B, the son C, and the daughter D as records. HOG features of facial images of father A, mother B, son C, and daughter D are registered in the database.

In step S140, the feature amount calculation unit 112 executes the clothing selection process. In the clothing selection process, the feature amount calculation unit 112 extracts the HOG feature amount of the clothes worn by the father A, the mother B, the son C, and the daughter D from the frame image data. As a result, the feature amount calculation unit 112 can identify a person by using the HOG feature amount of the face image of the father A, the mother B, the son C, and the daughter D, and the HOG feature amount of the clothing image. .. This is because each person often wears the same outfit, and different people tend to wear different outfits.

In step S150, the feature amount calculation unit 112 executes the database registration process. In the database registration process, the feature amount calculation unit 112 stores the database of the father A, the mother B, the son C, and the daughter D in the person registration data storage area 142 of the storage unit 140. In the database, father A, mother B, son C, and daughter D are recorded, and in addition to the HOG feature amount of each face image, the HOG feature amount of the clothes image, the machine learning data of the face image, and the machine learning data of the clothes image. Includes, height and other user-enterable attribute data.

The user can also register the data of the face image and the clothes image of each person by using the still image data captured by the digital camera. The feature amount calculation unit 112 can use such image data to generate a HOG feature amount of a face image and a HOG feature amount of a clothing image and register them in a database. In this embodiment, it is assumed that the HOG feature amount is generated based on the YUV image data having a small calculation load in image recognition.

FIG. 11 is a flowchart showing the contents of the feature amount calculation process according to the third embodiment. In the feature amount calculation process (step S400), the feature amount calculation unit 112 executes a candidate area high-speed detection process having a relatively small processing load to detect and detect a candidate area which is an area having a predetermined feature. The feature amount for identifying a person in each candidate area is calculated (the processing load is relatively large).

In step S410, the feature amount calculation unit 112 executes the candidate region high-speed detection process. In the candidate area high-speed detection process, the feature amount calculation unit 112 detects the candidate area from the frame image data. The candidate area high-speed detection process employs a detection processing method configured in which the processing speed is prioritized over the person recognition process (described later).

Specifically, the feature amount calculation unit 112 executes a widely used face detection process, for example, a multi-stage filter process using a Har-like detector. However, the multi-stage filter processing is configured to reduce the number of stages and give priority to the calculation speed. Therefore, in the candidate area high-speed detection process, it is allowed to detect an area not desired by the user as a candidate area.

In step S420, the feature amount calculation unit 112 executes the candidate area cutting process. In the candidate area cutting process, the feature amount calculation unit 112 acquires candidate area information which is information indicating the position and size of each candidate area detected in the frame image data which is the bitmap image data, and is a candidate. Bitmap image data of each candidate area is generated using the area information.

In step S430, the feature amount calculation unit 112 compares the total number of pixels (area) of the bitmap image data of the plurality of candidate regions with the total number of pixels (total area) of the frame image data. The inventor of the present application has found that a plurality of candidate regions may overlap each other and the total number of pixels of the plurality of candidate regions may exceed the number of pixels of the frame image data, resulting in an increase in the amount of calculation.

When the total number of pixels of the plurality of candidate regions is less than the number of pixels of the frame image data, the feature amount calculation unit 112 proceeds to step S440, and the total number of pixels of the plurality of candidate regions is the total number of pixels of the frame image data. In the above case, the process proceeds to step S450. Thereby, the feature amount calculation unit 112 can solve the above-mentioned problem of increasing the calculation amount.

In step S440, the feature amount calculation unit 112 calculates each feature amount based on the bitmap image data of each candidate area, and outputs the maximum value as the feature amount. The maximum value is set because there should be only one image of the same person in each bitmap image data. In step S450, the feature amount calculation unit 112 calculates and outputs the feature amount based on the bitmap image data of the frame image data as a whole.

Specifically, the feature amount calculation unit 112 executes the person recognition process. The person recognition process is a process of quantitatively expressing the possibility (probability) of a specific person as a feature quantity. In the person recognition process, it is assumed that the feature selected in step S10 for each of the plurality of frame image data stored in the frame memory 114 (in this example, the daughter D is selected in the feature selection process (step S10)). .) Is included or not. The feature amount calculation unit 112 attempts to detect the daughter D from the frame image data as YUV image data including the luminance data and the color difference data.

The feature amount calculation unit 112 can execute the detection and identification of a person by using, for example, a well-known OpenCV (Open Source Computer Vision Library). The feature amount calculation unit 112 detects a person from the frame image data, and determines whether or not the face of the detected person is the face of the daughter D. This determination is made using the HOG features of the face image of daughter D.

When the accuracy of determining whether or not the face is the daughter D is low, the feature amount calculation unit 112 determines whether or not the daughter D is the daughter D by using the HOG feature amount of the clothes image of the daughter D. The HOG feature amount of the clothes image can be used as a supplement, especially when the daughter D is photographed from the side and the size of the face image is small. The feature amount calculation unit 112 can also quantify the smile degree of the daughter D and use it for calculating the feature amount. In this way, the feature amount calculation unit 112 can execute complicated processing with a large amount of calculation for each candidate area.

As described above, in the frame image extraction process according to the third embodiment, the candidate area high-speed detection process is executed to detect the candidate area which is an area having a predetermined feature, and the calculation load is large in each of the detected candidate areas. Perform feature calculation. As a result, in the third embodiment, the calculation target of the feature amount can be reduced, so that the calculation load of the feature amount is not excessive as a whole, and the processing load is relatively large and the calculation method is highly accurate. Can be adopted.

In the present embodiment, the candidate region high-speed detection process (step S410) and the person recognition process (steps S440 and S450) are different types of processing, but the same type of processing differs only in accuracy. It may be.

Further, the present embodiment can be implemented in combination with at least one of the first embodiment and the second embodiment, or can be implemented without being combined with either the first embodiment or the second embodiment. is there. Further, the frame image extraction process according to the first to third embodiments may be executed not by the image forming apparatus 100 but by the smartphone 200 that functions as an image processing apparatus.

D. Fourth Embodiment:
FIG. 12 is a flowchart showing the contents of the still image acquisition process according to the fourth embodiment. The still image acquisition process according to the fourth embodiment is a still image according to the first to third embodiments in that at least the frame image data output process (step S80) is changed to the print output process (step S800). It is different from the acquisition process.

FIG. 13 is a flowchart showing the contents of the print output process according to the fourth embodiment. In this example, the image forming apparatus 100 is installed in a convenience store or the like, and can be printed out in response to the input of money or other deposit processing.

In step S810, the user inserts money through a slit for inserting money (not shown) in the image forming apparatus 100. In this example, it is assumed that the user has inserted a coin of 500 yen into the image forming apparatus 100. In the image forming apparatus 100, it is assumed that the print size that can be used includes an L plate (127 mm × 89 mm) and a 2 L plate (127 mm × 178 mm). In this example, the L plate (127 mm × 89 mm) is set as the initial setting as the print size, and the unit price is assumed to be 30 yen.

In step S820, the control unit 110 functions as a printable number calculation unit and calculates the printable number. The control unit 110 can print 16 sheets (= 500 yen (amount deposited in the deposit process) ÷ 30 yen (unit price)) based on the initial setting (L version) until the print size is set. Calculate as.

In step S830, the control unit 110 executes the group number calculation process. The group is a plurality of frame image data (constituting a continuous image data file) that are continuously continuous in time series grouped by the grouping process (step S68 and step S680). The groups include, for example, the first frame image data group FE1, the second frame image data group FE2, the third frame image data group FE3 (first embodiment), and the three frame image data groups FE1a, FE2a, FE3a (second embodiment). Form).

In step S840, the control unit 110 determines whether or not the number of printable sheets is equal to or greater than the number of groups. If the number of printable sheets is less than the number of groups, the control unit 110 advances the process to step S850, and if the number of printable sheets is more than the number of groups, the control unit 110 advances the process to step S860. In step S850, the control unit 110 executes the first print target image selection process. The second print target image selection process will be described later.

FIG. 14 is a flowchart showing the contents of the first print target image selection process according to the fourth embodiment. In step S851, the feature amount calculation unit 112 calculates the average feature amount for each group. The average feature amount can be calculated as, for example, the average value of the feature amounts of the top 10 frame image data of each group.

FIG. 15 is an explanatory diagram showing an operation display screen in the first print target image selection process according to the fourth embodiment. In FIG. 15A, representative still images are displayed for each group as a scene selection screen. In this example, it is assumed that a plurality of frame image data groups included in each group constitute each scene. On the operation display screen of FIG. 15, it is possible to enlarge / reduce or move the displayed image by operating (pinch-in, pinch-out, dragging, etc.) on the operation display unit 130 that functions as a touch panel.

In FIG. 15A, the display 131 of the operation display unit 130 has three representative still images F55, F43, and F17, a scroll icon SC1 for scrolling in the high evaluation direction, and scrolling in the low evaluation direction. The scroll icon SC2 and the display area RC for displaying the remaining number of sheets (printable number of sheets) and the number of remaining scenes are displayed. The display 131 functions as a touch panel, and the screen can be scrolled by swiping.

In step S852, the display 131 displays representative still images in descending order of average feature amount. The display 131 displays based on the display data generated by the control unit 110 that functions as the display control unit. The display data is data for displaying a representative still image in association with a characteristic area. The display data is configured to display each characteristic region of the plurality of representative still images so that the relationship with each of the plurality of representative still images can be understood. The order in which the representative still images are displayed is not limited to the average feature amount, and may be any order determined by using the feature amount.

In this example, the display 131 displays three representative still images F55, F43, and F17 in descending order of average feature amount from the left side. The representative still image F55 is a still image represented by the frame image data F55 representing the third frame image data group FE3. The representative still image F43 is a still image represented by the frame image data F43 representing the second frame image data group FE2. The representative still image F17 is a still image represented by the frame image data F17 representing the first frame image data group FE1 (see FIG. 4 of the first embodiment).

In this example, the display 131 displays the feature images Q55, Q43, and Q17, which are images of the feature region, on the upper side of the representative still images F55, F43, and F17. As a result, the user can confirm the criterion (or reason) for which the group including the representative still images F55, F43, and F17 is automatically selected. In this example, it is assumed that each group of the representative still images F55, F43, and F17 is extracted based on the feature amount for extracting the still image including the daughter D having a high smile degree as a subject.

Specifically, the feature image Q55 is an image of a candidate region used for calculating the feature amount when the representative still image F55 is extracted in the frame image extraction process according to the third embodiment. The feature image Q43 is an image of a candidate region used for calculating the feature amount when the representative still image F43 is extracted. The feature image Q17 is an image of a candidate region used for calculating the feature amount when the representative still image F17 is extracted.

In step S853, the user selects a group as a scene selection process. In this example, it is assumed that the user touches the representative still image F43 (or the feature image Q43) displayed on the display 131 to select a scene, and touches the scene selection icon M1 to confirm.

FIG. 15B displays two still images F42 and F44 before and after the representative still image F43 as an image selection screen. In FIG. 15B, the display 131 has three still images F42, F43, F44, feature images Q42, Q43, and Q44, a display area RC for displaying the remaining number of sheets and the number of remaining scenes, and a time series. A scroll icon SC3 for scrolling to the front side and a scroll icon SC4 for scrolling to the rear side are displayed.

As a result, the following trade-off problems can be solved. That is, after extracting still image data from moving image data, in order to select a desired still image from a large number of still images, there is a request to reduce the size of each still image and display a list. There is. On the other hand, if the size of each still image is reduced, the characteristic image area (for example, a face image) of each still image becomes small, which is a trade-off problem that it becomes difficult to select the still image.

In step S854, the user selects frame image data as an image selection process. In this example, it is assumed that the user touches the still image F42 (or the feature image Q42) displayed on the display 131 to select the still image, and touches the image selection icon M2 to confirm.

In step S855, the user executes the print layout setting process. In the print layout setting process, the user extracts a part of the still image F42 and sets the print layout of the print size of the L plate or the 2L plate.

FIG. 16 is a flowchart showing the contents of the print layout setting process according to the fourth embodiment. FIG. 17 is an explanatory diagram showing an operation display screen in the print layout setting process according to the fourth embodiment. FIG. 17A shows an enlarged face of a person displayed in the still image F42. Also on the operation display screen of FIG. 17, it is possible to enlarge / reduce or move the displayed image by operating (pinch-in, pinch-out, dragging, etc.) on the operation display unit 130 that functions as a touch panel.

In this example, the display 131 includes a feature image Q42 in which the still image F42 displays the face of the daughter D, an extracted image F42c in which the face of the son C is displayed, and two other persons who are not registered in the person registration data storage area 142. It is shown that the two extracted images F42a and F42b that display the face of the above are included. The three extracted images F42a, F42b, and F42c are those extracted by the face detection process by the feature amount calculation unit 112 in response to the change of the still image F42.

In step S8551, the user functions as a print layout setting unit and executes a subject person selection process. In the subject person selection process, the user refers to the face of the son C and the face of the daughter D, and changes to the scroll icon SC5 for changing to the front still image and the rear still image in chronological order. You can change the still image in the group by touching the scroll icon SC6. In this example, it is assumed that the user selects the face of the daughter D and the face of the son C and touches the person selection icon M3 to confirm. The three extracted images F42a, F42b, and F42c are extracted and changed according to the change of the still image.

FIG. 17B is an enlarged view of the still image F42L to which the print layout setting process has been applied to the still image F42. In FIG. 17B, the extracted images F42a and F42b are further shown so that the faces of two other people are excluded from the print target area which is the print target area set in the print layout. it's shown. The still image F42L employs a print layout in which the face of daughter D and the face and body of son C are selected, while the faces of two others are excluded. The still image F42L is automatically subjected to the print layout setting process by the following method. The user can touch the layout adjustment M4 to adjust (change) the print layout, and touch the vertical / horizontal change M5 to change the vertical / horizontal of the L version.

In step S8552, the control unit 110 executes the subject number counting process. In the subject number counting process, the control unit 110 counts the number of selected faces as the number of subjects. In step S8553, the control unit 110 determines whether or not the number of subjects is one. If the number of subjects is not one, the control unit 110 advances the process to step S8554, and if the number of subjects is one, the control unit 110 advances the process to step S8555.

FIG. 18 is an explanatory diagram showing an example of the print layout setting process according to the fourth embodiment. FIG. 18 shows a still image F101 displaying the faces of two people H1 and H2 in order to explain the print layout setting process in an easy-to-understand manner.

In step S8554, the control unit 110 sets a rectangle that includes all the subjects. Specifically, the control unit 110 automatically sets a rectangle as a bounding box BB1 that includes the faces of all the two persons H1 and H2 and their surroundings.

FIG. 19 is an explanatory diagram showing another example of the print layout setting process according to the fourth embodiment. FIG. 19 shows still images F102 and F103 displaying the faces of one person H3 in order to explain the print layout setting process in an easy-to-understand manner. In the still image F102, the person H3 has his face and line of sight directed in the shooting direction, which is the direction of the photographer. In the still image F103, the person H3 has his face and line of sight directed to the left.

When the number of subjects is one, the control unit 110 executes the print layout setting process based on the line-of-sight direction of the subjects. In step S8555, the control unit 110 executes the line-of-sight direction estimation process. In the line-of-sight direction estimation process, the control unit 110 can estimate by adopting, for example, the method proposed in Japanese Patent Application Laid-Open No. 8-322796.

Japanese Patent Application Laid-Open No. 8-322996 is based on analysis of a face image, that is, an image of an eye part is cut out from an image of a face part, a feature amount representing eye position information is extracted, and a comparison between the feature amount and a standard pattern is performed. We are proposing a method for estimating the line-of-sight direction. The feature amount includes the density distributions in the horizontal direction and the vertical direction corresponding to the black eyes of the eye portion of the shade image.

In step S8556, the control unit 110 arranges a space in the line-of-sight direction to set the print layout. In photography, a composition with a sense of stability is generally adopted so that the space is arranged in the direction of the subject's line of sight or the direction of the face. In this way, the control unit 110 can set the print layout according to the subject direction, which is at least one of the line-of-sight direction and the face direction, and can automatically adopt a stable composition.

In the still image F102, the control unit 110 automatically sets a rectangle as a bounding box BB2 that includes the person H3 and its surroundings from the still image F102. Since the face and the line of sight of the person H3 are directed toward the photographer, the control unit 110 adopts a composition in which the person H3 is evenly spread in both side directions from the bounding box BB2.

On the other hand, in the still image F103, the control unit 110 automatically sets a rectangle as a bounding box BB3 that includes the person H3 and its surroundings from the still image F103. Since the face and the line of sight of the person H3 are directed to the left side, the control unit 110 adopts a composition in which a relatively large space is arranged on the left side from the bounding box BB3.

The larger the angle of the subject direction with respect to the shooting direction, the larger the ratio of the space arranged on the subject direction side to the space arranged on the opposite side of the subject direction may be increased. Further, when the number of persons is plural and the subject directions of the plurality of faces are oriented in the same direction, a wide space may be arranged on the side in the same direction.

In step S8557, the control unit 110 executes the margin setting process. In the margin setting process, the control unit 110 automatically arranges a general margin assumed when a person is photographed by a camera and sets the frame frames Fr1, Fr2, and Fr3. This setting is made assuming the aspect ratio of the L plate (127 mm × 89 mm) as the initial setting.

Since the aspect ratio of a moving image is generally different from the aspect ratio of a still image, the composition intended at the time of shooting the moving image may not be maintained when editing the aspect ratio of the still image. Further, in moving image shooting, both the photographer and the subject may move, and it is generally difficult to shoot with consideration for composition.

In response to such a problem, in the present embodiment, the control unit 110 realizes an appropriate composition assumed when a person is photographed with a still camera, and automatically arranges a general margin to form a frame frame. Fr1, Fr2, and Fr3 can be set to the L version. As a result, the intended composition is maintained when shooting a moving image, and support such as semi-automatic setting of a stable and preferable composition is performed when shooting a moving image without being conscious of the composition, which burdens the user. Can be mitigated.

In step S856 (see FIG. 14), the print setting process is executed. In the print setting process, the user sets the number of prints and the content of the image processing, and touches the setting completion M6 (see FIG. 17B) to complete the print setting process.

In step S857, the control unit 110 executes the printable number update process. In the printable number update process, the control unit 110 calculates the printable remaining number, assuming that the print process is executed for the still image for which the setting has been completed. In this example, since printing of one still image F42L is set, the number of printable sheets (remaining number of sheets) is updated from 16 to 15 in the display area RC, and the number of remaining scenes is updated from 18 to 17. Will be done.

The control unit 110 executes the processes of steps S851 to S857 up to the final group as the first print target image selection process (step S858). In this example, the number of remaining scenes is the number of unprocessed groups that are displayed after the group that contains the still image data selected for printing.

On the other hand, when the number of printable sheets is equal to or greater than the number of groups, the control unit 110 proceeds to step S860 (see FIG. 13). In step S860, the control unit 110 executes the second print target image selection process. The second print target image selection process is different from the first print target image selection process in that the representative still images are displayed in chronological order for each group, and the representative still images are displayed in descending order of the average feature amount.

When the number of printable sheets is equal to or greater than the number of groups, at least one print output process can be performed for each group, so that the control unit 110 can display representative still images for each group in chronological order. On the other hand, when the number of printable sheets is less than the number of groups, the user is desired to select the group to perform the print output processing, so that the representative still images are displayed in descending order of the average feature amount.

Further, the control unit 110 can display the number of printable sheets (remaining number of sheets) and the number of remaining scenes in the display area RC of the operation display unit 130. As a result, the user can smoothly select a group (scene). For example, when the number of printable sheets is larger than the number of remaining scenes (number of remaining groups), the user can select additional images from each group.

When extracting still image data from moving image data, it is generally difficult to predict the number of still images extracted. For example, when printing is performed using an image forming device installed in a convenience store or the like while traveling, the number of extracts cannot be predicted, so the burden of selecting a still image to be printed while considering the fee to be paid is a burden. Is big. This embodiment can reduce the burden of selecting desired still image data from a plurality of still image data extracted from moving image data.

In step S870, the control unit 110 forms an image on the print medium by using the image forming unit 120 in response to the pressing of the operation processing unit (start switch) 132. The print target is the still image data for which the print setting process is completed in step S856 (see FIG. 14).

The control unit 110 may form an image on the print medium immediately upon completion of the print setting process. However, according to the above-described embodiment, there is an advantage that the still image data can be reselected by allowing the print setting process to be canceled.

As described above, in the still image acquisition process according to the fourth embodiment, a scene (group) is selected from the still image data extracted from the moving image data while checking the state of the face of the subject, and the scene (group) is desired from the group. Still images can be selected smoothly.

In the still image acquisition process according to the fourth embodiment, it is also possible to automatically perform layout setting processing with an appropriate composition in consideration of the state of the subject from the selected still image. As described above, the still image acquisition process according to the fourth embodiment can realize an appropriate print output of a still image while reducing the burden on the user in the print process using the moving image data.

E. Modification example:
The present invention can be implemented not only in the above embodiments but also in the following modifications.

In the above embodiment, the present invention is applied to an image forming apparatus, but it can also be applied to an apparatus functioning as an image processing apparatus such as a smartphone or a personal computer.

100 Image forming device 110 Control unit 111 Frame image generation unit 112 Feature amount calculation unit 113 Frame image extraction unit 114 Frame memory 115 Period setting unit 120 Image forming unit 130 Operation display unit 140 Storage unit 150 Communication interface unit 200 Smartphone

Claims (7)

An image forming device that executes a printing process according to a deposit process.
A still image data generator that generates multiple still image data from moving image data,
A feature amount representing a feature related to a person's face in the still image data is calculated, the still image data is extracted based on the feature amount, and a scene is continuously generated from the extracted plurality of still image data in a time series. A still image data extraction unit that generates at least one group that is a continuous image data file composed of a plurality of still image data constituting the above.
Using the amount deposited in the deposit process and the unit price of the print process, the number of printable sheets is calculated by dividing the amount by the unit price, and the number of groups, which is the number of the generated groups, is calculated. In addition, a printable number calculation unit for determining whether or not the printable number is equal to or greater than the group number
When the number of printable sheets is less than the number of the groups, a group of representative still images representing the continuous image data file is set as an image forming target for each group in the order determined by using the feature amount. When the number of printable sheets is equal to or greater than the number of the groups, the representative still image groups are set as image formation targets for each group in the order determined based on the time series of the groups. The operation display that can be selected and displayed
An image forming apparatus comprising. An image forming device that executes a printing process according to a deposit process.
A still image data generator that generates multiple still image data from moving image data,
A feature amount representing a feature related to a person's face in the still image data is calculated, the still image data is extracted based on the feature amount, and a scene is continuously generated from the extracted plurality of still image data in a time series. A still image data extraction unit that generates at least one group that is a continuous image data file composed of a plurality of still image data constituting the above.
Using the amount deposited in the deposit process and the unit price of the print process, the number of printable sheets is calculated by dividing the amount by the unit price, and the number of groups, which is the number of the generated groups, is calculated. At the same time, a printable number calculation unit for determining whether or not the printable number is equal to or greater than the group number, and
When the number of printable sheets is less than the number of the groups, a group of representative still images representing the continuous image data file is set as an image forming target for each group in the order determined by using the feature amount. Operation display unit that can be selected and displayed
Equipped with a,
The operation display unit determines the order in descending order of the average feature amount, which is the average value of the feature amounts of the still image data included in the continuous image data file in each of the groups.
The average feature amount, the image forming apparatus is a predetermined mean value of the feature amount of the higher number in each of the groups. The image forming apparatus according to claim 1 or 2 .
The operation display unit displays the number of printable sheets and the number of remaining groups, accepts an input for including the still image data included in the group in the print target, and updates the printable number of sheets in response to the input. And
An image forming apparatus in which the number of remaining groups is the number of groups displayed in an order after the group including still image data included in the print target. An image forming method that executes a printing process according to the deposit process.
A still image data generation process that generates multiple still image data from moving image data,
A feature amount representing a feature related to a person's face in the still image data is calculated, the still image data is extracted based on the feature amount, and a scene is continuously generated from the extracted plurality of still image data in a time series. A still image data extraction step that generates at least one group that is a continuous image data file composed of a plurality of still image data constituting the above.
Using the amount deposited in the deposit process and the unit price of the print process, the number of printable sheets is calculated by dividing the amount by the unit price, and the number of groups, which is the number of the generated groups, is calculated. At the same time, a printable number calculation step for determining whether or not the printable number is equal to or greater than the group number , and
When the number of printable sheets is less than the number of the groups, a group of representative still images representing the continuous image data file is set as an image forming target for each group in the order determined by using the feature amount. When the number of printable sheets is equal to or greater than the number of the groups, the representative still image groups are set as image formation targets for each group in the order determined based on the time series of the groups. Operation display process to display selectably and
An image forming method comprising. An image forming method that executes a printing process according to the deposit process.
A still image data generation process that generates multiple still image data from moving image data,
A feature amount representing a feature related to a person's face in the still image data is calculated, the still image data is extracted based on the feature amount, and a scene is continuously generated from the extracted plurality of still image data in a time series. A still image data extraction step that generates at least one group that is a continuous image data file composed of a plurality of still image data constituting the above.
Using the amount deposited in the deposit process and the unit price of the print process, the number of printable sheets is calculated by dividing the amount by the unit price, and the number of groups, which is the number of the generated groups, is calculated. At the same time, a printable number calculation step for determining whether or not the printable number is equal to or greater than the group number, and
When the number of printable sheets is less than the number of the groups, a group of representative still images representing the continuous image data file is set as an image forming target for each group in the order determined by using the feature amount. Operation display process to be displayed selectably and
With
The operation display step includes a step of determining the order in descending order of the average feature amount, which is the average value of the feature amounts of the still image data included in the continuous image data file in each of the groups.
An image forming method in which the average feature amount is a higher average value of a preset number of the feature amounts in each of the groups. An image forming program for controlling an image forming apparatus that executes a printing process according to a deposit process.
Still image data generator, which generates multiple still image data from moving image data,
A feature amount representing a feature related to a person's face in the still image data is calculated, the still image data is extracted based on the feature amount, and a scene is continuously generated from the extracted plurality of still image data in a time series. Still image data extraction unit that generates at least one group that is a continuous image data file composed of a plurality of still image data constituting
Using the amount deposited in the deposit process and the unit price in the print process, the number of printable sheets is calculated by dividing the amount by the unit price, and the number of groups, which is the number of generated groups, is calculated. At the same time, a printable number calculation unit for determining whether or not the number of printable sheets is equal to or greater than the number of the groups , and a determination using the feature amount when the number of printable sheets is less than the number of the groups. A group of representative still images representing the continuous image data file is selectively displayed as an image formation target for each of the groups in the order in which they are printed, and when the number of printable sheets is equal to or greater than the number of the groups, the group. An image forming program that causes an image forming apparatus to function as an operation display unit that selectively displays a group of representative still images as an image forming target for each of the groups in an order determined based on the time series of . An image forming program for controlling an image forming apparatus that executes a printing process according to a deposit process.
Still image data generator, which generates multiple still image data from moving image data,
A feature amount representing a feature related to a person's face in the still image data is calculated, the still image data is extracted based on the feature amount, and a scene is continuously generated from the extracted plurality of still image data in a time series. Still image data extraction unit that generates at least one group that is a continuous image data file composed of a plurality of still image data constituting
Using the amount deposited in the deposit process and the unit price of the print process, the number of printable sheets is calculated by dividing the amount by the unit price, and the number of groups, which is the number of the generated groups, is calculated. At the same time, a printable number calculation unit for determining whether or not the printable number is equal to or greater than the group number, and
When the number of printable sheets is less than the number of the groups, a group of representative still images representing the continuous image data file is set as an image forming target for each group in the order determined by using the feature amount. The image forming device is made to function as an operation display unit for display in a selectable manner.
The operation display unit determines the order in descending order of the average feature amount, which is the average value of the feature amounts of the still image data included in the continuous image data file in each of the groups.
An image forming program in which the average feature amount is a higher average value of a preset number of the feature amounts in each of the groups.