JP6512208B2 - Image processing apparatus, image processing method and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

In recent years, the number of pixels of imaging devices has been increased, or the standard for moving images has been increased from standard image quality to full high vision, and further increased to 4K, so that sufficient image quality can be obtained even if portions are cut out from one screen. It is coming.
In such a flow, there is disclosed a technique of performing cutout in a range that does not degrade the image quality while tracking a person of interest (for example, see Patent Document 1).

JP, 2013-115597, A

  However, the conventional technique of tracking and extracting the person of interest is effective when both the temporal position and the spatial position in the image are close even if one or more persons of interest are in the image. Even if there is a problem, the effect can not be obtained when any one of the temporal position and the spatial position in the images of a plurality of persons is separated. This applies not only to the person but also to the case where the portion of interest in the image is apart.

  An object of the present invention is to generate an effective image regardless of the positional relationship of a plurality of objects present in the image.

In order to achieve the above object, an image processing apparatus according to an aspect of the present invention is
By cutting out a frame period in which the first subject or the second subject is present from one moving image in which a first subject and a second subject existing in a separated frame period are photographed, the video Cutting means for cutting out an image,
The moving image corresponding to the first object cut out by the cutting out means is combined with the moving image corresponding to the second object according to the ranking of the objects based on the temporal combination criteria Means for generating one new moving image,
And the like.

  According to the present invention, an effective image can be generated regardless of the positional relationship of a plurality of objects present in the image.

It is a block diagram showing the composition of the hardware of the image processing device concerning one embodiment of the present invention. FIG. 5 is a functional block diagram showing a functional configuration for executing image generation processing among the functional configurations of the image processing apparatus of FIG. 1. It is a figure which shows an example of a layout table. FIG. 4A is a schematic view showing an example of layout, FIG. 4A is a view showing a layout 2A, FIG. 4B is a view showing a layout 2B, FIG. 4C is a layout 3G, and FIG. 4D is a layout 3H. . 5 (A) and 5 (B) are spatial clipping criteria 1, FIGS. 5 (C) and 5 (D) are spatial clipping criteria 2, FIG. 5 (E). FIG. 5F is a schematic diagram showing the concept of the spatial cutout criterion 4 and FIG. 5G, FIG. 5H, and FIG. It is a schematic diagram which shows the concept of the whole image generation procedure. It is a schematic diagram which shows the concept of the whole image generation procedure. It is a schematic diagram which shows the concept of the whole image generation procedure. It is a schematic diagram which shows the concept of the whole image generation procedure. It is a flowchart explaining the flow of the image generation process which the image processing apparatus of FIG. 1 which has the functional structure of FIG. 2 performs. It is a flowchart explaining the flow of the cutting-out process of step S3 in FIG. It is a flowchart explaining the flow of the spatial coupling process of step S7 in FIG. It is a flowchart explaining the flow of the temporal connection process of step S8 in FIG. It is a flowchart explaining the flow of the image generation process which the image processing apparatus in 2nd Embodiment performs. It is a flowchart explaining the flow of the cutting-out process of step S103 in FIG. It is a flowchart explaining the flow of the joint process of step S106 in FIG. It is a schematic diagram which shows the concept which puts together several subjects and it cuts out spatially.

  Hereinafter, embodiments of the present invention will be described using the drawings.

First Embodiment
[Constitution]
FIG. 1 is a block diagram showing a hardware configuration of an image processing apparatus 1 according to an embodiment of the present invention.
The image processing apparatus 1 is configured as, for example, a digital camera.

  The image processing apparatus 1 includes a central processing unit (CPU) 11, a read only memory (ROM) 12, a random access memory (RAM) 13, a bus 14, an input / output interface 15, an imaging unit 16, and an input unit. 17, an output unit 18, a storage unit 19, a communication unit 20, and a drive 21.

  The CPU 11 executes various processes in accordance with a program stored in the ROM 12 or a program loaded from the storage unit 19 into the RAM 13. For example, the CPU 11 executes an image generation process according to a program for an image generation process described later.

  Data and the like necessary for the CPU 11 to execute various processes are also stored in the RAM 13 as appropriate.

  The CPU 11, the ROM 12 and the RAM 13 are connected to one another via a bus 14. An input / output interface 15 is also connected to the bus 14. An imaging unit 16, an input unit 17, an output unit 18, a storage unit 19, a communication unit 20, and a drive 21 are connected to the input / output interface 15.

  Although not shown, the imaging unit 16 includes an optical lens unit and an image sensor.

The optical lens unit is configured of a lens that collects light, such as a focus lens or a zoom lens, in order to capture an object.
The focus lens is a lens that forms an object image on the light receiving surface of the image sensor. The zoom lens is a lens that freely changes the focal length in a certain range.
The optical lens unit is also provided with peripheral circuits for adjusting setting parameters such as focus, exposure, white balance, etc., as necessary.

The image sensor includes a photoelectric conversion element, an AFE (Analog Front End), and the like.
The photoelectric conversion element is composed of, for example, a complementary metal oxide semiconductor (CMOS) type photoelectric conversion element or the like. A subject image is incident on the photoelectric conversion element from the optical lens unit. Therefore, the photoelectric conversion element photoelectrically converts (captures) an object image, accumulates an image signal for a certain period of time, and sequentially supplies the accumulated image signal as an analog signal to the AFE.
The AFE performs various signal processing such as A / D (Analog / Digital) conversion processing on this analog image signal. A digital signal is generated by various signal processing, and is output as an output signal of the imaging unit 16.
Hereinafter, such an output signal of the imaging unit 16 is referred to as a “captured image”. The captured image is appropriately supplied to the CPU 11 and the like.

The input unit 17 includes various buttons and the like, and inputs various information in accordance with a user's instruction operation.
The output unit 18 is configured by a display, a speaker, and the like, and outputs an image and sound.
The storage unit 19 is configured by a hard disk, a flash memory, or the like, and stores data of various images.
The communication unit 20 controls communication with other devices (not shown) via a network including the Internet.

  A removable medium 31 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory or the like is appropriately attached to the drive 21. The program read from the removable media 31 by the drive 21 is installed in the storage unit 19 as needed. The removable media 31 can also store various data such as image data stored in the storage unit 19 in the same manner as the storage unit 19.

FIG. 2 is a functional block diagram showing a functional configuration for executing an image generation process among the functional configurations of such an image processing apparatus 1.
In the image generation processing, from one moving image in which a plurality of persons are photographed, the face portion of each person is spatially or temporally cut out based on registered face information, and the cut out moving images are It refers to a series of processes that combine to generate a new moving image.

  When the image generation process is executed, as shown in FIG. 2, in the CPU 11, the image selection unit 51, the cutout reference identification unit 52, the cutout processing unit 53, the connection reference identification unit 54, and the connection processing unit 55 in the CPU 11. , And the layout selection unit 56 function.

  Further, in one area of the storage unit 19, an image storage unit 71, a layout storage unit 72, a generated image storage unit 73, and a face information storage unit 74 are set.

The image storage unit 71 stores data of moving images captured by the image processing device 1 or another device.
The layout storage unit 72 stores layout data used when generating a new moving image, and a layout table in which conditions for selecting a layout are defined.

FIG. 3 is a diagram showing an example of the layout table.
As shown in FIG. 3, the layout table has the highest priority indicating the number of cut-out images indicating the number of portions cut out from the moving image and the portion to which the highest priority is set among the portions cut out from the moving image. The extraction target of and the corresponding layout are stored in association with each other.
Specifically, in the layout table, numerical values such as “2, 3” as the number of cut-out images, “faces of persons A, B, C”, “faces of front, right, left etc.” Attributes of the extraction target such as “direction” and “no top priority extraction target” and layout data such as “layout 2A, 2B... 2N, layout 3A, 3B. There is.

FIG. 4 is a schematic view showing an example of the layout, FIG. 4 (A) is a layout 2A, FIG. 4 (B) is a layout 2B, FIG. 4 (C) is a layout 3G, and FIG. 4 (D) is a layout 3H. FIG.
As shown in each schematic diagram of FIG. 4, the layout defines the background, the number of images to be combined with the background, and the size and positional relationship. As in the layout 2A of FIG. 4A and the layout 2B of FIG. 4B, when an image is combined on the entire screen, only a frame is displayed as a background.

Specifically, in the layout, a combination position of a predetermined number of images to be combined with the background is set, and priority is set to each combination position. Also, the size of the image to be combined is set at each combining position.
For example, layout 2A in FIG. 4A is a layout in which two images are arranged in the same size at the combining position arranged in the left and right direction, and the combining position on the left is priority 1 and the combining position on the right is priority It is in the second place. In addition, layout 3G of FIG. 4C is a layout in which three images are arranged at the synthesis positions aligned in the upper left, center, lower right, with the central synthesis position larger and the upper left and lower lower synthesis positions smaller. The center combining position has priority 1, the upper left combining position has priority 2, and the lower right combining position has priority 3.

Returning to FIG. 2, the generated image storage unit 73 stores data of a new moving image obtained by combining the cut out moving images according to the layout.
The face information storage unit 74 stores authentication data of the face of the person (front, right, left). The identification data of the face of the person is used to identify whether the face detected in the moving image is the face of a specific person. Note that various known face detection techniques and face authentication techniques can be used for the process of detecting a face from a moving image and the process of authenticating whether or not the detected face is a specific person.

The image selection unit 51 selects, from among the data of the images stored in the image storage unit 71, the data of the moving image corresponding to the user's instruction input. Hereinafter, data of the selected moving image is referred to as "original moving image".
The cutout reference specifying unit 52 specifies a cutout reference when cutting out a face portion of a person from a moving image.
In the present embodiment, a spatial cutout criterion and a temporal cutout criterion are defined as the cutout criterion, and the cutout criterion identifying unit 52 uses the spatial cutout previously used when the moving image cutout is performed. Identify either the reference or the temporal clipping criteria. Note that the extraction criterion specification unit 52 may receive an identification of an arbitrary extraction criterion by the user.
Specifically, the spatial clipping criterion and the temporal clipping criterion are set as follows.

(Spatial cutout criteria 1)
Each face detected and identified as a specific person on the screen of each frame constituting the moving image is to be extracted.
(Spatial cutout criteria 2)
A face detected on a screen of each frame constituting a moving image and a face authenticated as a specific person are extracted as extraction targets.
(Spatial cutout criteria 3)
A face authenticated as a specific person in the screen of each frame constituting a moving image is not to be extracted.
(Spatial cutout criterion 4)
The face / right / left direction of one face detected on the screen of each frame constituting a moving image or one face identified as a specific person is separately extracted. However, if a plurality of faces are detected or authenticated by giving priority to the authenticated face over the detected face, the priority set to the face registered as the authentication target, or the detected or authenticated face Determine the extraction target according to the size of.

(Temporal cutout criteria 1)
When a face is detected in consecutive frames constituting a moving image or when a face is recognized as a specific person, regardless of whether or not the detected face and the recognized face exist, the same time period Cut out a moving image corresponding to each face by the length. However, during the frame period in which the face is not detected or authenticated, the original moving image is inserted as it is, so that the same frame number is obtained.
(Temporal cutout criteria 2)
When a face is detected in consecutive frames constituting a moving image or when a face is recognized as a specific person, regardless of whether or not the detected face and the recognized face exist, the same time period Cut out a moving image corresponding to each face by the length. However, during the frame period in which no face is detected or recognized, the temporally compressed moving image is cut out without cutting out, and the frame rate of another short moving image is adjusted to match the length of the longest moving image. By reducing it, it is made to be the same time even for different number of frames.

(Temporal cutout criteria 3)
When a face is detected in consecutive frames constituting a moving image or when a face is identified as a specific person, only the frame period of the detected face and the portion where the identified face is present is cut out, Cut out a temporally compressed video. However, in the part to be extracted, the time during which the face is continuously detected or recognized is a part that is equal to or more than a predetermined threshold time. Note that the face of the detected or authenticated face may be cut out temporally by distinguishing between front / right / left.

  The cutout processing unit 53 executes cutout processing to cut out the face portion of the person from the moving image in accordance with the cutout reference specified by the cutout reference specifying unit 52. Specifically, the cutout processing unit 53 searches for an original moving image, and identifies a subject to be cutout. Then, the cutout processing unit 53 spatially cuts out a rectangular area including the face of a specific person in the screen for each frame, or includes the face of the specific person according to the cutout criteria specified by the cutout criteria specifying unit 52. Cut out the frame period in time. Note that depending on the layout, a portion of the region that has been cut out may be cut, and the size, aspect ratio, and the like may be changed, in the case of combining with the layout to be described later. Therefore, in the case where the subject to be cut out is spatially cut out in the cut-out processing unit 53, it may be cut out including a certain margin area around the subject.

The combination reference specification unit 54 specifies a combination reference when combining moving images cut out by the cut-out processing unit 53.
In the present embodiment, a spatial coupling criterion and a temporal coupling criterion are defined as the coupling criterion, and the coupling criterion specifying unit 54 is a spatial coupling previously used when the combination of moving images is performed. Identify either a reference or a temporal binding reference. Note that the combination criteria specification unit 54 may receive specification of any combination criteria by the user.
Specifically, the spatial coupling criteria and the temporal clipping criteria are set as follows.

(Spatial coupling criterion 1)
A layout corresponding to the number of cut-out images and a cut-out target registered in advance and having the highest priority is selected, and assigned to the area of the synthesis position in the order of the priority of the cut-out moving images.
(Spatial coupling criterion 2)
The layout corresponding to the number of cut-out images and the cut-out target having the largest cut-out portion is selected, and assigned to the area of the synthesis position in the order of priority of the cut-out moving images.
(Spatial coupling criterion 3)
Select the layout corresponding to the number of cut-out images and the cut-out target with the longest cut-out time (in the case of changing the length of time after cut-out, the original cut-out time), and order of priority of cut-out moving images , Assign to the area of the composite position.

(Temporal coupling criteria 1)
It combines in order of the priority registered beforehand. However, if a plurality of faces are detected or authenticated by giving priority to the authenticated face over the detected face, the priority set to the face registered as the authentication target, or the detected or authenticated face The order of combination is determined according to the size of.
(Temporal coupling criteria 2)
Combine in order of the size of the cut out part.
(Temporal coupling criteria 3)
It combines in order of the length of cutting out time (When changing the length of time after cutting out, original moving image is not inserted, or original cutting out time which is not reducing frame rate).

The combination processing unit 55 combines the moving images cut out by the cut-out processing unit 53 in accordance with the combination standard specified by the combination standard specification unit 54. Specifically, the combining processing unit 55 performs spatial combining processing for combining the extracted moving images according to the spatial combining criteria, or temporal combining processing for combining the extracted moving images according to the temporal combining criteria Run.
The layout selection unit 56 refers to the layout table and selects layout data based on the number of cut-out images and the highest-priority cut-out target when the connection processing unit 55 performs spatial connection of moving images.

[Concept of image generation procedure]
FIG. 5 is a schematic view showing the concept of the spatial clipping standard, and FIGS. 5A and 5B are spatial clipping standard 1, FIG. 5C and FIG. 5D are spatial clipping standard 2, and FIG. 5 (E) and 5 (F) are schematic diagrams showing the concept of the spatial segmentation standard 4, and FIGS. 5 (G), 5 (H) and 5 (I) are the spatial segmentation standard 4. FIG. In addition, the broken line in each schematic diagram of FIG. 5 has shown the spatial cutting out area | region.
In the spatial cutout criterion 1, as shown in FIG. 5A, when the moving image includes the faces of the persons A to C who are authenticated as the specific person, the faces of the persons A to C are regarded as cutout objects, respectively. Become. Further, as shown in FIG. 5B, when the moving image includes the faces of the persons A and B who have been authenticated as the specific person and the faces of the other person X, the faces of the persons A and B are respectively The face of the person X is not to be extracted.

  In the spatial cutout criterion 2, as shown in FIG. 5 (C), when the moving image includes the faces of the persons A to C authenticated as the specific person, the person A is similar to FIG. 5 (A). Each face of ~ C is to be extracted. On the other hand, as shown in FIG. 5D, when the moving image includes the faces of the persons A and B identified as the specific person and the faces of the other person X, the faces and persons of the persons A and B are included. Each face of X is to be extracted.

In the spatial cutout criterion 3, as shown in FIG. 5 (E), when the moving image includes the faces of the persons A to C authenticated as the specific person, the faces of the persons A to C are to be extracted. Absent. Further, as shown in FIG. 5F, when the moving image includes the faces of the persons A and B authenticated as the specific person and the faces of the other person (unspecified person) X, the person A The faces of B and C are not to be extracted, and the face of person X is to be extracted.
In the spatial cutout criterion 4, as shown in FIGS. 5G to 5I, when the moving image includes the face of the person authenticated as a specific person or the face of another person, the highest priority is given. The face orientation is the front, the right orientation, and the left orientation are separately extracted.

6 to 9 are schematic diagrams showing the concept of the entire image generation procedure. 6 to 9 illustrate the concept of the case where a moving image is generated by the temporal connection criterion and the spatial connection criterion, taking the case where the moving image is extracted by the temporal cutout criterion as an example.
Specifically, FIG. 6 shows a case where a moving image is cut out according to temporal cut-out criterion 1, FIG. 7 shows a case where a moving image is cut out according to temporal cut-out criterion 2, FIG. In the case where cut-out frames overlap between the cut-out moving images, FIG. 9 is the case where the moving image is cut out according to temporal cut-out reference 3 and the cut-out frames do not overlap between cut-out moving images Is a diagram illustrating an example of

As shown in FIG. 6, according to the temporal cutout standard 1, when the face of the persons A, B, and C authenticated as the specific person is included in the original moving image shown in FIG. Moving images corresponding to the faces of the persons A, B, and C are cut out by the length (see FIG. 6B). In the original moving image shown in FIG. 6A, the faces of the persons A, B, and C are authenticated in the period of the entire frame, but as in the original moving image shown in FIG. 7A. If there is a period of frame that has not been detected or authenticated, the original moving image is inserted in that period. Therefore, the time, the number of frames, and the frame rate of each moving image to be cut out are the same as the original moving image.
At this time, it is assumed that the priorities registered in advance are higher in the order of person A, person B and person C, and the size of the cut out part is larger in the order of person B, person A and person C.

Then, as shown in FIG. 6C, when combining the cut out moving images according to the temporal combination criterion 1, according to the priority registered in advance, in the order of person A, person B, person C, The cut out moving images are combined.
On the other hand, when the cut out moving images are combined according to the temporal combination criterion 2, the cut out moving images are combined in the order of the person B, the person A and the person C according to the size of the cut out portion. Also, when combining according to the temporal combining criteria 3, the time of the moving image to be cut out is the same for person A, person B and person C, but the original moving image as in the original moving image shown in FIG. If the length of the original clipped moving image before inserting the image is different, the combination is performed in the order of the person A, the person B, and the person C according to the order of the lengths.
Further, as shown in FIG. 6D, when combining the cut out moving images according to the spatial connection criterion 1, the area of the synthesis position of the person A in the priority 1 according to the priority registered in advance. The person B is joined to the area of the synthesis position of priority 2 and the person C is joined to the area of the synthesis position of priority 3. At this time, since the number of cut-out images is 3 and the person A is the top-priority cut-out target, the layout 3A is selected with reference to the layout table.

  On the other hand, when combining the cut out moving images according to the spatial connection standard 2, the person B is a region of the combining position of priority 1 and the person A is a combining position of priority 2 according to the size of the cut out portion. , The person C is combined with the area of the priority 3 composite position. Further, when combining according to the spatial combining criterion 3, the time of the moving image to be cut out is the same for person A, person B and person C, but the original moving image as in the original moving image shown in FIG. If the length of the original clipped moving image before inserting the image is different, it is combined with the area of the synthesis position having the highest priority in the order of person A, person B and person C according to the order of length. At this time, since the number of cut-out images is 3 and the person B is the cut-out target of top priority, the layout 3B is selected with reference to the layout table.

In addition, as shown in FIG. 7, in the temporal clipping reference 2, when the original moving image shown in FIG. 7A includes the faces of persons A, B, and C authenticated as a specific person, the same process is performed. Moving images corresponding to the faces of the persons A, B, and C are cut out by the length of time (see FIG. 7B). In the original moving image shown in FIG. 7A, the faces of the persons B and C are not authenticated in the period of some frames, but there is a period of frames in which no face is detected or authenticated. Is compressed without cutting out its period. Therefore, to match the length of time, the frame rate is reduced compared to the original moving image.
At this time, the priority registered in advance is higher in the order of person A, person B and person C, and the size of the extracted part is larger in order of person B, person A and person C and the frame of the extracted moving image It is assumed that the rate is higher in the order of person A, person B and person C (that is, the length before matching the length of time is longer in the order of person A, person B and person C).

Then, as shown in FIG. 7C, when combining the cut out moving images according to the temporal combination criterion 1, according to the priority registered in advance, in the order of person A, person B, person C, The cut out moving images are combined.
On the other hand, when the cut out moving images are combined according to the temporal combination criterion 2, the cut out moving images are combined in the order of the person B, the person A and the person C according to the size of the cut out portion.
Furthermore, when combining the extracted moving images according to the temporal combination criterion 3, the length of the extraction time is the same, but in this case, the length of the original extracted image before reducing the frame rate Accordingly, the cut-out moving images are combined in the order of the person A, the person B, and the person C.

Further, as shown in FIG. 7D, when combining the cut out moving images in accordance with the spatial connection criterion 1, the area of the synthesis position of the person A in the priority 1 according to the priority registered in advance. The person B is joined to the area of the synthesis position of priority 2 and the person C is joined to the area of the synthesis position of priority 3. At this time, since the number of cut-out images is 3 and the person A is the top-priority cut-out target, the layout 3A is selected with reference to the layout table.
On the other hand, when combining the cut out moving images according to the spatial connection standard 2, the person B is a region of the combining position of priority 1 and the person A is a combining position of priority 2 according to the size of the cut out portion. , The person C is combined with the area of the priority 3 composite position. At this time, since the number of cut-out images is 3 and the person B is the cut-out target of top priority, the layout 3B is selected with reference to the layout table.

  Furthermore, if the clipped moving images are combined according to the spatial combining criterion 3, the length of the cutout time is the same, but in this case, the length of the original clipped moving image before reducing the frame rate Accordingly, the person A is combined with the area of the synthesis position of priority 1, the area of the person B with the synthesis position of priority 2, and the person C with the area of the synthesis position of priority 3. At this time, since the number of cut-out images is 3 and the person A is the top-priority cut-out target, the layout 3A is selected with reference to the layout table.

In addition, as shown in FIG. 8, with temporal segmentation standard 3, when the original moving image shown in FIG. 8A includes the faces of persons A, B, and C authenticated as a specific person, the person Only the parts including the faces A, B and C are cut out as moving images corresponding to the faces of the persons A, B and C (see FIG. 8B). In the example shown in FIG. 8, in the original moving image, frames in which the faces of the person A, the person B, and the person C are included temporally overlap. In the original moving image shown in FIG. 8A, the faces of the persons B and C are not authenticated in the period of some frames, but there is a period of frames in which no face is detected or authenticated. Is less than the original video.
At this time, the priority registered in advance is higher in the order of person A, person B and person C, and the size of the extracted part is larger in order of person B, person A and person C and the length of the extracted moving image The length is longer in the order of person A, person B, and person C.

Then, as shown in FIG. 8C, when combining the cut out moving images according to the temporal combination criterion 1, according to the priority registered in advance, in the order of person A, person B, person C, The cut out moving images are combined.
On the other hand, when the cut out moving images are combined according to the temporal combination criterion 2, the cut out moving images are combined in the order of the person B, the person A and the person C according to the size of the cut out portion.
Furthermore, when the cut out moving images are combined according to the temporal combination criterion 3, the cut out moving images are combined in the order of the person A, the person B and the person C according to the length of the cutout time.

Further, as shown in FIG. 8D, when combining the cut out moving images according to the spatial connection criterion 1, the area of the combining position of the person A in the priority 1 according to the priority registered in advance. The person B is joined to the area of the synthesis position of priority 2 and the person C is joined to the area of the synthesis position of priority 3. At this time, since the number of cut-out images is 3 and the person A is the top-priority cut-out target, the layout 3A is selected with reference to the layout table.
On the other hand, when combining the cut out moving images according to the spatial connection standard 2, the person B is a region of the combining position of priority 1 and the person A is a combining position of priority 2 according to the size of the cut out portion. , The person C is combined with the area of the priority 3 composite position. At this time, since the number of cut-out images is 3 and the person B is the cut-out target of top priority, the layout 3B is selected with reference to the layout table.

  Furthermore, when combining the cut out moving images according to the spatial connection criterion 3, according to the length of the cut-out time, the area of the synthesis position of person A is priority 1 and the area of the synthesis position of person B is priority 2 , Person C is joined to the area of the synthesis position of priority 3. At this time, since the number of cut-out images is 3 and the person A is the top-priority cut-out target, the layout 3A is selected with reference to the layout table.

Further, as shown in FIG. 9, in the case of the temporal cutout reference 3, when the original moving image shown in FIG. 9A includes the face in the front, right and left directions of the person who is authenticated as the specific person, Only portions including faces facing front, right and left are respectively cut out as moving images (see FIG. 9B). In the original moving image shown in FIG. 9A, the face of the person is authenticated in the entire frame period, and each frame including the front, right and left faces does not overlap in time. Therefore, the total length of the moving image to be cut out is the same as the length of the original moving image.
At this time, the priority registered in advance is higher in the order of front, right and left, and the size of the cut out part is larger in the order of right, front and left and the length of the cut out is front, right It is assumed to be longer in the left direction.

Then, as shown in FIG. 9C, when combining the cut out moving images according to the temporal connection standard 1, according to the priority registered in advance, the parts are cut out in the order of front, right and left. Moving pictures are combined.
On the other hand, when the cut out moving images are combined according to the temporal combination standard 2, the cut out moving images are combined in the order of rightward, front and left according to the size of the cut out portion.
Furthermore, when the cut out moving images are combined according to the temporal combination criterion 3, the cut out moving images are combined in the order of front, right and left according to the length of the cut out time.

Further, as shown in FIG. 9D, when combining the cut out moving images according to the spatial connection standard 1, according to the priority registered in advance, the area of the synthesis position of the priority 1 in the front, The rightward direction is combined with the area of the synthesis position of priority 2 and the leftward direction is connected with the area of the synthesis position of priority 3. At this time, since the number of cut-out images is 3 and the front is the cut-out target of top priority, the layout 3G is selected with reference to the layout table.
On the other hand, when combining the cut out moving images in accordance with the spatial connection standard 2, the area facing to the right is a combination position of priority 1 and the area from the front is a combination position of priority 2 according to the size of the cut out part. , Left facing is combined with the area of the synthesis position of priority 3. At this time, since the number of cut-out images is 3 and the rightward direction is the cut-out target of top priority, the layout 3H is selected with reference to the layout table.

  Furthermore, when combining the cut out moving images according to the spatial connection criterion 3, according to the length of the cut-out time, the front is an area of the synthesis position of priority 1; the right is an area of the synthesis position of priority 2; Is coupled to the area of the synthesis position of priority 3. At this time, since the number of cut-out images is 3 and the front is the cut-out target of top priority, the layout 3G is selected with reference to the layout table.

[Operation]
Next, the operation will be described.
[Image generation processing]
FIG. 10 is a flow chart for explaining the flow of image generation processing executed by the image processing apparatus 1 of FIG. 1 having the functional configuration of FIG. 2.
The image generation process is started by an operation by the user for starting the image generation process on the input unit 17.

In step S <b> 1, the image selection unit 51 selects moving image data corresponding to the user's instruction input from among the image data stored in the image storage unit 71. The data of the selected moving image is the original moving image.
In step S <b> 2, the clipping reference identification unit 52 identifies the clipping reference when clipping the face portion of the person from the moving image. Here, although the cutout criterion selected last time is specified as the cutout criterion used this time, it may be specified by the operation for selecting the cutout criterion to the input unit 17 by the user.

In step S <b> 3, the cutout processing unit 53 executes cutout processing to cut out the face portion of the person from the moving image in accordance with the cutout reference identified by the cutout reference identification unit 52. The details of the clipping process will be described later.
In step S4, the combination reference specification unit 54 specifies a combination reference when combining moving images cut out by the cut-out processing unit 53. Here, although the previously selected combination criterion is specified as the combination criterion to be used this time, it may be specified by an operation for selection of the combination criterion to the input unit 17 by the user.

In step S5, the combining processing unit 55 determines whether to spatially combine the cut out moving images based on the combining reference identified in step S4.
If the extracted moving images are not spatially combined, it is determined as NO in step S5, and the process proceeds to step S8.
On the other hand, in the case of spatially combining the cut out moving images, YES is determined in step S5, and the process proceeds to step S6.

In step S6, the layout selection unit 56 refers to the layout table on the basis of the number of clipped images clipped out in step 3 and the clipping object of top priority based on the combination standard identified in step S4. Select data
In step S7, the combination processing unit 55 performs a spatial combination process of combining the cut out moving images in accordance with the spatial combination criteria. The details of the spatial coupling process will be described later.

In step S8, the combining processing unit 55 performs temporal combining processing for combining the cut out moving images in accordance with a temporal combining criterion. The details of the temporal connection process will be described later.
After steps S7 and S8, the image generation process ends.

[Cut-out process]
FIG. 11 is a flowchart for explaining the flow of the clipping process of step S3 in FIG.
In step S31, the cutout processing unit 53 searches for an original moving image, and specifies a subject to be cutout.

In step S <b> 32, the clipping processing unit 53 selects a subject with the highest priority as a processing target based on the specified spatial clipping reference or temporal clipping reference.
In step S33, the cutout processing unit 53 spatially or temporally cuts out the selected subject to generate an intermediate moving image. An intermediate moving image is a set of temporarily stored frame data that is not in a specific file format.

In step S34, the clipping processing unit 53 determines whether or not the clipping processing of all the subjects to be clipped is completed.
If the clipping process of all the subjects to be clipped is not completed, it is determined as NO in step S34, and the process proceeds to step S35.

In step S35, the clipping processing unit 53 selects a subject with the next priority as a processing target.
After step S35, the process proceeds to step S33.
On the other hand, when the clipping process of all the subjects to be clipped is completed, YES is determined in step S34, and the process returns to the image generation process of FIG.

[Spatial connection processing]
FIG. 12 is a flow chart for explaining the flow of the spatial coupling process of step S7 in FIG.
In step S51, the combining processing unit 55 combines the intermediate moving images (intermediate moving images whose sizes and aspect ratios have been adjusted) corresponding to the priorities of the combining positions of the selected layout according to the layout.

In step S52, the combining processing unit 55 files the moving image obtained by combining the intermediate moving image. Note that, in the present embodiment, as a file format at the time of converting a moving image into a file, for example, MPEG (Moving Picture Experts Group) 4, H.264, or the like can be used. H.264 or H. It can conform to H.265 and the like.
After step S52, the process returns to the image generation process.

Temporal join processing
FIG. 13 is a flow chart for explaining the flow of the temporal connection process of step S8 in FIG.
In step S71, the combining processing unit 55 connects and combines intermediate moving images in the order of the priorities of the identified temporal combination criteria.

In step S72, the combining processing unit 55 files the moving image obtained by combining the intermediate moving image.
After step S72, the process returns to the image generation process.

As a result of such processing, in the moving image (original moving image) selected by the user, even in the case where a plurality of persons are included as subjects, a plurality of cutouts according to the spatial cutout criterion or the temporal cutout criterion The subject is cut out. Then, the extracted moving images are combined according to the priority according to the spatial combining criterion or the temporal combining criterion, and a new moving image is generated.
Therefore, even when a plurality of persons are separated in the screen, each person can be appropriately cut out from the moving image to generate a new moving image.
Therefore, it becomes possible to generate an effective image regardless of the positional relationship of a plurality of subjects present in the screen.

Also, segmentation objects segmented according to spatial segmentation criteria can be combined according to temporal coupling criteria to generate a new moving image.
Therefore, it becomes possible to display the subject appearing at a distant position in the moving image according to a preset reference and display it continuously in time.
Also, segmentation objects segmented according to the temporal segmentation criteria can be combined according to the spatial coupling criteria to generate a new moving image.
Therefore, it is possible to display moving images of a plurality of subjects in a spatially arranged state by conforming to a preset reference regardless of whether the objects are simultaneously shown in the moving image or not. Become.

  In the present embodiment, a moving image has been described as an example of the image cutting target, but the image is cut out for continuous still images such as a plurality of continuously shot still images as well as the moving image. You may do it.

Second Embodiment
Next, a second embodiment of the present invention will be described.
The image processing apparatus 1 according to the second embodiment spatially cuts out a subject for a still image, spatially combines the cut out still images, and generates a new still image.
That is, the image processing apparatus 1 according to the second embodiment has the hardware configuration and the functional configuration for executing the image generation processing in common with the first embodiment, and the contents of the image generation processing are mainly performed in the first embodiment. It is different from the form.

FIG. 14 is a flowchart illustrating the flow of the image generation process performed by the image processing apparatus 1 according to the second embodiment.
The image generation process is started by an operation by the user for starting the image generation process on the input unit 17.

In step S101, the image selection unit 51 selects data of a still image corresponding to a user's instruction input from among the image data stored in the image storage unit 71. Hereinafter, data of the selected still image will be referred to as "original still image".
In step S <b> 102, the cutout reference specification unit 52 specifies a cutout reference (spatial cutout reference) when cutting out the face portion of the person from the still image. In the present embodiment, among the spatial cutout criteria 1 to 4 in the first embodiment, any one of the spatial cutout criteria 1 to 3 is specified. Here, as in the first embodiment, the previously selected clipping criterion is specified as the clipping criterion to be used this time, but it may be specified by an operation for selecting the clipping criterion to the input unit 17 by the user.

In step S103, the clipping processing unit 53 executes clipping processing to clip out the face portion of the person from the still image in accordance with the clipping reference identified by the clipping reference identifying unit 52. The details of the clipping process will be described later.
In step S104, the combination reference specification unit 54 specifies a combination reference (spatial connection reference) when combining the still images cut out by the cut-out processing unit 53. In the present embodiment, one of the spatial coupling criteria 1 and 2 is specified among the spatial coupling criteria 1 to 3 in the first embodiment. Here, as in the first embodiment, the combination criterion selected last time is specified as the combination criterion to be used this time, but may be specified by an operation for selecting the cutout criterion to the input unit 17 by the user.

In step S105, the layout selection unit 56 refers to the layout table on the basis of the combination criteria specified in step S104, based on the number of cut-out images cut out in the cut-out process in step S103 and the highest priority cut-out target. Select data
In step S106, the combining processing unit 55 performs a combining process of combining the cut out still images in accordance with the spatial combining criteria. The details of the combining process will be described later.
After step S106, the image generation process ends.

[Cut-out process]
FIG. 15 is a flowchart for explaining the flow of the clipping process of step S103 in FIG.
In step S121, the clipping processing unit 53 searches for the original still image, and specifies the subject to be clipped.

In step S122, the clipping processing unit 53 selects a subject with the highest priority as a processing target based on the specified spatial clipping standard.
In step S123, the cutout processing unit 53 spatially cuts out the selected subject and generates an intermediate still image. The intermediate still image is a set of temporarily stored pixel data that is not in a specific file format.

In step S124, the clipping processing unit 53 determines whether or not the clipping processing of all the subjects to be clipped is completed.
If the clipping process of all the subjects to be clipped is not completed, it is determined as NO in step S124, and the process proceeds to step S125.

In step S125, the cutout processing unit 53 selects a subject of the next priority as a processing target.
After step S125, the process proceeds to step S123.
On the other hand, when the clipping process of all the subjects to be clipped is completed, YES is determined in step S124, and the process returns to the image generation process of FIG.

[Join process]
FIG. 16 is a flowchart for explaining the flow of the combining process of step S106 in FIG.
In step S141, the combining processing unit 55 combines the intermediate still images (intermediate still images whose sizes and aspect ratios have been adjusted) corresponding to the priorities of the combining positions of the selected layout according to the layout.

In step S142, the combining processing unit 55 files the still image obtained by combining the intermediate still images. In the present embodiment, as a file format at the time of converting a still image into a file, for example, it is possible to conform to JPEG (Joint Photographic Experts Group), GIF (Graphic Interchange Format) or the like.
After step S142, the process returns to the image generation process of FIG.

As a result of such processing, even if a plurality of persons are included as subjects in the still image selected by the user, a plurality of clipping targets are clipped according to the spatial clipping standard. Then, the clipped still images are combined according to the priority according to the spatial combining criteria to generate a new still image.
Therefore, even when a plurality of persons are separated in the screen, each person can be appropriately cut out from the still image, and a new still image can be generated.
Therefore, it becomes possible to generate an effective image regardless of the positional relationship of a plurality of subjects present in the screen.

[Modification 1]
In the above-mentioned embodiment, when subjecting subjects, such as a person's face, etc. to be spatially cut out, the case where each subject was cut out individually was mentioned as an example and explained. On the other hand, when a subject such as the face of a person is spatially cut out, a plurality of close subjects in the screen may be cut out together in one region.
FIG. 17 is a schematic view showing a concept of collectively cutting out a plurality of subjects.
In FIG. 17, the moving image frame or still image includes the faces of persons A to C who have been authenticated as a specific person, and the distances between the faces of persons A and B within the set threshold value. positioned.
At this time, the faces of the person A and the person B can be collectively cut out spatially in one region.
As a result, since a subject with high spatial relevance is cut out without being separated, it is possible to make the relation between subjects more appropriate according to the situation of the subject when generating a new image. it can.

The image processing apparatus 1 configured as described above includes a cutout processing unit 53 and a combination processing unit 55.
The cutout processing unit 53 cuts out a plurality of moving images from one moving image based on spatial and / or temporal cutout criteria.
The combination processing unit 55 combines the extracted moving images temporally and / or spatially based on the spatial and / or temporal connection criteria to generate one new moving image.
Thereby, one moving image can be generated according to the spatial and / or temporal connection criterion from a plurality of moving images segmented according to the spatial and / or temporal cutout criterion.
Therefore, even when a plurality of subjects are separated in the screen, each subject can be appropriately cut out from the moving image, and a new moving image can be generated.
Therefore, it becomes possible to generate an effective image regardless of the positional relationship of a plurality of subjects present in the screen.

Further, the combining processing unit 55 combines a plurality of moving images cut out based on the spatial cut-out reference based on the temporal connection reference.
As a result, it becomes possible to match the subject appearing at a distant position in the moving image with the preset temporal connection reference and to display the subject continuously in time.

Further, the combining processing unit 55 combines a plurality of moving images cut out based on the temporal cut-out reference based on the spatial connection reference.
Thereby, regardless of whether or not the subject is simultaneously captured in the moving image, the moving images of a plurality of subjects are displayed in a spatially arranged state by being adapted to a preset spatial connection reference. It is possible to

Also, the clipping processing unit 53 spatially cuts out a predetermined subject portion included in each screen constituting the moving image as a spatial clipping reference.
As a result, it is possible to appropriately cut out the area of the predetermined subject on each screen of the moving image.

Further, the clipping processing unit 53 cuts out, in time, a frame including a predetermined subject among frames constituting a moving image, as a temporal clipping reference.
As a result, it is possible to appropriately cut out the period of a frame in which a predetermined subject is included in a moving image.

The predetermined subject is the face of a person registered in advance.
As a result, even when a plurality of persons registered in advance are captured separately in the screen, each subject can be appropriately cut out of the moving image.

In addition, the predetermined subject further includes the face of an unspecified person.
Thus, in a moving image in which the face of an unspecified person is taken, a plurality of persons can be appropriately cut out from the moving image.

In addition, the faces of persons not to be extracted by the extraction processing unit 53 are registered in advance.
Thus, it is possible to generate an effective moving image by excluding a specific person from among a plurality of persons appearing in the moving image.

Further, the combining processing unit 55 spatially divides the screen into regions based on the priorities and assigns the regions to the priorities, as the spatial combining reference, based on the priorities.
In this way, it is possible to generate an effective moving image in correspondence with the priority of the moving image of the subject that has been cut out and the spatial priority.

The image processing apparatus 1 further includes a layout selection unit 56.
The layout selecting unit 56 selects a layout in which the number, the size, or the positional relationship in which a plurality of images are combined corresponding to the number of the moving images cut out or the subject included in the images is defined.
The combining processing unit 55 combines the plurality of cut out images into the selected layout to generate one new image.
Thereby, the extracted moving image can be automatically combined into an appropriate layout to generate a new moving image.

Further, the combining processing unit 55 temporally connects each of the cut out moving images based on the priority as a temporal combining reference.
Thus, moving images of a plurality of subjects can be connected to generate a new moving image in the order corresponding to the priority of the cut out moving images.

Further, the priority is a priority registered in advance corresponding to a subject included in the image, a spatial size of the clipped subject portion, or a temporal length of the clipped moving image.
Thereby, an appropriate priority can be set according to the attribute of the extracted moving image.

The image processing apparatus 1 further includes a cutout processing unit 53, a layout selection unit 56, and a combination processing unit 55.
The cutout processing unit 53 cuts out a plurality of images from one image based on spatial cutout criteria.
The layout selection unit 56 selects a layout in which the number, size, or positional relationship of combining a plurality of images is defined based on the features of the extracted images.
The combining processing unit 55 combines the plurality of cut out images into the selected layout to generate one new image.
Thus, it is possible to generate one still image or moving image according to the spatial connection criterion from a plurality of images extracted according to the spatial extraction criterion.
Therefore, even when a plurality of subjects are separated in the screen, each subject can be appropriately cut out from the still image, and a new still image or moving image can be generated.
Therefore, it becomes possible to generate an effective image regardless of the positional relationship of a plurality of subjects present in the screen.

  The present invention is not limited to the above-described embodiment, and modifications, improvements, and the like in the range in which the object of the present invention can be achieved are included in the present invention.

  In the above-mentioned embodiment, although the face of a person was explained as the object of clipping, it is not restricted to this. That is, it may be another part of the person, and it may be possible to authenticate or detect other images, objects or parts thereof from the image.

  In the above embodiment, the size and aspect ratio of the new still image or moving image generated by combining the images extracted from the original still image or moving image are the same as or different from the original still image or moving image It can be

  Further, in the above-described embodiment, it has been described that the layout defines the background, the number of images to be combined with the background, and the size and positional relationship, but the present invention is not limited to this. That is, in the layout, a part of the background, the number of images to be synthesized, the size, and the positional relationship can be defined, or other elements can be defined.

  In the above-described embodiment, the image generation processing is performed on the data of the image stored in the image storage unit 71. However, the present invention is not limited to this. For example, the image generation processing may be performed on a still image or a moving image captured by the imaging unit 16.

  Further, in the above-described embodiment, in the spatial cutout criterion 4, the front / right / left direction of one face detected in a moving image or one face authenticated as a specific person is separately extracted. It is not restricted to this. For example, a smile, an angry face, a cry face, or the like of one face detected in a moving image or one face recognized as a specific person may be separately extracted.

  Further, in the above-described embodiment, various types of layout can be adopted as long as they can combine cut-out images. For example, the type of layout can be a form of PinP (Picture In Picture) in which images are displayed by overlapping smaller screens in the screen.

Moreover, in the above-mentioned embodiment, although the image processing apparatus 1 to which this invention is applied was demonstrated as an example of the digital camera, it is not specifically limited to this.
For example, the present invention can be applied to electronic devices in general having an image generation processing function. Specifically, the present invention is applicable to a laptop personal computer, a printer, a television receiver, a video camera, a portable navigation device, a portable telephone, a smart phone, a portable game machine, and the like.

The series of processes described above can be performed by hardware or software.
In other words, the functional configuration of FIG. 2 is merely illustrative and not particularly limited. That is, it is sufficient if the image processing apparatus 1 has a function capable of executing the above-described series of processes as a whole, and what functional block is used to realize this function is not particularly limited to the example of FIG. .
Further, one functional block may be configured by hardware alone, may be configured by software alone, or may be configured by a combination of them.

When the series of processes are executed by software, a program that configures the software is installed on a computer or the like from a network or a recording medium.
The computer may be a computer incorporated in dedicated hardware. The computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

  The recording medium including such a program is not only configured by the removable medium 31 of FIG. 1 distributed separately from the apparatus main body to provide the program to the user, but also the user in a state incorporated in advance in the apparatus main body The recording medium etc. provided to The removable medium 31 is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, or a magneto-optical disk. The optical disc is composed of, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versatile Disk), a Blu-ray (registered trademark) Disc (Blu-ray Disc), or the like. The magneto-optical disk is configured by an MD (Mini-Disk) or the like. The recording medium provided to the user in a state of being incorporated in the apparatus main body is, for example, the ROM 12 of FIG. 1 in which the program is recorded, the hard disk included in the storage unit 19 of FIG.

  In the present specification, in the step of describing the program to be recorded on the recording medium, the processing performed chronologically along the order is, of course, parallel or individually not necessarily necessarily chronologically processing. It also includes the processing to be performed.

  While some embodiments of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. The present invention can take other various embodiments, and furthermore, various changes such as omissions and substitutions can be made without departing from the scope of the present invention. These embodiments and modifications thereof are included in the scope and the gist of the invention described in the present specification, etc., and are included in the invention described in the claims and the equivalent scope thereof.

The invention described in the claims at the beginning of the application of the present application is appended below.
[Supplementary Note 1]
Clipping means for clipping a plurality of moving images from one moving image based on spatial and / or temporal clipping criteria;
Generation means for temporally or spatially combining the plurality of extracted moving images based on spatial and / or temporal connection criteria to generate one new moving image;
An image processing apparatus comprising:
[Supplementary Note 2]
The generation means combines a plurality of moving images cut out by the spatial cut-out criterion by a temporal combination reference,
The image processing apparatus according to claim 1, characterized in that:
[Supplementary Note 3]
The generation means combines a plurality of moving images cut out based on the temporal cut-out reference based on a spatial connection reference,
The image processing apparatus according to claim 1, characterized in that:
[Supplementary Note 4]
The clipping unit spatially clips, as the spatial clipping reference, a predetermined subject portion included in each screen constituting a moving image.
The image processing apparatus according to any one of appendices 1 to 3, characterized in that
[Supplementary Note 5]
The clipping unit temporally clips out a frame including a predetermined subject among frames constituting a moving image as the temporal clipping reference.
The image processing apparatus according to any one of appendices 1 to 4, characterized in that
[Supplementary Note 6]
The predetermined subject is a face of a person registered in advance.
The image processing apparatus according to any one of appendices 4 or 5, characterized in that
[Supplementary Note 7]
The predetermined subject further includes the face of an unspecified person,
The image processing apparatus according to claim 6, characterized in that
[Supplementary Note 8]
A face of a person not to be cut out by the cutting out means is registered in advance.
The image processing apparatus according to claim 6, characterized in that
[Supplementary Note 9]
The generation means spatially divides each of the cut out moving images on the basis of priority, and assigns each of the regions to which priority has been given, as the spatial connection criterion.
The image processing apparatus according to any one of appendices 1 to 8, characterized in that
[Supplementary Note 10]
The image processing apparatus further comprises selection means for selecting a layout in which the number, size, or positional relationship of combining a plurality of images corresponding to the number of cut out moving images or a subject included in the images is selected,
The generation unit combines a plurality of extracted images with the selected layout to generate a new image.
Appendix 9. The image processing apparatus according to appendix 9.
[Supplementary Note 11]
The generation unit temporally connects each of the cut out moving images based on priority as the temporal connection criterion.
The image processing apparatus according to appendix 4, characterized in that
[Supplementary Note 12]
The priority is a pre-registered priority corresponding to a subject included in an image, a spatial size of a clipped subject portion, or a temporal length of a clipped moving image.
The image processing apparatus according to any one of appendices 9 to 11, characterized in that
[Supplementary Note 13]
Clipping means for clipping a plurality of images from one image based on a predetermined clipping criterion;
Selection means for selecting a layout in which the number, size or positional relationship of combining a plurality of images is defined based on the features of the extracted images;
Generation means for combining the plurality of extracted images with the selected layout to generate a new image;
An image processing apparatus comprising:
[Supplementary Note 14]
The selection means selects the number of images or the layout corresponding to a subject included in an image as the feature of the extracted image.
The image processing device according to appendix 13, characterized in that
[Supplementary Note 15]
The cutting out unit cuts out a predetermined subject portion from an image including the predetermined subject.
The image processing apparatus according to any one of appendices 13 or 14, characterized in that
[Supplementary Note 16]
A clipping process of clipping a plurality of moving images from one moving image based on spatial and / or temporal clipping criteria;
Generation processing for combining the extracted moving images temporally or spatially based on spatial and / or temporal connection criteria to generate one new moving image;
An image processing method comprising:
[Supplementary Note 17]
On the computer
A clipping function of clipping a plurality of moving images from one moving image based on spatial and / or temporal clipping criteria;
A generation function of temporally or spatially combining the plurality of extracted moving images based on spatial and / or temporal connection criteria to generate one new moving image;
A program that is characterized by realizing
[Supplementary Note 18]
A clipping process of clipping a plurality of images from one image based on a predetermined clipping criterion;
A selection process of selecting a layout in which the number, size, or positional relationship of combining a plurality of images is defined based on the features of the extracted images;
Generation processing of combining the plurality of extracted images with the selected layout to generate a new image;
An image processing method comprising:
[Supplementary Note 19]
On the computer
A clipping function for clipping a plurality of images from one image based on a predetermined clipping criterion;
A selection function of selecting a layout in which the number, size, or positional relationship of combining a plurality of images is defined based on the features of the extracted images;
A generation function of combining the plurality of extracted images into the selected layout to generate a new image;
A program that is characterized by realizing

DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus, 11 ... CPU, 12 ... ROM, 13 ... RAM, 14 ... Bus, 15 ... Input-output interface, 16 ... Imaging part, 17 ... An input unit 18, an output unit 19, a storage unit 20, a communication unit 21, a drive 31, a removable medium 51, an image selection unit 52, and the like. Segmentation reference specification unit 53: Segmentation processing unit 54: Joint reference identification unit 55: Joint processing unit 56: Layout selection unit 71: Image storage unit 72 Layout storage unit, 73 ... generated image storage unit, 74 ... face information storage unit

Claims (8)

By cutting out a frame period in which the first subject or the second subject is present from one moving image in which a first subject and a second subject existing in a separated frame period are photographed, the video Cutting means for cutting out an image,
The moving image corresponding to the first object cut out by the cutting out means is combined with the moving image corresponding to the second object according to the ranking of the objects based on the temporal combination criteria Means for generating one new moving image,
An image processing apparatus comprising: The clipping unit is a clipping target when the first subject or the second subject is detected or authenticated for a predetermined threshold time or more.
The image processing apparatus according to claim 1, The temporal connection criterion may be a priority pre-registered in the subject, a spatial size of the extracted subject portion, or a temporal length of the extracted moving image.
The image processing apparatus according to claim 1, wherein the image processing apparatus comprises: When the first subject and the second subject present in individual frames making up a moving image are located at a distance equal to or greater than a predetermined threshold value in the frame, the clipping means determines the first condition. Cut out the subject and the second subject independently and spatially
The image processing apparatus according to any one of claims 1 to 3, characterized in that: When the first object and the second object present in individual frames making up a moving image are located at a distance within a frame that is less than a predetermined threshold value, the cutting-out means may perform the first operation. Collectively and spatially cutting out the subject and the second subject,
The image processing apparatus according to claim 4, characterized in that: By cutting out a frame period in which the first subject or the second subject is present from one moving image in which a first subject and a second subject existing in a separated frame period are photographed, the video a switching origin sense to cut out the image,
The moving image corresponding to the first object cut out by the cutting process and the moving image corresponding to the second object are combined according to the ranking of the objects based on the temporal combination criteria Generation processing to generate one new moving image,
An image processing method comprising: On the computer
By cutting out a frame period in which the first subject or the second subject is present from one moving image in which a first subject and a second subject existing in a separated frame period are photographed, the video With a clipping function that cuts out an image,
The moving image corresponding to the first object cut out by the cutting out function is combined with the moving image corresponding to the second object according to the ranking of the objects based on the temporal combination criteria Generation function to generate one new moving image,
A program that is characterized by realizing Clipping means for clipping a moving image of each predetermined subject by cutting out a frame period in which a plurality of predetermined subjects are present from one moving image in which at least one subject is present in separated frame periods;
A new moving image is generated by combining the moving images cut out by the cutting unit according to the predetermined objects in accordance with the ranking of the predetermined objects based on the temporal combination criteria. Means for generating
An image processing apparatus comprising:

Images