JP2021027486A - Image processing device, control method thereof, and program - Google Patents

Abstract

To provide an image processing device capable of suppressing deterioration of the image quality of a composite image when synthesizing a plurality of images.SOLUTION: In an image processing device 300, a plurality of target images (frames) 401 to 404 that are continuous in time series are acquired, main subject images 406 to 409 including a person 405 are respectively separated from target image 401 to 404, and the remaining images are cut out as background images 410 to 413, a background image candidate that is one background image is extracted from each of the background images 410 to 413, and the background image candidates and main subject images 406 to 409 are combined to generate a composite image 414.SELECTED DRAWING: Figure 2

Description

The present invention relates to an image processing device that synthesizes a plurality of images, a control method and a program thereof.

In recent years, there has been known a technique of synthesizing the movement of a subject into one image based on a plurality of images acquired from a moving image or a plurality of images acquired by continuous shooting. For example, when a moving subject is continuously photographed in an imaging environment with a uniform background to obtain a plurality of images, and then these images are combined, a composite image in which only the moving subject is displayed is obtained. Be done. In this composite image, the movement of the subject can be confirmed in time series with one image, so that the movement of the subject can be expressed like a moving image even though it is a still image. Usually, such a composite image requires a great deal of time and effort for an operator to manually create such a composite image using a drawing tool on a PC or the like. Therefore, some techniques for automatically generating such a composite image have been proposed.

By the way, when acquiring a plurality of images in order to obtain such a composite image, if there is an object whose background is not uniform and has a movement different from that of the subject, another object is superimposed on the composite image and noise is generated. Is displayed. For example, as shown in FIG. 7, when each image of the plurality of image groups 101 obtained by continuous shooting includes a bird 103 which is an object different from the person 102 who is the subject, the composite image 104 shows the images of each image. The noise 105 superimposed while the bird 103 is moving is displayed.

On the other hand, a technique for deleting a moving object that causes noise has also been proposed. For example, in the technique of Patent Document 1, in a plurality of images obtained by continuous shooting, a moving object is deleted by deleting an object other than an object existing at the same position in at least two images. An application that deletes all moving objects from a video is also known (see, for example, ViRD Co., Ltd., "VANISH360", http://vird.co.jp/product/vanish360/).

Further, for example, the configuration shown in FIG. 8 is known as a configuration for generating a composite image in which a subject having a movement different from the main subject (hereinafter referred to as “moving object”) is deleted.

In FIG. 8, in this configuration, still image data or moving image data is recorded on the recording medium 201, and the drive unit 202 acquires the still image data or moving image data from the recording medium 201. The decoding unit 203 decodes the acquired data and outputs the decoded still image data or moving image data to the memory 204. When generating the composite image, the user instructs the system controller 205 to read the plurality of composited images. In response to the instruction, the frame reading unit 206 reads out a plurality of frames (a plurality of images that are continuous in time series) corresponding to the instruction in the still image data or the moving image data from the memory 204. After that, the main subject group detection unit 207 detects the main subject from each frame, and the moving object deleting unit 208 deletes the moving object other than the main subject from each frame. After that, the video compositing processing unit 209 synthesizes each frame from which the moving object has been deleted, the image processing unit 210 adjusts the angle of view and color conversion of each combined frame to generate a composite image, and the display device 211 synthesizes them. Display the image.

Japanese Unexamined Patent Publication No. 2000-13681

However, in the configuration of FIG. 8, since each frame in which the moving body is deleted is combined, if each frame contains a region having a large difference from each other, the regions are overlapped. As a result, there is a problem that the image quality of the composite image is deteriorated. Further, even in the case of synthesizing the frames without deleting the moving body in the configuration of FIG. 8, the same problem occurs because the regions having a large difference from each other are overlapped.

An object of the present invention is to provide an image processing device capable of suppressing deterioration of the image quality of a composite image when synthesizing a plurality of images, a control method and a program thereof.

In order to achieve the above object, the image processing apparatus of the present invention separates a subject from each of an image acquisition means for acquiring a plurality of images that are continuous in time series and the plurality of acquired images to obtain a background. An image saving means that acquires and saves the plurality of subjects and the plurality of backgrounds acquired from the plurality of images, and the plurality of saved backgrounds are compared with each other, and the difference from the other backgrounds is the smallest. It is characterized by including a background extraction means for extracting the background of the above, and a composite image generation means for generating a composite image by synthesizing the plurality of subjects on the extracted one background.

According to the present invention, when a plurality of images are combined, deterioration of the image quality of the combined image can be suppressed.

It is a block diagram which shows schematic structure of the image processing apparatus which concerns on 1st Embodiment. It is a flowchart which shows the image composition processing in 1st Embodiment. It is a process drawing for demonstrating the image composition processing in 1st Embodiment. It is a figure for demonstrating the case where a moving body which is highly related to a main subject is not included in a composite image. It is a flowchart which shows the image composition processing in 2nd Embodiment. It is a figure for demonstrating the image composition processing in 2nd Embodiment. It is a figure for demonstrating the process in which noise is generated in the conventional image composition processing. It is a block diagram which shows schematic structure which executes the conventional image composition processing.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in the embodiments.

First, the first embodiment will be described. FIG. 1 is a block diagram schematically showing a configuration of an image processing apparatus according to the present embodiment. In FIG. 1, for example, an image processing device 300, which is an image pickup device such as a camera, includes a drive unit 302 to which a recording medium 301 is connected or mounted, a decoding unit 303, a memory 304, and a system controller 305. Further, the image processing device 300 includes a frame reading unit 306 (image acquisition means, still image acquisition means, frame acquisition means), a main subject group detection unit 307 (subject detection means, main subject detection means), and a background separation unit 308. (Image storage means, subject range specifying means). Further, the image processing device 300 includes a separation background storage unit 309 (background storage means), a separation subject storage unit 310 (subject storage means), a background comparison extraction unit 311 (background extraction means), and a separation background reading unit 312. To be equipped. Further, the image processing device 300 includes a video compositing processing unit 313 (compositing image generation means), a separated subject reading unit 314, an image processing unit 315, and a display device 316.

In the image processing device 300, the recording medium 301 records still image data or moving image data, and the drive unit 302 acquires still image data or moving image data from the recording medium 301. The decoding unit 303 decodes the acquired data and outputs the decoded still image data or moving image data to the memory 304. The system controller 305 controls the operation of each component of the image processing device 300. The frame reading unit 306 reads a plurality of frames (a plurality of images that are continuous in time series) in the still image data or the moving image data from the memory 304. The main subject group detection unit 307 detects the main subject from each frame, and the background separation unit 308 cuts out an image in a range including the main subject detected from each frame as the main subject image and separates the main subject image from each frame. And cut out the remaining image as a background image. The separated background storage unit 309 stores the background image cut out from each frame in the memory 304, and the separated subject storage unit 310 stores the main subject image cut out from each frame in the memory 304.

Further, the separated background reading unit 312 reads each background image stored in the memory 304 from the memory 304. The background comparison extraction unit 311 extracts a background image candidate (one background) to be combined from each background image. The separated subject reading unit 314 reads each main subject image from the memory 304, and the video compositing processing unit 313 synthesizes the background image candidate extracted by the background comparison extraction unit 311 and each read main subject image. The image processing unit 315 performs angle-of-view adjustment and color conversion of each of the combined main subject images and background image candidates to generate a composite image, and the display device 316 displays the composite image.

Next, an image composition process as a control method of the image processing device according to the present embodiment will be described. FIG. 2 is a flowchart showing an image composition process according to the present embodiment, and FIG. 3 is a process diagram for explaining the image composition process according to the present embodiment.

First, the system controller 305 receives an instruction from the user regarding the selection of the main subject and a plurality of consecutive images to be combined in time series (step S501). In the present embodiment, the image to be combined is hereinafter referred to as a target image. In the present embodiment, as each target image, target images (frames) 401 to 404 including a person riding a bicycle and a house or forest reflected as a background are selected. In addition, a person 405 riding a bicycle is selected as the main subject.

Next, the frame reading unit 306 reads the target images 401 to 404 from the memory 304 in order (step S502). After that, the main subject group detection unit 307 detects the person 405 from each of the read target images 401 to 404, and detects the rectangular region including the person 405 in each target image 401 to 404 as the main subject image. Specifically, the main subject images 406 to 409 including the person 405 are defined in the target images 401 to 404, and the area information of the main subject images 406 to 409 is stored in the memory 304 (FIG. 3A). For example, the area information (x1, y1, Lx1, Ly1) of the main subject image 406 in the target image 401 is stored in the memory 304. Here, (x1, y1) indicates the coordinates of the upper left of the main subject image 406 in the figure, (Lx1) indicates the horizontal length of the main subject image 406 in the figure, and (Ly1) indicates the horizontal length of the main subject image 406 in the figure. Indicates the vertical length of. Next, the background separation unit 308 cuts out the main subject images 406 to 409 from the target images 401 to 404, separates the main subject images 406 to 409 from the target images 401 to 404, and cuts out the remaining images as the background image ( Step S503). The cut-out background images 410 to 413 are stored in the memory 304 by the separated background storage unit 309, and the cut-out main subject images 406 to 409 are stored in the memory 304 by the separated subject storage unit 310. At this time, each of the target images is associated with the main subject image cut out from the target image, the area information of the main subject image, and the background image cut out from the target image. In the present embodiment, the main subject image 406 is cut out from the target image 401 to generate the background image 410, and the main subject image 407 is cut out from the target image 402 to generate the background image 411. Further, the main subject image 408 is cut out from the target image 403 to generate the background image 412, and the main subject image 409 is cut out from the target image 404 to generate the background image 413 (FIG. 3 (B)).

Next, the system controller 305 determines whether or not all the target images have been read from the memory 304 (step S504). Here, if all the target images have not been read, the process returns to step S502, and if all the target images have been read, the process proceeds to step S505.

In step S505, the separation background reading unit 312 reads each background image 410 to 413 stored in the memory 304, and the background comparison extraction unit 311 extracts background image candidates from each background image 410 to 413. At this time, the background comparison extraction unit 311 compares one of the background images with each of the background images located before and after the one background image in chronological order (adjacent in chronological order) to determine the degree of matching. Calculate the comparison result value shown. Further, the background comparison extraction unit 311 compares the comparison result values of the background images, and extracts one background image having the highest comparison result value as a background image candidate.

In the calculation of the comparison result value in step S505, the higher the degree of matching (the smaller the difference), the higher the comparison result value is calculated by the commonly used image matching technique. In step S505, it suffices if the degree of agreement between the background images can be obtained by some index, and thus other generally known techniques may be used. Here, when each background image (referred to as "front background image" and "back background image", respectively) located before and after in time series exists in one background image, one background image and one background image are used. The comparison result value of is obtained, and further, the comparison result value of one background image and the back background image is obtained. Then, the average of the two comparison result values is used as the evaluation value and associated with one background image. When only the back background image exists in one background image, the comparison result value between one background image and the back background image is obtained, and this comparison result value is used as an evaluation value and associated with one background image. Further, when only the foreground image exists in one background image, the comparison result value between one background image and the foreground image is obtained, and this comparison result value is used as an evaluation value and associated with one background image.

For example, in the present embodiment, as shown in FIG. 3C, the comparison result value C1 is calculated as the degree of agreement between the background image 410 and the background image 411, and the comparison result is the degree of agreement between the background image 411 and the background image 412. The value C2 is calculated. Further, the comparison result value C3 is calculated as the degree of agreement between the background image 412 and the background image 413. Then, the background image 411 is associated with the average of the comparison result value C1 and the comparison result value C2 as the evaluation value T2, and the background image 412 is associated with the average of the comparison result value C2 and the comparison result value C3 as the evaluation value T3. Be done. Further, the comparison result value C1 is associated with the background image 410 as the evaluation value T1, and the comparison result value C3 is associated with the background image 413 as the evaluation value T4. In step S505, for example, when the evaluation value T1 is the highest, the background comparison extraction unit 311 extracts the background image 410 as a background image candidate and stores it in the memory 304.

Next, the separated subject reading unit 314 reads the main subject images 406 to 409 from the memory 304, and the video compositing processing unit 313 reads the background image candidate (background image 410) extracted by the background comparison extraction unit 311 and the main subject images 406 to 406. 409 is synthesized. The combination of the background image 410 and the main subject images 406 to 409 is not performed based on the average value or weighting of each image, but is simply performed by overwriting the main subject images 406 to 409 on the background image 410. After that, the image processing unit 315 adjusts the angle of view and color conversion of the combined main subject images 406 to 409 and the background image 410 to generate a composite image 414 (see FIG. 3D) (step S506). End the process.

By the way, in the present embodiment, as shown in FIG. 3A, the target images 401 to 404 include the bird 415 as a moving body different from the person 405. Here, when the target images 401 to 404 are simply combined as in the conventional composite image generation process, the moving birds 415 in each target image 401 to 404 are superimposed and displayed, and noise over a predetermined range (for example, The noise 105) in FIG. 7 may be displayed. On the other hand, according to the process of FIG. 2, background images 410 to 413 are generated from the target images 401 to 404, and the main subject images 406 to 409 are overwritten only on one of the background images 410, so that the bird 415 Not superimposed and displayed. As a result, in the composite image 414, only one stationary bird 415 is included (see FIG. 3D), and the bird 415 is not displayed like noise. Further, even if the target images 401 to 404 include regions having a large difference from each other, since the target images 401 to 404 are not combined, the regions having a large difference from each other are not overlapped. As a result, deterioration of the image quality of the composite image 414 can be suppressed when the plurality of target images 401 to 404 are combined.

In step S506 of the process of FIG. 2, the main subject images 406 to 409 were overwritten on the background image 410 as the background image candidate. However, first, the background separation unit 308 identifies and removes all the images in the region corresponding to the main subject images 406 to 409 from the background image 410, thereby removing the background image candidate 416 (see FIG. 3D) (others). Background) may be generated. In this case, the background image candidate 416 (other background) is temporarily stored in the memory 304 by the separated background storage unit 309, and then the background image candidate 416 read from the memory 304 is overwritten with the main subject images 406 to 409. To.

Next, a second embodiment of the present invention will be described. Since the configuration and operation of the second embodiment are basically the same as those of the first embodiment described above, the description of the overlapping configuration and operation is omitted, and the different configurations and operations are described below. Give an explanation.

In the first embodiment, a rectangular region including the main subject is cut out from each target image as the main subject image, but in this case, a moving object highly related to the main subject may not be included in the composite image. For example, as shown in FIG. 4A, when target images (frames) 603 to 605 including a person 602 throwing the ball 601 are selected as each target image, the person 602 is selected as the main subject. This case will be described.

At this time, the ball 601 corresponds to a moving body that is highly related to the person 602. Further, the main subject images 606 to 608 are cut out from the target images 603 to 605, but since the ball 601 is separated from the person 602 in the target image 605, the main subject image 608 cut out from the target image 605 includes the ball 601. I can't. Here, for example, when the target image 603 from which the main subject image 606 is cut out is extracted as a background image candidate, the target image 603 does not include the ball 601 separated from the person 602. On the other hand, since the main subject image 608 does not include the ball 601 separated from the person 602, the ball that should exist in the region 610 when the background image candidate and the main subject images 606 to 608 are combined to generate the composite image 609. 601 is not included in the composite image 609.

In the second embodiment, correspondingly, the image processing device 300 includes a main subject-related determination unit 320 (related subject detection means, motion detection means, relevance determination means). The main subject-related determination unit 320 reads out a plurality of target images from the memory 304 and detects a moving object from each target image. Further, in each target image, the main subject-related determination unit 320 does not include the detected moving object in the main subject image, and the distance between the main subject image and the rectangular region (moving object image) including the detected moving object is a predetermined distance. If it is shorter than, the moving object is determined to be a subject related to the main subject. In the present embodiment, a moving object determined to be a subject related to the main subject will be hereinafter referred to as a related subject. In addition, the main subject group detection unit 307 collectively generates a main subject group by combining the main subject detected from each target image and each moving object determined to be a related subject in each target image. After that, the background separation unit 308 cuts out an image corresponding to the main subject group detected from each target image as a main subject group image, and cuts out an image other than the main subject group image from each target image as a background image. At this time, the separated subject storage unit 310 stores the main subject group image cut out from each target image in the memory 304. That is, in the first embodiment, only the main subject is cut out from each target image, but in the second embodiment, not only the main subject but also the related subject is cut out from each target image.

Next, an image composition process as a control method of the image processing device according to the present embodiment will be described. FIG. 5 is a flowchart showing the image composition process in the present embodiment, and FIG. 6 is a diagram for explaining the image composition process in the present embodiment.

First, the system controller 305 receives an instruction regarding the selection of the main subject and each target image by the user (step S701). In the present embodiment, the target images 603 to 605 as shown in FIG. 4A are selected. Further, the person 602 who throws the ball 601 is selected as the main subject.

Next, the main subject-related determination unit 320 reads the target images 603 to 605 from the memory 304, and detects a moving object from the target images 603 to 605 (step S702). As a moving body detection method, a generally known moving body detection technique (for example, a method of detecting a moving body by a motion vector of an object between images) is used. After that, in each target image 603 to 605, the main subject-related determination unit 320 does not include the detected moving object in the main subject image, and the distance between the main subject image and the rectangular region (moving object image) including the detected moving object. Is shorter than a predetermined distance, the moving object is determined to be a related subject. For example, as shown in FIG. 6, in the target image 603, the rectangular region including the ball 601 is detected as the moving object image 611, and the distance between the main subject image 612 including the person 602 and the moving object image 611 is shorter than a predetermined distance. , Ball 601 is determined to be a related subject. In the target image 604, a rectangular region including the ball 601 is detected as a moving object image 613, and when the distance between the main subject image 614 including the person 602 and the moving object image 613 is shorter than a predetermined distance, the ball 601 is determined to be a related subject. To. In the target image 605, a rectangular region including the ball 601 is detected as a moving object image 615, and when the distance between the main subject image 616 including the person 602 and the moving object image 615 is shorter than a predetermined distance, the ball 601 is determined to be a related subject. To.

Whether or not the ball 601 is a related subject is determined independently for each of the target images 603 to 605, but the ball is based on the total number of moving objects and the continuity of the moving objects in the series of target images 603 to 605. It may be determined whether or not 601 is a related subject. In this case, the main subject-related determination unit 320 (moving object quantity determining means, moving object continuous determining means) determines that the distance between the main subject image and the moving object image is shorter than a predetermined distance over each target image 603 to 605. It is determined whether or not the number of moving objects (balls 601) is equal to or greater than a predetermined threshold value. Further, in the main subject-related determination unit 320, the moving objects (balls 601) determined to have a distance between the main subject image and the moving object image shorter than a predetermined distance are in chronological order with each other over each target image 603 to 605. Determine if it is continuous. After that, the main subject-related determination unit 320 determines whether or not each ball 601 detected from each of the target images 603 to 605 is a related subject based on these determination results.

Next, the main subject group detection unit 307 generates a main subject group by collecting the main subject detected from each target image 603 to 605 and each moving object determined to be a related subject in each target image (step S703). After that, the frame reading unit 306 reads the target images 603 to 605 in order from the memory 304 (step S704).

Next, in step S705, the system controller 305 determines whether or not the moving object detected from each of the target images 603 to 605 is determined to be a related subject. When the moving object is determined to be a related subject (YES in step S705), the background separation unit 308 cuts out the main subject image and the moving object image together as the main subject group image from the target image (step S706). Further, the background separation unit 308 separates the main subject image and the moving object image from the target image, and cuts out the remaining image as the background image. On the other hand, when the moving object is not determined to be a related subject (NO in step S705), the background separation unit 308 cuts out only the main subject image from the target image, separates the main subject image from the target image, and leaves the remaining image. Is cut out as a background image (step S707). The cut-out background image is saved in the memory 304 by the separated background storage unit 309, and the cut out main subject group image and the main subject image are saved in the memory 304 by the separated subject storage unit 310.

At this time, each of the target images is associated with the main subject image or the main subject group image cut out from the target image, the area information of the main subject image or the main subject group image, and the background image cut out from the target image. .. In the present embodiment, for example, when the moving object detected from the target image 604 is determined to be a related subject, the main subject image 614 and the moving object image 613 are collectively cut out as the main subject group image from the target image 604. Further, for example, if the moving object detected from the target image 605 is not determined to be a related subject, only the main subject image 616 is cut out from the target image 605 as the main subject image.

Next, the system controller 305 determines whether or not all the target images 603 to 605 have been read from the memory 304 (step S708), and if all the target images 603 to 605 have not been read, the process proceeds to step S704. go back. If all the target images have been read, the process proceeds to step S709.

In step S709, the separation background reading unit 312 reads each background image stored in the memory 304, and the background comparison extraction unit 311 extracts background image candidates from each background image by the same method as in step S505 in the process of FIG. .. After that, the separated subject reading unit 314 reads each main subject group image or each main subject image from the memory 304, and the video composition processing unit 313 reads the background image candidate extracted by the background comparison extraction unit 311 and each main subject group image or each main subject group image. Combine with each main subject image. Again, the background image and each main subject group image or each main subject image are not combined based on the average value or weighting of each image, but simply on the background image, each main subject group image or each main subject. Overwrite the image. After that, the image processing unit 315 performs angle-of-view adjustment and color conversion of each of the combined main subject group images or each main subject image and the background image to generate a composite image 617 (step S706), and ends this processing.

According to the process of FIG. 5, when a moving object is detected from the target image and the distance between the main subject image of the target image and the moving object image including the detected moving object is shorter than a predetermined distance, the target image is changed to the main subject image. The moving object images are collectively cut out as the main subject group image. Further, the cut out main subject group image is combined with the background image candidate to generate a composite image. As a result, if the distance to the main subject image is shorter than a predetermined distance, the moving object image is included in the composite image. Specifically, in the composite image 617 of FIG. 6, the ball 601 exists in a region 618 slightly distant from the person 602. Here, a moving object image in which the distance to the main subject image is shorter than a predetermined distance is considered to be an image of a moving object having a high relevance to the main subject. That is, by the processing of FIG. 5, it is possible to prevent the moving object that is highly related to the main subject from being included in the composite image.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications and modifications can be made within the scope of the gist thereof.

The present invention supplies a program that realizes one or more functions of each of the above-described embodiments to a system or device via a network or storage medium, and one or more processors of the computer of the system or device implements the program. It can also be realized by the process of reading and executing. The present invention can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Further, in each of the above-described embodiments, the case where the present invention is applied to an imaging device such as a camera has been described, but the present invention is a device that processes a plurality of consecutive images in a time series including a moving subject. If so, it can be applied. For example, the present invention can also be applied to PCs and mobile devices (tablets, smartphones, etc.).

300 Image processing device 304 Memory 305 System controller 306 Frame reading unit 307 Main subject group detection unit 308 Background separation unit 309 Separated background storage unit 310 Separated subject storage unit 311 Background comparison extraction unit 313 Image composition processing unit 320 Main subject-related judgment unit

Claims (11)

An image acquisition means for acquiring a plurality of consecutive images in chronological order,
An image storage means that separates a subject from each of the acquired plurality of images to acquire a background, and stores the plurality of the subjects and the plurality of backgrounds acquired from the plurality of images.
A background extraction means for comparing the plurality of stored backgrounds with each other and extracting one background having the smallest difference from the other backgrounds.
An image processing apparatus comprising: a composite image generation means for synthesizing a plurality of subjects to generate a composite image on the extracted background. The image processing apparatus according to claim 1, wherein the background extraction means compares two adjacent backgrounds in time series with each other. The image acquisition means
A still image acquisition means for acquiring a plurality of continuous still images in chronological order from a plurality of still images,
The image processing apparatus according to claim 1, further comprising a frame acquisition means for acquiring a plurality of consecutive frames in time series from a moving image. The image storage means
A subject detecting means for detecting the subject in each of the acquired plurality of images,
A subject range specifying means for specifying a range including the detected subject in each of the images, and
A subject saving means for cutting out the specified range from each of the images and saving the subject as the subject.
The image processing apparatus according to any one of claims 1 to 3, further comprising a background storage means for cutting out an area different from the specified range from each image and storing it as the background. .. The subject detection means
A main subject detecting means for detecting a main subject in each of the plurality of acquired images,
Each of the acquired plurality of images has a related subject detecting means for detecting a related subject related to the main subject.
The image processing apparatus according to claim 4, wherein the main subject and the related subject detected from the same image are detected as a subject group. The related subject detection means
A moving object detecting means for detecting a moving object which is a moving subject different from the main subject in the plurality of images,
It has a relevance determining means for determining whether or not the distance between the detected moving object and the main subject is shorter than a predetermined distance.
The image processing apparatus according to claim 5, wherein the related subject detecting means detects a moving object whose distance to the main subject is determined to be shorter than the predetermined distance as the related subject. The relevance determination means
A moving object quantity determining means for determining whether or not the number of moving objects determined to have a distance to the main subject shorter than the predetermined distance is equal to or greater than a predetermined threshold value.
It has a moving body continuous determination means for determining whether or not moving objects determined to have a distance to each of the main subjects shorter than the predetermined distance are continuous with each other in time series.
The image processing apparatus according to claim 6, wherein the related subject detecting means detects the related subject based on the determination result of the moving object quantity determining means and the determination result of the moving object continuous determining means. The image processing apparatus according to claim 4, wherein the subject range specifying means specifies the entire range including the detected subject in each of the images. The background preservation means is characterized in that all of the specified range is removed from each of the preserved backgrounds, and the background from which all of the specified ranges are removed is preserved as another background. 8. The image processing apparatus according to 8. An image acquisition process that acquires multiple images that are continuous in chronological order,
An image storage step of separating a subject from each of the acquired plurality of images to acquire a background, and storing the plurality of the subjects and the plurality of backgrounds acquired from the plurality of images.
A background extraction step of comparing the plurality of saved backgrounds with each other and extracting one background having the smallest difference from other backgrounds.
A control method for an image processing apparatus, which comprises a composite image generation step of synthesizing a plurality of subjects to generate a composite image on the extracted background. A program that causes a computer to execute the control method of an image processing device.
The control method is
An image acquisition process that acquires multiple images that are continuous in chronological order,
An image storage step of separating a subject from each of the acquired plurality of images to acquire a background, and storing the plurality of the subjects and the plurality of backgrounds acquired from the plurality of images.
A background extraction step of comparing the plurality of saved backgrounds with each other and extracting one background having the smallest difference from other backgrounds.
A program characterized by having a composite image generation step of synthesizing a plurality of subjects to generate a composite image on the extracted background.

Images