JP5531865B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

Conventionally, there is a desire to remove an animal body from an image including an unnecessary animal body such as a passerby or a passing object.
As a conventional technique that meets such demands, Patent Document 1 discloses a technique for estimating a state in which there is no moving object image by averaging or majority voting for each pixel from data of a plurality of frames captured at substantially the same angle of view. Yes.
In addition, as another conventional technique that meets such demands, it is solved as a multi-value label optimization problem on an MRF (Markov Random Field) with a frame number as a label for data of a plurality of frames captured at substantially the same angle of view. Therefore, Non-Patent Document 1 describes a technique for minimizing the discontinuity of an object.

Japanese Patent Laid-Open No. 11-225344 “Interactive Digital Photomontage” A. Agarwala et al. ACM SIGGRAPH, 2004

However, when the technique described in Patent Document 1 is employed, in the region of an animal image that is not substantially moved but is not completely stationary, or in the region where there is a large amount of traffic of a plurality of animal images, Some of them are left uninterrupted, or vague traces remain.
On the other hand, when the technique of Non-Patent Document 1 is adopted, the graph cut method can be used to remove the animal image without leaving much traces, but the multi-value that makes the calculation process enormous. A graph cut of the label is required.

  The present invention has been made in view of such a situation, and uses a continuously acquired image to generate a more natural composite image excluding the animal image as an obstacle. The purpose is to reduce the burden of image processing.

In order to achieve the above object, the invention described in claim 1
Image acquisition means for acquiring a plurality of pieces of image data having substantially the same angle of view continuous in time;
Generating means for generating reference image data for image synthesis based on a plurality of image data acquired by the image acquisition means;
Selection means for selecting at least two or more image data from a plurality of image data acquired by the image acquisition means based on the reference image data generated by the generation means;
Boundary setting means for setting a boundary between two or more image data selected by the selection means;
Based on the boundaries set by the boundary setting means, to provide an image processing apparatus, characterized in that it comprises a <br/> and synthesizing means for synthesizing two or more image data selected by said selection means.
The invention according to claim 2
The generating means generates reference image data in which the influence of a moving object imaged on the plurality of image data is less than that of the plurality of image data.
The invention according to claim 3
The generation unit generates the reference image data by calculating an average value or a median value of each pixel of the plurality of image data acquired by the image acquisition unit.
In order to achieve the above object, the invention described in claim 4
Image acquisition means for acquiring a plurality of pieces of image data having substantially the same angle of view continuous in time;
Reference image acquisition means for acquiring reference image data for image composition;
Based on the reference image data acquired by the reference image acquisition means, a plurality of image data acquired by the image acquisition means that is at least two or more image data that can be determined to be similar to the image of the reference image data. Selecting means for selecting from the image data of
Boundary setting means for setting a boundary between two or more image data selected by the selection means;
Combining means for combining two or more image data selected by the selection means based on the boundary set by the boundary setting means;
An image processing apparatus is provided.

The invention according to claim 5
The selection means includes
It is characterized in that at least two or more pieces of image data are selected so that the combined result image is most similar to the image of the reference image data.
The invention according to claim 6
First dissimilarity calculating means for calculating the dissimilarity between the image data selected by the selecting means with respect to adjacent pixels in each image data;
Second dissimilarity calculating means for calculating the dissimilarity of the two or more image data selected by the selecting means with respect to the reference image data;
Function setting means for setting a function having the difference calculated by the first difference calculation means and the difference calculated by the second difference calculation means as variables;
Further comprising
The boundary setting unit sets a boundary for two or more selected image data so that the function set by the function setting unit has a minimum value.
It is characterized by that.

In order to achieve the above object, the invention described in claim 7
Image acquisition means for acquiring a plurality of pieces of image data having substantially the same angle of view continuous in time;
Reference image acquisition means for acquiring reference image data for image composition;
First selection means for selecting, from a plurality of image data acquired by the image acquisition means, at least one image data having a pixel value similar to the reference image data acquired by the reference image acquisition means;
A plurality of images acquired by the image acquisition unit, the image data different from the image data selected by the first selection unit, such that the resultant image becomes an image more similar to the image of the reference image data. Second selection means for selecting at least one from the data;
Boundary setting means for setting a boundary between two or more image data selected by the first selection means and the second selection means;
Combining means for combining the two or more image data based on the boundary set by the boundary setting means;
An image processing apparatus is provided.

Further, the invention according to claim 8 is
First difference calculation means for calculating a difference between the image data selected by the first selection means and the second selection means for adjacent pixels in each image data;
Second dissimilarity calculating means for calculating the dissimilarity of the two or more image data selected by the first selecting means and the second selecting means with respect to the reference image data;
Function setting means for setting a function having the difference calculated by the first difference calculation means and the difference calculated by the second difference calculation means as variables;
Further comprising
The boundary setting unit sets a boundary in the two or more image data so that the function set by the function setting unit has a minimum value.
It is characterized by that.
In the invention described in claim 9, the reference image acquisition means includes generation means for generating reference image data for image composition based on a plurality of image data acquired by the image acquisition means. It is characterized by.

Further, the invention described in 請 Motomeko 10,
It further includes imaging means for imaging a plurality of times at substantially the same angle of view continuously in time, and outputting image data obtained as a result of the plurality of times of imaging,
The image acquisition means acquires a plurality of image data output from the imaging means as a plurality of image data having substantially the same angle of view that is continuous in time.
It is characterized by that .
The invention according to claim 11 is
Processing means for processing to reduce the amount of information of the plurality of image data;
Is further provided.

In order to achieve the above object, the invention according to claim 12 is
An image acquisition step of acquiring a plurality of image data having substantially the same angle of view that is continuous in time;
A generating step for generating reference image data for image synthesis based on the plurality of image data acquired by the image acquiring step;
A selection step of selecting at least two or more image data from the plurality of image data acquired in the image acquisition step based on the reference image data generated in the generation step ;
A boundary setting step for setting a boundary between two or more image data selected in the selection step;
And a synthesis step of synthesizing two or more image data selected in the selection step based on the boundary set in the boundary setting step.
In order to achieve the above object, the invention according to claim 13 is
An image acquisition step of acquiring a plurality of image data having substantially the same angle of view that is continuous in time;
A reference image acquisition step of acquiring reference image data for image synthesis;
Based on the reference image data acquired in the reference image acquisition step, at least two or more pieces of image data that can be determined to be similar to the image of the reference image data are acquired in the image acquisition step. A selection step of selecting from a plurality of image data;
A boundary setting step for setting a boundary between two or more image data selected in the selection step;
A combining step of combining two or more image data selected in the selection step based on the boundary set in the boundary setting step;
An image processing method is provided.
In order to achieve the above object, the invention according to claim 14
An image acquisition step of acquiring a plurality of image data having substantially the same angle of view that is continuous in time;
A reference image acquisition step of acquiring reference image data for image synthesis;
A first selection step of selecting at least one image data similar in pixel value to the reference image data acquired by the reference image acquisition step from a plurality of image data acquired by the image acquisition step;
A plurality of images acquired by the image acquisition step, the image data different from the image data selected by the first selection step so that the image of the synthesis result is an image more similar to the image of the reference image data A second selection step of selecting at least one from the data;
A boundary setting step for setting a boundary between two or more image data selected by the first selection step and the second selection step;
A combining step of combining the two or more image data based on the boundary set by the boundary setting step;
An image processing method is provided.

In order to achieve the above object, the invention according to claim 15 is
Computer
Image acquisition means for acquiring a plurality of pieces of image data having substantially the same angle of view continuous in time;
Generating means for generating reference image data for image synthesis based on a plurality of image data acquired by the image acquiring means;
Selection means for selecting at least two or more image data from a plurality of image data acquired by the image acquisition means based on the reference image data generated by the generation means;
Boundary setting means for setting a boundary between two or more image data selected by the selection means;
Combining means for combining the two or more image data based on the boundary set by the boundary setting means
Provide a program to function as
In order to achieve the above object, the invention described in claim 16 provides:
Computer
Image acquisition means for acquiring a plurality of pieces of image data having substantially the same angle of view continuous in time;
Reference image acquisition means for acquiring reference image data for image synthesis;
Based on the reference image data acquired by the reference image acquisition means, a plurality of image data acquired by the image acquisition means that is at least two or more image data that can be determined to be similar to the image of the reference image data. Selection means for selecting from the image data of
Boundary setting means for setting a boundary between two or more image data selected by the selection means;
Based on the boundaries set by the demarcation means, combining means for combining the two or more image data,
Provide a program to function as
In order to achieve the above object, the invention described in claim 17
Computer
Image acquisition means for acquiring a plurality of pieces of image data having substantially the same angle of view continuous in time;
Reference image acquisition means for acquiring reference image data for image synthesis;
First selection means for selecting, from a plurality of image data acquired by the image acquisition means, at least one image data having a pixel value similar to the reference image data acquired by the reference image acquisition means;
A plurality of images acquired by the image acquisition unit, the image data different from the image data selected by the first selection unit, such that the resultant image becomes an image more similar to the image of the reference image data. A second selection means for selecting at least one from the data;
Boundary setting means for setting a boundary between two or more image data selected by the first selection means and the second selection means;
A combining means for combining the two or more image data based on the boundary set by the boundary setting means;
Provide a program to function as

  ADVANTAGE OF THE INVENTION According to this invention, the more natural synthetic | combination image except the said animal body image used as an obstruction can be produced | generated using the image acquired continuously.

It is a block diagram which shows the structure of the hardware of the imaging device which concerns on one Embodiment of this invention. It is a functional block diagram which shows the functional structure of the image process part of the imaging device of FIG. It is a figure which shows an example of the specific process result of the image process part of FIG. It is a flowchart explaining the flow of the image compositing process mainly performed by the image processing unit of FIG.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a hardware configuration of an imaging apparatus 1 as an embodiment of an image processing apparatus according to the present invention. The imaging device 1 can be configured by a digital camera, for example.

  The imaging apparatus 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, an image processing unit 14, a bus 15, an input / output interface 16, and an imaging unit. 17, an operation unit 18, a display unit 19, a storage unit 20, a communication unit 21, and a drive 22.

The CPU 11 executes various processes according to a program recorded in the ROM 12 or a program loaded from the storage unit 20 to the RAM 13.
The RAM 13 also appropriately stores data necessary for the CPU 11 to execute various processes.

The image processing unit 14 includes a DSP (Digital Signal Processor), a VRAM (Video Random Access Memory), and the like, and performs various types of image processing on image data in cooperation with the CPU 11.
For example, the image processing unit 14 performs image processing such as noise reduction, white balance, and camera shake correction on captured image data input from the imaging unit 17 described later.
Here, in this embodiment, the processing unit of the image by the imaging apparatus 1 is one still image, and the still image as such a processing unit is referred to as a “frame” in this specification. Hereinafter, unless otherwise specified, an image means a frame.
In the present embodiment, the image size (resolution) when image processing is performed by the imaging apparatus 1 including the image processing unit 14 is, as a general rule, the size recorded on the removable medium 31 described later. To do. Hereinafter, such a size is referred to as a “normal size”.
In the present embodiment, the image processing unit 14 further includes a continuous shot image acquisition unit 41 and an image composition unit 42. Although details will be described later with reference to FIG. 2, the reduction unit 51 of the continuous-shot image acquisition unit 41 changes the size of the image data from a normal size to a smaller size, for example, a QVGA (Quarter-Video Graphics Array). ). The image data of such a reduced size is a processing target of the image composition unit 42.

  The CPU 11, ROM 12, RAM 13, and image processing unit 14 are connected to each other via a bus 15. An input / output interface 16 is also connected to the bus 15. An imaging unit 17, an operation unit 18, a display unit 19, a storage unit 20, a communication unit 21, and a drive 22 are connected to the input / output interface 16.

  Although not shown, the imaging unit 17 includes an optical lens unit and an image sensor.

The optical lens unit is configured with a lens that collects light, such as a focus lens and a zoom lens, in order to capture an image of the subject.
The focus lens is a lens that forms a subject image on the light receiving surface of the image sensor. The zoom lens is a lens that freely changes the focal length within a certain range.
The optical lens unit is also provided with a peripheral circuit for adjusting setting parameters such as focus, exposure, and white balance as necessary.

The image sensor includes a photoelectric conversion element, an AFE (Analog Front End), and the like.
The photoelectric conversion element is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) type photoelectric conversion element. A subject image is incident on the photoelectric conversion element from the optical lens unit. Therefore, the photoelectric conversion element photoelectrically converts (captures) the subject image, accumulates the image signal for a predetermined time, and sequentially supplies the accumulated image signal as an analog signal to the AFE.
The AFE executes various signal processing such as A / D (Analog / Digital) conversion processing on the analog image signal. A digital signal is generated by various signal processing and output as an output signal of the imaging unit 17.
Note that such an output signal of the imaging unit 17 is referred to as “captured image data” in this specification. Therefore, captured image data is output from the imaging unit 17 in units of frames and is supplied to the CPU 11, the image processing unit 14, and the like as appropriate. Hereinafter, unless otherwise specified, the captured image means a frame.

The operation unit 18 includes various buttons and the like, and accepts user instruction operations.
The display unit 19 is composed of a liquid crystal display or the like and displays various images.
The storage unit 20 is configured by a DRAM (Dynamic Random Access Memory) or the like, and temporarily stores image data output from the image processing unit 14 or the like. The storage unit 20 also stores various data necessary for processing by the image processing unit 14 and the like.
The communication unit 21 controls communication with other devices (not shown) via a network including the Internet.

  A removable medium 31 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately attached to the drive 22. The program read from the removable medium 31 by the drive 22 is installed in the storage unit 20 as necessary. The removable medium 31 can also store various data such as image data stored in the storage unit 20 in the same manner as the storage unit 20.

  FIG. 2 is a functional block diagram showing a functional configuration of the image processing unit 14 in such an imaging apparatus 1.

  As shown in FIG. 2, the image processing unit 14 includes a continuous-shot image acquisition unit 41 and an image composition unit 42.

The continuous image acquisition unit 41 acquires data of a plurality of images obtained as a result of continuous shooting.
Continuous shooting means that the imaging device 1 or the like captures images at substantially the same angle of view a plurality of times in succession in time.
Here, what is described as the imaging device 1 or the like is that the image data acquired by the continuous-shot image acquisition unit 41 is not necessarily the data of the captured image output from the imaging unit 17 of the imaging device 1. This is because data of an image captured by another imaging device may be used. However, for simplification of description, data of a plurality of captured images obtained as a result of continuous shooting by the imaging unit 17 in a state where the imaging device 1 is fixed are acquired by the continuous image acquisition unit 41. .
Note that it is sufficient that the continuous shooting timing is before the acquisition timing of the continuous-shot image acquisition unit 41, and it is not particularly necessary to be immediately before the acquisition timing of the continuous-shot image acquisition unit 41.
In the present embodiment, it is assumed that the size of data of a plurality of images obtained as a result of continuous shooting is a normal size.

The continuous shot image acquisition unit 41 includes a reduction unit 51.
The reduction unit 51 reduces the size (resolution) of the data of the plurality of images acquired by the continuous shot image acquisition unit 41 from the normal size to a smaller size (eg, QVGA size).

  Hereinafter, each of the plurality of image data reduced by the reduction unit 51 in this way is referred to as “reduced continuous shooting data”, and the code of P [i] is used to distinguish each of the reduced continuous shooting data individually. Is used. Here, i indicates a number (hereinafter referred to as “frame number”) uniquely assigned to each of a plurality of image data acquired by the continuous-shot image acquisition unit 41. When a plurality of n (n is an integer value of 2 or more) image data is acquired by the continuous image acquisition unit 41, any frame number from i = 1 to n is shown. That is, the reduced continuous shooting data P [i] indicates reduced data of the image data of frame number i.

Note that the reduction unit 51 is not an essential component for the continuous-shot image acquisition unit 41.
However, one of the objects of the present invention, that is, the burden of image processing for generating a more natural composite image excluding the animal body image that becomes an obstacle using continuously acquired images. In order to achieve the purpose of reducing the weight, it is necessary to suppress the burden of calculation processing of the image composition unit 42 described later as much as possible.
Therefore, in the present embodiment, the image composition unit 42 described later does not directly use the data (normal size data) of a plurality of images acquired by the continuous shot image acquisition unit 41, but reduces the data volume. By using the normalized continuous shooting data P [1] to P [n], the burden of calculation processing is suppressed. For this reason, in the present embodiment, the reduction unit 51 is provided in the continuous-shot image acquisition unit 41.
In addition, the reduction unit 51 is provided because the image composition unit 42 uses the reduced image data, and the data of a plurality of images acquired by the continuous shot image acquisition unit 41 is captured without using a tripod. This is because misalignment that occurs in the case of a problem is not a problem. Of course, if necessary, alignment by geometric transformation of the image may be performed, and a better result is obtained.

The image synthesis unit 42 selects and acquires data of a plurality of images to be synthesized from the reduced continuous shooting data P [1] to P [n]. The image composition unit 42 sets boundaries for dividing each of the data of a plurality of images to be synthesized. The image synthesis unit 42 divides each of the data of the plurality of images to be synthesized along the boundary, and synthesizes a part of the data of each divided image so as to be connected along the boundary. Then, the data of the composite image is generated.
The image composition unit 42 appropriately sets this boundary, thereby generating composite image data from which the moving object image that is an obstacle included in at least a part of the plurality of images obtained by continuous shooting is removed. It becomes possible.

  The image composition unit 42 executes such a series of processes to generate composite image data, so that a reference image generation unit 61, a reference most approximate image selection unit 62, a complementary image selection unit 63, and 2 A value labeling unit 64 and a combining unit 65 are provided.

FIG. 3 shows a specific example of the processing result of the image composition unit 42 having such a functional configuration.
Specifically, FIG. 3A illustrates a specific example of each processing result of the reference image generation unit 61, the reference closest approximate image selection unit 62, and the complementary image selection unit 63.
3B and 3C show specific examples of processing results (including intermediate results) of the binary labeling unit 64 and the synthesis unit 65. FIG.
Hereinafter, the functional configuration of the image composition unit 42 will be described with reference to FIG. 3 as appropriate.

The reference image generation unit 61 generates data of a reference image B that serves as a reference for image composition based on the reduced continuous shooting data P [1] to P [n].
The data generation method of the reference image B may be a method that can remove the moving object image to some extent. For example, the median or average of the reduced continuous shooting data P [1] to P [n] for each pixel and each color component A general method such as taking can be adopted. Alternatively, for each pixel, search for frame data that minimizes the sum of the distance (absolute difference value or difference squared) of each pixel from another frame, and take the pixel value of that frame as a reference. A technique for generating image B data can also be employed.
In the reference image B, the animal image that is an obstacle is considerably removed, but a blurred trace remains, so it is unsuitable as an image of the final output.In this embodiment, in this embodiment, the final output is estimated. It is used as a reference. Here, for example, it is assumed that the data of the image 93 shown in FIG. 3A is generated as the data of the reference image B.

In this embodiment, the reference closest approximate image selection unit 62 selects the image having the pixel value closest to the data of the reference image B as one of the compositing targets from the reduced continuous shooting data P [1] to P [n]. Select and retrieve data. The image data selected as one of the compositing targets by the reference closest approximate image selection unit 62 in this way is hereinafter referred to as “reference closest approximate image data”.
However, it is sufficient if a synthesized image closest to the reference image is obtained as a result of synthesizing the reference closest approximate image that is one of the synthesis targets and another image to be synthesized (a complementary image described later). The reference closest approximation image does not necessarily need to be the closest approximation to the reference image B. In the following description, it is assumed that the reduced continuous shooting data P [p] of the frame number p among the reduced continuous shooting data P [1] to P [n] is selected as reference closest approximate image data.
In this case, the frame number p is obtained by the following equation (1).
In Expression (1), d (α, β) in Σ represents a function that outputs the degree of difference between the pixel value α and the pixel value β. Here, the pixel value of the pixel u of the reference image B is employed as the pixel value α. As the pixel value β, the pixel value of substantially the same pixel u among the pixel values constituting the reduced continuous shooting data P [i] is employed. Here, as the degree of difference, a luminance difference absolute value, a sum of absolute differences of color components, a sum of squares of differences, or the like can be employed.
According to the formula (1), the frame number i of the frame having the smallest difference value for all pixels is obtained as the frame number p of the data P [p] of the reference most approximate image. Become.
Here, for example, it is assumed that the data of the image 91 shown in FIG. 3A is acquired as the data P [p] of the reference most approximate image.

In this embodiment, the complementary image selection unit 63, when optimally cut and pasted from the reduced continuous shooting data P [1] to P [n] in combination with the data P [p] of the reference closest approximate image, The complementary image data that can be closest to the data is selected and acquired. The image data selected as one of the compositing targets by the reference closest approximate image selection unit 62 in this way is hereinafter referred to as “complementary image data”. In the following, it is assumed that the reduced continuous shooting data P [q] of the frame number q is selected as the complementary image data among the reduced continuous shooting data P [1] to P [n].
In this case, the frame number q is obtained by the following equation (2).
In Expression (2), d (A, B) in Σ represents a function that outputs the minimum value of the parameters A and B. According to the equation (2), the degree of difference between the reduced continuous shooting data P [i] or the reference closest approximate image data P [p] and the reference image B data is compared for each pixel. It is done. Then, the frame number i of the frame in which the sum of the smaller ones of these differences is the smallest is obtained as the frame number q of the complementary image data P [q].
Here, for example, it is assumed that the data of the image 92 shown in FIG. 3A is acquired as the complementary image data P [q].

In order to cut and paste the data P [p] of the reference closest approximation image and the data P [q] of the complementary image, the binary labeling unit 64 applies p (that is, data P [of the reference closest approximation image) to each pixel. p] -derived pixels) or q (that is, pixels derived from complementary image data P [q]).
Such processing of the binary labeling unit 64 is hereinafter referred to as “binary labeling”.
The binary labeling method is not particularly limited, and various methods can be employed. In the present embodiment, a method using a graph cut method is employed.
The binary labeling unit 64 calculates a smoothing term and a data term, which are two terms of a cost function described later, from the data P [p] of the reference most approximate image and the data P [q] of the complementary image.
The binary labeling unit 64 searches for binary labeling that divides all pixels into the p side and the q side so as to minimize the cost function by a graph cut method described later.
Refer to Non-Patent Document 1 for details of the cost function and the graph cut method.
The binary labeling unit 64 sets a boundary line for optimally cutting and pasting the data P [p] of the reference closest approximate image and the data P [q] of the complementary image by executing this binary labeling. Is possible.
The binary labeling unit 64 executes such a series of processes to set an optimal boundary line, so that a first difference calculation unit 71, a second difference calculation unit 72, and a cost function setting unit 73 are set. And a boundary setting unit 74.

Specifically, FIG. 3B shows a specific example of the processing result of the second difference degree calculation unit 72.
That is, in the example of FIG. 3A, as described above, the data P [p] of the reference closest approximate image is represented by the data of the image 91, and the data P [q] of the complementary image is represented by the data of the image 92. The data of the reference image B is represented by the data of the image 93.
For the data of each of the images 91 to 93 in FIG. 3A, the data obtained as the data term Cp is represented by a gray scale image as the image 101 shown in FIG. An image 102 shown in FIG. 3B represents data obtained as Cq as a grayscale image.

In the present embodiment, the first difference calculation unit 71 calculates the function C (u, v) of the adjacent pixel pair (u, v) according to the equation (3). Although the meaning of the expression (3) will be described later, the function C is hereinafter referred to as “smoothing term”.
According to Expression (3), the sum of the degrees of difference between the data P [p] of the reference closest approximate image and the data P [q] of the complementary image in the adjacent pixel pair (u, v) is expressed as a function C (u, v ), That is, a method of using a smoothing term is adopted. This method is a general calculation method also found in Non-Patent Document 1.
The smaller this smoothing term is, the more similar the data P [p] of the reference most approximate image and the data P [q] of the complementary image become, so if the boundary line is drawn in such a region, It is possible to make the cuts inconspicuous. However, in order to determine which side of the two areas divided by the boundary line should be excluded, that is, the area of the moving body image, the data term calculated by the second difference calculating unit 72 is Necessary.

In the present embodiment, the second dissimilarity calculator 72 calculates the function Cp (u) and the function Cq (u) of the pixel u according to the equations (4) and (5). Although the meanings of these functions Cp (u) and Cq (u) will be described later, these functions Cp and Cq are hereinafter collectively referred to as “data terms”.
According to the equations (4) and (5), the degree of difference from the data of the reference image B at each pixel position u for each of the data P [p] of the reference most approximate image and the data P [q] of the complementary image Is used as a data term. This technique is a technique unique to the present embodiment that has not been seen in the past.
In the reference image B, it is considered that the trace of the moving object image that is an obstacle is reduced. Therefore, the smaller this data term is, the higher the possibility of being the background region. Therefore, if a boundary line that cuts out a region where this data term is small can be drawn in a region where the smoothing term is small, the synthesized image data close to the data of the reference image B is obtained without crossing the animal image. It becomes possible.

The cost function setting unit 73 sets a cost function using the smoothing term C calculated by the first difference calculation unit 71 and the data terms Cp and Cq calculated by the second difference calculation unit 72. This cost function determines the optimal boundary line for dividing each pixel into either a region to be cut out from the data P [p] of the reference closest approximation image or a region to be cut out from the data P [q] of the complementary image. This is a function for searching.
In order to set the cost function, the cost function setting unit 73 assumes a graph composed of a set of nodes and a set of edges extending between the nodes as follows.
That is, first, the cost function setting unit 73 uses, as nodes, all the pixels of the synthesized image data that are virtually created by cutting and pasting the data P [p] of the reference closest approximate image and the data P [q] of the complementary image. Edges are set between all adjacent pixel pairs (u, v).
Next, the cost function setting unit 73 assumes a p-side space and a q-side space on the front and back sides of the graph, and adds one virtual node to each space.
Hereinafter, a node virtually added to the p-side space is referred to as a “source node”, and a node virtually added to the q-side space is referred to as a “sink node”.
Further, the cost function setting unit 73 adds an edge extending from the source node to all pixels and an edge extending from all pixels to the sink node.
The cost function setting unit 73 gives the weight of the smoothing term C (u, v) calculated by the first dissimilarity calculation unit 71 to the edge extending between each adjacent pixel pair (u, v).
The cost function setting unit 73 gives the weight of the data term Cq (u) calculated by the second difference calculation unit 72 to the edge extending from the source node to each pixel. Further, the cost function setting unit 73 gives the weight of the data term Cp (u) calculated by the second difference calculation unit 72 to the edge extending from each pixel to the sink node.
The graph thus weighted is hereinafter referred to as a “weighted graph”.
The cost function setting unit 73 prepares a weighted graph necessary for calculating the cost function and supplies the graph to the boundary setting unit 74.

Based on the above contents, the graph cut and the cost function will be described so that the boundary setting unit 74 can be easily understood.
In a graph G, when all the edges belonging to a subset E of the edge set of G are separated, if the node set of G is cut into two subsets, the subset E of the edge is expressed as “cut E of graph G "
The sum of the weights given to all the edges belonging to the cut E is hereinafter referred to as “the cost function of the cut E”.

  The boundary setting unit 74 examines all cuts E that separate the weighted graph supplied from the cost function setting unit 73 into a p-side node set and a q-side node set.

When the weighted graph is cut into the p-side node set and the q-side node set by cut E, all the p-side pixels and q-side pixels that are adjacent on the boundary line between the p-side pixel and the q-side pixel Since the edges that connected the pair are separated, they belong to cut E.
Therefore, for all adjacent p-side pixels u and q-side pixels v, the edge weight (smoothing term) C (u, v) connecting them is added to the cost function of cut E.
As the weight C (u, v) is smaller, there is no difference between the data P [p] of the reference most approximate image and the data P [q] of the complementary image in the vicinity of the pixel pair (u, v). It will not stand out.

In addition, since the edge connecting the source node belonging to the p side and all the pixels placed on the q side is inevitably separated, it belongs to the cut E.
Therefore, for all the pixels u on the q side, the edge weight (data term) Cq (u) connecting the source node is added to the cost function of the cut E.
The smaller this weight Cq (u), the smaller the difference between the complementary image data P [q] and the reference image B data at the pixel u, and therefore the complementary image data P [q] pixel at this pixel u. It means that it was good to adopt the value.

Conversely, the edge connecting the sink node belonging to the q side and all the pixels placed on the p side is inevitably separated, and therefore belongs to the cut E.
Therefore, for all the p-side pixels u, the edge weight (data term) Cp (u) connecting the sink node is added to the cost function of the cut E.
As the weight Cp (u) is smaller, there is no difference between the data P [p] of the reference most approximate image and the data of the reference image B at the pixel u. Therefore, the data P [ It was good to adopt the pixel value of p].

  If a cut E that minimizes the cost function calculated in this way is obtained from all possible cuts of the weighted graph, the cut E is between the p-side pixel and the q-side pixel. The boundary line is set, and the nearly optimal binary labeling for cutting and pasting the data P [p] of the reference most approximate image and the data P [q] of the complementary image is achieved.

The series of methods described above is a method called a graph cut, and several specific algorithms are known.
In addition, even in the same graph cut, the multi-value label graph cut described in Non-Patent Document 1 requires the above calculation to be repeated many times, and the calculation process becomes enormous. On the other hand, since the graph cut of the present embodiment described above only cuts and pastes the reduced continuous shooting data of two frames, it becomes a graph cut of a binary label and has an effect of reducing the burden of calculation processing. Is possible.

  The boundary setting unit 74 supplies the binary label obtained in this way to the synthesis unit 65.

The synthesizing unit 65 uses the binary label determined by the boundary setting unit 74 and adopts the corresponding pixel value from the data P [p] of the reference most approximate image for the p-side pixel, and complementary for the q-side pixel. By adopting the corresponding pixel value from the image data P [q], the composite image data is generated.
The composition unit 65 supplies the composite image data to the display control unit 81.

Specifically, FIG. 3C shows a specific example of processing results of the boundary setting unit 74 and the combining unit 65.
That is, in the example of FIG. 3A, the data P [p] of the reference most approximate image is represented by the data of the image 91, the data P [q] of the complementary image is represented by the data of the image 92, and the reference image B These data are represented by the data of the image 93.
In this case, the binary label created by the boundary setting unit 74 is displayed so that the area where the pixel value of the data P [p] of the reference most approximate image should be adopted is represented in white and the area to be discarded is represented in black. The obtained image is the image 111 of FIG. Conversely, an image 112 shown in FIG. 3C is displayed such that the area where the pixel value of the complementary image data P [q] should be adopted is represented in white and the area to be discarded is represented in black. As described above, the image 111 and the image 112 have a black and white reversed relationship.
As the black and white boundary line (dotted line in the figure) of the image 111 and the image 112, the data P [p] of the reference most approximate image and the data P [q] of the complementary image are cut and pasted, so that FIG. ) Of the composite image 113 is obtained.

Next, with reference to FIG. 4, an image composition process mainly executed by the image processing unit 14 in FIG. 2 will be described.
FIG. 4 is a flowchart for explaining the flow of the image composition process.

  The image composition process is started in response to a user's instruction to start image composition input from a button or selection menu provided on the operation unit 18, and the following processes in steps S1 to S13 are executed.

  In step S1 of FIG. 4, the reduction unit 51 of the continuous shot image acquisition unit 41 reduces the data of a plurality of images (here, n images in this case) obtained as a result of the continuous shooting, thereby reducing the reduction sequence. Copy data P [1] to P [n] are generated.

In step S2, the reference image generation unit 61 generates data of the reference image B from the reduced continuous shooting data P [1] to P [n] generated in the process of step S1.
For example, in the example of FIG. 3 described above, the data of the image 93 is generated as the data of the reference image B in the process of step S2.

In step S3, the reference closest approximate image selection unit 62 selects data P [p] of the reference closest approximate image from the reduced continuous shooting data P [1] to P [n] generated in the process of step S1. .
For example, in the example of FIG. 3 described above, the data of the image 91 is generated as the data P [p] of the reference most approximate image in the process of step S3.

In step S4, the complementary image selection unit 63 selects complementary image data P [q] from the reduced continuous shooting data P [1] to P [n] generated in the process of step S1.
For example, in the example of FIG. 3 described above, the data of the image 92 is generated as the data P [q] of the complementary image in the process of step S4.

In step S5, the binary labeling unit 64 generates the data of the reference image B generated in the process of step S2, the data P [p] of the reference most approximate image selected in step S3, and the complementary selected in step S4. A cost function is set from the image data P [q], and optimum binary labeling is obtained by cost minimization.
For example, in the example of FIG. 3 described above, the data of the image 101 is generated as data representing the degree of difference between the data of the reference image B and the data P [p] of the reference most approximate image in the process of step S5. Further, the data of the image 102 is generated as data representing the degree of difference between the data of the reference image B and the data P [q] of the complementary image. These data are used as data terms of the cost function.

In step S6, the synthesizer 65 synthesizes each data of the reference most approximate image data P [p] and the complementary image data P [q] according to the binary label obtained in the process of step S5. Data of a composite image (referred to as a “reduced composite image” to be distinguished from a normal-size composite image described later) is generated.
For example, in the example of FIG. 3 described above, the data of the image 111 in which the region to be cut out by selecting from the data P [p] of the reference closest approximate image is displayed in white according to the binary label obtained in the process of step S5. Will be generated. In addition, according to the binary label, data of an image 112 in which a region to be cut out by selecting from the complementary image data P [q] is generated in white is generated. As a result, in the process of step S6, the data of the image 113 is generated as the data of the reduced composite image.
When the data of the reduced combined image is supplied from the combining unit 65 to the display control unit 81, the process proceeds to step S7.

  In step S <b> 7, the display control unit 81 causes the display unit 19 to display an image represented by the data supplied from the combining unit 65, that is, a reduced combined image.

Here, the display form of the reduced composite image is not particularly limited. For example, the display control unit 81 uses the screen of the display unit 19 with the size of the reduced composite image as it is (smaller than the display size of the display unit 19). Can be displayed as part of Further, for example, the display control unit 81 can enlarge the size of the reduced composite image to the display size of the display unit 19 and display it as the entire screen of the display unit 19.
In any case, a reduced composite image with reduced resolution is displayed on the display unit 19 as compared with a plurality of images obtained by continuous shooting.
The composite image finally obtained as a recording target by the user is a composite image having substantially the same image quality as that of a plurality of images obtained by continuous shooting, that is, the same size as the size of the plurality of images obtained by continuous shooting. Although it is a (normal size) composite image, the normal size composite processing of an image with a large number of pixels generally takes a long processing time, and it is not appropriate to perform the processing uniformly at a stage where it is unknown whether the user likes it. Whether or not there is a significant wrinkle such as discontinuity of the subject due to the composition is often visible even in the reduced composite image, so this display is performed as a preview.
Therefore, the user can instruct whether or not to adopt a normal size composite image corresponding to the reduced composite image displayed on the display unit 19 by operating the operation unit 18 as a recording target.

In step S <b> 8, the display control unit 81 displays an image indicating whether or not the composite image has been employed on the display unit 19, thereby notifying the confirmation of whether or not the composite image has been employed.
Note that the notification method is not particularly limited to image display. For example, a method of outputting a voice (message) indicating whether or not a composite image has been adopted from a speaker (not shown) may be employed.

In step S9, the image composition unit 42 determines whether or not an instruction to adopt a composite image has been given.
When the user operates the operation unit 18 to instruct to adopt a normal size composite image corresponding to the reduced composite image displayed on the display unit 19 as a recording target, YES is determined in step S9. And the process proceeds to step S10.

In step S10, the image composition unit 42 generates data of a normal size composite image.
In other words, the reference closest approximate image selection unit 62 acquires data of a reference closest approximate image having a normal size. The normal-size reference closest approximate image data refers to the reference closest approximate image data (one of the reduced continuous shooting data) among the plurality of image data obtained as a result of continuous shooting, and is reduced by the reduction unit 51. This is the image data before the image.
The complementary image selection unit 63 acquires normal-size complementary image data. Normal size complementary image data refers to an image before complementary image data (one of the reduced continuous shooting data) is reduced by the reduction unit 51 among a plurality of image data obtained as a result of continuous shooting. Data.
The binary labeling unit 64 performs interpolation enlargement (upsampling) of the binary label used in the process of step S6 to a normal size. The label value of 0 or 1 may be simply interpolated by the nearest neighbor method, or the label value of 0 or 1 is interpolated by, for example, linear interpolation, and the intermediate position of different label values is [0,1] It may be a real value α value (described later).
The synthesizing unit 65 applies the binary label expanded to the normal size to each data of the reference approximate image I [p] having the normal size and the complementary image I [q] having the normal size, thereby synthesizing the normal size. Generate image data. Specifically, when A (u) is the 0 or 1 label value at the pixel position u or the α value of [0,1] interpolated as described above, 0 is the frame p side, and 1 is the frame q side. In this case, the output pixel value can be calculated as R (u) = I [p] (u) × (1−A (u)) + I [q] (u) × A (u).
When the normal size composite image data is supplied from the compositing unit 65 to the display control unit 81, the process proceeds to step S11.

In step S <b> 11, the display control unit 81 displays an image represented by the data supplied from the combining unit 65, that is, a normal size combined image as the entire screen of the display unit 19.
As a result, the image composition process ends.

  In contrast, if the reduced composite image displayed on the display unit 19 in step S7 is not a desired image, the user is instructed not to employ the composite image as a recording target. In such a case, it is determined as NO in Step S9, and the process proceeds to Step S12.

In step S <b> 12, the reference closest approximate image selection unit 62 acquires data of a reference closest approximate image having a normal size.
When the data of the standard closest approximate image having the normal size is supplied from the reference closest approximate image selecting unit 62 to the display control unit 81 via the synthesizing unit 65, the process proceeds to step S13.

In step S <b> 13, the display control unit 81 displays an image represented by the data supplied from the reference closest approximation image selection unit 62 via the synthesis unit 65, i.e., a normal size reference closest approximation image as a substitute image. Since this is an image at the moment with the smallest number of moving objects, it is considered to be an image that is close to the user's desired image, and since it is a single image, it is an image free from wrinkles due to synthesis.
As a result, the image composition process ends.

As described above, the imaging apparatus 1 according to the present embodiment selects two complementary images from a plurality of images having substantially the same angle of view that are temporally continuous, and is optimal for each by a technique such as graph cut. By setting the boundary and cutting and pasting, it is possible to obtain a composite image from which the moving object is removed by light calculation processing.
In the technique of Patent Document 1 or the like, there is a problem that a faint trace of a moving object remains by averaging. However, in the present embodiment, since the synthesis is performed by cutting and pasting instead of averaging, such a trace does not remain. In addition, according to the graph cut method, a boundary that crosses the moving object can be avoided as much as possible, so that it is difficult for the fragments of the moving object to remain. Furthermore, the graph cut of the multi-valued label requires enormous calculation processing, but the graph cut of the binary label can make the calculation processing extremely light compared to the graph cut of the multi-value label.

  In addition, this invention is not limited to the above-mentioned embodiment, The deformation | transformation in the range which can achieve the objective of this invention, improvement, etc. are included in this invention.

For example, in the above-described embodiment, first, the data of the reference closest approximate image is selected, and then the complementary image data to be combined therewith is selected. However, as described above, the result of combining the two image data is only the result. Since it is only necessary to approximate to the data of the reference image, an embodiment in which any two frames are examined from the reduced continuous shooting data and the optimum combination (p, q) is selected from the two frames is considered. . Specifically, it can be realized by using the following formula (6) instead of formulas (1) and (2).
However, since the processing time increases when all the arbitrary combinations are examined, an embodiment in which the first frame and the last frame are combined can be considered more easily. For).
Furthermore, the images to be combined are not necessarily limited to two frames, and three or more frames can be combined. In this case, the labeling is ternary or more, and the processing time increases. However, if the number of frames is sufficiently smaller than n, the processing can be performed at a sufficiently high speed.
In addition, there can be various variations in the design of the degree of difference, such as the difference absolute value, the difference square value, the inversion of the correlation value, the luminance alone or the sum of the color components, or the conversion or clipping of the sum by a monotone function. .

For example, in the above-described embodiment, the processing target of the image compositing process is obtained by reducing the data (normal size data) of a plurality of images obtained as a result of continuous shooting, but is not particularly limited thereto.
For example, if the processing time is allowed, data of a plurality of images (normal size data) obtained as a result of continuous shooting may be used as the processing target of the image composition processing.

  For example, in the above-described embodiment, the image processing apparatus to which the present invention is applied has been described as an example configured as an imaging apparatus such as a digital camera. However, the present invention is not particularly limited to the imaging apparatus, and the above-described image processing is performed regardless of the presence or absence of the imaging function (data of a plurality of images obtained as a result of continuous shooting may be acquired from another apparatus). It can be applied to general electronic devices that can execute the above. Specifically, for example, the present invention can be applied to a personal computer, a video camera, a portable navigation device, a portable game machine, and the like.

  The series of processes described above can be executed by hardware or can be executed by software.

  When a series of processing is executed by software, a program constituting the software is installed on a computer or the like from a network or a recording medium. The computer may be a computer incorporated in dedicated hardware. The computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

  The recording medium including such a program is not only constituted by the removable medium 31 of FIG. 1 distributed separately from the apparatus main body in order to provide the program to the user, but also in a state of being incorporated in the apparatus main body in advance. It is comprised with the recording medium etc. which are provided in this. The removable medium is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disk is composed of, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versatile Disk), or the like. The magneto-optical disk is configured by an MD (Mini-Disk) or the like. In addition, the recording medium provided to the user in a state of being preinstalled in the apparatus main body is configured by, for example, the ROM 12 in FIG. 1 in which the program is recorded, the hard disk included in the storage unit 20 in FIG.

  In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in time series along the order, but is not necessarily performed in time series, either in parallel or individually. The process to be executed is also included.

DESCRIPTION OF SYMBOLS 1 ... Imaging device, 11 ... CPU, 12 ... ROM, 13 ... RAM, 14 ... Image processing part, 15 ... Bus, 16 ... Input / output interface, 17 ... Image pick-up unit, 18 ... operation unit, 19 ... display unit, 20 ... storage unit, 21 ... communication unit, 22 ... drive, 41 ... continuous-shot image acquisition unit, 42 ..Image composition unit, 51... Reduction unit, 61... Reference image generation unit, 62... Closest approximation image selection unit, 63... Complementary image selection unit, 64. 65... Composition unit, 71... First difference calculation unit, 72... Second difference calculation unit, 73... Cost function setting unit, 74. Display control unit

Claims (17)

Image acquisition means for acquiring a plurality of pieces of image data having substantially the same angle of view continuous in time;
Generating means for generating reference image data for image synthesis based on a plurality of image data acquired by the image acquisition means;
Selection means for selecting at least two or more image data from a plurality of image data acquired by the image acquisition means based on the reference image data generated by the generation means;
Boundary setting means for setting a boundary between two or more image data selected by the selection means;
A combining unit that combines two or more image data selected by the selection unit based on the boundary set by the boundary setting unit;
An image processing apparatus comprising: The generation means generates reference image data in which the influence of a moving object imaged on the plurality of image data is less than that of the plurality of image data.
The image processing apparatus according to claim 1. 3. The reference image data according to claim 1, wherein the generation unit generates the reference image data by calculating an average value or a median value of each pixel of the plurality of image data acquired by the image acquisition unit. The image processing apparatus described. Image acquisition means for acquiring a plurality of pieces of image data having substantially the same angle of view continuous in time;
Reference image acquisition means for acquiring reference image data for image composition;
Based on the reference image data acquired by the reference image acquisition means, a plurality of image data acquired by the image acquisition means that is at least two or more image data that can be determined to be similar to the image of the reference image data. Selecting means for selecting from the image data of
Boundary setting means for setting a boundary between two or more image data selected by the selection means;
A combining unit that combines two or more image data selected by the selection unit based on the boundary set by the boundary setting unit;
An image processing apparatus comprising: The selection means includes
5. The image processing apparatus according to claim 4, wherein at least two or more pieces of image data are selected such that an image resulting from the synthesis is most similar to an image of the reference image data. First dissimilarity calculating means for calculating the dissimilarity between the image data selected by the selecting means with respect to adjacent pixels in each image data;
Second dissimilarity calculating means for calculating the dissimilarity of the two or more image data selected by the selecting means with respect to the reference image data;
Function setting means for setting a function having the difference calculated by the first difference calculation means and the difference calculated by the second difference calculation means as variables;
Further comprising
The boundary setting unit sets a boundary for two or more selected image data so that the function set by the function setting unit has a minimum value.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. Image acquisition means for acquiring a plurality of pieces of image data having substantially the same angle of view continuous in time;
Reference image acquisition means for acquiring reference image data for image composition;
First selection means for selecting at least one image data pixel value is similar to the reference image data obtained by the reference image acquiring means, from a plurality of image data obtained by the image obtaining means,
A plurality of images acquired by the image acquisition unit, the image data different from the image data selected by the first selection unit, such that the resultant image becomes an image more similar to the image of the reference image data. Second selection means for selecting at least one from the data ;
Boundary setting means for setting a boundary between two or more image data selected by the first selection means and the second selection means;
Combining means for combining the two or more image data based on the boundary set by the boundary setting means;
The image processing apparatus comprising: a. First difference calculation means for calculating a difference between the image data selected by the first selection means and the second selection means for adjacent pixels in each image data;
Second dissimilarity calculating means for calculating the dissimilarity of the two or more image data selected by the first selecting means and the second selecting means with respect to the reference image data;
Function setting means for setting a function having the difference calculated by the first difference calculation means and the difference calculated by the second difference calculation means as variables;
Further comprising
The demarcation means such that said function function set by the setting means becomes a minimum value, the image processing apparatus according to claim 7, characterized in that to set the boundary to the two or more image data . The reference image acquisition means includes generation means for generating reference image data for image composition based on a plurality of image data acquired by the image acquisition means. Or the image processing apparatus of 8. It further includes imaging means for imaging a plurality of times at substantially the same angle of view continuously in time, and outputting image data obtained as a result of the plurality of times of imaging,
The image acquisition means acquires a plurality of image data output from the imaging means as a plurality of image data having substantially the same angle of view that is continuous in time.
The image processing apparatus according to any one of claims 1-9, characterized in that. Processing means for processing to reduce the amount of information of the plurality of image data;
The image processing apparatus according to any one of claims 1 to 10, further comprising a. An image acquisition step of acquiring a plurality of image data having substantially the same angle of view that is continuous in time;
A generating step for generating reference image data for image synthesis based on the plurality of image data acquired by the image acquiring step;
A selection step of selecting at least two or more image data from the plurality of image data acquired by the image acquisition step based on the reference image data generated by the generation step;
A boundary setting step for setting a boundary between two or more image data selected by the selection step;
A synthesis step of synthesizing two or more image data selected by the selection step based on the boundary set by the boundary setting step;
An image processing method comprising: An image acquisition step of acquiring a plurality of image data having substantially the same angle of view that is continuous in time;
A reference image acquisition step of acquiring reference image data for image synthesis;
Based on the reference image data acquired in the reference image acquisition step, at least two or more pieces of image data that can be determined to be similar to the image of the reference image data are acquired in the image acquisition step. A selection step of selecting from a plurality of image data;
A boundary setting step for setting a boundary between two or more image data selected in the selection step;
Based on the boundary set in the boundary setting step, a combining step of combining two or more image data selected in the selection step;
An image processing method comprising: An image acquisition step of acquiring a plurality of image data having substantially the same angle of view that is continuous in time;
A reference image acquisition step of acquiring reference image data for image synthesis;
A first selection step of selecting at least one image data similar in pixel value to the reference image data acquired by the reference image acquisition step from a plurality of image data acquired by the image acquisition step;
A plurality of images acquired by the image acquisition step, the image data different from the image data selected by the first selection step so that the image of the synthesis result is an image more similar to the image of the reference image data A second selection step of selecting at least one from the data;
A boundary setting step for setting a boundary between two or more image data selected by the first selection step and the second selection step;
A combining step of combining the two or more image data based on the boundary set by the boundary setting step;
An image processing method comprising: Computer
Image acquisition means for acquiring a plurality of pieces of image data having substantially the same angle of view continuous in time;
Generating means for generating reference image data for image synthesis based on a plurality of image data acquired by the image acquiring means;
Selection means for selecting at least two or more image data from a plurality of image data acquired by the image acquisition means based on the reference image data generated by the generation means;
Boundary setting means for setting a boundary between two or more image data selected by the selection means;
A combining means for combining the two or more image data based on the boundary set by the boundary setting means;
Program to function as. Computer
Image acquisition means for acquiring a plurality of pieces of image data having substantially the same angle of view continuous in time;
Reference image acquisition means for acquiring reference image data for image synthesis;
Based on the reference image data acquired by the reference image acquisition means, a plurality of image data acquired by the image acquisition means that is at least two or more image data that can be determined to be similar to the image of the reference image data. Selection means for selecting from the image data of
Boundary setting means for setting a boundary between two or more image data selected by the selection means;
A combining means for combining the two or more image data based on the boundary set by the boundary setting means;
Program to function as. Computer
Image acquisition means for acquiring a plurality of pieces of image data having substantially the same angle of view continuous in time;
Reference image acquisition means for acquiring reference image data for image synthesis;
First selection means for selecting, from a plurality of image data acquired by the image acquisition means, at least one image data having a pixel value similar to the reference image data acquired by the reference image acquisition means;
A plurality of images acquired by the image acquisition unit, the image data different from the image data selected by the first selection unit, such that the resultant image becomes an image more similar to the image of the reference image data. A second selection means for selecting at least one from the data;
Boundary setting means for setting a boundary between two or more image data selected by the first selection means and the second selection means;
A combining means for combining the two or more image data based on the boundary set by the boundary setting means;
Program to function as.