JP2006003982A - Image processor, image processing method, recording medium, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To connect two images, where the sampling point of a pixel is not coincident, without performing re-sampling processing. <P>SOLUTION: A graphic structure is set in the inputted images X, Y in a step S1. In a step S2, an average density in the node of the graphic structure is calculated so as to obtain the images A1, A2 in response to each value of the average density. In a step S3, an area which is superimposed on the image A2 in the graphic structure of the image A1, is defined as A3. A source/sink node is set in a step S4 and connected to each node which is positioned on the outer periphery of the area A3 in a step S5. In a step S6, a cost is calculated concerning each node and edge of the area A3 and an edge connected to the source/sink node. In a step S7, the problem of Min-cut and Max-flow, is solved based on the costs. The present invention is applied to image processing such as image retouch, panoramic image generation, and texture generation. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus and method, a recording medium, and a program, and more particularly, to an image processing apparatus and method, a recording medium, and a program that are suitable for use when a plurality of images are combined to generate a new image. .

  For example, in the case of generating a landscape when looking down from the top of the mountain as one image (so-called panoramic image), a panoramic image is generated by combining a plurality of images taken in different directions. It will be. In addition to panoramic images, a plurality of images are connected to generate a new image very frequently as general image processing.

  As shown in FIG. 1, when two images X and Y are combined, it is necessary to adjust the size of the subject in the images X and Y and adjust the direction of the subject. As shown, the image is subjected to processing such as enlargement or reduction, deformation, and rotation (hereinafter simply referred to as deformation processing). However, in the case of FIG. 2, only the image B is rotated and deformed with relatively regularity, but in an actual deformation process, it is often deformed into a more complicated shape. Here, it is assumed that the image subjected to the deformation process is Y ′.

  In this rotation / deformation process, a square lattice structure of sampling points indicated by black dots on the image Y in FIG. 1 is usually non-square before the process, as indicated by black dots on the image Y ′ in FIG. It will be transformed into a lattice structure. Note that the image X may be similarly deformed.

  By the way, in the conventional method for combining two images (for example, the Graphcut method; see Non-Patent Document 1), only when the positions of the lattice points (sampling points) in the overlapped region match. Two images can be combined. Therefore, in the state shown in FIG. 2, since the sampling points of the image Y ′ are non-square lattice structures, they cannot be combined, so that the image Y ′ overlaps with the sampling points of the image X as shown in FIG. The sampling points are converted to a square lattice structure (hereinafter, this conversion process is referred to as resampling process) to generate an image Y ”and solve the Min-cut Max-flow problem to determine the connection line. Thus, the images X and Y ″ are combined.

Kwatra et al. “Graphcut textures: image and video synthesis using graph cuts” Siggraph2003 proceedings p.277-286

  However, as described above, when the resampling process is performed on the image Y ′, there is a problem that the image quality of the generated image Y ″ is deteriorated.

  As a method for solving the problem of the image degradation of the image Y ″, there is a method of keeping the resolution of the original image Y as high as possible. In that case, a memory for storing the images Y, Y ′, Y ″, etc. There is a problem that it may be necessary to increase the capacity of the image or to prepare an image Y having a high resolution.

  The present invention has been made in view of such a situation, and an object of the present invention is to allow two images whose sampling points do not match to be combined without performing a resampling process.

  The image processing apparatus according to the present invention includes a graph structure setting unit for setting a graph structure in which pixels existing in an original image are nodes and connected by edges in an overlapping region of two images, and two images Calculation means for calculating a cost corresponding to each node and edge of the overlap area based on the pixel value of the pixel, and an acquisition means for acquiring a dividing line for dividing the overlap area based on the cost calculated by the calculation means, And a coupling means for coupling the first and second images with the dividing line as a boundary.

  The graph structure determining means is a graph structure setting means for setting a graph structure in which each pixel is a node and connected by an edge with respect to two images, and two images in which the graph structure is set. When the first image and the second image are arranged at a position where the first image and the second image are combined, the graph structure region provided in the first image And determining means for determining an area overlapping the second image as an overlapping area.

  The graph structure setting means can set a graph structure composed of nodes and edges, each of which is a combination of triangular regions for two images.

  The graph structure setting means obtains a triangulated alpha shape for each of two images, and can set a graph structure composed of nodes and edges, which is a combination of triangular regions. .

  The graph structure setting unit compares the two images with respect to the average density of the nodes, and designates one of the two images as the first image and the other as the second image based on the comparison result, It is possible to set a graph structure by connecting only image nodes with edges.

  The graph structure setting means can compare two images with respect to the average density of the nodes, and can designate the higher average density of the two images as the first image and the lower image as the second image. .

  The calculation means calculates, as the cost of the node in the overlapping region, the pixel value of the sampling point of the first image corresponding to the node, and the pixel value of the sampling point closest to the node among the sampling points of the second image The difference can be calculated.

  A Voronoi diagram composed of Voronoi polygons can be used to determine the sampling point closest to the node of the sampling points of the second image.

  The acquisition unit can acquire a dividing line that divides the overlapping region by solving the Min-cut Max-flow problem based on the cost calculated by the calculating unit.

  The image processing method of the present invention includes a graph structure setting step for setting a graph structure in which pixels existing in an original image are used as nodes in an overlapping region of two images and connected by edges, and two images A calculation step for calculating a cost corresponding to each node and edge of the overlapping region based on the pixel value of the pixel, and an acquisition step for acquiring a dividing line for dividing the overlapping region based on the cost calculated in the processing of the calculation step And a combining step of combining two images with the dividing line as a boundary.

  The program of the recording medium of the present invention includes a graph structure setting step for setting a graph structure in which pixels existing in an original image are nodes and connected by edges in an overlapping region of two images, Based on the pixel value of the sampling point of the pixel of the image, a calculation step for calculating a cost corresponding to each node and edge of the overlap region, and a dividing line for dividing the overlap region based on the cost calculated in the processing of the calculation step It includes an acquisition step of acquiring, and a combining step of combining two images with the dividing line as a boundary.

  The program of the present invention includes a graph structure setting step for setting a graph structure in which pixels existing in an original image are nodes and connected by edges in a region where two images overlap, and pixels of two images A calculation step for calculating a cost corresponding to each of the nodes and edges of the overlapping region based on the pixel value, and an acquisition step for acquiring a dividing line for dividing the overlapping region based on the cost calculated in the processing of the calculation step; A computer is caused to execute processing including a combining step of combining the first and second images with the dividing line as a boundary.

  In the present invention, a graph structure in which each pixel is a node and connected by an edge is set for each of the two images, and one of the two images with the graph structure set is the first image. When the other is designated as the second image and is arranged at a position where the first and second images are combined, the area of the graph structure provided in the first image, and the second image The overlapping area is determined as the overlapping area. Further, based on the pixel values of the sampling points of the first and second images, costs corresponding to the nodes and edges of the overlapping region are calculated, and a dividing line for dividing the overlapping region is acquired based on the calculated costs. The first and second images are combined with the dividing line as a boundary.

  According to the present invention, two images whose pixel sampling points do not match can be combined without performing resampling processing.

  Embodiments of the present invention will be described below. Correspondences between constituent elements described in the claims and specific examples in the embodiments of the present invention are exemplified as follows. This description is to confirm that specific examples supporting the invention described in the claims are described in the embodiments of the invention. Therefore, even if there are specific examples that are described in the embodiment of the invention but are not described here as corresponding to the configuration requirements, the specific examples are not included in the configuration. It does not mean that it does not correspond to a requirement. On the contrary, even if a specific example is described here as corresponding to a configuration requirement, this means that the specific example does not correspond to a configuration requirement other than the configuration requirement. not.

  Further, this description does not mean that all the inventions corresponding to the specific examples described in the embodiments of the invention are described in the claims. In other words, this description is an invention corresponding to the specific example described in the embodiment of the invention, and the existence of an invention not described in the claims of this application, that is, in the future, a divisional application will be made. It does not deny the existence of an invention that is added by correction.

The image processing apparatus according to claim 1 (for example, the image processing apparatus 1 in FIG. 4)
Graph structure setting means (for example, FIG. 4) sets a graph structure (for example, the graph structure shown in FIG. 5) in which the pixels existing in the original image in the overlapping area of the two images are nodes and are connected by edges. Graph structure setting unit 2),
Designating means for designating one of the two images set with the graph structure as a first image (for example, image A1 in FIG. 6) and the other as a second image (for example, image A2 in FIG. 6). For example, the graph structure setting unit 3 in FIG. 4 for executing step S2 in FIG.
When the first image and the second image are arranged at a position where the first image and the second image are combined, an area of the graph structure provided in the first image and an area overlapping the second image is an overlapping area (for example, A determination unit (for example, the graph structure setting unit 3 in FIG. 4 that executes step S3 in FIG. 8) to determine the area A3 in FIG. 6;
Calculation means (for example, the cost calculation unit 5 in FIG. 4) for calculating costs corresponding to the nodes and edges of the overlapping region based on the pixel values of the sampling points of the first and second images,
Based on the cost calculated by the calculation means, an acquisition means (for example, a combined line determination unit in FIG. 4) for acquiring a dividing line (for example, a combined line) that divides the overlapping area;
And a coupling means (for example, a coupling unit 7 in FIG. 4) for coupling the first and second images with the dividing line as a boundary.

The image processing method according to claim 9 comprises:
A graph structure setting step (for example, step S1 in FIG. 8) for setting a graph structure in which pixels existing in the original image in the overlapping region of the two images are nodes and connected by edges;
A designation step (for example, step S2 in FIG. 8) of designating one of the two images set with the graph structure as a first image and the other as a second image;
When the first image and the second image are arranged at a position where the first image and the second image are combined, a region having a graph structure provided in the first image and overlapping with the second image is determined as an overlapping region. A determination step (eg, step S3 in FIG. 8);
A calculation step (for example, step S6 in FIG. 8) for calculating costs corresponding to the nodes and edges of the overlapping region based on the pixel values of the sampling points of the first and second images,
An acquisition step (for example, step S7 in FIG. 8) for acquiring a dividing line that divides the overlapping region based on the cost calculated in the processing of the calculating step;
A combination step (for example, step S8 in FIG. 8) for combining the first and second images with the dividing line as a boundary.

  Note that the correspondence relationship between the program recorded in the recording medium of the present invention and the constituent elements described in the claims of the program of the present invention and the specific examples in the embodiment of the present invention is the image processing of the present invention described above. Since it is the same as that of the method, its description is omitted.

  Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings.

  FIG. 4 shows a configuration example of an image processing apparatus according to an embodiment of the present invention. The image processing apparatus 1 generates an image Z by combining two input images X and Y, and includes a graph structure setting unit 2, a point density determination unit 3, a source / sink node setting unit 4, The cost calculation unit 5, the combination line determination unit 6, and the combination unit 7 are included.

  It is assumed that the images X and Y input to the image processing apparatus 1 have been subjected to deformation processing, and sampling points provided on the image do not have a square lattice structure. Hereinafter, an image in which sampling points do not have a square lattice structure is referred to as ISI (Irregularly Sampled Image). Of course, both or one of the images X and Y may not be ISI.

  The graph structure setting unit 2 regards each of the input images X and Y as a triangular region having nodes as vertices by regarding the sampling points on the image as nodes and connecting the nodes with edges. By dividing, a graph structure (a structure in which nodes are connected by edges) is set and output to the point density determination unit 3.

  As a method for dividing each of the images X and Y into triangular regions, for example, an Alpha Shape of a point set (in this case, a set of sampling points) obtained by dividing a Delaunay triangle is applied. Alpha Shape is a mathematical concept that represents an area that includes all the points that make up a point set. It can be changed into various forms by changing the input value alpha, but it is a term used as a concept that represents the shape of a point set. . In general, the point set can be obtained from the Delaunay triangulation structure, and the required "A set of point Alpha shape obtained by Delaunay triangulation" is obtained when the Alpha Shape is obtained. Become.

  Note that the graph structure setting unit 2 may divide the images X and Y into triangular regions by other methods instead of Delaunay triangulation. However, the use of Delaunay triangulation has the advantage that the resulting triangle shapes are relatively uniform. Alternatively, the graph structure may be set by dividing the images X and Y into regions having shapes other than triangles. However, dividing into regions of shapes other than triangles may be impossible depending on the arrangement of points, but it is always possible to divide into regions of triangles regardless of the arrangement of points. There is.

  The point density determination unit 3 calculates the average density of the nodes (sampling points) of the images X and Y for which the graph structure is set, and determines the larger image as A1 and the smaller image as A2. To do. Further, as shown in FIG. 5, the point density determination unit 3 determines the positional relationship between the images A1 and A2 based on the setting from the user, and then overlaps the image A2 in the graph structure of the image A1. The region is specified, the region is determined as A3, and the images A1 and A2 and the region A3 are output to the source / sink node setting unit 4. Of course, this region A3 also has a graph structure. At this time, the positional relationship between the images A1 and A2 is arbitrary by the user, and each sampling point has a non-square lattice structure. Therefore, the sampling points of the image A1 and the sampling points of the image A2 do not have to overlap.

  As shown in FIG. 6, the source / sink node setting unit 4 sets a source (Source) node that represents an area not included in the area A <b> 3 among the areas of the image A <b> 1. In addition, a sink node representing an area not included in the area A3 in the area of the image A2 is set. Further, the source / sink node setting unit 4 connects each node located on the outer periphery of the area A3 to the node on the image A1 other than the area A3 and connects to the source node, and the image A1 other than the area A3. Those not connected to the upper node are connected to the sink node.

  The cost calculation unit 5 calculates the cost of each node and each edge of the area A3 and the edge connected to the source / sink node. As the cost of the node in the area A3, the distance from the pixel value of the node (that is, the sampling point of the image A1) and the sampling point of the image A2 (not shown in FIGS. 5 and 6) to the node The difference from the pixel value of those close to is calculated.

Specifically, the difference in pixel value is, for example, an addition value of the square of the difference for each RGB color ((R ₁ −R ₂ ) ² + (G ₁ −G ₂ ) ² + (B ₁ −B) ₂ ) Adopt ² ). Here, R ₁ , G ₁ , B ₁ are pixel values at the sampling points of the image A 1, and R ₂ , G ₂ , B ₂ are pixel values at the sampling points of the image A 2.

  In addition, as a method of detecting a node closest to the node of the region A3 (that is, the sampling point of the image A1) and the sampling point of the image A2 (hereinafter referred to as the nearest point search method) For example, there is a method using a Voronoi diagram obtained by dividing the image A2 into Voronoi polygons as shown in FIG. This Voronoi diagram is obtained by dividing the region of the image A2 so that the nearest sampling points are common.

  Therefore, if it is determined to which of the Voronoi polygons constituting the Voronoi diagram the node in the region A3 belongs, the sampling point of the image A2 that is closest to the node in the region A3 is detected. be able to.

  Note that a method other than a method using a Voronoi diagram may be applied to the nearest point search method. However, in the graph structure setting unit 2, an algorithm for obtaining an alpha shape often performs Delaunay triangulation in the process. Therefore, if a method using a Voronoi diagram is used, a Voronoi diagram can be easily obtained using this result. There is an advantage that can be.

  The cost of the edge of the area A3 is calculated by adding the costs of the nodes at both ends of the edge. An infinite value is set as the cost of the edge connected to the source / sink node.

  The combined line determination unit 6 solves the Min-cut Max-flow problem on the basis of the cost of each node and each edge of the region A3 calculated by the cost calculation unit 5 and the edge connected to the source / sink node. An edge set (Min-cut that gives Max-flow) obtains a connection line that classifies all nodes connected to the source (source set) and one connected to the sink (sink set).

  For the region A3, the combining unit 7 generates the image Z by using the image A1 on the source node side of the determined combined line and the image A2 on the sink node side of the determined combined line. To do.

  Next, image combining processing by the image processing apparatus 1 will be described with reference to the flowchart of FIG.

  In step S1, the graph structure setting unit 2 divides each of the input images X and Y into triangular regions having the nodes as vertices by regarding the sampling points on the images as nodes and connecting the nodes with edges. Thus, the graph structure is set and output to the point density determination unit 3.

  In step S2, the point density determination unit 3 calculates the average density of each node of the images X and Y in which the graph structure is set, and determines the larger image as A1 and the smaller image as A2. To do. In step S3, the point density determination unit 3 determines the positional relationship between the images A1 and A2 based on the setting from the user, and then identifies a region overlapping the image A2 in the graph structure of the image A1. The region is determined as A3, and the images A1 and A2 and the region A3 are output to the source / sink node setting unit 4.

  In step S4, the source / sink node setting unit 4 sets a source node representing an area not included in the area A3 in the area of the image A1. In addition, a sync node representing an area not included in the area A3 in the area of the image A2 is set. In step S5, the source / sink node setting unit 4 connects the nodes connected to the nodes on the image A1 other than the region A3 to the source nodes for each node located on the outer periphery of the region A3. Those not connected to the node on the image A1 are connected to the sink node.

  In step S6, the cost calculation unit 5 calculates the cost of each node and each edge of the region A3 and the edge connected to the source / sink node. Specifically, the cost of the node in the area A3 includes the pixel value of the node (that is, the sampling point of the image A1) and the sampling point of the image A2 (not shown in FIGS. 5 and 6). The difference from the pixel value closest to the node is calculated. The cost of the edge of the area A3 is calculated by adding the costs of the nodes at both ends of the edge. An infinite value is set as the cost of the edge connected to the source / sink node.

  In step S7, the combined line determination unit 6 solves the Min-cut Max-flow problem based on the cost of each node and each edge of the area A3 calculated by the cost calculation unit 5 and the edge connected to the source / sink node. By solving, we obtain a connection line that classifies the edge set (Min-cut giving Max-flow) into one that connects all nodes to the source (source set) and one that connects to the sink (sink set). .

  In step S8, for the region A3, the combining unit 7 adopts the image A1 for the source node side with respect to the determined combined line, and the image A2 for the sink node side with respect to the determined combined line. An image Z is generated.

  The image Z generated in this way has not been subjected to the resampling process, that is, the images A and B whose image quality is not deteriorated are combined at the connection line, so that the joint is most noticeable. It can be expected that there is no state. This is the end of the description of the image combining process performed by the image processing apparatus 1.

  The image processing apparatus 1 according to the present embodiment can be applied to image processing such as image retouching, panoramic image generation, texture generation, and the like.

  By the way, the series of processes described above can be executed by hardware, but can also be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, a general-purpose personal computer configured as shown in FIG. 9 is installed from the recording medium.

  The personal computer 20 includes a CPU (Central Processing Unit) 21. An input / output interface 25 is connected to the CPU 21 via the bus 24. A ROM (Read Only Memory) 22 and a RAM (Random Access Memory) 23 are connected to the bus 24.

  The input / output interface 25 includes an input unit 26 including an input device such as a keyboard and a mouse for a user to input operation commands, a CRT (Cathode Ray Tube) or an LCD (Liquid Crystal Display) for displaying an image such as a combined processing result. Communication unit via a network typified by the Internet, which includes an output unit 27 such as a display, a storage unit 28 such as a hard disk drive for storing programs and various data, a modem, a LAN (Local Area Network) adapter, etc. Is connected to the communication unit 29. Also, a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory), a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor A drive 30 for reading / writing data from / to a recording medium 31 such as a memory is connected.

  A program for causing the personal computer 20 to execute the above-described series of processing is supplied to the personal computer 20 while being stored in the recording medium 31, read by the drive 30, and installed in a hard disk drive built in the storage unit 28. Has been. The program installed in the storage unit 28 is loaded from the storage unit 28 to the RAM 23 and executed by a command of the CPU 21 corresponding to a command from the user input to the input unit 26.

  In this specification, the steps executed based on the program are executed in parallel or individually even if they are not necessarily processed in time series, as well as processes executed in time series according to the described order. It also includes processing.

  The program may be processed by a single computer, or may be distributedly processed by a plurality of computers. Furthermore, the program may be transferred to a remote computer and executed.

  In this specification, the system represents the entire apparatus constituted by a plurality of apparatuses.

It is a figure for demonstrating the conventional method of connecting two images. It is a figure for demonstrating the conventional method of connecting two images. It is a figure for demonstrating the conventional method of connecting two images. It is a block diagram which shows the structural example of the image processing apparatus to which this invention is applied. It is a figure which shows the relationship between image A1, A2 and area | region A3. It is a figure which shows a source / sink node. It is a figure which shows an example of a Voronoi diagram. It is a flowchart explaining an image combination process. And FIG. 16 is a block diagram illustrating a configuration example of a personal computer.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Image processing apparatus, 2 Graph structure setting part, 3 Point density determination part, 4 Source / sink node setting part, 5 Cost setting part, 6 Connection line determination part, 7 Connection part, 20 Personal computer, 21 CPU, 31 Recording medium

Claims (11)

In an image processing apparatus that combines two images whose pixel sampling points do not match,
In the overlapping region of the two images, a graph structure setting means for setting a graph structure in which pixels existing in the original image are nodes and connected by edges;
Calculation means for calculating costs corresponding to the nodes and edges of the overlapping region based on pixel values of pixels of the two images,
Obtaining means for obtaining a dividing line for dividing the overlapping area based on the cost calculated by the calculating means;
An image processing apparatus comprising: a combining unit that combines the two images with the dividing line as a boundary. The image processing apparatus according to claim 1, wherein the graph structure setting unit sets a graph structure that is a combination of triangular regions and includes nodes and edges for each of the two images. The graph structure setting means obtains an alpha shape obtained by triangulating each of the two images, and sets a graph structure that is a combination of triangular regions and includes nodes and edges. The image processing apparatus according to claim 2. The graph structure setting means compares the two images with respect to the average density of the nodes, and designates one of the two images as a first image and the other as a second image based on the comparison result, The image processing apparatus according to claim 1, wherein only the nodes of the first image are connected by edges to set a graph structure. The graph structure setting unit compares the two images with respect to the average density of the nodes, and the higher average density of the two images is the first image, and the lower image is the second image. The image processing apparatus according to claim 4, wherein the image processing apparatus is specified. The calculation means calculates, as the cost of the node of the overlap region, the pixel value of the pixel of the first image corresponding to the node and the pixel of the pixel closest to the node among the pixels of the second image The image processing apparatus according to claim 4, wherein a difference from the value is calculated. The image processing apparatus according to claim 6, wherein a Voronoi diagram composed of Voronoi polygons is used to determine the pixel closest to the node among the pixels of the second image. The acquisition unit acquires the dividing line that divides the overlapping region by solving a Min-cut Max-flow problem based on the cost calculated by the calculating unit. The image processing apparatus described. In an image processing method for combining two images whose pixel sampling points do not match,
A graph structure setting step for setting a graph structure in which pixels existing in the original image are nodes and connected by edges in the overlapping region of the two images; and
A calculation step of calculating costs corresponding to nodes and edges of the overlapping region based on pixel values of pixels of the two images,
Based on the cost calculated in the processing of the calculation step, an acquisition step of acquiring a dividing line that divides the overlapping area;
An image processing method comprising: combining the two images with the dividing line as a boundary. A program for combining two images whose pixel sampling points do not match,
A graph structure setting step for setting a graph structure in which pixels existing in the original image are nodes and connected by edges in the overlapping region of the two images; and
A calculation step of calculating costs corresponding to nodes and edges of the overlapping region based on pixel values of pixels of the two images,
Based on the cost calculated in the processing of the calculation step, an acquisition step of acquiring a dividing line that divides the overlapping area;
And a combining step of combining the two images with the dividing line as a boundary. A recording medium on which a computer-readable program is recorded. A program for combining two images whose pixel sampling points do not match,
A graph structure setting step for setting a graph structure in which pixels existing in the original image are nodes and connected by edges in the overlapping region of the two images; and
A calculation step of calculating costs corresponding to nodes and edges of the overlapping region based on pixel values of pixels of the two images,
Based on the cost calculated in the processing of the calculation step, an acquisition step of acquiring a dividing line that divides the overlapping area;
A program that causes a computer to execute processing including a combining step of combining the two images with the dividing line as a boundary.

Images