JP2006302195A - Image processing method and image processing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing method and an image processor for highly precisely separating an object in an image. <P>SOLUTION: This image processing method for separating an object in a pickup image comprises an imaging step for acquiring a plurality of pickup images whose visual points are different, a three-dimensional point processing step for searching a three-dimensional shape (three-dimensional point group) as the object of image pickup based on a plurality of pickup images, a grouping step for grouping a three-dimensional group, a projection step for projecting the grouped three-dimensional point group on the reference image of the pickup image, an excessive division step for excessively dividing the reference image into regions, a coupling degree processing step for searching the number of three-dimensional points of each group in the adjacent excessive division regions, and for searching the degree of coupling between the adjacent excessive division regions based on the number of three-dimensional points of each group and a separation step for separating the object by coupling or separating the adjacent excessive division regions based on the degree of coupling. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image processing method and an image processing apparatus that divide a region in order to individually separate a plurality of objects in an image.

Various image processing methods for separating a plurality of objects in a captured image have been developed. As this processing method, there is a method of separating an object by acquiring edge information, color information, texture information, and the like from a captured image and dividing them into similar regions using these information (Patent Document 1). , See Patent Document 2).
JP 2003-87545 A JP 2004-178273 A

  However, even for the same object, edge information, color information, texture information, and the like differ depending on the pattern, color, or shadow of the surface. In such a case, even the same object is divided into different regions and cannot be accurately separated into individual objects.

  Therefore, an object of the present invention is to provide an image processing method and an image processing apparatus capable of separating an object in an image with high accuracy.

  An image processing method according to the present invention is an image processing method for separating an object in a captured image, an imaging step for acquiring a plurality of captured images from different viewpoints, and a three-dimensional image to be captured based on the plurality of captured images A three-dimensional point processing step for obtaining a three-dimensional point indicating a shape, a grouping step for grouping the three-dimensional points obtained in the three-dimensional point processing step, and a plurality of captured images of the three-dimensional points grouped in the grouping step A projection step for projecting the image on the reference image, an over-division step for over-dividing the reference image of the plurality of captured images into regions, and a projection step for each over-divided region and over-division performed in the over-division step Based on the three-dimensional points of each group, the number of three-dimensional points for each group in each adjacent hyperdivision region is obtained, and the number of three-dimensional points adjacent to each group is determined. A coupling degree processing step for obtaining a coupling degree between regions, and a separation step for separating an object by coupling or separating adjacent over-divided regions based on the coupling degree obtained in the coupling degree processing step. And

  In this image processing method, a plurality of captured images having different viewpoints are acquired in the imaging step. In the image processing method, the three-dimensional shape (three-dimensional point group) of the imaging target is restored based on a plurality of captured images having different viewpoints in the three-dimensional point processing step, and the three-dimensional point group is restored in the grouping step. Group. Further, in the image processing method, the grouped three-dimensional point group is projected onto a reference image among a plurality of captured images. Thereby, a two-dimensional position on the reference image is obtained for each grouped three-dimensional point. In the image processing method, the reference image is overdivided into each region in the overdivision step. In the area division, for example, edge information, color information, texture information, and the like are acquired from the captured image, and divided into similar areas using these pieces of information. Since one region is divided into a plurality of regions by this region division, it is called overdivision. In the image processing method, each overdivided region in each of the overdivided regions in the adjacent overdivided region from each overdivided region projected from the grouped three-dimensional point projected on the reference image and the overdivided region overdivided the reference image in the combination degree processing step. 3D points are obtained. Furthermore, in the image processing method, based on the number of three-dimensional points for each group of each overdivided region in the adjacent overdivided region in the combination degree processing step, the degree of coupling between the adjacent overdivided regions (that is, its adjacent It is a measure of whether or not the over-divided regions may be combined as a region of the same object, and is also a measure of whether or not they may be separated as boundaries of different objects). In other words, if there are many three-dimensional points in the same group between adjacent over-divided regions, it is likely that they are the same object region, while the number of three-dimensional points in the same group between adjacent over-divided regions If there is a small number of regions, there is a high possibility that they are regions of different objects, and the degree of coupling is determined according to the number of three-dimensional points for each group that occupy between such over-divided regions. Furthermore, in the image processing method, each of the overdivided regions is combined or separated based on the degree of connection in the separation step (that is, the overdivided regions that may be regarded as the same object region are combined and regarded as the boundary of different objects). Good over-divided areas are separated) and separated into individual objects by this combined or adjacent over-divided area. As described above, in this image processing method, the degree of coupling is defined based on the number of three-dimensional points (projection points) for each group between adjacent overdivided areas, and the adjacent overdivided areas are determined according to the degree of coupling. By determining whether or not they are the same object region, it is possible to accurately combine the overdivided regions for each object, accurately separate different objects, and separate each object with high accuracy. Furthermore, in this image processing method, when obtaining the degree of coupling, by removing a small number of three-dimensional points for each group, it is possible to separate objects with high accuracy without being affected by noise.

  An image processing apparatus according to the present invention is an image processing apparatus that separates an object from a captured image, and includes a plurality of imaging units that acquire a plurality of captured images from different viewpoints, and a tertiary of an imaging target based on the plurality of captured images. Three-dimensional point processing means for obtaining a three-dimensional point indicating the original shape, grouping means for grouping the three-dimensional points obtained by the three-dimensional point processing means, and imaging of the three-dimensional points grouped by the grouping means Projecting means for projecting on the reference image of the image, over-dividing means for over-dividing the reference image of the plurality of captured images into regions, and projecting with each over-divided region and the projection means over-divided by the over-dividing means Based on the three-dimensional points of each group, the number of three-dimensional points for each group in each adjacent overdivision region is obtained, and the degree of coupling between adjacent overdivision regions is obtained from the number of three-dimensional points in each group. Degree of coupling And management means, characterized in that it comprises separating means for separating an object by binding or separation between over-divided regions adjacent on the basis of the degree of coupling obtained by the coupling of the processing means. This image processing apparatus has the same effects as the image processing method described above.

  According to the present invention, by defining the degree of coupling based on the number of grouped three-dimensional points in adjacent over-divided regions, individual objects in the captured image can be separated with high accuracy.

  Embodiments of an image processing method and an image processing apparatus according to the present invention will be described below with reference to the drawings.

  In the present embodiment, the present invention is applied to an image processing apparatus that individually separates objects in a captured image. An image processing apparatus according to the present invention includes a stereo camera, images a predetermined target with the stereo camera, and separates a plurality of objects in the captured image into individual objects in regions corresponding to the objects. To divide.

  An image processing apparatus 1 according to the present embodiment will be described with reference to FIG. FIG. 1 is a configuration diagram of an image processing apparatus according to the present embodiment.

  The image processing apparatus 1 captures an image with a stereo camera, and over-divides a reference image (for example, a left captured image) in the left and right captured images into regions. Then, the image processing apparatus 1 combines / separates adjacent excessively divided areas and separates them into individual objects. In particular, in the image processing apparatus 1, in order to separate individual objects with high accuracy, three-dimensional point groups representing imaging targets in the captured image are grouped, and the three-dimensional points for each group in each adjacent overdivided region are grouped. The degree of coupling based on the number is calculated, and it is determined whether the over-divided areas are to be coupled or separated based on the degree of coupling. The image processing apparatus 1 includes two cameras 2 and 3 constituting a stereo camera, a monitor 4 and an image processing unit 5.

  In this embodiment, the cameras 2 and 3 correspond to a plurality of imaging units described in the claims, and each process performed in the image processing unit 5 is a three-dimensional point processing unit described in the claims. It corresponds to grouping means, projection means, over-division means, coupling degree processing means, and separation means.

  The cameras 2 and 3 are color stereo cameras that are arranged on an imaging target in parallel with a predetermined interval and capture the imaging target. The image captured by the camera 2 is a left captured image (referred to as a reference image), the image captured by the camera 3 is a right captured image (referred to as a reference image), and two captured images from different viewpoints are simultaneously displayed. can get. The cameras 2 and 3 are digital cameras including an image sensor such as a CCD [Charge coupled device], and transmit a captured image formed of digital image data to the image processing unit 5 as an image signal. Information about the positional relationship of the cameras 2 and 3 such as the interval, internal parameters and external parameters of the cameras 2 and 3 are stored in the image processing unit 5 in advance. FIG. 2 shows an example of left and right captured images, in which three objects, a bin, a rectangular parallelepiped box, and a bell pepper, are placed on a handkerchief having a pattern. In FIG. 2, a picture or a character is not drawn on each object, but actually, each object has a picture or a character, and a shadow is also present.

  The monitor 4 may be a liquid crystal monitor or a CRT [Cathode Ray Tube] monitor, and displays and outputs the processing result of the image processing unit 5. The monitor 4 displays an image or the like according to the output signal from the image processing unit 5.

  The image processing unit 5 is configured on a computer such as a personal computer, and performs each process by executing a dedicated application program. The image processing unit 5 takes in left and right captured images (stereo images) from the cameras 2 and 3, respectively. Then, the image processing unit 5 restores the three-dimensional shape (point cloud) based on the left and right captured images (three-dimensional shape restoration processing), groups the three-dimensional point cloud (grouping processing), and performs the grouping. The obtained three-dimensional point is projected onto the reference image (projection process). Further, the image processing unit 5 overdivides the reference image into regions (region overdivision processing). Then, the image processing unit 5 calculates the degree of coupling with respect to the over-divided area based on the number of three-dimensional points (projection points) for each group of adjacent over-divided areas (joint degree calculation process). Finally, the image processing unit 5 combines or separates adjacent over-divided regions based on the degree of combination, and separates them into individual objects (object separation processing).

  The three-dimensional shape restoration process will be described. The image processing unit 5 obtains a three-dimensional point group indicating the shape of the three-dimensional object based on the left and right captured images. In this three-dimensional shape restoration, for example, three-dimensional coordinate point data may be acquired by edge restoration or texture restoration, or a correlation calculation is performed between the left and right captured images, and the parallax obtained from the calculation result The depth distance to the object may be calculated using, and 3D coordinate point data may be acquired from the depth distance. FIG. 3 shows a three-dimensional point group indicating the three-dimensional shape of each object in the left captured image of FIG.

  The grouping process will be described. The image processing unit 5 groups the three-dimensional point group by a conventional segmentation technique, and assigns a group number to each three-dimensional point. By this grouping, three-dimensional points with the same group number are collected in a space corresponding to each object. Further, due to segmentation noise, three-dimensional points of different groups are scattered in a group of three-dimensional points having the same group number. FIG. 4 shows a point cloud obtained by grouping the three-dimensional point cloud of FIG. 3 and is expressed in a different color for each group. As can be seen from FIG. 4, dots of the same color are collected for each object, and dots of different colors are scattered in the collection of the same color.

  The projection process will be described. The image processing unit 5 projects the three-dimensional points grouped on the reference image. Each projection point is assigned a group number, and its position is defined by the position on the two-dimensional coordinate of the reference image. FIG. 5 shows an image obtained by projecting the grouped three-dimensional points in FIG. 4 onto the left captured image in FIG. 2, and is represented by a different color for each group. As can be seen from FIG. 5, three-dimensional points of the same color are distributed over the entire area of each object, and three-dimensional points of different colors are scattered therein.

  The area overdivision process will be described. The image processing unit 5 overdivides the reference image into regions using image information such as edge information, color information, and texture information of the reference image. Since excessive region division is performed, the region where each object exists in the captured image is divided into a number of regions. The method for dividing the region is not particularly limited, and for example, division can be performed at high speed by using the Mean-shift method. When performing region division by various methods, the region is excessively divided by setting a high threshold (that is, by making the division condition stricter). FIG. 6 shows an overdivided region image obtained by overdividing the left captured image of FIG. 2 into regions, and each object is divided into a number of regions.

  The coupling degree calculation process will be described. The image processing unit 5 sequentially selects two adjacent excessively divided areas i and j from all the excessively divided areas. A grouped three-dimensional point (projection point) having the two-dimensional coordinates of the reference image is arranged on each of the adjacent excessively divided regions i and j (see FIG. 7). Then, the image processing unit 5 counts the number of appearances of projection points for each group (that is, for each group number) for each of the excessively divided areas i and j. Here, when the adjacent over-divided areas i and j are the same object area, the two over-divided areas i and over-divided areas j both occupy the majority of projection points with the same group number. On the other hand, when the adjacent excessively divided regions i and j are different object regions, the projection points having different group numbers occupy the majority in the two excessively divided regions i and the excessively divided regions j. In the example shown in FIG. 7, the majority of the projection points (white circles) having the same group number occupy both the overdivided region i and the overdivided region j. Further, the over-divided area i and the over-divided area j are dotted with projection points (black circles) of group numbers other than the group numbers that occupy the majority due to the influence of noise.

  Further, the image processing unit 5 obtains areas Ai and Aj for the respective excessively divided regions i and j. Further, the image processing unit 5 extracts the maximum values Ni and Nj from the number of appearances of the projection points of each group for each of the excessively divided areas i and j, and the maximum value Ni. Recognize the group numbers Gi and Gj of Nj.

  Then, the image processing unit 5 uses the areas Ai and Aj of the adjacent excessively divided regions i and j, the maximum number of appearances Ni and Nj, and the group number Gi and Gj with the maximum number of appearances to obtain the formula (1). To calculate rij between adjacent over-divided regions i and j. rij is always a negative value when the group number Gi and the group number Gj are the same number, and is always a positive value when the group number Gi and the group number Gj are different numbers. Further, in order to express the degree of coupling with a value of 0 to 1, the image processing unit 5 uses rij for the adjacent overdivided areas i and j, and uses the expression (2) between the overdivided areas i and j. The degree of coupling uij is calculated. The degree of coupling uij is determined by calculating the ratio of the number of projection points (three-dimensional points) of the group of the maximum number of appearances of each overdivided region i, j and the area of the two overdivided regions i, j to one object. This indicates the degree of whether or not the regions can be combined (in other words, the degree of whether or not the two over-divided regions i and j can be separated as boundaries between different objects), and is a value of 0 to 1. The closer the coupling degree u is to 1, the higher the probability that the adjacent over-divided region is the same object, and the closer the coupling degree u is to 0, the higher the probability that the adjacent over-divided region is a different object.

  In the expression (2), max (rij) is the maximum value of rij between the overdivision regions i and j of all adjacent combinations, and min (rij) is the overdivision region i, j is the minimum value of rij, and ρ is an adjustment parameter (threshold). However, uij = uji = 0 is set between over-divided areas where the degree of coupling uij does not exist.

  The object separation process will be described. The image processing unit 5 combines / separates adjacent over-divided areas for all over-divided areas based on the degree of coupling. In other words, adjacent over-divided areas having a high degree of coupling are determined to be the same object, and adjacent over-divided areas having a low degree of coupling are determined to be different objects and separated. The image processing unit 5 generates a region composed of a plurality of adjacent overdivided regions by combining / separating the adjacent overdivided regions, and separates the generated regions into individual objects. FIG. 8 shows a region-divided image showing the result of separating the object in the captured image of FIG. 2. The region A1 corresponds to a bin, the region A2 corresponds to a rectangular box, and the region A3 corresponds to a bell pepper. Each is divided.

  In the image processing unit 5, an output result such as an image showing the individually separated objects or information on the individual objects is set as an output signal, and the output signal is transmitted to the monitor 4. If necessary, the image processing unit 5 uses a three-dimensional point group generated in the process, a grouped three-dimensional point group, a projection image on which the grouped three-dimensional point group is projected, an over-divided region image. Are set as output signals, and the output signals are transmitted to the monitor 4.

  The operation of the image processing apparatus 1 will be described with reference to FIG. In particular, the overall process flow in the image processing unit 5 will be described with reference to the flowchart of FIG. 9, and the flow of the coupling degree calculation process will be described with reference to the flowchart of FIG. 10. FIG. 9 is a flowchart showing the main processing according to the present embodiment. FIG. 10 is a flowchart showing the degree-of-association calculation processing according to the present embodiment.

  Each of the cameras 2 and 3 captures an imaging target and transmits each captured image to the image processing unit 5 as an image signal.

  The image processing unit 5 receives each image signal from the cameras 2 and 3 and takes in left and right captured images (stereo images) (S1 in FIG. 9).

  The image processing unit 5 restores the three-dimensional shape based on the left and right captured images, and acquires a three-dimensional point group (S2 in FIG. 9). Then, the image processing unit 5 groups the three-dimensional point group (S3 in FIG. 9). Further, the image processing unit 5 projects the three-dimensional points grouped on the reference image (S4 in FIG. 9). Here, three-dimensional points are grouped for each object in the captured image, and the positions of the grouped three-dimensional points on the two-dimensional coordinates of the reference image are determined.

  Further, the image processing unit 5 overdivides the reference image into regions (S5 in FIG. 9).

  Then, the image processing unit 5 calculates the degree of coupling between the adjacent excessively divided areas i and j in all the excessively divided areas (S6 in FIG. 9). At this time, the image processing unit 5 counts the number of appearances of each group number of the projection points belonging to each of the excessively divided areas i and j (S10 in FIG. 10). Further, the image processing unit 5 obtains areas Ai and Aj of the respective excessively divided regions i and j (S11 in FIG. 10). The image processing unit 5 extracts the maximum values Ni and Nj of the number of appearances and the group numbers Gi and Gj of the maximum values Ni and Nj from the number of appearances of each overdivided region i and j for each group (S12 in FIG. 10). ). Further, the image processing unit 5 calculates rij by the equation (1) using the areas Ai, Aj, the maximum number of appearances Ni, Nj, and the group numbers Gi, Gj in the adjacent excessively divided regions i, j ( S13 in FIG. Furthermore, the image processing unit 5 calculates the degree of coupling uij between the adjacent excessively divided regions i and j by using the expression r2 using the rij (S14 in FIG. 10). Here, the degree of coupling between the adjacent over-divided areas i and j of all combinations in all the over-divided area groups is obtained.

  When the degree of coupling between all adjacent overdivided areas is calculated, the image processing unit 5 combines or separates the adjacent overdivided areas with respect to all the overdivided areas based on the degree of coupling. Then, the image processing unit 5 generates a region composed of a plurality of excessively divided regions and separates them into individual objects (S7 in FIG. 9). Here, the combined states of all the excessively divided regions are determined, and the objects are separated into individual objects by the region including the overdivided region group in each combined state.

  The image processing unit 5 generates an output signal indicating an object separation result (image or the like) and transmits the output signal to the monitor 4. The monitor 4 displays a separation result corresponding to the received output signal.

  According to the image processing apparatus 1, by defining the degree of coupling by focusing on the number of three-dimensional points for each group in adjacent overdivided areas, it is possible to accurately determine whether adjacent overdivided areas are different or the same object. Can be judged. Therefore, the area corresponding to each object can be accurately divided, and individual objects can be separated with high accuracy. In particular, the image processing apparatus 1 uses only the three-dimensional point of the group having the largest number of appearances among the three-dimensional points included in the overdivided region when obtaining the degree of coupling. Separation with high accuracy is possible.

  As mentioned above, although embodiment which concerns on this invention was described, this invention is implemented in various forms, without being limited to the said embodiment.

  For example, in the present embodiment, each process is performed by executing an application program (software) on a computer, but each processing unit may be configured by hardware.

  In this embodiment, a plurality of imaging units are configured by a stereo camera using two cameras, but a plurality of imaging units may be configured by three or more cameras.

  In the present embodiment, the grouped three-dimensional point group is projected on the reference image. However, the grouped three-dimensional point group is projected on the overdivided region image obtained by overdividing the reference image. May be. Incidentally, since the reference image and the over-divided area image are the same two-dimensional coordinate system, each projection point has the same coordinates regardless of whether it is projected on the reference image or the over-divided area image. It will be.

  Further, in this embodiment, the degree of coupling between adjacent overdivided areas is obtained for all overdivided areas and separated into individual objects based on the degree of coupling, but the object to be separated from all overdivided areas It is also possible to extract overdivided regions that are likely to exist, obtain the degree of coupling between adjacent overdivided regions only for the extracted overdivided regions, and separate individual objects from the degree of coupling.

  Further, in the present embodiment, rij is calculated from equation (1) from the number of projection points and the area of the group having the maximum number of adjacent overdivided regions, and the degree of coupling is obtained from equation (2) using rij. Although the configuration is adopted, the degree of coupling between adjacent excessively divided regions may be obtained from the number of projection points for each group by another method.

  Further, in this embodiment, it is configured to determine whether to combine or separate adjacent over-divided areas based on the degree of coupling, but in addition to the degree of coupling, for areas composed of adjacent over-divided areas Next, apply a 3D primitive to the area that has a fitting accuracy equal to or higher than the threshold value as an object to be separated, and use a 3D primitive fitting method that separates areas with a fitting accuracy less than the threshold value based on the degree of coupling. The over-divided areas may be combined / separated.

It is a block diagram of the image processing apparatus which concerns on this Embodiment. It is an example of the captured image of right and left. It is an example of the three-dimensional point group which shows the three-dimensional shape of the object in the captured image of FIG. It is an example which grouped the three-dimensional point group of FIG. FIG. 5 is an example in which the grouped three-dimensional point group of FIG. 4 is projected onto the reference image of the captured image of FIG. 2. It is an example of the excessive division area image of the reference | standard image of the captured image of FIG. It is an example which shows the adjacent overdivision area | region which projected the grouped three-dimensional point. It is an example of the area | region division image which shows the result of isolate | separating the object in the captured image of FIG. It is a flowchart which shows the main process which concerns on this Embodiment. It is a flowchart which shows the coupling degree calculation process which concerns on this Embodiment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus, 2, 3 ... Camera, 4 ... Monitor, 5 ... Image processing part

Claims (2)

An image processing method for separating an object in a captured image,
An imaging step of acquiring a plurality of captured images with different viewpoints;
A three-dimensional point processing step for obtaining a three-dimensional point indicating a three-dimensional shape of an imaging target based on a plurality of captured images;
A grouping step for grouping the three-dimensional points obtained in the three-dimensional point processing step;
A projecting step of projecting the three-dimensional points grouped in the grouping step onto a reference image among a plurality of captured images;
An overdivision step of overdividing a reference image of a plurality of captured images into regions;
Based on each overdivision area overdivided in the overdivision step and the three-dimensional point of each group projected in the projection step, the number of 3D points for each group in each adjacent overdivision area is determined, and each group A degree-of-association processing step for obtaining the degree of coupling between adjacent over-divided regions by the number of three-dimensional points of
A separation step of separating an object by combining or separating adjacent over-divided regions based on the degree of coupling obtained in the degree of coupling processing step. An image processing apparatus for separating an object from a captured image,
A plurality of imaging means for acquiring a plurality of captured images from different viewpoints;
Three-dimensional point processing means for obtaining a three-dimensional point indicating a three-dimensional shape of an imaging target based on a plurality of captured images;
Grouping means for grouping the three-dimensional points obtained by the three-dimensional point processing means;
Projecting means for projecting the three-dimensional points grouped by the grouping means onto a reference image among a plurality of captured images;
An overdividing unit that overdivides a reference image of a plurality of captured images into regions;
Based on each overdivision area overdivided by the overdivision means and the three-dimensional point of each group projected by the projection means, the number of three-dimensional points for each group in each adjacent overdivision area is determined, and each group A degree-of-association processing means for obtaining the degree of coupling between adjacent over-divided regions according to the number of three-dimensional points;
Separating means for separating an object by combining or separating adjacent over-divided regions based on the degree of coupling obtained by the degree of coupling processing means.

Images