JP2006004122A - Image processor, robotics device, and vehicle guidance device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To stably restore a shape regarding an object existing in a wide range even outdoors or under intensive illumination, in an image processor. <P>SOLUTION: In the image processor, a first calculation means 1b extracts a change in the positional relation of characteristic points on an image that an image input means 1a has inputted in chronological order. A second calculation means 1c calculates the position and the direction of a view point when the image and the three-dimensional positions of the characteristic points are caught from a change in the positional relation. A third calculation means calculates the three-dimensional shape data on an object caught in an image from the inputted image and three-dimensional positions of the characteristic points. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus that restores a three-dimensional shape of an object or a surrounding obstacle from an image, a robot apparatus equipped with the image processing apparatus, and a vehicle guidance apparatus.

  Examples of conventional apparatuses for image processing of captured images are described in Patent Document 1 and Patent Document 2. The captured image processing apparatus described in Patent Document 1 captures a target object to obtain two-dimensional luminance image data, and restores a three-dimensional shape from the two-dimensional luminance image data. At that time, an edge is detected from the two-dimensional luminance image data, and a parameter indicating the smoothness of the surface of the target object is calculated from the edge. Then, a reflection map parameter as shadow information for accurately restoring the three-dimensional shape is determined using the parameter, and the three-dimensional shape of the target object is restored using this parameter.

  On the other hand, in Patent Document 2, in an image processing apparatus, a subject image from a predetermined viewpoint position captured by an imaging unit is stored in a first storage unit, and an object image from a viewpoint position closest to the captured image is standardized. It is described that it is generated based on a three-dimensional shape model and stored in a second storage means. Then, it is described that a difference is extracted from the difference between the images stored in the first and second storage means, and the standard three-dimensional shape model is corrected based on the difference.

JP-A-5-181980

JP-A-8-233556

  In the conventional image processing apparatuses described in Patent Documents 1 and 2, since the viewing angle is very small when the object is far from the camera, the shape of the object cannot be accurately restored. In addition, in an object outdoors or under bright illumination, halation is likely to occur due to bright reflected light, and it is difficult to identify the luminance value. Furthermore, since the brightness of an object in a distant place or a dark place is generally low, it is difficult to distinguish a difference in brightness at a low level, and it is difficult to restore an object shape in a distant place or a dark place.

  The present invention has been made in view of the above-described problems of the prior art, and the object thereof is to enable stable shape restoration of an object in a wide range from a distant place to a close distance, even outdoors or under strong illumination. It is to be. Another object of the present invention is to reduce the time required for shape restoration.

  The object is to provide an image input means, a first calculation means for extracting a change in the positional relation of the characteristic part on the image input from the image input means, and a tertiary of the characteristic part from the change in the positional relation. An image processing apparatus comprising: second calculation means for calculating an original position; and third calculation means for calculating three-dimensional shape data of an object captured in an image using a three-dimensional position of the characteristic portion Is achieved.

  Further, the above object is to provide a feature part extraction circuit for extracting a feature part from the first image as a set of a plurality of images input in time series by the image input means, and a second image in the set of images. Find the position of the corresponding part by searching for the neighborhood centering on the position of the characteristic part of the first image, and for the third and subsequent images in the set of images, the part obtained for the previous image A feature part tracking circuit that searches for a neighborhood centered on the position of the image to obtain the position of the corresponding part, and a coordinate calculation that calculates the three-dimensional position of the feature part when the position of the feature part in the set of images is obtained When the circuit and the coordinate calculation circuit obtain the three-dimensional position of the feature portion, the correspondence relationship between the feature portion and the shape data portion stored in the storage means is obtained, and the new coordinate calculation is performed. And a data integration circuit for converting the three-dimensional coordinates of the feature portion, and these image input means, feature portion extraction circuit, feature portion tracking circuit, coordinate calculation circuit, and data integration circuit operate in parallel in a pipeline manner. This is achieved by an image processing apparatus.

  According to the image processing apparatus of the present invention, it is possible to restore the three-dimensional shape of an object simply by obtaining images obtained by viewing the object from various directions. However, it is possible to restore the three-dimensional shape even for an object at a close distance. In addition, since no special reference light is used, the shape can be stably restored even outdoors or under strong illumination. Moreover, since all the processes can be performed automatically, it does not take time. Further, since it can be configured as a chip such as an LSI, an inexpensive and compact device can be obtained.

  Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 shows an embodiment of an image processing apparatus according to the present invention. Reference numeral 1 denotes an image processing apparatus according to the present invention, 2 denotes a camera that captures an image of a target object, and 3 denotes a processing apparatus that uses the shape of the target object restored by the image processing apparatus. Specifically, the processing device 3 is, for example, a control device for a mechanical device in which the image processing device 1 is incorporated, and controls the movement of the mechanical device based on object shape data restored by the image processing device. . In addition, 1a is an image input unit provided in the image processing apparatus 1, 1b is a first calculation unit that extracts a change in the positional relationship of feature points on an image input in time series, and 1c is a feature point. Second calculation means for calculating the three-dimensional position of each point and the position and direction of the viewpoint (camera position and orientation) when capturing each input image from the change in positional relationship, 1d is the three-dimensional position of the input image and feature point The third calculation means for calculating the three-dimensional shape data of the object captured in the image from the above, 1e is an input / output means for exchanging information about the restored three-dimensional shape data of the object with the control device 3, 1f is the third The fourth calculation means for obtaining the position / orientation of an object having the same shape by comparing the three-dimensional shape data of the object calculated by the calculation means and the three-dimensional shape data input from the input / output means.

  FIG. 2 is a diagram showing in more detail each component of FIG. In FIG. 2, the feature point / edge extracting means 12 and the feature point tracking means 13 constitute the first calculation means 1b, and the object recognition means 23 and the space search means 24 constitute the fourth calculation means. The coordinate calculation means 16 corresponds to the second calculation means 1c, and the line / surface extraction / registration means 19 corresponds to the third calculation means 1d. Reference numerals 11, 14, 15, and 17 denote storage means for temporarily storing data. Image data input by the image input means, observation matrix generated by the feature point tracking means, and feature point / edge extraction means, respectively. The feature point / edge map generated by, and the three-dimensional coordinate data of the feature point calculated by the coordinate calculation means are stored.

  Reference numeral 20 denotes CAD data storage means which is integrated and accumulated with shape data (hereinafter referred to as CAD data) stored in the CAD data storage means every time the line / surface extraction / registration means generates shape data. . Reference numeral 18 denotes camera position / posture storage means for storing camera position / posture data calculated by the coordinate calculation means. The data to be stored may be only the latest data or data related to all of the input image series.

  Reference numeral 22 denotes storage means for storing a shape model of an object to be searched from CAD data, and the object recognition means 23 is a part of the CAD data stored in 20 that matches the shape data stored in 22. Search from inside. The space search means 24 limits the range to be searched by the object recognition means 23 from the CAD data stored in 20.

  Details of each part will be described. FIG. 3 shows a specific configuration example of the image input means 10. In general, a camera using a CCD image sensor or the like is used to obtain image data. This camera has a lens 2a for forming an image on an image sensor, a CCD element 2b, a readout circuit 2c for reading out the CCD output, and a signal generation for converting the CCD output into a video signal, for example, as shown as 2 in FIG. The circuit 2d is configured.

  In general, CCD elements are sequentially read in the direction of rows of light receiving elements arranged in a grid, and the signal generation circuit 2d is a signal that represents the brightness of the pixels that are sequentially read in an analog manner by the amplitude of the signal. A so-called video signal 2e is generated by synthesizing a (luminance signal) and a synchronization signal representing a line or image segment.

  In the example of the image input means shown in FIG. 3, a synchronizing circuit 10a that separates the synchronizing signal from the video signal, and an A / D converter 10b that separates the luminance signal from the video signal and converts the brightness of the pixel into a digital value. And a writing circuit 10c for writing the digital value of brightness into the storage element 11 in synchronization with the synchronization signal.

  FIG. 4 shows another specific configuration example of the image input means 10. In addition to the above-described camera that outputs a video signal, there is a camera that outputs an image as a digital signal. In such a camera, for example, as shown in FIG. 4, a digital video signal composed of a multi-bit digital signal 2g representing a pixel luminance value and a synchronization signal 2f is output. In such a camera, since the data transfer efficiency is improved, image data can be obtained at high speed. In addition, since the luminance value is digitized inside the camera, there is an advantage that image deterioration due to signal distortion or noise does not occur.

  The example of the image input means shown in FIG. 4 includes a writing circuit 10 c that calculates a pixel data write address from the digital video signal and writes the pixel data to the storage element 11. The configuration of the image input means 10 may be the configuration shown in FIG. 3 or the configuration shown in FIG. 4 or both. By configuring the image input unit as described above, the image processing apparatus can be directly connected to a camera or the like. For this reason, it is not necessary to provide a special additional circuit or the like when the image processing apparatus is incorporated in a mechanical device or the like.

  FIG. 5 shows still another specific configuration example of the image input means 10. In FIG. 5, a light receiving element 10f such as a CCD element is formed on the image processing apparatus 1, and a lens 10d for forming an image is provided thereon. The luminance information detected by the light receiving element 10f is directly or converted into a digital value and transferred to the storage element 11. With such a configuration, it is not necessary to transfer the image data from the camera to the image processing apparatus, so that the image data can be obtained at a very high speed.

  Details of the feature point / edge extraction means 12 will be described with reference to FIG. Here, the feature point means a point whose position can be uniquely specified in a space, such as a vertex of a polygon, an intersection of a line and a line, a break point of a curve, or a branch point of a line. When the above feature points are extracted from an image in which a three-dimensional object is drawn like a line drawing, these points are often the vertices of the object or the figure drawn on the object surface. Therefore, by obtaining the three-dimensional positions of these points and connecting them with lines, it is possible to reproduce the shape of the original object or the shape of the figure drawn on the object surface.

  On the other hand, an edge means a ridge line in the shape of a three-dimensional object, a boundary line of a graphic drawn on the object surface, or a contour line in a two-dimensional image of the object. That is, it corresponds to a line obtained when a three-dimensional object is drawn like a line drawing. When connecting the above feature points with a line, it is possible to correctly reproduce the object shape by connecting the feature points along such an edge. Note that an edge often appears in a place where the brightness greatly varies spatially in an image.

  In FIG. 6, 12a is a means for calculating the spatial differentiation in the U direction (horizontal direction) of the luminance value of the image, and 12b is a means for calculating the spatial differentiation in the V direction (vertical direction) of the luminance value. Specifically, when the luminance of the image at the coordinates (u, v) is I (u, v), the U-direction differential Iu (u, v) and the V-direction differential Iv (u, v) are, for example, Calculate with the formula. (Unless otherwise specified in the following description, (u, v) represents the horizontal and vertical coordinates on the image.)

This is to calculate a change in spatial brightness in the image, and the specific calculation method is not limited to the above.

  12c is a means for calculating an index Ef as a feature point, for example, calculated by the following equation.

Here, Guu, Gvv, and Guv are obtained by integrating the square of the U-direction derivative, the square of the V-direction derivative, and the product of the U-direction derivative and the V-direction derivative in the vicinity region W as shown below.

The index Ef represents the degree to which the direction of change in brightness changes in the neighborhood region W, and the specific calculation method is not limited to the above.

  The feature point / edge extraction unit 12 extracts, for example, pixels having Ef equal to or larger than a certain set value as the feature point using the index Ef, and outputs a list 15a of coordinates (u, v) on the image. To do. 12d is a means for calculating an index Ee as an edge, for example, calculated by the following equation.

The index Ee represents the degree of spatial change in brightness at the pixel (u, v), and the specific calculation method is not limited to the above.

  The feature point / edge extracting means 12 uses the above-mentioned index Ee, for example, a value indicating that it is an edge for a pixel whose Ee is equal to or greater than a certain set value, and a value indicating that for a pixel that is not. Image data (edge pattern) 15b in which a value indicating that is not an edge is set is generated. The feature point / edge map 15 is composed of the feature point coordinate list 15a and the edge pattern 15b.

  Details of the feature point tracking means 13 will be described with reference to FIG. The feature point tracking unit 13 obtains the position of the corresponding feature point in the second image I2 whose position of the feature point is unknown based on the first image I1 whose position of the feature point is already known. At this time, the position of the corresponding feature point is obtained by searching the second image I2 for a neighboring region centered on the position of the known feature point.

  A specific search procedure is shown in FIG. Assuming that the coordinates of the known feature points are (u0, v0), a neighborhood region centered on (u0, v0), for example, an image IW1 of I1 with u and v coordinates in the range of ± w is obtained (s1). At this time, the coordinates (u0, v0) are not necessarily coincident with the coordinates of the grid points constituting the pixels of the image I1, and in s1, the coordinates (u0, v0) are centered from the pixel values of the grid points of I1. It is assumed that the pixel values of the lattice points thus calculated are interpolated (subpixel processing).

  A U-direction differential image and a V-direction differential image of IW1 are obtained and set as Iu and Iv (s2). This process is performed using, for example, the U-direction differential calculation means 12a and the V-direction differential calculation means 12b shown in FIG. First, the estimated position (u, v) of the corresponding feature point of the image I2 is set as (u0, v0) (s3), and the number of repetitions t is set as 0 (s4). An adjacent region centered on coordinates (u, v) with respect to I2, for example, an image in which u and v coordinates are in the range of ± w is obtained as IW2 (s5). In this procedure, sub-pixel processing is performed in the same manner as in procedure s1.

  A matrix G and a vector b are calculated from IW1, IW2, Iu, and Iv (s6). Specifically, G and b are calculated by the following equations.

Here, the definitions of Guu, Gvv, and Guv are the same as in (Expression 3), but the range to be integrated is a neighborhood region centered at (u0, v0), for example, the range where u and v coordinates are ± w, respectively. . In addition, the integration calculation of b indicates that IW1, Iu, and Iv are integrated in the vicinity region centered on (u0, v0), and IW2 is integrated in the vicinity region centered on (u, v).

  If the matrix G has an inverse matrix (s7), the correction amount (du, dv) of (u, v) is calculated by the equation shown in step s8. If G does not have an inverse matrix, the correction amount is set to 0 (s9). Then, (u, v) is corrected by (du, dv) (s10), and if the correction amount is smaller than the predetermined value dm (s11), the process is terminated. If the correction amount is larger than dm, the number of repetitions t is increased by 1 (s12). If the number of repetitions is smaller than the predetermined number tm, the process is continued. When the number of repetitions reaches the predetermined number tm, the process ends (s13). ).

  As described above, in the feature point tracking unit 13, the position of a known feature point is used as an initial value estimated value, and the feature value corresponding by repeating the correction of the estimated value with the correction amount calculated from the neighboring area is repeated. Specify the position of. In this iterative process, the process ends when the correction amount becomes a predetermined value or less, or when the number of repetitions reaches a predetermined number.

  If the threshold value dm regarding the correction amount is reduced, the position of the corresponding point can be specified with high accuracy. However, the number of repetitions increases in some cases, and it takes time to calculate the corresponding points. On the other hand, by providing the threshold value tm regarding the number of repetitions, the calculation can always be completed within a predetermined time, and the corresponding point position can be obtained with high accuracy for a feature point having a fast convergence. Further, it is possible to prevent an infinite loop when the repeated calculation does not converge.

  For the time-series image input by the image input means 10, for example, the position of the feature point is obtained by the feature point / edge extraction means 12 for the first image, and I1 is the first image, I2 As the second image, the position of the corresponding feature point of I2 is obtained by the above procedure. Next, assuming that I1 is the second image, I2 is the third image, and the position of the feature point of I1 is the feature point obtained above, the position of the corresponding feature point is obtained for the third image. . Thereafter, by repeating the same procedure, the position of the feature point can be tracked over the entire time-series image.

  With the above procedure, the feature point tracking means generates the following observation matrix.

Here, uij and vij are the position coordinates on the image of the j-th feature point in the i-th image.

  Details of the coordinate calculation means 16 will be described with reference to FIG. The observation matrix shown in (Equation 6) can be considered as an enumeration of the position coordinates of n feature points on an image obtained by observing an object from m different viewpoints. As described above, there are several methods for calculating the three-dimensional coordinates of a point and the position and orientation of the viewpoint from the appearance of n points at m different viewpoints. In these methods, different calculation methods are used depending on a model for projecting a point on an image plane, for example, whether it is an orthographic projection model or a perspective model. Here, for example, an example in which an orthographic projection model is used will be described.

  First, the average value of each row of the observation matrix is subtracted (s20). Specifically, the calculation is performed according to the following formula.

  Next, the matrix obtained above is subjected to singular value decomposition to obtain a U matrix and a V matrix as the following forms (s21). Note that the symbol T represents transposition of a matrix.

Here, the diagonal elements of the diagonal matrix S are arranged in descending order of the singular values.

  Next, a rotation matrix R is calculated from a 2m × 3 matrix U3 obtained from the left 3 matrix of the U matrix (s22). Specifically, when the submatrix obtained from the upper left 3 × 3 matrix of the diagonal matrix S is S3, R is calculated by the following equation, for example.

  Similarly, the shape matrix P is calculated from the matrix V3 of n rows and 3 columns obtained from the left three matrices of the V matrix (s23). Specifically, for example, the following formula is used for calculation using the partial matrix S3.

  Next, a transformation matrix A for orthonormalizing the rotation matrix R is obtained (s24). This can be obtained as a solution satisfying the following simultaneous equations, where ri is a vector created from each row of R.

上記変換行列Ａを右から掛けたものを回転行列、Ａの逆行列をＰの左から掛けたものを形状行列とすることにより（ｓ２５）、Ｒのｉ行とｉ＋ｍ行が第ｉ番目の画像における視点の方向を規定する二つのベクトル座標となり、Ｐのｊ列がｊ番目の特徴点の３次元座標となる。また、視点の平行移動成分は、観測行列Ｍの各行の平均値として与えられる。

By multiplying the transformation matrix A from the right by the rotation matrix and multiplying the inverse matrix of A by the left of P by the shape matrix (s25), the i-th and i + m rows of R are the i-th image. Are the two vector coordinates that define the direction of the viewpoint, and the j column of P is the three-dimensional coordinate of the j-th feature point. Further, the translation component of the viewpoint is given as an average value of each row of the observation matrix M.

  Details of the line / surface extraction / registration means 19 will be described with reference to FIG. First, a Delaunay triangle is generated from the feature point group (s30). Here, the Delaunay triangle is a triangle formed by a dividing method in which the minimum angle of the triangle is as large as possible when the space is divided into triangles by connecting point groups in a two-dimensional space. Mathematically, it is given as a dual Voronoi diagram composed of point clouds. Here, a Delaunay triangle is generated for a two-dimensional position of a feature point in an arbitrary image used for coordinate calculation.

  A pair of feature points connected by edges is obtained using the edge pattern of the image used to generate the Delaunay triangle (s31). This is obtained, for example, by starting from a certain feature point, following an edge extending in a straight line, and determining that the other feature point is connected if another feature point is reached.

  For the portion where the edge between the pair obtained above intersects with the Delaunay triangle side, correction is made so that the Delaunay triangle side does not intersect the edge (s32). For example, in FIG. 10A, it is assumed that a Delaunay triangle is generated in a shape like a dotted line 41 for the feature point group 40. On the other hand, it is assumed that an edge line connecting feature points is obtained as shown by a solid line 42 in FIG. At this time, since the Delauna triangle sides 41a and 41b intersect the edge line 42a, 41a and 41b are corrected to, for example, a line overlapping the line 41c and the edge 42a shown in FIG.

  With respect to the triangular surfaces obtained by the above procedure, those that are adjacent to each other and have almost the same surface direction are integrated (s33). For example, in FIG. 10C, since the directions of the faces of the triangles 43a, 43b, 43c, 43d, and 43e are almost the same, they are integrated. By performing the same processing for the other surfaces, the result shown in FIG. 10D is obtained. However, in the above processing, when the side where the surface is in contact with the edge line overlaps, the integration is not performed.

  The color of the surface obtained above is obtained (s34). This is obtained as an average value of the colors of the pixels in the region corresponding to the surface on the image. In many cases, a boundary such as a graphic drawn on the object surface is extracted as an edge, and a vertex of the boundary is extracted as a feature point. Therefore, a figure drawn on the object surface by the above processing is also generated as shape data.

  Through the above procedure, the shape data constituted by the generated points, lines, and surfaces is integrated into CAD data (s35). The detailed procedure of this integration process is shown in FIG. A point in the CAD data corresponding to the point (feature point) of the newly generated shape data is extracted (s40). For example, when the feature point / edge extracting unit 12 extracts a feature point, an identification number for identifying each feature point is added to the coordinate list 15a of the feature point. Then, when registering in the CAD data, the identification number is stored together with the point data. Next, when feature points are extracted again, the feature point coordinate list 15a created last time is referred to, and feature points having almost the same coordinates are assigned the same identification numbers as the previous time. As a result, the corresponding point can be extracted by searching the CAD data for a point having the same identification number as the point identification number in the newly generated shape data.

  A coordinate conversion is performed for conversion so that the newly generated shape data point matches the coordinates of the corresponding point extracted from the CAD data (s41). This is obtained by solving simultaneous equations concerning the coordinates of the points. The coordinates of all points of the newly generated enlightenment data are converted by the above coordinate conversion and then registered in the CAD data (s42). At this time, new points that correspond to the points in the CAD data are not registered. Correction processing is performed on data of lines and surfaces generated by integration of point data (s43). For example, if the end point of the line or the vertex point of the surface in the newly generated shape data corresponds to a point in the CAD data, processing such as replacement to the point is performed. Is.

  An example of shape data integration processing is shown in FIG. First, it is assumed that the feature points 40a, 40b, 40c, and 40d are extracted from the image at the camera position 201 by the feature point / edge extraction unit 12. These are also in the field of view at the camera positions 202 and 203, and are tracked by the feature point tracking means 13, and the positions on the corresponding images are obtained. Assume that the three-dimensional coordinates of the feature points 40a, 40b, 40c, and 40d are calculated from these position data and registered in the CAD data as shape data. Next, feature points 40c, 40d, 40e, and 40f are extracted from the image at the camera position 204, tracking processing is performed on the subsequent images at the camera positions 205 and 206, and the three-dimensional coordinates of these feature points are obtained. Suppose that it is calculated. At this time, the coordinates of the feature points 40c and 40d that are common between the two are not necessarily the same three-dimensional coordinates. This is because only the relative positional relationship between the camera and the feature point is obtained in the calculation process for restoring the three-dimensional coordinates from the positional relationship between the feature points on the screen. Therefore, coordinate conversion is performed so that the coordinates of the feature points 40c and 40d calculated from the camera positions 201, 202, and 203 overlap with the coordinates of the feature points 40c and 40d calculated from the camera positions 204, 205, and 206, thereby obtaining the feature points 40e, Correct shape data can be obtained by converting and integrating the coordinates of 40f.

  Next, details of the data input / output means 21 will be described with reference to FIG. The data input / output means includes, for example, an address signal line 21a for designating an address, a data line 21b for setting or reading data, a chip select control line 21c for designating access to the image processing apparatus 1, and input or reading of data. A read / write control line 21d to be designated, a strobe control line 21e to designate the timing of data writing or reading, an interrupt control line 21f to request interrupt processing from the control device connected to the image processing apparatus 1, and the like are provided.

  Using these signal lines, for example, shape data having a shape as shown in FIG. 14 is input / output. In FIG. 14, 44 is the coordinate data of the points constituting the shape, 45 is the line data, 46 is the surface data, and 47 is the object data. The line data 45 is a set of numerical data, each representing the number of the start point and the end point. The surface data 46 is numerical data composed of an arbitrary number of sets, each representing a line number. The object data 47 is numerical data composed of an arbitrary number of sets, each representing a surface number. For example, the example shown in FIG. 14 has one plane P1, four lines L1, L2, L3, L4, four points (x1, y1, z1), (x2, y2, z2), ( x3, y3, z3) and (x4, y4, z4) represent the shape data of the object.

  Next, details of the object recognition means 23 and the space search means 24 will be described with reference to FIG.

  The object recognizing means 23 searches the CAD data 20 for a shape portion that matches the shape data input from the input / output means 21 and temporarily stored in the storage means 22, and obtains its position and orientation. The search performed at this time may be performed over the entire CAD data 20, but for example, the current camera position and orientation may be specified and the search may be performed from a range visible from the current camera. By limiting the search range as in the latter case, the search process can be performed in a shorter time. For example, in FIG. 17A, for example, the camera mounted on the work machine in which the image processing apparatus 1 is incorporated is in the position and orientation shown by 52, and the work machine is placed on the objects 51a, 51b, 51c in the range visible from this camera. When working on the object, it is not necessary to search the entire CAD data space 51 for the purpose of searching for a specific object, but only the space 52a within the range visible from the camera position and orientation 52 is searched. Just do it. As a result, there is no need to search for an object such as 51d or 51e that is not related to the current work, and the search time can be shortened.

  Step s50 in FIG. 16 is processing for limiting the search range as described above, and extracts shape data included in a space designated by a camera viewpoint or the like.

  This extraction process can be performed at high speed by using, for example, a spatial search tree as shown in FIG. The lower node 55 at the time of the space search tree shown in FIG. 17B corresponds to a partial space obtained by dividing the space 51 in FIG. 17A, and is in a partial space surrounded by a dotted line indicated by a corresponding number. Correspond. The node 54 in the second stage corresponds to a space obtained by integrating adjacent ones of the partial spaces, and the uppermost node 53 corresponds to the entire space 51.

  Using this spatial search tree, for example, an object included in the search range 52a in FIG. 17A is extracted by the following procedure. First, starting from the highest node in the space search tree, the search is sequentially advanced to lower nodes, leaving only the nodes corresponding to the partial space overlapping the search range 52a. For example, in the example of FIG. 17, the space corresponding to the node 4 (spaces indicated by 11, 12, 15 and 16 in FIG. 17A) in the second stage node does not overlap with 52 a, so It does not search for connected nodes. As for the nodes 1, 2, and 3, only the first level nodes connected therebelow leave only those having overlapping portions. As a result, 1, 2, 5, 6, 3, 7, 9, 10, and 13 nodes are left. Finally, an object included in the partial space corresponding to the remaining node can be used as an object included in the space 52a. Precisely, there may be objects that are not included in the space 52a among the objects obtained above, but this number can be made sufficiently small by appropriately setting the division number of the space.

  Next, a point cloud is set on the shape data (model) stored in the storage means 22 and the shape surface of the shape data extracted in s50 (s51). For example, as shown in FIG. 18, a point group 56 that is uniformly distributed on the surface of the shape data is set.

  Next, the model point cloud is superimposed on the shape data point group at various positions and orientations to obtain the best position and orientation (s52). Various methods have been proposed for this superposition search procedure, such as A. Johnson and M. Hebert, "Using spin images for efficient object recognition in cluttered 3D scenes," IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 21 , No. 5, May, 1999, pp. 433-449.

  Next, if the degree of coincidence between the shape data in the position / orientation obtained as a result and the model point cloud is larger than a predetermined value (s53), the position / orientation data is output (s54); The fact that no is found is output (s55). Next, the overall operation flow of the image processing apparatus 1 will be described with reference to FIG.

  In FIG. 19, the horizontal flow represents the passage of time from left to right, and the vertical flow mainly represents the data flow. Also, the squares in the figure mainly represent the timing at which the processing shown at the left end of each row in which they are arranged.

  First, images captured by the camera are sequentially taken into the image processing apparatus at the timing indicated by 60 by the image input means 10. For example, when these image data are set as shown in 60a, the feature point / edge extraction processing 61 is executed at the timing when the head image is captured. The coordinate data on the image of the feature points obtained as a result is the first row of the observation matrix 63. The feature point tracking process 62 is executed for the second and subsequent images in the image set 60 a, and the coordinate data of the feature points in each image obtained as a result is the second and subsequent rows of the observation matrix 63.

  When the feature point tracking processing for all the images in the image set 60a is completed by the above procedure, the coordinate calculation processing 64 for calculating the three-dimensional coordinates of the feature points and the position and orientation of the camera from the generated observation matrix is executed. Is done. Then, when the three-dimensional coordinates of the feature points are obtained, line / surface extraction and registration processing 65 is executed, and new shape data is registered in the CAD data 66. On the other hand, when an object search command is input via the input / output means 21 (68), execution of the object recognition process is started from that point (67), and when the recognition process is completed, Position and orientation data is output (69).

  In general, the coordinate calculation process and the line / surface extraction / registration process require much longer time than the feature point extraction process and the feature point tracking process. For this reason, if the next image acquisition is started after all the processes from image acquisition to shape data registration are completed, the time from the acquisition of the first image to the acquisition of the next image becomes longer. . Then, it becomes very difficult to find the correspondence of feature points between the previous image and the subsequent image.

  On the other hand, images are collected as a set, and the length of the set is made longer than the length of the coordinate calculation process or line / surface extraction registration process, and the pipeline process as shown in FIG. The time interval between images can be shortened, and the correspondence between feature points can be obtained by a simple tracking process. In addition, when calculating the three-dimensional coordinates of feature points, information relating to a large amount of image data can be used, so that the accuracy of calculation results can be increased.

  Next, each device equipped with the above-described image processing apparatus will be described in detail. FIG. 20 shows a block diagram when the image processing apparatus is applied to a manipulator type robot. The robot 70 includes a plurality of joints and a gripping mechanism that grips an object at the tip. A camera 71 is provided near the gripping mechanism, and an image captured thereby is input to the image processing apparatus 1 according to the present invention. Then, the shape data of the object obtained by the image processing apparatus 1 is used by the control means 72 of the robot.

  For example, it is assumed that an operation of inserting the object 75 currently held by the robot into a hole 74 provided in the object 73 is performed. First, the robot's gripping mechanism is moved around the object 73, and images obtained by viewing the object 73 from various directions are obtained from the camera 71. By processing these images with the image processing apparatus 1, three-dimensional shape data of the object 73 is generated. Here, the control means 72 inputs the shape model of the object 73 to the image processing apparatus 1 and instructs to obtain the position and orientation of the object. Then, the image processing apparatus 1 searches for the object 73 by the built-in object recognition unit, obtains its position and orientation, and outputs it to the control unit 72. The control means 72 can correctly insert the object 75 into the hole 74 by controlling the position and orientation of the gripping mechanism based on the position and orientation data.

  FIG. 21 is a schematic diagram when the image processing apparatus is mounted on the moving mechanism. . The robot 80 includes a moving mechanism 83 and a camera 81, and inputs an image captured by the camera 81 to the image processing apparatus 1 according to the present invention. Then, the object shape data obtained by the image processing apparatus 1 is used by the robot control means 82. For example, it is assumed that the robot is currently moving in the house using the moving mechanism 83. By processing the image around the robot captured from the camera 81 by the image processing apparatus 1, the three-dimensional shape data of the obstacle around the robot and the position and orientation of the camera 81 are calculated. Here, the control means 82 takes out the obstacle shape data and the camera position / orientation data from the image processing apparatus 1, and calculates the current positional relationship of the robot itself with respect to surrounding obstacles. By using this, the robot 80 can move freely in the house without hitting surrounding obstacles.

  FIG. 22 shows a schematic diagram of a vehicle equipped with an image processing device. The vehicle 90 includes cameras 91 and 92, a vehicle guidance device 93, a display device 94, and the image processing device 1 according to the present invention, and inputs an image captured by the camera 91 or 92 to the image processing device 1 according to the present invention. . Then, the vehicle guidance device 93 uses the object shape data obtained by the image processing device 1.

  For example, assuming that the vehicle 90 is currently traveling on a road, the camera 91 captures an image of another vehicle traveling in front of the vehicle 90 and inputs the image to the image processing apparatus 1. Then, the image processing device 1 calculates the shape of the preceding vehicle and the position and orientation of the camera 91. The vehicle guidance device 93 reads the shape data and the camera position / orientation data from the image processing device 1 and calculates the relative positional relationship of the shape of the preceding vehicle with respect to the own vehicle using the camera position / orientation data. . Then, as shown in FIG. 23, the positional relationship between the host vehicle 94a and the preceding vehicle 94b is displayed on the display device 94.

  For example, if the vehicle 90 is about to enter the garage in the back, the camera 92 captures an image of an obstacle behind the vehicle 90 and inputs it to the image processing apparatus 1. Then, the image processing apparatus 1 calculates the position and orientation of the rear obstacle shape data and the camera 92. The vehicle guidance device 93 reads the shape data and the camera position / orientation data from the image processing device 1 and calculates the relative positional relationship of the shape of the rear obstacle with respect to the vehicle using the camera position / orientation data. To do. Then, as shown in FIG. 23, the positional relationship between the host vehicle 94a and the rear obstacle 94c is displayed on the display device 94. The display as shown in FIG. 23 helps the driver of the vehicle 90 during driving on the road or in the garage.

  In the above embodiments, various arithmetic processes in the image processing apparatus 1 are executed by a general digital circuit. However, if a semiconductor chip such as an LSI is used as the digital circuit, a compact image processing apparatus is obtained. The various arithmetic processes may be a computer program used by a computer. In that case, the image processing apparatus 1 is a computer. The feature points may be line segments or patterns in the image.

1 is a block diagram of an embodiment of an image processing apparatus according to the present invention. It is a detailed block diagram of the image processing apparatus shown in FIG. It is a schematic diagram of the image input means shown in FIG. It is a schematic diagram of the other Example of the image input means shown in FIG. It is a schematic diagram of the further another Example of the image input means shown in FIG. FIG. 3 is a schematic diagram of feature points and edge extraction means shown in FIG. 2. It is a flowchart explaining operation | movement of the feature point tracking means shown in FIG. It is a flowchart explaining operation | movement of the coordinate calculation means shown in FIG. It is a flowchart explaining operation | movement of the means to extract and register the line or surface shown in FIG. It is a figure explaining the extraction method of a line or a surface. It is a flowchart explaining the method of registering shape data into CAD data. It is a figure explaining the method of registering shape data to CAD data. It is a block diagram of one Example of the input / output means with which the 1st thru | or 4th calculation means is provided. It is a figure explaining the shape data input / output from the input / output means shown in FIG. It is a figure explaining the content of shape data. It is a flowchart explaining operation | movement of an object search and an object recognition means. It is a figure explaining an object and space search. It is a figure explaining the point group of a shape surface. It is a time chart explaining operation | movement of an image processing apparatus. It is a block diagram of one Example of the manipulator type robot which concerns on this invention. It is a schematic diagram of one Example of the moving mechanism which concerns on this invention. It is a mimetic diagram of one example of a vehicle guidance device concerning the present invention. It is a figure which shows the example of a display screen of the display apparatus used for the vehicle guidance apparatus shown in FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus, 2 ... Camera, 3 ... Control apparatus, 10 ... Image input means, 11 ... Image storage means, 12 ... Feature point / edge extraction means, 13 ... Feature point tracking means, 14 ... Observation matrix storage means, DESCRIPTION OF SYMBOLS 15 ... Feature point / edge map storage means, 16 ... Coordinate calculation means, 17 ... Three-dimensional coordinate data storage means, 18 ... Camera position / posture storage means, 19 ... Line / surface extraction / registration means, 20 ... CAD data storage means, 21: I / O means, 22: Object model storage means, 23 ... Object recognition means, 24 ... Space search means.

Claims (27)

  An image input means; a first calculation means for extracting a change in the positional relationship of the characteristic part of the image input from the image input means; and a three-dimensional position of the characteristic part from the change in the positional relationship of the input image And a third calculation means for calculating the three-dimensional shape data of the object displayed in the image using the three-dimensional position of the characteristic part calculated by the second calculation means. An image processing apparatus comprising:   A storage means for storing a three-dimensional shape, a three-dimensional shape stored by comparing the three-dimensional shape data of the object calculated by the third calculation means with the three-dimensional shape data stored in the storage means in advance. The image processing apparatus according to claim 1, further comprising a fourth calculation unit that obtains the position and orientation of an object having a shape similar to data.   The image processing apparatus according to claim 1, wherein the image input unit includes a conversion unit that converts an analog input signal or a digital input signal into digital image data.   The image processing apparatus according to claim 1, wherein the image input unit includes a light receiving element, and includes a unit that converts a signal captured by the light receiving element into image data.   The third calculation means calculates the three-dimensional shape data using the coordinate values of the points representing the object, the numbers of the start and end points constituting the line segment, and the numbers of the lines constituting the surface. The image processing apparatus according to claim 1.   A reference coordinate system is set in advance for the input image, the position of the object or viewpoint in the reference coordinate system is displayed with three parameters, and the posture or viewpoint of the object is displayed with three parameters. The image processing apparatus according to claim 1, wherein the third calculation means calculates three-dimensional shape data of the object using these parameters.   The image input means inputs an address signal, a data signal, and a control signal, and at least one of the first to third calculation means reads image data input from the image input means or register contents. The image processing apparatus according to claim 1, wherein a control signal is output in a state where an address corresponding to the address signal is given.   The first calculation unit includes a feature part extraction unit that extracts a characteristic part of an image from an image input by the image input unit, and a feature part tracking unit that tracks a change in the position of the characteristic part. The image processing apparatus according to claim 1.   The said 3rd calculation means calculates a some line segment from the point which the said 2nd calculation means calculated, and calculates the surface which this some line segment comprises. Image processing device.   The said 3rd calculation means connects the some line segment which comprises the outline calculated | required about the object displayed on an input image by the end point, and forms an outline. Image processing apparatus.   The third calculation unit connects adjacent points in the characteristic portion calculated by the second calculation unit to form a line segment and calculates a surface formed by the line segment. The image processing apparatus according to claim 1, wherein when the contour line of the object displayed on the screen intersects, the line segment and the calculated surface are corrected so that the line segment and the contour line do not intersect.   The shape data already calculated is stored in the storage means, and a new coordinate calculation is performed using the correspondence between the feature portion newly calculated by the second calculation means and the shape data portion stored in the storage means. The image processing means according to claim 2, further comprising an integration means for integrating the obtained characteristic portion with the already calculated shape data.   The fourth calculating means uses a part of the stored 3D shape data to determine the position and orientation of the object having a shape similar to the stored 3D shape data. 3. The image processing apparatus according to claim 2, further comprising means for limiting the position and orientation to be obtained.   The three-dimensional shape data stored in the storage means and the shape of the object are both displayed as a point group, and the fourth calculation means indicates the degree of coincidence of the point group in the stored three-dimensional shape data and the shape of the object. The image processing apparatus according to claim 2, further comprising comparison means for comparing.   15. The comparison unit according to claim 14, wherein when the degree of coincidence of the point group is smaller than a predetermined value, it is determined that a shape similar to the three-dimensional shape data stored in the storage unit is not found. The image processing apparatus described.   When the fourth calculation means obtains an object similar to the three-dimensional shape stored in the storage means, the fourth calculation means replaces the three-dimensional shape stored in the storage means with the obtained object shape. The image processing apparatus according to claim 2.   The fourth calculation unit, when obtaining an object similar to the three-dimensional shape stored in the storage unit, stores identification information and position information of the similar object in the storage unit. Item 3. The image processing apparatus according to Item 2.   A feature portion extraction circuit for extracting a feature portion from a first image of the plurality of images input by the image input means; and a second image of the plurality of images, a portion corresponding to the feature portion extracted from the first image; A feature portion tracking circuit for obtaining a feature portion and obtaining a feature portion corresponding to the feature portion of the immediately preceding image in the third and subsequent images of the plurality of images is provided in the third calculation means. The image processing apparatus according to claim 1.   The image processing apparatus according to claim 18, wherein the second calculation unit includes a coordinate calculation circuit that calculates a three-dimensional position of the feature portion by obtaining a position of the feature portion in each of the plurality of images. .   Using the three-dimensional position of the feature portion obtained by the coordinate calculation circuit and the three-dimensional shape data stored in the storage means, the three-dimensional position of the feature portion obtained by the coordinate calculation circuit is adapted to the three-dimensional shape data. The image processing apparatus according to claim 19, further comprising a data integration circuit for converting the data as described above.   21. The image according to claim 20, wherein the image input means, the feature portion extraction circuit, the feature portion tracking circuit, the coordinate calculation circuit, and the data integration circuit are arranged so as to operate in parallel in a pipeline manner. Processing equipment.   An image input means; a first calculation means for extracting a change in the positional relationship of the characteristic part of the image input from the image input means; and a three-dimensional position of the characteristic part from the change in the positional relationship of the input image And a third calculation means for calculating the three-dimensional shape data of the object displayed in the image using the three-dimensional position of the characteristic part calculated by the second calculation means. An image processing apparatus including: a control device that transmits data to and from the image processing apparatus; and a manipulator having an arm and a hand. The image capturing unit is provided on the arm or the hand of the manipulator. A robot apparatus characterized by that.   An image input means; a first calculation means for extracting a change in the positional relationship of the characteristic part of the image input from the image input means; and a three-dimensional position of the characteristic part from the change in the positional relationship of the input image And a third calculation means for calculating the three-dimensional shape data of the object displayed in the image using the three-dimensional position of the characteristic part calculated by the second calculation means. And a display device for displaying the output of the image processing device, wherein the display device performs image processing based on object shape data obtained by the image processing device. A vehicle guidance device for displaying a position of an obstacle around a vehicle on which the device is mounted.   24. The vehicle guidance device according to claim 23, wherein the display device displays obstacles around the vehicle as a top view.   24. The vehicle guidance device according to claim 23, wherein the display device displays a position of a vehicle traveling in front of a vehicle on which the image processing device is mounted.   The vehicle guidance device according to claim 23, wherein the display device displays a position of an object in a parking space of a vehicle on which the image processing device is mounted.   The vehicle guidance according to claim 23, wherein the display device corrects and displays the position of the vehicle on a map based on a positional relationship between buildings around the vehicle on which the image processing device is mounted. apparatus.

Images