JPH04102178A - Object model input device - Google Patents

Abstract

PURPOSE:To uniquely obtain a model concerning an object by teaching a strict condition, which can not be comprehended only by a computer, from the outside in the case of conversion from the shape and color data of a face on a picture to the shape and color data of the face on a three-dimensional space. CONSTITUTION:Drawing data generated by respective color recognition parts 1 are corrected the visual coordinates by plural cameras 55 and temporarily stored in a two-dimensional data storage part 9. Afterwards, faces are made correspondent among plural pictures by a face correspondence decision part 10, three-dimensional coordinate values are calculated by a plane processing part 11 and the data are integrated with three-dimensional data from a freely curved surface processing part 12 by a three-dimensional data integration part 19. Further, a structure definition part 20 defines the model structure and the defined result is displayed at a recognized result output part 21 in an arbitrary direction while changing a view point for each part designated from the outside. Thus, the other model can be easily prepared from the graphic model.

Description

[Detailed Description of the Invention] [Industrial Application Field] The present invention relates to a device for inputting and identifying three-dimensional shape and color information of an object as an image, and particularly when the object itself or a photograph of the object exists. The present invention relates to an object model input device with which a model of the object can be easily input and identified.

[Conventional technology]

Conventionally, regarding this type of device, Japanese Patent Application Laid-Open No. 1983
As described in Japanese Patent No. 134979, dimensions and shapes of each part of the object are input from a drawing of the object as an input object.

[Problem to be solved by the invention]

However, in the case of the above-mentioned conventional technology, it is effective if a drawing of the object is available, but if the dimensions and shape of each part are not known in advance, such as when a drawing is not available, Dimensions and shapes cannot be input.

On the other hand, in recent years, the need to input a graphic model of an object into a computer has arisen in a wide variety of fields, including not only CAD/CAM but also animation creation and visual simulation.

However, in this case, the size and shape of the object are not necessarily disclosed in the drawing (7); rather, it is extremely rare for an object to be disclosed in the drawing. In order to input a graphic model into a computer, it is necessary to clarify the dimensions and colors of each part of the object, and then input information about those dimensions, shapes, and colors one after another, which takes a lot of time. ing.

SUMMARY OF THE INVENTION An object of the present invention is to provide an object model input device that can easily input a graphic model of an object into a computer by processing an image of the object itself or a photograph.

Another object of the present invention is to provide an object model input device that can easily create another model from the graphic model.

[Means to solve the problem]

The two-legged purpose is to provide a means for inputting an image of an object, and from that image to recognize the surfaces constituting the object on a surface-by-surface basis based on instructions from the outside, and to generate shape data and color data corresponding to the surfaces on the image. means for displaying the shapes and colors corresponding to these surfaces on a display and changing them based on external operation input; and a means for converting the data into color data, a means for externally inputting the information necessary for the conversion, and a means for displaying the converted data in three-dimensional space on a display three-dimensionally, and based on external operational input. This is achieved by including means for changing the image and means for adding a surface that does not exist in the input image to the surface already obtained based on an external operation input.

Another object of the present invention is to further provide the object model input device configured as described above with means for defining one or more recognized surfaces constituting an object as one group, and defining each group as one or more groups. This is achieved by providing means for collectively changing shape, position, and color data corresponding to one or more surfaces to which the surface belongs.

[Effect]

Generally, an image of an object is considered to accurately capture physical information such as the shape and color of the object within the resolution of the sensor used to input the image. However, the relationship between a three-dimensional object and a normal two-dimensional image is a so-called one-to-n transformation, and it is impossible to uniquely create a model of a three-dimensional object from a two-dimensional image. However, the shapes and colors of many objects have regularities, and if these are used as constraints, it is relatively easy to create a model of a three-dimensional object. At that time, it is difficult to make the computer understand the regularity, but for humans, the regularity is often common sense. Therefore, in the process of creating an object model, if a situation arises that cannot be determined arbitrarily, you can ask the operator questions and confirm, and continue processing according to the operator's response instructions. This makes it possible to input a model of an object relatively easily.

More specifically, if an image input from the outside is divided into regions based on color, divided regions with different colors can be detected as surfaces. Furthermore, even if the surfaces are of the same color, the brightness differs in the bent portions, so the contour shape of the surface can be detected by determining the edges in the shading of the image. In this way, each surface that makes up an object can be detected from the image, but images generally contain noise, and the surface of the object itself also has dirt, etc. that should normally be ignored. Considering these circumstances, not all detected surfaces are necessarily correct. To solve this problem, it is possible to easily confirm the detected surface by displaying it on a display. If there is an error in this confirmation, the operator can instruct the image processing method and threshold value for surface detection, or modify the shape and color of the surface directly on the image using an editor. For example, each face can be detected correctly.

However, the shape of each surface detected in the above manner is only on a two-dimensional image, and is not the original three-dimensional shape of the object, but the line drawing of the two-dimensional shape is based on the sensor that inputs the image and the position of the object. It has a perspective shape created by the relationship.

Therefore, if you ask the operator for parallel lines or perpendicular lines on the actual object, which are the constraint conditions necessary for converting it into a three-dimensional shape, and then use the response instructions, you can use this information to determine the direction of the surface. Since we know that , the shape in three-dimensional space can be found. -Once the shape of some surfaces in three-dimensional space is determined, the ridge lines of the adjacent surfaces are also determined, so all the surfaces constituting the object and their respective A three-dimensional shape can be found. Note that it is easy for the operator to modify the result of conversion into the three-dimensional shape. If appropriate corrections are made by the operator, correct shape results in three-dimensional space can be easily obtained.

By the way, the image obtained by capturing an object is only for the front side of the object, and it is not possible to obtain an image for the back side of the object, but the shape of this invisible part can also be edited using three-dimensional shape editing means. This allows for easier input. In general, objects are often symmetrical or have a certain width, so if the shape and color of half of the object are obtained, the shape and color of the remaining part can be easily defined with simple shape editing operations. That is what it is. Of course, if you take images of an object from multiple different directions so that all sides of the object appear on the image (7. If you obtain multiple images of the same object, you can view the entire object without performing equivalent graphic editing operations. It is also possible to input a model of a curved surface.In addition, for objects with free-form surfaces, if you use a range finder (e.g. a laser range finder) to directly obtain distance data, you can easily create curved surfaces. It can be supplemented by

Now, in the object model input process as described above, if you name and define multiple surfaces that make up a specific element of the object as one group, you can deform or shift the specific elements all at once. Using this, it becomes possible to easily create another deformed object model based on the input model.

〔Example〕

Hereinafter, the present invention will be explained in detail with reference to FIGS. 1 to 13.

First, an overview of the configuration and operation of the object model input device according to the present invention will be explained. FIG. 1 shows the general configuration of an example thereof. In this case, the color image recognition unit 1 provided for a color TV camera (hereinafter simply referred to as a camera) 55 is capable of recognizing the shape and color of multiple objects constituting an object from a single color image. It has become. From the image input unit 2, the color image of the target object (including two-dimensional images such as color photographs) captured by the camera 55 is generally converted into color data of three primary colors, that is, red (R), green ( G) and blue (B) data, these color data are converted into hue and saturation data by the color region dividing section 3, and then the image is divided into regions. On the other hand, the color data from the image input section 2 is converted into brightness data and subjected to differentiation processing in the grayscale image differentiation processing section 4, and edge candidate points are extracted by this differentiation processing. After that, the color area dividing section 3,
The line drawing data as a processing result from each of the grayscale image differential processing sections 4 is edited and corrected by the line drawing editing section 5, and further edited and corrected by the line drawing modeling section 6 into straight lines, circular arcs, elliptical arcs, quadratic curves, etc. The line drawings are approximated and expressed using parameters. The closed surface extracting section 7 extracts the regions closed by this line drawing as overlapping parts that constitute the target object.

Although a plurality of cameras 55 are arranged in FIG. 1, each of these cameras 55, whose relative positional relationship is known in advance, is operated in a coordinated manner to more reliably detect the three-dimensional shape of the target object. The target object is imaged from different directions. The images captured by each of these cameras 55 are processed in the corresponding color image recognition section 1 in a predetermined manner as described above. Of course, those cameras 5
In the case where the imaging signals from 5 are sequentially taken into the common color image recognition section, it is not necessary to provide a color image recognition section corresponding to the camera.

As described above, the position of each object on the two-dimensional image that makes up the target object is known, and the correspondence between this and the actual position on the three-dimensional space is determined by visual coordinate calibration processing. It has become something that can be done. When a target object is imaged by a plurality of cameras 55, the color image recognition unit 1
The line drawing data generated in each process is subjected to visual coordinate calibration processing.
Once stored in the dimensional data storage unit 9, the plane correspondence determining unit 1-O correlates the planes between the plurality of images, and then the plane processing unit 11 calculates three-dimensional coordinate values. There is. However, when there is only one camera 55 and therefore only one two-dimensional image is obtained, after the line drawing data is processed by the visual coordinate calibration unit 8,
The three-dimensional coordinate values are directly determined by the plane processing unit 11 via the plane correspondence determining unit 10 without any processing being performed by the plane correspondence determining unit 10. Surface correspondence determination section 1
The reason why no processing is performed at 0 is that with only one two-dimensional image, it is impossible to determine the correspondence between surfaces. Now, when a plurality of two-dimensional images are obtained, it is likely that a plurality of two-dimensional coordinate values for line drawing data are obtained at the same time. A three-dimensional data detection unit 13 detects three-dimensional coordinate values.

When there is only one two-dimensional coordinate value or when only one two-dimensional image is obtained, the two-dimensional coordinate value is first determined by the surface normal determination unit 15 in the single-view surface processing unit 14. After the normal of the surface is determined, the single-view surface three-dimensional data detection unit 16
Three-dimensional coordinate values are detected. In this way, three-dimensional coordinate values are obtained from the plane processing section 11, or these three-dimensional coordinate values are integrated with three-dimensional data from the free-form surface processing section 12 in the three-dimensional data integration section 19. ing. When at least a part of the target object is composed of a curved surface, three-dimensional coordinate values for the curved surface can be obtained from the range finder input section 17 in the free-form surface processing section 12 via the curved surface shape input section 18. , these data are integrated with those from the plane processing section 11 by the three-dimensional data integration section 19. By integrating the three-dimensional coordinate values in the three-dimensional data integrating section 19, the overall model shape of the input target object can be known, but furthermore, the structure defining section 20 defines the model structure, The definition results are displayed in the recognition result output section 21 from an arbitrary direction by changing the viewpoint for each part specified from the outside. In addition, when executing the various processes described above, various commands are input from the command input section 57 by external operations by an operator as necessary, but all of these operations are now displayed on the display 56. ing.

That is, the various processes described above are performed in an interactive manner with the operator via the command input section 57 and display 56.

The configuration and operation of the object model input device according to the present invention have been outlined above, and the operation of each part of the configuration will be described in more detail as follows.

That is, first, the target object is imaged from the side, etc. by the camera 55, and the image signal from the camera 55 is obtained as color image data via the image input section 2. This color image data is the color data of the three primary colors (
Although expressed as R, G, B), the color image data is branched and input to the color area dividing section 3 and the grayscale image differential processing section 4. Of these, the color area dividing unit 3 divides each surface of the target object into color areas based on differences in hue. The figure shows an example of the display screen at that time. The operation method on the screen shown in FIG. 3 will be explained based on the method procedure. Once color image data is taken into the color area dividing section 3 (step 100), the color image data is converted into hue and saturation data. In addition, the original image of the target object is displayed in the image display window 22 (step 104).

Next, based on this display, the operator moves the mouse 2.
6, areas that are considered to be of the same color are specified, including shaded areas (step 10).
6). When the area specification is completed (step 108), the hue in the specified area is displayed in the color of the original image in the hue graph window 24 (step 110). A color palette is created by setting a separable hue range and representative colors within that range in the hue graph window 24 (step 112). As a result of these settings, an area having a hue within the setting range is displayed in the area dividing window 25 in a state filled with its representative color (step 114). Next, in the display within the region division window 25, only the portion having the number of pixels equal to or greater than a certain value is recognized as a region, and fine noise is removed (step 116). At that time, the value of the number of pixels as a threshold value can be variably adjusted with the dial 27, and the display screen in the area division window 25 changes continuously with the threshold value, but noise is appropriately removed. Dial operation is stopped when the dial is pressed. The area removed as noise is considered to be the same color area as the surrounding area, and interpolation is performed using that color. As shown in Figure 2, color area segmentation is completed [7
If the above process (color palette creation process, noise removal process) is repeated every time a region that is considered to be the same color is specified (step 118), the region division window 25 A new color area is added to the . Since the operating methods of the mouse, dial, etc. and menu selection are all displayed on the menu display window 23, the user can perform color area division while following the displayed instructions.

Incidentally, color region division can be performed not only based on hue, but also based on the intensity of RGB values, saturation, and brightness. When considering the surrounding environmental conditions of the object, such as lighting conditions, it is generally desirable to perform color region segmentation based on hue. However, if color region segmentation based on hue cannot be performed satisfactorily, This is because color area division may be performed satisfactorily by selectively using the color area. Even when using something other than hue, as shown in Figures 2 and 3,
Color area segmentation is performed. For example, in the case of using RGB values, three primary color components R and G are used within a specified area.

The average and variance of each B are determined, a color palette is created based on the range calculated from the variance, and color area division is performed. Division based on RGB values is mainly necessary to compensate for low-chromatic parts that cannot be divided into regions using hue. 1. Therefore, for example, as described above, two types of area division results are finally obtained (for example, area division results based on hue and RGB values).
By displaying the image and the original image on the screen at the same time, the operator can select an appropriate portion from among the two types of area division results and edit one area division image.

For example, when one of the divided images is clicked with a mouse within the area to be adopted, the clicked area is overwritten in the other divided image, and the divided area images are edited in this way. In this way (7), two types of divided pictures can be easily edited, and the boundaries of divided areas in these divided pictures can be extracted as edges.

On the other hand, the color image data from the image input section 2 is simultaneously processed by the grayscale image differential processing section 4, so that edges are detected.

Color image data is converted to brightness data that indicates gradation values, and then subjected to differential processing to obtain a differential image, and there are several differential processing methods used at this time. It has become. The operator selects multiple arbitrary differential processing methods from the differential processing methods displayed on the menu and performs differential processing at the same time.
The resulting differential image is binarized using a certain threshold. All edge images obtained as a result of the binarization process are displayed simultaneously on a multi-window screen, with the threshold being variable and adjustable by the operator using a dial. The edge image also changes according to the threshold value, but by stopping the threshold value change while viewing the processing results on the multi-window, the change in the edge image can also be stopped. . Next, noise is removed from the edge image extracted in this way, so that only edges with lengths longer than a certain value are recognized as line segments, and edges less than that length are removed as noise. It has become. If the threshold value for noise removal is also continuously set variably by the operator, the threshold value can be set arbitrarily while viewing the processing results on the window according to the variably set threshold value. It can be done. After the multiple edge images obtained in the above manner are divided into a specified number of screen divisions (for example, 9 divisions, 16 divisions, or 25 divisions), the operator can select any one of the corresponding divisions. By selecting one edge image and combining it with another selected edge image, one edge image can be easily created.

Therefore, the colored edge image extracted as a region boundary by the color region dividing section 3 and the grayscale image differential processing section 4
By editing and modifying the edge image extracted in the line drawing editing section 5, one line drawing having color information can be easily generated. That is, first, these two images are displayed superimposed, that is, the sum (logical sum) of the two images is displayed.
OR) is required and displayed.

Misrecognized lines can be easily identified from this display, but misrecognized lines can now be deleted by specifying their endpoints with the mouse and filling them with the same color as the surrounding pixels. There is. In this way, for the line drawing edited by the line drawing editing section 5, the line numbers making up the line drawing are approximated to straight lines or various curves in the line drawing modeling section 6, and then expressed using parameters. It looks like this. Fourth
The figure shows the processing in the line drawing editing section 5 and the effects of the processing in FIGS. 5(a) to 5(C).
First, a threshold value T for evaluating the variation and a threshold value T for evaluating the coefficient of the quadratic term are set (step 200). Next, in the line segment section of interest, the line segment is approximated by a quadratic curve using the least squares method (step 202).
The variation with respect to the next curve is determined by the average value of the square of the error within the line segment section (step 204). As shown in FIG. 5(a), the line segment P,
The error for the approximate quadratic curve between P, is shown,
If it is determined that the obtained variation is smaller than the threshold T1 (step 206), linearity is evaluated by the coefficient of the quadratic term of the approximate quadratic curve (step 212). .

That is, if the coefficient of the quadratic term is smaller than the threshold value T2, it is assumed that the line is a straight line, and the straight line is recalculated by the least squares method (step 214). Furthermore, if the coefficient of the quadratic term is 12 or more, the approximate quadratic curve is stored as it is for that line segment (step 216). however,
If the determined dispersion is greater than or equal to the threshold T1, the inflection point of the line segment is determined from the quadratic differential function of the line segment (step 208), and the section is divided at the inflection point. (step 210). For each line segment section divided at an inflection point, the same process as above is performed until the dispersion in all sections is within the threshold T□, that is, it can be approximated to a quadratic curve or a straight line. It is repeated until. FIG. 5(b) shows the line segment P. After finding the inflection point P2゜P3 at P, the line segment P. This figure shows the case where approximation is performed for 28 times. Figure 5(c)
Also P. This figure shows the case where approximation is performed for 23. However, even after coordinate transformation, quadratic curves or lines that cannot be approximated to a straight line (such as closed curves) are now approximated to circular arcs or elliptical arcs using the method of least squares. There is.

As described above, end points are further searched for each parameter-expressed line segment, and the color of the end point of each line segment is displayed in a color different from that of the line segment. At that time, if there is another line within a certain number of pixels specified by the operator from the end point, or if the extensions of several lines intersect within a certain number of pixels, the line will automatically extend to that position. has been extended and the results will be displayed again. Furthermore, for a line with a partially missing part, the operator specifies the end point and the point of the extension light using a mouse, etc., and connects these two points to complete the missing part and create a line with the missing part completed. The coordinates of the end points are memorized.

The line drawing obtained through the above series of processing is processed by the closed surface extraction unit 7.
The surface is extracted as a closed surface and recognized as a surface shape on a two-dimensional image, but the surface extraction is performed as shown in FIG. In other words, if we take the line drawing shown in Figure 6(a) as an example, first arrows with different directions are attached to the line numbers that make up the line drawing, and then arrows pointing from point a are attached. The arrows are followed in the direction, and the arrows that have already been passed are erased. If, during the passage, the path branches off to the enemy at a certain point, the pursuit is carried out by taking the path furthest to the left in the direction of travel. For example, in the case of a point, tracking is performed in the direction of point C. This results in closed plane a-b-c
-defa is extracted, but extraction of this closed surface is equivalent to extraction of the background. This is because everything other than the closed surface is considered to be the background. Next, see Figure 6 (
As shown in b), if the remaining arrows are traced again from point a, the closed surface afba will be extracted. By following the above rules and following the lines until all the arrows are erased, the background (other than the closed plane a-b-c-de-f-a shown in Figure 6(a)) ) and closed surfaces (the closed surfaces shown in FIGS. 6(e) to 6(e), respectively) are extracted. Each extracted closed surface is labeled as a set of lines constituting itself. After that, among the extracted closed surfaces, the one that includes a pattern is designated by the operator. As a result, for a surface including a pattern, the RGB values of each pixel within that surface are stored as is. In other words, the pattern is saved, and for a monochromatic surface for which no pattern is specified, the average value of the RGB values of each pixel within that surface is stored.

As described above, after obtaining the two-dimensional shape information of the target object, the following processing is performed to develop it into three-dimensional data.

First, in the visual coordinate calibration unit 8, a three-dimensional coordinate system (camera coordinate system) as a reference is established, and a point P in this three-dimensional coordinate system is set.
A conversion formula (A) for converting (X, Y, Z) to coordinates v(1, J) on the image corresponding to that point is determined. For this purpose, we first model the visual system.

As an example, as shown in Fig. 7, the position of the camera viewpoint (lens focus) is
, a three-dimensional coordinate system (camera coordinate system) C is taken in which the visual axis (the visual axis of the lens) is the Z axis. Also, assume that there is an image screen at a distance f from the viewpoint that is orthogonal to the visual axis at the origin, and this is defined as the image coordinate system (■). In this case, the relationship between the image coordinate system and the three-dimensional coordinate system is V=M' *P ・
... Conversion formula (A) However, ■ Image coordinate P: Three-dimensional coordinate M': Represented by a perspective transformation matrix. Here, if we know the perspective transformation matrix M', we can determine to which position on the image the point P on the three-dimensional coordinates is projected, or conversely, we can determine the straight line (line of sight) in the three-dimensional space projected onto the point V on the image. L) It allows us to know the position of a point on the top.

Therefore, in order to obtain the perspective transformation matrix M',
The processing shown in FIG. 8 is performed.

First, four points that are visible on the image and whose mutual spatial positional relationship is clear are selected, and their positions on the image are determined (802). Here, we prepared a visual calibration mat with red, blue, green, and yellow marks drawn at each vertex of a 30 cm square to make the mutual positional relationship clear, and looked at this mat together with the object. and the positions of the centers of gravity of these marks on the image screen V r (Ir, Jr), Vb (Ib, Jb), V
g (Ig, Jg), Vy (Iy, Jy) 04 points are calculated.

Then, from the formula V=M'*P, 3 of these four points
Find the position on the dimensional coordinate system C as shown below (8
03). In FIG. 8, Vr on the image.

Four line-of-sight equations Lr (
Kr), L b (Kb), L g (Kg),
Ly(Ky) is calculated from V=M'*P. Here, P r' , P b' , P g'P on the space
y′ are line-of-sight equations L r (Kr) and L b , respectively.
(Kb), Lg (Kg), Ly (Ky), and from the mutual positional relationship, the side Pr'
Pb' and side Py'Pg' are parallel and have the same length (equal vectors), and side Pr'Pb' and side Pg' Pb
Three conditional expressions hold true: the angle formed by ' is 90 degrees (inner product 0), and the spatial distance between Pr' and Pb' is 30r-rn.

Therefore, by solving Kr, Kb, Kg, and Ky from these, the positions Pr and P on the three-dimensional coordinate system C are
b', P g', P y' and f can be determined.

As described above, the temporary three-dimensional coordinate system x'-y'-z
After determining the conversion formula V=M'*P, the origin and direction of the three-dimensional coordinate system are aligned to clarify the correspondence with images seen from other viewpoints (step 804).
. Therefore, taking the center position of the four marks as the origin, coordinate transformation is performed to a three-dimensional coordinate system X-Y-Z such that the direction from red to blue is +Y and the direction from green to blue is +X5, and the conversion formula V Find =M*P. 7) This transformation determinant M is stored 17 for each image as visual calibration data.

By aligning the three-dimensional coordinate systems of each image in this way, if the same point in space is visible in two images, by solving the intersection of the two line-of-sight equations,
Coordinates on the three-dimensional coordinate system X-Y-Z can be determined. Note that a visual calibration mat is used in the above example, but if it cannot be used, select one plane of the target object visible on the image and calculate the actual three-dimensional space of that plane. The operator may instruct the size and shape on the screen. For example, if a simple reference plane such as a rectangle is selected and the operator specifies the side size, etc., results similar to those obtained when a visual calibration mat is used can be obtained.

After the visual system has been calibrated as described above, the surface correspondence determination unit 10 performs correspondence between surfaces obtained from a plurality of images. , no processing is performed in the surface handling section 10. However, if this is not the case, the correspondence between the faces is processed, but if this process is explained using the example shown in Figure 9, Figure 10 shows the process in that case. be. First, if the same plane is visible in two or more images, one set of such planes is designated by the operator. More specifically, first, the operator clicks the plane SO in the image 91 and the plane SO' in the image 92 with a mouse, thereby indicating the correspondence between the pair of planes (step, knob 901). Next, the correspondence of vertices on these planes is determined (step 902). In this case, since the arrangement of the vertices on each plane is known from the two-dimensional image data, one vertex (for example, VO) on the plane SO is the vertex ■0° to V3' on the plane SO°. If you decide which one of these corresponds, you will know the correspondence of all the vertices on that plane. Therefore, it is necessary to evaluate which of the vertices VO° to V3° on the plane SO′ corresponds to the vertex vO. To explain this evaluation method, for example, in FIG. , Vl' actually correspond, the line of sight passing through the vertex VO in the image 91 and the vertex Vl in the image 92
The line of sight passing through ' will either intersect in three-dimensional space or approach closely (due to calculation errors, etc.). That is, the shortest distance between lines of sight should be short.

Therefore, the vertex vO is sequentially made to correspond to the vertex VO° to the vertex V3° on the plane SO°, and the sum of the shortest distances between the lines of sight at each of the four vertices in each case is calculated as the error amount. By choosing the one with the least amount of error,
The combination of vertices with the best correspondence is sought. Next, after determining the correspondence between a set of planes (including vertex relationships), from the connections between the respective lines and planes on the images 91 and 92, the images 91 and 92 are created using that plane as the starting point. The correspondence between the surfaces above is expanded (step 903).

For example, from the correspondence of the vertices on the image 91 and the image 92, the correspondence of the sides L2 and Llo is determined. From the correspondence relationship between sides L2 and Llo, these sides L2
, Ll' can be easily searched for. These planes S2 and S1° appear to actually correspond when interpreted from the two-dimensional image display, but as shown in Figure 11, when the connection of the planes is displayed discontinuously, It is possible that the system may not be compatible. Furthermore, when creating a line drawing using a two-dimensional image, the edges may not be cut out well. Therefore, it is evaluated whether or not the searched surfaces S2 and SL' actually have a corresponding relationship (step 904). First, if the number of vertices of the searched surfaces S2 and SL' are different, it is determined that there is no correspondence, but if the number of vertices is the same, the sum of the shortest distances between the lines of sight at each corresponding vertex is determined. is now in demand. If this sum is large, it is determined that there is no correspondence between these surfaces, but if this sum is sufficiently small, it is determined that a correspondence exists. If it is determined that a correspondence exists, the correspondence between these surfaces is stored. As can be seen from the above, each time the correspondence between new surfaces such as surfaces S2 and Sl' is known, the correspondence among the surfaces adjacent to these surfaces is determined in the same way as above. , one after another, the correspondence between the surfaces is expanded. The problem arises when the detection of correspondence between surfaces comes to a dead end in the process of expanding this correspondence. In such a case, the corresponding surfaces detected so far are displayed on the display screen (step 905). Based on this display, if there are other corresponding surfaces [7], those surfaces are designated by the operator.

Specified surface as new starting point As such, it is sufficient to repeat the above process.

If the line drawing editing is the cause of the failure in surface correspondence detection, the operator can return to the line drawing editing section 5 and correct the line drawing. In this case, which part is incorrect depends on which aspects of the correspondence were detected.
It is easy to know based on the

Now, after determining the surface correspondence between the plurality of images, the plane processing section 11 determines the three-dimensional coordinates of the plane portion. In the plane processing unit 11, first, the viewed surface three-dimensional data detection unit 13 solves the intersection of the line of sight of each corresponding vertex for the plane that can be seen from a plurality of viewpoints determined by the surface normal determination unit 10, thereby determining the plane part. The three-dimensional coordinate values corresponding to each vertex are calculated. For planes that can only be seen in one image, the three-dimensional coordinate values corresponding to the vertices are determined by the important plane processing unit 14.

In the important plane processing section 14, first, the surface normal direction determining section 15 is configured to give surface normal directions for all important planes. By the way, what should be noted about this normal direction is that in reality, the normal direction of many planes is often the same direction as the normal direction of a certain important plane, or a direction perpendicular to it. , is a fact. Therefore, first, the viewed surface three-dimensional data detection unit 13 obtains the
For important planes that have the same normal direction as a plane whose position and orientation are already clear in space, the normal direction is copied according to an instruction from the operator. Furthermore, if the angle with respect to a plane whose position and orientation are clear is known, the operator can give that angle. With such simple operations, the normal direction of most important planes can be easily determined. However, for planes whose normals cannot be easily determined as described above, the operator selects three arbitrary vertices on the plane and instructs the positional relationship of these vertices ( For example, two sides and the angle between them). In this way, for the three vertices on the plane,
By knowing the positional relationship between the coordinates on the image and the three-dimensional space, the position and orientation on the three-dimensional coordinates of this plane can be calculated using the coordinate transformation formula (A ). In addition, at this time, if necessary, the operator instructs a set of parallel sides when the surface is rectangular, or specifies features of the shape of the surface such as symmetry.

After the normal to each single view plane is determined by the surface normal determining unit 15 as described above, the important plane three-dimensional data detecting unit 16
, the three-dimensional coordinate values of the vertices on each of these single viewing planes are determined. In this process, first, the viewed surface three-dimensional data detection unit 13 detects the focused surface Sl that is in contact with the surface SO whose position and orientation have already been determined, and 3 From the dimensional coordinate values and the normal data for the surface obtained by the surface normal determination unit 15, the position and orientation of the surface S1 in the three-dimensional space-A can be determined. Next, the solution of the plane equation of the surface 81, the line of sight equation passing through each vertex of 81, and the three-dimensional coordinate values of each vertex (4) and (5) are obtained, and these are stored in the three-dimensional data storage unit. It has become. In this way, after the three-dimensional data for the important plane Sl has been obtained, it is sufficient to sequentially expand the data to the next plane S3 in the same way.

By the way, although the above method is mainly aimed at surfaces composed of planes, it cannot be directly applied to free-form surfaces. Therefore, when the target object has a free-form surface, the free-form surface processing section 1.2 directly uses the three-dimensional data obtained from the range finder input section 17, or the curved surface shape input section 18 uses the free-form surface processing section 1. If the object represents a simple shape such as a cylinder, the operator will specify the shape. Once the shape is known, the pose that best fits it on the image is found, and furthermore, the position of that curved surface in 3D space is found from the 3D position data of the surfaces that are in contact with that curved surface. It has become something that can be done.

By the way, with each of the above methods, a fixed target object is imaged from various viewpoints, and a three-dimensional model of the target object is created from the images obtained. This is a problem if the shape of the target object is too large. In this way, when a common reference coordinate system cannot be established on each image, the three-dimensional data integration unit 19 performs processing for this. That is, in this process, the overlapping part between the two images is first searched for, and the operator instructs the corresponding points on each image for the three points that are not on a straight line in that part. There is. From these correspondences, correspondence of coordinate systems between these images is required. Therefore, it is only necessary to sequentially accumulate such corrections between images. To be more specific, if we take a very large object, such as a landscape, we can obtain several partial images by slightly overlapping the landscape, and then connect these partial images. , it is also possible to obtain the landscape as a whole.

Now, the structure definition unit 20 defines the structure of the input object model by decomposing it into parts. Specifically, all sides of the part that are considered to be the same part are clicked with the mouse,
Each part is given a unique name. The model structure is defined by grouping surfaces as elements that constitute each part of the displayed model. For example, assuming that the model is an airplane, individual surfaces such as surfaces forming the main wing, surfaces forming the fuselage, and surfaces forming the tail are grouped into one or more groups.

As described above, by integrating the three-dimensional coordinate values in the three-dimensional data integrating unit 19, the overall model shape of the input target object is known, and furthermore, the structure defining unit 20 defines the model structure. However, the definition results can be highlighted from any direction by changing the viewpoint for each part designated from the outside by the recognition result output unit 21. In the recognition result output unit 21, if a certain part of the displayed model is specified with the mouse, the entire part including that part can be displayed on the display 15 in a different color from other parts. , FIG. 12 shows a case where an airplane 50 as a sample is input as a model by the object model input device according to the present invention, and then its main wing is output and displayed as a processing result. As shown in the figure, by moving the viewpoint with the mouse, the display can be viewed from any direction.

Once you understand the structure of the input model, you can not only highlight each part of the car, but also change the shape, position, color data, etc. of the entire part at once if the corresponding part is specified. be. Therefore, it is easy to generate various new models as modified models based on the input model.

Finally, to briefly explain the arrangement of various sensors such as cameras with respect to the target object, FIG. 13 shows an example of the arrangement of the various sensors. As shown in the figure, a transparent net 52 is attached to the inside of the frame of the stage 51, but the sample (in this example, a model airplane)
50 is placed on the transparent net 52 so that a color image of the sample 5o can be obtained from one or more cameras 55. In addition, slit floodlights 59 are appropriately arranged around these cameras 55, and if necessary, a range finder for distance measurement or two-dimensional information drawn on a plane (especially patterns) can be used. )
A scanner for reading will also be installed. Image data from the camera 55, curved surface shape data from the range finder, and two-dimensional information from the scanner are processed by the image processing means 58 as appropriate according to manual operations from the operator via the command input section 57. The processing results are output and displayed on the display 56.

〔Effect of the invention〕

As explained above, according to claim 1, when converting surface shape and color data on an image into shape and color data in a three-dimensional space, constraints that cannot be understood by a computer alone are set. By teaching from the outside, a model of the target object can be uniquely obtained. For example, if half of the shape is found from one image, just by teaching symmetry, the shape of parts that are not obvious on the image can also be obtained.

In addition, in the case according to claim 2.3, after inputting a three-dimensional model, by defining the structure of the input model in units of groups, a certain part of the target object can be highlighted from any direction. In addition, it is now possible to perform bracket transformation, and this bracket transformation allows a new model to be easily created from the input model.

Furthermore, according to claim 4.5, even if the target object has a two-dimensional shape, that is, a color photograph, etc., or even if a part of the target object is a curved surface, the The shape of the target object can be obtained.

Furthermore, according to claim 6, when a plurality of images are obtained, by combining the surface shape and color data obtained from each of the images,
In the case of only one image, a three-dimensional object model of the target object can be easily obtained by teaching the normal direction of the surface from the outside.

According to claim 7, a three-dimensional object model can be obtained while preserving the pattern drawn on the surface.

Further, in the case according to claims 8 to 16, when recognizing a surface constituting a target object, surface shape data on an image can be easily obtained by color region division from a color image and edges extracted from a grayscale image. , and you can definitely get it.

[Brief explanation of the drawing]

FIG. 1 is a diagram showing a general configuration of an example of an object model input device according to the present invention, FIG. 2 is a diagram showing color region division processing based on hue in one component, and FIG. 3 is a diagram showing the processing. FIG. 4 is a diagram showing an example of a display screen for explaining the
Similarly, FIGS. 5(a) to (e) are diagrams showing the curve approximation processing in the line drawing modeling unit as one component, and FIGS. 6(a) to (e) are diagrams for explaining the processing contents. is a diagram for explaining the closed surface extraction method with a specific example, and FIG.
Similarly, FIGS. 8(a) and 8(b) are diagrams showing examples of the visual system model in the visual coordinate calibration unit as one component.
Figures 9 and 11 are diagrams for explaining the method and processing for determining the conversion formula shown in the figure, and Figures 9 and 11 are diagrams for explaining the processing in the surface correspondence determining unit, which is also one component. ,
FIG. 10 is a diagram showing the surface correspondence determination process, FIG. 12 is a diagram showing an example of the recognition result display in the recognition result output section for the same component, and FIG. 13 is a diagram showing the recognition result display for the same component. 1
2 is a diagram showing how various sensors such as cameras are arranged with respect to a target object in No. 2; FIG. 1... Color image recognition section, 2... Image human horn section, 3
. . . Color area division unit, 4 . . . Grayscale image differential processing unit, 5 . . . Line drawing editing unit, 6 .
...Closed surface extraction section, 8...Visual coordinate calibration section, 9...
two-dimensional data storage section, 10... surface correspondence determination section, 11.
...Plane processing section, 2...Free-form surface processing section, 13...
・Viewed surface 3D data detection unit, 14... Single view surface processing unit, 15... Surface normal determination unit, 16... Emphasis 3D data detection unit, 17... Range finder data input ], 8... Curved surface shape input unit, 19... Three-dimensional data integration unit, 20... Structure definition unit, 21... Recognition result output unit, 55... Color TV camera

Claims (1)

[Claims] 1. A means for inputting an image of an object, and a means for recognizing surfaces constituting the object surface by surface based on instructions from the outside from the image, and generating shape data corresponding to the surfaces on the image and means for holding color data, means for displaying the shape and color corresponding to the surface on a display and changing it based on external operation input, and means for displaying the shape and color data of the surface on the image in three-dimensional space. means for converting the data into shape and color data, means for externally inputting the information necessary for the conversion, and displaying the data converted in three-dimensional space on a display.
In addition to displaying the image dimensionally, it is equipped with a means for changing the image based on an external operational input, and a means for adding a surface that does not exist in the input image to a surface already obtained based on the external operational input. An object model input device configured as follows. 2. Means for defining one or more recognized surfaces constituting an object into one or more groups, and shape, position, and color data corresponding to the one or more surfaces belonging to each group. 2. The object model input device according to claim 1, further comprising means for collectively changing the object model input device. 3. In relation to the means for defining one or more recognized surfaces constituting an object into one or more groups, for each group of defined surfaces, a part name is assigned as a component of the object. 3. The object model input device according to claim 2, further comprising means for defining, and means for highlighting all the faces of the same group as one part from an arbitrary direction in three-dimensional space. 4. The means for inputting the image of the object is a range finder,
4. The object model input device according to claim 1, comprising a scanner, one or a plurality of color TV cameras whose relative positions are known in advance, or a combination thereof. 5. Regarding means for converting surface shape and color data on an image into shape and color data in a three-dimensional space,
5. The object model input device according to claim 4, further comprising means for directly obtaining three-dimensional shape data on a portion of a free-form surface from a range finder, and means for fitting the curved surface to a basic primitive to obtain three-dimensional data. 6. Regarding means for converting surface shape and color data on an image into shape and color data in a three-dimensional space,
Determine the three-dimensional coordinate position of the reference surface given the shape by using a visual calibration mat or by teaching the shape of the reference surface from the outside and solving the line-of-sight equation determined from each image. means for finding correspondence between images with respect to other surfaces, determining or calculating three-dimensional coordinate positions, and displaying these on a display;
If there is a misrecognition, there is a means to correct the line drawing that caused the misrecognition based on an external instruction, and when there is only one image, the data in three-dimensional space can be corrected by teaching the normal direction of the surface from the outside. Claims 1 to 3 further include means for converting the
5. The object model input device according to any one of 5. 7. Shape data is created in connection with a means for recognizing surfaces constituting an object surface by surface from the input image based on instructions from the outside, and retaining shape and color data of the surfaces on the image. means for externally specifying whether it is a single color or a pattern as the color data of the surface; and a means for determining and holding the RGB value of the surface from the RGB values of each pixel in the surface when the color is specified as monochrome. 7. The object model input device according to claim 1, further comprising means for holding the RGB values of each pixel in the plane as they are when a pattern is specified. 8. Recognizes surfaces constituting an object surface by surface from an input image based on instructions from the outside, and retains shape and color data of the surface on the image. A method for dividing color regions using the intensity, saturation, and brightness of RGB values, a means for extracting edges from a grayscale image, and a method for dividing a surface on an image by combining the data of the extracted color region boundaries and grayscale edges. 8. The object model input device according to claim 1, further comprising means for creating shape data. 9. In relation to the means for performing color region division, there is a means for creating a color palette to be used for color region division from a region range of the same color specified on an image from the outside, and a means for creating a color palette to be used for color region division from a region range of the same color specified on an image from the outside, and Claim 8, further comprising means for displaying, means for externally specifying an area range in which color area division is inappropriate, and means for creating a new color palette within the specified area. The object model input device described. 10. In relation to the means for performing color area division, there is a means for making an area having less than a specified number of pixels into the same color area as the surrounding area, a means for inputting the specified number of pixels from the outside, and a means for inputting the specified number of pixels from the outside. 10. The object model input device according to claim 8, further comprising means for sequentially displaying results of color region division that change according to the number of pixels. 11. In relation to the means for performing color region segmentation, there is provided a means for performing color region segmentation by selectively selecting information on hue, intensity of RGB values, saturation, and brightness for the entire image or a specific region. The object model input device according to any one of claims 8 to 10. 12. In relation to the means for performing color region segmentation, hue;
A means for processing RGB intensity, saturation, and luminance information in different ways, a means for simultaneously displaying a plurality of color region division results on a screen in a multi-window, and a means for displaying a selected portion in the plurality of color region division results. Claims 8 to 8 further include means for externally specifying and means for combining the specified portions to obtain one color area segmentation result.
12. The object model input device according to claim 11. 13. Regarding means for extracting edges from grayscale images,
means on one side to obtain a differential image by differentiating the gray scale image; means to binarize the differential image using a threshold value and display a binarized edge image; a means for sequentially displaying binarized edge images corresponding to the corresponding value, a means for selectively specifying one of the means from the outside when there is a plurality of means for obtaining a differential image, and a binary value corresponding to the means for obtaining a differential image. The present invention is equipped with means for simultaneously displaying converted edge images on a multi-window screen, means for externally selecting any part of the displayed edge image, and means for combining the selected parts to obtain one binarized edge image. The object model input device according to any one of claims 8 to 12, wherein: 14. Regarding means for extracting edges from grayscale images,
A means for removing line segments having a length equal to or less than a threshold value from among the extracted edges, and a means for sequentially displaying edge images according to the threshold value specified as a variable from the outside are provided. An object model input device according to any one of claims 7 to 12. 15. In relation to the means for creating surface shape data on an image, there is a means for detecting the end point of a line, and another line exists within a certain number of externally specified pixels from the end point. a means for extending the line to the existing position and displaying the result, a means for displaying the endpoint, a means for accepting an external operation to connect the endpoint with another line, and a surface surrounded by the line when the endpoint is connected. The object model input device according to any one of claims 8 to 14, further comprising means for extracting the surface and labeling the surface. 16. Related to the means for creating surface shape data on an image, the extracted line is divided into straight lines, circular arcs, elliptical arcs, 2
Claims 8 to 1 further comprising means for approximating any of the following curves, and means for storing and retaining parameters of the approximated line.
5. The object model input device according to any one of 5.