JP2011138267A - Three-dimensional image processor, three-dimensional image processing method and medium to which three-dimensional image processing program is recorded - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a technology which is sufficiently reliable for automatically modeling a three dimensional object from a two dimensional image using a software program. <P>SOLUTION: A three dimensional image processor reads a two dimensional image, displays the two dimensional image, and specifies a position to becomes a three dimensional reference on the same two dimensional image by receiving an operation input, and a conversion relationship acquiring means acquires a conversion relationship between actual three dimensional space and an two dimensional image space. Next, a shape of a boundary of a surface of the subject matter is specified by discrete point positions of a number corresponding to the shape of respective segments for respective segments on receiving an operation input on the displayed image and a value of height is acquired on receiving the operation input. Then, the boundary is specified by calculating the position information in three dimension of the whole section based on the discrete point positions that are specified for respective segments and that number. In this case, the three dimensional model of the subject matter is calculated based on the value of height, the calculated position information of the boundary of respective segments and the conversion relationship. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a 3D image processing apparatus, a 3D image processing method, and a medium on which a 3D image processing program is recorded.

In many robotics or computerized visualization applications, it is important to model a particular type of object (eg, a product being manufactured in a robotics application) very quickly.
The first method is an attempt to model an object from a video. There, the user can intuitively trace the boundary of the surface of the object, and the software automatically rebuilds the model. In the present specification, unless otherwise specified, an image taken by a camera is referred to as a video.
As Patent Literature 1, “system and method for three-dimensional modeling” is known.

This patent describes a push / pull technique that allows a user to create a volume from a predetermined two-dimensional region.
This patent describes a method by which a user can interactively create a 3D volume from a 2D shape.
Further, as Patent Document 2, “a system and a method for modeling a three-dimensional object from one video” is known.
This patent describes a method (e.g. a face model) in which the predecessor model is already available for the modeled object.

This automatically finds a feature from the video and matches it to the model so that the found feature can be described as most appropriate.
Further, as Patent Document 3, “an interactive model generation apparatus and method using a plurality of videos” is known.
This patent only describes an architecture that allows a user to work interactively with a software program to generate a three-dimensional model of an object from multiple videos. In this case, no implementation method is provided for generating primitive functions.

  In addition, as a non-patent document 1, Journal of Mathematical Imaging and Vision (2009/11) discloses a technique “a simple method of calculating the center of radial distortion (distortion) and correcting the radial distortion”.

US Patent US 6,628,279 US Publication No. 2007/0127810 US Publication 2002/0080139

Journal of Mathematical Imaging and Vision (2009/11), easy method for calculating the center of radial distortion and correcting radial distortion

The technique shown in Patent Document 1 is based on using a predefined set of primitive functions in order to specify a two-dimensional region. Therefore, the more complex the object to be modeled, the less intuitive the modeling operation.
With the technique shown in Patent Document 2, a specific type of object can only be modeled, and a general type of object cannot be modeled.
The technique shown in Patent Document 3 is limited to building a model with a small set of primitive functions of simple shapes, and can only model simple objects. Also, the predetermined primitive function used to calculate the rotation matrix between multiple videos can itself be a source of error for the system, thus making the previous calculation less reliable.

  The technique shown in Non-Patent Document 1 merely describes a method for calculating a primary radial distortion parameter based on the center of radial distortion and a line that should be a straight line in real space.

  The present invention is a three-dimensional image processing apparatus for calculating a three-dimensional image from a two-dimensional image, the image reading means for reading the two-dimensional image, and displaying the read two-dimensional image, receiving the operation input and the same. On the display image, a reference position specifying means for specifying a position serving as a three-dimensional reference on the image, a conversion relation acquiring means for acquiring a conversion relation between the three-dimensional real space and the two-dimensional image space, Tracing means that accepts operation input and specifies the shape of the boundary of the surface of the object for each segment by the number of discrete point positions corresponding to the shape of each segment, and accepts operation input and sets the height value Height value acquisition means to be acquired, and boundary specification means for specifying the boundary by calculating the three-dimensional position information of the entire section based on the discrete point positions designated for each segment and the number thereof And the height value and calculation The position information of the boundary of each segment, it is constituted comprising a three-dimensional model reconstruction means for calculating the three-dimensional model of the object based on the above conversion relationship.

  In the above configuration, when the image reading means reads a two-dimensional image, the reference position specifying means displays the read two-dimensional image, receives an operation input, and designates a three-dimensional reference position on the image. Then, the conversion relationship acquisition means acquires the conversion relationship between the three-dimensional real space and the two-dimensional image space. Next, the tracing means accepts an operation input on the display image and designates the shape of the boundary of the surface of the object with the number of discrete point positions corresponding to the shape of each segment for each segment, The height value acquisition means receives an operation input and acquires a height value. As a result, when the boundary specifying unit calculates the three-dimensional position information of the entire segment based on the discrete point positions and the number of points specified for each segment and specifies the boundary, the three-dimensional model is restored. The construction means calculates a three-dimensional model of the object based on the height value, the calculated position information of the boundary of each segment, and the conversion relationship.

As described above, the method for calculating the three-dimensional model of the object is not necessarily limited to an actual apparatus, and it can be easily understood that the method functions, and the method is also effective. There is no.
By the way, such a three-dimensional image processing apparatus may exist alone, or may be used in a state of being incorporated in a certain device. Is included. Therefore, it can be changed as appropriate, such as software or hardware.

When the software of the three-dimensional image processing apparatus is implemented as an embodiment of the idea of the invention, it naturally exists on a recording medium on which such software is recorded, and it must be used.
Of course, the recording medium may be a magnetic recording medium, a magneto-optical recording medium, or any recording medium that will be developed in the future. In addition, the duplication stages such as the primary duplication product and the secondary duplication product are equivalent without any question. In addition, even when the communication method is used as a supply method, the present invention is not changed.

Further, even when a part is software and a part is realized by hardware, the idea of the invention is not completely different, and a part is stored on a recording medium and is appropriately changed as necessary. It may be in the form of being read.
When the present invention is realized by software, a configuration using hardware or an operating system may be used, or may be realized separately from these. For example, even if it is a variety of arithmetic processing, if it can be processed by calling a predetermined function in the operating system, it can also be input from hardware without calling such a function. is there. It can be understood that the present invention can be implemented only by this program in the process in which the program is recorded on the medium and distributed even though it is actually realized under the intervention of the operating system.

  When the present invention is implemented by software, the present invention is not only realized as a medium storing a program, but the present invention is naturally realized as a program itself, and the program itself is also included in the present invention.

The present invention can provide a novel method for a user to quickly and easily trace a boundary line in a video in order to obtain a detailed model of a wide variety of objects consisting of a non-tilted top and bottom surface, In this technique, an object having a bent or hollow part can also be modeled.
Further, according to the present invention, a three-dimensional image can be converted from a two-dimensional image using the intersection of parallel lines orthogonal to each other in three dimensions from the two-dimensional image information, and the origin, height and outer edge of the three-dimensional image. Can be calculated.

It is a component diagram of a system. It is a flowchart of processing software. It is a figure which designates the position used as a three-dimensional reference on a two-dimensional image. It is a figure which determines one unique circle. It is a figure which shows the interpolation condition. FIG. 10 is a diagram in which a fitting process is performed. FIG. 10 is a diagram in which a fitting process is performed. It is a figure which shows the example in which a negative trace is used. It is a figure which shows the example in which a negative trace is used. It is a figure which shows another example in which a negative trace is used. It is a figure which shows another example in which a negative trace is used. It is a figure which shows the influence of radial distortion. It is a figure which shows the situation curved for radial distortion. It is a figure which shows the condition where only the edge pixel of a right direction remains. It is a figure which shows the area | region designated as the area | region in a segment. It is a figure which shows the area | region designated as the area | region in a segment.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings in the following order.
FIG. 1 is a component diagram of the system.
The three-dimensional object reconstruction step is executed by a processing software program in a processing device (for example, a computer).
The user supplies a video and a set of points that will trace the boundaries of the video's perimeter (if applicable, inside the top of the object). The manner in which the set of points is traced will be described in more detail later.

Processing software creates a reconstructed three-dimensional model of the object. It can also be used in various robotics, modeling or entertainment applications.
The processing software 1 includes a trace and point manager 2, a 3D model generator 3, and a texture manager 4. The trace and point manager 2 manages traces and points input by the user. The three-dimensional model generator 3 creates a three-dimensional model from given traces, points, and heights. The texture manager 4 manages the textures recorded for each surface of the 3D model.

FIG. 2 represents a flowchart of the algorithm that exists in the processing software.
Before describing each step in detail, a general outline of the object model process will be described.
Initially, the video is captured in a processing software program (image load) (step 1). Next, the user needs to define spatial coordinates for the same image for calibration (step 2).
Then, as an option, radial distortion correction may be performed (optional radial distortion correction) (step 3). The spatial coordinate information allows the absolute three-dimensional position in the real space to be calculated when the user specifies the video position and the height value in the real space.

  Next, the user traces the boundary line inside or around the object that is the image (traces the boundary of the surface) (step 4). Further, the position of the trace may be automatically adjusted based on the image information as an option (automatic point adjustment as an option) (step 5). Based on the trace, the processing software automatically reconstructs the 3D model of the boundary line using the height value (construction of 3D model) (step 6). If the user wants to trace more boundaries (does more traces be needed?), The tracking step is repeated (step 7).

Once all the boundaries have been traced, the texture of each visible surface is acquired from the video (step 8).
Then, in order to refine the model, or to obtain the appearance of parts of the object that were not previously visible, the user optionally captures more images of the object from different fields of view (step 9). For each additional video, the user needs to recalibrate (step 10).

In addition, the radial distortion is optionally corrected (radial distortion correction as an option) (step 11), and the trace position is automatically adjusted based on the image information (automatic point adjustment as an option) (step 12). Also good.
Then, the existing model is corrected from the current visual field (additional model correction) (step 13), and the texture of each surface visible in the current video is acquired (step 14). In addition, in the additional model modification, it is possible to add a new boundary from an image corresponding to a new visual field in addition to the boundary that can be visually recognized in the previous image. When a new boundary is generated in this way, the construction of the three-dimensional model in step 6 may be executed.

Details of each step will be described below.
First, (Table 1) shows a list of parameters used in this specification.

After capturing the video in step 1, camera calibration is performed in step 2. Here, Step 1 corresponds to image reading means, and Step 2 corresponds to reference position specifying means.
This is done by identifying two sets of parallel line segments in the image (in real space, each set line segment is perpendicular to the other set line segment). This is the same as the origin in the space reflected in the video. That is, since each line segment consists of two end points, the positions of four lines and the origin are determined, and two-dimensional position coordinates are provided for a total of nine points. As described above, two pairs of parallel line segments, and each pair designates a position serving as a three-dimensional reference by designating a line segment orthogonal to each other and the origin.

Based on these values, the processing software calculates a projection matrix for the image space from the real space, depending on the scale.
FIG. 3 is a diagram illustrating a process of designating a position serving as a three-dimensional reference on the video.
The square tags of the wavy line R and the wavy line G represent the end points of two sets of line segments (a total of four line segments). The round tag with a solid line represents the position of the origin when reflected in the image space. The processing software then automatically calculates the X, Y, and Z axes as reflected in the image space (shown as solid line G, solid line B, and solid line R, respectively).

In this embodiment, the mathematical framework for the calibration in step 2 is as follows. In the following, a conversion relationship between a three-dimensional real space and a two-dimensional image space is obtained, which corresponds to a conversion relationship acquisition unit.
In the pinhole camera model, a point in 3D space is mapped (acquired and pasted together) to a 2D point that is an image via the 3x4 element camera projection matrix P shown in Equation (1). Attached). Here, P is a projection matrix having 11 degrees of freedom, and can be decomposed into an internal camera matrix (K portion) and an external camera matrix ([RT] portion) shown in Equation (2). .

R and T are a 3x3 element rotation matrix and a 3x1 element translation vector, respectively. K is a 3 × 3 camera eigenmatrix as shown in Equation (3). Αu and αv are related to focal length and pixel squareness, s is a skew parameter, and x0 = (u0, v0) is a principle point of the image.
Here, it is assumed that the pixels of the image are square. That is, αu = αv = α is established. Further, since there is no distortion in the video, s = 0 is set, and in principle, the point x0 is located at the center of the video.

このフレームワークでは、ユーザーが映像中に原点と２つのセットの二対の線分を供給することで、カメラパラメータ（K、R、T）は縮尺に依存して解くことができる。ここで、各々のセットの２本の線分は互いに平行であり、２つのセットの線分は相互に垂直である。各々の線分は２点で表されるので、合計9ポイントがユーザーによって供給される必要がある。２セットの平行線から、各々のセットの消点を計算することができる。

In this framework, the camera parameters (K, R, T) can be solved depending on the scale by the user supplying the origin and two sets of two line segments in the video. Here, the two lines of each set are parallel to each other, and the lines of the two sets are perpendicular to each other. Since each line segment is represented by 2 points, a total of 9 points need to be supplied by the user. From two sets of parallel lines, the vanishing point of each set can be calculated.

The vanishing point of each set is the intersection of the two line segments of each set. These two vanishing points are defined as x1 = (u1, v1) and x2 = (u2, v2). From these two points and the video principal point x0 = (u0, v0), a third vanishing point x3 = (u3, v3) is determined. This is because the video principal point x0 is located at the centroid of the triangle formed by x1, x2 and x3.
First, it has been proved that the image principal point is a triangular vertice made by x1, x2 and x3. Considering the point at infinity in each of the three perpendicular directions, the equation shown in equation (1) is shown in equation (4). From there, the rotation matrix can be inferred as shown in equation (5).

回転行列の列要素は相互に直角であり、λi≠0（i = 1,2,3）なので、式（６）に示す式のとおりとなり、また、これらの式でαを消去すると式（７）に示す式となる。

この式のセットから、映像主点x0は、x1、x2とx3によってつくられる三角形の垂心であるということが証明される。第３の消点x3の位置を見つけるため、式（７）の中の最初の２つの式が使われる。
いくつかの単純な代数操作で、式（８）に示される式は演繹され、それから（u3、v3）の値が決められる。

Since the column elements of the rotation matrix are perpendicular to each other and λi ≠ 0 (i = 1, 2, 3), the equation is as shown in equation (6), and when α is eliminated in these equations, equation (7 ).

From this set of equations, it is proved that the video principal point x0 is a triangular centroid created by x1, x2 and x3. To find the position of the third vanishing point x3, the first two equations in equation (7) are used.
With some simple algebraic operations, the equation shown in equation (8) is deduced, and then the value of (u3, v3) is determined.

固有行列に残された最後のパラメータは焦点距離である。式（６）に示す３つの式のいずれかを使い、焦点距離を解くことができる。
たとえば最初の式を用いて、式（９）で示すように焦点距離を解くことができる。

つぎに、外部カメラ行列を解くことができる。回転行列の直行性と正規性を適用して、式（５）に示す制約は式（１０）に示すように表すことができる。そして、それは式（１１）に示すように解くことができる。以上で、回転行列は解かれる。

The last parameter left in the eigenmatrix is the focal length. The focal length can be solved using any of the three formulas shown in formula (6).
For example, using the first equation, the focal length can be solved as shown in equation (9).

Next, the external camera matrix can be solved. Applying the orthogonality and normality of the rotation matrix, the constraint shown in Equation (5) can be expressed as shown in Equation (10). And it can be solved as shown in equation (11). Thus, the rotation matrix is solved.

外部カメラ行列の並進ベクトルを発見するために、映像における原点が使用される。
原点の射影は、式（１２）で示すように表すことができる。

ここにおいて、これはλ0における自由度の一つである。このパラメータは原点から離れたカメラの並進に直接関連するものであるから、それは映像におけるピクセル対メーターの縮尺としての働きをする。事実、λ0の値が大きなものであるほど、カメラはより原点から離れ、その場面におけるオブジェクトはその映像上により小さく射影されることになる。この縮尺要素は、ある一つの線分（２つの終点を指定する）を映像に引き、実空間におけるその線分の実際の長さを特定することによって得ることができる。この縮尺値により、射影行列は１つのみとなる。

The origin in the video is used to find the translation vector of the external camera matrix.
The projection of the origin can be expressed as shown in equation (12).

Here, this is one of the degrees of freedom at λ0. Since this parameter is directly related to the translation of the camera away from the origin, it serves as a pixel-to-meter scale in the image. In fact, the larger the value of λ0, the further the camera is from the origin, and the object in the scene will be projected smaller on the video. This scale element can be obtained by drawing a certain line segment (designating two end points) on the image and specifying the actual length of the line segment in real space. With this scale value, there is only one projection matrix.

The radial distortion correction in step 3 will be described later.
The next step 4 is that the user traces along the boundary line outside or inside the top of the object. This trace corresponds to a tracing means. A trace is defined as a closed loop that defines the top boundary. That is, on the display image, an operation input performed by the user is received, and the shape of the boundary of the surface of the object is designated by the number of discrete point positions corresponding to the shape of each segment for each segment.

In this model, one trace is a predetermined solid having the same shape and the height of the top surface and the bottom surface existing in different xz planes. The user can then specify a height value between the top and bottom for each trace. Of course, the height value acquisition unit is configured by receiving the operation of inputting the height value by the user in this way. This height value is also input in step 4.
The reconstructed three-dimensional model of the object can consist of multiple traces, and furthermore, those traces can consist of different top and bottom height values.

A trace will later consist of one or more segments, and each segment will consist of two or more points provided by the user.
Here, there are two types of points that can be provided by the user, which are segment composition points, or segment division points. In the present specification, unless otherwise specified, a point located at the boundary between segments is referred to as a segment division point, and points constituting other segments are referred to as segment constituent points.
Each segment includes one or two segment division points (the first point of the trace is automatically taken as the segment division point). A segment that contains only one segment division point must have at least one segment constituent point. As a segment containing two segment split points, it may contain zero or more segment constituent points. In other words, each segment contains at least two points.

Depending on the number of points in the segment, as shown in Table 2, the curve of that segment (along the top surface of the object) is automatically interpolated for each segment in a different way.
For a segment containing two points, the segment is represented by a straight line. For a segment containing three points, a circle passing through the three points is fitted. Further points along the circle are then automatically interpolated between the two segment division points. Similarly, an axis parallel ellipse is fitted to a segment containing 4 points. For segments containing 5 points, a general ellipse is fitted. In cases where the number of segment points exceeds 5, this fitting system is an overdetermined system. In other words, there are more conditions than unknowns. In this case, the linear least squares method is used to fit the general ellipse.

特に、同一直線上にない３つの点があれば、（表３）に示すように使われているパラメータを使い、図４で示すようにユニークな一つの円が決定されることができる。その３点から、点P1と点P2、点P2と点P3とを結ぶことによって２本の線（laとlb）が引かれることができる。続いて、laとlbに直角で、それぞれの中間を通過する線（la'とlb'）ができる。la'とlb'の方程式を、式（１３）に示している。

In particular, if there are three points that are not on the same straight line, a unique circle as shown in FIG. 4 can be determined using the parameters used as shown in (Table 3). Two lines (la and lb) can be drawn from the three points by connecting the points P1 and P2 and the points P2 and P3. Subsequently, there is a line (la 'and lb') that is perpendicular to la and lb and passes between each. The equation for la ′ and lb ′ is shown in equation (13).

４つの点は式（１４）に示される形の軸平行楕円の方程式を一義的に定める。その楕円は中心（x0、y0）とし、それぞれxとy軸に沿った半径aとｂとなっている。 The center of the circle is the intersection of la ′ and lb ′, which solves the equation of α and β shown in equation (14) and replaces that value with the equation shown in equation (13). .
The radius of the circle is the distance from the center to any of the three points P1, P2 or P3.

The four points uniquely define the equation of the axis parallel ellipse having the form shown in the equation (14). The ellipse has a center (x0, y0) and radii a and b along the x and y axes, respectively.

５点あるいはそれ以上の点については、式（１８）で示すように分析的フォームによって記述される一般楕円が適合する。ちょうど５点の場合は、楕円の方程式が一義的に決定される。５点以上の場合は、このシステムは過剰決定体系となり、少なくとも最小二乗法で近似的に楕円を決めることができる。 To solve the ellipse parameters, the analysis form shown in Equation (16) is solved by four unknowns of four equations (the value of F is arbitrarily set to -1). By geometric analysis, the characteristic of the ellipse can be obtained as shown in equation (17).

For five or more points, the general ellipse described by the analytical form fits as shown in equation (18). In the case of exactly 5 points, the elliptic equation is uniquely determined. In the case of 5 points or more, this system becomes an overdetermined system, and an ellipse can be approximately determined by at least the least square method.

According to the geometric analysis, the center (x0, y0) of the ellipse is calculated as shown in equation (19).
In this case, the lengths of the major and minor axes and the inclination of the ellipse are not calculated. An analytical form of the equation is used to obtain the position of the point on the ellipse at a particular angle from the center (x0, y0). For the general ellipse, the axis parallel ellipse and the circle case, when at least three points are on one straight line, a unique solution may not exist. In that case, the order of general ellipse, axis parallel ellipse, circle, and straight line will continue to the next shape until such a curve is calculated successfully (the fitting as a straight line from the start point to the end point should always succeed) Is).

カーブを直線として表すのであれば、補間法は必要とされない。そのセグメントは、単に２つのセグメント分割点によって表現される。円、軸平行楕円、一般楕円の場合では、補間法は必要とされる。
図５で示すように、補間点の数nはまず式（２０）に示される方程式によって計算される。そこでは、始点角度θsと終点角度θeは適切な角度（2πを加えるか、引くことによって）となっている。それから、n個の点が、始点角度θsと終点角度θeとの間で等間隔となるようにして補間される。２つの終点がデフォルトで存在するので、そのセグメントは結局、（n＋2）点を持っていることになる。

If the curve is represented as a straight line, no interpolation method is required. The segment is simply represented by two segment division points. In the case of a circle, an axis parallel ellipse, or a general ellipse, an interpolation method is required.
As shown in FIG. 5, the number n of interpolation points is first calculated by the equation shown in equation (20). There, the start point angle θs and the end point angle θe are appropriate angles (by adding or subtracting 2π). Then, n points are interpolated so as to be equidistant between the start point angle θs and the end point angle θe. Since there are two endpoints by default, the segment will eventually have (n + 2) points.

図６と図７は、このフィッティングプロセスが実行される例を示している。なお、このフィッティングプロセスは、各セグメントごとに指定された離散的な点位置とその数に基づいて、当該区分全体の三次元での位置情報を算出して上記境界を特定する処理に相当し、境界特定手段を構成する。図６において、セグメント構成点、またはセグメント分割点は、それぞれ一点鎖線の十字と二点鎖線の十字で示されている。
セグメント１、３と５は、各々５点から成る。その結果、一般楕円が適合され、中間点は自動的に補間される。これらの補間された点は、天面から底面まで引かれている垂直の破線で示されている。

6 and 7 show an example in which this fitting process is executed. Note that this fitting process corresponds to a process of calculating the three-dimensional position information of the entire section and specifying the boundary based on the discrete point positions designated for each segment and the number thereof, The boundary specifying means is configured. In FIG. 6, segment constituent points or segment division points are indicated by alternate long and short dash lines.
Segments 1, 3 and 5 each consist of 5 points. As a result, the general ellipse is fitted and the midpoint is automatically interpolated. These interpolated points are shown as vertical dashed lines drawn from the top to the bottom.

On the other hand, since segment 2 contains only 4 points, the axis parallel ellipse is fitted and some points are interpolated.
Finally, since segment 4 consists of 3 points, the circles through these points are fitted.
Thus, as the correspondence between the shape of each segment and the number of discrete point positions, the straight line has two points, the circle has three points, the axis parallel ellipse has four points, and the general ellipse has five points or more.
Traces are one of two types and they are positive or negative. Positive traces add volume to the three-dimensional model, whereas negative traces subtract volume from the three-dimensional model (eg, when forming a cavity). That is, the designation of the boundary of the surface of the object includes designation of a solid shape boundary and designation of a hollow shape boundary. Then, in step 6, a solid shape calculation process and a hollow shape calculation process are executed, and a three-dimensional model is reconstructed with the combined result of both. Step 6 corresponds to a three-dimensional model reconstruction unit. The case where it is necessary to reconstruct the three-dimensional model due to the additional model modification in step 13 corresponds to the three-dimensional model restructuring means.

As a user operation, positive tracing is achieved by searching for points in a counterclockwise manner, and negative tracing is accomplished by searching for points in a clockwise manner.
8 and 9 show examples in which a negative trace is used.
In FIG. 8, the alternate long and short dash line trace is a positive trace, which follows the outer boundary of the object. On the other hand, the dash-dot line trace is a negative trace, which is traced along the internal boundary of the object. A segment containing 2, 3, 4 or 5 points can be found in either of the two traces.

10 and 11 show another example in which a negative trace is used.
In FIG. 10, there are four traces here. A positive trace on the outside and three negative traces on the inside.
Two of the negative traces have only one segment split point, but it has two other segment constituent points. Since there are a total of 3 points, the circle is fitted.
Since a wide variety of objects are symmetrical in the real world, at a point specified by the user, the point specified by the user can be automatically generated by flipping along a specific axis. In this way, the reconstructed three-dimensional model is guaranteed to maintain its left-right symmetry without manually correcting the position of the points by the user.

As described above, when an operation input by the user is received, a point position that maintains left-right symmetry is automatically generated by turning over the designated point position along a specific axis.
In the three examples shown above, the entire object is symmetrical, or at least a portion of the object is symmetrical. This feature simplifies the tedious and tedious task of requiring the user to carefully place video points at symmetrical positions.
The process software program can then not only add and remove traces, but also the user can add, modify and remove points. The user can also change the height value of the top and bottom at a later time, and the position of all points is re-evaluated.

In step 7, iterative determination is made and after all traces have been added, in step 6 the 3D model of the object is reconstructed and completed. That is, a three-dimensional model of the object is calculated based on the height value, the calculated position information of the boundary of each segment, and the conversion relationship.
The next step 8 is to obtain the texture of the model. Because of the visible surfaces in the video, the texture of each surface is mapped from the video. This process corresponds to a texture pasting unit that obtains a texture from a two-dimensional image based on position information of each surface when calculating the three-dimensional model and pastes the texture on each surface of the three-dimensional model. By executing Step 11 (Radial distortion correction), Step 12 (Automatic point adjustment), and Step 13 (Additional model correction), the boundary position information is corrected. Therefore, it is necessary to obtain a texture corresponding to the corrected boundary information and paste it on a predetermined surface. In this sense, when the boundary position information is corrected, step 14 of acquiring and pasting the texture again corresponds to the texture pasting means.

  If there are more images available in step 9 (in the same object from different fields of view), the positions of the nine points of the image (the two origins of two sets of two lines, respectively, the origin and the origin) Is repeated for the new video (step 10). In step 11 (and step 3) and step 12 (and step 5), radial distortion correction and automatic point adjustment are performed, which will be described later.

Since the model has already been reconstructed, if the texture is obtained and pasted in step 14, it should automatically be correctly projected onto the video.
In this case, the user may modify the existing model, add a trace for a part of the model that was previously disturbed, or obtain the texture of the surface visible in this frame. If there is a sufficient field of view, the texture of each surface of the entire model can be acquired.

7, 9, and 11 show the results of three-dimensional reconstruction in which textures are mapped to the (viewing) plane that can be seen in the video.
In this way, two-dimensional images from a plurality of different fields of view for the same object are read, the operation input is received for each image, and is pasted on each corresponding surface of the three-dimensional model. In addition to letting the user trace the surface boundaries, the object can also be modeled using a combination of predefined shapes (including general boxes, general cones, and spheroids). That is, the object is modeled using a combination of predefined shapes, general boxes, general cones, and spheroids.

Table 4 shows the modifiable properties for each of these forms.
All of these shapes allow modeling of a wide variety of objects, as well as the flexibility of tracing.

  By the way, in many situations, distortion is introduced into the image due to various effects of the lens. One of the most conspicuous effects is radial distortion, and a straight line in real space (not passing through the video principal point) is displayed so as to curve in the video. The method described above for reconstructing a 3D model of an object is based on straight lines or other primitive functions (in the calibration phase as well as in the reconstruction phase), thus reducing the side effects introduced by radial distortion, resulting in Is improved.

Hereinafter, as described in step 3 and step 11, a radial distortion removing unit that is a method of converting the positional information by removing the radial distortion included in the positional information of the two-dimensional image will be described.
The influence of radial distortion is shown in FIG.
The parameters used in the following description are shown in (Table 5).

  The radial distortion can be calculated based on the technique as described in Non-Patent Document 1.

歪められていない映像ポジション(x-u, yu)がゆがめられた映像ポジション(xd, yd)に対応する場合、rd{=(xd**2+yd**2)**(1/2)}はゆがめられた点の原点までの距離であり、λはラジアルディストーションパラメータである。 Here, the center of distortion and a term of radial distortion are calculated. In particular, it is first assumed that the center of distortion is located at the center of the video. Then, the radial distortion can be described using a division model as shown in equation (21).

If the undistorted video position (xu, yu) corresponds to the distorted video position (xd, yd), rd {= (xd ** 2 + yd ** 2) ** (1/2)} is The distance to the origin of the distorted point and λ is a radial distortion parameter.

ここで、kは傾きで、bがy切片である。あるいは、水平よりも垂直に近い線にとっては、yに関してxを表している方程式が得られる。式（２２）に示される式は式（２３）として書き直すことができる、そして、いくつかの代数操作で、式（２４）に示す式が得られる。この式は、このモデルのもとで、ゆがめられた直線が円になることを示している。 A straight line in an undistorted image can be expressed as in Equation (22).

Here, k is a slope and b is a y-intercept. Alternatively, for a line closer to vertical than horizontal, an equation representing x with respect to y is obtained. The equation shown in equation (22) can be rewritten as equation (23), and with some algebraic operations, the equation shown in equation (24) is obtained. This equation shows that the distorted straight line becomes a circle under this model.

歪曲の中心が中央にないならば、歪曲(x0, y0)のセンターはそれぞれxdとydをxd - x0とyd - y0と入れ替えることによって式（２４）に示される式に取り込むことができる。いくつかの代数操作で、式（２５）に示される式が得られる。あるいは、式（２６）に示される式を記述できる。

３本の線を使って、(Ai, Bi, Ci), i=1,2,3の値は式（２５）から計算されることができ、それから、歪曲(x0, y0)の中心は式（２７）で示すように計算することができる。最後に、ラジアルディストーションの値は、式（２８）で示すように解ける。あるいは、４本の線を用いた最小二乗法を使用できる。

If the center of distortion is not in the center, the center of distortion (x0, y0) can be incorporated into the equation shown in equation (24) by replacing xd and yd with xd-x0 and yd-y0, respectively. Several algebraic operations yield the equation shown in equation (25). Alternatively, the equation shown in equation (26) can be described.

Using three lines, the values of (Ai, Bi, Ci), i = 1,2,3 can be calculated from equation (25), and then the center of the distortion (x0, y0) is It can be calculated as shown in (27). Finally, the radial distortion value can be solved as shown in equation (28). Alternatively, a least square method using four lines can be used.

以上のようにして、二次元画像の位置情報に含まれる放射歪曲を除去して位置情報を変換することができるようになる。
次の問題は、円をエッジピクセルに対して安定してフィットさせることである。円の方程式が式（２９）で示すように解析的な形に書かれると仮定する。

エッジ点(xi, yi)から円までの距離は、式（３０）で示すように表すことができる。

さらに、式（３１）で示すように角座標を使ってa2とa3を書くことにより、式（３２）で示す目的の機能は、標準的なLevenberg-Marquardtアルゴリズムを使用することで、三次元空間(a1, a4, θ)で解くことができる。

As described above, it is possible to remove the radial distortion included in the position information of the two-dimensional image and convert the position information.
The next problem is to stably fit the circle to the edge pixels. Assume that the equation of the circle is written in an analytical form as shown in equation (29).

The distance from the edge point (xi, yi) to the circle can be expressed as shown in Expression (30).

Furthermore, by writing a2 and a3 using angular coordinates as shown in equation (31), the intended function shown in equation (32) can be achieved using a standard Levenberg-Marquardt algorithm in a three-dimensional space. It can be solved by (a1, a4, θ).

最後に、エッジピクセルを見つける処理を行う必要がある。すなわち、二次元画像からエッジ画素を検出し、同エッジ画素の位置情報から上記境界の位置情報を修正するものであり、ステップ５やステップ１２で行なっているポイント調整手段に相当する。
４本の線の各々ごとに、各線を囲んでいるピクセルからなる長方形の一片の中で、キャニーエッジディテクション(canny edge detection)を最初に実行する。エッジピクセルの強さと法線方向も記録される、そして、法線の向きからの角度差異が設定された閾値を超えるエッジピクセルは排除される。

Finally, it is necessary to perform processing for finding edge pixels. That is, the edge pixel is detected from the two-dimensional image, and the position information of the boundary is corrected from the position information of the edge pixel, which corresponds to the point adjusting means performed in step 5 or step 12.
For each of the four lines, canny edge detection is first performed in a rectangular piece of pixels surrounding each line. Edge pixel strength and normal direction are also recorded, and edge pixels whose angular difference from the normal direction exceeds a set threshold are eliminated.

The remaining edge pixels are considered to belong to the current line. If enough pixels as edges (above the threshold) are found in at least three of the four lines, the radial distortion correction process can proceed.
For example, as shown in FIG. 13, assume an image including an alternate long and short dash line that is to be curved for radial distortion. Assume that the user entered a two-dot chain line during the calibration phase. The dashed box indicates the region in which edge pixels are used for the calculation of radial distortion parameters.

When edge detection is performed, all edge pixels with a standard deviation greater than 45 degrees with respect to the input line are immediately discarded as isolates.
The direction of the remaining edge pixels may indicate one or two opposite general directions (in the case of double edges appearing close to each other). Of these two groups of edge pixels, only a larger number of groups are retained. That is, only the edge pixels in the correct direction remain, as shown in FIG.

If there is no radial distortion and there is no user input error, all edge pixels will be exactly on a straight line. With radial distortion, not all edge pixels are on a straight line. However, if the video is not correctly distorted with the correct value of the radial distortion parameter (and does not take any other form of distortion), the edge pixel will be on a straight line.
At this point, the distortion center (x0, y0) and the radial distortion parameter λ are calculated. The image can then be warped to correct radial distortion. At the same time, the positions of the points entered by the user (the origin and the end points of the four line segments) are adjusted as well, so they match the undistorted positions on the image. By correcting the radial distortion, the image information (for example, the edge) can be used more reliably.

For images where the foreground and background are hardly disturbed, the accuracy of the trace (and each segment) can be improved using edge information. For example, a straight line in real space appears as a straight line in an image.
First, canny edge detection is applied to the video, while recording the normal direction of each edge pixel. Each segment then collects the edge pixels to which it belongs.
An edge pixel belongs to that segment with a certain possibility (how to calculate that possibility will be described later) if the following two conditions are met:

An edge pixel falls within a certain distance (threshold) from its segment.
The normal direction of an edge pixel is perpendicular (with a certain threshold) to the segment tangent on the point of the segment closest to that edge pixel.
To calculate the distance from an edge pixel to a segment, the concept “in-segment” is first introduced.

If the segment is a straight line, the intra-segment area is defined as the area between the start and end points and extends outwardly perpendicular to the line. Otherwise, the in-segment area is the area between the start point and end point of the curve and extends outward from the center.
15 and 16 are visually shown as an area surrounded by an alternate long and short dash line that is designated as an in-segment area. If edge pixel j is within the intra-segment region of segment i, the distance of edge pixel to segment dij is the closest distance to that segment. This distance is calculated based on the known equations for that segment.

これから、エッジピクセルjがセグメントiに属しているという可能性は式（３４）で示すように計算される。 If the edge pixel is outside the intra-segment region, the distance to the segment dij is the shorter of the distances to the two endpoints.
Each segment assigns a weight to every edge pixel it collects. For the segment i and the edge pixel j, a weight wij is given as shown in the equation (33). Here, σ is the value of standard deviation and is equal to 4.

From this, the possibility that edge pixel j belongs to segment i is calculated as shown in equation (34).

今、各々のセグメントの方程式は、ある特定のピクセルがそのセグメントに属する可能性を計算したのと同様に、そのセグメントが集めたエッジピクセルの位置に基づいて再計算される。
セグメントが直線、円、軸平行楕円または一般楕円であるかどうかにかかわらず、セグメントの複雑さは変わらない。それから、その複雑さ（そのセグメントを定める変数の数）より多くのエッジピクセルがそのセグメントに属し得るということもある。これらの場合、最小二乗法によって求められる。

Now, the equation for each segment is recalculated based on the position of the edge pixel collected by that segment, just as the probability that a particular pixel belongs to that segment is calculated.
Regardless of whether a segment is a straight line, a circle, an axis-parallel ellipse, or a general ellipse, the complexity of the segment does not change. Then, there may be more edge pixels belonging to the segment than its complexity (the number of variables that define the segment). In these cases, the minimum square method is used.

The least square method usually takes a form as shown in the equation (35).
Let n points be in the current segment. Then W is an nxn matrix with the possibility of each edge pixel belonging to the current segment present diagonally. X is a list of variables to be solved, and S and b are edge-pixel specific values related to those variables.

それから、更新された各セグメントの方程式に基づき、クリックされた点の位置を修正する。セグメント構成点については、新しい候補位置は、その点に最も近いセグメント上（新しい方程式に従って）の点である。オリジナル位置と新しい候補位置の間の距離が閾値以下にあるならば、その点の位置は候補位置に修正される。 For example, from the analytical form shown in Equation (36), the circle has the constraints that A = C and B = 0.
In this case, S, X, and b can be written as shown in Expression (37).
Similar equations can be written for other shapes.

Then, the position of the clicked point is corrected based on the updated equation for each segment. For segment constituent points, the new candidate position is the point on the segment closest to that point (according to the new equation). If the distance between the original position and the new candidate position is less than or equal to the threshold, the position of that point is modified to the candidate position.

  For segment split points, the new candidate position depends on the start and end points of the segment. Two segments may intersect at 0, 1, 2, 2, or 4. The candidate positions are calculated differently for each case, as shown as Case 1 to Case 3 below.

Case 1:
When two segments do not intersect In this case, the candidate position is the middle of the closest point between the two segments on each segment Case 2:
If two segments intersect at one point:
In this case, the intersection is a candidate position. Case 3:
If two segments intersect at two points:
The point closest to the clicked point on the line connecting the two intersections becomes the new candidate position. Case 4:
If two segments intersect at four points:
Of the four intersections, two points closest to the clicked point are selected. Based on the two selected points, calculation is performed by the method of Case 3 to determine a new candidate position.

If the distance between the candidate position and the new position exceeds the threshold value, the position of the clicked point is not corrected.
Otherwise, the position of the clicked point is moved toward the candidate position. This process can be repeated arbitrarily to improve the accuracy of the reconstructed model (calculate edge pixel likelihoods, assign equations to segments and move clicked point locations) .
As described above, the edge pixel is detected from the two-dimensional image, and the position information of the boundary is corrected from the position information of the edge pixel. That is, the position of each click point is automatically adjusted based on the image information (edge).

By the way, in many robotics, modeling and entertainment applications, it is very beneficial to be able to quickly reconstruct 3D models of real space objects.
One way to achieve this result is to try to model a 3D object from one or more 2D images with the help of a software program. Even in the prior art, a completely automatic 3D object reconstruction method has not yet matured and is not sufficiently reliable. As a result, a simple and intuitive semi-automatic technique is very beneficial for the aforementioned applications.

The present technology provides a quick and easy solution for the reconstruction of 3D models of a wide variety of objects.
Users can intuitively reconstruct fairly complex objects with just a few mouse clicks. Users can also take advantage of object symmetry to further speed up the reconstruction process.
Since the image is used, the texture of each surface can be used immediately. Thus, no additional effort is required to apply the texture to the reconstructed model.

Therefore, this technique can greatly improve the efficiency of model reconstruction of 3D objects.
Furthermore, radial distortion introduces unnecessary errors in the video. One of the big artifacts caused by radial distortion is that real-space straight lines appear curved in the image.
By correcting the radial distortion based on information already provided by the user (no need for new input from the user), the video can be corrected immediately and more accurately represent the real space. it can. As a result, a more precise model can be reconstructed based on the video.

Furthermore, by automatically adjusting the position of each click point based on the image information (edge), human error can be further reduced. This step proceeds in parallel with the radial distortion correction step. This is because the radial distortion correction effect improves the quality of this automatic adjustment step. As a result, a more precise model can be reconstructed.
In the present invention, a three-dimensional model of an object having a non-tilted top surface and bottom surface, which is formed by a plurality of points obtained by tracing a boundary by a user, is generated.

Hereinafter, the features of the present embodiment will be described in summary.
In this embodiment, many artifacts are semi-automatically created from one or more images by taking advantage of the fact that they are composed of edges that can be modeled by straight, circle, or ellipse segments. A method for modeling is disclosed.
Also, based on one of the videos, the user defines an attribute that allows the software to calculate a projection matrix from real space to image space. This calibration step provides the user with flexibility and ensures the user's complete control over the correctness of the projection matrix.

In the present embodiment, a method for independently modeling different sections of object shapes is disclosed.
This is accomplished by modularizing the overall shape through traces and segments and interpolating each segment separately so that it can be represented independently of a particular part of the curve. Quite complex shapes of objects can also be modeled.
In the present embodiment, a radial distortion correction method based on data given by a user for calibration purposes is disclosed.

Radial distortion creates unwanted objects (trash) in the video that will reduce the accuracy of the reconstructed model.
In the present embodiment, a method for increasing or decreasing the volume from the reconstructed form is disclosed.
This is accomplished by joining multiple parts with positive traces and hollow parts with negative traces.
In the present embodiment, a method for drawing an object or a part of an object in a simple manner based on a predefined primitive function is disclosed.

Objects or parts of objects with a general shape can be modeled using a general box shape, a general cone shape, and a spheroid.
This embodiment discloses a method for simplifying modeling by introducing left-right symmetry to the reconstructed model.
This is accomplished by the user flipping some of the trace points along a particular axis, taking advantage of the symmetry property that occurs in many different objects in real space.
In this embodiment, a method for improving the accuracy of trace points using image information is disclosed.

By using image information such as edges, the accuracy of trace points can be improved. This is particularly useful in correcting radial distortion.
In the present embodiment, a method for refining a reconstructed model is disclosed.
This is accomplished by allowing the user to freely add, delete, modify and trace points at any time. The height value of the top and bottom can be freely modified at any time.
In the present embodiment, a method for reconstructing an object with a realistic appearance is disclosed.

The texture can be mapped to a model reconstructed from the video.
In the present embodiment, a method for refining the reconstructed model with more images is disclosed.
Components that have previously been blocked from view by looking at the object from a particular viewpoint can also be modeled. By looking at an object from a specific viewpoint, the texture could not be mapped, but a poorly mapped surface can also be mapped and remapped.

Finally, the application field of the present invention and the merits in each case will be described.
(1) Robotics Modeling objects for pick-and-place applications Rebuilding realistic models of objects for robotics simulations in pick-and-place applications Modeling objects for three-dimensionalization With robotics simulation applications Of realistic models of objects in real space for 3D modeling

merit:
In order to make robot simulation realistic, objects used for pick-and-place and objects in real space must be modeled so that three-dimensionalization is close to what is visible in real space.
For pick-and-place simulation, the user may want to test how the robot interacts with the object, or how many properties of the object affect that interaction. As a result, a precise model of the object is required.

This technique can easily and accurately model a wide range of objects.
(2) In computer vision, if there is a reconstructed model of the object to be modeled for training the object and pose recognition routine, simulate the situation of different obstacles under different viewpoints, different lighting be able to. A machine learning algorithm can be applied to learn special features of the object from these videos. For example, a new video object task or pose recognition may be performed for a new video.

merit:
The appearance of an object changes significantly under different viewpoints, different lighting, and different amounts of obstacles. For this reason, creating a video set of objects and applying a cognitive application is usually very tedious and time consuming.
But with this method, once the model of the object is reconstructed, it can simulate its appearance in whatever way the user wants, and can produce video under any specified circumstances . This greatly simplifies the task of collecting data by machine learning.

(3) In games, entertainment, virtual reality Modeling objects for display in games, entertainment (eg movies) or virtual reality environments Reconstructed objects are virtual environments such as games, movies and virtual reality programs Can be placed inside. For example, make the object realistically three-dimensional in the scene.
In games and virtual reality applications, the user can interact with the model of the object in each scene.

merit:
In order to represent an object more realistically in a virtual application, a good model of that object is required and it takes a very long time to build it.
However, this method allows a good model (geometrically and superficially) to be created quickly and easily. As a result, it can be used in a virtual environment with little user effort.
In order to automatically improve the accuracy of tracing, the program should be able to handle information such as edges and corners that are clues to the video.
In the present embodiment, the three-dimensional object reconstruction step is executed by a processing software program in a processing device (for example, a computer). In this sense, the present invention is realized as an actual 3D image processing apparatus, and a medium for storing the program can be realized. Similarly, a medium storing a 3D image processing program can also be realized. It has been realized. Furthermore, it is also realized as three-dimensional image processing as a process executed by a computer. Although not explicitly described individually, each process executed by the computer realizes a function of performing the same process on the computer.

1 ... processing software, 2 ... trace and point manager, 3 ... three-dimensional model generator, 4 ... texture manager.

Claims (14)

A three-dimensional image processing apparatus for calculating a three-dimensional image from a two-dimensional image,
Image reading means for reading a two-dimensional image;
Reference position specifying means for displaying a read two-dimensional image, receiving an operation input, and specifying a three-dimensional reference position on the image,
A conversion relationship acquisition means for acquiring a conversion relationship between a three-dimensional real space and a two-dimensional image space;
On the display image, a tracing means for accepting an operation input and designating the shape of the boundary of the surface of the object with the number of discrete point positions corresponding to the shape of each segment for each segment;
A height value acquisition means for receiving an operation input and acquiring a height value;
Based on the discrete point positions designated for each segment and the number thereof, boundary specifying means for calculating the three-dimensional position information of the entire section and specifying the boundary,
A three-dimensional model comprising a three-dimensional model reconstructing means for calculating a three-dimensional model of the object based on the height value, the calculated position information of the boundary of each segment, and the conversion relationship. Image processing device.   In the reference position specifying means, in the two-dimensional image, two pairs of parallel line segments and each pair is orthogonal to each other, and the origin is used as a three-dimensional reference position. The three-dimensional image processing apparatus according to claim 1, wherein the three-dimensional image processing apparatus is specified.   The tracing means is characterized in that as the correspondence between the shape of each segment and the number of discrete point positions, the straight line has 2 points, the circle has 3 points, the axis parallel ellipse has 4 points, and the general ellipse has 5 points or more. The three-dimensional image processing apparatus according to claim 1 or 2.   The designation of the boundary of the surface of the object in the tracing means includes designation of a solid shape boundary and designation of a hollow shape boundary, and the three-dimensional model reconstruction means includes a solid shape calculation process, The three-dimensional image processing apparatus according to any one of claims 1 to 3, wherein a hollow shape calculation process is executed, and a three-dimensional model is reconstructed based on a result of combining the two.   2. A texture pasting unit that obtains a texture from a two-dimensional image based on positional information of each surface when calculating a three-dimensional model and pastes the texture on each surface of the three-dimensional model. The three-dimensional image processing apparatus according to claim 4.   The texture pasting means reads two-dimensional images from a plurality of different fields of view for the same object, accepts the operation input for each image, and pastes it on each corresponding surface of the three-dimensional model. The three-dimensional image processing apparatus according to any one of claims 1 to 5.   The texture pasting means acquires the texture corresponding to the corrected boundary information and pastes it on a predetermined surface when the boundary position information is corrected. Item 7. The three-dimensional image processing device according to any one of items 6 to 6.   8. The method according to claim 1, further comprising radiation distortion removing means for converting the positional information by removing the radial distortion included in the positional information of the two-dimensional image in the process of specifying the shape of the boundary. The three-dimensional image processing apparatus according to any one of the above.   The point adjustment means for detecting an edge pixel from the two-dimensional image and correcting the position information of the boundary from the position information of the edge pixel in the process of specifying the shape of the boundary. The three-dimensional image processing apparatus according to claim 8.   The trace means accepts an operation input and automatically generates a point position maintaining left-right symmetry by turning over a specified point position along a specific axis. Item 10. The three-dimensional image processing apparatus according to any one of Items 9 to 9.   11. The object according to claim 1, wherein the object is modeled by using a combination of a predefined shape, a general box, a general cone, and a spheroid. The three-dimensional image processing apparatus described in 1.   The three-dimensional image processing apparatus according to any one of claims 1 to 11, wherein the tracing means can add, delete, and correct a point position in each segment. A three-dimensional image processing method for calculating a three-dimensional image from a two-dimensional image,
Load a 2D image
After displaying the read 2D image, accept the operation input, specify the 3D reference position on the image,
Get the transformation relationship between 3D real space and 2D image space,
On the display image, accept the operation input and specify the shape of the boundary of the surface of the object with the number of discrete point positions corresponding to the shape of each segment for each segment,
Accept the operation input, get the height value,
Based on the discrete point positions specified for each segment and their number, the position information in the three dimensions of the entire segment is calculated to identify the boundary,
A three-dimensional image processing method, comprising: calculating a three-dimensional model of an object based on a height value, calculated position information of a boundary of each segment, and the conversion relationship. A medium recording a 3D image processing program for calculating a 3D image from a 2D image,
Image reading for reading 2D images,
A reference position specifying function for displaying a read two-dimensional image, receiving an operation input, and specifying a three-dimensional reference position on the image,
A conversion relationship acquisition function for acquiring a conversion relationship between a three-dimensional real space and a two-dimensional image space;
On the display image, a trace function that accepts an operation input and specifies the shape of the boundary of the surface of the object for each segment by the number of discrete point positions corresponding to the shape of each segment;
A height value acquisition function that accepts an operation input and acquires a height value;
Based on discrete point positions designated for each segment and the number thereof, a boundary specifying function for calculating the three-dimensional position information of the entire section and specifying the boundary,
A computer realizes a three-dimensional model reconstruction function for calculating a three-dimensional model of an object based on the height value, the calculated position information of the boundary of each segment, and the conversion relationship. A medium on which a 3D image processing program is recorded.

Images