KR20040049832A - Method and Apparatus for Visualization and Manipulation of Real 3-D Objects in Networked Environments - Google Patents

Abstract

PURPOSE: A device for visualizing and operating a 3-dimensional object under network environment is provided to remove a rendering process and a model fabrication process, thereby visualizing or dealing with a 3-dimensional object in real time through a network by using a general PC as a client. CONSTITUTION: A turn table firmly locates a photographed target object in an upper part, and rotates at predetermined rotative angle by responding to a turn table control signal. A camera photographs an image of the photographed target object. A camera position controlling device supports the camera such that the camera can rise or fall to photograph the target object by responding to the control signal, and maintains a distance between the target object and the camera. At least one motor rotates the turn table, and moves the camera. An image manager supplies the control signal, and generates a 3-dimensional image of the target object by using images from the camera.

Description

Method and Apparatus for Visualization and Manipulation of Real 3-D Objects in Networked Environments

Traditionally, this has been a domain of computer graphics and rendered images based on meshes or other object models (Girod00, Robb00). The problem with this existing paradigm is that the computational cost of the rendering process is expensive. Although there are ways to improve the rendering process to an optimal level, it is basically based on approximations, such as the expansion of grid / mesh elements. The images obtained through these techniques are of course less realistic, especially because of the difficulty in modeling naturally complex objects (Girod00).

Most of the visualization paradigms currently available must follow a two-step process of modeling and rendering (Girod00). First, in the modeling step, parameters representing an object are extracted and stored. The rendering step then renders an image of the object for the particular view requested by the user based on the parameters of the stored object (preferably in real time). The modeling step can be performed offline or a very complex algorithm can be applied, but the rendering step must be performed in real time for realistic manipulation of the object.

On the other hand, since most of the clients on the Web are PCs with limited computing power, the rendering process for naturally complex objects generally cannot be performed on the client side.

On the other hand, even if the rendering step on the server side (for example, supercomputer) will limit the number of rendering requests that the server can serve at the same time, this also does not solve the problem.

An object of the present invention is to provide a method and apparatus for visualization and manipulation of three-dimensional objects in a network environment that can provide a solution that replaces the modeling step with an image acquisition step and the rendering step with a simple image compression step. .

It is another object of the present invention to provide a method and apparatus for visualizing and manipulating a three-dimensional object in a network environment capable of eliminating parts requiring calculations and manual parts extracting modeling parameters from existing visualization techniques. . Still another object of the present invention is to provide a method and apparatus for visualizing and manipulating a 3D object in a network environment in which a user can view and manipulate a real 3D object in real time through the Internet.

1 is a diagram showing the geometry of an image acquisition device

2 is a diagram showing a set of all straight lines starting at the position of the "fuzz-ball" camera and passing through the origin (center of the object).

Figure 3a is a diagram showing the position to shoot the image in the slice sampling strategy, a diagram showing the position (point) and three-dimensional trajectory (line) to shoot all the images

Fig. 3B is a diagram showing the positions at which images are taken in the slice sampling strategy, showing all the shooting positions mapped by Sanson projection.

4A is a diagram showing the image capture position in an anisotropic spiral strategy, showing a three-dimensional trajectory of the image capture position.

4B is a diagram showing image capture positions in an anisotropic spiral strategy, showing Sandson projection of image capture positions.

FIG. 5A is a diagram showing the image capturing position in an isotropic spiral sampling strategy, showing a position (point) and a three-dimensional trajectory (line) of capturing an image.

FIG. 5B is a diagram showing image shooting positions in an isotropic spiral sampling strategy, showing the shooting positions mapped by the Sandson projection. FIG.

6 is a diagram showing the real of the three-dimensional imaging system

7 is a diagram showing the main screen of the three-dimensional image editor program

8A is a diagram showing how the NEWS mode operates in a display program.

8B is a diagram showing how the AVIATION mode operates in a display program.

9 is a diagram showing an image of an object according to an angle viewed by a user as a main goal of a 3D visualization of a 3D image visualization object.

In the present invention, the server with three-dimensional data only transmits data to the requesting client. Hereinafter, an image acquisition apparatus and a 3D player for accessing and manipulating an object in real time by accessing a 3D image database will be described in detail.

Image Acquisition Device

1 is a diagram showing the geometry of an image acquisition device.

Referring to FIG. 1, an image is taken at various angles θ and φ by placing the object at the origin and rotating the camera about a distance R apart around the object. On the other hand, in order to reproduce an object image corresponding to a specific viewing angle that a user may request, the object must be represented as an image at all possible viewing angles.

The definition of all possible view angles can be understood by drawing a subset of all possible straight lines (all possible view angles) from the origin. All of these lines will be fuzz balls as shown in FIG. 2 is a diagram showing a set of all straight lines starting from the position of the fuzz-ball, camera and passing through the origin (center of the object).

2 and θ is θ = And φ = (0.5 + n) pi / 7, where m = 0,1, , 31 and n = 0,1, , 6. Note that basically the spacing between the lines controls the viewing resolution in the corresponding directions θ and φ. View resolution determines the smoothness of rotating the view angle by moving the mouse to change θ and φ.

When the camera shoots from the position of each straight line towards the origin, the camera must move around the surface surrounding the object. Therefore, in view of minimizing the moving distance of the camera, the optimum surface is the surface having the smallest surface area while including the object.

Since the camera must be away from the object, the optimum surface is a sphere with radius R, for example by a distance R defined at the origin.

If an object is projected to every point on the surface (the camera is looking towards the origin), then the object can be projected at all possible viewing angles. Even after setting the sphere with radius R to be the optimal surface, we still need to determine the parameters and divide the view angles θ and φ discontinuously. The parameterized angle determines the camera path, namely θ (t) and φ (t), 0 = t <= T [ T is the total acquisition time] By dividing the view angle into discrete values, i.e., θ _k and φ _k , k = 0,1, Images of, N-1, finite sequences can be obtained. Two angle sequences, θ _k and φ _k , k = 0,1, N-1 sets the camera's unique path.

This sequence should also be designed efficiently to reduce shooting time. Since the camera's position is fixed at a distance R from the origin, it would be best if the camera's movement could include the surface of a sphere with radius R at one time. We propose these three designs.

Slice Sampling Strategy

3A shows one travel path in which the camera can contain a sphere of radius R at one time. FIG. 3b shows a three dimensional view of FIG. 2 mapped using a Sanson projection (Kreyszig91) on a horizontal plane. Sanson projection is often used for geographic mapping applications. Let N _θ and N _φ represent the average (or exact) number of frames taken in the range [0,2 pi] and [0, pi], respectively. N _θ and N _φ need not be integers. The total number of images taken at this time is N N _θ N _φ . Consider the following image capture points:

Where operator Represents a float operation, which takes the largest integer less than or equal to the argument. Thus N _θ and _φ N determines the resolution of the view (viewing resolution) for θ and φ direction. For example, N _{θ =} N _φ means that the view resolution is the same for the θ and φ directions. 3A and 3B show results using photographing points of Equations 1 and 2.

The most convenient motor control method is to discontinue φ while rotating θ at a constant angular velocity. It will rotate as much. Since T is the time taken to shoot N frames, the two motors are controlled as follows.

Here, T _θ is a time taken for the camera to rotate once in the θ direction. Thus, if two motors are controlled according to equations 3 and 4 and the image acquisition point is taken as:

The image capture points coincide exactly with the points given by equations (1) and (2). 3A and 3B show photographing angles where the points indicate the point where the actual camera takes the image. The exact setting of the parameters can be seen from the description of the drawings.

* Anisotropic Spiral Sampling Strategy

The motor control according to the equations (3) and (4) proceeds in several steps intermittently in the θ direction for each rotation in the θ direction. In practice, motor control in this manner is difficult, especially when time becomes an important threshold. Eliminating this interruption results in the following motor control formula:

When the image acquisition time is taken by Equation 5 and motor control is performed according to Equations 6 and 7, the following image capturing positions can be obtained.

4A to 4B show image capturing positions when the same parameters as those of FIGS. 3A to 3B are taken. Similar to FIGS. 3A and 3B, FIGS. 4A and 4B show a three-dimensional trajectory and a Sandson projection of the image capturing position. The parameters used are (N _θ , Nψ) = (16,16), (θ _o , ψ _o ) = (0, ) to be. Like the slice sampling strategy, this method shoots a total of 256 frames.

Although simple, slice sampling (Equations 1 and 2) or anisotropic spiral sampling (Equations 8 and 9) above do not provide the same view resolution everywhere. There are more imaging points near the two poles as in FIGS. 3A-3B and 4A-4B. Therefore, the view resolution is higher towards the anode. Usually the three-dimensional object to be projected needs to be projected isotropically over the entire projection plane. You don't have to take a lot of images because you're biased in two plays.

Since the imbalance of view resolution (low resolution at the equator of the sphere) is unnatural, this can be eliminated by redesigning the scanning strategy. Another scanning strategy developed in the next section makes the view resolution the same for the entire sphere in which all image capture points exist.

Isotropic Spiral Sampling Strategy

In the previous technique, two motors were rotated at a constant speed and images were taken at equal intervals in time. As can be seen in FIGS. 3A-3B and 4A-4B, the image capture points are denser as they get closer to the poles. However, if the image is taken at a low frequency around the pole, it is possible to obtain an isotropic view resolution with the motor control equation given by Equations 6 and 7, ie, rotating the motor at a constant speed. Naturally, in this case, the image capturing time will not be the same interval in time.

In the isotropic spiral scanning strategy, we first determine the time and parameters of the two angles, (θ (t), φ (t)), where 0 = t <= T and t is [0, T]. Assuming that the radius of the sphere containing the object is R and the image capture points are on the sphere, the amount of micro-change in the path of the camera is as follows.

Here, by the equation (10), the position vector r always exists on the sphere of radius R. Minute (微少), there is a method to maintain a constant for a distance d r (vector) the parameter t is sampled at regular intervals for the camera position vector r to the parameter t. Therefore, neighboring shooting points maintain the same distance along the path.

However, the gap in the vertical direction φ may vary depending on the actual angle of φ. That is, the gap in the vertical direction will change according to sin (φ). So the micro distance drthis If you always design a constant for t, the isotropic imaging points will not be provided. So look for other solutions such as:

Since it is simple to implement, we continue to use the following motor control equations.

Here, as before, T and T _θ are the time taken for the camera to rotate one turn in the? And θ directions, respectively. Substituting these equations into equation 10 yields the following equation.

As can be seen from Equations 11 and 12, the motor controlling θ by design rotates faster than the motor controlling φ. So for a very small value t and an initial value φ ₀ The component is assumed to be negligible. Therefore, the following approximation can be used:

Integrating the above equation, we can find the next arc length, which is the moving distance of the camera.

Therefore, the total distance the camera moves (from t = 0 to t = T ) is

Since the purpose is to sample the camera trajectories separated by equal intervals, the total distance divided by the total number of frames N (-1) is determined as the interval between image capture points. Therefore, the image capture points can be represented as follows.

The following results can be obtained by solving Equation 15 and Equation 17 with equations and solving for the shooting time.

Shooting time t and index k Has the same relationship as N can also be used as a parameter for the user to determine the view resolution. In previous methods (FIGS. 3A-3B and 4A-4B) To maintain view resolution when

here, Indicates a rounding operation. In the previous two methods, the total number of images taken was N N _θ N _φ . In the spiral sampling strategy, the total number of frames compared to the previous methods Decreases at the rate of. This reduction is the cost of reducing the number of frames around the anode, which were unnecessarily too many shots in the other two methods.

5A and 5B show the image capture points accordingly. The shooting positions are visually equally spaced, which in effect provides an isotropic view resolution.

The parameters used are (N _θ , N _φ ) = (16,16), to be. Unlike the previous approaches, this approach totals 163 Take a picture of the dog. Where isotropic view resolution and total number of frames It should be noted that the ratio decreases.

* Image Capture and Frame Tagging

Whatever approach is used to obtain the image sequence, the image acquisition device will store the following two data in a central (or distributed) database for future network access:

Compressed image sequence f ( k ), k = 0,1, , N-1.

Θ _k and φ _k corresponding to each image, k = 0,1, , N-1.

Note that the positions θ _k and φ _k from which each frame was obtained are stored with the frame.

6 is a picture of the 3-D image acquisition equipment. The object is placed on the turntable and photographed by a camera attached to the carrier. The conveying body driven by the motor moves along the C arm of the vertical axis to which the timing belt is attached. The turntable is driven by another independent motor. These two motors that control the turntable θ and the camera carrier φ are preprogrammed by a method defined in the operator console PC.

Both motors are equipped with an encoder and record the angles (θ) and (φ) at the moment the encoder captures all images.

Two motors control two angles that determine the position of the camera. The camera moves along the C-arm (φ direction) of the vertical axis while the turntable rotates to control the value of θ. The machine thus acquires image sequences and stores the two angles θ and φ generated by the encoder mounted on the motor.

* The 3-D Image Editor

Sequence of images f ( k ), k = 0,1, After acquiring N-1, the obtained three-dimensional images can be edited variously, including background removal, frame insertion / deletion, color correction, and unsharp masking. 7 shows a 3-D image editor. The editor program is divided into three main windows. The first is a Sanson projection window (upper left in FIG. 7) showing the position (marked with a dot) where the actual image was obtained by the Sanson projection method. The second is a 3-D view window (lower left in Fig. 7) showing the acquired image in three dimensions, and the second is a thumbnail window showing thumbnails of all frames.

The three windows are interconnected internally. For example, when a user clicks on a point in the Sanson projection window, the color of the selected point changes. At the same time, the frame corresponding to the point appears in the 3-D view window, and the image is highlighted in the thumbnail window. Similarly, actions in other windows (3-D view window and thumbnail window) cause the remaining windows to perform the corresponding action.

Hereinafter, the 3D player will be described.

Generation of the look-up-table

Each of the images (video frames) is stored with corresponding angles θ, φ in the acquisition process. For efficient operation, the 3-D player requires a lookup table, LUT (θ, φ), which maps the view angles (θ, φ) to the frame number. The time required for this task is minimal, so it can be calculated while receiving image data from the server. The LUT is created according to the following procedure.

The LUT of size L _θ xL _φ is filled once by the following procedure.

Iterations: k = 0,1, , N-1

End of loop

here, Denotes a truncation operation that converts a real number to an integer less than or equal to the argument. Undoubtedly, the above operation results in an unfilled cell (with -1). These cells are filled with the value of the nearest frame. For example, the following process occurs.

Iterate over all satisfying LUT (i, j) =-1

Closest to satisfy Find it.

End of repetition

The nearest frame value mentioned above can be thought of as any value with the smallest l ₂ -norm :

here Wow Are frame numbers corresponding to each.

The operation used in the above Loop can be performed by repeating morphological operations such as 'dilate' (Serra82). Note that other distances or intervals can also be used to find the closest cell.

As soon as the user inputs (θ and φ), the 3-D player decompresses the frame corresponding to the next frame number and outputs it to the screen.

In visual applications, the accuracy of angle information θ and φ can be an important factor (Parker97, Schlkoff89). However, in the proposed 3-D visualization application, the accuracy of the two angles is not a serious problem, as long as the image is only for visualization purposes, as long as the image is densely acquired. Therefore, the equation (23) solves the task of showing the nearest frame, that is, the frame closest to the current (θ and φ).

* Display of 3-D Objects by Random Access

The 3-D player uses the 3-D image sequence f ( k ), k = 0,1, when the user clicks or drags the mouse over an object on the screen. Can show N-1. Initially, the user is shown one of the sequence of images, and when the user drags the mouse, the screen is reprinted with images in the proper order according to the rules that make the screen feel like a real 3-D object. The handling of visible objects is based on one of two modes: NEWS (north-east-west-south) and AVIATION. These two modes will behave differently for the same mouse movement.

In NEWS mode, dragging left (or right) turns the object west (or east). Dragging down (or up) returns the object to the south (or north) direction. In AVIATION mode, on the other hand, dragging in one direction will return the object in that direction. In other words, in AVIATION mode, the mouse drag direction always corresponds to the direction of two orthogonal circles, one of which corresponds to the direction of the current object. FIG. 8A is a diagram showing an operation method of the NEWS mode in the display program, and FIG. 8B is a diagram showing an operation method of the AVIATION mode in the display program. 8A and 8B show the difference in operation between the two methods. 8A and 8B, δU and δR input by the user by dragging the mouse are regarded as variations in the U axis and the R axis, respectively. In the NEWS mode, the U and R axes correspond to north and east, respectively. In AVIATION mode, the U and R axes lie on a plane defined by two orthogonal members. One circle then forms an angle between the North Pole and Ψ.

Mathematically, the two modes of operation can be summarized as follows.

Suppose the current object has an angle relative to the north pole and the user has dragged the mouse by as much as. Then for NEWS mode the renewal formula is:

In the AVIATION mode,

Update function θ (in AVIATION mode ) φ ( ) PSI ( There are several systems available. In particular, the following north reference systems commonly used in navigation systems are applicable.

In both cases, the PSI does not change with the movement of the mouse, and the user must manually rotate the image to view the desired angle. Of course, other methods are possible. In FIG. 7, the lower left window (3-D view window) is an ActiveX control that implements two methods that can control a three-dimensional object, that is, NEWS and AVIATION modes.

In order for web surfers to see 3D objects, only the ActiveX control (3-D view window) needs to be visible. In either case, the frames displayed on the screen in both modes Becomes Where the subscript PSI represents the angle of rotation of the image (object). The 3-D viewer must therefore support rotation of the object as well as random access to any frame. The 3-D viewer also provides general functions like zoom and pan up, down, left and right.

* Pre-Decode for Fast Display

Since the image sequence is encoded as video, a task of decoding certain frames in advance is required to facilitate real time operation. For example, the MPEG standard encodes video sequences in units of GOP (bhaskaran95). An example of the structure of a 12-frame GOP may be IBBPBBPBBPBB. Where each alphabet represents a frame type. I-frames can be decoded independently; P-frames must have a previous anchor frame (I- or P-frame) decoded first; And in the B-frame, two anchor frames must be decoded before and after.

In our special case of random access to any frame within a GOP, we need to decode some frames before and after that frame before decoding that particular frame within the GOP. Of course, when playing a coded video like a normal video or video, the above-mentioned problem does not occur because all the frames must be decoded sequentially. But in our case, we should be able to decode non-sequential frames.

In the case of a GOP having an order of IBBPBBPBBPBB, if only I-frames are decoded in advance, four decoding processes are required to perform random access to the last two B-frames. However, if all I-frames and P-frames are decoded in advance and stored in memory, only one decoding process is sufficient (or no need for one) to solve one image in the GOP.

This is the method chosen in the present invention. In other words, since all I-frames and P-frames are pre-decoded in the local buffer, you only need to decode at most once to see a particular frame. In order to reduce the size of the memory used, previously decoded images are compressed using a lossless compression method.

One other possibility would be the case of encoding with a GOP structure such as IBBBBBBBBBBB and decoding the I-frames in advance. In this case, if all I-frames in the video have been decoded, random access can be made to the desired frame once or without decoding.

Multimedia Streaming

When viewing or manipulating a 3D image on the screen using a display application (ActiveX control), there is no need to wait for all the data to come in. When data is transmitted over the network, only one or more frames are required to view or manipulate the image.

The implemented application first receives the data header including the total number of frames, the details of the GOP structure, the bit rate, and then decodes it. When all the necessary information is obtained through the above process, an independent thread (low priority) is activated to receive the remaining data. When a thread receives data, it communicates with the main program as to which frames were sent and which were not. The order of transmission of the MPEG sequences is also changed to send all I-frames first, all P-frames and then all B-frames last.

The same frame type, e.g., I-frame, among the higher priority frames is transmitted before the lower frame. Upon receiving the frame, the application program decodes the I-frame and P-frames in advance. The program that displays the image on the screen marks frames that have been sent to use only the frames that are already present.

The file consists of not only video data but also other data to be downloaded at the user's request, for example zoom data or another video data. For example, if a user wants to see a more detailed image from an angle, the download of separate multimedia data begins when the user asks. Another example would be to download a specific video (opening the refrigerator door) at the user's request.

9 conceptually shows a one-stop model for how to handle our three-dimensional objects. 9 includes a 3-D image center having a three-dimensional image acquisition device connected to the Internet. An individual with a three-dimensional object will look for a 3-D image center to visualize the object in three dimensions.

According to FIG. 9, at all possible angles, using sliced sampling or spiral sampling, as soon as an image of the object is obtained, the image sequences are sent to a central (or distributed) database for storage. The individual using the service leaves the service center with only the URL of the 3D object file.

The user publishes the URL on his web so that other web surfers can access this 3-D data. Even if you don't put the URL on a specific web page, you can see the three-dimensional object you created by simply typing the URL in a web browser. The URL (three-dimensional data) may or may not be protected by a password. The customers of the 3-D Image Center can be not only individuals, but also large and small companies who want to see objects in three dimensions over the Internet.

In the following, the effects of the present invention will be mentioned.

First, most existing three-dimensional visualization techniques go through a two-step process. First create a model of the object and save it. When the user wants to see an object from a certain angle, it renders it based on a pre-made model before showing it to the user. This image rendering process makes it difficult for users to manipulate objects in real time for complex and natural objects. According to the present invention, not only rendering but also model making process is eliminated. As a result, a normal PC can be used as a client to visualize or manipulate three-dimensional objects in real time over the network.

Second, the method of visualizing or manipulating objects in three dimensions does not gain the popularity that video or images enjoyed. The reason is probably that most individuals who wanted to show 3D objects through the web have to invest a lot of effort and time to put them on their homepage. The three-dimensional object one-stop service concept according to the present invention can fundamentally reduce the effort and time that an individual or an expert has to put into the homepage.

Third, the system according to the present invention can be used by non-professionals because it removes all the technical details that all users of today need to know in order to visualize real objects in three dimensions.

Claims (8)

A turn table unit fixedly positioning a photographing object on an upper portion and rotating at a predetermined rotational angle in response to a turntable control signal; A camera unit for capturing an image of the photographing object; A camera position adjusting device configured to support the camera unit to raise or lower the camera unit in response to a camera position control signal and to maintain the distance between the object and the camera unit while the camera unit is raised or lowered; At least one motor for rotating the turntable unit and moving the camera unit; And And an image manager which provides the control signals and generates a 3D image of the object to be photographed using the images from the camera unit. The photographing apparatus of claim 1, wherein the camera unit maintains a constant distance from the object during the ascending or descending on the camera position adjusting device while simultaneously adjusting a photographing angle aligned with the inner center of the object to be photographed by the camera position adjusting device. 3D image generating apparatus characterized by having. The apparatus of claim 1, wherein the image manager comprises a storage configured to store the digital image. 3. The digital image of claim 2, wherein the digital image is stored together with rotation angle information of the turntable part corresponding to the turntable control signal, and an angle corresponding to a movement position of the camera part corresponding to the camera position control signal. 3D image generating device. The apparatus of claim 1, wherein the image manager is a computer. The apparatus of claim 1, wherein the camera position adjusting device is a "C" type arm. The apparatus of claim 1, wherein the operation control signals are updated every time the digital image is generated. A program of instructions that can be executed in a computer is tangibly embodied to perform a three-dimensional image generating method, the recording medium readable by the computer, comprising: a turntable unit, a camera unit, a camera position adjusting device and at least one Generate motion control signals for motor control of The image manager receives a plurality of digital images corresponding to the object to be photographed from the camera unit, Storing the received plurality of digital images in a storage unit in the image manager; Executing the steps of generating a 3D image using the plurality of digital images in the image manager; Here, when the camera unit ascends or descends on the camera position adjusting device, the distance between the object and the camera unit is kept constant and at the same time the camera position adjusting device has a photographing angle that is in line with the inner center of the object to be photographed. A recording medium characterized by executing steps.