KR19980035726A - Method and apparatus for generating panoramic image of virtual reality system - Google Patents

Abstract

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a virtual reality system (VR), and more particularly, to a panoramic image generator for generating a panoramic image by using a compression method. A first process of rearranging the lines in line order to generate one panoramic image; And a second process of performing vector quantization on the panoramic image of the first process, generating a codeword and a codebook, and storing the codeword and the codebook in a memory. Image data storage means (60); Panoramic image generating means (61) for generating a single panoramic image; VQ processing means 62 for generating codebooks and codewords; Panoramic image storage means (63); 3D graphics processing units 64-1 and 64-2 for rendering and texture mapping to output RGB image signals; Signal converters 65-1 and 65-2; Image display and motion detection means (66); Position and direction tracker 67; And a panoramic image generating device of a virtual reality system comprising a viewing angle calculator 68, which generates a single panoramic image, vector quantizes it to effectively reduce memory consumption, and transmits only an index. It has an effect.

Description

A method of panoramic images generating in a virtual reality system and pretty an apparatus

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a virtual reality system (VR). More particularly, the present invention relates to a method and an apparatus for generating a panoramic image using a compressor method.

The virtual reality system vividly shows the unfinished or non-existent virtual situation as if it is a real situation, and gives users a lot of knowledge and experience by freeing the desired action in it. It is implemented based on computer technology and image technology. Unlike the existing computer that provides a two-dimensional environment of use in the form of a window and treats the user as an observer or a foreigner, VR provides a three-dimensional environment of use (virtual world) to the user. It feels like you are in this virtual world and allows you to directly manipulate objects that appear in virtual reality. Because of these differences, the use of virtual reality (VR) technology enables vivid knowledge and experiences that are difficult to provide by traditional information provision and interaction methods, so education / training, entertainment / culture, science / medicine, design, etc. It is expected to create a huge new market in various fields such as manufacturing, telecommunications, and defense.

Such VRs are used interchangeably with terms such as virtual environment, virtual presence, artificial world, virtual world, cyberspace, depending on the point of view. However, in general, a sensor provides a three-dimensional virtual world similar to the real world created by a computer, an input means that can be freely manipulated in real time with the virtual world, and a real sense in response to the user's manipulation. It is a technology that enables artificial experience and experience by providing a means of sensory feedback.

Such a virtual reality (VR) system is composed of effect generators, a reality engine, an application, and a three-dimensional model, each of which is described as follows.

Effectors are a kind of hardware that represent input and output sensors and devices to control the virtual environment.

The Reality Engine is external hardware such as a synthesizer that delivers sensory information necessary for an effect generator, and constitutes a reality engine.

An application is software that describes the simulation and its power, structure, and texture of the interactions between objects and users.

In addition, the three-dimensional model (Geometry) refers to the accumulated 3D model database on the properties of the object, that is, the shape, color, position and the like. It describes the shape, location, structure, and color of the house, and this information is processed by the application and used to create the virtual world.

As an example, as shown in FIG. 1, the effectors include a head mounted display (HMD: 1), a transmitter (2), a joystick (3), etc., for the user to write and to experience the world of virtual reality visually. consist of. The HMD 1 includes a sensor 1-1 for detecting a head direction of a user, left and right speakers for providing stereo sound, left and right LCDs for displaying an image, an optical lens, and the like.

In addition, the reality engine is a hardware existing inside the computer system. The first 3D graphics board 4, the second 3D graphics board 5, the 3D sound board 6, the sound source 7, and the communication port 8, a computer interface board (9), the first signal converter 10, the second 3D graphics board for converting the RGB signal of the first 3D graphics board into NTSC TV signal with hardware existing outside the computer Is composed of a second signal converter 11 and a position and direction tracker 12 for converting the RGB signal into an NTSC TV signal.

The application is composed of a memory 13 storing user application software and a simulation manager 14 for simulating a virtual world.

In addition, the three-dimensional model (Geometry) is composed of a hard disk 15 is stored in the database for building the three-dimensional modeling of each object of the virtual reality.

Such a system tracks the movement of the user's head by the transmitter 2, the sensor 1-1, and the position tracker 12 so that the reality engine is operated by the operation or control of the application or simulation manager. It detects and calculates motion and displays and presents the virtual world again at the appropriate time.

Therefore, the reality engine tracks the subject's head or hand movements and creates a new image on the display, which must go through the entire cycle in real time. In studies of human perception, a response of less than 100 milliseconds is generally considered in real time. If it takes 100 milliseconds to rotate the loop once, then it fills one screen every 0.1 seconds and then 10 screens per second.

The application is software for editing, executing, and controlling the virtual world. The simulation manager reads a database built for the virtual world and monitors the behavior of the subject and the interactions between objects in real time, and then develops the situation. At the same time, it provides a tracked object and a backdrop, allowing users to immerse themselves in that environment.

Geometric models are currently created with tools such as 3D Auto-CAD, where the constructed files are computed and reconstructed from time to time when represented in the virtual world under the control of the application.

However, the problem that occurs in the representation of the virtual world is the number of polygons required to construct such a geometric model and the speed processing for reconstructing the polygons. Polygons are polygons that are created as wireframes that separate the borders and display them as lines. For example, the backdrop may consist of 200 to 20,000 distributed polygons. Each time the user moves, the position and shadows of each triangle in the foreground must be recalculated and redrawn. Thus, the more complex the background, the greater the number of polygons required.

In addition, new texture mapping techniques are being used to achieve a strong sense of realism. Texture mapping is a technique of painting an object's surface to create a realistic model.

A method of generating a conventional panoramic image is shown in FIG. 2. As shown in Fig. 2A, if a single panoramic image is composed of three frames Sa, Sb, and Sc (assuming each frame is composed of 640 x 480 pixels), the image data of these frames is shown in 3a. Similarly, the video is stored separately on the RAM.

That is, the Sa frame consists of lines a1 to a480, these data are stored sequentially from a1 to a480 on the video RAM, and the Sb frame consists of lines from b1 to b480, and these data are stored on the video RAM. From b1 to sequentially b480, the Sc frame consists of lines from c1 to c480, and these data are stored from c1 to c480 sequentially on the video RAM, requiring considerable memory.

In addition, when such a panoramic image is displayed in the conventional virtual reality system, since the scan is scanned for each line, the lines of the Sa frames are all scanned, and then sequentially scanned in the order of Sb frames and Sc frames.

Therefore, since only one frame screen is displayed, the screen moves abruptly. For example, when the user's viewpoint gradually moves from the Sa screen to the Sb screen, the right part of the Sa screen and the left part of the Sb screen should be visible and gradually moved to the Sb screen. There was a problem that the screen does not match.

In order to solve this problem, the invention related to the proposed panoramic image generator is registered in Korean Patent Application No. 96-43227, which will be described below.

In summary, in the present invention, a screen bent in three frames as shown in FIG. 2A is configured as one broadband frame (ie, a panoramic image Sp) as shown in FIG. 3A to correspond to a user's view point. By forming and displaying image data, a panoramic screen is naturally provided corresponding to a viewpoint.

A structure of a memory in which images are stored line by line to form a wideband frame as shown in FIG. 3A is illustrated in FIG. 3B.

A1, a2, ... a480, b1, b2 when three frames make one panoramic image as shown in FIG. 2A. As shown in FIG. 3B, image data stored in order of b480, c1, c2, ... c480 is synthesized by combining a1 + b1 + c1 to synthesize d1, a2 + b2 + c2. a480 + b480 + c480 is synthesized and stored in the order of d480. In this case, as shown in FIG. 3A, the panoramic image is synthesized into a wideband frame, that is, a single image, and scanned in line order, thereby providing a natural screen.

However, such a wide ocean-like panoramic screen maps photographs or images observed from different angles into geometric planar fragments, that is, texture mapping of images onto a spherical body or cylinder according to the point of view of the same scene (background). Because of this, several images are required.

Therefore, these multiple images for displaying the panoramic images are obtained from the same background, but despite the high correlation, there is a problem in that a large amount of memory is required by storing the multiple images. .

In order to solve the above problem, the present invention synthesizes an image divided into several frames to generate one panoramic image, and then saves the memory by compressing and storing the same by using a vector compression technique having high compression efficiency. The objective is to provide a panoramic image generator of a virtual reality system that can be enabled.

In order to achieve the above object, the method of the present invention includes a first process of generating a panoramic image by rearranging images stored separately for each frame in a line order; And a second step of generating a codeword and a codebook by performing vector quantization on the panoramic image of the first step and storing the codeword and the codebook in a memory.

In order to achieve the above object, the apparatus of the present invention, the panoramic image generating means for receiving the image data divided into a plurality of frames to generate a panoramic image; VQ processing means for compressing the generated panoramic image to generate a codebook and a codeword; Panoramic storage means for storing the generated codebook and codeword; A three-dimensional graphic processing unit which reads out image data of one frame corresponding to the viewpoint of a user from the panoramic image storage unit, renders and texture-maps and outputs an RGB image signal; A signal converter for converting the RGB output of the 3D graphic processor into a TV image signal; Image display and motion detection means for receiving and displaying the output of the signal converter and detecting a user's movement; A position and direction tracker which receives a user's motion signal from the image display and motion detection means and calculates the position and direction of the user; And a viewing angle calculator for calculating a view volume of the panoramic image according to the movement of the user input from the position and direction tracker and outputting the view volume to the 3D graphic board.

Therefore, by rearranging the panoramic images line by line and storing the information about the codebook and the codeword through vector quantization, the memory capacity is reduced and high speed transmission is possible.

1 is a block diagram showing a typical virtual reality system,

2A and 2B illustrate a concept of generating a panoramic image on a frame-by-frame basis in a conventional manner;

3A and 3B illustrate a concept of generating a single panoramic image on a line basis;

4 is a flowchart illustrating a method of generating a single panoramic image by converting FIG. 2B into FIG. 3B;

5 is a flowchart illustrating a codebook generation algorithm of the vector quantization technique used in the present invention;

6 is a block diagram of a virtual reality system displaying a compressed panoramic image according to the present invention.

Explanation of symbols on the main parts of the drawings

60: image data storage unit 61: panoramic image generating unit

62: VQ processing unit 63: Panoramic image storage unit

64-1, 64-2: 3D Graphics Board 65-1, 65-2: Signal Converter

66: three-dimensional sound processor 67: position and direction tracker

68: viewing angle calculation unit 70: HMD

71: transmitter 72: input device

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

To this end, in the first process of the present invention, image data divided and stored in units of frames as shown in FIG. 2B are continuously stored line by line as shown in FIG. 3B through address conversion.

The second process of the present invention vector-quantizes a single panoramic image synthesized as shown in FIG. 4B through the first process, thereby reducing the required memory capacity and network by reducing the number of bytes required per pixel. To reduce the transmission speed.

Here, a pixel refers to a point of a graphic image on computer graphics, and has a pixel of 640 x 480 (pixel) in a standard VGA. The number of bits per pixel depends on how much of the image color is represented. VGA displays have eight bits per pixel, so they can display 256 colors.

A process of performing the address conversion of the first process will be described with reference to the flowchart of FIG. 4.

According to an exemplary embodiment of the present invention, a single panoramic image is generated by k frames including n lines. From the first line of the first frame to the k-th frame, the first memory is divided into frames. The n th line of the k th frame is sequentially read in line number order and stored in the second memory to generate a single panoramic image.

In the initialization step 50 of FIG. 4, the frame variable i and the line variable j are initialized to 1, and in the reading step 51, the j line data of the i frame is read from the first memory, and the storage step 52 is read. i-frame j-line data is sequentially stored on the second memory.

Subsequently, after the frame variable i is increased by one (53) and the increased frame variable i is not larger than the last frame number k, the reading step 51 and the storing step 52 are repeated for the same line number of another frame. If the increased frame variable i is greater than the last frame number k, the process is repeated for the next row.

That is, if the increased frame variable i is greater than the last frame number k, the row variable j is increased by 1 (55), and if the increased row variable j is not greater than the last row number n after the initialization of the frame variable i to 1, The reading step 51 and the storing step 52 are repeated for the row number, and the process ends when the increased row variable j is greater than the last row number k (57).

When such a process is performed, one panoramic image is stored in the second memory.

Next, to help understand the second process of vector quantizing one panoramic image obtained in the first process, the concept of low vector quantization will be described.

Vector quantization used in the present invention has been widely used due to the fact that it is possible to reconstruct an image having a good image quality with a high condensation rate, and that the decoder is simple, so that hardware can be easily applied in real time. The vector quantization method is divided into a codebook generation process and a compression process using the generated codebook. In particular, the codebook generation process is an important process that determines the vector quantization performance.

Vector quantization is any vector on the Euclidean space R ^K in the K dimension. Is a representative vector belonging to a finite size subset Y on R ^K. It is defined as the mapping Q corresponding to.

In Equation 1 Represents a set of representative vectors, that is, a codebook, and index N represents the number of representative vectors.

Input vector Representing vector Distortion between two vectors Minimizing is the optimal idea.

The LBG algorithm is frequently used to design an optimal codebook for a given training vector, and to find a representative vector for an optimal correspondence in the designed codebook. As a distortion measure, a square error formula such as Equation 2 is used.

here, Denote a vector of the original image vector and the reconstructed image, respectively.

In this way, the codebook is divided into RANDOM CODES, BINARY-SPLITTING, pairwise nearest neighbor (PNN), and CLUSTERING depending on how the initial value is set using the LBG algorithm.

That is, in vector quantization, there is a slight difference in the form of the block and the progress of the algorithm. However, in most cases, the MSE between the block and the vectors is basically calculated and the block is assigned to the vector having the minimum value. It replaces the pixels in this block and compresses them. This means that the codebook looks for a block to replace one block, with the least distortion.

In addition, when the vector quantization compression technique is applied, the compression ratio is expressed by Equation 3 below.

As shown in Equation 3, the compression ratio is inversely proportional to the log value of the representative vector, that is, the size of the codebook, and is proportional to the number of colors, the number of bits representing the color, the vector width, and the vector height.

5 is a flowchart for explaining an algorithm of VQ used in the present invention, and FIG. 5 is a codebook generation algorithm by a clustering algorithm (CLUSTRIN) method.

The clustering algorithm is a kind of iteration method of constructing and clustering a training set (TS) with sample vectors called a training vector because the probability distribution function of the signal to be quantized is not well known. Design the codebook.

The codebook is designed by clustering the training vectors by a predetermined number of codewords.

This method forms larger color groups by successively grouping pairs of color clusters with the least distortion measure and constructs a palette with the center values of these color groups.

In this embodiment, the color palette is composed by using the grouping method. In order to speed up the performance, the color palette is searched for each color group on the three-dimensional histogram composed of red, green, and blue to simulate the center color in the block. Color groups with small distortion measures were grouped.

The initial codebook vector receives the color group with high probability of occurrence as a component vector of the codebook and performs the following process.

Referring to the variable used in the flowchart of FIG. 5, TH represents a degree of distortion between the actual vector and the codebook vector, and Nc is a constant representing the maximum number of codebook vectors to be desired. N is the number of vectors stored so far in the codebook, i is the index of the training vector, and j is the index of the vector stored in the codebook.

(510) Initialize the threshold (TH) and the maximum number of codebooks (Nc).

(520) Initialize the number n of the vectors stored so far in the codebook and the index i of the training vector to 0.

(530) A color vector (train vector: TV [i]) of the actual image is input.

(540) The index j of the codebook vector is initialized to zero.

(550) The first vector (CV [0]) of the codebook is input.

A distortion D between the codebook vector CV [j] and the input training vector TV [i] is obtained and the threshold TH and the distortion D are compared.

If the distortion (D) is smaller than the threshold (TH), the training vector is considered to be similar to the codebook vector, and the training vector index is increased by 1 to receive the next training vector (i = i + 1). Repeat from step (3).

If the distortion (D) is greater than the threshold (TH), the training vector is different from the current codebook vector, and thus increases the codebook vector index by one (j = j + 1).

In order to check whether there is a next vector in the codebook, the number n stored in the current codebook is compared with the size of the codebook vector index j in step 580, and if there is a next code vector in the codebook (nj), the (650) Follow the steps.

(600) If there is no next code vector in the codebook (nj), then compare the currently stored number n of codebooks with the maximum number Nc of codebooks.

As a result of the comparison in the step (600), if the current vector number n is smaller than the maximum vector number Nc, the training vector is stored in the codebook, and the number of vector of the codebook is increased by one (n = n + 1). Repeat from step 570.

As a result of the comparison of the step (600), if the current number of vectors n is greater than the maximum number of vectors Nc, the canal ends because the number of codebooks is completed.

As described above, rather than scalar quantization of values of color signals as one-dimensional signals, better performance can be obtained by performing quantization using each color signal value as a vector component using redundancy of the color signals. That is, if the original image has 65,000 pixel values, the codebook is designed to map the pixel values within a certain range to 256 levels by limiting them to 256 pixel values, and transmit only the index of the codebook in the encoding process. This increases the compression efficiency and the transmission efficiency.

6 is a configuration diagram of a virtual reality system for displaying a panoramic image according to the present invention. The apparatus of the present invention includes an image data storage unit 60, a panoramic image generation unit 61, and a VQ. Processing unit 62, panoramic image storage unit 63, first three-dimensional graphics board 64-1, second three-dimensional graphics board 64-2, first signal converter 65-1, first Computer system composed of two signal transducers 65-2, three-dimensional sound processor 66, position and direction trackers 67, viewing angle calculator 68, CPU 69, HMD 70, transmitter 71 ) And an input device 72.

The image data storage unit 60 is provided with image data divided into a plurality of frames, and a hard disk drive (HDD) or other auxiliary storage devices may be used.

The panoramic image generator 61 receives image data divided into a plurality of frames and generates a single panoramic image.

The VQ processor 62 generates a codebook and a codeword by compressing the single panoramic image generated by the panoramic image generator 61 by vector quantization.

The panoramic image storage unit 63 stores the generated codebook and codeword.

The first 3D graphic processor 64-1 indexes the codebook stored in the panoramic image storage 63 with image data of one frame corresponding to the user's viewpoint input from the viewing angle calculator 68. The codeword corresponding to (N) is read, rendered, and texture mapped to generate the left RGB image signal.

The second 3D graphic processor 64-2 indexes image data of one frame corresponding to the viewpoint of the user input from the viewing angle calculator 68 to the codebook stored in the panoramic image storage unit 63. The codeword corresponding to (N) is read, rendered, and texture mapped to generate the right RGB image signal.

In this case, the technique used to correspond to the change in the viewpoint of the panoramic image in the three-dimensional graphics is to texture-map the panoramic image inside the cylindrical coordinate axis, and then the view volume ( view volum) is calculated and only the relevant data is extracted and displayed. To this end, the head direction of the user is detected by the sensor of the HMD 70 and the position and direction are calculated by the position and direction tracker 67.

The first and second signal converters 65-1 and 65-2 are signal converters for converting RGB outputs of the 3D graphics boards 64-1 and 64-2 into TV video signals.

The HMD 70 receives and displays the outputs of the signal converters 65-1 and 65-2, detects an image display and a movement for detecting a user's movement, and transmits the image to the position and direction tracker 67. Do it.

The position and direction tracker 67 receives a user's motion signal from the HMD 70, calculates the position and direction of the user, and transmits the position and direction to the viewing angle calculator 68.

The viewing angle calculator 68 calculates a view volume of the panoramic image according to the user's movement input from the position and direction tracker 67 and outputs the view volume to the 3D graphics boards 64-1 and 64-2. do.

In this case, the view volume calculation of the viewing angle calculator 68 calculates an amount of change in the angle of the user head, calculates the amount of pixel change on the display according to this value, and sets a new view point by adding the amount of pixel change to the current view point. Based on this value, the view volume is calculated.

The CPU 69 executes a predetermined program to control various components constituting the system.

The input device 72 is for a user to freely operate in the virtual reality world, and a joystick, a mouse, a data globe, and the like may be used.

In the HMD 70, the transmitter 71 transmits a radio signal according to the control signal of the position and direction tracker 67 to detect the direction of the user, and the sensor inside the HMD 70 receives the position and When the signal received by the direction tracker 11 is transmitted, the position and direction tracker 67 analyzes the received signal to detect the position and direction of the user.

As described above, the encoding operation of the VQ processing unit for vector quantizing and displaying a panoramic image composed of a plurality of frames is compared with the N representative vectors of the codebook when one input vector is input for encoding. Select, quantize, and export one index.

At this point, the VLSI structure of the VQ requires a systolic array structure having regularity and modularity due to the huge amount of data and computation, local connection between processing elements, and data flow.

In addition, the decoding process may be simply implemented in the form of a lookup table by a memory such as a ROM, unlike a large amount of computation required in the encoding process.

As described above, according to the present invention, rearranged image data divided into a plurality of frames to generate a panoramic image, vector quantization, and extract only the corresponding data according to a viewpoint to form a frame and then display the image. As a result, the virtual reality system can display a panoramic image naturally according to the user's view point.

In addition, by using vector quantization of color components, memory consumption can be effectively reduced by using only a codebook and a memory for codewords storing only a few representative vectors.

Claims (5)

A first process of generating one panoramic image by rearranging the images divided by frames and storing the stored images in line order; And a second process of performing vector quantization on the panoramic image of the first process to generate a codeword and a codebook and storing the codeword and the codebook in a memory. The method of claim 1, wherein the first process is performed in order of line number from the first memory of the first frame to the n-th line of the k-th frame to the k-th frame of the first memory. A method of generating a panoramic image of a virtual reality system, comprising: performing an address conversion process of reading and storing in a second memory to generate a single panoramic image. The method of claim 1, wherein the second process comprises a process of designing a codebook having an index of leveling M pixel values of an actual image with N representative vectors by using an RGB color component as a vector amount. How to create a panoramic image of a virtual reality system. Image data storage means (60); Panoramic image generating means (61) for receiving image data divided into a plurality of frames of the image data storing means (60) and generating a single panoramic image; VQ processing means (62) for compressing the generated panoramic image to generate a codebook and codeword; A panoramic image storing means (63) for storing the generated codebook and codeword; 3D graphics processing unit 64-1, 64-2 for reading image data of one frame corresponding to the user's viewpoint from the panoramic image storage means 63, rendering and texture mapping and outputting an RGB image signal ; Signal converters 65-1 and 65-2 for converting the RGB output of the three-dimensional graphics processing units 64-1 and 64-2 into TV video signals; Image display and motion detection means (66) for receiving and displaying the outputs of the signal converters (65-1, 65-2) and detecting a user's movement; A position and direction tracker 67 which receives a user's motion signal from the image display and motion detection means 66 and calculates the position and direction of the user; And The viewing angle calculator 78 calculates a view volume of the panoramic image according to the movement of the user input from the position and direction tracker 67 and outputs the output to the 3D graphics processing units 64-1 and 64-2. Panoramic image generating device of a virtual reality system, characterized in that comprising a. The nth line of the k th frame from the first line of the first frame from the image data storage unit 60 in which k frames of n rows are stored. And an address conversion function for generating a single panoramic image stored in line order by sequentially reading up to line number order.