KR20110107691A - Spatial augmented reality-based interactive display system and method of the same - Google Patents

Abstract

The present invention relates to a space augmented reality-based interactive display system and a method thereof, by outputting a guide post to a fixed object or a fluid object fixed on the exhibition space, an observer can obtain information on the exhibition object by simply viewing the exhibition object. It can guide you to a large number of objects in a space. In addition, a multi-touch screen is manufactured using a beam projector and an infrared LED, and a display environment for augmented reality is realized using a semi-projection mirror and a large image output device, thereby increasing immersion and realism.
According to an aspect of the present invention, there is provided a spatial augmented reality-based interactive display system comprising: a multi-touch screen configured to receive a signal for content progression from a user; An exhibition space for displaying a still or moving subject; A camera for real-time photographing a state of the subject in the exhibition space; An image output device for outputting an image for spatial enhancement made by information input through the multi-touch screen; A semi-projection mirror for overlapping the appearance of the exhibition space and the output image of the image output device; A beam projector for outputting a content progress screen to the multi-touch screen; An infrared LED array bar outputting infrared light under the multi-touch screen to form an interaction film; An infrared camera configured to input an infrared light emitted from the interaction layer when the hand touches the multi-touch screen; And receiving the image from the camera to calculate the location information of the subject in real time, and receiving the image from the infrared camera to calculate the user's touch point, such that the information input through the multi-touch screen is the same as the movement of the subject. And a control PC outputting through the image output apparatus or executing contents of a corresponding touch point through the image output apparatus to move in a shape.

Description

Spatial Augmented Reality-based Interactive Display System and Its Method {SPATIAL AUGMENTED REALITY-BASED INTERACTIVE DISPLAY SYSTEM AND METHOD OF THE SAME}

The present invention relates to a spatial augmented reality-based interactive display system and a method thereof, and more particularly, information on each object so that an observer can easily obtain information about the object only by viewing the object on display. The present invention relates to a spatial augmented reality-based interactive display system and a method for outputting the same to the side of an object that is displayed.

In addition, the present invention provides a multi-touch screen using a beam projector and infrared LED, a spatial augmented reality based interactive display system using a semi-projection mirror and a large image output device to implement a display environment for spatial augmented reality and its It is about a method.

In general, information about exhibits in a museum or exhibition hall is briefly explained in the form of pictures or letters on a panel attached to the wall of the exhibition hall, and thus there is a limit to obtaining in-depth knowledge or information about the exhibit. Existing exhibition systems generally require the general composition and guidance of similar items rather than individual exhibitions. Posts are provided separately from the exhibition space where viewers observe each exhibition to obtain information about the exhibits. Browsing this information can be time-consuming and difficult. In particular, many observers may need to move without obtaining the information they want.

Although the voice guidance system is established to solve these existing limitations, when the voice guidance system is operated simultaneously in various places inside the exhibition hall, it is difficult for the observer to listen to the voice guidance required by the viewer, and if a user passes the desired information, the user spends a lot of time again. There is a problem of being guided. In addition, children may gather in front of the exhibition booths while taking notes on the panel, which may disturb other visitors.The names of the exhibits are displayed in English only. An unknown problem occurs.

In order to solve this problem, Korean Patent Publication No. 2005-0071114 (Article guidance service system and method using a wireless terminal), Korean Patent Publication No. 2002-0059150 (Exhibit information guide service method), and domestic published patent No. 2003-0055236 (Method and system for providing exhibition site information service) and Korean Patent Publication No. 2003-0057053 (Method for providing tour and exhibition viewing information using mobile communication) tag apparatus around the exhibit or exhibit The present invention discloses a system for providing a guide of an exhibition through a tag device when installed and approaching the exhibition using a visitor terminal (PDA, mobile phone, etc.).

However, this conventional technology not only entails huge initial costs by providing information on exhibitions using a mobile communication terminal but also provides only simple information. Accordingly, conventionally, an exhibition environment system incorporating virtual reality technology has been developed to maximize the immersion of viewers. However, even in this case, there is a limit to the representation of some special types of exhibits due to the huge installation space and enormous cost. In particular, expensive equipment such as HMD (Head Mount Display) is not only additionally necessary, but also has a limited number of viewers because it is limited in maximization of immersion.

In addition, the conventional exhibition system provides information about only a simple fixed object by providing a guide plate composed of photographs and texts provided separately from the exhibition space as a method for guiding the exhibits, Many limitations have arisen in obtaining the desired information in time.

Recently, researches on multi-touch screens that can naturally interact with a user's hand without any tools have been actively conducted. As a method for displaying a 3D image, which is more realistic than a conventional 2D image, the research on display using spatial augmented reality has been actively conducted in order to provide a more realistic image than a 3D image.

The technical problem to be solved by the present invention to solve the above-mentioned problem is to configure the form of outputting a guide post to a fixed object or a fluid object fixed on the exhibition space to observe the information of the exhibition object by simply viewing the exhibit. A spatial augmented reality based interactive display system and method for guiding a large amount of objects that can be acquired and exist in a space are provided.

In addition, the other technical problem to be achieved by the present invention is not only to guide the exhibition, but also to induce the interest of the audience, and in addition, the user can directly participate in the form of learning through the interaction with the system The present invention provides a spatial augmented reality-based interactive display system and method for providing an educational platform.

In addition, another technical problem to be achieved by the present invention is to produce a multi-touch screen using a beam projector and an infrared LED, and to enhance the space to realize a display environment for space augmented reality using a semi-projection mirror and a large image output device The present invention provides a reality-based interactive display system and method thereof.

In addition, another technical problem to be achieved by the present invention is to propose a spatial augmented reality-based interactive display system and method that enhances immersion and realism by showing a minimum of the virtual image based on the actual image.

In addition, another technical problem to be achieved by the present invention is to present a spatial augmented reality-based interactive display system and method for enhancing the immersion and reality by enabling the user input information to be confirmed as a real image created on the basis of spatial augmented reality There is.

The problem of the present invention is not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

As a means for solving the above-described technical problem, the invention as set forth in claim 1 is a space-augmented reality-based interactive display system, comprising: an exhibition space for displaying a still or moving subject; A camera for photographing the state of the subject in real time; An image output device for outputting guide information of the subject; A semi-projection mirror for superimposing the appearance of the exhibition space and guide information output from the image output device; And a control PC which receives an image signal from the camera, calculates position information of the subject through image processing, and outputs guide information to be displayed on the subject to the image output apparatus. Is provided.

In addition, as another means for solving the above technical problem, the invention described in claim 2, "Spatial Augmented Reality-based interactive display system, comprising: a multi-touch screen for receiving a signal for content progression from the user; An exhibition space for displaying a still or moving subject; A camera for real-time photographing a state of the subject in the exhibition space; An image output device for outputting an image for spatial enhancement made by information input through the multi-touch screen; A semi-projection mirror for overlapping the appearance of the exhibition space and the output image of the image output device; A beam projector for outputting a content progress screen to the multi-touch screen; An infrared LED array bar outputting infrared light under the multi-touch screen to form an interaction film; An infrared camera configured to input an infrared light emitted from the interaction layer when the hand touches the multi-touch screen; And receiving the image from the camera to calculate the location information of the subject in real time, and receiving the image from the infrared camera to calculate the user's touch point, such that the information input through the multi-touch screen is the same as the movement of the subject. It provides a spatial augmented reality-based interactive display system comprising a; control PC for outputting through the image output device or moving the contents of the touch point through the image output device to move in a shape.

The invention according to claim 3, wherein the interactive display system comprises: a total reflection mirror (206) for reflecting an image output from the beam projector to the multi-touch screen; A total reflection mirror (208) for reflecting the subject image to the camera; White LED lighting for adjusting the brightness of the exhibition space to be the same or similar to the brightness of the image output from the image output device; And a vertical moving device for adjusting a depth of an image felt by a user by adjusting the height of the image output device. ”The present invention provides a spatial augmented reality-based interactive display system.

The invention according to claim 4, wherein the multi-touch screen is composed of an acrylic plate and a rear screen, and the infrared LED array bar is provided with a plurality of infrared LEDs at regular intervals, Space augmented reality-based interactive display system characterized by blocking the light to the wall to improve the straightness of light.

The invention as set forth in claim 5, wherein the positional information of the subject is calculated by calculating an x, y coordinate of the subject through an image corresponding to the front side of the subject among the input images of the camera. And calculating the z coordinate of the subject using an image corresponding to the upper surface of the subject among the input images.

According to the invention of claim 6, "The method of claim 1, wherein the guide information: based on spatial augmented reality, characterized in that it comprises any one of the image, audio, text, graphic image, user input value according to the exhibition guide Interactive display system.

In addition, as another means for solving the above-described technical problem, the invention of claim 7 is a space-augmented reality-based interactive display method comprising the steps of: (a) real-time photographing the state of the subject in the exhibition space with a camera; (b) image processing the image signal of the camera to calculate position information of the subject and outputting guide information to be displayed on the subject to an image output apparatus; And (c) displaying the exhibition space and the guide information output from the image output apparatus by overlapping them through a semi-projection mirror. ”The present invention provides a spatial augmented reality-based interactive display method.

In addition, as another means for solving the above-described technical problem, the invention according to claim 8, "In the space augmented reality based interactive display method, (a) real-time photographing the state of the subject in the exhibition space with a camera; (b) receiving a signal for content progress from a user through a multi-touch screen; (c) image inputting infrared light emitted from an interaction film formed in the infrared under the multi-touch screen when the hand touches the multi-touch screen with an infrared camera; (d) image processing the image signal of the camera to calculate position information of the subject, receiving an image from the infrared camera, calculating a user's touch point, and moving information of the subject through the multi-touch screen; Outputting the image output apparatus to move in a shape such as or outputting a content of a corresponding touch point through the image output apparatus; And (e) superimposing the appearance of the exhibition space and the image information output from the image output apparatus through a semi-projection mirror. ”The present invention provides a spatial augmented reality-based interactive display method.

The invention as set forth in claim 9, wherein the interactive display method comprises: calculating a position coordinate of a subject through a background modeling tracking algorithm for one subject, Spatial augmented reality-based interactive display method characterized in that for calculating the position coordinates of the subject through a CamShift tracking algorithm.

The invention as set forth in claim 10, wherein the interactive display method comprises: pre-calculating a degree to be distorted by the tilt of the translucent mirror, and inversely correcting the image to be deformed upon output of the image. Spatial augmented reality-based interactive display method characterized in that the correction by using a reflection transformation matrix to correct according to the inclination of the semi-projection mirror.

According to the present invention, by applying the next-generation display technology to the conventional exhibition hall guide system, the space augmented reality technology applied only to a fixed object can be efficiently used for producing various 3D contents. This is also effective in terms of education in which quick and accurate information can be obtained from simple viewing aquariums.

In addition, localization of a space-augmented reality-based stereoscopic image display system can replace high-cost imports, thereby reducing market costs and contributing to overseas exports and preoccupation of overseas markets.

In addition, the present invention can be utilized as an exhibition and advertisement system capable of delivering various and efficient messages, and has high applicability to related related content markets (future games, etc.), thereby enabling the development of various types of projects.

In addition, by augmenting and displaying the minimum virtual image based on the actual image, the composition can be configured at a lower cost than the virtual reality environment, and the efficiency of learning can be improved by increasing the immersion and the realism.

In addition, by generating the main character of the content through the user's input it can cause interest and interest.

In addition, there is no damage to others viewing other exhibits, such as a voice guidance system, and there is no inconvenience for all observers to obtain information they want when viewing a group.

In addition, it is possible to induce interest and curiosity about the displayed contents and to easily obtain the corresponding information, and it is possible to demonstrate in a relatively small space without additional equipment by using the space augmented reality technology.

The effects of the present invention are not limited to those mentioned above, and other effects that are not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a block diagram of a spatial augmented reality based interactive display system according to an embodiment of the present invention
2 is an internal flow chart illustrating the system operation principle of the present invention.
3 is a diagram illustrating a process of determining a location coordinate of a subject by separating a background and a subject;
4 is a diagram illustrating a state of detecting two different subjects through a CamShift algorithm.
5 is a diagram illustrating an error caused by a distorted input image.
6 is a diagram illustrating a method of calculating depth information based on a corrected image.
7 is a photograph showing a camera input image to which distortion correction is applied
FIG. 8 is a view for explaining a method of tracking a flexible movement to a position of a subject toward a calculated coordinate value when a change in the position of the subject occurs while moving to a position of the subject
9 is a block diagram of a spatial augmented reality based interactive display system according to another embodiment of the present invention
10 is a product photograph of an IR-LEDs Array Bar.
11 to 39 are capture screens showing an implementation example of the spatial augmented reality based interactive display system according to the present invention.
11 to 15 are capture screens showing an example of realizing the information input by the user through the multi-touch screen as content in the exhibition space inside the system,
16 to 39 are capture screens showing an example of the implementation of the educational learning content for the exhibition environment of the exhibition space.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present invention. In order to clearly explain the present invention in the drawings, parts not related to the description are omitted, and similar parts are denoted by similar reference numerals throughout the specification.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

Space Augmented Reality Based Interactive  Embodiment of the display system

1 is a block diagram of a spatial augmented reality based interactive display system according to a preferred embodiment of the present invention, Figure 2 is an internal flow chart showing the operating principle of the system of the present invention.

1 and 2, the spatial augmented reality-based interactive display system according to the present invention includes an image output device 110, a semi-projection mirror 120, a CCD camera 130, and an actual exhibition space 140. do.

First, the image output device 110 is composed of an LCD display device, and finally outputs the image calculated by the algorithm of FIG. Next, the semi-projection mirror 120 installed in an inclined state is a surface on which the image output from the image output apparatus 110 is projected, and the image of the real space behind the mirror is reflected along with the image reflected on the mirror. Allow viewers to watch the scene at the same time. In addition, the camera 130 installed on the ceiling of the system provides an image for calculating the position information of the moving object by capturing a real time picture of the actual exhibition space 140 in real time. Lastly, the actual exhibition space 140 installed at the rear of the semi-projection mirror 120 is allocated as a space for a moving subject to arrange a floating object for exhibition.

The present invention displays a space to display the flow object and the flow object on the actual exhibition space 140 on the back of the mirror, and the real-time image with the camera 120 is installed on the ceiling and the top of the exhibition space 140 It receives the input, outputs the image generated by the internal calculation to the image output device 110, and guides the exhibit by projecting on the semi-projection mirror 120.

As shown in FIG. 2, the spatial augmented reality-based interactive display system of the present invention outputs an image to the image output apparatus 110 through internal calculation. Hereinafter, to facilitate understanding, the exhibition space 140 will be described taking an aquarium as an example.

First, the position coordinates of the object are calculated in real time based on the front region of the aquarium 140 in the image received through the camera 130 (steps S100 to S110). At the same time, depth information of the object is calculated based on only the area of the water surface corresponding to the upper surface of the aquarium 140 of the received image (step S120).

Here, the method of calculating the position coordinates of the real time object is as follows.

In this system, the position coordinates of an object are calculated through a tracking algorithm called Background Modeling for one object, and the position is calculated through a tracking algorithm called CamShift for several objects. Below is the contents of the algorithm.

In order to separate the background from the image input to the CCD camera 130 and calculate the position coordinates of the subject, the background and the color model of each subject should be learned. First, the average image and the standard deviation image are obtained for the background image to be learned in the RGB color model. In this process, since the channel characteristics of the color are used, the R, G, and B colors of the image are respectively divided.

However, when the average image and the standard deviation image are simply used based on the RGB color model, an error occurs in calculating the subject image position due to the shadow generated by the subject. To solve this problem, the RGB color model is normalized to obtain an average image and a standard deviation image as shown in Equation 1 below.

는 각각의 컬러 채널이며,

는 정규화 된 각각의 컬러 채널이다. 상기 수학식 1에서

를 나타낸다. 정규화된 rgb 컬러 모델에서는 밝기 성분이 제거되기 때문에 밝기에 대한 컬러 구분이 없다. 그 결과 입력되는 피사체의 영상으로부터 그림자 영역을 제거할 수 있다.In Equation (1)

Are the respective color channels,

Is each color channel normalized. In Equation (1)

Indicates. In the normalized rgb color model, there is no color distinction for brightness because the brightness component is removed. As a result, the shadow area may be removed from the input image of the subject.

는 학습 될 배경 영상,

은 학습할 전체 프레임 수를 나타낸다.

는 각 채널에 대한 평균 영상,

는 각 채널에 대한 표준편차 영상이다. 상기 수학식 3에서

는 정규화 된 각 채널에 대한 평균영상,

는 정규화 된 각 채널에 대한 표준편차 영상이다.The average image and the standard deviation image of the normalized rgb color model are obtained in the same manner as in Equation 2 above.

Background image to be learned,

Represents the total number of frames to learn.

Is the average image for each channel,

Is the standard deviation image for each channel. In Equation 3

Is the average image for each normalized channel,

Is the standard deviation image for each normalized channel.

When the background average image and the standard deviation image are obtained through the background learning, the image is compared with the input image in real time. Each pixel difference is calculated from the calculated image and the input image in real time to detect pixels having different values from those of the existing image. At this time, the detected pixels are designated as blobs, and the largest range of blobs is designated as a subject through size comparison, and the remaining blobs are regarded as an error value due to noise and removed. Finally, the average coordinates of the pixels constituting the blob of the subject are calculated to obtain the center coordinates of the subject.

3 is a diagram illustrating a process of determining a position coordinate of a subject by separating a background and a subject, (a) detecting an average image of a background, (b) inserting a subject (fish), and (c) separating a background and a subject and (d) indicate acquisition of the center point of the subject, respectively.

However, when using this background modeling technique, since the change of the background image is calculated and used, there is no problem in separating the background image and the subject when there is one target object, but when there are two or more subjects to be separated, the two subjects overlap even a little. If it is removed and separated, a problem may occur in which the calculated position coordinates and the subject do not coincide.

As mentioned above, CamShift-based tracking technique is used to compensate for the problem that occurs when using the background modeling technique.

4 is a diagram illustrating a state of detecting two different subjects through a CamShift algorithm.

If the color model is properly adapted to different lighting conditions, color information will be an efficient tool for identifying features. MeanShift and CamShift algorithms, which use these color information, can perform real-time tracking with relatively few calculations. However, in the case of the MeanShift algorithm, there is a problem in that the center of the subject cannot be found because the subject size change according to the change of the distance between the subject and the camera cannot be newly updated. CamShift can compensate for the weaknesses of MeanShift, and can track irregular subject movements, and acquire the position coordinates of the tracking subject unless another object overlaps 100% with the current tracking subject.

Next, in order to calculate the depth information of the object based on the surface area of the water, the present invention assigns four vertices of the corners from the top of the object to be displayed, allocates the front and back boundary lines, and divides the corresponding sections at regular intervals. Apply the depth information value from front to back.

However, since the image input in the form of FIG. 5 (a) is distorted due to the angle at which the camera looks at the subject as shown in FIG. 5 (b), the input image is unilaterally illustrated in FIG. 5 (c). In the case of division, an error different from the actual distance as shown in FIG. When divided consistently from the camera's point of view, the distance per unit is substantially shorter in the front and gradually longer in the rear. In order to correct an irregularly allocated error due to such image distortion, a homography matrix is used to correct an error of an input image with respect to an angle at which a camera looks at a subject.

For a plane, the point on the plane and the corresponding point of the other plane can be expressed in a 3x3 matrix. The 3x3 matrix here represents the projection transformation relationship between the two planes, which is called homography. Multiplying such a homography matrix by a 9x1 vector lexically by an nx9 matrix that combines points on the plane with corresponding points on the other plane results in zero. In this process, a 9 × 1 vector in which the homography matrix is arranged in a dictionary can be obtained through SVD (Singular Value Decomposition). However, since the n × 9 matrix that combines the points on the plane with the corresponding points on the other plane is not a square matrix, SVD cannot be applied immediately. A pseudo inverse is used to solve this.

에서

는

행렬을 사전식으로 배열한

벡터이며,

는

와

를 조합한

행렬이다. 여기서,

는 SVD를 통하여 계산을 하며, 호모그래피 행렬인

는

의 가장 작은 고유값에 해당하는 고유벡터들로 구성된다.As shown in Equation 4,

in

Is

Lexicographically arranging the matrix

Vector,

Is

Wow

Combining

It is a matrix. here,

Is computed through SVD, which is a homography matrix

Is

It consists of eigenvectors corresponding to the smallest eigenvalues of.

Using the homography matrix calculated through the above process, a corrected image as shown in FIG. 6 (b) is obtained, and the images are distributed at regular intervals as shown in FIG. Depth information values are calculated according to the position of the object by generating a reference plane as shown in FIG. This can be seen in the figure.

7B is generated by applying a homography matrix to the actual input image of FIG. 7A, and the z-coordinate is designated based on the subject position.

In this way, the guide image suitable for the target is output by using the calculated position coordinates of the object, x, y, and z. In this case, a change in the direction of the moving subject is changed to obtain a more natural result. In order to prevent the disconnection of the output image due to the frame degradation that may occur in this process, it is possible to check the results of the more natural movement by outputting the image by a flexible motion tracking technique through parallel processing.

Here, in the present invention, instead of the method of outputting the guidance text to the corresponding coordinates in real time, the algorithm is applied to track the flexible movement to the position of the subject toward the calculated coordinate value.

FIG. 8 shows that when a change occurs in the position of the subject while moving to the position of the subject (a), the direction is changed in consideration of mass and acceleration (b) to reach the position of the subject with flexible movement (c).

When the user sets the mass to an appropriate value for an object that tracks a moving subject, the object shows a smooth movement that is appropriate for its acceleration and mass. When the stationary object detects the movement of the subject and starts tracking, the agility changes according to the set acceleration, and when the subject moves in the direction of rotation, the rotating form changes according to the set mass. As a result, when the object arrives around the object that has stopped moving, the object that has stopped tracking stops the movement and observes a change in the position coordinate of the object.

When the image generated through this process is directly projected on a mirror installed at an inclination, distortion occurs due to the tilt of the mirror. The distortion caused by the tilt of the mirror can be corrected to an appropriate value according to the tilt using the reflection transformation matrix.

Here, there are matters to be considered when manufacturing a space augmented reality environment using the semi-projection mirror 120. The image reflected on the semi-projection mirror 120 is a distortion caused by the tilt of the mirror. The reflection matrix is used as a method to solve such a problem. The degree of distortion by the tilted mirror is calculated in advance, and the image to be distorted at the time of output is corrected in reverse and then output.

The reflection transformation matrix (Equation 3) applied to the present invention is used in a model-view transformation process of OpenGL widely used in the field of computer graphics.

는 일반적인 평면의 방정식을 통하여 계산할 수 있다. 상기 반투영 거울(120)이 바라보는 법선 벡터

와 상기 반투영 거울(120)상에 존재하는 임의의 점

와 원점 0, 0, 0이 있을 경우 평면의 방정식은

과 같다. 즉, 평면의 방정식을 이용하면 기울어진 거울로 인하여 생길 수 있는 왜곡을 보정 하기에 알맞은 반사 변환 행렬을 구할 수 있다.Of Equation 5

Can be calculated through the equation of a general plane. Normal vector viewed by the translucent mirror 120

And any point present on the translucent mirror 120

And origin zero, zero, zero, the equation of the plane

Is the same as In other words, by using the equation of the plane, it is possible to obtain a reflection transformation matrix suitable for correcting the distortion caused by the tilted mirror.

In general OpenGL, the coordinate system set for modeling by object is the model coordinate system (Object Coordinates). When the individual objects are collected, the coordinate system that can encompass them at once is the world coordinate system and the world coordinate system is expressed based on the viewpoint of the object. However, when the reflection transformation matrix is combined, small changes occur in the global coordinate system and the viewpoint coordinate system.

As described above, the reflected global coordinate system is generated by applying the reflection transformation matrix to the generated global coordinate system. The previously calculated view coordinate system will also be used as new reflected eye coordinates. In this way, the distortion of the mirror may be solved by modifying and outputting a distorted image due to the tilt of the mirror.

Finally, when projecting the corrected image through the reflection transformation matrix to the semi-projection mirror 120 is installed, the observer is a spatial augmented guide to the information about the moving subject in the real space installed on the back of the mirror Can be easily obtained by

Space Augmented Reality Based Interactive  Another embodiment of a display system

9 is a configuration diagram of a spatial augmented reality based interactive display system according to another exemplary embodiment of the present invention.

As shown in FIG. 9, the spatial augmented reality-based interactive display system includes an actual exhibition space 201, an image output device 202, a multi-touch screen 203, and an infrared LED array bar in a spatial augmented reality display or system. (IR-LEDs Array Bar: 204), Infrared Camera (205), Total Reflection Mirror (206), Translucent Mirror (207), Total Reflection Mirror (208), Camera (209), White LED Light (210), It comprises a vertical moving device 211, a beam projector 212, the control PC (213).

Here, the spatial augmented reality display 201 allows a user to view a spatially enhanced image. That is, the multi-touch screen 203 allows the user to view a spatial augmented reality environment that appears to be contained in the actual exhibition space 201 installed inside the system.

The image output device 202 outputs an image for spatial enhancement made by the control PC 213 to the semi-projection mirror 207 with information input by a user through the multi-touch screen 203.

The multi-touch screen 203 transmits information as a monitor and receives user information for content progress. The image output from the beam projector 212 is reflected and displayed on the total reflection mirror 206 on the multi-touch screen 203, and guides the user on how to use the content. The multi-touch screen 203 is composed of an acrylic plate and a rear screen.

The IR-LEDs array bar 204 is configured to detect the touch point of the user input through the multi-touch screen 203, and the top and bottom (or left and right) of the multi-touch screen 203. Installed on the right side) to output infrared light to form an interactive surface under the multi-touch screen 203. In this case, when the hand touches the multi-touch screen 203, infrared light is emitted from the interaction layer at the point where the hand is touched.

As shown in the photo of FIG. 10, the infrared LED array bar 204 installs a plurality of infrared LEDs (IR-LEDs) 220 at regular intervals (for example, 3 cm intervals). In this case, the reason for disposing the infrared LED 220 at regular intervals is to make the amount of infrared light uniform for all regions. In addition, the reason for setting the spacing of the infrared LEDs 220 to 3 cm is a distance of 3 cm, which was found to be adequate as a result of testing a distance that can be supplied with sufficient amount of infrared light while considering the amount of heat generated by each LED. However, it is not necessarily limited to this value and may vary depending on the LED device or other environment.

The infrared LED 220 generally emits light at a 45 degree angle. Therefore, in the infrared LED array bar 204 of the present invention, by blocking both sides of the infrared LED 220 to the wall in order to prevent the light of the LED diverging at 45 ° to configure the light to go out only vertically Improved straightness.

Meanwhile, only one infrared LED array bar 204 may be installed at the top or the bottom of the multi-touch screen 203.

The infrared camera 205 is for inputting an image of a user's touch point to the multi-touch screen 203, and the infrared LED array bar 204 when a hand is touched on the multi-touch screen 203. The infrared light emitted from the interaction film (region) formed by the image is input.

The total reflection mirror 206 reflects the image output from the beam projector 212 to the multi-touch screen 203. In this case, the reason for reflecting the image output from the beam projector 212 using the total reflection mirror 206 is that as the distance between the beam projector 212 and the multi-touch screen 203 increases, the image of the wide screen Because you can get And, in order to prevent the reduction of light, a total reflection mirror is used instead of a general mirror.

The semi-projection mirror 207 allows the user to view the space-augmented reality environment by overlapping the screen of the image output device 202 and the object of the actual exhibition space 201.

The total reflection mirror 208 serves a similar purpose as the total reflection mirror 206 but performs the opposite operation. That is, the total reflection mirror 208 provides the camera 209 to receive the image of a wide angle of view in a narrow and short space. If the lens of the camera 209 is used as a wide-angle lens, since the image is distorted, an additional device for correcting the distorted image is required, which places an unnecessary load on the system. Thus, the total reflection mirror 208 was used to remove the constraint of the installation space of the camera 209.

The camera 209 inputs a state of the exhibition space 201 through the total reflection mirror 208 as an image, and inputs a base image for calculating a position of a moving object in real time.

The white LED light 210 is an illumination that controls the brightness of the exhibition space 210 such that the brightness of the image output from the image output device 202 and the brightness of the exhibition space 201 are similar. That is, the display space 210 and the image output device 202 by reducing the difference in brightness is installed to increase the realism, the variable resistor is installed for the adjustment of the brightness.

Since the image of the exhibition space and the image output from the image output device 202 are superimposed on one through the semi-projection mirror 207, the brightness of the exhibition space 201 and the image output device 202 are output. The more the brightness of the image is the same, the less heterogeneity is felt. However, since there is a limit in controlling the image brightness of the image output device 202, the image output from the image output device 202 is displayed on the brightness of the exhibition space 201 using the white LED light 210. Adjust the brightness to be similar to or the same as.

The vertical moving device 211 is a vertical moving device for adjusting the height of the image output device 202 according to the situation because the depth of the image felt by the user varies depending on the height of the image output device 202. Here, the augmented image due to the difference in depth of the augmented image may appear to match the object of the exhibition space 201 (a suitable setting), or may appear to be floating in the air or buried inside a background object. Therefore, the height of the image output device 202 is adjusted to an appropriate height using the vertical moving device 211.

The beam projector 212 outputs an image to be output to the multi-touch screen 203 through the total reflection mirror 206.

The control PC 213 receives the object image of the exhibition space 201 from the camera 209 in real time, calculates the position of the moving object in real time, and outputs the image through the beam projector 212. The image output on the multi-touch screen 203 is displayed. In addition, the touch point of the user input to the multi-touch screen 203 is input through the infrared camera 205, the coordinate value of the touch point is calculated, and the content of the corresponding point is executed by the image output device 202. Alternatively, the image or text information input by the user is displayed through the image output apparatus 202. In addition, the image information input by the user to the multi-touch screen 203 is displayed on the image output device 202 to move in the same shape as the movement of the object in the exhibition space 201 according to the space augmented reality program. In addition, the control PC 213 adjusts the height of the image output device 202 through the vertical moving device 211, and controls the brightness of the white LED light 210. In addition, the control PC 213 processes the output image and various operations required by the system.

The infrared camera 205 inputs infrared light emitted from the interaction film (area) formed by the infrared LED array bar 204 when the user touches the multi-touch screen 203 as an image to control the PC. 213). The control PC 213 calculates coordinate (x, y) values of the hand region through image processing of the signal transmitted from the infrared camera 205.

Here, the method of calculating the coordinate (x, y) value of the hand region is as follows.

This function operates by calculating the coordinates that recognize the highest point of the coordinates after correcting the distortion using a homography matrix through the blob / labeling only in the region where the LED is reflected through the image binarization and the region that is not. do. Here, the blob binds the areas where the LED is reflected when the image is binarized. That is, since the human hand does not reflect light to one pixel (point) but reflects light to a plurality of pixels (planes), it serves to bind the areas where the LED is continuously reflected. The labeling is an operation of numbering the regions bounded by the blobs as they are. By assigning a number for each region, it is possible to calculate the coordinate values from when the region is created to disappear. At this time, since the image is two-dimensional rather than three-dimensional, only the coordinate information value of (x, y) is calculated by calculating only the (x, y) coordinate.

After calculating the coordinate value of the touch point, the content of the corresponding point is executed by the image output device 202 or the picture or text information input by the user is displayed through the image output device 202. In addition, the image information input by the user to the multi-touch screen 203 is displayed on the image output device 202 to move in the same shape as the movement of the object in the exhibition space 201 according to the space augmented reality program.

Hereinafter, an example of actually implementing a spatial augmented reality based interactive display system of the present invention will be described with reference to the accompanying drawings.

Space Augmented Reality Based Interactive  Example of display system

11 to 39 are capture screens showing an implementation example of a spatial augmented reality-based interactive display system according to the present invention. FIGS. 11 to 15 show contents input by a user through a multi-touch screen in an exhibition space inside the system. 16 to 39 are capture screens showing an example of the implementation of the educational learning content for the exhibition environment of the exhibition space.

First, as shown in FIG. 11, when the "experience" button is pressed on the content screen of the multi-touch screen 203, the spatial augmented reality display 201 is displayed as shown in FIGS. 12 and 13. 12 and 13 illustrate a state in which an object in the exhibition space 201 and an image in the image output device 202 are overlapped through the semi-projection mirror 207.

Thereafter, the content screen of the multi-touch screen 203 is changed as shown in FIG. 14. 14 to 22 are contents that allow a user-drawn fish to move like a real fish in the exhibition space 201.

Pressing the “draw” button at the bottom of the screen of FIG. 14 brings up a drawing screen as shown in FIG. 15. After selecting a color for drawing with a finger on the drawing screen (FIG. 80) (FIG. 16), when the fish is drawn as shown in FIG. 17, the fish is drawn in the color selected by the user. Subsequently, when the "Draw" button is pressed, the fish drawn by the user in the exhibition space 201 moves as if it is actually alive as shown in FIG. At this time, the movement of the fish drawn by the user moves like an actual fish, as described above, the image signal of the camera 209 which captured the image of the object in the exhibition space 201 in real time to the image on the control PC 213 It is possible to extract the position information (coordinates) of the moving object through the process and apply it to the fish-shaped picture input by the user and output it through the image output device 202.

Next, the screen of FIG. 19 is a content of feeding the fish using a finger and feeding the food. When the user moves the finger on the screen as shown in FIG. 'S picture moves as the finger moves and eats. At this time, the food is an image of the content output through the image output device 202.

23 to 39 are exemplary implementations of educational learning content for an exhibition environment of an exhibition space.

The screen of FIG. 23 shown in the multi-touch screen 202 shows the upstream, midstream, and downstream of the river. As shown in FIG. When touched, a brief description of the touched midstream is displayed on the screen through a balloon, and then the midstream of the river appears in the exhibition space 201 viewed through the semi-projection mirror 207. Of course, the state of the river shown in the exhibition space 201 is an image output through the image output device 202, as described above.

Next, FIGS. 26 to 29 are education learning contents for perception and earthquake, and show the appearance and generation process of perception sequentially through the multi-touch screen 202. 30 to 39 are educational learning contents that a user can directly experience.

As shown in FIG. 30, when the mantle displayed as a picture on the screen of the multi-touch screen 202 is touched according to voice guidance, the screen is changed as shown in FIG. 31 and a detailed description and a figure of the mantle are displayed. Here, when the selection button at the bottom is pressed, the mantle and the strata structure are displayed in a picture as shown in FIG.

When the mantle is touched on the screen of FIG. 32 according to the voice guidance, a stratum pushing view window is displayed as shown in the screen of FIG. 33. After pressing a selection button in the strata push window, the strata are pushed and displayed as shown in FIG. 35 when the strata are pushed with both hands through the multi-touch screen 202 as shown in FIG. 34.

Subsequently, the magma inside the crust erupts out of the crust so that the volcano erupts, as shown in FIG. As shown in the screen, a volcanic eruption is produced in the exhibition space 201 viewed through the translucent mirror 207.

The spatial augmented reality-based interactive display system and method of the present invention configured as described above can solve the technical problem of the present invention by outputting a guide post to a fixed object or a flexible object in the exhibition space.

In addition, another spatial augmented reality-based interactive display system and method thereof of the present invention produce a multi-touch screen using a beam projector and an infrared LED, and display environment for spatial augmented reality using a semi-projection mirror and a large image output device By implementing this, the technical problem of the present invention can be solved.

Preferred embodiments of the present invention described above are disclosed to solve the technical problem, and those skilled in the art to which the present invention pertains (man skilled in the art) various modifications, changes, additions, etc. within the spirit and scope of the present invention. It will be possible to, and such modifications, changes, etc. will be considered to be within the scope of the following claims.

The spatial augmented reality-based interactive display system and method thereof can be used as an exhibition and advertisement system capable of delivering various and efficient messages, and have high application ability to related related content markets (future games, etc.). Development is possible.

110: image output device 120: semi-projection mirror
130: CCD camera 140: exhibition space
200: Spatial Augmented Reality Based Interactive Display System
201: Space augmented reality display or actual exhibition space inside the system
202: video output device 203: multi-touch screen
204 IR-LEDs Array Bar
205: Infrared Camera
206: total reflection mirror 207: semi-projection mirror
208: total reflection mirror 209: camera
210: white LED light 211: vertical moving device
212: beam projector 213: control PC
220: Infrared LEDs

Claims (10)

In the spatial augmented reality based interactive display system,
An exhibition space for displaying a still or moving subject;
A camera for photographing the state of the subject in real time;
An image output device for outputting guide information of the subject;
A semi-projection mirror for superimposing the appearance of the exhibition space and guide information output from the image output device; And
A control PC which receives an image signal from the camera, calculates position information of the subject through image processing, and outputs guide information to be displayed on the subject to the image output apparatus;
Space augmented reality-based interactive display system comprising a.
In the spatial augmented reality based interactive display system,
A multi-touch screen receiving a signal for content progression from a user;
An exhibition space for displaying a still or moving subject;
A camera for real-time photographing a state of the subject in the exhibition space;
An image output device for outputting an image for spatial enhancement made by information input through the multi-touch screen;
A semi-projection mirror for overlapping the appearance of the exhibition space and the output image of the image output device;
A beam projector for outputting a content progress screen to the multi-touch screen;
An infrared LED array bar outputting infrared light under the multi-touch screen to form an interaction film;
An infrared camera configured to input an infrared light emitted from the interaction layer when the hand touches the multi-touch screen; And
Receives an image from the camera and calculates the position information of the subject in real time, receives an image from the infrared camera, calculates a user's touch point, and displays the information input through the multi-touch screen as the movement of the subject. A control PC for outputting through the image output apparatus or executing contents of a corresponding touch point through the image output apparatus so as to move to the image output apparatus;
Space augmented reality-based interactive display system comprising a.
The system of claim 2, wherein the interactive display system comprises:
A total reflection mirror (206) for reflecting the image output from the beam projector to the multi-touch screen;
A total reflection mirror (208) for reflecting the subject image to the camera;
White LED lighting for adjusting the brightness of the exhibition space to be the same or similar to the brightness of the image output from the image output device; And
Vertical shifting device for adjusting the depth of the image felt by the user by adjusting the height of the image output device;
Spatial augmented reality-based interactive display system further comprises.
The method of claim 2,
The multi-touch screen is composed of an acrylic plate and a rear screen,
The infrared LED array bar is a space augmented reality based interactive display system, characterized in that a plurality of infrared LEDs are installed at regular intervals, and both sides of the infrared LEDs are blocked by a wall to improve the straightness of light.
The method of claim 1 or 2, wherein the location information of the subject is:
The x and y coordinates of the subject are calculated from an image corresponding to the front side of the subject among the input images of the camera, and the z coordinates of the subject are calculated from an image corresponding to the top side of the subject among the input images. Augmented reality based interactive display system.
The method of claim 1, wherein the guide information is:
A spatial augmented reality-based interactive display system comprising any one of an image, a voice, a text, a graphic image, and a user input value according to the exhibition guide.
In the spatial augmented reality based interactive display method,
(a) real-time photographing of the subject of the exhibition space with a camera;
(b) image processing the image signal of the camera to calculate position information of the subject and outputting guide information to be displayed on the subject to an image output apparatus; And
(c) superimposing the appearance of the exhibition space and the guide information output from the image output apparatus through a semi-projection mirror;
Spatial augmented reality-based interactive display method comprising a.
In the spatial augmented reality based interactive display method,
(a) real-time photographing of the subject of the exhibition space with a camera;
(b) receiving a signal for content progress from a user through a multi-touch screen;
(c) image inputting infrared light emitted from an interaction film formed in the infrared under the multi-touch screen when the hand touches the multi-touch screen with an infrared camera;
(d) image processing the image signal of the camera to calculate position information of the subject, receiving an image from the infrared camera, calculating a user's touch point, and moving information of the subject through the multi-touch screen; Outputting the image output apparatus to move in a shape such as or outputting a content of a corresponding touch point through the image output apparatus; And
(e) superimposing the appearance of the exhibition space and the image information output from the image output apparatus through a semi-projection mirror;
Spatial augmented reality-based interactive display method comprising a.
The method of claim 7 or 8, wherein the interactive display method is:
Spatial augmentation characterized by calculating the position coordinates of a subject through a background modeling tracking algorithm for one subject, and calculating the position coordinates of a subject through a CamShift tracking algorithm for a plurality of subjects Reality based interactive display method.
The method of claim 7 or 8, wherein the interactive display method is:
After calculating the degree of distortion by the tilt of the translucent mirror in advance and using a reflection transformation matrix to reversely correct the image to be deformed when outputting the image, characterized in that for correcting according to the tilt of the semi-projection mirror Space augmented reality based interactive display method.

Images