KR20090040839A - Augmented reality computer device - Google Patents

Abstract

An augmented reality computer apparatus for outputting right and left image of the respective 3D graphic game image on a screen for glasses and implementing the three-dimensional image is provided to output the image synthesized image by using the technique of augmented reality. A minicomputer comprises a mark. A camera unit takes a picture of label. An image processor computes the relative position between the camera and the mark by processing the photographed image. The image composition part synthesizes the image taken a picture in the camera part and the image outputted in minicomputer into enlargement by using companion positional information between the camera calculated in the image processor and mark. A glasses-type display unit outputs the composite video synthesized from the image composition part.

Description

Augmented Reality Computer Device {AUGMENTED REALITY COMPUTER DEVICE}

The present invention relates to an augmented reality computer device by displaying the output image of a small computer, such as a mobile phone, navigation, portable game machine umpc (ultra mobile PC) by using augmented reality technology on the display screen in the form of a large computer like a large computer screen Users can work or play on the wide screen of a large computer instead of the small screen of a small computer. Augmented reality technology is a technology that synthesizes and outputs a graphic image generated by a computer on a live image taken with a camera. In this case, the two graphic images are not simply synthesized at a fixed fixed position on the real image but are characterized by synthesizing the graphic image as if it were a real object. For example, if a person is standing in front of a camera wearing a hat with a specific mark attached to it, detecting a mark attached to the hat from an image captured by the camera and synthesizing a monster face at the mark position. When a person shakes his head, the synthesized monster's face is also shaken, giving a feeling that a real monster is alive. Marks are usually in the form of black squares with white margins. The rectangle of the mark is detected from the image taken by the camera, and the three-dimensional relative position between the camera and the rectangle is calculated from the size and the distortion of the rectangle. And the 3D graphic image of the monster's face is synthesized on the human face by parallel and rotational movement to the mark position. Displaying such a 3D graphic image on a screen is a known technique used in a 3D game. As the performance of mobile phones gradually increases, desktop computer programs and games are also used in mobile phones. In addition, the technology to connect to the Internet on the go (WiBro) has been developed to be able to work, games or web surfing by connecting to the Internet in the future, regardless of place with a mobile phone. In other words, mobile phones are expected to be able to do all the work that was done on the existing desktop. However, the screen of a small computer, such as a mobile phone is very small compared to the monitor of a desktop computer has the disadvantage that the output image of the program that was working on the desktop can not be output as it is on the mobile phone screen.

An object of the present invention is to solve the problem that it is difficult to recognize the contents of a small computer because the screen of the small computer is small when a complex and detailed screen such as the output image of the desktop computer is output on the small screen of the conventional small computer.

In order to achieve the above object, the present invention augments the screen of a small computer on an image photographing a direction in which a person's eyes are directed by a camera attached to the head mount display (near eye display) to photograph a wearer's gaze direction. By synthesizing using a real technology and output to the display in the form of glasses provides a device that can give an effect as if the screen of a small computer is larger.

Using the augmented reality computer device according to the present invention can use the large screen of the desktop computer while maintaining the advantages of light, small space and easy to carry a small computer, such as a mobile phone, anytime, anywhere to run a complex screen program Or enjoy the game. In addition, the screen is not displayed on the small computer screen, but on the spectacles-type display, so that other people can not see the screen to maintain the security of the work. When playing a 3D game, the 3D image can be enjoyed by outputting the left and right images of the 3D graphic game image on the screens in front of the left and right eyes of the glasses display. In addition, by using augmented reality technology, images from external and small computer screens are synthesized and output to the glasses-type display, so that emergency situations occurring in the external environment can be detected immediately, so that an emergency response can be made without taking off the glasses-type display in an emergency. have.

Example 1

An augmented reality computer device according to the present invention includes a camera unit for photographing a small computer including a mark, an image processing unit for calculating a three-dimensional relative position between the mark and the camera by image processing the image image photographed by the camera unit; It includes an image synthesizer for enlarging the small computer screen using the position information of the mark calculated by the image processing unit to the image taken by the camera unit to synthesize the mark position, and the glasses display unit for outputting the synthesized image synthesized by the image synthesis unit; It is characterized by. At this time, it is preferable that the camera unit for capturing the gaze direction of the user wearing the spectacle display for outputting the composite image is integrally attached to the spectacle display. The mark is preferably composed of the feature points of the small computer. For example, it may be a small square screen of a small computer, four vertices of a square body, or a square mark output on a small screen of a small computer.

Figure 1 shows the configuration of the augmented reality computer device according to the present invention. In FIG. 1, a small screen (mo) of a mobile phone, which is a small computer, is photographed by two cameras (ca1, ca2) on the left and right sides of the eyeglass display wearer's visual direction on the outer surface of the spectacle display (ds). Transfers the captured image (cd). (One camera may be used.) The image processing program of the main body of the main body of the small computer or the square (mo) that forms the monitor of the small computer from the captured image (cd) We measure the three-dimensional relative position between the camera and the rectangle by detecting the size and distortion of the rectangle. (You can simply detect the two-dimensional position, not the three-dimensional relative position. For example, you can attach a special pattern mark to a small computer and calculate its position by image recognition.)

The mathematical formula for measuring the relative position between the camera and the rectangle is a known technique known as the perspective n point problem. This method is a problem of obtaining a relative position between a camera and a point from n points photographed by photographing n points of known relative positions. Description of this

 http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/MARBLE/high/pia/solvi ng . htm

You can find it at What characterizes the vertices of the rectangle is the perspective 4-point problem, which is a special case of the perspective n-point problem above.

Alternatively, a square (or a small computer's position) may be detected through stereo image analysis by comparing left and right camera images. In addition, by attaching an infrared LED to a small computer and detecting the LED in the captured image, the LED may be used as a feature point. That is, there is no limitation on the rectangle detection method. The main body con receives the screen information md of a program running on the small computer from the small computer. (The computer screen is transmitted as if the computer screen is transmitted to the beam projector.) Even when the main body con and the small computer are integrated. It's okay. Then, the microcontroller, computer, or digital signal processor (DSP) included in the main body displays the image to be output on the screen of the small computer on the image taken by the camera, performs parallel movement, affine conversion, etc. To pass). In this case, when combining two images, when the user wearing the spectacle display sees the synthesized image, it is preferable to synthesize the synthesized so that the screen of the synthesized small computer looks as if it is enlarged at the position where the actual small computer screen is located. That is, when a user wearing a spectacle-type display is holding a small computer and shaking it, the enlarged synthesized display image (amo) attached to the monitor (mo) of the small computer also shakes the image as if it is a physical large display attached to the small computer. Synthesize In Fig. 1, an amo of a synthesized small computer is shown as being enlarged and attached to an actual small computer screen mo. In other words, the user sees the enlarged amo on the spectacle display ds instead of the small computer screen mo. In the event of a fire or accident, a small computer can be pulled down to lower the large amo from the field of vision so you can see the surroundings and evacuate immediately. (In contrast, when the computer screen is output to the conventional eyeglass display, the computer screen is output to the whole field of view, so it is inconvenient to remove the eyeglass display to see the surroundings.) For this purpose, an appropriate mark indicating a characteristic point may be output to the monitor mo of the small computer, or a mark such as an infrared led may be attached to the small computer. You can also synthesize images from keyboards, mice, and stylus pens used in desktop computers. In this case, when the synthesized keyboard image is pressed with a finger, the image processor detects the movement of the hand in a stereo image of the left and right cameras to be a key input, and when the synthesized mouse is moved by hand, the mouse pointer is preferably moved. In addition, when writing or drawing by moving the stylus pen of the synthesized image, it is desirable to display the letter or figure in the synthesized image.

Example 2

2 is a view showing two square marks (mk1, mk2, mk3) centered on the screen (mo) of the small computer so as to easily recognize the screen of the small computer in the image processing unit. The image processing unit recognizes these marks mk1, mk2, and mk3 in the captured image and calculates a three-dimensional relative position between the small computer and the camera. The mark is divided into three parts. The first mark mk1 is a donut-shaped black rectangle with a square hole in the center, and the second mark mk2 is a small black rectangle with the center coinciding with the first mark. The third mark mk3 is located below the other two marks mk1 and mk2 to indicate the downward direction of the entire mark. The image processor detects an area having a small rectangle with a center coinciding with the rectangle in the photographed image, and then checks the position and direction of the entire mark by examining whether there is a third mark around the rectangle. In addition, after the boundary line is detected in the mark area, four sides of the first mark may be obtained by obtaining a straight line by Hough transform. Then, four vertices of the first mark can be obtained from the intersections of the four sides. In this case, since the size of the first mark (the length of the four sides) is determined, a three-dimensional relative position between the camera and the mark may be calculated using a formula of a perspective n point problem. The marks mk1, mk2, and mk3 described herein are merely examples, and any shape may be used as a mark as long as the marks are easily distinguishable from the surrounding environment. For example, a comb pattern with vertical lines or a chessboard pattern with alternating black and white squares may be arranged throughout the screen.

Example 3

3 is an example in which the rectangular mark of the second embodiment is modified.

In FIG. 3, the shape mk4 of one side of the donut-shaped rectangle mk1 is formed differently to indicate the reference point of the rectangle mark mk1. That is, in this embodiment, mk4 plays the role of mk3 of the second embodiment.

Example 4

You can also use the rectangular border of a small computer's display as a mark. In this case, the image processor may recognize the small computer from the captured image by using the 3d model information of the small computer, or may recognize the small computer by analyzing two stereo cameras ca1 and ca2.

1 is a block diagram of the present invention

2 is a mark output on a small computer screen

3 is a view that the deformed mark is output on the small computer screen

<Description of Major Symbols in Drawing>

amo: display image of small computer

ca1, ca2: Camera DS: Glasses Display

con: body mo: small computer display

md: small computer display image information

cd: Video information taken with the camera

ar: synthesized image information

mk1, mk2, mk3, mk4: Marks displayed on the screen

Claims (6)

In a computer device A small computer including a mark; A camera unit photographing the mark; An image processing unit which processes a video image photographed by the camera unit to calculate a relative position between a mark and a camera; An image synthesizer configured to enlarge and synthesize an image to be output to a small computer by using relative position information between the mark calculated by the image processor and the camera to the image photographed by the camera; An augmented reality computer device comprising a spectacle-type display unit for outputting a composite image synthesized by the image synthesis unit. The method of claim 1, An augmented reality computer device, characterized in that the glasses attached to the display unit for outputting the composite image to capture the direction of the user's eye wearing. The method according to any one of claims 1 to 2, The mark comprises a rectangle and The image processor detects the quadrangle and analyzes the size and shape of the quadrangle to calculate the relative position between the small computer and the camera. The image synthesis unit enlarges the image to be output to the small computer and augmented reality computer device, characterized in that to synthesize the image in the same form as attached to the mark position of the small computer. The method of claim 3, The rectangle mark includes a rectangle that is output to the display portion of the small computer. Augmented reality computer device, characterized in that. The method of claim 4 wherein the square is A large first rectangle shaped like a donut with a center; A small second rectangle located in the center of the opening of the first rectangle; A feature point representing the reference point of the large first rectangle Augmented reality computer device comprising a. The method of claim 3, The rectangle mark contains the outline rectangle of the display of the small computer Augmented reality computer device, characterized in that.

Images