KR20200055422A - Apparatus and Method for providing augmented reality contents by KALMAN filter - Google Patents

Abstract

Disclosed in the present invention are a device for providing augmented reality contents to a user and a method thereof, and more specifically, to a device for providing augmented reality contents. To this end, the device for providing augmented reality contents using a Kalman filter according to an embodiment of the present invention comprises: a position extracting unit which extracts position information of a user to calculate a position information measurement value; a correcting unit which corrects the position information measurement value of the user calculated in the position extracting unit to a position information correction value by using the Kalman filter; a content database unit in which contents to which the position information is allotted are stored; a reality image extracting unit which extracts a reality image; a movement information extracting unit which extracts the movement of the user; an augmented reality processing unit which calls the contents to which the position information corresponding to the position information correction value is allotted, and adjusts the size and the position of the contents to match the position information correction value, while allowing the same to be overlapped with the extracted reality image and correcting the position of the contents in accordance with the movement information; and a display unit which outputs the reality image and the contents. Therefore, precision in extracting the position information of the user can be increased.

Description

Apparatus and Method for providing augmented reality contents by KALMAN filter}

The disclosed content relates to an apparatus and method for providing augmented reality content to a user, and provides augmented reality information required for a user at a specific location.

Unless otherwise indicated herein, the content described in this section is not prior art to the claims of this application and is not admitted to be prior art by inclusion in this section.

Recently, with the spread of mobile devices capable of executing and providing various media such as smartphones, methods of providing content to users have been diversified. As these contents lead to an increase in technology demand for interaction between humans and computers from the viewpoint of efficiency and convenience, augmented reality, a field of Human Computer Interaction (HCI), has attracted great attention, especially mobile augmented reality ( Interest in Mobile Augmented Reality is growing.

Augmented reality (Augmented Reality), unlike Virtual Reality (Virtual Reality), refers to an environment that is provided to users by fusing images from the real world and images from the virtual world. In general, the image of the virtual world appearing in augmented reality uses computer graphics, and serves to provide additional information to the image of the real world.

Content that is provided to the user using augmented reality in the past, and extracts the user's location information using GPS, GNSS, etc. from the real image projected on the camera of the mobile device, and provides advertisement and navigation information from the extracted location information. have.

According to Korean Patent Publication No. 10-1282292, an advertisement service system using augmented reality and a method thereof are published. More specifically, the advertisement form and detailed information transmitted from the advertiser terminal and the object selection information of each advertisement In response to receiving the user ID and terminal information from the service management means and the user terminal that registers the database, the user information is searched and provided in the database, and the advertisement corresponding to the location information of the user terminal is searched in the database. Distribute a preset object as a medium, and upon receiving a collection signal from the user terminal, determine success / failure based on the collection success rate of the object, and transmit a success message or a failure message through a message transmission means. When a success message is transmitted to a user terminal, an advertisement service system and method using augmented reality including a service processing means for storing an object collected in the user's storage box in the database are posted.

The invention disclosed in the aforementioned Korean Patent Publication No. 10-1282292 extracts the location information of the user terminal using GPS, but it is difficult to grasp the location information indoors using only GPS and GNSS to extract the location information. There is an error even when simply extracting the location information, which means that a problem may occur in providing accurate content. Accordingly, although a location is extracted using a beacon or the like capable of providing separate location information, there is also a problem that an error occurs, and there is a need to improve it.

1. Korean Patent Publication No. 10-1282292

It is intended to provide an apparatus and method for providing augmented reality content to a user at a specific location, improving the accuracy of the user's location information extraction, and controlling the movement of the augmented reality content in response to the user's movement.

Disclosed is an apparatus for providing augmented reality content, a location extracting unit for extracting location information of a user to calculate a location information measurement value, and a user's location information measurement value calculated by the location extraction unit is positioned through a Kalman filter. Correction unit for correcting with the information correction value, content database unit storing the content to which the location information is assigned, reality image extraction unit for extracting the real image, motion information extraction unit for extracting the user's motion, and corresponding to the location information correction value Calls the content to which the location information is assigned, adjusts the size and position of the content to match the location information correction value, overlaps the extracted reality image, and corrects the location of the content according to the motion information An apparatus for providing augmented reality content using a Kalman filter including an augmented reality processing unit and a display unit for outputting the real image and content is presented as an embodiment.

In addition, the disclosed contents are a first step of calculating a location information measurement value by extracting the user's location information, a second step of correcting the calculated location information measurement value of the user to a location information correction value through a Kalman filter, the user A third step of calling the content to which the location information corresponding to the location information correction value of the is assigned, a fourth step of extracting the real image through the camera, a fifth step of synthesizing and displaying the called content and the real image According to an embodiment of the present invention, a method for providing augmented reality content using a Kalman filter including a sixth step of adjusting the position of the content is presented according to an embodiment of the user.

According to the disclosed embodiment, augmented reality content is provided to a user at a specific location, but the user's location information can be accurately extracted to accurately provide the content to which the location information is allocated, and augmented reality in response to a user's movement Content movement can also be precisely controlled.

The effects of the present embodiments are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those skilled in the art from the claims.

1 is a block diagram of an apparatus for providing augmented reality content using a Kalman filter according to an embodiment of the disclosed content.
2 is a schematic diagram for extracting location information of a user according to an embodiment of the disclosed content.
3 is a schematic diagram of extracting a user's location with three beacons according to an embodiment of the disclosed subject matter.
Figure 4 is a graph between the understanding of the fleece formula and the strength and distance of the transmitted and received signals.
5 (a) is a flowchart of a method for providing augmented reality content using a Kalman filter according to an embodiment of the disclosed content.
Figure 5 (b) is a flow diagram of a Kalman filter according to an embodiment of the disclosed content.

Advantages and features of the present invention, and methods for achieving them will be clarified with reference to embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the embodiments allow the disclosure of the present invention to be complete, and common knowledge in the technical field to which the present invention pertains. It is provided to fully inform the holder of the scope of the invention, and the invention is only defined by the scope of the claims. The same reference numerals refer to the same components throughout the specification.

In describing embodiments of the present invention, when it is determined that a detailed description of known functions or configurations may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted. In addition, terms defined below are terms defined in consideration of functions in an embodiment of the present invention, which may vary according to a user's or operator's intention or practice. Therefore, the definition should be made based on the contents throughout this specification.

Hereinafter, a preferred embodiment of an apparatus for providing augmented reality content using an improved Kalman filter will be described with reference to the accompanying FIG. 1.

The apparatus for providing augmented reality content using a Kalman filter extracts location information of a user to calculate a location information measurement value, and a location extraction unit (10) that calculates a location information measurement value, and a user's location information measurement value calculated by the location extraction unit through a Kalman filter Correction unit (20) for correcting the position information correction value, content database unit (30) in which the content to which the location information is assigned is stored, reality image extraction unit (40) for extracting a real image, and motion information for extracting the user's movement The extracting unit 50 calls the content to which the location information corresponding to the location information correction value is assigned, adjusts the size and location of the content to match the location information correction value, and overlaps the extracted reality image. , An augmented reality processing unit 60 for correcting the location of the content according to the motion information, and a display unit 70 for outputting the real image and content.

Referring to FIG. 2 in more detail, the location extracting unit 10 may measure a coordinate of a user using a Global Positioning System (GPS), Global Navigation Satellite System (GNSS), and the like, and a short-range wireless communication device. Can be used to measure the user's coordinates. In addition, the user's location information may be primarily extracted through the GPS and GNSS, and the user's location information may be extracted and corrected through the short-range wireless communication device. The short-range wireless communication device may use a device using NFC, and may use a Bluetooth Low Energy (BLE) beacon. BLE Beacon is a device that is low power and has a wider coverage than NFC, so it is a desirable device to extract user's location information.

The user's location information is extracted through the GPS, GNSS, etc. This is to extract the user's location information through communication with the satellite, so the time error of the satellite, the location error of the satellite, the refraction of the ionosphere and the convection layer, noise ) And multipath cause an error of about 10m. In order to overcome this, GPS and GNSS are used together to improve the precision, but this is also only for outdoor measurements, and when entering the room, the communication signal is weakened, causing more errors, or sometimes it is difficult to receive location information. Therefore, it is desirable to extract the user's information primarily through GPS, GNSS, etc., and secondarily extract the user's location information indoors through a short-range wireless communication device.

When the user's location information is measured using the BLE beacon, three or more BLE beacons may be provided to calculate the user's location information measurement value through triangulation. As shown in FIG. 3, three or more beacons to which specific location coordinates are assigned are installed in a place to provide the content. Each beacon allocates the coordinates of (X1, Y1), (X2, Y2), (X3, Y3). In addition, when the user's location is set to arbitrary coordinates (X, Y), if the distances D1, D2, and D3 of the beacons are measured, the user's location coordinate can be obtained through Pythagorean theorem.

Summarizing the above equations as X and Y to solve the system of equations, the user's coordinates (X, Y) can be obtained.

As a method of obtaining the distance between the beacon and the user, D1, D2, and D3, there is a Fris Formula. The fleece formula is a formula that expresses the power relation between transmission and reception of wireless communication through free space. Accordingly, it means that the distance of the transmitting and receiving device can be measured according to the strength of the received signal. As shown in (a) of FIG. 4, the signal radiating from the transmitting antenna has a spherical shape, and the area of the signal received by the receiving antenna among the radiating signal areas is called an effective opening area, and the effective opening area is Measure the incoming signal strength. If the power of the received signal is expressed by the expression using RSSI (Received Signal Strength Indication), which is a value based on the strength of the received signal, it is as follows.

In summary, the received power is proportional to the gain of two antennas (transmission and reception), proportional to the square of the wavelength, and inversely proportional to the square of the distance. This is called fleece definition,

는 파장을, d는 송수신 간 거리를,

는 송수신 안테나의 실효개구면적을 의미한다. 상기 식에서 Is defined as The P means transmit / receive radiation power, the G means transmit / receive antenna gain,

Is the wavelength, d is the distance between sending and receiving,

Is the effective opening area of the transmitting and receiving antenna. In the above formula

Is also called 'Free Space Path Loss (FSPL)'.

Accordingly, by measuring the strength of the signal received by the user's terminal, that is, power, according to the fleece definition equation, the distance between the beacon and the user's terminal can be measured to calculate the location information measurement value of the user. If the distance according to the strength of the received signal is expressed in a graph, it may be expressed as a graph of FIG. 4 (B).

The correction unit 20 may correct a user's location information measurement value calculated by the location extraction unit 10 to a location information correction value through a Kalman filter. The Kalman filter has an algorithm that corrects the measured value by removing noise included in the measured value using the old information and the new measured value. The Kalman filter is generally used for predicting a motion state with linearity, and by extending the linearity, an extended KALMAN filter can be designed to be used for predicting a motion state with nonlinearity. have. As shown in FIG. 5 (b), the initial value for the user's location information is selected (S201), the user's location information correction value and the error covariance are predicted (S202), and Kalman's gain is calculated through this (S203). )do. The calculated Kalman gain and the location information measurement value are calculated to calculate the location information correction value (S204), and the error covariance is calculated again through the calculated location information correction value (S205).

Through this, noise can be removed and the next position of the user can be predicted, so that accurate position measurement is possible.

The content database unit 30 stores the content to which the location information is assigned. Each content has location information, and when the user's location information correction value corresponds to the location information allocated to the content, it serves to provide the content to the user.

The reality image extracting unit 40 extracts a reality image viewed by a user through the device, which is an embodiment described in this specification. Generally, a real image is extracted through the camera included in the device, but any configuration capable of extracting a background image is not limited thereto. Also, the image here does not mean a simple picture, but includes a dynamic image, that is, an image.

The motion information extraction unit 50 extracts a value that changes according to the user's motion, that is, motion information. As an embodiment for extracting the user's motion information, the user's motion information may be extracted by at least one of a gyro sensor and an acceleration sensor composed of three or more axes. The gyro sensor is a sensor that uses the value of the angular velocity, which is the rotational speed of an object, and is also called an 'angular velocity sensor'. It calculates the Coriolis Force generated when an object rotates and converts it into an electrical signal. Accordingly, it is possible to identify a motion of the user rotating about any one axis. The acceleration sensor is a sensor that measures how much force is applied in one direction, based on the gravitational acceleration, and is a one-, two-, and three-axis acceleration sensor according to various directions. It is classified and can be calculated by electrically converting the force of linear motion in each direction. In order to measure all motion information such as up, down, left, and right of the user, it is preferable to have an acceleration sensor with 3 or more axes so that 3D motion information can be extracted. To extract more accurate user motion information, the gyro sensor and It is desirable to use 3 or more acceleration sensors together.

The augmented reality processing unit 60 calls the content to which the location information corresponding to the location information correction value is allocated, adjusts the size and location of the content to match the location information correction value, and the extracted reality image While overlapping, the position of the content may be corrected according to the motion information. In this case, the location, size, and angle of the provided content may be adjusted and synthesized according to the user's location information correction value, and the location of the content may be adjusted by extracting a feature from the user's real image.

For example, as illustrated in FIG. 6, a contour of components included in the extracted reality image may be extracted, and a location and size of content to be provided may be previously allocated according to the shape of the contour. As such, if a database is built by extracting the features of the real image, it will be possible to provide more accurate location and size content. In addition, a marker may be included in the extracted reality image to adjust the location and size of the content.

The display unit 70 may output the real image and content. It includes all display devices capable of outputting augmented reality content. For example, there are LCD, LED, OLED, AMOLED, and the like, but not limited to this example, will be apparent from the standpoint of a person skilled in the art.

Hereinafter, a method of providing augmented reality content using a Kalman filter will be described with reference to FIGS. 5 (a) and 5 (b).

The method of providing augmented reality content using Kalman Filter,

A first step of calculating the location information measurement value by extracting the user's location information (S100), a second step of correcting the calculated location information measurement value of the user to a location information correction value through a Kalman filter (S200), the A third step (S300) of calling the content to which the location information corresponding to the user's location information correction value is allocated, a fourth step of extracting the real image through the camera (S400), and synthesizing the called content and the real image And a fifth step of displaying (S500) and a sixth step of adjusting the location of the content according to the user's motion information (S600).

If you look specifically,

In the first step S100, the location information measurement value can be calculated by extracting the location information of the user. To calculate the user's location information measurement value, a user's coordinates may be measured using a Global Positioning System (GPS), a Global Navigation Satellite System (GNSS), or the like, and the user's coordinates using a short-range wireless communication device. Can be measured. In addition, the user's location information may be primarily extracted through the GPS and GNSS, and the user's location information may be extracted and corrected through the short-range wireless communication device. The short-range wireless communication device may use a device using NFC, and may use a Bluetooth Low Energy (BLE) beacon. BLE Beacon is a device that is low power and has a wider coverage than NFC, so it is a desirable device to extract user's location information.

The user's location information is extracted through the GPS, GNSS, etc. This is to extract the user's location information through communication with the satellite, so the time error of the satellite, the location error of the satellite, the refraction of the ionosphere and the convection layer, noise ) And multipath cause an error of about 10m. In order to overcome this, GPS and GNSS are used together to improve the precision, but this is also only for outdoor measurements, and when entering the room, the communication signal is weakened, causing more errors, or sometimes it is difficult to receive location information. Therefore, it is desirable to extract the user's information primarily through GPS, GNSS, etc., and secondarily extract the user's location information indoors through a short-range wireless communication device.

When the user's location information is measured using the BLE beacon, three or more BLE beacons may be provided to calculate the user's location information measurement value through triangulation. As shown in FIG. 3, three or more beacons to which specific location coordinates are assigned are installed in a place to provide the content. Each beacon allocates the coordinates of (X1, Y1), (X2, Y2), (X3, Y3). In addition, when the user's location is set to arbitrary coordinates (X, Y), if the distances D1, D2, and D3 of the beacons are measured, the user's location coordinate can be obtained through Pythagorean theorem.

Summarizing the above equations as X and Y to solve the system of equations, the user's coordinates (X, Y) can be obtained.

As a method of obtaining the distance between the beacon and the user, D1, D2, and D3, there is a Fris Formula. The fleece formula is a formula that expresses the power relation between transmission and reception of wireless communication through free space. Accordingly, it means that the distance of the transmitting and receiving device can be measured according to the strength of the received signal. As shown in (a) of FIG. 4, the signal radiating from the transmitting antenna has a spherical shape, and the area of the signal received by the receiving antenna among the radiating signal area is called an effective opening area, and the effective opening area is Measure the incoming signal strength. If the power of the received signal is expressed by the expression using RSSI (Received Signal Strength Indication), which is a value based on the strength of the received signal, it is as follows.

In summary, the received power is proportional to the gain of two antennas (transmission and reception), proportional to the square of the wavelength, and inversely proportional to the square of the distance. This is called fleece definition,

는 파장을, d는 송수신 간 거리를,

는 송수신 안테나의 실효개구면적을 의미한다. 상기 식에서 Is defined as The P means transmit / receive radiation power, the G means transmit / receive antenna gain,

Is the wavelength, d is the distance between sending and receiving,

Means the effective opening area of the transmitting and receiving antenna. In the above formula

Is also called 'Free Space Path Loss (FSPL)'.

Accordingly, by measuring the strength of the signal received by the user's terminal, that is, power, according to the fleece definition equation, the distance between the beacon and the user's terminal can be measured to calculate the location information measurement value of the user. If the distance according to the strength of the received signal is expressed in a graph, it may be expressed as a graph of FIG. 4 (B).

In the second step S200, the calculated location information measurement value of the user may be corrected to a location information correction value through a Kalman filter. The Kalman filter refers to an algorithm that corrects a measurement value by removing noise included in the measurement value using past information and a new measurement value. The Kalman filter is generally used for predicting a motion state with linearity, and by extending the linearity, an extended KALMAN filter can be designed to be used for predicting a motion state with nonlinearity. have. As shown in FIG. 5 (b), the initial value for the user's location information is selected (S201), the user's location information correction value and the error covariance are predicted (S202), and Kalman's gain is calculated through this (S203). )do. The calculated Kalman gain and the location information measurement value are calculated to calculate the location information correction value (S204), and the error covariance (S205) is calculated again through the calculated location information correction value.

Through this, noise can be removed and the next position of the user can be predicted, so that accurate position measurement is possible.

In the third step (S300), the content to which location information corresponding to the user's location information correction value is allocated can be called. Content stored for providing has respective location information, and when a user's location information correction value corresponds to location information allocated to the content, it can be called to provide the corresponding content to the user.

In the fourth step (S400), a real image can be extracted through a camera. An image viewed by the user through the camera may be extracted, but is not limited thereto, and the extracted reality image may be processed and provided. The image here does not mean a simple picture, but includes a dynamic image, that is, an image.

In the fifth step (S500), the called content and the real image may be synthesized and displayed. The content having location information corresponding to the user's location information correction value and the extracted reality image may be synthesized and displayed. In this case, the location, size, and angle of the provided content may be adjusted and synthesized according to the user's location information correction value, and the location of the content may be adjusted by extracting a feature from the user's real image.

For example, as illustrated in FIG. 6, a contour of components included in the extracted reality image may be extracted, and a location and size of content to be provided may be previously allocated according to the shape of the contour. As such, if a database is built by extracting the features of the real image, it will be possible to provide more accurate location and size content. In addition, a marker may be included in the extracted reality image to adjust the location and size of the content.

In the sixth step (S600), the location of the content may be adjusted according to the user's motion information. The location and size of the content should be corrected according to the user's movement. The user's motion information may be extracted by at least one of a gyro sensor and an acceleration sensor composed of three or more axes. The gyro sensor is a sensor that uses the value of the angular velocity, which is the rotational speed of an object. It is also called an angular velocity sensor. It calculates the Coriolis Force generated when an object rotates and converts it into an electrical signal. Accordingly, it is possible to identify a motion of the user rotating about any one axis. The acceleration sensor is a sensor that measures how much force is applied in one direction, based on the gravitational acceleration, and is a one-, two-, and three-axis acceleration sensor according to various directions. It is classified and can be calculated by electrically converting the force of linear motion in each direction. In order to measure all motion information such as up, down, left and right of the user, it is desirable to have an acceleration sensor of 3 or more axes so that 3D motion information can be extracted. To extract more precise user motion information, the gyro sensor and It is desirable to use 3 or more acceleration sensors together.

In addition, as an embodiment of the invention described herein, by using the above-described device and method, it is possible to provide augmented reality content according to the corresponding place in a tourist spot or a specific place. For example, GPS coordinate information of a place such as a historical event or a historic site may be previously allocated, and three or more beacons may be installed at the location. Each beacon has a unique ID value, and content to be provided to the ID value may be previously allocated.

And, when the user visits the device with the device described in this specification, the device can extract the user's location information through the GPS of the place and the beacon installed in the place, and the extracted location information Accordingly, pre-allocated content can be called to the device. At this time, a Kalman filter can be used for precise measurement of the user's location information.

When the user photographs the corresponding place through the camera of the device, the actual background through the camera and the content to which the corresponding location information is allocated may be superimposed on the display of the device and provided to the user. For example, when photographing a place where a historical event has occurred, the content reconstructed with the CG (Computer Graphic) can be reproduced by overlapping, and information about the event can be provided together as a text or a separate picture. In addition, as a separate content, when the user shoots with a self-camera at the location, if the real image and the face of the user are taken together, augmented reality is superimposed by superimposing the image on the historical event or the historical site on the captured image. It can also be provided.

At this time, since the location, size, and angle of the content must be changed according to the user's movement, the user's movement may be recognized through the gyro sensor or acceleration sensor included in the device, and the location of the content may be adjusted.

The disclosed content is only an example, and can be variously changed by a person having ordinary skill in the art without departing from the gist of the claims claimed in the claims, and thus the protection scope of the disclosed content is specified above. It is not limited to the embodiment.

10: location extraction unit
20: correction unit
30: content database unit
40: reality image extraction unit
50: motion information extraction unit
60: augmented reality processing unit
70: display unit

Claims (8)

In the device for providing augmented reality content,
A location extracting unit that extracts the user's location information and calculates the location information measurement value;
A correction unit that corrects a user's location information measurement value calculated by the location extraction unit to a location information correction value through a Kalman filter;
A content database unit storing the content to which the location information is allocated;
A reality image extraction unit for extracting a reality image;
A motion information extraction unit for extracting a user's motion;
Recall the content to which the location information corresponding to the location information correction value is assigned, adjust the size and position of the content to match the location information correction value, and overlap the extracted reality image, according to the motion information. An augmented reality processor that corrects the location of the content; And
A display unit for outputting the real image and content;
Device for providing augmented reality content using a Kalman filter comprising a.
The method according to claim 1,
The Kalman filter of the correction unit obtains Kalman's gain through the position information correction value and error covariance of the past time point, and obtains the positional information correction value through the positional information measurement value and the Kalman gain extracted from the position extraction unit, A device for providing augmented reality content using a Kalman filter, wherein the error covariance is corrected through the obtained position information correction value.
The method according to claim 2,
The location extracting unit primarily extracts the location information of the user through GPS, and secondly extracts the location information of the user through a beacon, wherein the beacon consists of three or more, and the RSSI value of the beacons A device for providing augmented reality content using a Kalman filter, which measures a distance according to and calculates a user's location information measurement value by a triangulation method.
The method according to any one of claims 1 to 3,
The motion information extracting unit extracts the motion information of the user by at least one of a gyro sensor and an acceleration sensor composed of three or more axes.
A first step of extracting a user's location information and calculating a location information measurement value;
A second step of correcting the calculated location information measurement value of the user to a location information correction value through a Kalman filter;
A third step of calling up content to which location information corresponding to the user's location information correction value is allocated;
A fourth step of extracting a real image through a camera;
A fifth step of synthesizing and displaying the called content and the real image; And
A sixth step of adjusting the location of the content according to the user's motion information;
Method of providing augmented reality content using a Kalman filter comprising a.
The method according to claim 5,
In the second step, the Kalman filter obtains Kalman's gain through the position information correction value and error covariance of the past time point, and obtains the positional information correction value through the position information measurement value and the Kalman gain extracted from the location extraction unit. , Compensating the error covariance through the obtained position information correction value The method of providing augmented reality content using a Kalman filter.
The method according to claim 6,
In the first step, the user's location information is first extracted through GPS, and secondly extracted through a beacon, but the beacon is composed of three or more, and the distance according to the RSSI value of the beacons is determined. A method of providing augmented reality content using a Kalman filter, which measures and calculates a user's location information measurement value by a triangulation method.
The method according to any one of claims 5 to 7,
The sixth step is a method for providing augmented reality content using a Kalman filter, characterized in that the user's motion information is extracted by at least one of a gyro sensor and an acceleration sensor composed of three or more axes.

Images