KR101288851B1 - Operating method of augmented reality system with smart-phone - Google Patents

Abstract

PURPOSE: A method of operation of a smart phone user customized augmented reality service system is provided to display a fused image without unfriendliness by analyzing season, weather, temperature and time of an image. CONSTITUTION: Analyzed contents are stored in an assigned region (S2300). Contents to be fused are uploaded (S2400). An accessed terminal unit selects the analyzed contents (S2500). The analyzed contents and the uploaded contents are fused. Unfriendliness minimization-processed fused contents are stored in an assigned region (S2600). A format of the fused contents is converted into an MPEG7 format (S2700). [Reference numerals] (AA) Start; (BB) Finish; (S2100) Collect the contents of a macro-environment; (S2200) Store the collected contents as original contents; (S2300) Extract spatiotemporal information and separate a background and objects from the original contents, and store the contents after structuralization; (S2400) Upload the contents from an accredited terminal; (S2500) Select the analyzed contents?; (S2600) Fuse the analyzed contents and the uploaded contents and minimize a difference between the same; (S2700) Converting a format of the fused contents into a Mpeg 7 format; (S2800) Does a terminal unit request the fused contents?; (S2900) Provide the fused content to the terminal unit

Description

Operating method of augmented reality service system customized for smart phone using digital map {Operating method of augmented reality system with smart-phone}

The present invention relates to a method of operating a user-customized augmented reality service system using a smart phone, more specifically, to analyze the spatio-temporal information of the content collected on the basis of location and upload the content to be fused to minimize the heterogeneity with the content to be fused The present invention relates to a method for operating a user-customized augmented reality service system using a smart phone to provide augmented reality content that is played as a video on a smartphone.

Advances in computer and information and communication technologies enable the analysis and processing of large amounts of data that have been difficult to attempt in the past, and have had a significant impact on video image data processing.

Augmented Reality (AR) is a field of Virtual Reality (VR) and is one of the fields of computer graphics that synthesizes virtual objects into images of the real environment and shows them as if they exist in the real environment.

Virtual reality (VR) is a computer graphic that excludes interaction with the real world and processes objects or objects to be exposed to interact with each other in a pre-built virtual space.

Augmented Reality (AR) exposes the real environment and virtual objects in a real-time composite state and complements real-world information (augmented information) with virtual world information. Is the environment of the real world.

In other words, augmented reality (AR) uses a special interface called computer graphics to represent the real world image or actual image that is actually observed, and registers and expresses a 3D virtual image so that the real world environment and the virtual environment of the virtual world are represented. As the distinction of, becomes obscure, the real environment is augmented so that the improved image can be seen, and it can be applied to observer quickly, quickly and conveniently for the situation such as information on a specific region or actual product.

As a prior art using augmented reality is a patent registration No. 10-0980202 (registered on August 30, 2010), there is a "mobile augmented reality system and method that can interact with three-dimensional virtual objects".

1 is a functional configuration diagram of a system using increase and decrease reality according to an embodiment of the prior art.

Hereinafter, the prior art will be described in detail with reference to the accompanying drawings, which includes a camera, an image processing unit, a display unit, and an interaction unit.

The camera is attached to the mobile communication terminal, and the image processing unit generates a three-dimensional virtual object on the hand using a camera. The display unit outputs a three-dimensional virtual object and a hand image. Dimensional virtual object.

The conventional art has the advantage of increasing the sense of reality by the augmented reality because the user can access the 3D virtual contents anytime and anywhere using the mobile terminal and feel the weight and feel by using the vibration sensor.

However, the prior art has a problem that the location-based area information can not be provided as a realistic content by augmented reality.
Location-based is a location based service (LBS) using a digital map, and since it is generally known, a detailed description thereof will be omitted.

Therefore, it is necessary to develop a technique to provide location-based local information as a realistic image without any difference by augmented reality.

Republic of Korea Patent Registration No. 10-0980202 (2010.08.30.) "Mobile augmented reality system and method that can interact with 3D virtual objects"

In order to solve the problems and necessity of the prior art as described above, the present invention extracts and stores spatiotemporal information from multimedia contents and analyzes the season, diary, temperature, and time of the selected image so that the fused image can be fused without heterogeneity. Provides an operation method of a user-customized augmented reality service system using a smartphone that can be played as a video on the terminal.

In order to achieve the object of the present invention, a method for operating a user-customized augmented reality service system using a smart phone includes a GPS information receiving device and collects contents of the collected surrounding environment in a region allocated together with location information by GPS. Photographing device to save; A location-based content server that receives the collected content from the photographing apparatus and stores it in an allocated area and provides the original content, and receives the original content from the location-based content server and analyzes spatiotemporal information in an area allocated as analysis content; An analysis content providing server for storing, The content fusion server for receiving the analysis content from the analysis content providing server, uploaded from the terminal unit and fused with the input content, and storing the fused content in the allocated area as fused content, The content fusion server The server receives the converged content from the service server, converts it into a format that can be played by the terminal unit, and stores the data in the allocated area. The terminal unit, the terminal unit A method of operating a customized augmented reality service system including an authentication content server, an authentication server for authenticating access to a content fusion server and a service server, wherein when a shooting control signal is input by the photographing apparatus, the surrounding content is photographed and collected. A photographing step of receiving and analyzing a GPS signal at the collecting point and including the identified coordinate information in a metadata area of the collected content and storing the collected coordinate information as collected content; A collection step of receiving the collected content from the photographing apparatus by the location-based content server and storing the collected content as original content in an allocated memory area; The original content is received from the location-based content server by the analysis content providing server, and when the terminal accesses and provides the original content, the original content is provided and the spatiotemporal information is extracted by analyzing the original content. Analyzing the data stored in the memory area allocated as the analysis content in the separated structured state; Receiving the analysis content from the analysis content providing server by the content fusion server and uploading the content by accessing the terminal unit and selecting the analysis content, the analysis content and the uploaded content are fused and heterogeneous and minimized fusion content is processed. A fusion step of storing the data in an allocated memory area; And receiving the fusion content from the contents fusion server by the service server, converting it into an MPEG 7 format, and storing the fused content in an allocated area. When the terminal accesses the fusion content, the fused content is converted into the MPEG 7 format. A service step of providing convergence content; It includes, but the extraction of the spatiotemporal information is made by extracting the time series information including the diary, season, time and spatial information including latitude, longitude, altitude from the metadata of each unit frame image constituting the original content, The image analysis is performed by extracting similarities from each unit frame image constituting the original content and designating a frame image including the most extracted similarities as an index frame image, and minimizing the heterogeneity when the analysis content is selected. Analyze the time value of each included frame image, select and fuse the frame image with the smallest difference from the time value of the content to be uploaded and fused from the terminal unit, and remove the object selected as unnecessary from the frame image of the selected analysis content. The object selected as needed. The brightness and saturation of the selected content and the uploaded content are similarly converted, the position of the object is adjusted, the shadow is removed, and the fused boundary is adjusted to be similar to the surroundings. It is converted into 7 formats and plays fused content provided from the service server as a video, and is one or more than one among a smartphone and a computer, and the spatiotemporal information can be extracted by analyzing an image included in the original content.

The present invention has the advantage of extracting the spatiotemporal information from the multimedia content and analyzing the season, diary, temperature, and time of the image to display the image fused without heterogeneity.

In addition, there is an advantage to be played as a video on the terminal provided with the fused image.

1 is a functional configuration diagram of a system using the increase and decrease reality according to an embodiment of the prior art,
2 is a functional configuration diagram of a user-customized augmented reality service system using a smartphone according to an embodiment of the present invention,
3 is a flow chart illustrating a method of operating a customized augmented reality service system according to an embodiment of the present invention,
And
4 is a flowchart illustrating a method of operating a smartphone constituting a customized augmented reality service system according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The present invention is capable of various modifications and various embodiments, and specific embodiments are illustrated in the drawings and will be described in detail in the detailed description. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. In the following description of the present invention, if it is determined that the detailed description of the related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted.

2 is a functional configuration diagram of a user-customized augmented reality service system using a smart phone according to an embodiment of the present invention, Figure 3 illustrates a method of operating a personalized augmented reality service system according to an embodiment of the present invention. 4 is a flowchart illustrating a method of operating a smartphone constituting a customized augmented reality service system according to an embodiment of the present invention.

Hereinafter, the system configuration of the present invention will be described in detail with reference to the accompanying FIG. 2. The user-customized augmented reality service system 1000 using a smartphone includes a photographing apparatus 1100, a location-based content server 1200, and an analysis content providing server. 1300, the content fusion server 1400, the service server 1500, the terminal unit 1600, and the authentication server 1700.

The photographing apparatus 1100 includes a GPS receiver which receives GPS information from a GPS satellite and outputs coordinate information or position information analyzed by latitude, longitude, sea level, time, etc. Collects the content of the content and stores the numerical map that associates the collected content with location information as metadata in the allocated area. Since metadata is generally known, a detailed description thereof will be omitted.
In addition, the digital map is a digital map that stores the coordinate information or location information analyzed by latitude, longitude, elevation, time, etc. for each point of the map in the state of being associated with the captured image or the corresponding image, and outputs it by searching. Since it is generally known, a detailed description thereof will be omitted.

The photographing apparatus 1100 may include a dedicated camera including a GPS receiver, a smartphone having a camera function and a GPS reception function, and the GPS receiver will be described as receiving weather information together.

The location-based content server 1200 receives the collected content from the photographing apparatus 1100 and stores the collected content in the allocated area and provides the original content when the analysis content server 1300 requests it.

The analysis content providing server 1300 receives the original content from the location-based content server 1200 and stores it in an area allocated as the analysis content analyzed by image analysis and spatiotemporal information, and the content fusion server 1400 or the terminal unit 1600. Original content is provided upon request.

In image analysis, an object or a character may be separated from an image of a background by using image processing technology, and may be separately structured and stored separately.

In addition, the image analysis may extract the similarity from each unit frame image constituting the original content, and designate the frame image including the most extracted similarity as the index frame image.

Spatiotemporal information is divided into time series information and spatial information, and time series information is composed of information such as weather (weather), season, time including date, and spatial information is coordinate information or location information including latitude, longitude, altitude, etc. Is made of.

The spatiotemporal information is extracted from metadata of each unit frame image constituting the original content, time series information including diary, season, and time, and spatial information including latitude, longitude, and altitude.

On the other hand, although limited, spatiotemporal information may be extracted by analyzing an image of the content. In other words, it analyzes the weather and analyzes the weather such as rain and snow, analyzes the time of cognition in the morning, lunch, and evening, and analyzes spatial information such as beach, mountain, desert and plain cognition. Naturally, the spatio-temporal information analyzed together can be selectively used.

The content fusion server 1400 is provided with the analysis content from the analysis content providing server 1300, uploaded from the terminal unit 1600 and fused with the input content, and stores the fused content as the fusion content in the allocated area and the service server ( 1500 and at the request of the terminal unit 1600.

There should be no heterogeneity in order to fuse the content and the content. In particular, there should be no heterogeneity of the object when providing the sensitization reality service by fusing the object of the uploaded content with the analysis content for providing the sensitization reality service. That is, it is preferable to process so that it is impossible to confirm where the fused part is.

In order to eliminate the heterogeneity of the object in the sensitized reality service, the time value at which each fused content is collected is used. In other words, there is no heterogeneity as the time at which each content is collected is closer or adjacent or identical.

In addition, in order to eliminate the heterogeneity, an object determined to be unnecessary from an underlying content may be removed or an object determined to be necessary may be added.

In addition, it minimizes the difference in brightness, saturation, and size of each fused content, adjusts the position of the fused object, matches or removes shadows, and processes the boundary line with the fused object to look natural.

The heterogeneity minimization technique can use both general and known techniques.

The service server 1500 converts the fusion content provided from the content fusion server 1400 into an MPEG 7 format that can be played as an optimized video in the terminal unit 1600, and stores the fused content in an allocated area. Provided by the request of the terminal unit 1600.

The terminal unit 1600 includes a smart phone or a computer, and the authentication content server 1300, the content fusion server 1400, the service server 1500 by authentication of the authentication server 1700 I can connect it.

The terminal 1600 receives the analyzed content by accessing the analysis content server 1300 or accesses the content fusion server 1400 to receive the fused content with the heterogeneity in a minimized state or accesses the service server 1500. By doing so, the converged content converted into the MPEG 7 format can be provided.

The terminal unit 1600 may play augmented reality content as a video using the fusion content of the MPEG 7 format provided from the service server 1500. The play of the video is performed because 24 unit frame images are displayed continuously per second, and the number of unit frames displayed per second can be added or subtracted.

The authentication server 1700 authenticates the terminal 1600 to access the customized AR service system. The authentication server 1700 determines whether the member who paid the membership fee to the terminal 1600 or a member corresponding to a predetermined rule and accesses the terminal 1600 when the terminal 1600 is determined to have a proper membership. The terminal unit 1600 that is not authenticated because there is no proper membership is blocked from access.

Hereinafter, the operation method of the customized augmented reality service system according to the present invention will be described in detail with reference to FIGS. 3 and 2. When the corresponding signal for controlling shooting is input, the photographing apparatus 1100 collects contents of the surrounding environment. In operation S2100, the memory is stored as the collected content in the allocated memory area.

When the content is collected, the photographing apparatus 1100 includes the latitude, longitude, and coordinate information or location information including the latitude, longitude, and the sea level of the collection point as metadata in the collection content and store the data using the GPS receiver.

The photographing apparatus 1100 provides the original content stored at the request of the location-based content server 1200, that is, the metadata is included and contents collected from the surrounding environment, and the location-based content server 1200 provided with the collection content is provided. Stored as original content in the allocated memory area (S2200).

The analysis content providing server 1300 receives the original content by requesting the location-based content server, and provides the original content when the terminal 1600 connected to the authentication content is requested.

Meanwhile, the analysis content providing server 1300 analyzes the provided original content, extracts spatiotemporal information, and stores it in an area allocated as an analysis content in a structured state in which a background and an object or a character are separated by image analysis (S2300). .

Spatio-temporal information extraction extracts time series information including diary, season, and time, and spatial information including latitude, longitude, and altitude from metadata of each unit frame image constituting original content.

If metadata is not included in the content and spatial information cannot be analyzed and extracted, spatial information may be extracted based on tags from Yahoo Flickr, Google Image Search, YouTube, and the like.

In addition, by analyzing the image of the content, time series information and spatial information by diary, season, time may be extracted.

That is, the analysis of the original content may use metadata, an object photographed in an image, or another known method.

Image analysis extracts similarities from each unit frame image constituting the original content and designates a frame image including the most extracted similarities as an index frame image.

The content fusion server 1400 receives the analysis content by accessing the analysis content providing server and uploads the content to be fused by accessing the terminal unit 1600 (S2400).

When the connected terminal selects the analysis content (S2500), the content fusion server 1400 converges the analysis content and the uploaded content and stores heterogeneous and minimized fusion content in the allocated area (S2600).

The heterogeneity minimization analyzes the time value of each included frame image when the analysis content is selected, and selects and fuses the frame image having the smallest difference from the time value of the content to be uploaded and fused from the terminal unit.

In addition, minimizing heterogeneity removes objects deemed unnecessary from the frame image of the selected analysis content and adds objects deemed necessary.

In minimizing heterogeneity, the brightness, saturation, and size of the selected content and uploaded content are similarly transformed, and the position of the object is removed, the shadow is removed or changed, and the boundary between the fused object and the background is similar to the surroundings. .

The content fusion server 1400 stores the fused content in the allocated area and provides it at the request of the terminal unit 1600 and the service server 1500.

The service server 1500 converts the format of the fusion content provided from the content fusion server into the MPEG 7 format, stores the allocated content in the allocated memory area, and provides the MPEG 7 format when a request is made from the connected terminal (S2700). S2800, S2900).

Hereinafter, with reference to the accompanying Figures 4 and 2 will be described in detail the operating method of the smart phone constituting a customized augmented reality service system according to the present invention.

If it is determined by the terminal 1600 to use the augmented reality service (S3100), access to the authentication server 1700 to obtain the authentication and access to the customized augmented reality service system 1000 (S3200).

If it is determined that the corresponding command to collect the content by the terminal unit is input (S3300), the location-based content is collected by the corresponding control command and provided to the customized augmented reality service system (S3400).

In this case, the terminal unit may function as the photographing apparatus 1100.

When it is determined that the corresponding command of the fusion request is input by the terminal unit (S3500), the terminal accesses a customized augmented reality service system, searches for and selects structured content, and requests fusion with the uploaded content (S3600).

In this case, the terminal unit may select content to be fused from contents previously uploaded to the system or newly upload content to be fused.

If it is determined that the corresponding command for receiving the fused content is input (S3700), the terminal unit accesses the customized AR service system, downloads the content converted to the MPEG 7 format, stores the content in the allocated area, and processes and plays the video ( S3800).

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art.

1000: Customized Augmented Reality Service System
1100: recording device 1200: location-based content server
1300: Analysis content providing server 1400: Content convergence server
1500: service server 1600: terminal
1700: Certificate Server

Claims (1)

A photographing apparatus having a GPS information receiving apparatus and storing the collected contents of the surrounding environment as collected contents together with the location information by the GPS; A location-based content server that receives the collected content from the photographing apparatus and stores it in an allocated area and provides the original content, and receives the original content from the location-based content server and analyzes spatiotemporal information in an area allocated as analysis content; An analysis content providing server for storing, The content fusion server for receiving the analysis content from the analysis content providing server, uploaded from the terminal unit and fused with the input content, and storing the fused content in the allocated area as fused content, The content fusion server The server receives the converged content from the service server, converts it into a format that can be played by the terminal unit, and stores the data in the allocated area. Terminal unit to play, the terminal unit In the method of operating a customized augmented reality service system comprising a server authentication authentication to authenticate the access to the analysis content providing server, the content convergence server and the service server,
When the photographing control signal is input by the photographing apparatus, the surrounding content is photographed and collected while receiving and analyzing the GPS signal at the collecting point to include the identified coordinate information in the metadata area of the collected content, and as the collection content. A photographing step of storing;
A collection step of receiving the collected content from the photographing apparatus by the location-based content server and storing the collected content as original content in an allocated memory area;
The original content is received from the location-based content server by the analysis content providing server, and when the terminal accesses and provides the original content, the original content is provided and the spatiotemporal information is extracted by analyzing the original content. Analyzing the data stored in the memory area allocated as the analysis content in the separated structured state;
Receiving the analysis content from the analysis content providing server by the content fusion server and uploading the content by accessing the terminal unit and selecting the analysis content, the analysis content and the uploaded content are fused and heterogeneous and minimized fusion content is processed. A fusion step of storing the data in an allocated memory area; And
The service server receives the fusion content from the content fusion server, converts it into an MPEG 7 format, stores it in an allocated area, and when the terminal accesses the fusion content, the fused content is converted into the MPEG 7 format. A service step of providing content; Including but not limited to:
The space-time information extraction
It is made by extracting time series information including diary, season, time and spatial information including latitude, longitude, and altitude from metadata of each unit frame image constituting the original content,
The image analysis is
Extracting similarity from each unit frame image constituting the original content and designating a frame image including the extracted similarity as the index frame image,
Minimizing the heterogeneity
When the analysis content is selected, the time value of each included frame image is analyzed and the frame image having the smallest difference from the time value of the content to be uploaded and fused from the terminal unit is selected and fused.
Remove the object selected as unnecessary from the frame image of the analysis content selected in the fusion step, add the object selected as necessary,
The brightness and saturation of the content selected in the fusion step and the content uploaded in the fusion step are similarly converted, the position of the object is removed, the shadow is removed, and the fused boundary is similar to the surroundings.
The terminal unit
The fused content provided in the MPEG 7 format and provided from the service server is played as a video, any one or more than one of a smartphone and a computer,
The space-time information is
Operating method of a smart phone user customized augmented reality service system using a digital map, characterized in that the image can be extracted by analyzing the image contained in the original content.

Images