KR102662437B1 - System for providing optical character recognition based tourist information service - Google Patents

Abstract

An OCR-based tourist information service provision system is provided, and when selecting or shooting an image to perform OCR (Optical Character Recognition), a user terminal and a user terminal that output search results using keywords for characters included in the image. When selecting or shooting an image to perform OCR on, a confirmation unit that confirms the selected image or image on the shooting screen, a search unit that searches letters included in the image using pre-built OCR, and a search using the searched letters as keywords. It includes a tourist information service providing server including an extraction unit that searches and extracts results from a pre-built internal database or an external database searched in real time, and a transmission unit that transmits the extracted results to the user terminal.

Description

ＯＣＲ based tourist information service provision system {SYSTEM FOR PROVIDING OPTICAL CHARACTER RECOGNITION BASED TOURIST INFORMATION SERVICE}

The present invention relates to an OCR-based tourist information service provision system, and provides a tourist information solution that can provide search results by linking with internal and external databases based on characters recognized in captured screens or photos.

Tourist information services are divided into human services such as tourist interpretation guides and cultural tour guides, and non-human services such as promotional materials, brochures, kiosks, etc. From the perspective of tourists, tourism information must satisfy their tourism needs by providing tourists with tourism-related information, and from the perspective of tourism resources, it collects and processes the information contained in each tourist resource and provides it to users. The value as a tourism resource must be conveyed by providing it in a valuable form. Desirable tourism information should be able to reduce uncertainty in tourists' decision-making by providing accurate information. Tourist information services target foreign and domestic tourists and include all service contact points that tourists encounter during their tourism experience. In other words, it includes all tourist guide, information, and interpretation services related to accommodation, food and beverage, transportation, tourist souvenirs, ICT, and hospitality services that tourists experience before, during, and after the trip.

At this time, a method of performing tourist guidance using GPS, QR code, and NFC tags was researched and developed. In relation to this, the prior art, Korean Patent No. 10-1376790 (announced on March 20, 2014) and Korean Publication Patent No. 2014-0117337 (published on October 7, 2014) prints tourist destination guide content based on GPS, provides photo or category-based tourist destination guide information, and provides tourist destination guide information based on QR code. A configuration that provides and a configuration that applies NFC tags to maps, guide maps, and schematics to link them to a web page along a URL link when each object is tagged with an NFC tag is disclosed.

However, in the former case, the QR code is scanned and then moved to the corresponding link, and the NFC tag only performs a search of the tagged area. In the case of tourist attractions without a QR code or NFC tag, tourist information is provided to tourists. You have to search on your own. However, if you go sightseeing abroad and do not know the local language, there are cases where even searching is impossible because the keyboard is different or the letters themselves are not recognized even after looking at the letters. In the case of images, you must be able to search the image itself, but this is not possible. Since it is not easy to specify, search results often come up with incorrect results. Accordingly, research and development of a system that can provide tourist information after recognizing and searching letters included in a shooting screen or stored photos is required.

One embodiment of the present invention builds an internal database according to at least one category for tourist information, then links it with an external database server, and recognizes characters in the captured screen or pre-stored photos using OCR on the user terminal. By sending search results corresponding to letters recognized through OCR to the user's terminal, relevant search results can be provided even if the user goes abroad and does not know the letters, and even in the case of domestically, tourist attractions, public transportation, and tourist attractions recognized as letters It is an OCR-based tourist information service that can find information related to restaurants and restaurants, and can increase the accuracy of search results while providing diversity in searches by adding location information to results recognized through deep learning even in the case of images rather than text. A delivery system can be provided. However, the technical challenge that this embodiment aims to achieve is not limited to the technical challenges described above, and other technical challenges may exist.

As a technical means for achieving the above-described technical problem, an embodiment of the present invention provides that when selecting or photographing an image to perform OCR (Optical Character Recognition), characters included in the image are searched as keywords. When selecting or shooting an image to perform OCR on the user terminal and the user terminal that outputs the results, a confirmation part that confirms the selected image or image on the shooting screen, and a search that searches for characters included in the image using a pre-built OCR It includes a tourist information service providing server that includes an extraction unit that searches and extracts search results using the searched letters as keywords from a pre-built internal database or an external database searched in real time, and a transmission unit that transmits the extracted results to the user terminal. do.

According to one of the above-described means for solving the problem of the present invention, an internal database is built according to at least one category for tourist information, then linked with an external database server, and a captured screen for shooting or a pre-stored photo is stored on a user terminal. By recognizing letters with OCR and sending search results corresponding to the OCR-recognized letters to the user's terminal, relevant search results can be provided even if the user goes abroad and does not know the letters, and even in the case of domestic use, they are recognized as letters. It can find information related to tourist attractions, public transportation, and restaurants, and even in the case of images rather than text, location information can be added to the results recognized through deep learning to provide diversity in searches and increase the accuracy of search results. .

1 is a diagram illustrating an OCR-based tourist information service providing system according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating a tourist information service providing server included in the system of FIG. 1.
Figures 3 and 4 are diagrams for explaining an embodiment in which an OCR-based tourist information service is implemented according to an embodiment of the present invention.
Figure 5 is an operation flowchart for explaining a method of providing OCR-based tourist information service according to an embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts unrelated to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected," but also the case where it is "electrically connected" with another element in between. . In addition, when a part is said to "include" a certain component, this means that it does not exclude other components, but may further include other components, unless specifically stated to the contrary, and one or more other features. It should be understood that it does not exclude in advance the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

The terms “about,” “substantially,” and the like used throughout the specification are used to mean at or close to that value when manufacturing and material tolerances inherent in the stated meaning are presented, and are used to enhance the understanding of the present invention. Precise or absolute figures are used to assist in preventing unscrupulous infringers from taking unfair advantage of stated disclosures. The term “step of” or “step of” as used throughout the specification of the present invention does not mean “step for.”

In this specification, 'part' includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Additionally, one unit may be realized using two or more pieces of hardware, and two or more units may be realized using one piece of hardware. Meanwhile, '~ part' is not limited to software or hardware, and '~ part' may be configured to reside in an addressable storage medium or may be configured to reproduce one or more processors. Therefore, as an example, '~ part' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and 'parts' may be combined into a smaller number of components and 'parts' or may be further separated into additional components and 'parts'. Additionally, components and 'parts' may be implemented to regenerate one or more CPUs within a device or a secure multimedia card.

In this specification, some of the operations or functions described as being performed by a terminal, apparatus, or device may instead be performed on a server connected to the terminal, apparatus, or device. Likewise, some of the operations or functions described as being performed by the server may also be performed in a terminal, apparatus, or device connected to the server.

In this specification, some of the operations or functions described as mapping or matching with the terminal mean mapping or matching the terminal's unique number or personal identification information, which is identifying data of the terminal. It can be interpreted as:

Hereinafter, the present invention will be described in detail with reference to the attached drawings.

1 is a diagram illustrating an OCR-based tourist information service providing system according to an embodiment of the present invention. Referring to FIG. 1, the OCR-based tourist information service providing system 1 includes at least one user terminal 100, a tourist information service providing server 300, at least one seller terminal 400, and at least one external database. It may include a server 500. However, since the OCR-based tourist guide service providing system 1 of FIG. 1 is only an embodiment of the present invention, the present invention is not limited to FIG. 1.

At this time, each component of FIG. 1 is generally connected through a network (Network, 200). For example, as shown in FIG. 1, at least one user terminal 100 may be connected to the tourist information service providing server 300 through the network 200. Additionally, the tourist information service providing server 300 may be connected to at least one user terminal 100, at least one seller terminal 400, and at least one external database server 500 through the network 200. Additionally, at least one seller terminal 400 may be connected to the tourist information service providing server 300 through the network 200. In addition, at least one external database server 500 may be connected to at least one user terminal 100, the tourist information service providing server 300, and at least one seller terminal 400 through the network 200.

Here, the network refers to a connection structure that allows information exchange between each node, such as a plurality of terminals and servers. Examples of such networks include a local area network (LAN) and a wide area network (WAN). Wide Area Network, Internet (WWW: World Wide Web), wired and wireless data communication network, telephone network, wired and wireless television communication network, etc. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), 5th Generation Partnership Project (5GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), and Wi-Fi. , Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), RF (Radio Frequency), Bluetooth network, NFC ( It includes, but is not limited to, Near-Field Communication (Near-Field Communication) network, satellite broadcasting network, analog broadcasting network, and DMB (Digital Multimedia Broadcasting) network.

In the following, the term at least one is defined as a term including singular and plural, and even if the term at least one does not exist, each component may exist in singular or plural, and may mean singular or plural. This should be self-explanatory. In addition, whether each component is provided in singular or plural form may be changed depending on the embodiment.

At least one user terminal 100 uploads a captured screen or a pre-stored photo using an OCR-based tourist information service-related web page, app page, program, or application, or allows access to the tourist information service providing server 300, , It may be a terminal of a tourist who receives and outputs the results of character recognition and search using OCR of the tourist information service providing server 300.

Here, at least one user terminal 100 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop equipped with a navigation system and a web browser, a desktop, a laptop, etc. At this time, at least one user terminal 100 may be implemented as a terminal capable of accessing a remote server or terminal through a network. At least one user terminal 100 is, for example, a wireless communication device that guarantees portability and mobility, and includes navigation, personal communication system (PCS), global system for mobile communications (GSM), personal digital cellular (PDC), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) ) It may include all types of handheld-based wireless communication devices such as terminals, smartphones, smartpads, and tablet PCs.

The tourist information service providing server 300 may be a server that provides an OCR-based tourist information service web page, app page, program, or application. Additionally, the tourist information service providing server 300 may be a server that builds an internal database to enable tourist information for at least one category and sets it to be linked with at least one external database server 500. In addition, when driving a shooting screen captured by the user terminal 100 or requesting OCR in a pre-stored photo, the tourist information service providing server 300 recognizes the characters in the shooting screen or the characters in the photo as OCR and then , It may be a server that extracts search results from an internal database or an external database server 500 that uses letters as keywords and then transmits them to the user terminal 100. At this time, the tourist information service providing server 300 may further increase search accuracy based on GPS, and may arrange search results in latest order to provide information based on recent tourist information.

Here, the tourist information service providing server 300 may be implemented as a computer that can access a remote server or terminal through a network. Here, the computer may include, for example, a laptop equipped with a navigation system and a web browser, a desktop, a laptop, etc.

When a price search event of the user terminal 100 occurs using an OCR-based tourist information service-related web page, app page, program, or application, at least one seller terminal 400 conversely offers a lower price to the user. It may be a seller's terminal that sells or provides products or services that induce visits and payments to the terminal 100.

Here, at least one seller terminal 400 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop equipped with a navigation system and a web browser, a desktop, a laptop, etc. At this time, at least one seller terminal 400 may be implemented as a terminal capable of accessing a remote server or terminal through a network. At least one seller terminal 400 is, for example, a wireless communication device that guarantees portability and mobility, and includes navigation, personal communication system (PCS), global system for mobile communications (GSM), personal digital cellular (PDC), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) ) It may include all types of handheld-based wireless communication devices such as terminals, smartphones, smartpads, and tablet PCs.

At least one external database server 500 allows access to the tourist information service providing server 300 and provides search results with or without using OCR-based tourist information service-related web pages, app pages, programs, or applications. It could be a server. Here, at least one external database server 500 may be implemented as a computer that can connect to a remote server or terminal through a network. Here, the computer may include, for example, a laptop equipped with a navigation system and a web browser, a desktop, a laptop, etc.

Figure 2 is a block diagram for explaining the tourist information service providing server included in the system of Figure 1, and Figures 3 and 4 show an embodiment in which the OCR-based tourist information service is implemented according to an embodiment of the present invention. This is a drawing for explanation.

Referring to Figure 2, the tourist information service providing server 300 includes a confirmation unit 310, a search unit 320, an extraction unit 330, a transmission unit 340, a deep learning recognition unit 350, and a location reflection unit. It may include a unit 360, a location confirmation unit 370, a category search unit 380, a public transportation information unit 390, and a search engine unit 391.

The tourist information service providing server 300 according to an embodiment of the present invention or another server (not shown) operating in conjunction with at least one user terminal 100, at least one seller terminal 400, and at least one external When transmitting an OCR-based tourist information service application, program, app page, web page, etc. to the database server 500, at least one user terminal 100, at least one seller terminal 400, and at least one external database server (500) can install or open an OCR-based tourist information service application, program, app page, web page, etc. Additionally, a service program may be run on at least one user terminal 100, at least one seller terminal 400, and at least one external database server 500 using a script executed in a web browser. Here, a web browser is a program that allows the use of web (WWW: World Wide Web) services and refers to a program that receives and displays hypertext written in HTML (Hyper Text Mark-up Language), for example, Netscape. , Explorer, Chrome, etc. Additionally, an application refers to an application on a terminal and includes, for example, an app running on a mobile terminal (smartphone).

Referring to FIG. 2, when the user terminal 100 selects or captures an image to perform OCR, the confirmation unit 310 may confirm the selected image or the image on the capture screen.

The search unit 320 can search for letters included in the image using a pre-built OCR. Currently, OCR (Optical Character Recognition) is being used in various fields (data input automation, document editing, document classification, etc.), and services that provide OCR such as Google, Kakao, and Naver are becoming more diverse. Compared to its versatility, there are limitations to this technology. When using the provided OCR, OCR is applied to the entire image, so if OCR needs to be applied only to a specific location, an appropriate template must be used. Otherwise, all text in the image may be displayed mixed up. Additionally, there is a problem that OCR is not applied properly when the image is rotated. Accordingly, one embodiment of the present invention applies OCR only to a desired location, extracts only the text for that portion, and then searches the extracted characters (letters) to provide search results. This model can be largely composed of three stages: performing additional learning on the pre-trained object detection model to detect the location where OCR is to be applied (first and second stages) step), the text can be extracted by applying OCR only to that location (third step).

<Data collection and modeling>

In order to build a model that extracts information from photos taken at tourist attractions, photos can be collected for at least one category, then an image dataset can be constructed and used for learning. To learn the YOLO model in the first and second stages (Stage 1 & Stage 2) to detect letters in each category, you can learn to detect objects (letters) by labeling photos of tourist attractions in each category. there is. The first and second stages are processes for object detection, and the third stage (Stage 3) is a stage where OCR is applied to the objects (letters) extracted in the first and second stages to extract text. The reason for dividing the model into three stages is to prevent the problem that when OCR is applied directly to the original image, other texts other than the desired text are mixed and output. The model used for object detection may be the COCO dataset, and a model that is additionally trained by adding learning data collected from the platform of the present invention to the pre-trained YOLO model can be used. The optimizer can use SGD (Stochastic Gradient Descent) and train the model by changing only epochs while keeping other hyperparameter values the same.

The OCR model can use the Google Cloud Vision API, but is not limited to this. Additionally, considering that the domain of the present invention is a tourism domain, performance may not be good if used immediately, so preprocessing may be performed before inputting data into the OCR model. As preprocessing, you can resize by 1.5 times and then apply INTER AREA using interpolation, and after converting to black and white, you can apply brightness control. You can input preprocessed input data into an OCR model and output text as a result. Post-processing such as removal of special characters, spaces, and duplicate words can be applied to the printed title text to produce the final result. At this time, if the number of letters to which OCR needs to be applied increases, the speed may slow down, so when the user confirms the capture screen or uploads a pre-saved photo, the user is sent based on the bounding box extracted by the object detection model in the first step. The user can select the desired object (letter) in the second step, and once the user confirms the desired object, the final result can be derived through the object detection model in the second step and the OCR model in the third step.

The extraction unit 330 may search and extract search results using the searched letters as keywords from a pre-built internal database or an external database searched in real time. For example, let's assume that a user arrived in Jeju Island and took a photo of the signboard of Yeondon, a hot spot on Instagram. At this time, the results the user wants would be information such as Yeondon's menu, price, taste rating, reviews, business hours, etc., but if a Yeondon franchise in Seoul appears in the search results, there is no reason to search and the search engine may feel that it is not smart. You can. If search results suitable for the situation are not found, the user may not use the OCR of the present invention. Accordingly, in one embodiment of the present invention, a method of semi-automatically collecting contextual information keywords related to the situation faced by the user in addition to keywords input through OCR can be used. The user's real-time situational information that can be automatically collected includes information such as weather, location, time, season, and traffic, and information that cannot be known from the outside, such as the user's current mood, is directly entered and collected in the database. The additional keywords collected can be used to determine under what circumstances the user collected the information, and can be combined with other information to provide effective search results tailored to the user's search situation.

<Generation of situation information>

As detailed above, situational information refers to all external information that can be received in real time in the situation the user is in, and can be information such as time, weather, location, season, and traffic. By using context information, you can find out under what circumstances the user collected information, and by using this for search, you can provide information tailored to the user's situation, making search more efficient and convenient. For example, when trying to get information about bars to go to on a rainy day, even if only the word bar is recognized through OCR, you can get search results related to bars to go to on a rainy day, such as a makgeolli restaurant or a green onion pancake restaurant that the user saved on a rainy day. You can. Additionally, when searching for a restaurant while traveling, you do not need to include location information in the search keyword, such as [Restaurants in Suyeong-gu, Busan]. Just search for the word [restaurant] and the searcher's location will be automatically determined to search for nearby restaurants. I can do it.

<Automatically secure situation information>

External context keywords can be obtained automatically using web page parsing. Parsing refers to a syntax analysis process in which a computer acting as a parser classifies sentence-level strings into tokens that are considered meaningful and reconstructs them into a parse tree. In other words, you can retrieve only the desired string from a web page containing a lot of information. Among various situational information, it is possible to intuitively recognize and classify external situations and obtain weather and location information, which have high information utilization. This is done twice: when saving information and when retrieving it. The process of obtaining location and weather information is done by specifying and retrieving the desired information from a website that provides local weather information based on GPS. In this way, you can obtain various situational information about the user, such as location and weather, as well as season, time, and country.

The user's location and weather information are automatically collected and used automatically during searches, but mood information must be selected by the user when entering a URL and related keywords or when searching. Mood information can be classified into three categories: good, normal, and bad. This information is optional and the user does not have to select it when entering or searching, but if you want to get information related to your current mood, you can set the mood information and then search to get more detailed search results that include the mood information. Likewise, if you also enter mood information when inputting, you can find out what mood the person who collected the URL was in at the time to obtain the information, making it possible to share more detailed information. Data parsing, that is, the process of securing situational information, is performed automatically, so search results with situational information that matches the location and weather information at the moment of search can be searched together, and these results can be provided with priority.

The transmission unit 340 may transmit the extracted result to the user terminal 100. When selecting or photographing an image to perform OCR (Optical Character Recognition), the user terminal 100 may output results in which characters included in the image are searched as keywords.

If there are no letters in the image on the screen selected or captured by the user terminal 100, the deep learning recognition unit 350 detects and confirms the object in the image using a pre-built deep learning algorithm, and selects the confirmed object. Search results through image search can be transmitted to the user terminal 100. If there are no letters in the image (photo), object detection must be performed based on deep learning to detect and identify the object and perform a search for the object. At this time, you are searching for a place in a tourist destination. For example, if you need to search using only images without recognizing the name or place name due to the lack of letters, deep learning must be used for place recognition. For example, a famous Tanghulu store in Taiwan is located in Shilin Night Market, and if you search for Tanghulu taken at Shilin Night Market, if Tanghulu itself exists, you must collect information other than Tanghulu and then specify the location to respond. Information about [Shilin Night Market] may appear as a search result.

For this purpose, you can use Patch-NetVLAD, the latest image search technology for place recognition, which extracts local features and global features using patch sizes of various sizes (multi-scale) and fuses them. By combining local features and global features, landmarks corresponding to various scales such as size change and angle change are specified. Landmarks estimate the image with the highest similarity score for place recognition through spatial scoring and nearest neighbor. This method uses only one color image as input, which significantly reduces the accuracy of the search when searching for a night image where object distinctions are not clear. Accordingly, in one embodiment of the present invention, Patch-NetVLAD is used as the basis of image search technology, and video segmentation information images and regions of interest are added to improve search accuracy through semantic information of each pixel, and to increase search accuracy through semantic information of each pixel and images taken at night or at night. You can also use a network that can adapt well to recognized shooting screens.

Alternatively, MAC or R-MAC can be used. MAC converts the image expression method into a vector of size K through Max-Pooling operation on image information, and calculates the cosine similarity between these vectors. Similarity comparisons are made by acquiring only the information of the location with the highest activation, which has the disadvantage of not reflecting the local information of the image. R-MAC compensates for local feature loss by dividing R areas of various sizes and calculating similarity according to various areas. In other words, rather than calculating the similarity of the entire image, a re-ranking process was added to first select a similar region (R) and then calculate the similarity for that region. You can also use image search using triplet loss, which has been developed from this.

<Image Search>

In the case of general Patch-NetVLAD, it learns only on daytime images and shows weak results on night images. Accordingly, as described above, in order to respond to environmental changes, both day and night images can be used for learning to be designed to be robust to weather or time changes. To obtain a night image, the day image can be augmented with a night image through AU-GAN (Uncertainty-Aware Generative Adversarial Network), and an image segmentation information image and a region of interest extraction module can be added to improve the performance of the network. The input daytime data set is converted into an image segmentation information augmented for the night time, and each image is extracted as a feature. Afterwards, it goes through the region of interest extraction module and is input to the image search module, and finally, the location can be estimated by deriving an image estimated to be at the same location as the input image.

<Image segmentation technology>

The general method has the problem of performance decreasing when the environment changes by conducting learning and verification using daytime images in the same environment. To solve this, image segmentation information images can be additionally used. If the image segmentation technology is trained simultaneously with the main network, the parameters to be learned become enormous and the learning time increases. Therefore, to obtain image segmentation information, you can use Panoptic-DeepLab, which pre-trained ResNext-101 as a baseline using already learned Cityscapes Datasets. By applying this technology, accuracy can be increased by focusing on important objects such as roads, buildings, and trees rather than unnecessary objects such as cars or people when estimating location. If you learn by focusing on cars or people that change momentarily, the possibility of estimating a different location with a similar person increases, so an image segmentation information module can be added to determine important fixed objects when estimating a location. As an input to the image segmentation information module, image segmentation information for day images, not night images, can be used. This is because in the case of night images, objects are not clearly visible and most of the information is distorted.

<Detection of area of interest>

The region of interest refers to the weight value for the position that should be considered important in each input image. As mentioned above, when estimating the location, it is necessary to capture and match features that are continuously maintained rather than temporary phenomena, so the importance of moving objects can be lowered and the importance of fixed objects can be determined more clearly to influence learning. Region of interest detection can be introduced. For example, the region of interest may consist of a total of four layers, and passes through the average pooling and max pooling layers and then the convolution layer. Afterwards, it goes through a sigmoid layer to multiply with the features extracted from VGG16, and finally the region of interest is detected. By using region-of-interest images, even if the environment changes, it is possible to create an architecture that is robust to environmental changes by determining the importance of day and night images and learning them.

The location reflection unit 360 detects the current location of the user terminal 100 using GPS or extracts the location from the GeoTag of the selected image, and then adds GPS or location as a search term along with a keyword corresponding to the letter. Search results can be extracted. Assuming that the user is currently located at a tourist attraction and searches based on the current location of the tourist attraction, the GPS of the user terminal 100 can be extracted and a search can be performed together.

The location confirmation unit 370 detects letters in the image captured by the user terminal 100 or the image on the capture screen, and then searches for the letters to determine the location if the letter is a letter for determining the location of the user terminal 100. )'s current location can be confirmed, and already established POI (Point of Interest) information can be extracted based on the confirmed current location and transmitted to the user terminal 100. If the user is in Jeju Island and searches for Yeondon, the search term [Jeju Island + Yeondon] can be automatically created and searched. Alternatively, when a user takes an image of the business name of Jeju Island Yeondon or searches for a phone number, the user's current location can be determined based on the phone number or business name, and information such as reviews, menus, and prices can be searched.

When selecting a category to be searched by OCR in the user terminal 100, the category search unit 380 preferentially extracts search results included in the category when searching letters searched by OCR as a keyword to search the user terminal 100. It can be sent to . For example, if you select a category such as public transportation, restaurants, tourist attractions, lodging, or events/events, the information stored in the category or information from an external database can be sorted by priority.

When the option to travel by public transportation is selected on the user terminal 100, the public transportation information unit 390 provides public transportation information including public transportation routes, dispatch times, fare information, and routes based on the current location of the user terminal 100. Traffic information can be provided. At this time, the current location can be determined to determine which station or stop the user is currently at, and the corresponding route can be provided preferentially.

The search engine unit 391 can set search options to provide search results by entering a search word in addition to selecting or shooting an image on the user terminal 100. In other words, a general text input search engine can also be provided.

In addition, in one embodiment of the present invention, a platform for recommending tourist destinations or planning a travel itinerary may be further provided. In other words, it provides various tourist information before a trip to help users plan their travel itinerary, and during the trip, it provides information such as the user's current location, season, and weather in the surrounding area, as well as the user's number of steps today, whether or not they eat, and what type of transportation they use. An architecture that extracts dynamic data that combines context awareness and user information such as nationality, gender, age, number of visits to Korea, and user preferences to search and recommend tourist information tailored to the user's situation. It can be designed.

Users can register basic information such as nationality, gender, age, interests, cities to visit, accommodations, etc. through the API module in order to receive customized tourist information recommendations before travel. General information can be used to make travel plans and get help during travel. You can view tourist information. After completing your trip, you can share your tourist course or write a review. Through this, you can refer to basic tourist information and evaluations of tourist courses and tourist destinations by domestic and foreigners. If the user interface system provides tourist information, the tourist destination registration platform is a system that creates tourist courses based on this. Before traveling, general users register the city they want to visit, accommodations, etc. and receive key tourist information. Using this, you can plan your own tour itinerary using a combination of the tourist attractions provided. Next, during the trip, the actual travel course is recorded in the system based on the tourist's location. The seller terminal 400 receives the tourist's tourist course preference.

<Situation awareness platform>

Open APIs for IoT sensors, weather, and traffic can be used to recognize factors that affect tourists' travel and reflect them when recommending tourist information. While static user information is stored in the DB through user information, information collected through Open API based on the user's GPS is stored in the DB as dynamic user information. User circumstances such as the tourist's location and fatigue, tour time, and hunger level, and external circumstances such as weather, population density, and traffic congestion, etc. correspond to dynamic user information and are integrated and stored in the user DB. It can be composed of a tourist destination analysis module, a user analysis module, and a search and recommendation module to recommend tourist information that suits the user's preferences to tourists. For the analysis module, user DB, interest DB, and tourist destination DB are used. In particular, the user DB includes situational awareness information. For domestic purposes, you can use statistical data from the foreign tourist survey report provided by the Korea Tourism Organization for search and recommendation purposes. The statistical data contains information on the main purpose of visit and destinations based on nationality, gender, and interests, which can be used as a reference when designing search and recommendation modules. The search and recommendation module has weight variables for search and recommendation, allowing different results to be recommended depending on the results of the analysis module.

The search and recommendation module allows tourists to view tourist information before traveling, register basic information and interests, and make plans through recommended tourist courses based on this. During your trip, you receive continuous tourist information recommendations by combining situational awareness information using IoT sensors and Open API, tourist behavior analysis information, and basic information registered before travel. Finally, after the trip, you can write a review based on the actual tourist course created during the trip and share it with other tourists. The user terminal 100 plans a tour schedule by recommending customized tourist destinations based on the user's information through the API module before traveling, and provides real-time customized tourist information that combines the user's surrounding situation information, behavioral tendencies, and preferences during the trip. It can be provided. Therefore, through the proposed tourism platform, tourists visiting Korea can reduce the inconvenience they feel while planning a trip and enjoy a satisfying trip by receiving recommendations for tourist destinations that fit the real-time situation.

Hereinafter, the operation process according to the configuration of the tourist information service providing server of FIG. 2 described above will be described in detail using FIGS. 3 and 4 as an example. However, it will be apparent that the embodiment is only one of various embodiments of the present invention and is not limited thereto.

Referring to Figure 3, (a) the solution (Metarism=Meta+Tourism) according to an embodiment of the present invention builds a database by collecting at least one information in at least one category, (b) an external database and An interconnected DBMS can be built so that it can also be interconnected. In addition, when the user terminal 100 confirms an image or selects a photo on the capture screen as shown in (c), the tourist information service providing server 300 extracts the location of the letter through object detection in the image or photo. After recognizing the letters with OCR, you can provide search results by entering the letters as search words in a search engine. At this time, the tourist information service providing server 300 may use place search deep learning to determine the place and then identify the object if it is not a photo or captured screen containing the letter (d). In addition, as shown in (a) of FIG. 4, the seller terminal 400 may inversely suggest a lower price for the search results on the user terminal 100 to encourage visits and payments, and provide a tourist information service. The server 300 may provide search results according to the exact location by linking GPS as shown in (b). The tourist information service providing server 300 may check the language of each country and automatically provide an exchange rate according to the language as shown in (c), and may also perform translation and interpretation as shown in (d).

Matters that are not explained about the OCR-based tourist information service provision method in FIGS. 2 to 4 are the same as or can be easily inferred from the information explained about the OCR-based tourist information service provision method in FIG. 1. Therefore, the following description will be omitted.

FIG. 5 is a diagram illustrating a process in which data is transmitted and received between components included in the OCR-based tourist information service providing system of FIG. 1 according to an embodiment of the present invention. Hereinafter, an example of the process of transmitting and receiving data between each component will be described with reference to FIG. 5, but the present application is not limited to this embodiment, and the process shown in FIG. 5 according to the various embodiments described above It is obvious to those skilled in the art that the process of transmitting and receiving data can be changed.

Referring to FIG. 5, when the tourist information service providing server selects or captures an image to perform OCR on the user terminal, it confirms the selected image or the image on the capture screen (S5100) and writes the letters included in the image. Search using the constructed OCR (S5200).

In addition, the tourist information service providing server searches and extracts search results using the searched letters as keywords from a pre-built internal database or an external database searched in real time (S5300) and transmits the extracted results to the user terminal (S5400) .

The sequence between the above-described steps (S5100 to S5400) is only an example and is not limited thereto. That is, the order between the above-described steps (S5100 to S5400) may change, and some of the steps may be executed simultaneously or deleted.

Matters that are not explained about the OCR-based tourist information service provision method of FIG. 5 are the same as or can be easily inferred from the content explained about the OCR-based tourist information service provision method through FIGS. 1 to 4. Therefore, the following description will be omitted.

The OCR-based tourist information service providing method according to an embodiment described with reference to FIG. 5 can also be implemented in the form of a recording medium containing instructions executable by a computer, such as an application or program module executed by a computer. . Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and non-volatile media, removable and non-removable media. Additionally, computer-readable media may include all computer storage media. Computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data.

The method of providing an OCR-based tourist guide service according to an embodiment of the present invention described above can be executed by an application installed by default on the terminal (this may include programs included in the platform or operating system, etc., installed by default on the terminal). It may also be executed by an application (i.e., program) installed by the user directly on the master terminal through an application providing server such as an application store server, an application, or a web server related to the service. In this sense, the OCR-based tourist information service providing method according to an embodiment of the present invention described above is implemented as an application (i.e., program) installed by default on the terminal or directly installed by the user and can be read by a computer such as the terminal. It can be recorded on any recording medium.

The description of the present invention described above is for illustrative purposes, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. will be. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as single may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

Claims (7)

When selecting or photographing an image to perform OCR (Optical Character Recognition), a user terminal that outputs results in which characters included in the image are searched as keywords; and
When selecting or capturing an image to perform OCR on the user terminal, a confirmation unit for confirming the selected image or image on the capture screen, a search unit for searching letters included in the image using a pre-built OCR, and the searched letters An extraction unit that searches and extracts search results with keywords from a pre-built internal database or an external database searched in real time, a transmission unit that transmits the extracted results to the user terminal,
If there are no letters in the image on the screen selected or captured by the user terminal, Patch-NetVLAD is used as the basis for image search technology, and the image segmentation information image and region of interest are used as input through a pre-built deep learning algorithm. A deep learning recognition unit that detects and confirms objects in the image and transmits the results of searching the confirmed objects through image search to the user terminal;
After detecting a letter in an image captured by the user terminal or an image on the screen, if the letter is a letter that determines the location, the current location of the user terminal is determined by searching for the letter and determining the location, and the confirmed location is determined. A tourist information service providing server including a location confirmation unit that extracts pre-established POI (Point of Interest) information based on the current location and transmits it to the user terminal;
Including,
After collecting photos for at least one category, building an image dataset, and learning to detect objects for each category through this, the search unit: i) images preprocessed by resize and black-and-white conversion as the first step; extract the bounding box using an object detection model, ii) as the second step, select the letter based on the bounding box using the object detection model, and iii) extract the text through the OCR model as the third step. do,
While collecting situational information including weather and location,
The tourist information service providing server obtains and extracts corresponding store information based on location from a plurality of store information obtained based on the captured image,
If the weather corresponds to a specific weather, store information stored in response to the specific weather is obtained as a search result,
If there are no letters in the image on the screen selected or captured on the user terminal,
The deep learning recognition unit extracts local features and global features using patch sizes of various sizes, and combines the local features and the global features to specify landmarks for place recognition,
For day images, the Patch-NetVLAD, an image search technology for place recognition, is used, and for night images, the day images are augmented with the night images through AU-GAN, and an image segmentation information image and a region of interest extraction module are added. ,
An OCR-based tourist information service providing system in which fixed objects are judged to be more important than moving objects in relation to the area of interest, which means a weight value according to important positions in the input image.
delete ◈Claim 3 was abandoned upon payment of the setup registration fee.◈ According to claim 1,
The tourist information service providing server is,
Location reflection that detects the current location of the user terminal using GPS or extracts the location from the GeoTag of the selected image, and then extracts search results by adding the GPS or location as a search word along with a keyword corresponding to the letter. wealth;
An OCR-based tourist information service provision system further comprising:
delete ◈Claim 5 was abandoned upon payment of the setup registration fee.◈ According to claim 1,
The tourist information service providing server is,
When the user terminal selects a category to be searched by OCR, a category search unit that preferentially extracts search results included in the category and transmits them to the user terminal when searching the characters searched by OCR as a keyword;
An OCR-based tourist information service provision system further comprising:
According to claim 1,
The tourist information service providing server is,
When the user terminal selects the option to travel by public transportation, a public transportation information unit that provides public transportation information including public transportation routes, dispatch times, fare information, and routes based on the current location of the user terminal;
An OCR-based tourist information service provision system further comprising:
According to claim 1,
The tourist information service providing server is,
In addition to selecting or photographing the image on the user terminal, a search engine unit sets search options to provide search results by entering a search term;
An OCR-based tourist information service provision system further comprising:

Images