KR20210155213A - Restaurant Information Service Providing System and Method - Google Patents

Abstract

The present invention relates to a system for providing restaurant information, which comprises: a location information obtaining unit for obtaining location information using a GPS; an image obtaining unit for obtaining a signboard image of a restaurant from a camera; a direction information obtaining unit for recognizing a photographing direction of the camera using a geomagnetic sensor to obtain direction information; a character recognition unit for recognizing characters in the signboard image; a restaurant recognition unit for recognizing the restaurant using the characters of the signboard image or the signboard image on the basis of the location information and the direction information; and an information providing unit for providing information on the restaurant recognized by the restaurant recognition unit. Accordingly, even when a user does not enter a restaurant, the user can easily obtain various pieces of information, such as menu, price, and review of a desired restaurant, thereby being assisted in restaurant selection.

Description

Restaurant Information Service Providing System and Method

The present invention relates to a system for providing restaurant information, and more particularly, to a system that assists in restaurant selection by providing a user with menu, price, and evaluation information about a nearby restaurant using a camera, a geomagnetic sensor, and GPS of a mobile terminal; It's about how it works.

Today's restaurant advertising methods are very diverse, starting from word of mouth among people, sharing information through social media, referencing blogger reviews, and searching for information through dedicated applications, and the information that can be obtained through them is also limitless.

However, despite the increase in the amount of information, it has become more difficult than before to find a restaurant that satisfies customers. The reason is that as the amount of information increases, there is also an increase in untrustworthy false information, and the reliability of information decreases.

Word of mouth or SNS information is distorted and inflated, so it is often different from the truth, and the reviews of bloggers are usually faked by the restaurant owner. As for information retrieval through a dedicated application, it is common for part-time students to leave reviews that are praiseworthy for being paid.

For this reason, when selecting a restaurant for the first time, such as a tourist destination, reliable information or information about a restaurant is often not obtained enough, so unnecessary time is wasted in selecting a restaurant.

Or, while walking down the street, if you find a restaurant that has a pretty appearance or sells your favorite menu, you will stimulate your curiosity and will want to know information about the restaurant, except for large restaurants that promote a lot on the Internet. If so, there is a limit to obtaining information about the actual restaurant, and since it is not possible to search for information about the restaurant on the street, most of them visit a restaurant or simply give up and pass by.

If you enter a restaurant only by looking at the actual exterior of the restaurant, you may experience unnecessary financial waste due to hygiene or wrong menu selection, higher-than-expected prices, etc. at the same time as you enter the restaurant. Various problems can arise.

Republic of Korea Patent Publication No. 10-2016-0002099 (published on January 08, 2016) Republic of Korea Patent Publication No. 10-2007-0036584 (published on April 03, 2007)

An object of the present invention is to provide a system and method for providing restaurant information that can easily obtain restaurant information using a mobile terminal in order to improve the above problems.

In addition, the present invention provides the user's desired restaurant information, restaurant information of a similar menu, and surrounding commercial area information within the application of the mobile terminal, so that the user can conveniently use the restaurant by providing the menu, price, rating, and location of the restaurant desired by the user. The purpose is to provide a system that can

The system for providing restaurant information in the present invention for solving the above-described problems includes: a location information obtaining unit for obtaining location information using a GPS; an image acquisition unit for acquiring an image of a signboard of a restaurant from a camera; a direction information obtaining unit for recognizing a photographing direction of the camera using a geomagnetic sensor to obtain direction information; a character recognition unit for recognizing characters in the signboard image; a restaurant recognition unit for recognizing the restaurant by using the letters of the sign image or the sign image based on the location information and the direction information; and an information providing unit providing restaurant information of the restaurant recognized by the restaurant recognition unit.

In addition, the restaurant information may include a menu of the restaurant, a price of the menu, a rating of the menu, evaluation information of the menu, and information on surrounding restaurants that sell a menu of the same category as the menu.

In addition, the character recognition unit may recognize a character using a convolutional neural network (CNN) trained on the Mnist dataset.

In addition, when it is determined that the user has stayed at a specific restaurant for more than a preset time based on the location information, it may further include a restaurant evaluation unit that requests the user to evaluate the restaurant.

In addition, when the evaluation of the restaurant is input, the restaurant evaluator may perform emotion analysis through text mining and evaluate the positivity of the restaurant by using the emotion analysis result.

In addition, the restaurant evaluation unit may give additional points to the evaluation of the restaurant or the restaurant evaluation as the number of visitors to the restaurant increases.

In addition, the restaurant recognition unit extracts candidate restaurants located within a predetermined range using the location information, primarily recognizes a restaurant matching the letter among the candidate restaurants, and uses the direction information to match the signboard image and A matching restaurant can be recognized secondarily.

A method of providing restaurant information according to another embodiment of the present invention includes the steps of: confirming a user's location using a GPS; recognizing a photographing direction of a camera using a geomagnetic sensor; obtaining an image of a signboard of a restaurant using the camera; recognizing the letters of the signboard in the signboard image; recognizing a restaurant using any one of the letters of the signboard and the signboard image, the location information of the user, and direction information of the camera; and providing restaurant information of the recognized restaurant to the user.

In addition, the restaurant information may include a menu of the restaurant, a price of the menu, a rating of the menu, evaluation information of the menu, and information on surrounding restaurants that sell a menu of the same category as the menu.

In addition, the step of recognizing the letter may recognize the letter using a convolutional neural network (CNN) trained using the Mnist dataset.

In addition, when it is determined that the user stayed at a specific restaurant for more than a preset time based on the location information, the method may further include the step of requesting an evaluation of the restaurant from the user.

In addition, in the requesting the evaluation of the restaurant, when the evaluation of the restaurant is input, emotion analysis may be performed through text mining, and the positive degree of the restaurant may be evaluated using the emotion analysis result.

In addition, in the step of requesting the evaluation of the restaurant, an additional point may be given to the evaluation of the restaurant as the number of visitors to the restaurant or the evaluation of the restaurant increases.

In addition, the step of recognizing the restaurant may include extracting candidate restaurants based on the location information; recognizing a restaurant matching the letter among the candidate restaurants; and recognizing a restaurant matching the signboard image by using the direction information.

In addition, the step of recognizing a restaurant matching the signboard image may recognize a restaurant through similarity matching using signboard images of the candidate restaurants and the signboard image acquired with the camera.

The restaurant information providing system and method of the present invention as described above can receive information about the restaurant even in a state that does not enter the restaurant.

In addition, you can check the reviews of the restaurant together, so you can easily obtain review information about the restaurant without searching for it.

In addition, by easily acquiring information about restaurants located nearby based on restaurant information for a desired menu, it is possible to receive help in selecting a restaurant.

The effects of the present invention are not limited to the above-mentioned effects, and various effects may be included within the range apparent to those skilled in the art from the following description.

1 is a view showing a simplified restaurant information providing system according to an embodiment of the present invention.
2 is a block diagram showing the structure of a restaurant information providing system according to an embodiment of the present invention.
3 is a diagram in which the system for providing restaurant information according to an embodiment of the present invention is implemented in the form of an application of a mobile terminal.
4 is a flowchart illustrating a method for obtaining restaurant information according to an embodiment of the present invention.

Since the present invention can apply various transformations and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the specific embodiments of the present invention, and it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. In describing the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted.

Terms such as first, second, etc. may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

Terms used in the present invention are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Spatially relative terms such as below, beneath, lower, above, etc. will be used to easily describe the correlation between one component and other components, as shown in the drawings. can Spatially relative terms should be understood as terms including different directions of components when used in addition to the directions shown in the drawings. For example, when a component shown in the drawing is turned over, a component described as below (below, beneath) other components may be placed above (above, upper) other components. Accordingly, the example term below may include both downward and upward directions. Components may also be oriented in other orientations, and thus spatially relative terms may be interpreted according to orientation.

As used in the present invention, the expression indicating a part such as “part” or “part” means a device in which the component may include a specific function, software that may include a specific function, or a device that may include a specific function. And it means that it can represent a combination of software, but it is not necessarily limited to the expressed function, which is provided to help a more general understanding of the present invention, and a person with ordinary knowledge in the field to which the present invention belongs If it is, various modifications and variations are possible from these base materials.

Therefore, the spirit of the present invention should not be limited to the described embodiments, and not only the claims to be described later, but also all those with equivalent or equivalent modifications to the claims will be said to belong to the scope of the spirit of the present invention. .

Hereinafter, a restaurant information providing system according to a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a view showing a simplified restaurant information providing system according to an embodiment of the present invention.

The system for providing restaurant information in an embodiment of the present invention is to obtain information of a restaurant while a user is moving on foot or in a boarding situation, and is composed of a mobile terminal 10, a server 20, and a signboard 30. In addition, the mobile terminal 10 may include a camera 13 , a GPS 11 , and a geomagnetic sensor 13 .

The mobile terminal 10 in an embodiment of the present invention may mean a portable electronic device that a user can move and operate, for example, a tablet capable of performing an operation similar to that of the mobile terminal 10 . , a laptop computer, a wearable device, and the like, and may include any wireless communication device capable of performing wireless communication 5 with the server 20 .

The mobile terminal 10 includes a display (not shown), and may provide various information to the user through the display (not shown).

Various information in an embodiment of the present invention includes the menu of the restaurant photographed by the user using the camera 13, the price of the menu, rating and evaluation information for each menu or the entire restaurant created by other users, and information captured by the user. It may include various information such as menus and price information for neighboring restaurants that provide a menu similar to the menu of the restaurant.

The similar menu may refer to a menu of a restaurant that divides the representative menu or food type of each restaurant into categories, and sells menus belonging to the same category within the divided category.

The camera 13 can be used to photograph the sign 30 of the restaurant. The mobile terminal 10 recognizes the letters, symbols, and the shape of the signboard based on the image taken by the camera 13 and uses it to classify the restaurant, and the signage photographed with the camera 13 is a rooftop sign, outdoor It may include all types of signboards used to advertise restaurants, such as signboards and standing signs.

The camera 13 may be photographed using a camera mounted on the mobile terminal 10 , but may be photographed using an external device in addition to the camera in the mobile terminal 10 . For example, all equipment capable of taking an image, such as a digital camera, DSLR, camcorder, etc. that is connected to the mobile terminal 10 and can transmit the photographed image to the server 20 can be utilized.

The GPS 11 is an abbreviation of a global positioning system, and may be a device capable of determining the latitude, longitude, and elevation (altitude above sea level) of the GPS device through communication with a satellite located in a satellite orbit of the earth. In general, all mobile terminals 10 sold in Korea include GPS, and in one embodiment of the present invention, the GPS included in the mobile terminal 10 can be used to detect the user's location. have.

The geomagnetic sensor 12 is a sensor for detecting the direction or movement of the mobile terminal 10 , and may refer to a sensor such as a Hall sensor included in the mobile terminal 10 .

The geomagnetic sensor 12 generally detects the direction or movement of the mobile terminal 10 by sensing the earth's magnetic field generated from the earth, and usually means a 3-axis geomagnetic sensor or a Hall sensor, but a barometric pressure sensor, an acceleration sensor, and a gyro. It may operate using an external sensor such as a scope sensor or a geomagnetic imaging sensor.

The mobile terminal 10 and the server 20 may transmit and receive data through the wireless communication 5 .

The server 20 may recognize a restaurant based on the image captured by the camera 13 or provide restaurant information related to the restaurant to the user.

Specifically, the server 20 obtains the image of the restaurant sign 30 photographed with the camera 13 through the network 5, and obtains the location information of the mobile terminal 10 using the GPS 11, , it is possible to obtain information about the shooting direction of the camera 13 by using the geomagnetic sensor 12 .

The server 20 may analyze the image of the restaurant sign 30 taken with the camera 13 to recognize the characters in the sign image, and recognize the restaurant using the recognized characters, location information, and direction information. .

An embodiment of the present invention discloses a system and method for recognizing a restaurant in the server 20, but is not limited thereto. In another embodiment, the mobile terminal 10 performs restaurant recognition, and the server 20 A function of searching for and outputting information stored in the database 25 may be provided according to a request of the mobile terminal 10 . That is, the restaurant information search system of the present invention has a purpose of recognizing a restaurant and providing restaurant information to the user, and the subject performing the operation of recognizing the restaurant is not limited to the mobile terminal 10 and the server 20. and may be operated together or changed as needed to perform the operation.

The database 25 may store information related to a restaurant. The information related to the restaurant may include information such as the name of the restaurant, the menu of the restaurant, the price for each menu, the location and contact information of the restaurant, star ratings of other users, evaluation information, and the location and contact information of the restaurant.

In addition, the database 25 may store the image of the signboard 30 including information related to the restaurant and model information of equipment used for photographing together. The image of the signboard 30 may be an image of the signboard 30 photographed by a user, or may be a single image, or may be a plurality of images photographed by a plurality of users. Such restaurant-related information may be stored by forming one table for each restaurant, or may be stored in the form of a table and used for retrieval. The restaurant may have a unique number and store related information, or it may be linked with the signboard 30 to connect the restaurant corresponding to the signboard 30 in a state where the signboard 30 has a unique number.

In addition, the database 25 may search/classify and store web page information including related information, such as menus related to restaurants, restaurant introduction review blogs, and recipe blogs introducing restaurant menus and similar menus. Through this, photo information of a restaurant selected by the user can be collected and provided to the user, or a cooking recipe that the user is interested in can be collected and used thereafter.

FIG. 2 is a block diagram showing the structure of the restaurant information providing system 100 embodying FIG. 1 .

The restaurant information providing system 100 may provide restaurant information to the user by using the information collected by the mobile terminal 10 .

Specifically, the restaurant information providing system 100 recognizes the restaurant photographed by the user using the information obtained from the camera 13, the GPS 11, and the geomagnetic sensor 12 of the mobile terminal 10, Based on the information, restaurant information may be fetched from the database 25 and provided to the user.

The location information obtaining unit 110 may recognize the location of the mobile terminal 10 . In general, the GPS 11 located inside the mobile terminal 10 may be used, and the user's latitude, longitude, and elevation information may be compared with a map through the GPS signal to determine the user's exact location.

In addition, the location information acquisition unit 110 may detect the movement of the user by periodically tracking the location of the user. The mobile terminal 10 detects the user's location in units of a predetermined time, and the detected location is transmitted to the location information acquisition unit 110 to detect the user's location, and the detected location information is linked to the map to move the user On the other hand, if the user does not detect a movement in a certain place, it may be determined that the user is taking an action while staying in a certain place.

The GPS 11 located inside the mobile terminal 10 is generally equipped with an A-GPS (Assisted GPS) developed by Qualcomm, and the A-GPS does not receive orbit information and time of the satellite from the satellite. It is a technology that greatly reduces the time it takes to set a location by receiving it through a network of

In an embodiment of the present invention, in order to minimize the error, the error can be reduced by repeatedly measuring the position several times.

The direction information obtaining unit 120 may recognize the direction of the camera 13 moved by the user to photograph the sign 30 of the restaurant by using the camera 13 .

Even if the location of the mobile terminal 10 is specified using the location information obtaining unit 110 , there is a limit to specifying the direction in which the user looks. In addition, in the case of a building, at least 2 to 3 or at most 10 or more restaurants may be located in one building, and in the case of a specific user, information on a restaurant located across the road may be requested. In order to solve these problems, the direction information obtaining unit 120 may recognize and use any one or more of a direction, a photographing angle, and a movement of the mobile terminal 10 in which the user uses the image obtaining unit 110 .

The shooting direction, shooting angle, and movement of the mobile terminal 10 may refer to all three-dimensional movements of the user's mobile terminal 10, and recognize movements in the x-axis, y-axis, and z-axis directions. It can operate in a combined form.

The direction information acquisition unit 120 may generally operate using the geomagnetic sensor 12 in the mobile terminal 10 , but a barometric pressure sensor, an acceleration sensor, a gyroscope sensor, a 3-axis/6-axis/9-axis degree of freedom sensor Any sensor capable of recognizing the direction, angle, and motion information of the mobile terminal 10, such as, may be utilized.

The image acquisition unit 110 may acquire an image of the restaurant sign 30 photographed with the camera 13 .

The sign 30 includes all kinds of billboards that can be used to advertise restaurants, such as a rooftop sign located at the top of the entrance to a shopping mall, an outdoor sign positioned perpendicular to the building, a standing sign positioned on a road or sidewalk, and a line signboard. can

The character recognition unit 140 may detect the characters in the signboard image by recognizing the characters in the sign image acquired by the image acquisition unit 110 .

In the character recognition method, only a character region is extracted by pre-processing a photographed image, and the extracted character region is applied to a neural network to recognize a character on a signboard.

The pre-processing process includes the steps of converting a photographed image to grayscale, removing noise using a filter, etc., dividing the grayscale image to specify the position of the character, and setting the position of the specific character in the region of interest (Region Of Interest). Interesting, ROI)) and deleting the background image part excluding the region of interest can be specified, but the present invention is not limited thereto and may include any process of converting the text region to be easily recognized by the neural network.

The neural network may be a convolutional neural network (CNN), and the CNN may be a neural network model trained in advance using a Mnist dataset.

The Mnist dataset may mean a training dataset configured based on data of the MNIST database (Modified National Institute of Standards and Technology database), and is a database used to learn various image processing systems. In an embodiment of the present invention, learning the CNN using the Mnist dataset is exemplified, but the present invention is not limited thereto, and learning can be performed using a training dataset for performing character recognition without distinction.

In an embodiment of the present invention, the neural network discloses a method using a convolutional neural network (CNN), but a support vector machine (SVM), k-Nearest Neighbors (k-NN), and a random forest (Random). Forest), Artificial Neural Network (ANN), Recurrent Neural Network (RNN), all machine learning and deep learning that can perform character recognition are available.

The restaurant recognition unit 150 includes the character information recognized by the character recognition unit 140 , the location of the mobile terminal 10 recognized by the location information acquisition unit 110 , and the camera 13 recognized by the direction information acquisition unit 120 . ) direction and information on the restaurant stored in the information providing unit 160 can be received and analyzed to classify the restaurant.

Specifically, a restaurant list of candidate restaurants located within a preset distance may be generated based on the location information of the mobile terminal 10 recognized by the location information acquisition unit 110 . Alternatively, by using the location information of the mobile terminal 10 recognized by the location information obtaining unit 110 and the direction in which the camera 13 recognized by the direction information obtaining unit 120 is looking, the information providing unit 160 It is possible to create a restaurant list of candidate restaurants existing in the user's location and direction. By matching the restaurant list with the character data recognized by the character recognition unit 140 , a restaurant corresponding to the restaurant sign 30 photographed by the user may be recognized.

However, there may be cases in which the character recognition unit 140 cannot accurately recognize characters, and in this case, there may be a problem in that it is not possible to accurately provide restaurant information to the user. In the present invention, in order to solve this problem, an image matching unit (not shown) may be included separately from the character recognition unit 140, and the restaurant recognition unit 150 uses an image matching unit (not shown) to allow a user to use an image matching unit (not shown). The matching score is calculated by performing similarity matching between the signboard image of the restaurant photographed by the acquisition unit 110 and the signboard image of the candidate restaurant, and for restaurants with the highest matching score or restaurants including a matching score higher than a preset standard Restaurant information may be provided to the user.

The image matching unit (not shown) uses a model operating through learning such as neural network, deep learning, machine learning, or any general method of performing image matching, such as a matching method using pixel analysis, can be used. .

When the restaurant recognition unit 150 recognizes a restaurant, the restaurant recognition unit 150 may transmit the recognized restaurant to the information providing unit 160 .

The information providing unit 160 may request the database 25 for information related to the restaurant recognized by the restaurant recognition unit 150 .

The restaurant-related information may include the recognized restaurant menu, menu price, and menu evaluation information, and the evaluation information for each menu may be evaluation information such as scored evaluation points, texts or images written by users. have.

In addition, the information providing unit 160 requests additional information about nearby restaurants belonging to the same category by using the location information and menu information of the restaurants stored in the database 25 together to help the user with menu selection convenience. can

For example, if the menu of a restaurant photographed by the user is a fried rice house in which 'Teppan fried rice' is the main menu, the restaurant recognition unit 150 classifies the 'fried rice' category in the information providing unit 160 to the 'fried rice' category. Upon requesting the restaurant information within, the information providing unit 160 may search for stored information of nearby restaurants and provide restaurant information including 'fried rice' to the restaurant recognition unit 150 . The reference of the nearby restaurant may operate based on a preset standard or a user's setting range.

The restaurant evaluation unit 170 is a specific restaurant in a state where the location of the mobile terminal 10 collected by the location information acquisition unit 110 based on the location information measured using the GPS 11 does not move for more than a preset time. When moving after being located in the restaurant information providing system 100 may determine that the user has used a specific restaurant. When it is determined that the user stayed in the restaurant because no movement is detected for more than a preset time, the restaurant information providing system 100 may request the user to evaluate the restaurant.

For example, when a user moves to a specific restaurant after obtaining restaurant information using the restaurant information providing system 100, and after detecting that there is no movement for 20 minutes or more, comes out of a specific restaurant and moves 15 meters or more , the restaurant information providing system 100 may determine that the user ate at the restaurant.

When it is determined that the user has used the restaurant, a restaurant evaluation may be requested from the user using a push notification of the mobile terminal 10 .

The restaurant evaluation can be performed in various fields such as service quality, hygiene status, decoration, taste, and star rating of the restaurant in relation to the restaurant used. , asterisks, points such as points input, short-answer type or multiple-choice type, etc. can request evaluation from the user in various ways.

When the user's restaurant evaluation is input, the restaurant information providing system 100 may evaluate the positive level of the restaurant by performing text mining on the text input by the user.

The text mining refers to using natural language processing technology based on linguistics, statistics, machine learning, etc. to formulate semi-structured/unstructured text data, and to extract and process useful information by classifying and analyzing the structured text. As a technology for the purpose of doing this, in an embodiment of the present invention, an evaluation score can be generated through analysis of how positive the restaurant evaluation information input by the user is by performing user emotion analysis through text mining. have.

The evaluation score may be added or subtracted using various criteria such as the number of users of the corresponding restaurant, the number of evaluation text inputs, the evaluation text input ratio, and the ratio of negative and positive evaluations.

3 is a diagram in which the restaurant information providing system 100 according to an embodiment of the present invention is implemented in the form of an application of a mobile terminal, and may be a screen illustrating an operation of the information providing unit 160 in an embodiment of the present invention. .

Figure 3 (a) is the main screen of the application, using the two methods of 'Find a restaurant in my location' menu using the 'Find a restaurant by camera' menu, which is a restaurant search method using a camera, and 'Find a restaurant in my location', which is a restaurant search method using location information. A method for retrieving restaurant information may be provided.

3 (b) is an example of a screen when the 'Find a restaurant in my location' menu (hereinafter, location search) is selected from the menu of (a). When the user selects a location search, the location information obtaining unit ( 110) may search for the user's location using GPS, and request the information providing unit 160 for all restaurant information within a preset distance based on the user's location and provide the information to the user.

Referring to (b) of FIG. 3 , basically, it is possible to provide the user with a list of restaurants and map information centered on the user's current location. You can provide restaurant information.

The restaurant information may include information about the menu, price, evaluation, etc. for the restaurant, and when the user selects a restaurant menu among restaurant information and selects the 'View more nearby restaurants' menu at the bottom of the restaurant information part, as shown in FIG. As in (d), it is possible to provide the name of a nearby restaurant, similar menus, distance from the current location, prices of similar menus, and evaluation information on similar menus in the form of a table.

Alternatively, the user sets the user's location virtual by moving the map part at the bottom of the screen, and then searches the restaurant list based on the virtual location or searches the location or restaurant name using the Search menu at the top of the screen. You can search restaurant information.

Alternatively, although not shown on the drawing, the user selects a menu on one side of the map portion and converts the entire screen into a map form in a pop-up form, and then uses the enlarge/reduce function to set an exact virtual location and search for a restaurant can do.

Alternatively, by installing a filter at the top of the restaurant list, you can search for a restaurant by selecting only a specific field or menu such as Korean, Chinese, Western, or Japanese food.

Alternatively, you can press and hold the restaurant information or use the menu on one side of the restaurant information section to check the detailed information about the restaurant selected by the user in a pop-up or new window in full screen, or to obtain additional information such as a blog reviewing a restaurant or a photo. can

(c) of FIG. 3 is an example of a screen when the 'find a restaurant with a camera' menu (hereinafter, camera search) is selected from the menu of (a). When the user selects a camera search, the image acquisition unit 110 can be activated to request the user to photograph the signboard, and at the same time, the location information obtaining unit 110 and the direction information obtaining unit 120 may be activated to detect the user's location and direction.

When the user's signboard shooting is completed, the information providing unit 160 calls the information providing unit 160 to provide information on the restaurant in the image captured by the user by analyzing the image captured by the user, location and direction information together with the screen indicating that the user is being analyzed. .

Referring to FIG. 3C , a sign image taken by the user may be provided at the bottom, while character data (text) obtained by analyzing the user's image may be provided on the left side of the screen. To the right of the text, information about a restaurant photographed by the user may be provided.

The restaurant information may include information about the menu, price, evaluation, etc. of the restaurant, and when the user selects a restaurant menu among restaurant information and selects the 'More nearby restaurants' menu at the bottom of the restaurant information part, as shown in FIG. As in (d), it is possible to provide the name of a nearby restaurant, similar menus, distance from the current location, prices of similar menus, and evaluation information on similar menus in the form of a table.

Alternatively, when it is determined that the restaurant recognition is wrong, the user can directly input the restaurant name by pressing the Search menu at the top of the screen, or press the camera button to recognize the restaurant through re-photography.

Alternatively, you can press and hold the restaurant information or use the menu on one side of the restaurant information section to check the detailed information about the restaurant selected by the user in a pop-up or new window in full screen, or to obtain additional information such as a blog reviewing a restaurant or a photo. can

4 is a flowchart illustrating a method for obtaining restaurant information according to an embodiment of the present invention.

If the user stops or the moving speed decreases while moving while maintaining a certain position, such as holding the mobile terminal 10 in his pocket or in his hand, the location information acquisition unit 110 uses the GPS 11 to determine the user's The location can be confirmed (S110).

When the movement of the mobile terminal 10 is measured by the geomagnetic sensor 13 in a state in which the user stops moving or the movement speed is reduced, the direction information acquisition unit 120 determines how the camera 13 of the mobile terminal 10 looks. The shooting direction can be recognized (S120).

The user may acquire a signboard image for a restaurant for which specific information is curious by using the camera 13 of the mobile terminal 10 to take a photo of the signboard (S130).

The character recognition unit 140 may analyze the sign image photographed using the camera 13 to recognize the characters in the sign image (S140).

In the step S140, the character recognition method uses a neural network learned in advance using the Mnist data set for the region of interest after performing preprocessing to cut only the signboard part as a region of interest (ROI). Thus, character recognition can be performed.

The restaurant recognition unit 150 may compare the letters recognized by the letter recognition unit 140 with the information provided by the information providing unit 160 to determine whether there is a restaurant with the same business name (S150).

In order to determine a restaurant with the same business name, the recognized location and direction information of the mobile terminal 10 are used together to help make accurate judgment even when there are multiple restaurants with the same business name, such as a chain store. have.

Specifically, the restaurant recognition unit 150 acquires a list of restaurants within a preset distance from the information storage unit 230 based on the location information obtained from the location information acquisition unit 110, and the direction information acquisition unit 120 A candidate restaurant may be set in the restaurant list based on the direction information obtained from , and the restaurant may be recognized by matching the restaurant name of the candidate restaurant with the characters recognized by the character recognition unit 140 .

In step S140 , if a restaurant matching the recognized character exists in the information providing unit 160 , the restaurant information providing system brings the restaurant information and surrounding restaurant information of the corresponding restaurant to the user from the information providing unit 160 , It can be provided to the user (S190).

Alternatively, in step S140, if a restaurant matching the recognized character does not exist in the information providing unit 160, similarity matching is performed between the sign image photographed by the user and the sign image existing in the information providing unit 160 It can be done (S160).

Specifically, the restaurant recognition unit 150 calculates a matching score by performing similarity matching between the signboard image of the restaurant photographed by the user with the image acquisition unit 110 and the signboard image of the candidate restaurant generated in step S150, and matching It is possible to provide the user with restaurant information about the restaurant having the highest score or the restaurants having a matching score higher than or equal to a preset criterion.

In step S160, the image similarity matching method uses a model operating through learning such as neural network, deep learning, machine learning, or any general method of image matching, such as a matching method using pixel analysis, can be used. have.

In step S160, if an image with a signboard image and a predetermined similarity or higher exists in the candidate restaurant, the signboard image captured by the user may be added to the information providing unit 160 to be used as a source for matching signboard images thereafter (S180) ).

When the addition of the sign image to the information providing unit 160 is completed, the restaurant information providing system may provide the user with restaurant information and surrounding restaurant information of the corresponding restaurant from the information providing unit 160 (S190).

Alternatively, in the step S160, if the image of the signboard image and the predetermined similarity or higher does not exist in the information providing unit 160, the restaurant information providing system provides a message to the user that the restaurant recognition has failed, while the S120 step It is possible to return to and request a new orientation and an image for recognition (S170).

As described above, in an embodiment of the present invention, the restaurant information providing system can receive information such as a specific menu, price, and evaluation of a restaurant without entering the restaurant, and by receiving restaurant reviews from users who actually used it, You can avoid wasting unnecessary time searching for restaurants.

In addition, in an embodiment of the present invention, the system for providing restaurant information can help the user to easily select a restaurant by providing information on a restaurant nearby that sells similar menus together with information on a restaurant desired by the user.

Meanwhile, the system and method for providing restaurant information according to an embodiment of the present invention can be implemented as one module by at least one of software, firmware, and hardware, and the above-described embodiments of the present invention are programs that can be executed on a computer. It can be implemented in a general-purpose computer that operates the program using a writeable, computer-readable recording medium. The computer-readable recording medium is implemented in the form of a magnetic medium such as a ROM, a floppy disk, a hard disk, an optical medium such as a CD or DVD, and a carrier wave such as transmission through the Internet. In addition, the computer-readable recording medium is distributed in a computer system connected to a network, so that the computer-readable code can be stored and executed in a distributed manner.

And, the component or '~ unit' used in the embodiment of the present invention is a task, class, subroutine, process, object, execution thread, software such as a program or FPGA ( It may be implemented in hardware such as a field-programmable gate array or ASIC (application-specific integrated circuit), or may be implemented in a combination of software and hardware. The component or '~ unit' may be included in a computer-readable storage medium, or a part thereof may be distributed and distributed among a plurality of computers.

So far, with respect to the present invention, the preferred embodiments have been looked at. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments are to be considered in an illustrative rather than a restrictive sense. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

10: mobile terminal
11: GPS
12: geomagnetic sensor
13: camera
20: server
25: Database
30: sign
100: restaurant information providing system
110: location information acquisition unit
120: direction information acquisition unit
130: image acquisition unit
140: character recognition unit
150: restaurant recognition unit
160: information provision unit
170: restaurant evaluation department

Claims (15)

a location information obtaining unit for obtaining location information using GPS;
an image acquisition unit for acquiring an image of a signboard of a restaurant from a camera;
a direction information obtaining unit for recognizing a photographing direction of the camera using a geomagnetic sensor to obtain direction information;
a character recognition unit for recognizing characters in the signboard image;
a restaurant recognition unit for recognizing the restaurant by using the letters of the signboard image or the signboard image based on the location information and the direction information; and
an information providing unit providing restaurant information of the restaurant recognized by the restaurant recognition unit;
A restaurant information provision system comprising a. According to claim 1,
The restaurant information is
The restaurant information providing system comprising the menu of the restaurant, the price of the menu, the rating of the menu, information on the evaluation of the menu, and information on surrounding restaurants selling a menu of the same category as the menu. According to claim 1,
The system for providing restaurant information, characterized in that the character recognition unit recognizes characters using a convolutional neural network (CNN) trained using the Mnist dataset. According to claim 1,
The restaurant information providing system further comprising a restaurant evaluation unit that requests the user to evaluate the restaurant when it is determined that the user stayed at a specific restaurant for more than a preset time based on the location information. 5. The method of claim 4,
The restaurant evaluation unit,
When the evaluation of the restaurant is input, emotion analysis is performed through text mining, and the positive level of the restaurant is evaluated using the emotion analysis result. 5. The method of claim 4,
The restaurant evaluation unit,
Restaurant information providing system, characterized in that the more the evaluation of the restaurant or the number of visitors to the restaurant, the additional points are given to the restaurant evaluation. According to claim 1,
The restaurant recognition unit,
Candidate restaurants located within a predetermined range are extracted using the location information and direction information, a restaurant matching the letter is primarily recognized among the candidate restaurants, and a restaurant matching the signboard image using the direction information A restaurant information providing system, characterized in that secondary recognition. confirming the user's location using GPS;
recognizing a photographing direction of a camera using a geomagnetic sensor;
obtaining an image of a signboard of a restaurant using the camera;
recognizing the letters of the signboard in the signboard image;
recognizing a restaurant using any one of the letters of the signboard and the signboard image, the location information of the user, and direction information of the camera; and
providing restaurant information of the recognized restaurant to a user;
A method of providing restaurant information, including 9. The method of claim 8,
The restaurant information is
A method for providing restaurant information comprising the menu of the restaurant, the price of the menu, the rating of the menu, information on the evaluation of the menu, and information on surrounding restaurants selling a menu of the same category as the menu. 9. The method of claim 8,
The step of recognizing the letter is,
A method for providing restaurant information, characterized in that letters are recognized using a convolutional neural network (CNN) trained using the Mnist dataset. 9. The method of claim 8,
When it is determined that the user stayed at a specific restaurant for more than a preset time according to the location information, requesting the user to evaluate the restaurant. 12. The method of claim 11,
The step of requesting an evaluation of the restaurant is,
When the evaluation of the restaurant is input, emotion analysis is performed through text mining, and the positive degree of the restaurant is evaluated using the emotion analysis result. 12. The method of claim 11,
The step of requesting an evaluation of the restaurant is,
Restaurant information providing method, characterized in that the more the evaluation of the restaurant or the number of visitors to the restaurant, the additional points are given to the restaurant evaluation. 9. The method of claim 8,
The step of recognizing the restaurant is,
extracting candidate restaurants based on the location information and direction information;
recognizing a restaurant matching the letter among the candidate restaurants; and
recognizing a restaurant matching the signboard image by using the direction information;
A method of providing restaurant information comprising a. 15. The method of claim 14,
The step of recognizing a restaurant matching the signboard image is,
Restaurant information providing method, characterized in that the restaurant information is recognized through similarity matching using the sign image of the candidate restaurants and the sign image acquired with the camera.

Images