KR20220083223A - Method and system for providing deep learning-based menu - Google Patents

Abstract

A system for providing a menu board based on deep learning according to an embodiment of the present invention includes: at least one processor; and at least one memory; including, at least one application stored in the memory and executed by the at least one processor to provide a deep learning-based menu board, wherein the at least one application obtains text information in a menu board image, and Obtaining an object image representing a specific object other than text in a menu board image, mapping the obtained text information and the object image to pair, and performing language conversion on the text information to obtain translated text information and provides menu board contents reconstructed based on at least two or more of the text information, the translated text information, and the paired object image.

Description

Method and system for providing menu board based on deep learning

The present invention relates to a method and system for providing a menu board based on deep learning. More specifically, it relates to a method and system for providing a menu board based on deep learning that extracts text and images in the menu board based on deep learning, and provides a reconstructed menu board by matching the extracted text and image with each other .

Conventional character recognition technology has been focused on image generation and document processing through scanners, but recently, image acquisition through a mobile camera and character recognition of natural scene images have become more mobile with the development of mobile terminal environments. The combination of information collection and services through the

In addition, as information communication and traffic science technology develop and globalization progresses in recent years, it is frequently encountered with predetermined information (eg, menu boards) provided based on various languages.

Therefore, in the prior art, research on a translation technique for photographing predetermined information implemented in different languages through a camera and converting text in the photographed image to translate is being actively conducted.

However, in the prior art, when translating and providing predetermined information such as a menu board, by extracting and converting data centering on the text in the menu board and providing it, other information other than text (for example, images, etc.) are lost.

For example, if there is a consumer who wants to translate and check a menu board implemented in a different language into the native language, the consumer translates the text on the menu board and displays the menu along with information such as the menu name or price. I want to check the image (photo) information as well.

However, at the level of the prior art, technology development for providing newly reconstructed information in consideration of image information other than text is insufficient.

In addition, when checking predetermined information such as a menu board, it is not included in the information, but the technology to provide additional information related to the text in the information is also insufficient, so it is necessary to introduce a technology for this. have.

KR 10-2142238 B1

The present invention seeks to implement a deep learning-based menu board providing method and system that extracts text and images in the menu board based on deep learning, and provides a reconstructed menu board by matching the extracted text and image with each other .

In detail, the present invention intends to implement a deep learning-based menu board providing method and system for pairing and providing matching text and images by performing deep learning based on text and images in the menu board.

In addition, the present invention is based on deep learning that provides a predetermined image corresponding to the text based on deep learning when there is no image corresponding to the text when pairing the text and the image as described above. We want to implement a method for providing a menu board and its system.

In addition, the present invention intends to implement a method and system for providing a menu board based on deep learning that converts a specific language-based text extracted from a menu board based on deep learning and translates it into another predetermined language.

However, the technical problems to be achieved by the present invention and embodiments of the present invention are not limited to the technical problems described above, and other technical problems may exist.

A system for providing a menu board based on deep learning according to an embodiment of the present invention includes: at least one processor; and at least one memory; including, at least one application stored in the memory and executed by the at least one processor to provide a deep learning-based menu board, wherein the at least one application obtains text information in a menu board image, and Obtaining an object image representing a specific object other than text in a menu board image, mapping the obtained text information and the object image to pair, and performing language conversion on the text information to obtain translated text information and provides menu board contents reconstructed based on at least two or more of the text information, the translated text information, and the paired object image.

The text information includes menu name information indicating a menu name among texts in the menu image, price text information indicating a menu price among texts in the menu image, and menu name information among texts in the menu image and at least one of side text information indicating text other than the price text information, wherein the side text information includes menu additional description text information indicating a description of the menu and a business name indicating a business store name providing the menu board It includes at least one or more of text information.

Also, when there is no object image mapped to the text information, the application detects a recommendation image matching the text information and performs pairing with the text information.

In addition, the recommendation image is an object image matching the menu name information among a plurality of object image data in the memory.

In addition, the application detects the recommended image by performing deep learning to input the plurality of object image data stored in the database module in the memory and the menu name information, or matching each of the plurality of object image data The recommended image is detected based on the tag word and the menu name information.

In addition, the application detects the recommended image by performing a search based on at least one of the menu name information and the side text information on a social network service (SNS) linked with the memory.

In addition, the application obtains the translated text information by converting the text information based on a target language according to a user input.

In addition, the application includes a first module for extracting feature information about the text in the menu image based on a deep learning neural network, and initial encoding information having location information about the text based on the extracted feature information A second module for generating a second module, a third module for obtaining an attention map maintaining two-dimensional information on the input image by performing a shape transformation based on the generated initial encoding information; Text information in the menu board image is acquired based on a fourth module that detects text in the menu board image through a learning model.

Meanwhile, a method for providing a menu board based on deep learning according to an embodiment of the present invention is a method for providing a menu board based on deep learning by an application executed in a computing device, the method comprising: acquiring text information in a menu board image; obtaining an object image representing a specific object other than text in the menu image; mapping the obtained text information and the object image to pairing; obtaining translated text information by performing language conversion on the text information; and providing menu board contents reconstructed based on at least two or more of the text information, the translated text information, and the paired object image.

A method and system for providing a menu board based on deep learning according to an embodiment of the present invention provide a reconstructed menu board by extracting text and an image in the menu board based on deep learning, and matching the extracted text and image with each other By doing so, there is an effect that the existing menu board can be created and provided in a form according to the user's requirements.

In addition, the method and system for providing a menu board based on deep learning according to an embodiment of the present invention perform deep learning based on text and images extracted from the menu board, and provide matching text and image by pairing By doing so, when reconfiguring and providing the menu board in the future, such as translating, it is possible to provide not only text information such as menu name or price, but also an image representing the menu, thereby minimizing the loss of other information other than text. can have an effect.

In addition, in the deep learning-based menu board providing method and system according to an embodiment of the present invention, when an image corresponding to the text is absent when pairing text with an image, the text is based on deep learning By performing the pairing by acquiring a predetermined image corresponding to It has the effect of making the contents of the plate more accurate and intuitive, and at the same time improving its usability.

In addition, the method and system for providing a menu board based on deep learning according to an embodiment of the present invention convert a specific language-based text extracted from the menu board based on deep learning and translate it into another language and provide it, Even if it is a menu board made in an unfamiliar language, it has the effect of making it easy to check and use the contents of the menu board.

However, the effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned can be clearly understood from the description below.

1 is a conceptual diagram illustrating a state in which a deep learning-based menu board providing service is provided from a consumer's point of view according to an embodiment of the present invention.
2 is a conceptual diagram illustrating a state in which a deep learning-based menu board providing service is provided from the perspective of a business owner according to an embodiment of the present invention.
3 is an internal block diagram of a computing device according to an embodiment of the present invention.
4 is a conceptual diagram illustrating the functional operation of a deep learning-based optical character recognition (OCR) module according to an embodiment of the present invention.
5 is an internal block diagram of an optical character recognition (OCR) module according to an embodiment of the present invention.
6 is an example of a diagram illustrating an internal operation method of the first module (CNN-based feature extraction unit) according to an embodiment of the present invention.
7 is an example of a diagram illustrating an internal operation method of a second module (CNN-based alignment unit) according to an embodiment of the present invention.
8 is an example of a diagram for explaining a method of combining dimensional data in a first module and a second module according to an embodiment of the present invention.
9 is an example of a view for explaining a method of performing the dimensional combining of the first method based on the dimensional data in the first module and the second module according to an embodiment of the present invention.
10 is an example of a state in which artifacts of encoding information (a second attention map) are minimized based on a third module (a first and second module combining unit) according to an embodiment of the present invention.
11 is an example of a diagram for explaining a method of performing a second method of dimensional combining based on dimensional data in a first module and a second module according to an embodiment of the present invention.
12 is an example of a diagram for explaining an internal operation method of a fourth module (CNN-based character recognition unit) according to an embodiment of the present invention.
13 is a flowchart illustrating a method for providing a menu board based on deep learning according to an embodiment of the present invention.
14 is an example of an image of a menu board according to an embodiment of the present invention.
15 is an example of a diagram for explaining a method of performing pairing according to an embodiment of the present invention.
16 is an example of a diagram for explaining a method of providing a recommendation image and performing pairing according to an embodiment of the present invention.
17 is an example of a state of displaying menu board contents according to an embodiment of the present invention.

Since the present invention can apply various transformations and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. Effects and features of the present invention, and a method for achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various forms. In the following embodiments, terms such as first, second, etc. are used for the purpose of distinguishing one component from another, not in a limiting sense. Also, the singular expression includes the plural expression unless the context clearly dictates otherwise. In addition, terms such as include or have means that the features or components described in the specification are present, and do not preclude the possibility that one or more other features or components will be added. In addition, in the drawings, the size of the components may be exaggerated or reduced for convenience of description. For example, since the size and thickness of each component shown in the drawings are arbitrarily indicated for convenience of description, the present invention is not necessarily limited to the illustrated bar.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, and when described with reference to the drawings, the same or corresponding components are given the same reference numerals, and the overlapping description thereof will be omitted. .

A deep learning-based menu board providing system (hereinafter, a menu board providing system) according to an embodiment of the present invention extracts text and images in the menu board, and matches the extracted text and image to create a new menu board reconstructed It is possible to provide a deep learning-based menu board service (hereinafter, integrated menu board recognition service).

In detail, in the embodiment, the menu board providing system extracts text and images from an image (menu board image) of a menu board produced in a specific language based on the integrated menu board recognition service, and extracts text and images from the extracted text and image. It is possible to provide a pairing service that maps the text and the image corresponding to each other by performing deep learning based on the deep learning.

In addition, in an embodiment, the menu board providing system performs deep learning based on the text extracted from the menu board image based on the integrated menu board recognition service to convert the menu board of the specific language into a menu board based on another language. A menu board translation service that translates (translates) can be provided.

In addition, in an embodiment, the menu board providing system provides a predetermined recommended image corresponding to the text based on deep learning when an image corresponding to the text is absent when pairing the text and the image An image recommendation service may be provided.

Meanwhile, the menu board providing system according to the embodiment may provide various service platforms based on the integrated menu board recognition service according to users.

Here, the user according to the embodiment may include a consumer who wants to check the menu board and place an order, or a business owner who registers or provides the menu board.

1 is a conceptual diagram illustrating a state in which an integrated menu board recognition service is provided from a consumer's point of view according to an embodiment of the present invention.

In detail, referring to FIG. 1 , in the embodiment, the menu board providing system may provide a menu board translation service in which the text in the menu board is translated into another language when the user is a consumer.

Specifically, in an embodiment, the menu board providing system may acquire a user (here, consumer) input for executing a menu board translation service based on a user interface based on the integrated menu board recognition service.

Further, in an embodiment, when the menu board translation service is executed based on the obtained user input, the menu board providing system converts the language of the menu board based on the user interface and receives a target language input to be translated. can be obtained

That is, the menu board providing system may translate the text in the menu board into the target language obtained as above.

Also, in the embodiment, when the target language is set as described above, the menu board providing system may acquire a menu board image that is an image of a menu board to be translated.

Also, in an embodiment, the menu board providing system may perform deep learning to convert the text in the menu board image into the target language by using the obtained menu board image as input data.

And the menu board providing system creates menu board contents, which are newly reconstructed menu boards, based on the output data and/or other data (in the embodiment, object images, etc.) obtained based on the deep learning performed as described above. can provide

2 is a conceptual diagram illustrating a state in which a deep learning-based menu board providing service is provided from the perspective of a business owner according to an embodiment of the present invention.

In addition, referring to FIG. 2 , in the embodiment, the menu board providing system extracts text and images in the menu board and reconfigures them to match each other when the user is a business owner so that it can be registered or provided on a predetermined platform. It is possible to implement a menu board registration/providing service.

In more detail, in an embodiment, the menu board providing system may acquire an input from a user (here, a business owner) executing a menu board registration/providing service based on a user interface based on an integrated menu board recognition service.

Also, in the embodiment, when the menu board registration/provided service is executed based on the obtained user input, the menu board providing system may acquire a menu board image, which is an image of a menu board to be registered/provided.

In addition, in an embodiment, the menu board providing system extracts at least one text and at least one image in the menu board image by using the obtained menu board image as input data, and matches corresponding images for each extracted text. Deep learning can be performed.

In addition, the menu board providing system may generate and provide menu board content, which is a reconfigured menu board, based on the deep learning performed as described above.

In an embodiment of the present invention, the menu board providing system for providing the integrated menu board recognition service as described above may be implemented based on a computing device.

Hereinafter, a computing device implementing the menu board providing system will be described in detail with reference to the accompanying drawings.

- Computing Device (200)

3 is an internal block diagram of a computing device 200 according to an embodiment of the present invention.

Referring to FIG. 3 , the computing device 200 according to an embodiment of the present invention may be a predetermined computing device in which a menu board application providing an integrated menu board recognition service is installed.

In detail, in terms of hardware, the computing device 200 may include a mobile type computing device and/or a desktop type computing device on which a menu board application is installed.

Here, the mobile type computing device may be a mobile device such as a smart phone or a tablet PC on which a menu board application is installed.

For example, the mobile type computing device may include a smart phone, a mobile phone, a digital broadcasting terminal, personal digital assistants (PDA), a portable multimedia player (PMP), a tablet PC, and the like.

In addition, the desktop-type computing device executes an integrated menu board recognition service based on wired/wireless communication, such as a fixed desktop PC, a laptop computer, and a personal computer such as an ultrabook in which a menu board application is installed. It may include a device in which a program is installed for the purpose.

Meanwhile, from a functional point of view, the computing device 200 may be implemented by including a plurality of modules.

In detail, in an embodiment, the computing device 200 may include a main module 210 , a sub-module 220 , a database module 230 , a communication module 240 , a memory 250 , and a processor 260 . .

<Main Module (210)>

In an embodiment, the main module 210 of the computing device 200 includes an optical character recognition module 100 that detects text in a menu image, and an image that detects an image (in the embodiment, an object image) in a menu image. A detection module 212 may be included.

One) Optical Character Recognition (OCR) Module; 100

The optical character recognition module 100 according to an embodiment of the present invention may detect text in a menu board image by performing deep learning.

In an embodiment, the optical character recognition module 100 may acquire at least one or more menu name information and at least one or more price text information by detecting text in the menu board image.

In addition, in the embodiment, the optical character recognition module 100 detects text in the menu board image, and side text information based on text other than the menu name and price text information (in the embodiment, menu additional description text information and/or business name text) information, etc.) can be obtained.

In the following embodiments, the optical character recognition module 100 is described as being implemented based on a plurality of modules (in the embodiment, first to fourth modules), but depending on the embodiment, Mask Text Spotter, CRAFT and/or It may be implemented using a known algorithm, such as may be implemented based on a deep learning-based character recognizer such as DAN, etc. In the embodiment of the present invention, the algorithm itself for detecting text in the menu image is not limited or limited. . However, in terms of the accuracy of optical character recognition, it would be the most preferable embodiment to implement the optical character recognition module 100 based on a plurality of modules (first to fourth modules) according to an embodiment of the present invention.

4 is a conceptual diagram illustrating the functional operation of the deep learning-based optical character recognition module 100 according to an embodiment of the present invention, and FIG. 5 is an internal block diagram of the optical character recognition module 100 according to an embodiment of the present invention. to be.

4 and 5, in an embodiment of the present invention, the optical character recognition module 100 is based on a plurality of deep learning modules that perform different functions, based on the characteristics and location information of the text in the input image. To provide a deep learning-based optical character recognition (OCR) service (hereinafter, character recognition service) that obtains encoding information (in the embodiment, attention map) and performs character recognition in the input image based on the obtained encoding information can

In detail, in an embodiment, the optical character recognition module 100 includes at least one encoder deep learning server 110, at least one encoding schema conversion server 120 and at least one decoder deep learning server as a plurality of deep learning modules ( 130) may be included.

At this time, in an embodiment of the present invention, the optical character recognition module 100 includes at least one encoder deep learning server 110 , at least one encoding schema conversion server 120 , and at least one decoder deep learning server 130 . However, according to the embodiment, at least one of the encoder deep learning server 110 , the encoding schema conversion server 120 and the decoder deep learning server 130 may be implemented as a separate device. Yes it may be possible too.

In an embodiment, the deep learning module as above may be implemented based on a deep-learning neural network.

Here, the deep learning neural network may include a deep learning neural network such as a convolutional neural network (CNN, Convolution Neural Network, for example, U-net Convolution Neural Network), The embodiment of the present invention does not limit or limit the deep learning neural network topology itself.

In addition, each of the deep learning modules as described above in the embodiment may be learned based on a predetermined training data set (Training Data Set).

For example, the deep learning module may be learned according to a predetermined method based on a training data set implemented as an image data set obtained by scanning a plurality of documents including various types of characters.

In this case, the predetermined method may include Hebbian Learning (Hebbian Rule), Perceptron Rule, Gradient Descent (Delta Rule, 　 Least Mean Square) and/or Back propagation method, and in an embodiment of the present invention, a deep learning neural It does not limit or limit the learning method of the network itself.

Coming back, in the embodiment, the optical character recognition module 100 including the deep learning module as described above works with a character recognition deep learning neural network implemented based on the learned deep learning neural network of each of the plurality of deep learning modules. can

In addition, in the embodiment, the optical character recognition module 100, based on the deep learning performed based on the interlocking deep learning neural network, the text in the input image to perform optical character recognition (OCR) (atypical text (eg, , curved text, etc.)) and location information can be extracted.

Also, the optical character recognition module 100 may acquire, based on deep learning, initial encoding information based on the information extracted as above, that is, a first attention map.

Also, in an embodiment, the optical character recognition module 100 may perform shape transformation on the obtained initial encoding information (first attention map) based on deep learning.

That is, the optical character recognition module 100 may perform encoding schema conversion based on initial encoding information.

In addition, the optical character recognition module 100 may obtain final encoding information to be input to the decoder, ie, a second attention map, through the shape conversion performed as described above.

In addition, in the embodiment, the optical character recognition module 100 includes characters (atypical characters (eg, curved characters, etc.) in the input image based on the obtained encoding information (second attention map) based on deep learning) can do) recognition can be performed.

In detail, the optical character recognition module 100 may perform optical character recognition (OCR) for detecting characters in an input image by performing decoding based on encoding information based on deep learning.

Hereinafter, each of the deep learning modules of the optical character recognition module 100 will be described in detail with reference to the accompanying drawings.

<Encoder Deep Learning Server>

In an embodiment of the present invention, the encoder deep learning server 110 interworks with a deep learning neural network, and features information and location information of text in the input image (which may include unstructured text (eg, curved text, etc.)) can be extracted.

Also, in an embodiment, the encoder deep learning server 110 may acquire initial encoding information, that is, a first attention map, based on the extracted feature information and location information.

In detail, in an embodiment, the encoder deep learning server 110 may be implemented based on an encoder deep learning neural network (hereinafter, an encoder).

In this case, the encoder according to the embodiment may include a CNN-based feature extraction unit 111 as a first module (hereinafter, feature extraction unit) and a CNN-based alignment unit 112 as a second module (hereinafter, an alignment unit). .

6 is an example of a diagram illustrating an internal operation method of the first module (CNN-based feature extraction unit 111) according to an embodiment of the present invention.

Referring to FIG. 6 , specifically, the feature extraction unit 111 (first module) of the encoder may extract feature information (FPN_Features) for text in an input image based on a deep learning neural network.

Here, the feature information according to the embodiment is outputted as a height (height, H), a width (width, W) and the number of output channels (C) when the input image is passed through the convolutional layer block means data.

For example, the feature extraction unit 111 (first module) may perform deep learning to extract feature information about text in an input image using a resnet45-based deep learning neural network composed of a plurality of convolutional layers. have.

At this time, the description of the internal parameter design for implementing the deep learning neural network of the feature extraction unit 111 (the first module) in the embodiment is replaced with reference to FIG. 3 .

7 is an example of a diagram illustrating an internal operation method of the second module (CNN-based alignment unit 112) according to an embodiment of the present invention.

Meanwhile, referring to FIG. 7 , in the embodiment, the alignment unit 112 (second module) of the encoder includes data output from the feature extraction unit 111 (first module) based on the deep learning neural network (ie, feature information). ), initial encoding information having location information on text in the corresponding input image, that is, a first attention map may be obtained.

In detail, in the embodiment, the alignment unit 112 (second module) is based on the output data of the feature extraction unit 111 (first module), and the first feed for the convolution layers of steps 1 to 4 A feed forward process can be performed.

In this case, in the embodiment, the first to fourth convolutional layers may operate in a batch normalization method, and the initial setting value of the hyper-parameter in the batch normalization is ‘c 64; eps 1e-05; momentum 0.1'.

Also, an activation function applied to each of the first to fourth convolutional layers may be a rectified linear unit (ReLU).

In addition, the first to fourth convolutional layers are 'k 3x3; s 2x2; p 1x1; It can be configured like c 64'.

As described above, when the first feed forward process for the convolutional layers of steps 1 to 4 is performed based on the output data FPN_Features of the feature extraction unit 111 (the first module), sorting in the embodiment The unit 112 (second module) may perform a second feed forward process based on the first to fourth steps of deconvolutional layers.

In detail, in the embodiment, the sorting unit 112 (second module) provides a second feed to the deconvolution layers of the first to fourth steps based on the output data from the first to fourth convolutional layers. A forward process can be performed.

In this case, in the embodiment, the first to fourth deconvolution layers may operate in a batch normalization method, and the initial setting value of the hyper-parameter in the batch normalization is ‘c 64; eps 1e-05; momentum 0.1'.

In addition, the first to third-stage deconvolution layers among the first to fourth-stage deconvolution layers may use a rectified linear unit (ReLU) as an activation function, and the fourth-stage deconvolution layer, A sigmoid function can be used as an activation function.

In addition, the first to fourth deconvolution layers are, 'k 3x3; s 1x1; It can be configured like c 64'.

In addition, in the embodiment, the first to fourth deconvolution layers may perform deep learning based on bilinear interpolate, and the scale value at this time may be set to '2'. .

In addition, in the first to fourth deconvolution layers, padding layers may be set to 'reflectionPad2d: 1'.

That is, in the embodiment, the alignment unit 112 (second module) uses the output data (FPN_Features) from the feature extraction unit 111 (first module) as input data, and a deep learning neural network that can be implemented as described above. As a basis, deep learning based on the first to fourth steps of convolution and deconvolution layers may be performed, and through this, initial encoding information (first attention map) for text in the corresponding input image may be obtained.

8 is an example of a diagram for explaining a method of combining dimensional data in a first module and a second module according to an embodiment of the present invention, and FIG. 9 is a first module and a second module according to an embodiment of the present invention. It is an example of a drawing for explaining a method of performing the dimensional combination of the first method based on the dimensional data in the module.

Meanwhile, referring to FIGS. 8 and 9 , in the embodiment, the encoder, based on the dimensional data in the feature extraction unit 111 (first module) and the dimensional data in the alignment unit 112 (second module), One way of dimensional combining can be performed.

In detail, in the embodiment, the feature extraction unit 111 (the first module) performs deep learning to determine the dimension of the input image as '1 * Number of output channels (C) * Height (Height, H) * Width (Width, W)' can be printed.

That is, the feature extraction unit 111 (the first module) may obtain the first dimension information by converting the dimension of the input data (in the embodiment, the input image, etc.) into '1*C*H*W'.

On the other hand, in the embodiment, the alignment unit 112 (second module) performs deep learning to determine the dimension of the input image as 'Max number of channels (T) * 1 * Height (Height, H) * Width (Width, W)' can be printed.

That is, the alignment unit 112 (second module) may obtain the second dimension information by changing the dimension of the input data (in the embodiment, feature information, etc.) to 'T*1*H*W'.

In addition, in the embodiment, the encoder including the feature extracting unit 111 (first module) and the alignment unit 112: second module is an encoder based on the first and second dimension information obtained as above. In 1) height (Height, H) and width (Width, W) information dimensional combination can be performed.

In detail, in the embodiment, the encoder includes height (H) and width (W) information based on '1*C*H*W' as the first dimension information and 'T*1*H*W' as the second dimension information 3D information implemented as 'T*C*(HxW)' may be obtained by performing dimensional combining.

In this case, the encoder according to the embodiment may further perform 2) dimensional combining for '(HxW)'.

Thus, in the embodiment, the encoder can finally obtain fourth-dimensional information in the form of compressed height (H) information of '(C*T)' after feature extraction.

That is, in the embodiment, the encoder may perform encoding that converts two-dimensional (2D) information based on existing 'H*W' into one-dimensional (1D).

<Encoding schema conversion server>

In an embodiment of the present invention, the encoding schema conversion server 120 cooperates with the deep learning neural network, and the shape for the initial encoding information (the first attention map) obtained from the sorting unit 112 (second module) conversion can be performed.

That is, in the embodiment, the encoding schema conversion server 120 may perform encoding schema conversion based on initial encoding information.

In addition, the encoding schema conversion server 120 may obtain final encoding information to be input to the decoder, ie, a second attention map, based on the form conversion performed as described above.

In an embodiment of the present invention, based on data (in the embodiment, feature information) output from the feature extraction unit 111 (first module), the alignment unit 112 (second module) for text in the corresponding input image When extracting encoding information having location information, if a general deconvolution method is used, certain artifacts may occur.

As described above, a deep learning neural network (ie, the fourth module 131 in the embodiment) performs decoding using encoding information having predetermined artifacts (eg, predetermined distortion, noise, etc. in the second attention map) as input data. In this case, it may be affected by noise and/or distortion including objects other than text due to artifacts throughout the input data, which in turn results in decoding, that is, optical character recognition (OCR). This may result in poor quality of the results.

10 is a view of minimizing artifacts of encoding information (a second attention map) based on a third module (first and second module combining unit 121) according to an embodiment of the present invention; it's ele

Thus, referring to Figure 10, in the embodiment of the present invention, the encoding schema conversion server 120, from the initial encoding information obtained from the feature extraction unit (111: the first module) and the alignment unit (112: the second module) In order to minimize the artifacts of , encoding schema transformation may be performed using a deconvolution block implementing a separate encoding process, and through this, the decoder (in the embodiment, the fourth module 131) ) to be input to the final form of encoding information (ie, the second attention map) may be generated.

In detail, in an embodiment, the encoding schema conversion server 120 may be implemented based on the encoding schema conversion deep learning neural network.

In this case, in the embodiment, the encoding schema transformation deep learning neural network may include a first and second module combining unit 121 (hereinafter referred to as a combining unit) that is a third module.

In more detail, the combiner 121 (third module) according to the embodiment is based on the deep learning neural network, and the data output from the aligner 112 (second module) (that is, the first attention map in the embodiment) is A separate encoding process can be performed based on

In general, in the case of atypical text that is implemented with characteristics such as curved in an input image, it is important to maintain two-dimensional information based on height (Height, H) and width (Width, W) as much as possible.

However, in the dimensional combination that implements the initial encoding information (the first attention map) in the encoder as described above, encoding is performed so that the two-dimensional (2D) information based on the existing 'H*W' is converted into one-dimensional (1D). , in this case, a loss may occur for each of the height (H) and width (W) individual information.

11 is an example of a diagram for explaining a method of performing a second method of dimensional combining based on dimensional data in a first module and a second module according to an embodiment of the present invention.

Thus, referring to FIG. 11 , in the embodiment of the present invention, the combining unit 121 (third module) is an aligning unit 112: second module after feature extraction of the feature extracting unit 111: first module in the encoder. The first attention map, in which H information of (C*T) is output in a compressed form, is (C*(H*W) )) can be implemented in the form

In the embodiment of the present invention, the description is based on that the coupling unit 121 (third module) is implemented as a device separate from the encoder, but this is only an example. ), including the aligning unit (112: second module) when the operation of the coupling unit (121: the third module) can be operated including the functional operation, etc., various embodiments are also possible.

Specifically, in the embodiment of the present invention, the combining unit 121 (third module) includes the first dimension information of the above-described feature extracting unit 111 (first module) and the second of the aligning unit 112 (second module). Based on the dimension information, the second method of dimensional combining may be performed.

In more detail, in the embodiment, the combining unit 121 (third module) includes '1*C*H*W', which is the first dimension information of the feature extracting unit 111: the first module, and the aligning unit 112: second Based on 'T*1*H*W', which is the second dimension information of the module), 1) The maximum number of channels (Max number of channels, T) and the number of output channels (Number of output channels, C) channel dimensional combination can be done

That is, in the embodiment, the combining unit 121 (third module) may perform T and C channel dimensional combining based on '1*C*H*W' and 'T*1*H*W', and this 3D information implemented as '(TxC)*HxW' can be obtained through this.

After obtaining the third dimension information as described above, in an embodiment, the combiner 121 (third module) may perform 2) dimension combination of height (Height, H) and width (Width, W) information.

In detail, in the embodiment, the combining unit 121 (third module) may perform dimensional combining of height (H) and width (W) information based on the third dimension information implemented as '(TxC)*HxW' and , through this, fourth-dimensional information implemented as '(TxC)*(HxW)' can be obtained.

Subsequently, the combiner 121 (third module) that has obtained the fourth dimension information may perform a dimensionality reduction process using 3) a 1x1 convolution layer (1x1 convolution layer) based on the obtained fourth dimension information. have.

Specifically, in the embodiment, the combining unit 121 (third module) performs a dimensionality reduction process for the fourth dimension information implemented as '(TxC)*(HxW)' based on a 1x1 convolution layer (1x1 conv layer). can be done by

In addition, the combiner 121 (third module) may acquire fifth dimension information implemented as 'C*(HxW)' by performing the dimension reduction process.

That is, in the embodiment, the combining unit 121 (third module) includes the first dimension information (1*C*H*W) obtained from the feature extracting unit 111: the first module and the aligning unit 112: second By performing the second method of dimensional combining based on the second dimension information (T*1*H*W) obtained from the module), the two-dimensional (2D) information based on the existing 'H*W' Dimensional combining can be performed.

In summary, in the embodiment of the present invention, the combiner 121 (third module) is a second attention map in a form that minimizes the loss of two-dimensional information about the input image, unlike the shape of the first attention map. (encoding information) can be created.

In addition, in the embodiment, the combiner 121 (third module) maintains two-dimensional information (height and width information) in the input image as much as possible, and at the same time, the second attention map, which is a shape matching the input data shape of the decoder. (encoding information) can be created.

Thus, the combiner 121 (third module) may cause the corresponding second attention map (encoding information) to operate as input data of the decoder (ie, the fourth module 131), and through this, 2 in the input image Decoding may be performed based on input data in a form in which dimensional information is maximally maintained.

As such, in the embodiment, the combiner 121 (third module) provides encoding information (second attention map) that minimizes the loss of two-dimensional information in the input image, thereby providing artifacts (eg, second attention) in the encoding information. It is possible to minimize certain distortion and/or noise in the map), and through this, it is possible to improve the deep learning-based optical character recognition (OCR) performance of text including unstructured text (eg, curved text, etc.) have.

<Decoder Deep Learning Server>

In an embodiment of the present invention, the decoder deep learning server 130 cooperates with the deep learning neural network, and based on the encoding information (second attention map) obtained from the combiner 121 (third module) Characters in the input image (which may include irregular characters (eg, curved characters, etc.)) may be recognized.

That is, in the embodiment, the decoder deep learning server 130 may perform optical character recognition (OCR) for detecting characters in an input image by performing decoding based on encoding information based on deep learning.

In detail, in an embodiment, the decoder deep learning server 130 may be implemented based on a decoder deep learning neural network (hereinafter, a decoder).

In this case, the decoder according to the embodiment may include a CCN-based character recognition unit 131 (hereinafter referred to as a character recognition unit) that is a fourth module.

12 is an example of a diagram for explaining an internal operation method of the fourth module (CNN-based character recognition unit 131) according to an embodiment of the present invention.

In more detail, referring to FIG. 12 , in the embodiment, the character recognition unit 131 (fourth module) includes encoding information (second attention map) output from the combiner 121 (third module) based on a deep learning neural network. ) to detect characters in the input image.

In this case, the character recognition unit 131 (fourth module) according to the embodiment may be implemented based on a decoder of a transformer model.

In detail, if a general attention method is used in the decoding process, the entire input data from the encoder (and/or the combiner 121: the third module) at every time step at which the decoder predicts the output character It should be noted once again.

In this way, the decoder does not refer to the entire input data at the same rate, but rather concentrates on the part of the input data that is related to the character to be predicted at the time. There may be problems with receiving

Therefore, in recent years, the transformer model method that minimizes the influence of the previous state is mainly used.

For reference, in the transformer model (transformer recognizer) method, there is an advantage that decoding is performed based on self-attention, which is performed on the self-attention, so that the influence of the previous state can be minimized.

Hereinafter, for a detailed description of the transformer model, ‘Vaswani, A shish, et al. "Attentioni's all you need." Advances in neural information processing systems. 2017.’ is replaced with the thesis.

In this case, the transformer model in the thesis includes an encoder and a decoder structure in the corresponding transformer model, and both the encoder and the decoder are implemented in a similar structure.

However, the character recognition based on the above structure is inefficient in extracting the positional information of the text in the input image, and the accuracy thereof may be deteriorated.

Thus, in the embodiment of the present invention, the character recognition unit 131 (fourth module) can be implemented using only the decoder of the transformer model, and the encoder ( and/or input the second attention map (encoding information) obtained from the coupling unit 121 (the third module) into the character recognition unit 131 (the fourth module) to perform deep learning-based optical character recognition (OCR). can be done

Here, the description of the internal design for implementing the deep learning neural network of the character recognition unit 131 (fourth module) according to the embodiment is replaced with reference to FIG. 9 .

That is, in the embodiment, the character recognition unit 131 (fourth module) may be implemented as shown in FIG. 9 based on the decoder of the transformer model, and the character recognition unit 131 (fourth module) implemented as above includes an encoder and / Alternatively, deep learning may be performed using the encoding information (the second attention map) output from the combiner 121 (third module) as input data.

In addition, the character recognition unit 131 (fourth module) may perform optical character recognition (OCR) for detecting characters in a corresponding input image by performing the deep learning.

Also, the character recognition unit 131 (the fourth module) may output and provide the detected character data in a predetermined manner (eg, display output, etc.).

As described above, in the embodiment of the present invention, the character recognition unit 131 (fourth module) is implemented based on the decoder of the transformer model that minimizes the influence of the previous state based on self-attention, so that the encoder according to the embodiment of the present invention And/or it is possible to implement a deep learning-based optical character recognition (OCR) model that combines the advantages of the combiner 121 (third module) and the decoder of the known transformer model.

In addition, as described above, in the embodiment of the present invention, an encoder for outputting initial encoding information (first attention map) by quickly and accurately extracting characteristics and position information of text in an input image, and initial encoding information output from the encoder From the combiner 121 (third module) that converts and generates encoding information (second attention map) in a form that minimizes artifacts in the initial encoding information, and the combiner 121: third module A character recognition deep learning neural network including a transformer decoder-based character recognition unit 131 (fourth module) that detects real text (characters) in the input image with high accuracy and speed by using the output encoding information as input data By providing, it is possible to further improve the performance of optical character recognition (OCR) for text in the input image (which may include unstructured text (eg, curved text, etc.)).

2) Image Detection Module (212)

The image detection module 212 according to an embodiment of the present invention may detect an object image in the menu board image based on deep learning.

Here, the object image according to the embodiment may be an image representing the shape of a specific object other than text in the menu board image.

For example, the image detection module 212, the first food picture, the second food picture, the third food picture in the menu board image, ... , it is possible to detect the nth food picture as an object image.

In an embodiment, the processor 260 of the computing device 200 includes the object image detected from the menu board image by the image detection module 212 as above, and the menu board image by the optical character recognition module 100 described above. It is possible to pair the detected text from . A detailed description of this will be described in the method for providing a menu board based on deep learning to be described later.

In an embodiment, such an image detection module 212 may be implemented based on a deep learning-based image detector such as YoLo v4 and/or EfficientDet, etc. In an embodiment of the present invention, it detects an object image from a menu image. It does not limit or limit the algorithm itself.

<Sub Module (220)>

In an embodiment, the sub-module 220 of the computing device 200 detects a text translation module 221 that converts and translates text detected from the menu board image into a different language, and a predetermined image corresponding to the text. and may include an image recommendation module 222 to provide it.

One) Translation Module (221)

In an embodiment of the present invention, the character translation module 221 converts text information (in the embodiment, menu name information, price text information, and/or side text information, etc.) detected from the menu board image based on deep learning into different languages can be translated into

For example, the text translation module 221 may perform deep learning based on text information detected from a menu board image obtained by photographing a menu board made based on the first language (eg, English).

In addition, the text translation module 221 may translate the text information based on the first language into text information based on a heterogeneous second language (eg, Korean) based on the performed deep learning.

In an embodiment, the character translation module 221 as above may be implemented based on a deep learning-based language translator such as seq2seq, etc. In the embodiment of the present invention, the algorithm itself for translating the text in the menu image is limited or do not limit

2) Image Recommendation Module (222)

The image recommendation module 222 according to an embodiment of the present invention includes the object image detected by the processor 260 of the computing device 200 from the menu board image by the above-described image detection module 212, and the above-described optical character. When pairing the text (in the embodiment, menu name information) detected from the menu board image by the recognition module 100, if there is no object image mapped to the menu name, based on deep learning, the It is possible to obtain and provide a predetermined object image (in an embodiment, a recommended image) recommended to be paired.

In detail, in the embodiment, the image recommendation module 222 performs deep learning using, as input data, a plurality of object image sets stored and/or managed by the database module 230, and menu name information in the absence of a mapped object image. can be done

Also, the image recommendation module 222 may read at least one object image corresponding to the menu name from the database module 230 based on the deep learning performed as described above.

In addition, the image recommendation module 222 may provide the read object image as a recommendation image to be paired with the menu name.

In another embodiment, the image recommendation module 222 may interwork with the external computing device 200 (eg, a web server, etc.) to perform crawling based on the menu name information.

In addition, the image recommendation module 222 may perform deep learning using at least one image set obtained as a result of the crawling performed and the menu name information as input data.

In addition, the image recommendation module 222 may acquire at least one object image corresponding to the menu name based on the performed deep learning.

Also, the image recommendation module 222 may provide the above-obtained object image as a recommendation image to be paired with the menu name.

<Other Modules>

In an embodiment of the present invention, the computing device 200 may include a database module 230 , a communication module 240 , a memory 250 , and a processor 260 as other modules.

One) Database Module (230)

The database module 230 according to an embodiment of the present invention may store and manage various application programs, applications, commands and/or data for implementing an integrated menu board recognition service.

In an embodiment, the database module 230 may store and manage an unspecified number of object image sets, menu board images, text information, recommendation images, pairing information, translated text information, and/or menu board contents.

The database module 230 may be various storage devices such as ROM, RAM, EPROM, flash drive, hard drive, and the like, and web storage that performs a storage function of the database module 230 on the Internet (internet) ) may be

In addition, in the embodiment, the database module 230 is integrated with an external database (eg, a database of a social network service (SNS), etc.) that can provide at least a part of a set of unspecified plurality of object images. Various data related to the menu board recognition service may be provided.

2) Communication Module (240)

The communication module 240 according to an embodiment of the present invention may transmit/receive various data and/or information for providing an integrated menu board recognition service.

In an embodiment, the communication module 240 may communicate with the external computing device 200 (eg, an external computing device and/or a web server, etc.) to exchange data related to the integrated menu board recognition service.

The communication module 240 is a technology standard or communication method for mobile communication (eg, Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), High Speed Downlink Packet Access (HSDPA), HSUPA) (High Speed Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), etc.) can transmit and receive.

In addition, the communication module 240, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World) Interoperability for Microwave Access) can also transmit and receive wireless signals through wireless communication methods.

Also, the communication module 240 may transmit/receive a wireless signal using a short-range wireless communication method. For example, the communication module 240 is Bluetooth (Bluetooth™), RFID (Radio Frequency Identification), infrared communication (Infrared Data Association; IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi - At least one of Wireless-Fidelity (Fi), Wi-Fi Direct, and Wireless Universal Serial Bus (USB) technologies may be used to support short-distance communication.

3) Memory (250)

The memory 250 according to an embodiment of the present invention may store any one or more of an operating system (OS), various application programs, data, and commands for providing an integrated menu board recognition service.

In an embodiment, the memory 250 may be implemented including the database module 230 as described above.

Also, in the embodiment, the memory 250 as described above may include a program area and a data area.

Here, the program area according to the embodiment may be linked between an operating system (OS) for booting the computing device 200 and functional elements, and the data area is generated according to the use of the computing device 200 . data may be stored.

The memory 250 may be various storage devices such as ROM, RAM, EPROM, flash drive, hard drive, and the like, and may be a web storage that performs a storage function of the memory 250 on the Internet. may be Also, the memory 250 may be a recording medium in a form detachable from the computing device 200 .

4) Processor (260)

The processor 260 according to an embodiment of the present invention may control the overall operation of each of the above-described units in order to implement the integrated menu board recognition service.

The processor 260 may be a system-on-a-chip (SOC) suitable for the computing device 200 including a central processing unit (CPU) and/or a graphics processing unit (GPU), and an operating system stored in the memory 250 . (OS) and/or application programs may be executed, and each component mounted in the computing device 200 may be controlled.

Also, the processor 260 may internally communicate with each component through a system bus, and may include one or more predetermined bus structures including a local bus.

In addition, the processor 260, ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers (controllers) , micro-controllers, microprocessors, and other electrical units for performing other functions.

In an embodiment of the present invention, the computing device 200 including the components as described above includes at least one main module 210 , a sub-module 220 , a database module 230 and/or a communication module 240 . It may consist of, at least one or more processors 260 for data processing, and at least one or more memories 250 for storing various application programs, data and/or commands for providing an integrated menu board recognition service. can

- How to provide a menu board based on deep learning

Hereinafter, a method in which the menu board application executed on the consumer's computing device 200 provides a menu board based on deep learning will be described in detail with reference to the accompanying drawings.

13 is a flowchart illustrating a method for providing a menu board based on deep learning according to an embodiment of the present invention.

Referring to FIG. 13 , in an embodiment of the present invention, the menu board application may acquire text information in the menu board image. (S101)

14 is an example of an image of a menu board according to an embodiment of the present invention.

In detail, referring to FIG. 14 , the menu board application in the embodiment may acquire a menu board image that is an image of a menu board produced based on a predetermined language (eg, English, etc.).

In this case, in the embodiment, the menu board image includes menu information (menu name, menu price, menu photo and/or menu description information, etc.) and/or business store information (business name information, etc.) sold by the corresponding business store. can do.

Also, in the embodiment, the menu board application that has obtained the above menu board image may acquire text information by detecting text in the obtained menu board image based on deep learning.

Here, the text information according to the embodiment is information obtained based on text sensed by performing deep learning using a menu board image as input data, and includes at least one or more menu name information 10, price text information 20, and / Alternatively, the side text information 30 may be included.

In detail, in the embodiment, the menu name information 10 may include at least one text indicating a menu name among texts in the menu board image.

For example, the menu name information 10 includes kimchi stew (first menu name, 10-1), soybean paste stew (second menu name, 10-2), budae-jjigae (third menu name, 10-3), ... , the nth menu name may be included.

Also, according to an embodiment, the price text information 20 may include at least one text indicating a price for a menu among texts in the menu board image.

For example, the price text information 20 is 7,000 (first menu price, 20-1), 7,000 (second menu price, 20-2), 8,000 (third menu price, 20-3), ... , the nth menu price may be included.

In this case, in the embodiment, when the menu board application acquires the menu name information 10 and the price text information 20 from the menu board image, the corresponding menu name and price text information 20 may be matched to each other and detected.

For example, the menu board application detects as a pair by matching the price text information 20 having the detection order and/or coordinate values corresponding to the sequence and/or coordinate values in which the menu name information 10 is detected. can

In addition, in the embodiment, the side text information 30 is information obtained based on text other than the menu name and price text information 20, and in the embodiment, it may include menu additional description text information and/or business name text information. can

For example, the additional explanatory text information of the side text information 30 may include text describing a menu and/or an origin text, etc., and the business name text information of the side text information 30 includes a corresponding menu board. It may include the business name of the first business branch provided.

Also, in an embodiment, the menu board application may acquire the object image 40 in the menu board image. (S103)

Here, the object image 40 according to the embodiment may mean an image displaying the shape of a specific object other than text in the menu image.

In detail, in the embodiment, the menu board application may acquire at least one object image 40 , which is an image displaying the shape of a specific object other than text, from the menu board image based on deep learning.

For example, the menu board application performs deep learning based on the menu board image, the first food photo 40-1, the second food photo 40-2, and the third food representing the actual image of a specific menu. Photo, … , it is possible to obtain the nth food picture as an object image (40).

In this way, the menu board application in the embodiment translates the menu board later by extracting not only the text in the menu board but also the object image 40 (in the embodiment, food photos, etc.) matching the text. When reorganized and provided, it is possible to provide not only text information such as menu name and price, but also an image representing the menu, and through this, a menu board in a form that minimizes loss of other information other than text can be created and provided. can

Also, in an embodiment, the menu board application may pair the text information obtained as above with the object image 40 . (S105)

15 is an example of a diagram for explaining a method of performing pairing according to an embodiment of the present invention.

In detail, referring to FIG. 15 , in the embodiment, the menu board application may pair the menu name information 10 and the object image 40 in text information obtained from the menu board image by matching each other.

Specifically, in the embodiment, the menu board application may 1) perform deep learning-based pairing.

In detail, the menu board application uses at least one menu name information 10 and an object image 40 as input data, selects a specific object image 40 corresponding to a specific menu name, matches each other, and provides output data. Pairing can be performed based on deep learning neural networks.

In more detail, in the embodiment, the menu board application may input at least one or more menu name information 10 and at least one object image 40 to the mapping deep learning neural network.

In this case, the mapping deep learning neural network may perform deep learning by 1:1 matching the object image 40 corresponding to each of at least one or more input menu names.

In addition, the mapping deep learning neural network may provide output data in the form of matching each of at least one menu name and each of the object images 40 as a result of the performed deep learning.

Thereafter, the menu board application that has received the above output data from the mapping deep learning neural network may perform pairing between the menu name information 10 and the object image 40 based on the received output data.

For example, the menu board application includes menu name information 10 including kimchi stew (first menu name), soybean paste stew (second menu name), and budae stew (third menu name), and a photo of kimchi stew (first food photo). ), an object image 40 including a picture of soybean paste stew (the second food picture) and a picture of the baguette stew (the third picture of the food) may be input to the mapping deep learning neural network.

And the menu board application maps 'kimchi stew' menu name information 10 and 'kimchi stew photo' object image 40 from the mapping deep learning neural network, and 'doenjang stew' menu name information 10 and 'doenjang stew' It is possible to obtain output data provided by mapping the 'picture' object image 40 and mapping the 'budaejjigae' menu name information 10 and the 'budaejjigae photo' object image 40 .

Accordingly, the menu board application may perform pairing in which the object image 40 corresponding to each of at least one menu name in the menu board image is 1:1 based on the obtained output data.

As such, in the embodiment, the menu board application performs the pairing between the menu name and the object image 40 based on deep learning, so that the mapping between the plurality of menu names and the plurality of object images 40 in the menu board is performed quickly and conveniently can be done

In another embodiment, the menu board application may perform 2) pairing based on distance information.

Here, the distance information according to the embodiment includes a first coordinate value (eg, center point coordinates of a menu name text area, etc.) for an area in which the menu name information 10 is sensed in the menu board image, and an object image in the menu board image. The distance between the second coordinate values (eg, coordinates of the center point of the area of the object image 40 ) for the area in which ( 40 ) is detected may be indicated.

In detail, in the embodiment, the menu board application may calculate distance information between a first coordinate value for the specific menu name information 10 and a plurality of second coordinate values for each of the at least one object image 40 .

Also, the menu board application may detect distance information having a minimum value among a plurality of calculated distance information.

In addition, the menu board application may map the object image 40 having the second coordinate value corresponding to the detected distance information to the corresponding menu name information 10 .

Thus, the menu board application may perform pairing in which each of the menu name information 10 and a predetermined object image 40 are mapped.

As such, in the embodiment, the menu board application can perform pairing based on the distance between the menu name and the object image 40, so that the object image 40 does not exist at a position exactly matching the menu name according to the configuration of the menu board. Even if not, the object image 40 corresponding to the corresponding menu name can be determined with high accuracy and paired.

16 is an example of a diagram for explaining a method of providing a recommendation image and performing pairing according to an embodiment of the present invention.

At this time, referring to FIG. 16 , in the embodiment, the menu board application may acquire a recommendation image to be paired with the corresponding text information when the object image 40 mapped to the text information is absent, and Based on this, pairing may be performed.

In detail, in general, in the menu board image (ie, menu board), object images 40 (eg, food photos, etc.) corresponding to the menu name and representing the appearance of the menu may all exist for each menu name, but depending on the situation The object image 40 for all menu names in the menu board image may not exist.

For example, depending on the situation, the menu board image includes only the object image 40 for the main menu of the business branch among the plurality of menus in the menu board image, and the object image 40 for the remaining menus. may not be included.

As in the above situation, when the object image 40 mapped to the predetermined menu name information 10 is absent, the menu board application in the embodiment of the present invention provides a recommended image to be paired with the corresponding menu name information 10 . can do.

Here, the recommended image according to the embodiment may be a predetermined object image 40 determined to correspond to the menu name information 10 among a plurality of unspecified object images 40 existing outside the corresponding menu board image. have.

Also, in an embodiment, the menu board application may perform pairing by mapping the recommendation image to the corresponding menu name information 10 .

In detail, in the embodiment, the menu board application may perform pairing by 1) acquiring a recommendation image based on a database.

In more detail, the database module 230 according to an embodiment of the present invention may include an unspecified number of object images 40 set data.

In addition, the menu board application in the embodiment, as described above, based on the menu name information 10 in which the corresponding object image 40 is absent, and the object image 40 set data, the object image 40 set Among the data, it is possible to detect a recommendation image that is an object image 40 determined to match the menu name information 10 .

In this case, in an embodiment, the menu board application may detect a recommended image based on the set data of the menu name information 10 and the object image 40 based on a deep learning neural network.

Specifically, the menu board application may input the menu name information 10 and the object image 40 set data to the above-described mapping deep learning neural network.

Here, the mapping deep learning neural network may perform deep learning based on the input menu name information 10 and set data, and as a result, the object image 40 corresponding to the menu name information 10 is the It can be determined as a recommended image by extracting from the set data.

In addition, the mapping deep learning neural network may provide output data in the form of mapping the determined recommendation image and the menu name information 10 .

Thereafter, the menu board application receiving the output data from the mapping deep learning neural network, based on the received output data (that is, the information on which the menu name information 10 and the recommended image are mapped), a corresponding object image ( The recommendation image may be paired with the menu name information 10 in which 40) is absent.

Thus, in the embodiment, the menu board application retrieves the object image 40 corresponding to the corresponding menu name information 10 from the outside of the menu board image, even if the menu name information 10 does not have a corresponding object image 40 in the menu board image. It can be obtained and provided by matching.

For example, the menu board application, when the object image 40 corresponding to 'Budaejjigae (ie, menu name)' is absent, deep learning based on the text of 'Budaejjigae' and object image 40 in the database set data can be performed.

In addition, the menu board application may extract and obtain a recommendation image, which is an object image 40 determined to correspond to the 'Budaejjigae' text, from the set data through the deep learning performed.

In addition, the menu board application may perform pairing by mapping the above-obtained recommendation image to the 'Budaejjigae (ie, menu name)' text.

In another embodiment, each of the set data of the unspecified plurality of object images 40 in the database module 230 may include a tag word corresponding to each.

Here, the tag word according to the present embodiment may be set based on an object name (in the embodiment, a menu name) in the corresponding object image 40 .

For example, when a predetermined object image 40 in the object image 40 set data includes a 'Budaejjigae' object, the tag word corresponding to the predetermined object image 40 is set to 'Budaejjigae' can be

In addition, in the present embodiment, the menu name application is based on the menu name information 10 in which the corresponding object image 40 is absent and the tag word for each of the object image 40 set data, the menu name information 10 It is possible to extract a recommended image for the object image 40 from the set data.

In detail, in the present embodiment, the menu name application may extract an object image 40 having a tag word corresponding to the menu name information 10 from the object image 40 set data.

For example, in the menu name application, when the menu name information 10 is 'Budaejjigae', an object image having 'Budaejjigae' among a plurality of object images 40 in the object image 40 set data as a tag word ( 40) can be extracted.

Also, in the present embodiment, the menu name application may determine the object image 40 extracted as above as a recommended image to be paired with the menu name information 10 .

In addition, the menu name application may perform pairing by mapping the determined recommended image to the menu name information 10 .

Illustratively, the menu name application sets the object image 40 having a tag word corresponding to the text 'Budaejjigae' when the object image 40 corresponding to 'Budaejjigae (ie, menu name)' is absent. It can be extracted from data, and the extracted object image 40 can be determined as a recommended image to be paired with the menu name information 10, and the determined recommended image is mapped to the 'Budaejjigae (ie, menu name)' text for pairing can be performed.

As such, in the exemplary embodiment of the present invention, in the menu board application, if there is no object image 40 matching the menu name, an unspecified number of object images 40 other than the object image 40 obtained from the corresponding menu board image. By selecting one of them and mapping it to the corresponding menu name, it is possible to detect and provide a picture representing the corresponding menu even if there are no pictures of each menu on the existing menu board. It is possible to recognize the contents of the menu board more accurately and intuitively, and at the same time improve its usability.

In another embodiment, the menu board application may perform pairing by acquiring a recommendation image based on 2) a social network service (SNS).

In detail, the database module 230 according to the present embodiment may interwork with a server of a predetermined social network service (eg, Instagram and/or Facebook, etc.).

In addition, in the present embodiment, the menu board application is based on the social network service based on the menu name information 10 and/or the side text information 30 of the text information in which the corresponding object image 40 is absent as described above. Search (crawl) can be performed.

In more detail, in the present embodiment, the menu board application includes the menu name information 10 in which the corresponding object image 40 is absent and/or the business name text information in the side text information 30 corresponding to the corresponding menu name information 10. Based on this, a social network service-based search may be performed.

For example, the menu board application may perform a social network service-based search based on a search word including 'budaejjigae' and 'business name of the first business branch'.

In addition, in the present embodiment, the menu board application may perform the above search to detect and obtain an object image 40 matching the menu name information 10 and/or business name text information from the database of the social network service. have.

At this time, in the present embodiment, when a plurality of object images 40 matching the menu name information 10 and/or business name text information are detected, the menu board application uses the detected candidate object image ( 40) can be selected.

In an embodiment, the menu board application performs deep learning based on the plurality of candidate object images 40 detected as above, and the size of the background area excluding the object area in the corresponding candidate object image 40 has a minimum value. The candidate object image 40 may be selected as a recommendation image.

As another embodiment, the menu board application may select the candidate object image 40 having the maximum number of 'likes' among the plurality of candidate object images 40 detected as described above as the recommended image.

In addition, in the present embodiment, the menu board application may perform pairing by mapping the recommended image selected as above to the menu name information 10 .

As described above, in the present embodiment, the menu board application can determine a recommended image to be paired with a menu name in conjunction with a predetermined social network service, so that a recommended image can be determined by performing a search based on a large amount of big data and considering the latest trends. Through this, mutual correspondence and accuracy between the recommended image and the menu name can be further improved.

Returning to FIG. 13 again, in an embodiment, the menu board application may obtain the translated text information 50 by performing language conversion on the acquired text information as described above. (S107)

Here, the translated text information 50 according to the embodiment includes text information (in the embodiment, menu name information 10, price text information 20 and It may be information translated by converting the side text information 30, etc.) into a predetermined other language (eg, Korean).

In detail, in an embodiment, the menu board application may acquire the translated text information 50 based on the character translation module 221 that performs a language conversion process based on deep learning.

In this case, for example, the text translation module 221 may be implemented based on a deep learning-based language translator such as seq2seq, and in the embodiment of the present invention, the algorithm itself for translating text information in the menu image is limited. or do not limit

In more detail, the menu board application in the embodiment transmits menu name information 10, price text information 20 and/or side text information 30 based on a specific language (eg, English) to the character translation module 221 . can be entered.

At this time, the text translation module 221 performs deep learning based on the input data to obtain the text information (ie, menu name information 10, price text information 20 and/or side text information 30). It may be converted into another predetermined language (eg, Korean) and provided as output data.

Then, in the embodiment, the menu board application is based on the output data provided from the text translation module 221 , for the menu name information 10 , price text information 20 and/or side text information 30 . The translated text information 50 may be obtained.

In this case, in the embodiment, the translated text information 50 obtained as above may be paired with the object image 40 mapped to the text information corresponding to the corresponding translated text information 50 .

As such, in the embodiment, the menu board application can provide a menu board implemented in a specific language by translating it into another language, so that even if the menu board is made in a different language that the consumer is not familiar with, the contents of the menu board can be easily checked. and can make use of it.

17 is an example of a state of displaying menu board contents according to an embodiment of the present invention.

Also, referring to FIG. 17 , in an embodiment, the menu board application may generate and provide menu board content based on the information obtained as above. (S109)

That is, in the embodiment, the menu board application includes the text information obtained as described above, the translated text information 50 and/or the corresponding text information and the paired object image 40-1 for each menu name of the translated text information 50; 40-2, 40-3, …: 40), it is possible to create and provide menu board contents.

Here, the menu board content according to the embodiment refers to text information obtained from an existing menu board image, translated text information 50 and/or an object image 40 paired for each menu name of the text information and the translated text information 50 . ) may mean a menu board that is newly reconfigured and provided based on the

In an embodiment, the menu board application is newly created by reconstructing based on the translated text information 50 obtained from the menu board image obtained by photographing the menu board, and the object image 40 paired with the corresponding translated text information 50 . You can create a translated menu board.

In addition, the menu board application may provide the generated translated menu board as menu board content in a predetermined method (eg, display output, etc.).

According to an embodiment, the menu board application, on the existing menu board image, the translated text information 50 obtained from the corresponding menu board image, and the object image 40 paired to the translated text information 50 in augmented reality It can also be displayed by overlapping (AR).

As such, in the embodiment, the menu board application includes translated text information (that is, information translated from menu name and price text) and an object image 40 corresponding thereto (ie, food photos corresponding to the corresponding menu name), etc. By providing a newly created menu board by combining

As described above, the method and system for providing a menu board based on deep learning according to an embodiment of the present invention extract the text and image in the menu board based on deep learning, and match the extracted text and image to the menu board reconstructed By providing this, there is an effect that the existing menu board can be created and provided in a form according to the user's requirements.

In addition, the method and system for providing a menu board based on deep learning according to an embodiment of the present invention perform deep learning based on text and images extracted from the menu board, and provide matching text and image by pairing By doing so, when reconfiguring and providing the menu board in the future, such as translating, it is possible to provide not only text information such as the menu name or price, but also an image representing the menu, thereby minimizing the loss of other information other than text. can have an effect.

In addition, in the deep learning-based menu board providing method and system according to an embodiment of the present invention, when an image corresponding to the text is absent when pairing text with an image, the text is based on deep learning By acquiring a predetermined image corresponding to , and performing the pairing, an image representing the corresponding menu can be detected and provided together even if an image for each menu does not exist in the existing menu board, and the reconfigured menu is provided. It has the effect of making the contents of the plate more accurate and intuitive, and at the same time improving its usability.

In addition, the method and system for providing a menu board based on deep learning according to an embodiment of the present invention convert a specific language-based text extracted from the menu board based on deep learning and translate it into another language and provide it, Even if it is a menu board made in an unfamiliar language, it has the effect of making it easy to check and use the contents of the menu board.

In addition, the embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention or may be known and used by those skilled in the computer software field. Examples of the computer-readable recording medium include hard disks, magnetic media such as floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floppy disks. medium), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory 250, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. A hardware device may be converted into one or more software modules to perform processing in accordance with the present invention, and vice versa.

The specific implementations described in the present invention are only examples, and do not limit the scope of the present invention in any way. For brevity of the specification, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connection or connection members of lines between the components shown in the drawings illustratively represent functional connections and/or physical or circuit connections, and in an actual device, various functional connections, physical connections that are replaceable or additional may be referred to as connections, or circuit connections. In addition, unless there is a specific reference such as “essential” or “importantly”, it may not be a necessary component for the application of the present invention.

In addition, although the detailed description of the present invention has been described with reference to a preferred embodiment of the present invention, those skilled in the art or those having ordinary knowledge in the art will have the spirit of the present invention described in the claims to be described later. And it will be understood that various modifications and variations of the present invention can be made without departing from the technical scope. Accordingly, the technical scope of the present invention should not be limited to the contents described in the detailed description of the specification, but should be defined by the claims.

Claims (10)

at least one processor; and
at least one memory; including,
At least one application stored in the memory and executed by the at least one processor to provide a deep learning-based menu board, the at least one application comprising:
Obtain text information in the menu image,
Obtaining an object image representing a specific object other than text in the menu board image,
Pairing by mapping the obtained text information and the object image,
To obtain translated text information by performing language conversion on the text information,
providing menu board contents reconstructed based on at least two or more of the text information, the translated text information, and the paired object image
A system for providing menus based on deep learning. The method of claim 1,
The text information is
menu name information indicating the menu name among the texts in the menu image, price text information indicating the price of the menu among the texts in the menu image, and texts other than the menu name information and the price text information in the text in the menu image At least one of the side text information indicating
The side text information is
Comprising at least one or more of menu additional description text information indicating a description of the menu and business name text information indicating the name of a business store providing the menu board
A system for providing menus based on deep learning. 3. The method of claim 2,
The application is
When there is no object image mapped to the text information, detecting a recommendation image matching the text information and performing pairing with the text information
A system for providing menus based on deep learning. 4. The method of claim 3,
The recommended image is,
an object image matching the menu name information among a plurality of object image data in the memory.
A system for providing menus based on deep learning. 5. The method of claim 4,
The application is
Detecting the recommended image by performing deep learning with the plurality of object image data stored in the database module in the memory and the menu name information as inputs;
Detecting the recommended image based on a tag word matched to each of the plurality of object image data and the menu name information
A system for providing menus based on deep learning. 5. The method of claim 4,
The application is
Detecting the recommended image by performing a search based on at least one of the menu name information and the side text information on a social network service (SNS) linked to the memory
A system for providing menus based on deep learning. The method of claim 1,
The application is
Converting the text information based on a target language according to a user input to obtain the translated text information
A system for providing menus based on deep learning. The method of claim 1,
The application is
A first module for extracting feature information about the text in the menu board image based on a deep learning neural network;
a second module for generating initial encoding information having location information on the text based on the extracted feature information;
a third module for obtaining an attention map maintaining two-dimensional information on the input image by performing a shape transformation based on the generated initial encoding information;
Obtaining text information in the menu board image based on a fourth module for detecting text in the menu board image through a deep learning model based on the obtained attention map
A system for providing menus based on deep learning. As a method for providing a menu board based on deep learning by an application running on a computing device,
obtaining text information in the menu board image;
obtaining an object image representing a specific object other than text in the menu image;
mapping the obtained text information and the object image to pairing;
obtaining translated text information by performing language conversion on the text information; and
providing menu board contents reconstructed based on at least two or more of the text information, the translated text information, and the paired object image
How to provide a menu board based on deep learning. 10. The method of claim 9,
The application is
When there is no object image mapped to the text information, detecting a recommendation image matching the text information and performing pairing with the text information
How to provide a menu board based on deep learning.

Images