JP2016181198A - Computer program, information search system, and control method of the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To achieve an improved user interface that contributes to acceleration of translation processing and search processing in providing a search service.SOLUTION: A computer program according to the present invention causes a computer operated by a user to function as: an imaging part that acquires an image including a search target word; a storage part that stores the image; a display part that displays the image; a mark providing part that enables, on the displayed image, provision of a mark for a user to designate the search target word at, for example, a position corresponding to the search target word; and a recognition part that recognizes the search target word on the basis of the provided mark and acquires text data of the search target word.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a computer program, an information retrieval system, and a control method thereof.

  For example, when a foreign traveler or a foreigner who has just lived (hereinafter simply referred to as “foreigner”) eats out in a foreign country, the restaurant menu is written in the language of that country. In many cases, it is usually difficult to quickly and accurately understand the contents of dishes and the like described in the food and drink menu. In that case, foreigners once translate the contents of multiple (very large number of menus in some cases) menus into their native language using a dictionary, translation device (eg, Patent Document 1), web translation engine, portal site, etc. Then, it is necessary to select a desired menu. The same applies to foreigners who want to know the contents of matters written in a language different from their native language.

JP 2014-016759 A

  However, the work of converting a plurality of items into a native language using a dictionary, a translation device, a translation engine, etc. is very time consuming. In the first place, when a foreigner cannot read a language different from the native language, there is a high possibility that a dictionary, a translation device, a translation engine, etc. cannot be used. For example, if a foreigner unfamiliar with Japanese does not know how to read the food and drink menu written in Japanese (there are many complicated fonts and handwritten ones), the translation device or translation engine I can't even type.

  On the other hand, using the reading method by the conventionally known OCR (Optical Character Reader), the matter included in the captured image including characters is converted into text data, and the text data is converted to the web. A method of examining the contents of the translation engine or search engine by inputting it into a portal site having these functions is also conceivable. However, in this case, generally, after all text characters in the image are recognized and text data is acquired, the text corresponding to the target matter is selected, and then a translation engine or search engine is used. Will be. Therefore, since it is necessary to recognize the entire information included in the image at the time of OCR conversion, the amount of data increases and the usability is never good.

  Therefore, the present invention has been made in view of such circumstances, and when obtaining information on a search target word described in a language different from the native language of a foreigner, processing load and data in recognition of the search target word An object of the present invention is to provide a computer program, an information search system, and a control method thereof that can contribute to speeding up of the translation process and the search process by reducing the amount, and can realize a further improved user interface. To do.

  In order to solve the above problems, a computer program according to an aspect of the present invention provides a computer operated by a user, an imaging unit that acquires an image including a search target word, a storage unit that stores the image, and a display that displays the image In the displayed image, for example, a mark assigning unit that enables a user to add a mark designating a search target word to a position corresponding to the search target mark, and the given mark Based on this, the search target word is recognized and functions as a recognition unit that acquires text data of the search target word.

  A computer program according to an aspect of the present invention delivers a computer to a translation system by passing text data of the search target word, to a search system by passing a translation result by the translation system, and from the search system to the search target word. It may be configured to function as a translation search unit that acquires related information, and the display unit may display information related to the search target word acquired by the translation search unit.

  More specifically, for example, the recognition unit can be configured to perform a predetermined range of image processing using the mark as a reference in the image (an image including a search target word).

  Here, the type of the mark is not particularly limited, and may be, for example, an open figure (line having both ends) or a closed figure (endless circle, rectangle, circle, triangle, square, polygon, etc.).

  Further, the language of the search target word is not particularly limited. For example, the present invention is extremely useful when it is Japanese.

  An information search system according to an aspect of the present invention is a system that provides information related to a search target word to a user via a client device operated by the user, and obtains an image including the search target word. A display unit for displaying the image, a display unit for displaying the image, and a mark for designating a search word by the user in the image, for example, at a position corresponding to the search target mark. A mark giving unit that makes it possible, a recognition unit that recognizes a search target word based on the mark, obtains text data of the search target word, and delivers the text data of the search target word to the translation system, and translates to the search system Translation test that delivers the translation results by the system and obtains information related to the search target word from the search system And a section, the display unit displays the information related to the searched word acquired by the translation search unit.

  A control method for an information search system according to an aspect of the present invention is a method for controlling an information search system that provides information related to a search target word to a user via a client device operated by the user, An image including the search target word is acquired by the imaging unit, the image is stored by the storage unit, the image is displayed by the display unit, and the user designates the search target word in the image by the mark providing unit. For example, it is possible to assign a mark to be performed at a position corresponding to the search target mark, and the recognition unit recognizes the search target word based on the mark and obtains text data of the search target word, and translates it. The search unit delivers the text data of the search target word to the translation system, and the translation result from the translation system is sent to the search system. It can pass, and to obtain relevant information from the search system in the search word, the display unit displays information related to the acquired search target words by the translation search unit.

  According to the present invention, an image including a search target word is acquired and stored, and in the displayed image, the user assigns a mark to a position corresponding to the search target word, and based on the provided mark. The search target word can be recognized and the text data can be acquired. Therefore, character recognition by image processing of the entire image becomes unnecessary, and recognition of the search target word intended by the user can be performed with certainty. Therefore, the search target word written in a language different from the native language of the foreigner can be obtained. When obtaining information, it is possible to reduce the processing load and data amount in recognition of the search target word. In addition, this contributes to speeding up of the translation process and the search process, and can realize a further improved user interface.

1 is a system block diagram schematically showing a preferred embodiment of a network configuration relating to an information search system according to the present invention. FIG. It is a block diagram which shows roughly the structure of suitable one Embodiment of the server apparatus in the information search system by this invention. It is a block diagram which shows roughly the structure of one suitable embodiment of the client apparatus in the information search system by this invention. It is a flowchart which shows the flow of the basic process in the search service provided by the information search system by this invention. (A)-(E) are top views which show an example of the transition of the display screen in a client apparatus.

  Hereinafter, embodiments of the present invention will be described in detail. The following embodiments are examples for explaining the present invention, and are not intended to limit the present invention only to the embodiments. The present invention can be variously modified without departing from the gist thereof. Furthermore, those skilled in the art can employ embodiments in which the elements described below are replaced with equivalent ones, and such embodiments are also included in the scope of the present invention. Furthermore, positional relationships such as up, down, left, and right shown as needed are based on the display shown unless otherwise specified. Furthermore, various dimensional ratios in the drawings are not limited to the illustrated ratios. In the following, an embodiment in which the search word is Japanese will be described as an example for easy understanding. However, as described above, the present invention is not limited thereto.

  FIG. 1 is a system block diagram schematically showing a preferred embodiment of a network configuration relating to an information search system according to the present invention. The information search system 100 includes a server device 10 that provides a search service connected to one or a plurality of client devices 30 (shown in the drawing) via a network 20.

  The server device 10 is a network node having a part of a function for providing a search service. For example, the server device 10 is configured by a host computer having a high arithmetic processing capability, and a predetermined server program operates on the host computer, thereby It expresses the function. In the present embodiment, the server device 10 includes a translation system (such as a translation engine or a portal site having a translation function) that translates a search target word, which will be described later, and a search system (such as a portal site having a translation engine or a translation function). It is a device to provide. Here, the computer for configuring the server device 10 is not limited to the host computer, and may be configured by, for example, a general-purpose communication terminal device. Further, the host computer constituting the server device 10 is not necessarily one, and may be constituted by a plurality of sub computers distributed on the network 20.

  On the other hand, the network 20 includes, for example, a wired network (a short-range communication network (LAN), a wide-area communication network (WAN), a value-added communication network (VAN), etc.) and a wireless network (mobile communication network, satellite communication network, Bluetooth, WiFi (Wireless Fidelity), HSDPA (High Speed Downlink Packet Access), etc.). On the other hand, the client device 30 is a network node having a function of receiving a search service, and is configured by, for example, a general-purpose communication terminal device used by a user.

  In the information retrieval system 100 configured as described above, a request (query, request, etc.) signal to the server device 10 is output from the client device 30 operated by the user, and the request signal is transmitted via the network 20 to the server device 10. Sent to. The server apparatus 10 that has received the request signal from the client apparatus 30 returns a response signal corresponding to the request to the client apparatus 30, and the response signal is transmitted to the client apparatus 30 via the network 20. Part of the provision of the search service is realized by the exchange of these signals.

  In the present specification, as described above, a network node that provides a part of the search service is referred to as a “server apparatus” regardless of the size of the arithmetic processing capability, and a network node that receives the search service is referred to as a “client”. Referred to as an "apparatus". Of course, a gateway server for converting a communication protocol between the server apparatus 10 and the client apparatus 30 may be interposed.

  Furthermore, examples of the general-purpose communication terminal device constituting the server device 10 or the client device 30 include mobile phones including desktop personal computers, notebook personal computers, tablet personal computers, laptop personal computers, smartphones, and the like. More specifically, as mobile phones, PDC (Personal Digital Cellular), PCS (Personal Communication System), GSM (registered trademark) (Global System for Mobile communications), PHS (Personal Handy phone System), PDA (Personal Digital) A handheld portable terminal such as Assistant) can be exemplified. Furthermore, as a data communication system in these cellular phones, for example, W-CDMA (Wideband Code Division Multiple Access), CDMA-2000 (Code Division Multiple Access-2000), IMT-2000 (International Mobile Telecommunication-2000), Wibro ( Standards such as “Wireless Broadband Internet”.

  FIG. 2 is a block diagram schematically showing a configuration of a preferred embodiment of the server device in the information search system according to the present invention. As shown in the figure, the server device 10 includes a processor 11, a communication interface 12, and a storage resource 13. The processor 11 is composed of an arithmetic and logic unit for processing arithmetic operations, logical operations, bit operations and the like and various registers (program counter, data register, instruction register, general-purpose register, etc.), and is stored in the storage resource 13. The computer program 40 according to the invention is interpreted and executed, and responses to requests from one or more client devices 30 are returned. The communication interface 12 is a hardware module for connecting to the client device 30 via the network 20 and is, for example, a modulation / demodulation device such as an ISDN modem, an ADSL modem, a cable modem, an optical modem, or a soft modem.

  The computer program 40 is a program for performing a series of information retrieval processing (mainly translation and retrieval) in response to requests from a plurality of client devices 30, and is a main program (such as the server program described above) of the server device 10. And a plurality of software modules that are called and executed as appropriate during the operation. Each of the software modules is a subprogram modularized to execute a specific process, and is created using, for example, a procedure, a subroutine, a method, a function, and a data structure. The “module” indicates a unit that can be compiled only by that part (however, it is not limited to this).

  Specifically, the computer program 40 performs various control arithmetic processing included in the search service (for example, processing for recognition, translation, and search of a search target word), and processing results by the control module 41 In addition to an image module 42 that performs image generation and display processing for displaying the image on the client device 30, a module that performs communication processing with the network 20 and the like is provided.

  Further, the storage resource 13 includes parameters necessary for the control module 41 to perform various control calculation processes included in the search service (for example, parameters necessary for executing an algorithm used for recognition of a search target word), Calculation parameters 50 such as a layout and a parameter necessary for screen magnification operation when the image module 42 performs image generation and display processing are stored.

  The storage resource 13 is a logical device provided by a storage area of a physical device (for example, a computer-readable recording medium such as a disk drive or a semiconductor memory). A plurality of physical devices may be mapped to one logical device, or one physical device may be mapped to a plurality of logical devices. The storage resource 13 may store data, logs, and the like indicating the access history of each client device 30, past search target words, and the like.

  FIG. 3 is a block diagram schematically showing the configuration of a preferred embodiment of the client device in the information search system according to the present invention. The client device 30 includes a processor 31, an audio output device 32, a communication interface 33, a storage resource 34, an input device 35, a display device 36, and a camera 37.

  The processor 31 includes an arithmetic logic unit and various registers (program counter, data register, instruction register, general-purpose register, etc.), and a program 60 stored in the storage resource 34 (for example, according to the present invention installed in a smartphone) The application program for information retrieval) is interpreted and executed, a request is transmitted to the server device 10 according to the operation information input to the input device 35, and a response from the server device 10 is received. Specifically, the program 60 is an application program for connecting to the server device 10 and receiving a translation and search service. This application program can be distributed from the server device 10 through the network 20, for example.

  The storage resource 34 is a logical device provided by a storage area of a physical device (for example, a computer-readable recording medium such as a disk drive or a semiconductor memory), and an operating system program, a driver program, Also stores various data. Examples of the driver program include an input device driver program for controlling the input device 35 and an output device driver program for controlling the audio output device 32 and the display device 36. Examples of the various data include history data of a search service menu display, history data of a web page on which information on a search target word is posted, and the like.

  The audio output device 32 is, for example, a sound player that can reproduce general sound data. The communication interface 33 provides a connection interface with the server device 10 and is configured by a wireless communication interface or a wired communication interface. Furthermore, the input device 35 provides an interface for receiving an input operation from the user, and examples thereof include a touch panel, a keyboard, and a mouse. The display device 36 provides a user with an image display interface such as a search service screen, and examples thereof include an organic EL display, a liquid crystal display, and a CRT display. Further, the camera is for capturing a still image of a subject including the search target word.

  An example of a specific procedure in providing a search service using the information search system 100 according to the present invention configured as described above will be further described with reference to FIGS. FIG. 4 is a flowchart showing a basic processing flow in the search service provided by the information search system according to the present invention. FIGS. 5A to 5E show transitions of display screens in the client device at that time. It is a top view which shows an example of a process. In FIG. 4, on the left side of the vertical broken line, the screen display processing in the client device corresponding to the transition and processing of the display screen of FIG. 5 is shown in time series, and on the right side of the vertical broken line, The operation of the client device by the user is shown in time series.

  First, the user activates a program 60, which is an application program for executing and controlling the information retrieval system according to the present invention, from application programs installed in the client device 30 (step SU1). At this time, the program 60 is selected by, for example, tapping or pressing an icon or the like displayed on the display 61 of the display device 36 in the client device 30.

  When the program 60 is activated, an activation screen shown in FIG. 5A is displayed on the display device 36 (display unit) of the client device 30 (step SC1), and the storage resource 34 (storage unit). Various stored data are loaded (loading of such data may be unnecessary). When the data loading is completed, the camera screen shown in FIG. 5B is automatically displayed on the display 61 without any user operation (step SC2).

  Here, in this embodiment, as shown in FIG. 5B, a food and drink menu written in Japanese is displayed on the display 61 as a subject, and the camera button 63 for photographing the subject is also the subject. It is displayed so as to overlap the image. When the user depresses the camera button 63 and photographs an image including a word (search target word) that is recalled from the food / drink menu with the meaning (search target word) (step SU2), the photographed image is captured. Is stored in the storage resource 34 and then immediately displayed on the display 61. In a predetermined area of the display screen, a message 64 prompting the user to draw a guideline 64a (mark) is displayed below the search target word. (FIG. 5C). That is, the search target word designation screen is displayed on the display 61 in accordance with a command from the processor 31 (mark assigning unit) (step SC3).

  At this time, the user can draw the guide line 64a below the search target word in the image by tracing the surface of the display 61 with, for example, a finger or a stylus pen (step SU3). FIG. 5C shows a state in which “cheese” handwritten and horizontally written in katakana as a search target word is designated as a recognition target character, and a guideline 64a is drawn below the character.

  When the user finishes drawing the guideline 64a, the processor 31 (recognition unit) determines a predetermined range based on the guideline 64a as the region 64b to be recognized by image processing. Subsequently, the processor 31 (translation search unit) performs character recognition by image processing, machine learning, and language processing (recognition result correction), and recognizes that the search target word described in the region 64b is “cheese”. (Step SC4 so far).

  The content of character recognition processing by image processing at this time is not particularly limited. For example, an image recognition system application represented by general OCR technology can be applied, or data learning can be performed instead of OCR technology. A so-called deep learning technique using the above can also be applied. Character recognition including such deep learning technology can be composed of, for example, steps of image processing, machine learning, and language processing (recognition result correction), and the processing content in each of these steps is as follows, for example.

<Image processing>
-Preprocessing by pixel spatial filter and image binarization by NLBlack filter-Text block corresponding to user-specified auxiliary line (guideline 64a) using density distribution (in the example of FIG. 5C, "cheese" Part ”and“ Extracting Object Rectangle Using Cutout / Projection Distribution Information ”Character Rectangle Extraction by Object Rectangle Combination Based on Aspect Ratio Quantities <Machine Learning>
-Data space expansion of font images by adding distortion and noise-Unsupervised image image space learning using Convolutional Denosing AutoEncoder (Deep Learning)-Supervised learning of character type discriminators by Deep Convolutional Neural Network (Deep Learning)-Character type frequency distribution Document space adaptive correction of Softmax score using <Language processing (recognition result correction)>
・ Creation of N-gram frequency distribution specialized for food and drink menus using information on the web ・ Recognition score correction algorithm using sliding windows combined with N-gram ・ By greedy method using general word dictionary Final correction of recognition results using optimal path search

  Thus, in step SC4, the search target word “cheese” is recognized, and the text data is obtained. Next, the processor 31 performs a process of handing over the text data of “cheese” to a translation system that exists on the web or that operates stand-alone. Translated into “Cheese” (step SC5). Furthermore, the processor 31 performs a process of handing over the obtained translation word “Cheese” to a search system that exists on the web or that operates on a stand-alone basis, and by the search engine, the translation word “Cheese” that is the translation word of the search target word is processed. Relevant information is acquired (step SC6). Then, the processor 31 displays an image including information related to the search target word thus obtained on the display 61 (step SC7).

  FIG. 5D and FIG. 5E are examples of the screen of the display 61 displayed in step SC7. In FIG. 5D, for example, “cheese” that is the Japanese search target word 66j and “Cheese” that is the English translation word 66e are displayed at the top of the screen. Also, in the area 66 below them, English text 66t that is a text search result of “Cheese” is displayed. Further, an image 67 t that is an image search result of “Cheese” is displayed in a region 67 below the region 67.

  Further, a transition button 62a is displayed between the region 66 and the region 67 in FIG. 5D, and when the user presses the transition button 62a, a further search result for the search target word 66j is obtained. It is displayed on the display 61. FIG. 5E is an example of such a further search result display screen (for example, a web page), and a text header including a search target word 66j and / or a translation word 66e is displayed in an upper region 68 of the screen. 68t is displayed, and further related information 69t is displayed in a region 69 below the 68t.

  Further, a return button 62b and a home button 62c are also provided on each of the screens shown in FIGS. By pressing the return button 62b, the user can redisplay the previous screen at any time for viewing, and by pressing the home button 62c, the user can quickly return to the camera screen shown in FIG. You can also. Therefore, when the user wants to examine information related to another search target word, the return button 62b returns to the camera screen shown in FIG. 5B or the search target word designation screen shown in FIG. 5C. Alternatively, the home button 62c can be pressed to return to the camera screen shown in FIG. 5B, and the series of processing from steps SC2 and SC3 described above can be executed again. On the other hand, when the use of the search service is terminated, the application is terminated by a user operation (step SU4).

  According to the program 60, the information search system 100, and the control method thereof configured as described above, a screen (search for displaying an image of a subject including the search target word 66j taken by the user with the camera of the client device 30 on the display 61). In the target word designation screen; FIG. 5C), the user can designate the search target word 66j by drawing the guideline 64a (mark). Then, based on the guideline 64a given by the user, character recognition is performed by image processing of the area 64b that is a predetermined range (step SC4), so that character recognition by image processing of the entire photographed image becomes unnecessary, and the user Can be reliably recognized.

  As a result, when obtaining information (FIGS. 5D and 5E) related to the search target word 66j written in a language different from the native language of the foreigner, the processing load in recognition of the search target word 66j and The amount of data can be reduced. In addition, this contributes to speeding up of the translation process and the search process, and can realize a further improved user interface.

  When the search target word 66j is Japanese, there are horizontal writing, vertical writing, and further diagonal writing, and they are often mixed. On the other hand, according to the present invention, when the search target word 66j is designated, the user draws the guide line 64a along the search target word 66j, and then performs the image processing based on the guide line 64a. Define. Therefore, even when the search target word 66j is Japanese, the present invention can accurately recognize the search target word 66j regardless of the direction of writing.

  Note that, as described above, each of the above embodiments is an example for explaining the present invention, and is not intended to limit the present invention to the embodiment. The present invention can be variously modified without departing from the gist thereof. For example, those skilled in the art can replace the resources (hardware resources or software resources) described in the embodiments with equivalents, and such replacements are also included in the scope of the present invention.

  More specifically, the mark (designation auxiliary line) for designating the search target word may be an open figure having both ends as in the guideline 64a, but is not limited thereto, and is endless or endless such as a circle or a rectangle. It may be a closed figure. In any case, since the range for character recognition can be defined by the image processing based on such a mark, the advantageous effects of the present invention described above can be obtained. Furthermore, the search target word is not limited to the food and drink menu.

  According to the present invention, character recognition by image processing of the entire image is not necessary, and the search target word intended by the user can be reliably recognized. Thereby, when obtaining information on a search target word written in a language different from the native language of the foreigner, it is possible to reduce a processing load and a data amount in recognition of the search target word. As a result, the translation process and the search process can be speeded up, and a further improved user interface can be realized. Therefore, the present invention can be widely used for activities such as design, manufacture, provision, and sales of programs, systems, and methods in the field of providing information search technology and search services.

DESCRIPTION OF SYMBOLS 10 Server apparatus 11 Processor 12 Communication interface 13 Storage resource 20 Network 30 Client apparatus 31 Processor 32 Audio | voice output device 33 Communication interface 34 Storage resource 35 Input device 36 Display device 40 Computer program 41 Control module 42 Image module 50 Calculation parameter 60 Program (computer) program)
61 Display 62a Transition button 62b Back button 62c Home button 63 Camera button 64 Message 64a Guideline 64b Area 66 Area 66e Translation word 66j Search target word 66t English text 67 Area 67t Image 68 Area 68t Text header 69 Area 69t Related information 100 Information search system SC1 to SC7 Steps SU1 to SU4 Steps

Claims (7)

The computer that the user operates
An imaging unit for acquiring an image including a search target word;
A storage unit for storing the image;
A display unit for displaying the image;
In the image, a mark giving unit that enables the user to give a mark designating the search target word; and
A recognition unit that recognizes the search target word based on the mark and obtains text data of the search target word;
A computer program that functions as a computer program. The computer,
Passing the text data of the search target word to the translation system, passing the translation result by the translation system to the search system, and functioning as a translation search unit for acquiring information related to the search target word from the search system,
The display unit displays information related to the search target word acquired by the translation search unit;
The computer program according to claim 1. The recognizing unit performs image processing of a predetermined range based on the mark in the image;
The computer program according to claim 1. The mark is an open figure or a closed figure;
The computer program according to claim 1. The search target word is Japanese.
The computer program according to claim 1. An information search system for providing information related to a search target word to the user via a client device operated by the user,
An imaging unit for acquiring an image including the search target word;
A storage unit for storing the image;
A display unit for displaying the image;
In the image, a mark giving unit that enables the user to give a mark specifying the search target word;
A recognition unit for recognizing the search target word based on the mark and obtaining text data of the search target word;
A translation search unit that delivers text data of the search target word to a translation system, delivers a translation result by the translation system to the search system, and acquires information related to the search target word from the search system;
The display unit displays information related to the search target word acquired by the translation search unit.
Information retrieval system. A method for controlling an information search system that provides information related to a search target word to a user via a client device operated by the user,
The imaging unit acquires an image including the search target word,
The storage unit stores the image,
The display unit displays the image,
The mark assigning unit makes it possible for the user to attach a mark designating the search target word in the image,
The recognition unit recognizes the search target word based on the mark, and obtains text data of the search target word,
The translation search unit passes the text data of the search target word to the translation system, passes the translation result by the translation system to the search system, and acquires information related to the search target word from the search system,
Displaying information related to the search word acquired by the translation search unit by the display unit;
Control method of information retrieval system.