JP6408055B2 - Information processing apparatus, method, and program - Google Patents

Description

  Embodiments described herein relate generally to an information processing apparatus, a method, and a program.

  In recent years, for example, information processing apparatuses capable of recognizing a specific recognition target from an image have been developed. As an example of such an information processing apparatus, for example, there is a character recognition apparatus that recognizes a character from the image by capturing an image of a subject on which the character is described.

  Here, in order to recognize a character from the image captured as described above, a certain processing time is required in the character recognition device.

  However, for example, a captured image is only displayed on the screen of the character recognition device while such character recognition processing (hereinafter referred to as character recognition processing) is being performed. Until the result of the character recognition process is displayed on the screen, it is impossible to grasp whether or not the character recognition process is properly executed in the character recognition device.

  For example, when the result of a several character recognition result is displayed, the user's visibility with respect to the result may fall.

  Such a character recognition device is not user-friendly and is not convenient. For this reason, for example, it is desired to realize a display that improves user convenience with respect to image recognition for a recognition target including characters.

JP 2013-122747 A JP 2003-078640 A JP 2000-207487 A

  Therefore, the problem to be solved by the present invention is to provide an information processing apparatus, method, and program capable of realizing a display that improves user convenience with respect to image recognition.

The information processing apparatus according to the embodiment includes first recognition means, first display processing means, second recognition means, and second display processing means. The first recognition means recognizes a first recognition target included in the first image. The first display processing means displays feedback information indicating that the recognition process is being executed before the recognition process in the first recognition means is completed, and the first recognition target is displayed after the recognition process is completed. The recognition result of is displayed. The second recognition unit recognizes a second recognition target included in a second image different from the first image. The second display processing means displays feedback information indicating that the recognition process is being executed before the recognition process in the second recognition means is completed, and the second recognition target is displayed after the recognition process is completed. The recognition result of is displayed. The recognition result of the first recognition target and the recognition result of the second recognition target are displayed as a history of recognition results. When the recognition result of the first recognition target is selected from the recognition result of the first recognition target and the recognition result of the second recognition target displayed as the history, the first image and the first image are captured. When the recognized position is displayed and the recognition result of the second recognition target is selected, the second image and the position where the second image is captured are displayed.

The block diagram which shows an example of a structure of the character recognition apparatus which concerns on embodiment. The flowchart which shows an example of the process sequence of a character recognition apparatus. The figure which shows an example of the display screen when the image imaged with the camera is displayed. The figure which shows an example of the display screen in case a likelihood map image is displayed as Fordback information. The figure which shows an example of the display screen in case the frame line showing each of a character candidate area | region is displayed instead of the likelihood map image. The figure which shows an example of the display screen in case the frame line of a character string area | region is displayed as feedback information. The figure which shows an example of the display screen in case a character recognition result is displayed. The figure which shows an example of the display screen when a frame line is highlighted according to acquisition of a text. The figure which shows an example of the display screen when a frame line is highlighted according to acquisition of a text. The figure which shows an example of the display screen when the result that the character cannot be recognized is displayed. The figure which shows an example of the display screen in case the frame line containing the character string area | region which cannot recognize a character is displayed with the broken line. The flowchart which shows an example of the process sequence of the character recognition apparatus in the case of using a character recognition result for a search process. The figure which shows an example of the display screen when a mosaic process is performed with respect to the area | region used as recognition object. The figure which shows an example of the display screen when a specific character string is displayed on the area | region used as recognition object.

Hereinafter, embodiments will be described with reference to the drawings.
FIG. 1 is a block diagram illustrating an example of the configuration of the information processing apparatus according to the present embodiment. The information processing apparatus according to the present embodiment has a function capable of recognizing a recognition target from an image, and can be realized as a portable electronic device such as a smartphone or a tablet computer. In the following description, an information processing apparatus that recognizes characters as the above-described recognition targets (hereinafter referred to as a character recognition apparatus) will be mainly described.

  As shown in FIG. 1, the character recognition device 10 includes a camera 11, a storage unit 12, a processing unit 13, and a display unit 14.

  The camera 11 is mounted on the back surface of the character recognition device 10 such as the smartphone described above, and captures an image of a subject arranged around the user who holds the character recognition device 10. The subject includes, for example, a signboard in the city where characters (a character string made up of) are written. That is, in this embodiment, the camera 11 captures an image including a plurality of characters.

  The storage unit 12 is realized by using a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), a ROM (Read Only Memory), or a RAM (Random Access Memory) provided in the character recognition device 10. . The storage unit 12 can store, for example, an image captured by the camera 11.

  The processing unit 13 is realized by a computer (for example, a processor) provided in the character recognition device 10 that executes a program stored in the storage device. Note that the program executed by the computer includes an operating system and an application program for recognizing characters from an image (hereinafter referred to as a character recognition application).

  The processing unit 13 includes a first display processing unit 131, a character recognition processing unit 132, a second display processing unit 133, a third display processing unit 134, and a search processing unit 135. A part or all of each of the units 131 to 135 included in the processing unit 13 is realized, for example, by causing a computer to execute the character recognition application described above, that is, by software. Note that some or all of these units 131 to 135 may be realized by hardware such as an IC (Integrated Circuit) or may be realized as a combined configuration of software and hardware. The character recognition application to be executed by the computer may be stored in a computer-readable storage medium and distributed, or may be downloaded to the character recognition device 10 through a network.

  The first display processing unit 131 acquires an image captured by the camera 11. Note that the first display processing unit 131 may acquire an image stored in the storage unit 12, for example. In addition, the first display processing unit 131 may acquire an image from, for example, a server device external to the character recognition device 10. The first display processing unit 131 displays the acquired image on the display unit 14.

  The character recognition processing unit 132 executes processing for recognizing characters from the image (image captured by the camera 11) displayed by the first display processing unit 131 (hereinafter referred to as character recognition processing).

  The second display processing unit 133 starts the character recognition processing by the character recognition processing unit 132 until the result of the character recognition processing is acquired (that is, during execution of the character recognition processing). Information indicating that the recognition process is being executed (hereinafter referred to as feedback information) is displayed on the display unit 14. Note that the feedback information includes, for example, information indicating the position of the character included in the image, and is displayed before the character recognition process ends as described above.

  The third display processing unit 134 displays the result of the character recognition processing on the display unit 14 when the character recognition processing by the character recognition processing unit 132 is completed.

  The search processing unit 135 executes a search process using the result of the character recognition process displayed by the third display processing unit 134. Note that the result of the search processing by the search processing unit 135 is displayed on the display unit 14.

  The display unit 14 is realized using, for example, a touch panel display provided in the character recognition device 10.

  The touch panel display described above incorporates, for example, a capacitive touch panel that can detect the contact position of a finger (or pen) on the screen of the touch panel display. According to this, the user can operate the character recognition device 10 by bringing a finger into contact with (touching) the screen of the touch panel display.

  Next, an example of a processing procedure of the character recognition device 10 according to the present embodiment will be described with reference to the flowchart of FIG.

  2 is executed by the processing unit 13 when the character recognition application described above is activated in the character recognition device 10, for example.

  First, when a character recognition application is activated in the character recognition device 10, the first display processing unit 131 acquires an image captured by the camera 11. Here, if the character recognition device 10 is a smartphone or the like as described above, a button (for example, a shutter button) for the user to instruct the camera 11 to take an image is provided on the character recognition device 10. Is provided. In this case, the first display processing unit 131 acquires a still image captured by the camera 11 when the user presses the shutter button, for example. The first display processing unit 131 displays the image acquired in this way on the display unit 14 (step S1). It is assumed that the image acquired by the first display processing unit 131 is stored (accumulated) in the storage unit 12, for example. Here, a description will be given assuming that a still image captured by the camera 11 is acquired and displayed on the display unit 14, but a moving image captured by the camera 11 may be acquired and displayed on the display unit 14. .

  Next, the character recognition processing unit 132 executes character recognition processing to recognize the character (string) included in the image displayed in step S1 (step S2).

  Here, in the character recognition process, a process of detecting a partial area including a character string (hereinafter referred to as a character string area) from the image displayed in step S1 is executed. Next, a process for recognizing a character string included in the detected character string area is executed, and text representing the character string is acquired as a result of the character recognition process (hereinafter referred to as a character recognition result).

  Note that the character recognition processing unit 132 may perform translation processing on the text acquired as described above. According to this translation process, it is possible to obtain a result of text being translated into a preset language (that is, a translation of the text).

  That is, the character recognition result in the present embodiment may be a text representing a character string included in the above-described image (character string region), or may be a translated sentence obtained by translating the text.

  Further, the present embodiment includes a process for detecting a character string area from the above-described image and a process for recognizing a character string included in the detected character string area (a process for acquiring text representing the character string). Although the processing is mainly described as character recognition processing, a series of processing obtained by adding the above-described translation processing to the text may be used as the character recognition processing.

  Here, a certain processing time is required until a character recognition result is acquired in the character recognition processing described above. For this reason, the second display processing unit 133 is executing the character recognition processing while the character recognition processing by the character recognition processing unit 132 is being executed (that is, until the character recognition result is acquired). Feedback information representing the presence is displayed (step S3). The feedback information is information for feeding back the processing status related to character recognition to the user who uses the character recognition device 10.

  Specifically, the second display processing unit 133 uses, for example, a partial area (that is, an image) on the image that is estimated to include characters in the character recognition process (process for detecting a character string area) as feedback information. It is assumed that a region where there is a character above) is displayed. A specific example of feedback information displayed on the display unit 14 will be described later.

  When the process of step S3 is executed, it is determined whether or not the character recognition process by the character recognition processing unit 132 is completed (step S4).

  When it is determined that the character recognition process is completed (YES in step S4), the character recognition processing unit 132 acquires the character recognition result described above. The third display processing unit 134 displays the character recognition result acquired by the character recognition processing unit 132 on the display unit 14 (step S5). As described above, when the translation process is included in the character recognition process, it is determined that the character recognition process is completed when the translation process is completed (a translated sentence is acquired).

  On the other hand, if it is determined that the character recognition process has not ended (NO in step S4), the process returns to step S3 and the process is repeated. That is, in the process shown in FIG. 2, the display of the feedback information described above is maintained until the character recognition process is completed. The feedback information may be configured to be updated (that is, different feedback information is displayed) according to the progress (processing status) of the character recognition process, for example.

  Hereinafter, a transition example of a screen (hereinafter referred to as a display screen) displayed on the display unit 14 when the character recognition process is executed in the character recognition device 10 will be specifically described.

  As described above, when the character recognition application is activated in the character recognition device 10 and an image is captured by the camera 11, the image (including the display screen) is displayed on the display unit 14.

  Here, FIG. 3 shows an example of a display screen in a case where an image captured by the camera 11 (hereinafter referred to as a captured image) is displayed.

  As shown in FIG. 3, the display screen is provided with a first area 141 and a second area 142.

  The first area 141 is an area where a captured image (still image) is displayed. In the example illustrated in FIG. 3, the first region 141 displays a captured image 300 including a signboard on which characters (columns) of “XX station this right turn” are described.

  The second area 142 is an area in which, for example, a result (character recognition result) obtained by performing character recognition processing on the captured image displayed in the first area 141 is displayed. At a point before the character recognition process is executed, the second area 142 is blank as shown in FIG.

  Next, a character recognition process is performed on the image 300 shown in FIG. In this character recognition process, for example, as described above, a process of detecting a character string area, a process of acquiring text representing a character string included in the character string area, and the like are executed.

  Here, in the present embodiment, while the character recognition process described above is being executed, feedback information indicating that the character recognition process is being executed is displayed.

  Hereinafter, this feedback information will be specifically described. In the present embodiment, the feedback information includes, for example, a likelihood map image (information based on likelihood information) representing a degree that there is a character for each pixel constituting the captured image (hereinafter referred to as likelihood).

  In this case, the second display processing unit 133 extracts, for example, a plurality of character candidate areas from the captured image (original image), and adds, for example, 1 to the likelihood of each pixel included in each of the character candidate areas (that is, , Increase the likelihood of the pixel). In addition, a character candidate area | region means the partial area | region of the image in which the character in a captured image may exist, for example. The extraction of the character candidate area is performed by extracting a characteristic part such as an edge based on the luminance value of each pixel of the captured image and specifying an area having character-like characteristics. As described above, a plurality of character candidate areas are extracted, but the likelihood of a pixel that is extracted many times as a character candidate area is high, and the likelihood of a pixel that is extracted as a character candidate area is low. Lower. The 2nd display process part 133 produces | generates a likelihood map image by providing the color according to the likelihood of each pixel obtained by such a process to each said pixel. That is, in the present embodiment, the feedback information includes color information. The method of creating the likelihood map is an example, and the likelihood map may be created using another known technique.

  In the present embodiment, the likelihood map image generated by the second display processing unit 133 is displayed on the display unit 14 until, for example, the character string region is detected after the execution of the character recognition process is started. The

  Here, FIG. 4 shows an example of a display screen when a likelihood map image is displayed as feedback information.

  As shown in FIG. 4, the likelihood map image 400 is displayed on the captured image 300 displayed in the first region 141, for example. Note that an area 401 in the likelihood map image 400 illustrated in FIG. 4 indicates an area including pixels with low likelihood (that is, an area where there is a low possibility that a character exists). On the other hand, a region 402 in the likelihood map image 400 indicates a region including pixels with a high likelihood (that is, a region having a high possibility that a character exists).

  Specifically, in the likelihood map image 400, for example, a region 401 whose likelihood is lower than a predetermined value is represented by black (or gray or the like), and a region 402 whose likelihood is higher than a predetermined value. Shall be represented in white.

  Here, for example, the region 402 has been described as being represented by one color (white), but the region 402 may be represented by a plurality of colors (colors) according to the likelihood of each pixel, for example. Absent.

  According to this, the user who uses the character recognition device 10 is that the character recognition process is being executed in the character recognition device 10, and the region 402 is an area where the character recognition device 10 is estimated to have a character. It is possible to grasp the fact (that is, the part in the image that the character recognition device 10 recognizes as being a character).

  In FIG. 4, the likelihood map image 400 is described as being superimposed on the captured image 300. However, for example, when the character recognition process is started, the captured image 300 is not displayed, and the likelihood map image is displayed. It may be configured such that only 400 is displayed.

  Further, here, the likelihood map image 400 has been described as being displayed, but instead of the likelihood map image 400, for example, each of a plurality of character candidate regions (partial regions of the image) illustrated in FIG. A frame line (that is, a frame line including a region where a character on the captured image 300 may be present) 500 may be displayed on the captured image 300 as feedback information. Note that a frame 500 shown in FIG. 5 may be further displayed on the likelihood map image 400 described above. By displaying such a frame 500, the approximate size of the detected character string can be presented to the user. In FIG. 5, the frame line 500 has a rectangular shape, but the frame line 500 may have another shape (for example, a circular shape).

  That is, in the present embodiment, the footback information (information indicating the position of the recognition target) includes information indicating a partial area of the image, information on a frame line including the partial area of the image, and the like.

  Here, it is assumed that the character string region described above is detected by the character recognition processing unit 132.

  In this case, the second display processing unit 133 hides the likelihood map image 400 described above, and uses a frame line (hereinafter referred to as a circumscribed rectangle) representing the character string region (the circumscribed rectangle) detected by the character recognition processing unit 132 as feedback information. , Displayed as a frame line of the character string area).

  FIG. 6 shows an example of a display screen when a frame line of a character string area is displayed as feedback information.

  As shown in FIG. 6, the frame line of the character string area is displayed on the captured image 300, for example. In the example shown in FIG. 6, the frame line 601 of the character string region including the character string “XXX station” described on the signboard included in the captured image 300 and the character string region including the character string “turn right” are displayed. A frame line 602 is displayed.

  The frame lines 601 and 602 of the character string area shown in FIG. 6 are displayed on the display unit 14 (first area 141) until, for example, a character recognition result is acquired in the character recognition process described above.

  According to this, the user who uses the character recognition device 10 is that the character recognition process is being executed in the character recognition device 10 and that the region included by the frame lines 601 and 602 is detected as the character string region. Can be understood.

  In addition, according to the detection accuracy (certainty) of the character string area included by each of the frame lines 601 and 602 shown in FIG. 6, each display mode (for example, shape or color) of the frame lines 601 and 602 May be changed. As the detection accuracy of the character string region, for example, the total likelihood of the pixels included in the character string region among the likelihoods of the pixels constituting the captured image 300 can be used.

  FIG. 6 shows a state in which both the frame lines 601 and 602 are displayed. However, the frame lines 601 and 602 may be displayed when the detection of all the character string regions is completed. For example, each character string area may be sequentially displayed each time it is detected.

  In FIG. 6, the frame lines 601 and 602 have a rectangular shape, but the frame lines 601 and 602 may have other shapes (for example, a circular shape).

  Here, it is assumed that the character recognition process ends and a character recognition result (for example, text representing a character string included in the character string area) is acquired by the character recognition processing unit 132.

  In this case, the third display processing unit 134 displays the character recognition result acquired by the character recognition processing unit 132 while hiding the frame line of the character string area.

  FIG. 7 shows an example of a display screen when a character recognition result is displayed. As shown in FIG. 7, the character recognition result is displayed in the second area 142 provided on the display screen.

  In the example illustrated in FIG. 7, the character string “(representing text) of“ XX station this right turn ”” is displayed in the second region 142 as the character recognition result for the captured image 300.

  Here, the description will be made assuming that the text representing the character string included in the character string area is acquired as the character recognition result. However, when the translation result of the text (translated text) is acquired as the character recognition result The translation of the text is displayed in the second area 142. Note that both the text and the translation of the text may be displayed in the second area 142.

  In the present embodiment, the first area 141 and the second area 142 are described as being provided on the display screen. However, for example, the first area is displayed on the entire display screen until a character recognition result is acquired. 141 is provided, and the second region 142 may appear on the screen when the character recognition result is acquired (that is, the character recognition result is displayed).

  Note that the process of acquiring text representing a character string included in the character string area is sequentially executed for each character string area. Specifically, for example, after the text representing the character string included in the character string region represented by the frame line 601 illustrated in FIG. 6 is acquired, the character string included in the character string region represented by the frame line 602 is changed. The representing text is obtained.

  Here, in the example illustrated in FIG. 7, it is described that the frame lines 601 and 602 of the character string area are not displayed when the character recognition result is acquired, but this is the case where the character recognition result is acquired. However, the display of the frame lines 601 and 602 of the character string area may be maintained. In this case, as described above, for example, when the text representing the character string included in the character string region included by the frame line 601 illustrated in FIG. 6 is acquired, the frame line 601 is highlighted, and the frame line 602 When the text representing the character string included in the included character string region is acquired, the frame line 602 can be highlighted. Specifically, when the text representing the character string “XXX station” included in the character string region included by the frame line 601 is acquired, the frame line 601 is highlighted as shown in FIG. The Thereafter, when the text representing the character string “this turn right ahead” included in the character string area included by the frame line 602 is acquired, the frame line 602 is highlighted as shown in FIG.

  When a character string (representing text) displayed in the second area 142 is designated (selected) by the user, a frame line that includes the character string area corresponding to the designated character string is highlighted. It may be configured as described above. Specifically, when the character string “XX station” displayed in the second area 142 is designated, the frame line 601 is highlighted, while when the character string “this right turn” is designated. The frame line 602 is highlighted.

  When the character string “XXX station” displayed in the second area 142 is designated, only the frame line 601 is displayed (that is, the frame line 602 is not displayed). Good. On the other hand, when the character string “turn right ahead” is designated, only the frame line 602 is displayed (that is, the frame line 601 is not displayed).

  According to the configuration described above, the user can easily grasp the character string region corresponding to the character recognition result (that is, the character string region including the character string from which the text is acquired).

  In addition, the character string area is detected, but the character string included in the character string area is not a recognizable language (for example, the character string is a sequence of characters that are not recognizable language words). In this case, as shown in FIG. 10, a result indicating that the character (column) cannot be recognized is displayed in, for example, the second area 142.

  In addition, when the frame line including the character string region is displayed as described above, the frame line including the character string region that cannot recognize the character string is displayed in another manner such as a broken line. It does not matter if such a configuration is used. For example, it is possible to recognize a character string included in the character string region included by the frame line 601 described above, but not to recognize a character string included in the character string region included by the frame line 602. In such a case, the frame line 602 can be displayed as a broken line as shown in FIG. Here, a frame line 602 that includes a character string region in which a character string cannot be recognized is indicated by a broken line. However, the frame line 602 may have, for example, different colors (colors indicating that a character cannot be recognized). ) Or may be hidden.

  Note that the above-described processing illustrated in FIG. 2 is executed each time an image is captured by the camera 11 when the user presses a shutter button, for example. In such a case, for example, when a first image is captured by the camera 11, a character recognition result for the first image is displayed in the second area 142, and then the second image is displayed by the camera 11. When the image is captured, a character recognition result for the second image is displayed in the second area 142. Here, for example, when it is desired to confirm the character recognition result for the first image after the second image is captured, the first image is captured again by the camera 11 and the character recognition process is executed (or the first image is recognized). For example, it is necessary to acquire one image from the storage unit 12 and execute character recognition processing). This is very cumbersome for the user.

  For this reason, in the present embodiment, it is assumed that a history of character recognition results is displayed in the second area 142. Specifically, when the second image is captured after the first image is captured as described above, the second region 142 includes the character recognition result and the first image for the first image. Both character recognition results for the second image are displayed. When the character recognition process for another image is further executed, the character recognition result for the other image is added to the second area 142. In this case, in the second area 142, the first image (the thumbnail image thereof) is associated with the character recognition result for the first image, and the second image (the thumbnail image) is associated with the character recognition result for the second image. (Thumbnail image) may be further displayed.

  The first image and the second image described above may be still images as described above. For example, the first image and the second image constitute a moving image displayed in real time on the display unit 14 (first region 141). It may be an image.

  Further, when the character recognition process is performed on each of the images constituting the moving image displayed mainly on the display unit 14 in real time, the same character recognition result may be obtained in duplicate. In such a case, it is not useful for the user that the same character recognition result is displayed in the second area 142 as a history of character recognition results. For this reason, in this embodiment, when the same character recognition result as the character recognition result (text) acquired in the past is acquired, the said character recognition result is put together (that is, as one character recognition result). ) It shall be displayed in the second area 142.

  Instead of simply displaying the same character recognition results together, for example, based on information such as matching between images or an imaging target (that is, a subject) based on the position of the character recognition device 10 (camera 11), etc. Only the character recognition results for the same character string area (or captured images including the same subject) may be displayed together. Specifically, for example, the similarity of the plurality of images is determined by correlating the plurality of images whose sizes are normalized, and the character recognition results obtained from the plurality of images determined to be similar are collected together. It can be configured to display.

  Similarly, for example, it is configured such that the character recognition results for the same character string region are collectively displayed by determining the identity of the imaging target based on the information of the position of the character recognition device 10 and the orientation of the camera 11, for example. It is also possible.

  That is, in this embodiment, when the imaging target (the first recognition target and the second recognition target) included in each of the plurality of images is the same, any one of the character recognition results for each of the plurality of images. It can be set as the structure by which a display process (For example, the display process of the recognition result of the 1st recognition object) is performed.

  In the case of the configuration for determining the identity of the imaging target, the determination process may be executed before the character recognition process is executed. According to such a configuration, when the imaging target is the same, it is possible to omit the character recognition processing for the imaging target (character string region), and to reduce the processing amount in the character recognition device 10. It becomes possible.

  The position of the character recognition device 10 described above can be acquired by, for example, a GPS function provided in the character recognition device 10 (for example, a smartphone). Further, the orientation information of the camera 11 can be determined (detected) by an acceleration sensor or the like provided in the character recognition device 10, for example.

  In addition, since a plurality of character recognition results are displayed in the second area 142, a new character recognition result cannot be added (displayed) to the second area 142 (that is, a new character recognition result). In the second area 142), it may be deleted from the second area 142 in the order in which past character recognition results are acquired.

  In the present embodiment, it has been described that both the character recognition result for the image (captured image) displayed in the first region 141 and the history of the character recognition result are displayed in the second region 142. The area where the character recognition result for the image displayed in the first area 141 is displayed and the area where the history of the character recognition result is displayed may be provided separately.

  In the present embodiment, the history of character recognition results is described as being displayed in the second area 142. However, when a moving image is displayed on the display unit 14 in real time, the moving image is displayed. Only the character recognition result for the (image displayed in real time) may be displayed, and the history of character recognition results (that is, past character recognition results) may not be displayed. That is, it is also possible to adopt a configuration in which the display of the history of character recognition results and the non-indicating lamp can be switched in accordance with the type of image (a still image or a moving image) to be subjected to character recognition.

  Here, the character recognition result obtained by executing the character recognition process as described above is displayed in the second area 142, but in the present embodiment, the character recognition result is displayed in the second area 142. Character recognition results can be used for search processing.

  Hereinafter, an example of a processing procedure of the character recognition device 10 when the character recognition result is used for the search process will be described with reference to the flowchart of FIG.

  Here, it is assumed that the character recognition result is displayed in the second area 142 by executing the process shown in FIG. In this case, a user who uses the character recognition device 10 can perform an operation (hereinafter referred to as a designation operation) for designating the character recognition result displayed in the second area 142. When the character recognition device 10 includes a touch panel display, the designation operation includes, for example, an operation of touching a character recognition result displayed in the second region 142 with a finger.

  When such a designation operation is performed by the user, the search processing unit 135 included in the processing unit 13 acquires a character recognition result designated by the designation operation (step S11). Note that the character recognition result acquired in step S11 includes, for example, text representing a character string included in the captured image described above.

  Next, the search processing unit 135 executes a search process based on the character recognition result acquired in step S11 (step S12). In this search process, for example, a Web page (site) is searched via the Internet using a character recognition result (text) as a search keyword.

  The result of the search process in step S12 (hereinafter referred to as search result) is displayed on the display unit 14 (step S13). In this case, the search result is displayed in the second area 142, for example.

  Here, the process shown in FIG. 12 will be specifically described. Here, it is assumed that text representing “XX station this right turn” is displayed in the second area 142 as a result of character recognition for the captured image 300 described above.

  In this case, it is assumed that the user can perform a designation operation for designating at least one of a plurality of words constituting the text displayed in the second area 142, for example. In this case, for example, the user can designate (touch) the part of “XX station” as a search keyword.

  According to this, the search processing unit 135 searches the Web page using “XX station” as a search keyword. In this case, in the second area 142, for example, search results including a timetable (including a web page) of XX station are displayed in, for example, a thumbnail format. As a search result, for example, a predetermined number of Web pages having a higher degree of matching (score) with a search keyword may be displayed.

  The user designates such a search result (thumbnail) in the second area 142 to display, for example, a timetable (including a Web page) of the station XX (including the first area 141 or the entire screen). ) Can be displayed.

  In the present embodiment, the character recognition result may be a translation of a text representing a character string included in the captured image. When searching for a Web page using such a character recognition result (translated sentence), the search process may be executed based on the translated sentence, or the search process may be executed based on the text before translation. May be.

  Here, the description has been made assuming that the Web page is searched. However, for example, a configuration in which an application (program) installed in the character recognition device 10 is searched may be used. As described above, when “XX station” is designated as a search keyword, an application used for, for example, railway transfer guidance is searched, and the application is automatically started. Good. In addition, for example, an application for displaying a map (information) may be searched, and a map around the station designated as a search keyword may be automatically displayed.

  In addition to the application, for example, various contents stored in the storage unit 12 may be searched. When the image captured by the camera 11 is stored (accumulated) in the storage unit 12 as described above, an image from which the character recognition result designated by the user is acquired is retrieved from the storage unit 12. Then, it may be displayed on the display unit 14 (for example, the first region 141). In this case, the image and the character recognition result for the image may be associated with each other in the character recognition device 10 (storage unit 12).

  In addition, when the character recognition result is designated by storing position information indicating the position where the image is captured in the storage unit 12 in association with the image, the character recognition result is acquired together with the acquired image. The position where the image is captured can be displayed. According to such a configuration, for example, when a plurality of signboard images in which names of restaurants are written are captured, the name of the restaurant desired by the user is specified in the second area 142. In this case, it is possible to grasp the image of the signboard in which the name of the restaurant is described and the position of the signboard (position where the image is captured).

  Here, the description has been made on the assumption that the image is searched from the storage unit 12, but for example, a configuration in which various contents stored in an external server device and the like are searched may be employed.

  When the search result is displayed in the second area 142 as described above, for example, the search result can be sufficiently displayed in the range (size) of the second area 142 shown in FIG. It may not be possible. For this reason, when the above-described search process is executed (that is, the search result is displayed in the second area 142), the second area 142 (range) may be enlarged. In addition, when the second area 142 is enlarged, the first area 141 becomes smaller. In this case, for example, at least the character string area (a character string included in) is displayed in the first area 141. In this way, the display range of the captured image 300 may be adjusted.

  As described above, in the present embodiment, a character including a process for detecting a character string area from an image (first process) and a process for recognizing a character included in the detected area (second process). When the recognition process is executed, feedback information indicating that the character recognition process is being executed is displayed while the character recognition process is being executed, and the result of the character recognition process when the character recognition process ends. Is displayed. In the present embodiment, an area on an image that is estimated to have a character in the process of detecting a character string area (that is, a character seems to be present) is displayed as feedback information.

  Specifically, a likelihood map image in which a color corresponding to the likelihood of each pixel constituting the image (the degree to which there is a character) is given to the pixel or a region in which a character in the image may exist The included frame is displayed as feedback information.

  In the present embodiment, with such a configuration, it is possible to present to the user the processing status related to character recognition, so that the user can grasp that the character recognition processing for the image is being executed, For example, when the response time (waiting time) is long, it is possible to prevent the user from feeling. That is, in the present embodiment, it is possible to realize a display that improves user convenience with respect to image recognition.

  In the present embodiment, since the region (position) recognized by the character recognition device 10 when there is a character can be presented to the user, the user is appropriately performing the character recognition process ( That is, it is possible to grasp whether or not the character intended by the user is recognized. Specifically, for example, when a likelihood map image 400 as shown in FIG. 4 is displayed, the user can grasp that the position of the character included in the captured image 300 is properly recognized. it can. On the other hand, for example, when a likelihood map image that entirely becomes the region 401 is displayed, the user can grasp that the position of the character is not properly recognized. For example, the image is captured again. It is possible to perform an operation such as performing character recognition processing again.

  In the present embodiment, an example of feedback information has been described. However, the feedback information only needs to indicate that the character recognition process is being executed. For example, the feedback information may be displayed in a different manner. It does not matter. Specifically, as shown in FIG. 13, for example, an image in a state where mosaic processing has been performed on a region 700 that is a recognition target (a region where a character is likely to exist) may be displayed. As shown in FIG. 14, for example, a specific character string such as “processing in progress” (information regarding a character string indicating that the character recognition process is being executed) may be displayed on the area 700. Although not shown, it may be expressed that the character recognition process is being executed by changing (moving) the image.

  In the present embodiment, an image is displayed in the first area 141 on the screen of the character recognition device 10 (display unit 14), and the character recognition result (character recognition processing result) is displayed in the second area 142 on the screen. With the configuration for displaying (result), it is possible to improve the visibility of the character recognition result of the user.

  In the present embodiment, the second region 142 is described as being provided below the first region 141. However, the second region 142 is provided above the first region 141, for example. Alternatively, it may be provided on the left side or the right side of the first region 141. Further, the position of the second area 142 on the screen may be changed according to the position of the character (character string area) included in the image displayed in the first area 141. Absent. Specifically, for example, the position of the second area 142 may be changed so as to be provided in the vicinity of the character string area.

  In the present embodiment, the area where the image is displayed (first area 141) and the area where the character recognition result is displayed (second area 142) are described as being provided separately. An image may be displayed on the entire screen, and a character recognition result may be displayed on a character (column) included in the image.

  Furthermore, in the present embodiment, a character recognition process for recognizing characters included in each of a plurality of images (first image and second image) is executed, and a plurality of images are displayed in the second region 142. The result of the character recognition process (the first character recognition process and the second character recognition process) is displayed as a history. In the present embodiment, with such a configuration, for example, even when it is desired to confirm a character recognition result for an image captured by the camera 11 in the past, the user can capture the image without capturing the image again. The character recognition result can be easily confirmed.

  Further, in the present embodiment, when a plurality of character recognition results displayed as histories are the same as described above, the plurality of character recognition results are displayed together (that is, any of the plurality of character recognition results). With the configuration in which the process of displaying one of them is executed), it is possible to effectively use the area where the history is displayed (second area 142).

  In the present embodiment, when the character recognition result displayed in the second area 142 is designated by the user, an image from which the character recognition result is acquired is displayed. In the present embodiment, with such a configuration, the user can easily confirm an image from which the character recognition result has been acquired, for example, even if the image has been captured in the past, by specifying the character recognition result. Is possible.

  In the present embodiment, when the character recognition result displayed in the second area 142 is designated by the user, a search process using the character recognition result (text) is executed, and the search result (search process) Result). In the present embodiment, with such a configuration, the user can easily search (acquire) information related to the character recognition result, and thus the convenience for the user can be improved.

  As described above, in the present embodiment, various examples of display that improve user convenience with respect to image recognition (character recognition) have been described. However, the character recognition apparatus 10 according to the present embodiment is not limited to this embodiment. A configuration including at least one of the described display examples may be used, but a configuration in which some of the display examples are combined may be used.

  In the present embodiment, for example, an image including a signboard arranged in a city is described as being captured by the camera 11. However, the subject captured by the camera 11 is written with characters. For example, it may be a guide board, a menu, a product tag, a book, or the like, or a display on which characters are displayed, for example.

  Moreover, although the character recognition apparatus 10 which concerns on this embodiment was demonstrated as what is implement | achieved as portable electronic devices, such as a smart phone and a tablet computer, for example, the said character recognition device 10 is an electronic device provided with the camera 11 grade | etc., For example. You may implement | achieve as a server apparatus connected so that communication is possible.

  Moreover, although the character recognition apparatus which recognizes a character from an image was demonstrated in this embodiment, this embodiment may be applied when recognizing recognition objects other than a character from an image. Specifically, the present embodiment may be applied to an information processing apparatus (recognition apparatus) that recognizes a face, a person, a sign, a symbol, a moving object, or another object from an image. A known technique using statistical learning or the like can be used for recognizing the object or the like.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and the equivalents thereof.

  DESCRIPTION OF SYMBOLS 10 ... Character recognition apparatus (information processing apparatus), 11 ... Camera, 12 ... Storage part, 13 ... Processing part, 14 ... Display part, 131 ... 1st display processing part, 132 ... Character recognition processing part, 133 ... 2nd display Processing unit 134... Third display processing unit 135.

Claims (11)

First recognition means for recognizing a first recognition target included in the first image;
First, feedback information indicating that the recognition process is being executed is displayed before the recognition process in the first recognition unit is completed, and a recognition result of the first recognition target is displayed after the recognition process is completed. Display processing means;
Second recognition means for recognizing a second recognition target included in a second image different from the first image;
The feedback information indicating that the recognition process is being executed is displayed before the recognition process in the second recognition unit is finished, and the recognition result of the second recognition target is displayed after the recognition process is finished. Display processing means and
Comprising
The recognition result of the first recognition target and the recognition result of the second recognition target are displayed as a history of recognition results,
When the recognition result of the first recognition target is selected from the recognition result of the first recognition target and the recognition result of the second recognition target displayed as the history, the first image and the first image are captured. If the recognition result of the second recognition target is selected, the second image and the position where the second image was captured are displayed.
Information processing device. The feedback information displayed by the first display processing means includes information indicating the position of the first recognition target,
The information processing apparatus according to claim 1 , wherein the feedback information displayed by the second display processing unit includes information indicating a position of the second recognition target . Information indicating the position of the first recognition target is Ri information der showing a part region of the first image,
The information indicating the position of the second recognition target is information indicating a partial region of the second image.
The information processing apparatus according to claim 2 . Information indicating the position of the first recognition target is Ri information der encompassing border part region of the first image,
The information indicating the position of the second recognition target is information on a frame line that includes a partial region of the second image.
The information processing apparatus according to claim 2 or 3 . Said first information indicating the position of the recognition target and the information indicating the position of the second recognition target, the information processing apparatus according to any one of claims 2 to 4 which is the color information. Information indicating the position of the first recognition target is Ri information der related to the character string indicating that the recognition processing in the first recognition unit is running,
The information indicating the position of the second recognition target is information relating to a character string indicating that the recognition process in the second recognition unit is being executed.
The information processing apparatus according to claim 2 . Information indicating the position of the first recognition target is Ri information der based on the likelihood information of the detection in the first recognition unit,
The information indicating the position of the second recognition target is information based on detection likelihood information in the second recognition means.
The information processing apparatus according to claim 2. Before SL when the first recognition target and the second recognition target is the same object, the first recognition target recognition result and the second recognition target recognition result information processing apparatus according to claim 1 wherein the displayed together . The information processing apparatus according to any one of claims 1 to 8 , further comprising display means for displaying the feedback information and the recognition result . A program executed by a computer of an information processing device,
In the computer,
Recognizing a first recognition target included in the first image;
Displaying feedback information indicating that the recognition process is being executed before the recognition process for the first recognition object is completed, and displaying a recognition result of the first recognition object after the recognition process is completed; ,
Recognizing a second recognition target included in a second image different from the first image;
Displaying feedback information indicating that the recognition process is being executed before the recognition process for the second recognition target is completed, and displaying a recognition result of the second recognition target after the recognition process is completed; was executed,
The recognition result of the first recognition target and the recognition result of the second recognition target are displayed as a history of recognition results,
When the recognition result of the first recognition target is selected from the recognition result of the first recognition target and the recognition result of the second recognition target displayed as the history, the first image and the first image are captured. If the recognition result of the second recognition target is selected, the second image and the position where the second image was captured are displayed.
program. A method executed by an information processing apparatus,
Recognizing a first recognition target included in the first image;
Displaying feedback information indicating that the recognition process is being executed before the recognition process for the first recognition object is completed, and displaying a recognition result of the first recognition object after the recognition process is completed; ,
Recognizing a second recognition target included in a second image different from the first image;
Displaying feedback information indicating that the recognition process is being executed before the recognition process for the second recognition target is completed, and displaying a recognition result of the second recognition target after the recognition process is completed; equipped with a,
The recognition result of the first recognition target and the recognition result of the second recognition target are displayed as a history of recognition results,
When the recognition result of the first recognition target is selected from the recognition result of the first recognition target and the recognition result of the second recognition target displayed as the history, the first image and the first image are captured. If the recognition result of the second recognition target is selected, the second image and the position where the second image was captured are displayed.
Method.

Images